Method for encoding input signal using neural network and corresponding device

Abstract

The invention relates to a method for encoding an input signal, the method comprising: encoding a potential representation of the input signal by modifying a first neural network encoding potential representation of the input signal obtained from a neural network encoder whose coefficients are frozen by using a metric based on the following, obtaining at least a second potential representation of the input signal:-the input signal,-the first neural network encoded potential representation, and-a decoded signal obtained from decoding of the first neural network encoded potential representation of the input signal using a neural network decoder whose coefficients are frozen.

Description

Technical Field

[0001] The present invention relates to the field of signal encoding and decoding using neural networks. Background Art

[0002] Digital technology is becoming more and more important in everyday life, especially in video streaming. The use of video streaming is increasing, so its environmental impact has become an important issue. Standardization organizations such as MPEG and ITU are committed to reducing the impact of video streaming and have released several video coding standards to reduce the size of the video while maintaining acceptable visual quality. Neural-based (NN) encoders and decoders have been introduced in video encoding and decoding processes and provide improved performance, enabling the amount of streaming data to be reduced, but their use in video codecs remains challenging in terms of configuration. Several proposed solutions for introducing neural networks in video compression still lack compression efficiency. In fact, in real-world video communication applications, encoders are often required to apply specific arbitrary rate-distortion optimization (RDO) strategies based on different factors. The factors that determine what RDO strategy to adopt can be related to signal characteristics, encoder device characteristics, bandwidth constraints, etc. In general, the goal of the RDO algorithm is to minimize the compressed signal bit rate at a given quality level. Or in other words, to improve the compressed signal decoding quality at a given bit rate level.

[0003] The functionality of a typical encoder (as opposed to a neural network encoder) is usually flexible enough to allow a certain degree of freedom for applying RDO strategies, as well as choosing between alternative compressed representations of the same input signal. However, with a neural network encoder, the process of compressing a video signal with the encoder, given a pre-trained encoder-decoder pair, is performed by a mapping from the pixel domain to the latent domain. This mapping is as unconstrained as using a lookup table, where, in order to find out what latent domain point corresponds to a given pixel domain input, the input should be passed through the layers of the NN encoder whose internal parameters have been trained. Therefore, the current solution to apply different mappings for the same input video is to retrain the NN encoder-decoder pair considering the training loss function corresponding to the desired strategy. However, this process is expensive in terms of time and resources, and given the high diversity of possible RDO strategies, retraining for all cases is not feasible.

[0004] The present disclosure proposes to address at least one of these disadvantages. Summary of the invention

[0005] In this context, the present disclosure proposes a method for encoding an input signal, the method comprising

[0006] - obtaining at least a second latent representation of the input signal by modifying a first neural network encoded latent representation of the input signal obtained from a neural network encoder with frozen coefficients, using a metric based on:

[0007] - the input signal,

[0008] - the first neural network encodes a latent representation, and

[0009] - a decoded signal obtained from decoding a first neural network encoded latent representation of the input signal using a neural network decoder with frozen coefficients.

[0010] The present disclosure also relates to an apparatus for encoding an input signal, the apparatus comprising one or several processors, which are configured, alone or in combination, to:

[0011] - obtaining at least a second latent representation of the input signal by modifying a first neural network encoded latent representation of the input signal obtained from a neural network encoder with frozen coefficients, using a metric based on:

[0012] - the input signal,

[0013] - the first neural network encodes a latent representation, and

[0014] - a decoded signal obtained from decoding a first neural network encoded latent representation of the input signal using a neural network decoder with frozen coefficients.

[0015] By means of the disclosed method and apparatus, an input signal is encoded using a neural network encoder and a neural network decoder, and a metric is then applied to the input signal. The metric is applied using a first encoded latent representation and also applied to a decoded signal obtained after decoding the first latent representation, the first latent representation being obtained after encoding the input signal by an encoder, the decoding being performed by the decoder. Typically, but not exclusively, the encoder and decoder may have been trained together. The neural network encoder and the neural network decoder have coefficients that are frozen, or fixed or determined, and are not modified by the present method.

[0016] By means of this, a specific RDO strategy can be applied to an input signal. The specific RDO strategy is defined by means of a metric. Therefore, a metric that depends on the input signal, or whose parameters depend on the input signal, can be applied. The decoder parameters (also called weights or coefficients) are determined (or frozen), and the decoder is then used with these parameters. The encoder with frozen coefficients ultimately provides a single potential representation of the input signal because the single potential representation is obtained after encoding with predetermined coefficients. When the encoder is trained on a general data set, the first potential representation cannot achieve the obtained encoded signal according to the predetermined RDO strategy as trained. The present disclosure provides an alternative potential representation with a specific RDO strategy without retraining either the encoder or the decoder, thereby providing an efficient and fast encoding method.

[0017] According to some embodiments, a first latent representation and a second latent representation are calculated to be decoded by a neural network decoding method that is typically trained together with a neural network encoding method used to encode the first latent representation.

[0018] According to some embodiments, obtaining the at least second potential representation comprises an iterative process comprising at least two iterations, wherein:

[0019] - for a first iteration step, the iteration process receives as input: a first latent representation of an input signal, the input signal, and the decoded signal obtained from decoding the first latent representation, and wherein,

[0020] - For the next iteration step, the first latent representation is replaced by the output of the previous iteration step.

[0021] According to some embodiments, the metric is obtained by computing at least a post-processing loss function through the iterative process, wherein the second latent representation is obtained as an output that minimizes the post-processing loss function.

[0022] According to some embodiments, the at least first neural network encoded representation of the input signal is obtained by an encoding method obtained after a training step using a training loss function, wherein the training loss function and the post-processing loss function are the same function.

[0023] According to some embodiments, the at least first neural network encoded representation of the input signal is obtained by an encoding method obtained after a training step using a training loss function, and the iterative process calculates at least one processing loss function different from the training loss function.

[0024] According to some embodiments, the iterative process calculates the post-processing loss function using different parameters for parts of the input signal.

[0025] By means of this particular embodiment, a particular strategy may be applied to at least some parts of the input signal. This enables improved encoding of some parts of the input signal. For example, different rate-distortion strategies may be applied to different parts of the signal.

[0026] According to some embodiments, at least a first neural network encoded representation of the input signal is obtained by an encoding method that calculates a training loss function, and at least some of the parameters used by the at least one post-processing loss function are different from the parameters of the training loss function.

[0027] According to some embodiments, the input signal represents an image or video signal, and the at least two parts of the input signal, at least one of the parts, represent a region of interest.

[0028] In this particular embodiment, more bits may be dedicated to encoding one or more regions of interest to improve the user experience when viewing or acquiring content. This may also be advantageous for broadcast content, and adapted to particular content such as gaming or sports, for example when the input signal represents video showing a lot of movement.

[0029] According to some embodiments, the training loss function and the post-processing loss function are functions of a distortion factor and a rate factor multiplied by a rate-distortion coefficient, and the parameters of the loss function and the processing function differ in terms of their rate-distortion coefficient values.

[0030] According to some embodiments, the training loss function and the post-processing loss function are functions of a distortion factor and a rate factor multiplied by a rate-distortion coefficient, and the loss function and the processing function parameters differ in their distortion factors.

[0031] According to some embodiments, the post-processing loss function receives as input an auxiliary signal, the auxiliary signal comprising information representing parts of the input signal.

[0032] According to some embodiments, the information included in the auxiliary signal is selected from at least one of the following:

[0033] a. a distortion mask comprising weights associated with at least some pixels of the input signal,

[0034] b. a spatial identity of at least one set of pixels of the input signal,

[0035] c. A patch to be applied to at least a portion of the decoded input signal.

[0036] According to some embodiments, the auxiliary signal is a distortion mask and the post-processing loss function calculates the distortion factor based on a pixel-wise multiplication of the distortion mask and a difference between the input signal and the corresponding decoded signal.

[0037] According to some embodiments, the auxiliary signal is a patch, the post-processing loss function calculates a superimposed signal based on the decoded signal and the patch, and calculates a distortion factor based on a difference between the calculated superimposed signal and the decoded signal.

[0038] According to another aspect, the present disclosure also relates to a computer program product, which comprises instructions, which, when the program is executed by one or more processors, cause the one or more processors to perform the method according to the present disclosure.

[0039] According to another aspect, the present disclosure also relates to a computer-readable storage medium having instructions stored thereon, the instructions being used to cause one or more processors to execute the method according to the present disclosure.

[0040] According to another aspect, the present disclosure also relates to a method for encoding an input signal, the method comprising

[0041] - obtaining at least a second latent representation of said signal by modifying a first neural network encoded latent representation of said input signal obtained from a neural network encoder with frozen coefficients, by using a metric based on:

[0042] - the input signal,

[0043] - random signals, and

[0044] - A decoded signal obtained from decoding the input signal, the encoded input signal being encoded by the neural network encoder, the decoding being obtained from a neural network decoder with frozen coefficients.

[0045] According to this specific aspect of the disclosure, there is no need to provide the method with a first latent representation. Instead, a random signal is used. With this embodiment, no encoder is required to post-process the input signal, since the first latent representation is no longer used. A decoder-only decoder is used to decode the post-processed signal.

[0046] According to another aspect, the present disclosure also relates to a device for encoding an input signal, the device comprising one or more processors, which are configured individually or in combination to

[0047] - obtaining at least a second latent representation of the input signal by modifying a first neural network encoded latent representation of the input signal obtained from a neural network encoder with frozen coefficients, using a metric based on:

[0048] - the input signal,

[0049] - random signals, and

[0050] - A decoded signal obtained from decoding the input signal, the encoded input signal being encoded by the neural network encoder, the decoding being obtained from a neural network decoder with frozen coefficients.

[0051] Based on another aspect, the present disclosure also relates to a signal comprising a bitstream comprising encoded data representing a second latent representation of a signal, the second representation being obtained by modifying a first neural network encoded latent representation of the input signal obtained from a trained neural network encoder using a metric based on:

[0052] - the input signal,

[0053] - the first neural network encodes a latent representation, and

[0054] - a decoded signal obtained from decoding a first neural network encoded latent representation of said input signal, the decoded signal being obtained from a neural network decoder trained together with said trained neural network encoder.

[0055] These and other aspects, features and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments, which is to be read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A general overview of a first embodiment of a system including a neural network encoder and decoder is presented.

[0057] Figure 2 A general overview of encoders and decoders according to some embodiments of the present disclosure is presented.

[0058] Figure 3 A first implementation of distortion calculation according to some embodiments of the present disclosure is presented.

[0059] Figure 4 A second embodiment of distortion calculation according to some embodiments of the present disclosure is presented.

[0060] Figure 5 Implementations of distortion calculation of superimposed signals according to some embodiments of the present disclosure are presented.

[0061] 6a and 6b illustrate implementations in which the post-processing loss function uses different parameters for some parts of the input signal.

[0062] Figure 7 An embodiment of the present disclosure using a random signal is shown.

[0063] Figure 8 An example of a hardware architecture for an encoder according to some embodiments of the present disclosure is shown.

[0064] Fig. 9 A general overview of a second embodiment of a system including a neural network encoder and decoder is presented.

[0065] Fig.10 Implementations of distortion calculations according to some embodiments of the present disclosure are presented. DETAILED DESCRIPTION

[0066] Figure 1 An overview of both an encoder 100 and a decoder 200 for encoding and decoding an input signal x is shown. A signal may be understood as any signal that is encoded (or compressed) for transmission in order to reduce the size of the data to be transmitted. Thus, a signal may be understood as an image, a video signal, and an audio signal, a combination of a video signal and an audio signal, or other signals, for which compression would be advantageous in gaining, for example, transmission bandwidth or storage capacity.

[0067] The following description will refer to encoding and decoding of images or videos, but this is given as an illustrative input signal and should not be limited thereto.

[0068] A system for providing a neural network encoding and obtaining coefficients for a neural network encoder and decoder can be found in Ballé, Johannes, Valero Laparra, and Eero P. Simoncelli, “End-to-end optimized image compression,” arXiv:1611.01704 (2016).

[0069] like Figure 1As shown, both the encoder 100 and the decoder 200 are neural network based encoders and decoders. The neural network 101 for encoding transforms the input signal from the pixel domain into a domain represented by the neural network latent variables. The transformation is followed by a quantization step 102 and an entropy encoding step 103. In turn, the neural network 201 for decoding performs a transformation of the neural network latent variables to pixels. Before the inverse transformation, entropy decoding 203 and inverse quantization 202 are performed. Typically, a pair of neural networks for encoding and neural networks for decoding are jointly trained.

[0070] The neural network encoder 101 and the neural network decoder 201 are each defined by a structure including, for example, a plurality of neuron layers and / or a set of weights respectively associated with the neurons of the networks involved. Later in the description, when referring to the neural network structure or parameters, weights or coefficients may be used indiscriminately.

[0071] A representation (e.g. a two-dimensional representation) of the current image x (or alternatively, a component or block of the current image x) is applied as input (i.e., at a layer at the input end (i.e., at an input layer)) to the artificial neural network of the encoding neural network 101. The encoding artificial neural network 101 then produces data (in this case, a data unit) at the output end.

[0072] The data (here, data units) are then applied as input to the decoding neural network 201. The decoding neural network 201 then generates a signal corresponding to the current image x (or alternatively, a component or block of the current image x). A representation (e.g., a two-dimensional representation) of is given as output.

[0073] The encoding neural network 101 is designed so that a data unit contains a smaller (at least smaller on average) amount of data than the previous representation of the current image x. In other words, the data in the data unit is compressed.

[0074] According to the neural network coding, the NN encoder 101 generates a first potential representation of the signal x, called z. Then, the latent variable z can be quantized into a quantization index I, and the index can be entropy encoded to form a bit stream B representing the input signal. The bit stream can be stored or transmitted. However, for the sake of simplicity, we will not describe the quantization step and the entropy encoding step here. Therefore, the encoder 100 can include a quantizer 102 and an entropy encoder 103, and the decoder 200 can include an entropy decoder 203 and a dequantizer 202.

[0075] As mentioned before, NN-based encoders and decoders are defined by a structure comprising, for example, multiple neuron layers and / or a set of weights respectively associated with the neurons of the network in question. A neural network architecture is defined by the number, size, type and interactions of its NN layers.

[0076] In some embodiments, the function of each neuron consists of a linear operation on its input, followed by a nonlinear activation operation. In the most advanced applications of NN encoding and decoding, especially in image and video processing, a more complex type of network layer, called a convolutional layer, is often used. In its simplest form, a convolutional layer is a rectangular filter that slides over its two-dimensional input signal and applies a convolution operation at each sliding position to generate its output layer.

[0077] A learning-based end-to-end encoder-decoder (i.e., a pair of NN encoder and NN decoder) is defined by a NN encoder and a NN decoder that are usually jointly optimized. The latent variable z is the common link between these NN encoders and decoders, which is both the output of the encoder and the input of the decoder.

[0078] Furthermore, the encoding neural network 101 and the decoding neural network 201 are pre-trained to minimize the difference between the input representation of the current image x and its output representation x̂, while also minimizing the amount of data transferred between the encoding neural network 101 and the decoding neural network 201.

[0079] In view of this, the NN encoder 101 and the NN decoder 201 are trained on a set of input signals. The NN encoder and the NN decoder obtained after this training can be regarded as universal NN encoders and decoders. Here, the term universal means that these encoders are not dedicated to encoding some specific signals. For example, they are not trained to encode the region of interest of the image differently from other regions.

[0080] The training of the NN encoder 101 and the NN decoder 201 may be performed jointly or not.

[0081] Once the neural network architecture is determined, the parameters are calculated in order to optimize the NN encoding and decoding. Typically, gradient descent and back propagation are used. A training loss function is calculated using the input signal and the decoded signal as input. The training of the neural network consists of several iterations, in which an input data set consisting of, for example, several images is input and used to obtain the most suitable coefficients for the NN encoder and decoder. The iterative process is performed to improve the overall performance of the neural network with respect to the previously defined loss function.

[0082] make Represents the set of all NN model parameters of the encoder and decoder, which is composed of the encoder and decoder Given such model parameters, for the input Encode to obtain the latent representation The process is expressed as Correspondingly, for the potential representation Decode to get the representation The process of (or reconstructing the signal) is expressed as In addition, once we calculate , you can use any entropy encoder engine (denoted as ) so that Write the bitstream and calculate its bit rate, which is .

[0083] The training loss function can be specified as:

[0084] [Mathematical formula 1]

[0085] [Mathematical formula 2]

[0086] Among them, the function "dist" calculates the distortion between the input signal and its corresponding decoded signal, and the function "rate" calculates the code rate of the underlying representation.

[0087] make represents the set of all NN model parameters of the NN encoder and decoder at the nth iteration. In other words, it consists of the model parameters of the encoder (enc) and the decoder (dec). Then

[0088] [Mathematical formula 3]

[0089] in represents the learning rate, represents the gradient, and X represents the training set of signal x.

[0090] Once trained, an optimized set of codec parameters is obtained (denoted as ). For the actual encoding, each input to the NN encoder 101 is encoded to obtain a latent representation z for further decoding using the NN decoder 201. Each input signal is encoded using a NN encoder, the parameters of which have been trained once. Therefore, the parameters of the NN encoder are not modified during the actual encoding phase or the actual decoding phase (i.e., once the parameter optimization is completed).

[0091] Compared to conventional encoders, no adaptation of the codec's tools and parameters is done to apply rate-distortion optimization while encoding. For an input signal x, only one potential representation z can be provided, thus avoiding any possibility to apply a specific RDO strategy depending on the input signal characteristics.

[0092] Figure 2 An embodiment of a system including a decoder and an encoder as proposed in the present disclosure is presented.

[0093] The present disclosure proposes a method for obtaining at least another potential representation of an input signal, which at least another potential representation represents an alternative to the first potential representation without retraining the codec 101 / 201, and which potential representation can also be decoded by the trained decoder 201. Therefore, the following encoding device or method enables the encoding / decoding framework to adapt to the content to be encoded and decoded.

[0094] The system according to the present disclosure may include at least an encoder 100, a decoder 200 and a post-processing function 300. It may be implemented as including Figure 2 The encoder 100, decoder 200, and post-processing function 300 may be implemented individually or in combination in a single integrated circuit, multiple ICs, and / or discrete components. In some embodiments, multiple components may also include a communication bus or interface for interacting with additional devices. In various embodiments, Figure 2 The system described in is configured to implement one or several aspects of the present disclosure.

[0095] The main functions of the encoder and decoder are not described in detail in the present disclosure because they represent well-known encoding steps and include, for example, intra-frame prediction or inter-frame prediction, motion estimation and compensation, prediction residual calculation, quantization and entropy coding... The same is true for the decoder function, which also includes: decoding the entropy coded bitstream to obtain transform coefficients, motion vectors and other coding information, dequantizing the coefficients, and decoding the prediction residual to combine with the prediction block to reconstruct the image.

[0096] According to some embodiments, the present disclosure provides a method and apparatus for encoding an input signal. The method includes:

[0097] - obtaining at least a second latent representation of the input signal by modifying a first neural network encoded latent representation of the input signal obtained from a neural network encoder with frozen coefficients, using a metric based on:

[0098] - Input signal,

[0099] - the first neural network encodes the latent representation, and

[0100] - a decoded signal obtained from decoding a first neural network encoded latent representation of the input signal, the decoded signal being obtained from a neural network decoder with its coefficients frozen.

[0101] The coefficients of the NN encoder and the coefficients of the NN decoder are frozen and therefore, there is no specific need for the encoder to be available, only a first latent representation of the input signal.

[0102] According to some embodiments, a first latent representation and a second latent representation are calculated to be decoded by a neural network decoding method that is trained together with a neural network encoding method used to encode the first latent representation.

[0103] The encoder 100 includes a NN encoder 101, and the decoder 200 includes a NN decoder 201. In some embodiments, they may have been trained together as a pair. In some embodiments, they may not be trained together.

[0104] They have been trained using a generic, non-specific training data set. Once trained, the coefficients of the NN encoder 101 and the NN decoder 201 are set and no longer modified; these coefficients define the NN encoder and decoder and can, for example, be stored or considered frozen and no longer modifiable. Thus, with the aid of the NN encoder 101 or encoder 100, only one potential representation of the input signal x is generated.

[0105] According to this embodiment, the post-processing function 300 is used to generate at least one additional potential representation of the signal x . is used by the decoder 200 to decode the signal x and obtain the decoded signal potential signal.

[0106] As previously described, the post-processing function 300 receives a first latent representation , the input signal x and the corresponding decoded signal as input.

[0107] In some embodiments, the post-processing function 300 may include one or several iterations, each iteration providing another potential representation of the input signal. A potential representation is decodable if its corresponding decoder model is able to process the potential representation through the layers of the decoder model and generate a pixel domain reconstruction of the potential representation. There are typically more than one potential representation of the input signal that can be decoded by a given decoder. Therefore, the learning-based end-to-end codec is free to decide which potential representation to select, as long as the selected potential representation is decodable.

[0108] According to some embodiments, when the post-processing function 300 comprises an iterative process (comprising at least two iterations), wherein,

[0109] - for a first iteration step, the iteration process receives as input: a first latent representation of an input signal, the input signal, and the decoded signal obtained from decoding the first latent representation, and wherein,

[0110] - For the next iteration step, the first latent representation is replaced by the output of the previous iteration step.

[0111] As in Figure 2 As can be seen from the above, for iteration i, the post-processing function 300 receives the input signal And output signal .Signal is the latent representation decoded by decoder 200 .

[0112] According to some embodiments, the input signal It can be:

[0113] - first potential representation,

[0114] - The output of the previous iteration of the post-processing function. This can be a previous iteration or the previous iteration.

[0115] According to some embodiments, an iterative process calculates a post-processing loss function. Figure 3 , Figure 4 , Figure 6 and Figure 7 An embodiment of this post-processing loss function is described.

[0116] In some embodiments, as described in reference Figure 3 As described, the post-processing loss function It can be specified as follows:

[0117] [Formula 4]

[0118] In other embodiments (such as Figure 4 In the embodiments described above, but not limited to these), the distortion function may be different and additional signals may be considered.

[0119] As previously described, a first latent representation of the input signal has been obtained by an encoding method that computes a training loss function. According to some embodiments, the parameters used for the post-processing loss function are the same parameters as the parameters of the training loss function. The advantage of this method is that it allows encoding to find an alternative latent representation that outperforms the latent representation obtained by the network with respect to the same metric / loss function that has been used in training. In other words, this allows an RDO-like search in the space of all latent variables to improve compression efficiency, as is usually done in conventional codecs.

[0120] or

[0121] As previously described, a first latent representation of the input signal has been obtained by an encoding method that computes a training loss function. According to some embodiments, at least some of the parameters used by the post-processing loss function are different from the parameters of the training loss function.

[0122] In such an implementation:

[0123] or

[0124] or: and

[0125] or: and

[0126] According to some embodiments, the training loss function And the post-processing loss function is a function of a distortion factor and a rate factor multiplied by a rate-distortion coefficient, wherein the parameters of the loss function and the processing function differ in terms of their rate-distortion coefficient values.

[0127] In this example:

[0128]

[0129] According to some embodiments, the training loss function And the post-processing loss function is a function of a distortion factor and a rate factor multiplied by a rate-distortion coefficient, wherein the parameters of the loss function and the processing function differ in terms of their rate-distortion coefficient values.

[0130] In this example:

[0131]

[0132] According to some embodiments, the post-processing function minimizes a post-processing loss function. The post-processing loss function can be expressed as , so that the post-processing function is consistent with iteration i and the input at iteration i Since the post-processing operates with fixed and pre-optimized encoder and decoder models, denote the parameters of such a pre-optimized model.

[0133] Post-processing loss function yes Then, the gradient can be calculated and expressed as .

[0134] Minimization of the post-processing loss function is performed by utilizing back-propagation and gradient descent of the post-processing loss function. Output signal Obtained as follows:

[0135] [Formula 5]

[0136] in represents the learning rate and Represents the gradient.

[0137] In some embodiments, Can be adjusted according to the selected minimization speed.

[0138] In some embodiments, the post-processing loss function may be modified from one iteration to the next, or at least in some iterations. In view of this, the distortion function may be modified from one iteration to another. Rate Distortion Coefficient .

[0139] Figure 3 and Figure 4 An embodiment of a method for calculating a post-processing loss function is presented.

[0140] Figure 3 A first embodiment of a method for calculating a post-processing loss function is presented. In this embodiment, in step E1, a distortion between an input signal and a first potential representation is calculated. The first potential representation may be:

[0141] - pass Figure 1 The encoder obtains a latent representation of the signal x, which corresponds to the encoding without further post-processing,

[0142] - Output of the previous iteration of the post-processing function ,

[0143] - The output of a previous iteration of the post-processing function , where j < i

[0144] The distortion is calculated by the following formula:

[0145] [Mathematical formula 6] ,in express The decoded version of .

[0146] In step E2, As input, we perform the bitrate calculation and obtain .

[0147] Then, in step E3, the post-processing loss function is calculated using the following formula:

[0148] [Formula 7]

[0149] In step E4, gradient descent and back propagation are used to minimize the loss function according to the following formula:

[0150] [Mathematical formula 8]

[0151] As reference Figure 2 As mentioned, the post-processing loss function can be the same as the training loss function or can be different. In view of this, we can have:

[0152] ( or )or

[0153] ( or )or

[0154] and )or

[0155] ( and or

[0156] and or

[0157] and

[0158] Figure 4 A second implementation of a method for calculating a post-processing loss function is presented.

[0159] Figure 4 More specifically, the computation of the post-processing function is addressed, where an auxiliary signal is introduced in order to apply a rate-distortion optimization (RDO) strategy specific to the input signal content, without the need to retrain the NN encoder 101 and the NN decoder 201 to apply a specific RDO strategy. The auxiliary signal can be:

[0160] - a distortion mask comprising weights associated with at least some pixels of the input signal,

[0161] - a spatial identity of at least one set of pixels of the input signal,

[0162] - a patch to be applied to at least a portion of the decoded input signal.

[0163] The auxiliary signal Aux may be transmitted as metadata, for example in a SEI (Supplemental Enhancement Information) message as used by some compression standards.

[0164] In view of this, in step S1, an auxiliary signal "Aux" can be introduced for calculating the distortion. In this embodiment, the distortion is calculated according to the following formula:

[0165] [Mathematical formula 9]

[0166] in express The decoded version of Represents a pixel-level matrix multiplication operation.

[0167] The code rate calculation step S2 may be the same as the code rate calculation step E2 and will not be repeated here.

[0168] The loss calculation step S3 can be the same as the loss calculation step E3, where the distortion The value of is the value calculated in step S1.

[0169] The back-propagation step S4 may be the same as the back-propagation step E4, wherein the loss function value is the loss function value calculated at step S3.

[0170] exist Figure 4 In this embodiment, the "Aux" signal can be a mask, such as a pixel-level region of interest mask M. The mask has an effect on the decoded signal For example, if a region of interest is detected in an input signal (e.g., an image or video), a weight can be applied to the region of interest. , and can apply weights to one or more other regions ,in The mask consists of the following original image: each pixel value of the ROI can be multiplied by the weight And each pixel value of non-ROI can be multiplied by the weight , and then the distortion value is:

[0171] [Formula 10]

[0172] in express The decoded version of Represents a pixel-level matrix multiplication operation.

[0173] exist Figure 4In another embodiment of the invention, the "Aux" signal may be an array containing information for identifying / locating different regions of the image and their associated distortion weights. In this embodiment, the distortion calculation may be performed by processing the identifiable regions one by one, calculating the weighted distortions of these regions, and progressively calculating the total weighted distortion.

[0174] exist Figure 4 In another embodiment of the Aux signal may be a patch that may be superimposed on the input signal x. It is known to add, for example, some logos on an image, where for example when the decoded picture is intended to be displayed (perhaps after broadcast), the appearance of the logo after decoding should avoid any decoding artifacts. Figure 5 An example is given above, which shows how to apply a patch to the decoded signal to obtain a superimposed signal. .

[0175] With the help of this post-processing function, when the patch is superimposed on the decoded input signal (for iteration i of the post-processing function, the superimposed signal is given ), this embodiment improves the appearance of the input signal after decoding. In view of this, the "Aux" signal can be used as a reference signal for calculating the distortion. The distortion calculated at step S1 becomes as follows:

[0176] [Mathematical formula 12] ,in express The decoded version of Represents a decoded image with overlay.

[0177] According to this embodiment, step S2, step S3 and step S4 remain unchanged, except that step S3 uses the distortion calculated by equation [Math. 12].

[0178] FIG. 6a and FIG. 6b illustrate an embodiment in which the post-processing loss function uses different parameters for some parts of the input signal. In other words, parts of the input signal (more precisely, the input image) can be identified. By means of the parameters, the distortion, bit rate or rate-distortion coefficient can be understood. Using different parameters enables different rate-distortion optimization strategies to be applied to different parts of the image. Thus, the rate-distortion strategy is adapted to the content of the input signal.

[0179] It should be noted that these parts, when aggregated, may represent the complete image. For these parts, each of these parts, some of these parts, the loss function parameters may be adapted according to one or several criteria. An example of a split of the parts may be: one part represents the foreground and one part represents the background. According to another example, faces may each be grouped into one part, and the other parts may be grouped into a second part. In other embodiments, each face may represent one part.

[0180] FIG. 6a shows an embodiment of calculating at least two partial loss functions. According to this embodiment, a rate-distortion strategy can be applied according to the content encoded by the input signal. More specifically, it is conceivable to apply different rate-distortion strategies on at least two different parts of the input signal. This achieves spatial code rate allocation.

[0181] When the input signal represents an image, a still image, or an image of a video signal, these parts may represent one or several regions of interest of the image. Therefore, by means of the present disclosure, different rate-distortion strategies may be applied to different parts of an image without the need to train a pair of encoders / decoders 101 and 201 more than once.

[0182] According to this embodiment of the present disclosure, multiple parts are defined in the input image. These parts represent content to which a dedicated RDO strategy can be applied in order to improve the coding efficiency of these parts, which can be regions of interest and, for example, require a higher bit rate than other regions of lower interest. In some embodiments, different parts can represent the background and foreground of the image.

[0183] The input signal (more specifically but not limited to the input image) can be divided into a number j of parts, for example, representing spatial regions of the image. The identification of these parts can be done by using a signal (e.g., an auxiliary signal as previously described), or can be encoded into the signal x (e.g., in a designated header of the signal x).

[0184] Therefore, it can be said that:

[0185] [Mathematical formula 13] , where j is the number of parts.

[0186] In step z1, a distortion is applied to a portion x of the image jFor some of the j loss functions, they may have the same parameters, such as, for example, the same distortion function or the same rate-distortion coefficients. There may be as many distortion steps / functions as there are parts in the image, here j functions. However, if some of the j parts may experience the same distortion due to, for example, similarity of their content or for some other reason, then less than j steps / functions z1 may be needed. The distortion may be calculated as

[0187] [Formula 14]

[0188] in Represents x j potential representation of, and express The decoded version of .

[0189] In step z2, a partial rate calculation is performed. As described with respect to step z1, there may be as many rate calculation steps / functions as there are parts in the image, here j functions. However, if some of the j parts may be subject to the same rate calculation due to, for example, similarity of their content or for some other reason, then less than j steps / functions may be needed.

[0190] In step z3, partial loss calculation is performed on the following parts of the image : At least some distortion or some rate calculation or both have been calculated for these parts. Given this, a partial loss calculation can be performed, such as:

[0191] [Mathematical formula 15]

[0192] Once j loss functions are calculated in step z3, in step z4, the global loss function is obtained as the fusion of the partial loss functions:

[0193] [Formula 16]

[0194] In step z5, back propagation is further performed using the aggregated loss function to minimize the loss function.

[0195] In an alternative, one function (eg a piecewise function) may be used to define at least two different rate-distortion strategies to be applied depending on the content encoded by the input signal. This alternative is illustrated in Figure 6b.

[0196] In this embodiment, a distortion calculation is applied to each portion of the image, where the distortion is calculated as a fusion of multiple partial distortions calculated on a partial level. There may be as many or fewer distortion functions as defined portions.

[0197] [Mathematical formula 17]

[0198] [Mathematical formula 18]

[0199] The post-processing function is then calculated using this distortion and the rate calculation calculated at step U2, which is the same rate calculation as that calculated for the image.

[0200] [Mathematical formula 19]

[0201] Another aspect of the present disclosure will now be disclosed, relating to a method for encoding an input signal, the method comprising

[0202] - obtaining at least a second latent representation of said signal by modifying a first neural network encoded latent representation of said input signal obtained from a neural network encoder with frozen coefficients, by using a metric based on:

[0203] - the input signal,

[0204] - random signals, and

[0205] - A decoded signal obtained from decoding the input signal, the encoded input signal being encoded by the neural network encoder, the decoding being obtained from a neural network decoder with frozen coefficients.

[0206] According to this aspect of the disclosure, there is no need to provide a first input latent representation for computing the post-processing function. The corresponding operation using a random signal may be computationally more complex which may make the minimization of the loss function.

[0207] In this aspect of the invention, the post-processing function as previously described uses a random signal for the first iteration i. Instead of using the first potential representation As input, the random signal has the same potential representation as the first Same size.

[0208] For the next iterations, i > 1, the random signal It can be replaced by the output of the loss function of the previous iteration.

[0209] According to this aspect, the metric is obtained by computing at least a post-processing loss function through the iterative process, wherein the second latent representation is obtained as an output that minimizes the post-processing loss function.

[0210] Figure 7 Implementation of this aspect of the present disclosure is shown. Figure 7As can be seen in , the method comprises a first step T1 for calculating the distortion between the input signal x and the decoded input signal.

[0211] In step T2, a random signal is used Perform bitrate calculation.

[0212] The calculations of step T1, step T3 and step T4 can be respectively Figure 3 The calculations of step E1, step E3 and step E4 are the same.

[0213] All embodiments given with reference to the first aspect of the invention may be implemented in this second aspect of the invention. More specifically, Figure 3 , Figure 4 The embodiment of FIG. 6 can be applied to this aspect of the present disclosure. For example, more than one post-processing function can be used, and these post-processing functions can be applied to different parts of the input signal, with their defined parameters, one post-processing function for each part or for several parts. An auxiliary signal can also be used, such as a patch, a mask...

[0214] In other words, one main difference of this aspect of the invention is the use of a random signal instead of using a first latent representation as input to the metric.

[0215] Therefore, this aspect of the invention also enables the application of the RDO strategy to signals encoded using a pair of NN encoder / decoders trained together, without having to perform another training of the pair of encoder / decoders to generate another latent representation.

[0216] Therefore, this aspect of the invention also enables the application of the RDO strategy to a signal encoded using an NN encoder with frozen coefficients, without having to perform another training of the NN encoder or NN decoder (also with frozen coefficients) to generate another latent representation that is more adapted to the content of the input signal.

[0217] In some embodiments, at least a first neural network encoded representation of an input signal is obtained by an encoding method that calculates a training loss function, and the training loss function and the post-processing loss function are the same function.

[0218] In some embodiments, at least a first neural network encoded representation of an input signal is obtained by an encoding method that calculates a training loss function, and the training loss function and the post-processing loss function are different functions.

[0219] In some embodiments, the iterative process computes at least two post-processing loss functions, which are applied to different parts of the input signal.

[0220] In some embodiments, when the input signal represents an image or video signal, the at least two portions of the input signal, at least one of which, represents a region of interest.

[0221] In some embodiments, the training loss function and the post-processing loss function are functions of a distortion factor and a rate factor multiplied by a rate-distortion coefficient, and the loss function and the processing function parameters differ in their rate-distortion coefficient values.

[0222] In some embodiments, the training loss function and the post-processing loss function are functions of a distortion factor and a rate factor multiplied by a rate-distortion coefficient, and the loss function and the processing function parameters differ in their distortion factors.

[0223] In some embodiments, the post-processing loss function receives as input an auxiliary signal comprising information representing the portion of the input signal.

[0224] In some embodiments, the information included in the auxiliary signal is selected from at least one of the following:

[0225] - a distortion mask comprising weights associated with at least some pixels of the input signal,

[0226] - a spatial identity of at least one set of pixels of the input signal,

[0227] - a patch to be applied to at least a portion of the decoded input signal.

[0228] In some embodiments, when the auxiliary signal is a distortion mask, the post-processing loss function calculates the distortion factor based on a pixel-wise multiplication of the distortion mask and a difference between the input signal and the corresponding decoded signal.

[0229] In some embodiments, the auxiliary signal is a patch, the post-processing loss function calculates a superimposed signal based on the decoded signal and the patch, and calculates the distortion factor based on a difference between the calculated superimposed signal and the decoded signal.

[0230] The present disclosure also relates to a device for encoding an input signal, the device comprising one or more processors, which are configured individually or in combination to

[0231] - obtaining at least a second latent representation of the input signal by modifying a first neural network encoded latent representation of the input signal obtained from a neural network encoder with frozen coefficients, using a metric based on:

[0232] - Input signal,

[0233] - the first neural network encodes the latent representation, and

[0234] - a decoded signal as a decoding of the first neural network encoded latent representation of the input signal, the decoded signal being obtained from a neural network decoder with its coefficients frozen.

[0235] Figure 8 An encoder (e.g. Figure 2 An example of a hardware architecture of an encoder 100).

[0236] To this end, the encoder has the hardware architecture of a computer. Figure 8 As shown, the encoder includes a processor 1. Although shown as a single processor 1, two or more processors may be used depending on the specific needs, expectations, or specific implementations of the encoder. In general, the processor 1 executes instructions and manipulates data to perform the operations of the encoder and any algorithms, methods, functions, processes, flows, and procedures described in this disclosure.

[0237] The encoder may also include communication means 5. Figure 3 5 is shown as a single communication device 5, but two or more communication devices may be used according to the specific needs, expectations or specific implementations of the decoder. The communication device may be used by the encoder to communicate with another electronic device linked to it in communication, and for example a decoding device. Typically, the communication device 5 may include logic encoded in software, hardware or a combination of software and hardware. More specifically, the communication device 5 may include software supporting one or more communication protocols associated with the communication, to transmit physical signals within and outside the decoder shown.

[0238] The encoder may further include a random access memory 2, a read-only memory 3 and a non-volatile memory 4. The read-only memory 3 of the encoder constitutes a recording medium consistent with the present disclosure, which can be read by the processor 1 and has recorded thereon a computer program PROG_EN consistent with the present disclosure, the computer program containing instructions for executing the steps of the method for encoding an input signal according to the present disclosure.

[0239] The program PROG_EN defines the functional modules of the encoder based on or controlling the above-mentioned elements 1 to 5 of the encoder, and these functional modules may include in particular:

[0240] - a NN-based encoder 101 for encoding an input video signal x into a first latent representation,

[0241] - Post-processing calculation module 300,

[0242] - quantizer module 102,

[0243] - Entropy coding module 103,

[0244] - a bitstream generator for generating a bitstream B,

[0245] - Transmission interface TX, used for transmitting the coded signal B.

[0246] An example of such a program is described below:

[0247] Input: x, , , E, D, C

[0248] parameter: 、N、

[0249] Output:

[0250]

[0251] For do

[0252]

[0253]

[0254]

[0255]

[0256]

[0257] )

[0258] End for

[0259]

[0260] Fig. 9 represents the first latent representation Figure 1 alternative.

[0261] According to some embodiments, the loss function is calculated using at least one additional video frame besides the current input signal, wherein encoding of the at least one additional frame depends on a decoded signal of the current input signal (eg for motion calculation).

[0262] One advantage of this embodiment is that, compared to a "local" optimization that only considers the current input signal, considering at least one next frame to be encoded enables a more "global" optimization of the current latent representation. Such a global optimization provides a second latent representation that is better not only for the current input signal, but also for additional frames that depend on the current input signal.

[0263] The next frame may be understood as the next frame associated with the temporal order or display order of the image frames, but is not limited thereto. The next frame may also be related to the order in which the pictures are encoded. In some embodiments, when a GOP (Group of Pictures) structure is used to encode the encoded image, the next frame may be related to the GOP structure of the encoded image.

[0264] Fig.10 An implementation method of using the next frame for calculation post-processing is shown. In this implementation method, for one iteration, the following method is performed:

[0265] - Get the first neural network latent representation of the current input signal x by using the neural network encoder with frozen coefficients ,

[0266] - using a neural network decoder with frozen coefficients to decode the first neural network latent representation of the current input signal Decode and obtain the first decoded signal of the current input signal ,

[0267] - obtaining a neural network latent representation of the at least one additional frame by using the neural network encoder, represented as Due to the above time dependency, this step must refer to the first decoded signal of the current input signal and is expressed in [Equation 20]. In this equation, the symbol Indicator function The calculation is subject to In other words, [Equation 20] refers to the current frame Encoded, the potential representation of the current frame is gives:

[0268] [Mathematical formula 20]

[0269] - decoding the neural network latent representation of the at least one additional frame using the fixed neural network decoder and obtaining a decoded signal of the at least one additional frame, represented as .

[0270] - make is a weight coefficient (between zero and one) to determine How much does it contribute to the final loss function. It has been evaluated that Compared with other values ​​of Values ​​of generally produce higher compression performance.

[0271] - Calculate the distortion of the first decoded signal of the current input signal , and the distortion of the decoded signal of at least one additional frame . Use by The defined weights add these two distortions and obtain the combined distortion:

[0272] [Mathematical formula 21]

[0273] - Calculate the bit rate of the first neural network potential representation of the current input signal , and the bitrate of the neural network latent representation of at least one additional frame . Use by The defined weights add these two bitrates and obtain the combined bitrate:

[0274] [Mathematical formula 22]

[0275] - Use the combined bitrate and combined distortion to calculate the loss function:

[0276] [Mathematical formula 23]

[0277] - obtaining the second latent representation of the input signal by minimizing the loss function :

[0278] [Mathematical formula 24]

[0279] As can be seen from the above formula, the code rate calculation is also modified here. In this embodiment, or

[0280] Embodiments of the present invention can be implemented by computer software that can be executed by a data processor (such as in a processor entity), or by hardware, or by a combination of software and hardware. In addition, in this regard, it should be noted that any block of the logic flow in the figure can represent a program step, or an interconnected logic circuit, block and function, or a combination of a program step and a logic circuit, block and function. The software can be stored on a physical medium (such as a memory chip or a memory block implemented in a processor), a magnetic medium (such as a hard disk or a floppy disk), and an optical medium (such as, for example, a DVD and its data variants, a CD).

[0281] A number of embodiments have been described. Features of these embodiments may be provided individually or in any combination across various claim categories and types. In addition, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination across various claim categories and types.

[0282] The embodiments and aspects described herein may be implemented in, for example, a method or process, a device, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of embodiment (e.g., discussed only as a method), the embodiments of the features discussed may also be implemented in other forms (e.g., a device or program). The device may be implemented, for example, in appropriate hardware, software, and firmware. The method may be implemented, for example, in a device, such as a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, such as a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate information communication between end users.

[0283] References to "one embodiment" or "an embodiment" or "one implementation" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in an implementation" and any other variations appearing in various places throughout this application are not necessarily all referring to the same embodiment.

[0284] In addition, the present application may involve “determining” or “obtaining” various information. Determining information may include, for example, one or more of the following: estimating information, calculating information, predicting information, or obtaining information from a memory.

Claims

1. A method for encoding an input signal, the method comprising: - by modifying a first neural network encoded latent representation ( ) of said input signal (x) obtained from a neural network encoder (101) with frozen coefficients, using a metric based on ), obtaining at least a second potential representation ( : - the input signal (x), - The first neural network encodes latent representations ),as well as - a decoded signal obtained by encoding a latent representation of the input signal (x) from a first neural network using a neural network decoder (201) with frozen coefficients obtained in the decoding of .

2. The method according to claim 1, wherein: Obtaining the at least second potential representation ( ) comprises an iterative process comprising at least two iterations (i), wherein, - For the first iteration step, the iteration process receives as input: a first latent representation ( ), the input signal (x), and the first potential representation ( ) is decoded by , and among them, - For the next iteration step, the first latent representation ( ) is derived from the output of the previous iteration step ( replace.

3. The method according to claim 2, wherein: Through the iterative process, by calculating at least the post-processing loss function ( ) to obtain the metric, where the second latent representation ( The post-processing loss function is obtained ) minimized output.

4. The method according to claim 3, wherein: By using the training loss function ) after the training step of obtaining the encoding method to obtain the at least first neural network encoding representation ( ), the training loss function ) and the post-processing loss function ) are the same function.

5. The method according to claim 2 or 3, wherein: By using the training loss function ) after the training step of obtaining the encoding method to obtain the at least first neural network encoding representation ( ), and the iterative process calculates the training loss function ) Different at least one processing loss function ).

6. The method according to claim 5, wherein: The iterative process computes the post-processing loss function using different parameters for some parts (j) of the input signal ).

7. The method according to claim 6, wherein: When the input signal (x) represents an image or a video signal, the at least two parts of the input signal, at least one of the parts, represent a region of interest.

8. The method according to any one of claims 5 to 7, wherein: The training loss function ) and the post-processing loss function ) is the distortion factor , and the rate-distortion coefficient ( Multiplying rate factor A function where the training loss function ) and the post-processing function The parameters of the rate-distortion coefficient value Different aspects.

9. The method according to any one of claims 5 to 8, wherein: The training loss function ) and the post-processing loss function ) is the distortion factor , and the rate-distortion coefficient ( Multiplying rate factor A function where the training loss function ) and the post-processing function The parameters of its distortion factor Different aspects.

10. The method according to any one of claims 5 to 9, wherein: The post-processing loss function ) receives as input an auxiliary signal (Aux), which includes information representing various parts of the input signal.

11. The method according to claim 10, wherein: The information included in the auxiliary signal (Aux) is selected from at least one of the following: a. a distortion mask comprising weights associated with at least some pixels of the input signal, b. a spatial identity of at least one set of pixels of the input signal, c. A patch to be applied to at least a portion of the decoded input signal.

12. The method according to claim 11, wherein: When the auxiliary signal (Aux) is a distortion mask, the post-processing loss function ) based on the distortion mask and the input signal (x) and the corresponding decoded signal The distortion factor is calculated by pixel-wise multiplication of the difference between them.

13. The method according to claim 11, wherein: The auxiliary signal (Aux) is the patch, and the post-processing loss function ) based on the decoded signal and the patches to calculate the superposition signal ( , and based on the calculated superposition signal ( With the decoded signal The distortion factor is calculated by taking the difference between .

14. A device for encoding an input signal, the device comprising one or more processors, which are configured individually or in combination to - by modifying a first neural network encoded latent representation ( ) of said input signal (x) obtained from a neural network encoder (101) with frozen coefficients, using a metric based on , obtaining at least a second potential representation ( : - the input signal (x), - The first neural network encodes latent representations ,as well as - Decoding signal , the decoded signal is obtained from a first neural network encoding latent representation of the input signal (x) using a neural network decoder (201) with frozen coefficients obtained in the decoding of .

15. The device according to claim 14, wherein: The at least one processor is further configured to obtain the at least second potential representation as follows ( : Through at least two iterations, - For the first iteration, use as input the following: the first latent representation of the input signal (x) ( , the input signal (x), and the first potential representation ( The decoded signal obtained in the decoding , and among them, - For the next iteration, the first latent representation ( The output of the previous iteration step ( replace.

16. The apparatus of claim 15, further configured to: during iterations, calculate at least a post-processing loss function ) to obtain the metric, where The second potential representation ( The post-processing loss function is obtained ) minimized output.

17. The device according to claim 16, wherein: From using the training loss function ) to obtain the at least first neural network encoding potential representation ( , and the training loss function ) and the post-processing loss function ) are the same function.

18. The device according to claim 15 or 16, wherein: From using the training loss function ) to obtain the at least first neural network encoding representation ( , and the iterative process calculates the training loss function ) Different at least one post-processing loss function ).

19. The device according to claim 18, wherein: The iterative process computes the post-processing loss function using different parameters for parts of the input signal ).

20. A signal comprising a bitstream comprising encoded data representing a second latent representation of a signal, the second representation being obtained by modifying a first neural network encoded latent representation of the signal obtained from a trained neural network encoder using a metric based on: - the signal (x), - The first neural network encodes latent representations ,as well as - Decoding signal , the decoded signal is obtained from a first neural network encoding latent representation of the signal (x) using a neural network decoder (201) with frozen coefficients obtained in the decoding of .

21. A computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 13.

22. A computer-readable storage medium having stored thereon instructions for causing one or more processors to execute the method of any one of claims 1 to 13.

Images