Method and apparatus for training neural network-based decoder

Abstract

The invention relates to a device and a method for training a neural network-based decoder. The method includes quantizing a parameter representing coefficients of the neural network based decoder using a training quantizer during the training. The invention also relates to a method and a device for encoding at least parameters representing coefficients of a neural network-based decoder. The invention also relates to a method for generating an encoded bitstream comprising: a neural network-based encoder and an encoded neural network-based decoder of the decoder, and a signal encoded using said neural network-based encoder.

Description

Technical Field

[0001] The present invention relates to the field of signal encoding and decoding using neural networks. Background Art

[0002] Digital technology is becoming more and more important in everyday life, especially video streaming. The use of video streaming is increasing, so its environmental impact becomes an important topic. Standardization organizations such as MPEG and ITU are committed to reducing the impact of video streaming and have released several video coding standards to reduce the size of the video while maintaining acceptable visual quality. Neural-based (NN) encoders and decoders have been introduced in video encoding and decoding processes and provide improved performance, thereby enabling the amount of streaming data to be reduced, but their use in video codecs remains challenging in terms of configuration. Several proposed solutions for introducing neural networks in video compression still lack compression efficiency. For example, a paper entitled "AIVC: Artificial Intelligence based video codec" by Théo Ladune and Pierrick Philippe discloses an end-to-end neural video codec that is presented as being able to implement any desired codec configuration. However, this neural network codec presents some disadvantages, and among these disadvantages, it can be considered that compression efficiency can be improved. To this end, the present disclosure proposes a neural network codec that solves at least one disadvantage of the prior art. Summary of the invention

[0003] In this context, the present disclosure proposes a computer-implemented method for training a neural network based decoder, the method comprising quantizing parameters representing coefficients of the neural network based decoder using a training quantizer during the training, wherein the parameters are coefficients of the neural network based decoder.

[0004] The present disclosure is particularly relevant in the context of a codec trained for a dedicated set of input signals, the dedicated set comprising the signal to be encoded. The decoder network learns to decode correctly even in the presence of quantization noise, and can then be compressed more to obtain an overall gain in compression efficiency, so that the gains obtained using the specific set of signals used to train the encoder are not offset when transmitted with the encoded signal set. Thus, one of the technical contributions of the present disclosure is encoding data for reliable and / or efficient transmission or storage (and corresponding decoding).

[0005] Using the coefficients of a neural network represents a simple implementation that does not require additional parameters.

[0006] According to some embodiments, the method comprises:

[0007] - running inference on training of a neural network based decoder using said quantized parameters,

[0008] - back-propagating gradients based on the result of the inference to update coefficients of the neural network based decoder, the updated coefficients of the trained neural network decoder being unquantized.

[0009] Thus, the gradients are not erased during subsequent quantization stages.

[0010] According to some embodiments, the method comprises:

[0011] - obtaining the difference between the coefficients of a reference decoder based on a neural network and the coefficients of said decoder based on a neural network to be trained,

[0012] - obtaining a dequantized version of said difference, said dequantized version being further obtained after applying said quantization,

[0013] - obtaining a trained decoder as a difference between coefficients of said neural network based reference decoder and coefficients of said dequantized version,

[0014] - using the neural network based encoder to be trained and the obtained trained neural network based decoder to perform said running inference.

[0015] Preferably, the neural network based decoder and the neural network based reference decoder are compatible in the sense of obtaining the difference between their coefficients, and for example they may have the same architecture and / or the same connections.

[0016] According to another aspect, the proposed disclosure relates to a computer-implemented method for encoding at least parameters representing coefficients of a neural network-based decoder into a bitstream, wherein the neural network-based decoder is trained using the training method of any embodiment of the present disclosure.

[0017] According to some embodiments, the method for encoding comprises quantizing the parameters representing the coefficients of the trained neural network-based decoder using a quantizer, the quantizer

[0018] - is equal to the trained quantizer, or

[0019] - is the quantizer whose reconstruction level includes the reconstruction level of the trained quantizer Q2.

[0020] According to some embodiments, the method for encoding comprises quantizing the parameters representing coefficients of the trained neural network based decoder using a quantizer, the trained quantizer being a function of distortion, and the quantizer being a function of rate-distortion.

[0021] According to some embodiments, the method for encoding comprises quantizing the parameters representing coefficients of the trained neural network based decoder using a quantizer, the trained quantizer minimizing distortion, and the quantizer minimizing rate distortion.

[0022] According to some embodiments of the method for encoding, the encoding conforms to

[0023] - Neural Network Representation (NNR), also known as MPEG-7 Part 17 or

[0024] -Neural Network Exchange Format (NNEF) or

[0025] -Open Neural Network Exchange (ONNX).

[0026] According to another aspect, the present disclosure relates to a computer-implemented method for generating an encoded bitstream, the method comprising:

[0027] - a neural network-based encoder and decoder based on the encoded neural network-based decoder encoded according to the method of the present disclosure, and

[0028] - A signal encoded using said neural network based encoder.

[0029] According to some embodiments of the method for generating a bitstream, the bitstream is generated by concatenating the encoded neural network-based decoder and the encoded signal.

[0030] Based on another aspect, the present disclosure relates to a device for training a neural network-based decoder, the device comprising one or more processors configured individually or in combination to quantize parameters representing coefficients of the neural network-based decoder using a training quantizer during the training.

[0031] According to some embodiments of the apparatus for training a neural network based decoder, the parameters are coefficients of the neural network based decoder.

[0032] According to some embodiments of the apparatus for training a neural network based decoder, the parameter is a difference between coefficients of the neural network based decoder and coefficients of a neural network based reference decoder.

[0033] According to some embodiments of the apparatus for training a neural network based decoder, the apparatus is further configured to:

[0034] - running inference on training of a neural network based decoder using said quantized parameters,

[0035] - back-propagating gradients based on the result of the inference to update coefficients of the neural network-based decoder, the updated coefficients of the trained neural network-based decoder being unquantized.

[0036] According to some embodiments of the apparatus for training a neural network based decoder, the apparatus is further configured to:

[0037] - obtaining the difference between the coefficients of a reference decoder based on a neural network and the coefficients of said decoder based on a neural network to be trained,

[0038] - obtaining a dequantized version of said difference, said dequantized version being further obtained after applying said quantization,

[0039] - obtaining a trained neural network based decoder as a difference between coefficients of said neural network based reference decoder and coefficients of said dequantized version,

[0040] - using the neural network based encoder to be trained and the obtained trained neural network based decoder to perform said running inference.

[0041] According to another aspect, the present disclosure relates to a device for encoding at least parameters representing coefficients of a neural network based decoder into a bitstream, wherein the neural network based decoder is trained using the training device of any embodiment of the present disclosure.

[0042] According to another aspect, the present disclosure relates to an apparatus for generating a coded bitstream, the apparatus comprising:

[0043] - a neural network-based encoder and decoder encoded with a device according to any embodiment of the present disclosure, and a neural network-based decoder encoded with a neural network-based encoder and decoder, and

[0044] - A signal encoded using said neural network based encoder.

[0045] According to another aspect, the present disclosure also relates to a computer program product, which includes instructions. When the program is executed by one or more processors, the instructions cause the one or more processors to perform any one of the methods in the embodiments of the present disclosure.

[0046] According to another aspect, the present disclosure also relates to a computer-readable storage medium, which includes instructions, which when executed by a computer, cause the computer to perform any one of the embodiments of the method according to the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Shows the overall overview of a neural network based encoder and decoder.

[0048] Figure 2A and Figure 2B Two different embodiments of an encoder for implementing two different embodiments of the training method according to the present disclosure are shown.

[0049] Figure 3 An example of a hardware architecture for an encoder according to some embodiments of the present disclosure is shown.

[0050] Figure 4 A first example of a training method for a NN-based decoder according to some embodiments of the present disclosure is presented.

[0051] Figure 5 A second example of a training method for a NN-based decoder according to some embodiments of the present disclosure is presented.

[0052] Figure 6 A third example of a training method for a NN-based decoder according to some embodiments of the present disclosure is presented.

[0053] Fig. 7A and Figure 7B Two different implementations of a decoder for decoding an encoded signal according to some embodiments of the present disclosure are shown.

[0054] Figure 8 An example of a hardware architecture of a decoder according to some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0055] Figure 1 An overview of a neural network (NN) for encoding and decoding a signal is presented. A signal may be understood as any signal that is encoded (or compressed) for transmission in order to reduce the size of the data to be transmitted. Thus, a signal may be understood as a video signal, an audio signal, a combination of both, or other signals for which compression would be advantageous in order to gain, for example, transmission bandwidth or storage capacity.

[0056] The following description will refer to encoding and decoding of images, but this is given as an illustrative input signal and should not be limited thereto.

[0057] like Figure 1As shown in , the neural network EN for encoding is associated with the neural network DN for decoding, which can be used to decode the data unit generated by the encoding neural network EN. The neural network EN and the neural network DN thus form a pair of neural networks.

[0058] The neural network EN and the neural network DN are each defined by a structure comprising, for example, a plurality of neuron layers and / or a set of weights respectively associated with the neurons of the network in question. Later in the description, when referring to the neural network structure or parameters, weights or coefficients may be used indiscriminately.

[0059] A representation (e.g. two-dimensional) of the current image IMG (or alternatively, a component or a block of the current image IMG) is applied as input to an artificial neural network (i.e. at a layer at the input end) of the encoding artificial neural network EN. The encoding artificial neural network EN then produces data (in this case data units) at an output end.

[0060] Subsequently, the data (here data units) are applied as input to a decoding neural network DN. The decoding neural network DN then generates as output a representation IMG' (e.g. two-dimensional) corresponding to the current image IMG (or alternatively to a component or block of the current image IMG).

[0061] The encoding neural network EN is designed so that the data unit contains a smaller (at least on average smaller) amount of data than the aforementioned representation of the current image IMG. In other words, the data in the data unit is compressed.

[0062] Furthermore, the encoding neural network EN and the decoding neural network DN are pre-trained so as to minimize the difference between the input representation of the current image IMG and its output representation IMG', while also minimizing the amount of data transferred between the neural network EN and the neural network DN.

[0063] Figure 2A and Figure 2B Two different embodiments of NN encoders and decoders for implementing two different embodiments of the training method according to the present disclosure are shown. These figures are schematic diagrams and do not show all modules of such encoders, but only the modules of interest for understanding the present description.

[0064] exist Figure 2A and 2BIn the embodiment of the present invention, an input video V to be encoded is received at the input of the NN codec. Each image of the sequence V is represented, for example, by means of at least one two-dimensional representation, such as a pixel matrix. In practice, each input image may be represented by a plurality of two-dimensional representations (or pixel arrays) (or pixel matrices) corresponding to a plurality of components of the image, for example a plurality of color components, or alternatively a luminance component and at least one chrominance component.

[0065] The present disclosure may be particularly relevant to the case where a neural network trained on a sequence for encoding a video signal is used to encode V. This subsequently improves the encoding because the signal is compressed more efficiently. However, this creates the need to transmit the decoder DV together with the encoded signal, because the decoder is no longer generic, but is specific to the sequence and is therefore unknown on the decoding side in terms of structure and / or parameters. Transmitting the decoder DV may offset the gains obtained through efficient training. In addition, when compressing the decoder for transmission, it is not desirable to compress the decoder too much, otherwise the decoding performance will degrade. Therefore, the bit rate of the decoder remains high and destroys any gains obtained on the compressed signal.

[0066] Therefore, the present disclosure provides a training method and apparatus for a neural network decoder, which is implemented by applying quantization during the training of the decoder. Due to the application of quantization during the training of the decoder, the decoder learns to decode correctly even in the presence of quantization noise. Thus, the decoder can be compressed more to obtain an overall gain in compression efficiency.

[0067] In some embodiments, the input video V may include, for example, a set of images that are determined to be homogeneous if the mean square error between corresponding luminance components of two consecutive images is less than a predetermined threshold for all consecutive image pairs of the sequence (e.g., for images whose luminance values ​​may vary between 0 and 255, the threshold may be between 25 and 75, such as equal to 50).

[0068] In some embodiments, the input video V may be divided into groups of images corresponding to a predefined duration, such as a group of images during 1 second of video.

[0069] In some embodiments, the input video V may be divided into multiple groups of images between two scene cuts.

[0070] The video V can also be divided into mini batches to reduce the computational cost. The number of images in each mini batch can be controlled by parameters.

[0071] Figure 2AA first embodiment E1 of an encoder comprising a NN-based encoder EV and a NN-based decoder DV is presented. The NN-based encoder EV and the decoder DV are trained on a video V and thus the coefficients of the network are obtained for this specific input video V.

[0072] The training process includes, for example:

[0073] - applying the current image IMG (or a plurality of images including the current image IMG, here the images of the current sequence V) as input to the encoder E1;

[0074] - Train the NN codec by running inference on the input video V to obtain coefficients (or weights) of the NN codec to reduce the throughput cost, which is called the rate-distortion cost.

[0075] according to Figure 2A In an embodiment, quantization is used during training to train the NN-based decoder DV. To this end, a trained inference is run on the NN codec and a trained quantizer Q2 is applied during training of the NN-based decoder DV. The parameters of the NN-based decoder are obtained by applying the quantizer Q2 to the coefficients of the NN-based decoder DV. This is about Figure 4 Described in more detail.

[0076] The encoder E1 comprises at least:

[0077] - a NN-based encoder EV specifically trained on an input signal V to generate a signal VV,

[0078] - a NN-based decoder DV, which is specifically trained on an input signal V and the training method is about Figure 4 or Figure 6 Disclosed.

[0079] - a network encoder responsible at least for encoding the parameters representing the decoder DV based on the training, thereby generating a signal TD

[0080] A bitstream generator responsible for concatenating at least the coded signal VV and coding parameters representative of the decoder DV based on training (signal TD) to output signal F.

[0081] The video V is encoded using a NN encoder EV according to known methods to obtain an encoded signal VV. According to the neural network encoding, EV generates latent variables L representing the input signal V. Then, the L variables can be quantized into quantization indices I, and the indices can be entropy encoded to form a bit stream B representing the input video. This bit stream can be stored or transmitted.

[0082] As explained later, the parameters of the trained decoder can be based on Figure 4 or Figure 6 The described method obtains the coefficients DV2(N) of the NN-based decoder DV.

[0083] The network encoder is used to compress the parameters of the trained decoder to obtain a NN-encoded decoder TD. According to some embodiments, the network encoder may be selected from:

[0084] - Neural Network Representation (NNR) or also known as MPEG-7 Part 17 or

[0085] -Neural Network Exchange Format (NNEF) or

[0086] -Open Neural Network Exchange (ONNX).

[0087] Of course, the above list is given as an example only and should not be considered an exhaustive list of formats that can be used.

[0088] It is advantageous to encode the parameters of the trained network DV rather than the difference values ​​using a reference decoder, since this eliminates the requirement for the reference decoder to be present in the decoding device before operating the decoding of the bitstream created using the encoder of the present invention. The decoder is thus backwards compatible with devices that do not include a potential reference decoder and is simpler in design.

[0089] The network encoder includes, among other functions, quantization of coefficients DV2 of a trained NN-based decoder DV using a quantizer Q1. The quantizer Q1 may:

[0090] - equal to the training quantizer Q2, or

[0091] - is the quantizer whose reconstruction level includes the reconstruction level of the trained quantizer Q2.

[0092] Hence, quantizer Q1 may use a subset of the quantization levels of, for example, Q2, and thus operate with lower complexity.

[0093] In some embodiments, the training quantizer Q2 may work on distortion and the quantizer Q1 may work on rate distortion.

[0094] In some embodiments, training quantization may minimize distortion, and quantizer Q1 may minimize rate distortion.

[0095] In some embodiments, the reconstruction levels of quantizer Q2 may be reconstruction levels of Q1 with even indices.

[0096] The encoder E1 thus implements a method for encoding into a bitstream parameters representing at least the coefficients of a decoder DV based on an NN, wherein the reference Figure 4 Or the training method described in 6 trains the decoder.

[0097] Then, the bitstream generator generates a bitstream including

[0098] - a neural network based encoder and decoder based on an encoded neural network decoder encoded according to the method of the present disclosure, and

[0099] - A signal encoded using said neural network encoder.

[0100] The TD signal can be connected in series with the coded signal VV by the bitstream generator to create a bitstream. The connection can be understood as assembling two signals into a single bitstream in any form, such as multiplexing. The provided bitstream can be understood by the decoder responsible for decoding the coded signal VV.

[0101] In practice, the bitstream generator may include an entropy encoder for entropy encoding the aggregated data.

[0102] However, for simplicity, the entropy encoding step will not be described here, but the data stream F may already be entropy encoded.

[0103] The bit stream F can then be sent to another electronic device (e.g. Fig. 7A The encoding device E1 may be used to store the encoded data in a storage unit (such as a hard disk or a memory of the encoding device E1) for subsequent reading and sending.

[0104] Figure 2B A second embodiment E2 of an encoder comprising a NN-based encoder EV and a NN-based decoder DV is presented. The NN-based encoder EV and the decoder DV are trained on a video V and thus the coefficients of the network are obtained for this specific input video V.

[0105] Furthermore, in this embodiment, the encoder E2 comprises an NN-based reference encoder EU and an NN-based reference decoder DU. The NN-based reference encoder EU and the NN-based reference decoder DU may be defined as NN-based generic encoders and decoders, respectively, in the sense that they have been trained on a set of video sequences different from the video they are used to compress. Their coefficients are known and are used in conjunction with the decoder D2 (reference Figure 7B description) and therefore does not need to be transmitted along with the encoded data.

[0106] The neural network based encoder and the neural network based reference encoder have an architecture that may be identical in terms of architecture and connections.

[0107] The neural network based decoder and the neural network based reference decoder have an architecture that may be identical in terms of architecture and connections.

[0108] according to Figure 2B In an embodiment, quantization is used during training to train the NN-based decoder DV. To this end, a trained inference is run on the NN codec and a training quantizer Q2 is applied during training of the NN-based decoder DV. The parameters of the NN-based decoder DV are obtained by applying the quantizer Q2 to parameters related to the difference between the coefficients of the NN-based decoder DV and the NN-based reference decoder DU. This is about Figure 5 and Figure 6 The encoder E2 comprises at least:

[0109] - a NN-based encoder EV specifically trained on an input signal V to generate a signal VV,

[0110] - a NN-based decoder DV, which is specifically trained on an input signal V and the training method is about Figure 5 and Figure 6 Disclosed,

[0111] - a NN-based reference encoder EU trained on several input sequences to generate a signal VV,

[0112] - a NN-based reference decoder DU trained on several input sequences,

[0113] a network encoder responsible at least for encoding parameters representing a decoder DV based on training, thereby generating a signal TD,

[0114] A bitstream generator responsible for concatenating at least the coded signal VV and coding parameters representative of the decoder DV based on training (signal TD) to output signal F.

[0115] The bitstream V is encoded using a NN encoder EV according to known methods to obtain an encoded signal VV. According to the neural network encoding, EV generates latent variables L representing the input signal V. Then, the L variables can be quantized into quantization indices I, and the indices can be entropy encoded to form a bitstream B representing the input video. This bitstream can be stored or transmitted.

[0116] As explained later, the parameters of the trained decoder may be coefficients DDV2(N) representing the Figure 5 and Figure 6 The method described obtains the difference between DU and DV2(N). This represents the advantage of transmitting less data for encoding the decoder DV.

[0117] The network encoder is used to compress the parameters of the trained decoder to obtain a NN-encoded decoder TD. According to some embodiments, the network encoder may be selected from:

[0118] - Neural Network Representation (NNR), also known as MPEG-7 Part 17 or

[0119] -Neural Network Exchange Format (NNEF) or

[0120] -Open Neural Network Exchange (ONNX).

[0121] Of course, the above list is given as an example only and should not be considered an exhaustive list of formats that can be used.

[0122] The network encoder includes, among other functions, quantization of coefficients DDV2 of a trained NN-based decoder DV using a quantizer Q1. The quantizer Q1 may:

[0123] - equal to the training quantizer Q2, or

[0124] - is the quantizer whose reconstruction level includes the reconstruction level of the trained quantizer Q2.

[0125] In some embodiments, the training quantizer Q2 may work on distortion and the quantizer Q1 may work on rate distortion.

[0126] In some embodiments, training quantization may minimize distortion, and quantizer Q1 may minimize rate distortion.

[0127] In some embodiments, the reconstruction levels of quantizer Q2 may be reconstruction levels of Q1 with even indices.

[0128] The encoder E2 thus implements a method for encoding parameters representing at least the coefficients of a decoder DV based on an NN into a bitstream, wherein the reference Figure 5 Or the training method described in 6 trains the decoder.

[0129] Then, the bitstream generator generates a bitstream including

[0130] - a neural network based encoder and decoder based on an encoded neural network decoder encoded according to the method of the present disclosure, and

[0131] - A signal encoded using said neural network encoder.

[0132] The TD signal can be connected in series with the coded signal VV by the bitstream generator to create a bitstream. The connection can be understood as assembling two signals into a single bitstream in any form, such as multiplexing. The provided bitstream can be understood by the decoder responsible for decoding the coded signal VV.

[0133] In practice, the bitstream generator may include an entropy encoder for entropy encoding the aggregated data.

[0134] However, for simplicity, the entropy encoding step will not be described here, but the data stream F may already be entropy encoded.

[0135] The bit stream F can then be sent to another electronic device (e.g. Figure 7B The encoding device E2 may be stored in a storage unit (such as a hard disk or a memory of the encoding device E2) for subsequent reading and sending.

[0136] The encoder E1 thus implements a method for encoding into a bitstream parameters representing at least the coefficients of a decoder DV based on an NN, wherein the reference Figure 4 Or the training method described in 6 trains the decoder.

[0137] Figure 3 An encoder according to some embodiments of the present disclosure and an example Figure 2A and Figure 2B Example of the hardware architecture of encoder E1 or E2.

[0138] To this end, the encoder has the hardware architecture of a computer. Figure 3 As shown, the encoder includes a processor 1. Although shown as a single processor 1, two or more processors may be used depending on the specific needs, expectations, or specific implementations of the encoder. In general, the processor 1 executes instructions and manipulates data to perform the operations of the encoder and any algorithms, methods, functions, processes, flows, and procedures described in this disclosure.

[0139] The encoder may also include communication means 5. Figure 3 5 is shown as a single communication device 5, but two or more communication devices may be used according to the specific needs, expectations or specific implementations of the decoder. The communication device may be used by the encoder to communicate with another electronic device linked to it in communication, and for example a decoding device. Typically, the communication device 5 may include logic encoded in software, hardware or a combination of software and hardware. More specifically, the communication device 5 may include software supporting one or more communication protocols associated with the communication, to transmit physical signals within and outside the decoder shown.

[0140] The encoder may further include a random access memory 2, a read-only memory 3 and a non-volatile memory 4. The read-only memory 3 of the encoder constitutes a recording medium consistent with the present disclosure, which can be read by the processor 1 and has recorded thereon a computer program PROG_EN consistent with the present disclosure, the computer program containing instructions for executing the steps of the method for training a neural network-based decoder according to the present disclosure.

[0141] The program PROG_EN defines the functional modules of the encoder based on or controlling the above-mentioned elements 1 to 5 of the encoder, and these functional modules include in particular:

[0142] - an NN-based encoder EV for encoding an input video signal V into an output signal VV,

[0143] a NN-based decoder DV adapted to be trained on the input signal V and transmitted together with the encoded signal V,

[0144] - a network encoder for encoding the trained NN-based decoder,

[0145] - A bitstream generator for concatenating the encoded trained NN-based decoder and the encoded signal VV.

[0146] - A transmission interface TX, which is used to transmit the coded signal F.

[0147] In some embodiments, the defined modules may also include:

[0148] -NN-based reference encoder EU,

[0149] - a NN-based reference decoder DU for providing differential parameters of a trained NN-based decoder,

[0150] Figure 4 Examples of training methods or computer-implemented training methods for NN-based decoders according to some embodiments of the present disclosure are presented. Figure 3 A subset of the steps performed during the training of the NN-based decoder DV according to an embodiment of the present disclosure may be represented.

[0151] Figure 4 A method for training a decoder (DV) based on a neural network is described, the method comprising quantizing parameters representing coefficients of the neural network using a training quantizer (Q2) during said training.

[0152] according to Figure 4 In an embodiment, the parameters are coefficients of the neural network (EV-DV).

[0153] As mentioned earlier, Figure 4 Can be displayed by Figure 2A The method implemented by encoder E1.

[0154] The method receives as input a signal V to be encoded. As mentioned above, the signal can be of any type, such as an audio signal to be compressed, or a video signal, or a combination of the two, or other types of signals. Also as mentioned above, in order to reduce the computation time, the signal V can be divided into data subsets, hereinafter referred to as mini-batches. The signal V is divided into N mini-batches, V(i) represents mini-batch i, and I is between 1 and N.

[0155] In this embodiment, for the purpose of illustrating the present disclosure, the input signal V represents a video signal comprising a set of images. Therefore, a small batch i is a subset of the images of the input video signal V. As previously mentioned, one object of the present disclosure is to provide a specific NN codec for encoding and decoding the signal V. When the signal V is encoded and stored or transmitted for subsequent decoding, the method is reinitialized and the NN codec is trained again specifically for this new video signal. There may be other ways to divide the signal into small batches, for example, by extracting a cropped region from the video (e.g., a 256×256 pixel window located at the same position in each frame) to represent a smaller size video. The window size and cropping position can be changed so that a variety of situations are encountered during the training process.

[0156] At step S1, the NN-based encoder EV and the NN-based decoder DV are initialized. The initialization values ​​of mini-batch 0 are EV(0) and DV(0). The initialization values ​​are the initialization values ​​of the coefficients of the NN-based encoder and decoder. The initialization can be performed using a predetermined method, or the coefficients can be randomly assigned.

[0157] Further after step S1 , quantization of the NN-based decoder DV(i) is performed in step S2 , applying a quantizer Q2 to the coefficients of the NN-based decoder DV(i).

[0158] Further after the quantizer Q2, at step S3, a dequantizer is also applied to Q2(V(i)) to obtain a dequantized signal DQ2(Q2(V(i))), referred to as DV2(i).

[0159] Then, during step S4, the signal DV2(i) = DQ2(Q2(V(i))) is used to train the NN-based decoder DV, and the NN-based encoder EV. Multiple inferences are run in step S5 until it is decided to back-propagate to update the NN coefficients.

[0160] At step S6, a gradient is obtained from the obtained coefficients.

[0161] However, in order to preserve the gradients, these gradients are not applied to the quantized coefficients of the NN-based decoder in step S7, but to the unquantized coefficients after the optimization process. Therefore, the gradients are applied to the EV(i) and DV(i) coefficients for determining the updated coefficients. The coefficients are updated, resulting in EV(i+1) and DV(i+1), which are the encoder and decoder to be trained during the next mini-batch i+1, respectively.

[0162] Thus, as explained above, according to some embodiments, the training method comprises:

[0163] - Run inference on the trained NN-based decoder (DV) using quantized parameters,

[0164] - Back-propagating gradients based on the result of the inference to update coefficients of the NN-based decoder (DV), the updated coefficients of the trained NN-based decoder (DV) being unquantized.

[0165] At step S8, i is incremented and the next small batch is input to the NN-based codec EV(i+1), DV(i+1).

[0166] Figure 5 Examples of training methods or computer-implemented training methods according to other embodiments of the present disclosure are presented.

[0167] Figure 4 A method for training a decoder (DV) based on a neural network is described, the method comprising quantizing parameters representing coefficients of the neural network using a training quantizer (Q2) during said training.

[0168] according to Figure 5 In an embodiment of the invention, the parameter is the difference between the coefficients of the neural network (EV-DV) and the coefficients of the reference neural network (EU-DU).

[0169] As mentioned earlier, Figure 5 Can be displayed by Figure 2B The NN codec implementation method.

[0170] The method receives as input a signal V to be encoded. As mentioned above, the signal can be of any type, such as an audio signal to be compressed, or a video signal, or a combination of the two, or other types of signals. Also as mentioned above, in order to reduce the computation time, the signal V can be divided into data subsets, hereinafter referred to as mini-batches. The signal V is divided into N mini-batches, V(i) represents mini-batch i, i is between 1 and N.

[0171] In this embodiment, for the purpose of illustrating the present disclosure, the input signal V represents a video signal comprising a set of images. Thus, a mini-batch i is a subset of the images of the input video signal V. As previously mentioned, one object of the present disclosure is to provide a specific NN codec for encoding and decoding the signal V. When the signal V is encoded and stored or transmitted for subsequent decoding, the method is reinitialized and the NN codec is again trained specifically for this new video signal.

[0172] At step T1, the NN-based encoder EV and the NN-based decoder DV are initialized. The initialization values ​​of mini-batch 0 are EV(0) and DV(0). The initialization values ​​are the initialization values ​​of the coefficients of the NN-based encoder and decoder. The initialization can be performed using a predetermined method, or the coefficients can be randomly assigned.

[0173] At step T2, the NN codec obtains a NN-based reference or universal codec EU_DU. The EU_DU codec is trained in a typical way: the cost function can be, for example, rate-distortion, the training set can include hundreds of videos representing the type of content that one might want to compress later, and the usual steps of inference and gradient backpropagation are applied. This produces EU and DU, which are called NN-based universal encoder and NN-based universal decoder (universal means that it can handle any type of video).

[0174] At step T3, the difference DDV(i) between the coefficients of the NN-based reference decoder DU(i) and the NN-based decoder DV(i) to be trained is calculated.

[0175] At step T4, a quantizer Q2 is applied to the difference value and Q2(DDV(i)) is obtained as output.

[0176] At step T5, dequantizer DQ2 is applied and DDV2(i) = DQ2(Q2(DDV(i))) is obtained.

[0177] At step T6, the signal DV2(i) is obtained, DV2(i) = DU – DDV2(i)

[0178] Then, during a step T7, the signals DV2(i) are used to train the NN-based decoder DV, and the NN-based encoder EV. A number of inferences are run during a step T8 until it is decided to back-propagate to update the coefficients of the NN-based decoder DV.

[0179] At step T9, a gradient is obtained from the obtained coefficients.

[0180] However, in order to preserve the gradients, these gradients are not applied to the quantized coefficients of the NN-based decoder at step T10, but to the unquantized coefficients after the optimization process. Therefore, the gradients are applied to the EV(i) and DV(i) coefficients for determining the updated coefficients. The coefficients are updated, resulting in EV(i+1) and DV(i+1), which are the encoder and decoder, respectively, that will be trained during the next mini-batch i+1.

[0181] Thus, as explained above, according to some embodiments, the training method comprises:

[0182] - Run inference on the trained NN-based decoder (DV) using quantized parameters,

[0183] - Back-propagating gradients based on the result of the inference to update coefficients of the NN-based decoder (DV), the updated coefficients of the trained NN-based decoder (DV) being unquantized.

[0184] At step T11, i is incremented and the next small batch is input to the NN codec EV(i+1), DV(i+1).

[0185] Therefore, according to Figure 5 In an implementation mode, the training method comprises:

[0186] - obtain the difference (DDV) between the neural network based reference decoder (DU) and the NN based decoder to be trained (DV),

[0187] - obtaining a dequantized version (DDV2) of said difference value, said dequantized version (DDV2) being further obtained after applying said quantization,

[0188] - obtaining a trained NN-based decoder (DV2) as a difference between a NN-based reference decoder (DU) and said dequantized version (DDV2),

[0189] - said running inference comprises a to-be-trained NN-based encoder (EV) and said obtained trained NN-based decoder (DV2).

[0190] Figure 6 The present invention is shown in accordance with the embodiments of the present disclosure and is specifically based on Figure 3 and Figure 4 A global overview of the example training method or computer-implemented training method given in .

[0191] Figure 6 A method for training a decoder (DV) based on a neural network is described, the method comprising quantizing parameters representing coefficients of the neural network using a training quantizer (Q2) during said training.

[0192] As mentioned before, the method uses as input a signal V. As shown on the figure and explained before, the signal V is divided into a set of mini-batches V(i).

[0193] At step E1, a set of initialization coefficients of the NN codec is obtained to start the training phase, referred to as EV(0)_DV(0), for the NN encoder and the NN-based decoder, respectively. The initialization values ​​are initialization values ​​of the coefficients of the NN-based encoder and decoder. Initialization can be performed using a predetermined method, or the coefficients can be randomly assigned.

[0194] At step E2 , quantization of the parameters of the NN-based decoder is performed.

[0195] According to a first embodiment, the parameters may be coefficients of a NN-based decoder. More precisely, the parameters are coefficients of a NN-based decoder DV.

[0196] According to another embodiment, the parameter may be the difference between the coefficients of the NN codec EV_DV and the coefficients of the reference NN codec EU_DU. More precisely, the coefficient may be the difference between the coefficients of the NN-based decoder DV and the coefficients of the NN-based reference decoder DU.

[0197] In this embodiment, the method also obtains a reference (or universal) NN codec comprising a reference encoder and decoder pair EU_DU based on a neural network. The reference NN codec has been previously trained on a large number of input videos and can therefore be considered universal. Because it can process any kind of input video. The EU_DU codec is trained in a typical manner: the cost function can be, for example, rate distortion, the training set can include hundreds of videos representing the type of content that you may want to compress later, and the usual steps of inference and gradient back propagation are applied. This produces what are called universal encoders and universal EU and DU.

[0198] The quantization step E2 more precisely comprises a quantizer applied to the parameters followed by a dequantizer, so that the NN-based decoder DV is trained with already quantized parameters and thus the noise generated by the quantizer is input during the training phase, so that the signal on the decoding side is more efficiently decoded than without a quantizer during training.

[0199] These parameters are then further used during the inference run at step E3 after undergoing quantization and dequantization. Further after this set of inferences, the loss function is obtained and back-propagated to obtain the gradient of the loss with respect to the parameters of the network to be optimized at step E4.

[0200] A typical loss function used in video coding may be a rate-distortion cost, such as D + λ R, where D is the distortion measured between the input signal V and the corresponding decoded signal V', R is the rate estimated by the entropy of index I, and λ is a factor input by the user to select the tradeoff between compression and quality. While R includes the estimated bitrate of index i, it does not necessarily include the rate of the neural network-based decoder (DV). In practice, it is advantageous not to consider the rate of the neural network-based decoder, as this allows for faster training while having little impact on the overall compression performance, as this rate tends to be constant.

[0201] At step E5, the gradients obtained at step E4 are used to obtain updates to the network coefficients to be used for the next mini-batch. To update the coefficients, the gradients are applied to EV(i) and DV(i) instead of to the coefficients obtained further after step E2.

[0202] Thus, as explained above, according to some embodiments, the training method comprises:

[0203] - Run inference on the trained NN-based decoder (DV) using quantized parameters,

[0204] - Back-propagating gradients based on the result of the inference to update coefficients of the NN-based decoder (DV), the updated coefficients of the trained NN-based decoder (DV) being unquantized.

[0205] At step E6, a new mini-batch is obtained as input to the training method, and the new mini-batch is used.

[0206] Once N mini-batches have been used, training ends.

[0207] Fig. 7A and Figure 7B Two different embodiments of decoders for decoding signals generated, for example, by encoders 2A and 2B, respectively, are shown.

[0208] refer to Fig. 7A , decoder D1 receives as input the bitstream F produced by encoder E1 (as explained above).

[0209] The decoder D1 may comprise entropy decoding of the bitstream F in order to be able to decode the data. However, as already indicated, this aspect will not be described in detail here.

[0210] The bitstream F comprises several parts, and at least a coded signal TD and a coded signal VV. The coded data VV may comprise a plurality of data units respectively associated with different images of a sequence of images. (In other words, each of these parts of the bitstream may correspond to a sequence of images).

[0211] The decoder D1 comprises a receiving interface (Rx interface) for receiving a signal and optionally applies a first decoding stage before transmitting the signal to the analyzer. The receiving interface can remove the network protocol layer for the incoming signal. The analyzer can identify in the bitstream F the different data that make up the bitstream. Furthermore, the analyzer can also be designed to identify data units within the bitstream F and transmit each data unit to the NN-based decoder DV.

[0212] The network decoder is designed to decode parameters corresponding to the TD signal encoded by the network encoder of the encoder E1 included in the bitstream F.

[0213] In this embodiment, the TD signal comprises coefficients of the NN-based decoder DV that has been trained by the encoder E1 according to the present disclosure (signal DV2, as previously described).Once decoded, the parameters are applied to the NN-based decoder DV to configure the NN-based decoder DV.

[0214] The network decoder is designed to decode the parameters corresponding to the TD signal encoded by the network encoder of the encoder E2 included in the bitstream F. The network decoder comprises several components, more precisely a dequantizer DQ(1) capable of dequantizing the received parameters of the NN-trained decoder TD and obtaining a NN-based decoder DV trained by DV2.

[0215] The NN-based decoder DV decodes the data units to obtain a two-dimensional representation IMG' of the current image, and the video stream V' is constructed as an image sequence IMG'.

[0216] refer to Figure 7B , the decoder D2 receives as input the bitstream F produced by the encoder E2 (as explained above).

[0217] The decoder D2 may comprise entropy decoding of the bitstream F in order to be able to decode the data. However, as already indicated, this aspect will not be described in detail here.

[0218] The bitstream F comprises several parts, and at least a coded signal TD and a coded signal VV. The coded data VV may comprise a plurality of data units respectively associated with different images of a sequence of images. (In other words, each of these parts of the bitstream may correspond to a sequence of images).

[0219] The decoder D2 comprises a receiving interface (Rx interface) for receiving the signal and optionally applies a first decoding stage before transmitting the signal to the analyzer. The receiving interface can remove the network protocol layer for the incoming signal. The analyzer can identify in the bitstream F the different data that make up the bitstream. Furthermore, the analyzer can also be designed to identify the data units within the bitstream F and transmit each data unit to the NN-based decoder DV.

[0220] The network decoder is designed to decode the parameters corresponding to the TD signal encoded by the network encoder of the encoder E2 included in the bitstream F. The network decoder comprises several components, more precisely a dequantizer DQ(1) capable of dequantizing the received parameters of the trained NN-based decoder TD and obtaining a NN-based decoder DV trained by DDV2.

[0221] In this embodiment, the TD signal includes parameters of the NN-based decoder DV that has been trained by the encoder E1 according to the present disclosure (signal DDV2, as described above). These parameters are the difference between the coefficients of the NN codec EV_DV and the coefficients of the reference NN codec EU_DU. In order to obtain the coefficients DV2 of the NN-based decoder DV, the parameters DDV2 are subtracted from the coefficients of the NN-based decoder DU. The obtained DV2 parameters are then used to configure the NN-based decoder DV. The NN-based decoder DV decodes the data unit to obtain a two-dimensional representation IMG' of the current image, and the video stream V' is constructed as an image sequence IMG'.

[0222] Figure 8 Decoders according to some embodiments of the present disclosure are shown, as well as for example Fig. 7A and Figure 7B The hardware architecture of decoder D1 or D2.

[0223] To this end, the decoder has the hardware architecture of a computer. Figure 3 As shown, the decoder includes a processor 1. Although shown as a single processor 1, two or more processors may be used depending on the specific needs, expectations or specific implementations of the decoder. In general, the processor 1 executes instructions and manipulates data to perform the operations of the decoder and any algorithms, methods, functions, processes, flows and procedures described in this disclosure.

[0224] The decoder may also include communication means 5. Figure 35 is shown as a single communication device 5, but two or more communication devices may be used according to the specific needs, expectations or specific implementations of the decoder. The communication device may be used by the decoder to communicate with another electronic device linked to it in communication, and for example, an encoding device. Typically, the communication device 5 may include logic encoded in software, hardware, or a combination of software and hardware. More specifically, the communication device 5 may include software supporting one or more communication protocols associated with the communication, to transmit physical signals within and inside the decoder shown.

[0225] The decoder may further include a random access memory 2, a read-only memory 3 and a non-volatile memory 4. The read-only memory 3 of the decoder constitutes a recording medium consistent with the present disclosure, which can be read by the processor 1 and has recorded thereon a computer program PROG_DN consistent with the present disclosure, the computer program containing instructions for decoding a signal encoded by the encoder to perform the steps of the method for training a neural network-based decoder according to the present disclosure.

[0226] The program PROG_DN for example defines the functional modules of the decoder based on or controlling the above-mentioned elements 1 to 5 of the decoder, and these functional modules include in particular:

[0227] - receiving interface Rx,

[0228] - an analyzer for analyzing an input signal and identifying different components of said signal,

[0229] - a NN-based decoder DV for decoding an input video signal VV into an output signal V',

[0230] a network decoder for decoding the trained NN-based decoder and providing decoded coefficients (DV2) to the trained NN-based decoder DV,

[0231] In some embodiments, the defined module may also include an NN-based reference decoder DU, which is used to obtain coefficients (DDV2) of the NN-based decoder DV from differential encoding parameters of the trained NN-based decoder DV when differential encoding is performed relative to the decoder DU as described above.

[0232] The present disclosure also relates to a bitstream comprising encoding parameters representing a neural network based decoder trained according to the training method of the present disclosure.

[0233] The present disclosure also relates to a device for encoding at least parameters representing coefficients of a neural network-based decoder into a bitstream, wherein, according to any one of the embodiments of the present disclosure, the neural network-based decoder is trained using the training device of the present disclosure. In some embodiments, the device may be a device for transmitting data, such as a broadcast device.

[0234] The present disclosure also relates to a device for generating a coded bit stream, the device comprising

[0235] - a neural network based encoder and decoder (DV) of the invention encoded with the present disclosure according to any of the embodiments of the present disclosure, and a neural network based decoder encoded with the present disclosure according to any of the embodiments of the present disclosure, and

[0236] - a signal encoded using said neural network based encoder (EV). In some embodiments, the device may be a device for transmitting data, such as a broadcasting device.

[0237] The present disclosure also relates to a bitstream comprising encoding parameters representing an NN-based decoder trained according to the training method of the present disclosure, and an encoded video signal encoded with an NN-based encoder, wherein the NN encoder and the NN decoder have been trained using the video signal.

[0238] The invention also relates to a device for training a NN-based decoder (DV), comprising means for quantizing parameters representing coefficients of said neural network-based decoder (DV) using a training quantizer (Q2).

[0239] According to some embodiments, means for quantizing may include a quantizer Q2 and a dequantizer DQ2 and additional logic circuitry to provide the quantization functionality on the NN-based decoder.

Claims

1. A computer-implemented method for training a neural network-based decoder (DV), the method comprising quantizing parameters representing coefficients of the neural network-based decoder (DV) using a training quantizer (Q2) during said training, wherein: The parameters are coefficients of the neural network based decoder.

2. A method according to any one of the preceding claims, comprising: - running inference on the training of the neural network based decoder (DV) using said quantized parameters, - back-propagating gradients based on the result of the inference to update coefficients of the neural network-based decoder (DV), the updated coefficients (DDV2, DV2) of the trained neural network-based decoder being unquantized.

3. The method according to claim 2, wherein: The method includes: - obtaining the difference (DDV) between the coefficients of a reference decoder based on a neural network (DU) and the coefficients of said decoder based on a neural network (DV) to be trained, - obtaining a dequantized version (DDV2) of said difference value, said dequantized version (DDV2) being further obtained after applying said quantization, - obtaining a trained neural network based decoder (DV2) as the difference between the coefficients of said reference neural network decoder (DU) and the coefficients of said dequantized version (DDV2), - using the neural network based encoder to be trained (EV) and the obtained trained decoder (DV2) to perform said running inference.

4. A computer-implemented method for encoding at least parameters representing coefficients of a neural network-based decoder (DV) into a bitstream (TD), wherein: The neural network based decoder (DV) is trained using a training method as claimed in any of the preceding claims.

5. The encoding method according to claim 4, comprising: quantizing the parameters representing the coefficients (DDV2, DV2) of the trained neural network-based decoder using a quantizer (Q1), the quantizer (Q1) - equal to the trained quantizer (Q2), or - is the quantizer whose reconstruction level includes the reconstruction level of the trained quantizer (Q2).

6. The encoding method according to claim 4, comprising: The parameters representing coefficients (DDV2, DV2) of the trained neural network based decoder are quantized using a quantizer (Q1), the training quantizer (Q2) being a function of distortion and the quantizer (Q1) being a function of rate distortion.

7. The encoding method according to claim 4, comprising: The parameters representing coefficients (DDV2, DV2) of the trained neural network based decoder are quantized using a quantizer (Q1), the trained quantizer (Q2) minimizing distortion and the quantizer (Q1) minimizing rate distortion.

8. The encoding method according to any one of claims 4 to 7, wherein: The encoding complies with - Neural Network Representation (NNR) or -Neural Network Exchange Format (NNEF) or -Open Neural Network Exchange (ONNX).

9. A computer-implemented method for generating a coded bitstream, the method comprising: - a neural network decoder encoded according to the method according to any one of claims 6 to 10 of a neural network based encoder and decoder, and - A signal encoded using said neural network based encoder.

10. The method according to claim 9, wherein: The bitstream is generated by concatenating the encoded neural network-based decoder (DV) and the encoded signal.

11. A device for training a neural network based decoder (DV), the device comprising one or more processors configured, individually or in combination, to quantize parameters representing coefficients of the neural network based decoder (DV) using a training quantizer (Q2) during said training, 13 wherein, The parameters are coefficients of the neural network based decoder (DV).

12. The device according to claim 11, further configured to: - running inference on the training of the neural network based decoder (DV) using said quantized parameters, - back-propagating gradients based on the result of the inference to update coefficients of the neural network-based decoder (DV), the updated coefficients of the trained neural network-based decoder (DV) being unquantized.

13. The device according to claim 12, further configured to: - obtaining the difference (DDV) between the coefficients of a reference decoder based on a neural network (DU) and the coefficients of said decoder based on a neural network (DV) to be trained, - obtaining a dequantized version (DDV2) of said difference value, said dequantized version (DDV2) being further obtained after applying said quantization, - obtaining a trained decoder (DV2) as the difference between the coefficients of said neural network based reference decoder (DU) and the coefficients of said dequantized version (DDV2), - using the neural network based encoder to be trained (EV) and the obtained trained neural network based decoder (DV2) to perform said running inference.

14. A device for encoding at least parameters representing coefficients of a decoder (DV) based on a neural network into a bitstream (NN), wherein: The neural network based decoder (DV) is trained using a training device as claimed in claim 11.

15. A device for generating a coded bit stream, the device comprising: - a neural network-based encoder (EV) and a decoder (DV) encoded with the encoding of the device according to claim 14; and - Encode the signal using the neural network based encoder (EV).

16. A computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 3, or the method according to claims 4 to 8, or the method according to claims 9 to 10.

17. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 3, or the method of claims 4 to 8, or the method of claims 9 to 10.

Images