Method and device for encoding and decoding images

Abstract

The invention concerns a method and device for encoding and decoding a signal comprising a plurality of samples. The decoding method comprises the following steps: - decoding (E31, FMD) a first group of feature maps (Fmdi); - decoding (E35, NND) a set of parameters (Wdk) representative of a neural network (MLP'); - for at least one sample, called the current sample (Pdn), of the signal to be decoded, associated with a position (xn, yn) in the signal to be decoded: - constructing (E33, E34, XTR', TT ') a feature vector (Zdn) from the feature maps of said first group (FMdi), as a function of said position (xn, yn) of said current sample, and: - processing (E36, MLP') said feature vector (Zdn) using an artificial neural network (MLP') defined by the decoded parameters (Wdk) to provide a vector representative of a decoded value of the current sample (Pdn).

Description

Description Title of the invention: Method and device for coding and decoding images. Prior art [1] The invention relates to the general field of coding single or multidimensional signals. It relates more particularly to the compression of digital images and video. [2] Digital images are generally subject to source coding aimed at compressing them in order to limit the resources required for their transmission and / or storage. There are many coding standards, such as, for still images, the JPEG family standards, and for animated images, or video, the ITU / MPEG standards (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) as well as their extensions (MVC, SVC, 3D-HEVC, etc.) [3] Encoding an image is generally performed by dividing the image into several rectangular blocks, and encoding these blocks of pixels according to a given processing sequence. In existing video compression techniques, the processing of a block typically includes a prediction of the pixels of the block carried out using previously coded and then decoded pixels present in the image being encoded, in which case we speak of "Intra prediction", or previously coded images, in which case we speak of "Inter prediction". This exploitation of spatial and / or temporal redundancies makes it possible to avoid transmitting or storing the value of the pixels of each block of pixels, by representing at least some of the blocks by a residual representing a difference between the prediction values ​​of the pixels of the block and the actual values ​​of the pixels of the predicted block. [4] As video formats continue to evolve to compress more and more and adapt to the variety of expected formats and communication networks, the possibilities for prediction are becoming ever greater and classic coding and decoding algorithms very complex. [5] In addition to these classic approaches proposed by compression standards (JPEG, MPEG, ITU), approaches based on artificial intelligence, and in particular neural ones, tend to develop. [6] Some of these neural approaches can be seen as a simple extension of the notion of competition of the aforementioned compression techniques, such as the prediction and transformation mode competition in video coding. [7] Other approaches use the concept of "autoencoder". Autoencoders are learning algorithms based on artificial neural networks, which allow the construction of a new representation of a dataset. The architecture of an autoencoder consists of two parts: the encoder and the decoder. The encoder consists of a set of layers of neurons, which process the data in order to construct new representations called "encoded", also called "latent representations". In turn, the layers of neurons in the decoder receive these representations and filter them in order to try to reconstruct the initial data. The differences between the reconstructed data and the initial data make it possible to measure the error made by the autoencoder. Training consists of modifying the parameters of the autoencoder in order to reduce the reconstruction error measured on the different samples of the dataset.The performance of such autoencoder-based systems comes at the cost of a considerable increase in memory footprint and complexity compared to conventional approaches as proposed by compression standards. They can have millions of parameters and may require up to a million MACs (multiply-accumulate) to decode a pixel. This makes such decoders significantly more complex than conventional decoders, which could hamper the adoption of learning-based compression. [8] More recently, a simple neural network-based encoding technique was described in the article "Compression with Implicit Neural representations" by Emilien Dupont et al. (arXiv:2103.03123). The proposed encoding technique consists of fitting a neural network to an image, quantizing the network weights, and transmitting them. At decoding time, the neural network is evaluated at each pixel position to reconstruct the image. However, such a technique remains inefficient in terms of compression. There is therefore a need for a solution that allows an image or sequence of images to be encoded / compressed in a simple and efficient manner. Statement of the invention [9] The invention relates to a coding method according to claim 1 and a decoding method according to claim 8.

[0010] For the purposes of the invention, encoding, or "coding", means the operation which consists of representing a set of samples in a compact form carried for example by a digital binary train. Decoding means the operation which consists of processing a digital binary train to restore decoded samples.

[0011] A signal "sample" means a value taken from the signal. Sampling the signal produces a series of discrete values ​​called samples. In the case of an image signal, the sample is called a pixel, which can be, for example, a color pixel traditionally represented by a triplet of values, for example (R, G, B) or (Y, U, V). Generally speaking, a sample is represented by a vector of one or more components, or data.

[0012] A "signal comprising a plurality of samples" means a signal with one (audio, sound), two (image) or more than two (stereoscopic, multiscopic image, image associated with a depth map, video, etc.) dimensions. Depending on this dimensionality, the sample has one, two or more coordinates in the signal. In the case of an image signal, the position of the sample is identified by its abscissa (x) and ordinate (y) coordinates.

[0013] By "feature maps" we mean an abstract representation of the signal comprising a plurality of variable data, discrete or not, which are also called values, for example real or integer numbers. As is known, these maps are also referred to as the "latent representation" of the input signal.

[0014] By "data characteristic vector constructed from the characteristic maps as a function of a position" is meant a vector consisting of one or more elements, or data, preferably discrete, the data being constructed from the characteristic maps at a position determined by the position of the sample being processed in the signal. This characteristic vector is the one that is applied to the input of the neural network. For example, in the case of a one-dimensional audio signal, such a vector can be constituted from a plurality of values ​​taken from each of the characteristic maps at the same coordinate as the sample to be coded. In the case of an image, such a vector can be constituted from a plurality of values ​​taken from each of the characteristic maps at the same abscissa and ordinate coordinates as the sample to be coded.It can also be made up from a plurality of values ​​taken from each of the feature maps at a position that depends not only on the position of. the sample but also the size of the feature map. Once these values ​​are taken from the feature maps, they can be processed to form the feature vector, before input into the neural network, for example by quantization, filtering, interpolation, etc.

[0015] By "neural network" we mean a neural network such as a convolutional neural network, a multi-layer perceptron, an LSTM (for "Long Short Term Memory" in English), etc. The neural network is defined for example by a plurality of layers of artificial neurons which comprise a set of activation, weighting and addition functions (for example, a layer can calculate y = f(Ax+b), where y and b are vectors of dimension N, x is a vector of dimension M, A is a matrix of dimension MxN, and f is the activation function). Hereinafter we speak of "neural network" or "synthetic neural network".

[0016] By "neural network parameter" we mean one of the values ​​that characterizes the neural network, for example a weight associated with one of the neurons (filter coefficient, weighting, bias, value affecting the operation of non-linearity), etc.

[0017] By "processing by a neural network" is meant the application of a function expressed by a neural network to the input characteristic vector to produce an output vector representative of the sample to be encoded (resp. to be decoded). This output vector may include one or more data representative of the sample.

[0018] By "performance measurement" is meant a measurement between at least one value of a sample to be encoded and a decoded value of said sample. The measurement can evaluate, for example, a distortion, or a perceptual error. It can be carried out on a sample or a plurality of samples (for example, the complete image). The measurement can also include a measurement of the flow rate, in particular associated with the coding of the neural network and / or the coding of the characteristic maps. The measurement can be a joint measurement between the flow rate and the distortion through their weighting. As is well known in the state of the art, the value of this measurement is generally minimized until a target value is reached.

[0019] A "construction step" means a step that aims to construct the representative parameters of the image, before their actual coding. The construction sub-steps can be repeated as many times as necessary to obtain an acceptable performance measurement.

[0020] Generally speaking, it is considered that the steps of an coding or decoding method should not be interpreted as being linked to a notion of temporal succession. In other words, the steps may be carried out in a different order than that indicated in the independent coding or decoding claim, or even in parallel.

[0021] The coding method according to the invention constructs the coding parameters, from the input signal, for example an image, by training a neural network on characteristic vectors associated with a position of a sample to be coded. These characteristic vectors are constructed from characteristic maps. During the training, or construction, the parameters of the neural network and the values ​​of the characteristic maps are updated according to a performance measurement, for example of the rate-distortion type. When the training is finished, that is to say when the performance measurement obtained is satisfactory, the actual coding of the parameters of the neural network and / or the values ​​of the characteristic maps can be carried out and stored or transmitted to the decoder.

[0022] Advantageously, the training process makes it possible to refine the parameters of the neural network and / or the values ​​of the feature maps until an adequate representation in terms of performance is obtained, for example a desired balance between the generated bit rate and the distortion suffered by the input signal. The training of the values ​​of the feature maps and the parameters of the neural network can be joint. Advantageously, the coding method according to the invention makes it possible to efficiently compress the signal.

[0023] Advantageously, the decoding process is simple since it is sufficient to decode the feature maps and the neural network to reconstruct a decoded version of the signal, for example an image.

[0024] Such a neural network can advantageously be of very simple structure with few parameters.

[0025] Moreover, decoding can be done sample by sample, in a progressive manner.

[0026] According to embodiments of the coding and / or decoding method:

[0027] - The method comprises a step of constructing a second group of feature maps and the feature vector is further constructed from said feature maps. characteristics. Advantageously, these additional maps of the second group, constructed identically to the encoder and the decoder, are neither stored nor transmitted at the encoder level, nor decoded at the decoder level. They thus make it possible to benefit from additional data to improve compression without degrading the flow rate. They can, for example, include coordinates, data available in the maps of the first group, data concerning other images already processed by the encoder or decoder, etc.

[0028] - At least one of the characteristic maps has a lower resolution than the signal to be coded (resp. decoded). Advantageously, according to this mode, the compression of the characteristic maps is more efficient since at least one of the characteristic maps has fewer values ​​than if it were at the resolution of the signal. For example, in the case of a digital image, one of the characteristic maps can have a resolution of 1 / 2, that is to say it has half as many values ​​on the abscissa and ordinate as the input signal has samples, i.e. in total 4 times fewer values ​​than a characteristic map at the resolution of the signal.

[0029] - At least one of said characteristic maps has a resolution identical to that of the signal to be coded (resp. decoded). Advantageously, according to this mode, at least one of the characteristic maps, at the same resolution as the input signal (resp. to be decoded), allows significant fidelity and respect for the details of the initial resolution of the signal.

[0030] - The construction of said characteristic vector comprises a sub-step of extracting a value from said at least one characteristic map at a position identical to that of the current sample in the signal.

[0031] Advantageously, it is thus possible to extract a value from a feature map of the first or second group, at the same position as the sample in the signal (input signal for coding, signal to be decoded for decoding), to constitute an element of the feature vector. This method is simple to implement. For example, if we have J feature maps as input, with the same resolution as the signal, a simple extraction of the values ​​of the maps at the coordinates of the current sample (at the same abscissa and the same coordinate in the feature map) makes it possible to directly construct the feature vector of J elements.

[0032] - The construction of said characteristic vector comprises the following sub-steps: - extraction of a plurality of values ​​from the feature maps based on of said position of said current sample; - processing of said extracted values ​​to obtain the characteristic vector.

[0033] Advantageously, it is possible to process the values ​​extracted from the feature maps to construct the feature vector. Such processing may, for example, correspond to a quantization of the extracted data, or to scaling, or to filtering, etc. During coding, quantization is essential for the proper functioning of the system if the feature maps contain, for example, floating point or real values. It is necessary to quantize them before coding them and / or providing them as input to the neural network. During decoding, however, inverse quantization is not necessary, depending on the embodiments.

[0034] - The coding (resp. decoding) of said first group of feature maps comprises an entropic coding (resp. decoding) sub-step. Advantageously, entropic coding makes it possible to exploit signal redundancies that may persist in the feature maps, the signal thus being compressed more efficiently.

[0035] Correlatively, the invention also relates to a coding device and a decoding device.

[0036] The characteristics and advantages of the coding or decoding method apply in the same way to the coding or decoding device according to the invention and vice versa.

[0037] The invention also relates to a computer program on a recording medium, this program being capable of being implemented in a computer or a control device in accordance with the invention. This program comprises instructions adapted to the implementation of the corresponding method. This program can use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0038] The invention also relates to an information medium or a recording medium readable by a computer, and comprising computer program instructions mentioned above. The information or recording media can be any entity or device capable of storing the programs. For example, the media can comprise a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a floppy disk or a hard disk, a DNA sequence, or a flash memory. On the other hand, the Information or recording media may be transmissible media such as an electrical or optical signal, which may be conveyed via an electrical or optical cable, by radio link, by wireless optical link or by other means.

[0039] The program according to the invention can in particular be downloaded from an Internet-type network.

[0040] Alternatively, each information or recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of a method according to the invention. Brief description of the drawings

[0041] Other characteristics and advantages of the present invention will emerge from the description given below, with reference to the appended drawings which illustrate exemplary embodiments thereof which are not limiting in nature. [Fig. 1] Figure 1 schematically represents a coding device used in the context of the invention; [Fig. 2] Figure 2 schematically represents a decoding device used in the context of the invention; [Fig. 3] Figure 3 illustrates an example of a synthetic artificial neural network used in the context of the invention for coding and decoding; [Fig. 4] Figure 4 is a flowchart representing an example of an encoding method that can be implemented by the encoding device of Figure 1; [Fig. 5] Figure 5 illustrates an encoding method used in one embodiment of the invention; [Fig. 6] Fig. 6 illustrates an encoding method used in another embodiment of the invention; [Fig. 7] Figure 7 is a flowchart representing an example of a decoding method that can be implemented by the decoding device of Figure 2; [Fig. 8] Figure 8 illustrates a decoding method used in one embodiment of the invention. Description of the embodiments

[0042] Figure 1 schematically represents an ENC coding device.

[0043] This ENC coding device includes a GEN module for generating feature maps, a TT processing and quantification module, an XTR data extraction module, an MLP module corresponding to an artificial neural network, an NNC neural network coding module, an FMC module for coding feature maps, an EVAL module for evaluating a performance function, an MA J update module, and an optional FME module for generating additional feature maps.

[0044] The ENC coding device may be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the above-mentioned modules may then be realized by the cooperation of the processor and computer program instructions stored in the above-mentioned memory and designed to perform the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0045] The ENC coding device in Figure 1 receives as input a succession of samples to be coded, noted P n , for example a temporal succession of sound samples, or a set of image data noted I(P n ). In this second case, the image signal I(P n ) can represent a two-dimensional image, or a plurality of two-dimensional images (video, color components, stereoscopic, multiscopic components, etc.). P ndenotes a sample n of the input signal comprising N samples. In one embodiment, the signal is a color image signal represented by means of at least one two-dimensional representation, such as a matrix of pixels, of width W and height H (with N = WxH) each pixel comprising a red (R), green (G), blue (B), or, alternatively, a luminance (Y) component and at least one chrominance component. The location of each pixel is defined by its abscissa and ordinate (x and y) coordinates in the image. In one embodiment, the image is in grayscale represented by means of a two-dimensional representation, such as a matrix of pixels, each pixel comprising a grayscale, or luminance, component. In this case the vector representing the pixel is reduced to a single component, or data.

[0046] As will be described in more detail later with reference to Figures 3 to 8:

[0047] The GEN module for generating characteristic maps is configured to generate, and in particular initialize, a plurality of M characteristic maps denoted FMi. The optional FME module can generate one or more additional maps, L in number, which will not be coded or transmitted, denoted FMEi.

[0048] The XTR module performs an extraction of values ​​in the FMi and optionally FMEi characteristic maps, for a sample P n to be encoded, depending on its coordinates in the input signal.

[0049] The TT module performs processing of the extracted values, to generate the Z vector n . Z n is a J-tuple, that is, it has J elements, or data Zi. The vector Z n of index n refers to the characteristic vector of the pixel P' n .

[0050] The TT module performs a quantification of the data extracted from this set of M FMi maps, or from the vector Z n constituted from this data. To prepare for subsequent coding (e.g., entropy-based), the coding system quantizes the latent representation of the input data by using a quantizer Q to generate an ordered collection of quantized values. Recall that quantizing a value refers to mapping that value to a member of a discrete set of possible code symbols. For example, the set of possible code symbols may consist of integer values, and the quantization system performs simple rounding from a real value to an integer value. In another example, quantization consists of multiplying by a given value and then rounding.

[0051] The MLP module is a neural network defined by K parameters Wk, capable of processing the vector Zn , or J-Uplet, as input, to generate as output a second vector representative of the sample P n to be coded. The neural network is, according to one embodiment, an MLP, or Multi Layer Perceptron, consisting of an input layer adapted to the input format (the J-tuple), optionally one or more hidden layer(s), and an output layer adapted to the output format of the output vector, generally a vector comprising A elements. According to one embodiment, A is equal to 3 and the output vector is the triplet (R, G, B) of the pixel P'n coded then decoded.

[0052] The NNC module performs the coding of the neural network, in particular its Wk parameters. During the process of training, or building, the coding, that is to say as long as the stage of evaluating a performance is not satisfactory, the NNC module performs a coding simulation, followed by a decoding, intended for the evaluation module. It updates the values ​​of the parameters according to the results of a performance measurement carried out by the EVAL module. Subsequently, it performs the effective coding of the parameters of the neural network Wk. The coded parameters are denoted Wck. As is known, the coding simulation can be identical to the actual coding, or an approximation of it.

[0053] The EMC module performs the coding of the EMi maps, i.e. the quantized values ​​of each map of the first group (excluding the FMEi maps). During the process of training, or building, the coding, i.e. until the performance evaluation step is satisfactory, the FMC module performs a coding simulation, followed by a decoding, intended for the evaluation module. Subsequently, it performs the actual coding of the values ​​of the FMi maps. The coded maps are denoted FMci. In a known manner, the coding simulation can be identical to the actual coding, or produce an approximation thereof. The coding module compresses the quantized data using, for example, entropy-type coding.

[0054] The EVAL module performs an evaluation and minimization of a coding performance. The evaluation function is for example of the rate-distortion type. The minimization can be carried out by gradient descent, or any other method within the reach of a person skilled in the art.

[0055] The MAJ module updates the values ​​of the FMi cards to be encoded based on the results of the performance function. Optionally, it can update the values ​​of the additional FMEi cards.

[0056] Figure 2 schematically represents a DEC decoding device.

[0057] The DEC coding device of Figure 2 receives as input a first group of encoded data organized into M FMci feature maps (also called FM layers) and the encoded Wck parameters of the neural network

[0058] This DEC decoding device includes a neural network decoding NND module, a feature map decoding FMD module, a data extraction XTR' module, an inverse processing TT' module, a neural network corresponding MLP' module, an optional additional feature map generation FME' module.

[0059] The maps decoded by the FMD module, numbering M, are denoted FMdi. The parameters decoded by the NND module are denoted Wdk.

[0060] The decoder's FME' module can also generate one or more additional cards, denoted FME'i, L in number, identical to the additional FMEi cards generated by the encoder.

[0061] The XTR' module is identical to the XTR module in Figure 1. It performs an extraction of values ​​from the M decoded FMdi feature maps, and possibly from the L additional FME'i maps, to construct the vector Z ntaking the form of a J-tuple for a sample P n to be decoded, depending on its coordinates in the signal to be decoded. In one embodiment, J=M. In one embodiment, J=M+L.

[0062] The TT' module optionally performs an inverse quantization corresponding to the quantization performed at the encoder. The inverse quantization is not necessary if the encoder's quantizer Q has performed a simple rounding of the real values ​​submitted to it. The inverse quantization is not necessary if the neural network is able to take into account a quantization of its input data. Otherwise, the decoder performs the inverse operation of the quantizer Q. Furthermore, like the encoder's TT module, the TT' module optionally performs a processing of the values ​​extracted from the feature maps, to generate the decoded vector Zd n .

[0063] The MLP' module is a neural network defined by K parameters Wdk, capable of processing the vector Zd n , or J-Uplet, as input, to generate as output a second vector representative of the sample P n to decode, generally a vector comprising A elements. According to one embodiment, A=3 and the output vector is the triplet (R, G, B) of the pixel Pd n decoded. The MLP' module is identical in structure to the MLP module, and its parameters are either identical if the coding of its Wk parameters is lossless, or different if the coding is lossy.

[0064] When all samples P n of the signal have been decoded, we have a reconstructed signal I(Pd n ), according to an example an image I comprising N samples decoded in the form of N vectors Pd n .

[0065] The DEC decoding device may be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the above-mentioned modules may then be realized by the cooperation of the processor and computer program instructions stored in the above-mentioned memory and designed to perform the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0066] Figure 3 illustrates an example of a synthetic artificial neural network used for encoding and decoding in embodiments of the invention.

[0067] The MLP synthetic artificial neural network for coding and the MLP' synthetic artificial neural network for decoding are defined by an identical structure, comprising for example a plurality of layers of artificial neurons, and by a set of weights and activation functions associated respectively with the artificial neurons of the network concerned.

[0068] A vector representation of a current sample (a vector Z n or Zd n from the feature maps FMi and FMEi or FMdi and FME'i) is applied as input (i.e. on an input layer) of the MLP or MLP' synthetic artificial neural network. The artificial neural network produces as output a vector P' n or Pd n representative of the decoded sample, according to one embodiment the color components (R, G, B) constituting a color pixel of an image.

[0069] The concatenation of all these reconstructed pixels in an image (2D, 3D) constitutes the decoded, or reconstructed, image.

[0070] At the encoder, the MLP synthesis artificial neural network is trained on the image, so as to minimize the differences between the input representation of the current image I(P n ) and its representation I(P' n ) output, while also minimizing the amount of data to be encoded. The EVAL module performs a performance measurement in this regard.

[0071] Once training is complete, the network parameters are encoded, either lossless, in which case the MLP' neural network is identical to MLP, or lossy, in which case the MLP' network may be slightly different from MLP.

[0072] Figure 4 is a flowchart representing an example of an encoding method that can be implemented by the encoding device of Figure 1.

[0073] According to this embodiment, the signal is a two-dimensional image, each sample to be coded is therefore a pixel P n of coordinates (x n , y n ).

[0074] Encoding takes place in two main phases:

[0075] In a first phase, called the construction phase, learning is carried out in order to determine, for an input signal I(P n ), the values ​​of the FMi maps and the Wk parameters to optimize a global cost function. The learning is for example carried out by gradient descent, followed by an update of the parameters of the MLP neural network and the values ​​of the FMi feature maps. As is known in the state of the art, the cost function can be of the rate-distortion type, or rate, or distortion, or perceptual. To measure the rate R, it is necessary to simulate the coding of the FMi maps, then to measure the associated coding rate (the size of the flow B 1). According to one embodiment, the coding of the parameters Wk is not simulated because their influence is less important than that of the feature maps. According to one embodiment, the coding of the parameters Wk is also simulated and the associated rate is measured (the size of the flow B2). To measure the distortion D, it is necessary to simulate the coding then the decoding of at least part of the image I, to obtain at least one pixel P' nresulting from a simulation of coding then decoding, then measuring the difference between this part of the image I(P n ) as input and a corresponding part of the image I(P' n ) encoded then decoded.

[0076] Then, in a second phase, called the coding phase, the FMi maps and the Wk parameters are encoded to produce the coded values ​​FMci and Wck before transmission or storage. They constitute the compressed representation of the input signal I (P n ).

[0077] We will now describe the steps of a method according to one embodiment of the invention.

[0078] During a step E20, a signal I(P n ) to be coded, comprising a plurality of N samples P n , is provided as input to the process.

[0079] In step E21, the M FMi maps of the first group are initialized by the GEN module. Subsequently, the Wk parameters of the MLP neural network and the values ​​of the FMi maps must be optimized during the construction phase.

[0080] According to one embodiment, the FMi cards are of the same resolution as the input signal I(P n ) and therefore each have the same number of values ​​N as there are samples P n to code.

[0081] According to one embodiment, the FMi cards have a resolution less than or equal to that of the input signal I(P n ) and therefore include, for at least one of them, a number N' of values ​​to be coded less than N.

[0082] According to one embodiment, the first FMi card is at the resolution of the signal and each subsequent card is at half the resolution of the previous one.

[0083] According to one embodiment, several FMi cards have the same resolution, lower than the signal resolution.

[0084] According to one embodiment, the FMi cards are initialized by predefined constant values.

[0085] According to another embodiment, the feature maps are initialized by a set of random real values.

[0086] According to one embodiment, one or more FMEi maps, forming a second group of L additional feature maps, are generated and added to the first group. They are used to construct the feature vector but will not be stored or transmitted.

[0087] These FMi characteristic maps of the first group are subsequently updated, or refined, during a step E22 by the MAJ update module of the encoder during its learning during the construction phase.

[0088] In a step E23, the FMi maps of the first group are coded by the FMC module of the encoder. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is an actual coding and the coded values ​​constitute the stream Bl. The simulation can be identical to the actual coding but it can also be different (for example, simplified). For this coding, any known technique aimed at compressing the values ​​of the maps can be used.

[0089] In one embodiment, the FMi cards are coded in order (FMi, FM2,..., FM4), and the variables of each card in a predefined order, for example lexicographic. Each card undergoes entropy coding. The entropy coding produces a compressed stream Bl whose flow rate is subsequently measured during a step E28.

[0090] During a step E24, values ​​are extracted by the XTR module from the FMi and possibly FMEi cards. This extraction is carried out according to the coordinates (x n , y n ) of sample P n of the input signal.

[0091] According to one embodiment, the coordinate values ​​(x n , y n ) of each card are extracted.

[0092] According to one embodiment, the coordinate values ​​are scaled to extract the lower resolution layers.

[0093] According to one embodiment, values ​​close to those previously cited are also extracted for the subsequent processing of step E25.

[0094] At a step E25, a vector Z n is constructed by the TT module from the values ​​extracted from the FMi and possibly Fme maps for each sample P n of coordinates (x n , yn ) of the input signal. The processing includes quantization of the values ​​extracted from the FMi maps, if necessary. The processing may include other operations, such as filtering, scaling, application of any function, preferably monotonic, etc.

[0095] In one embodiment, Z n has as many values ​​as there are input cards or FMi cards (and optionally FMEi). In this case we have J=M(+L).

[0096] In one embodiment, Z n is a J-uplet (zi, Z2,.. ., zj), consisting of the values ​​of the FMi (and optionally FMEi) maps located at the coordinates (x n , y n ) of the current pixel P n , as will be illustrated in Figure 5. The value located at the coordinates (x n , y n ) of the FMi map gives the final value (zi) of element i of the vector Z nrelating to this FMi or FMEi card. In this case the TT module does not apply any processing other than quantification.

[0097] In one embodiment, Z n is a J-tuple constructed from values ​​taken from the FMi (and optionally FMEi) maps at coordinates that may be different depending on the maps. For example, if the FMi (and / or FMEi) maps are at different resolutions because they have been downsampled, the coordinates are adapted (by scaling) to match the resolution of each map, as will be illustrated in Figure 6.

[0098] In one embodiment, Z nis a J-tuple constructed from values ​​taken from the FMi (and FMEi) maps by applying processing to one or more values ​​of the maps, for example filtering the values ​​close to the targeted value in a map. For example, in an FMi map that is at the same resolution as the input signal, we can extract the values ​​located at the coordinates (x n , y n ), (x n -l, y n ), (x n , y n -l) and (x n -l, y n -l) and apply processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (zi) of element i of vector Z n relative to this FMi or FMEi card. According to another example, in an FMi card which is at half the resolution of the input signal, we can consider the values ​​located at the coordinates (x n / 2, y n / 2), (x n / 2-l, y n / 2), (x n / 2, y n / 2-l) and (xn / 2-l, y n / 2-l) and apply processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (zi) of element i of vector Z n relating to this FMi or FMEi card.

[0099] The samples to be coded are, for example, processed in sequential order, from n=1 to n=N.

[0100] During a step E26, the vector Z n is processed by the MLP neural network to generate as output a vector representative of the sample P n to code, according to one embodiment, the triplet (R, G, B) of the sample P' n (sample P n coded then decoded).

[0101] The structure and Wk parameters of the neural network are initialized, for example, during the first iteration of this step. These parameters are subsequently updated, or refined, during the construction phase, during subsequent iterations of the process.

[0102] According to one embodiment, the parameters of the neural network are initialized by predefined values ​​known to give a satisfactory result (for example, following training on a corpus of images).

[0103] According to another embodiment, the parameters Wk of the neural network are initialized by a set of random values.

[0104] In step E27, the Wk parameters of the MLP neural network are quantized and coded. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is an actual coding and the coded values ​​constitute the B2 stream. The simulation can be identical to the actual coding but it can also be different (for example, simplified). Any known technique can be used for this purpose, for example the neural network coding standard proposed by the MPEG-7 part 17 standard, also called NNR (Neural Network Representation). Note that in this case, the amount of degradation that the coding brings to the Wk parameters must be chosen.

[0105] During step E28, a performance measure is evaluated.

[0106] For this purpose, the coding simulation rates associated with the feature maps (simulation of the B1 flow by coding and decoding of the FM0 maps and optionally with the parameters of the neural network (simulation of the B2 flow by coding and decoding of the Wk parameters) are measured.

[0107] According to one embodiment, the cost function is of the rate-distortion type, noted (D+L*R), where D is for example the quadratic error measured between the input signal and the decoded signal (or the error measured on a subset of samples of the signal). According to another example, D is calculated from a perceptual function such as the SSIM (for Structural SIMilarity), or MSSSIM (for Multi-scale Structural SIMilarity). According to one embodiment, R is the simulated rate of the stream B1; according to another embodiment, R is the total rate used to code this image, that is to say the sum of the simulated rates of B1 and B2. L is a parameter that adjusts the bitrate-distortion tradeoff. Other cost functions are possible.

[0108] As long as the cost function has not reached its minimum, the performance measure is not satisfactory, and the method is repeated from step E22. This minimization can be performed by a mechanism known as gradient descent with parameter updates during step E22 for the feature map values ​​and E26 for the network parameters.

[0109] In a step E29, if the cost function has reached its minimum, the training stops. If an encoded version corresponding to the last simulation of the parameters of the neural network (Wk) and the feature maps (FMi) is available, the streams B1 and B2 can be formed from it. According to another embodiment, the actual encoding of the updated parameters of the neural network (Wk) and the values ​​of the feature maps (FMi) is carried out in this step to produce the encoded parameters Wck and FMci which constitute streams B1 and B2.

[0110] The streams B1 and B2 can be concatenated to produce a final stream BS. According to one embodiment, the stream B2 of the encoded parameters of the neural network is stored or transmitted before the stream B1, so that it can be decoded before the stream B1.

[0111] Figure 5 illustrates an encoding method used in one embodiment of the invention for encoding an image I(P n ) (not shown).

[0112] In this embodiment, there are 4 FMi cards. In a preferred embodiment, there are 7. The FMi cards have the same resolution as the image I(P n ), and therefore each have WxH values, where W represents the width of the image in pixels and H its height (N=WxH).

[0113] According to a variant presented in dotted lines, the feature maps are 5 in number: an additional FMEo map has been introduced, which will neither be stored nor transmitted. This additional map typically contains data that can assist the MLP network in the task of signal reconstruction. Thus, the added maps can be one or more from the following non-limiting list:

[0114] - A map showing the abscissa of that point at each point.

[0115] - A map showing the ordinate of that point at each point.

[0116] - A map with positional coding at each point (as described for example at the internet address https: / / skosmos.loterre.fr / P66 / fr / page / -K0D65X2X-X ).

[0117] - A map representing an image distinct from the image being processed, capable of providing information on the image to be coded, for example the previously processed image if the current image is part of a series of images to be coded such as a video, a set of medical images, a multi-view representation, etc.

[0118] - A map representing a feature map of an image distinct from the image being processed that may provide information about the image to be coded, for example a previously processed map if the current image is part of a series of images to be coded such as a video, a set of medical images, a multi-view representation, etc.

[0119] - A card containing the value of an already decoded sample from the same card, for example the previous sample in the decoding order.

[0120] In this embodiment, the vector Z n is a 4-tuple (Z1...Z4) consisting of the values ​​extracted from the FMi maps located at the coordinates (x n , y n ) of the current pixel P n The extracted values ​​are quantified by the modulus TT and the vector Z nconsisting of the quantified values ​​is processed by the MLP neural network to generate as output a second vector, according to the example a triplet (R, G, B) representative of the sample P n to be coded. The output vector is in this case the triplet (R, G, B) of the pixel P' n encoded then decoded. The triplet is inserted into the decoded image I(P' n ) at positions (x n , y n ) of the color components (R', G', B').

[0121] In one embodiment, the 4 FMi maps are quantized.

[0122] According to the variant shown in dotted lines, the vector Z n is a 5-tuple (Z0...Z4), the value zo being extracted from the additional map FMEo.

[0123] Figure 6 illustrates an encoding method used in another embodiment of the invention.

[0124] In this embodiment, there are 4 FMi cards. As in the previous embodiment, the first FMi card has the same resolution as image I, and therefore has WxH values, where W represents the width of the image in pixels, and H its height. The second FM2 card has half the resolution (in each dimension) of the FMi card. Each additional card has half the resolution of the previous card. This structure makes it possible to reduce the number of variables in feature maps which facilitates coding and learning while minimizing coding cost.

[0125] In this embodiment, a 4-tuple Z n is constructed from the values ​​of the FMi maps located respectively at the coordinates (x n , y n ), (x n / 2, y n / 2), (x n / 4, y n / 4), (x n / 8, y n / 8) of the current pixel P n (x n , y n). These extracted values ​​are processed and quantized to generate the 4-tuple Z n . According to one embodiment, for the FM2 map, the value located at the coordinates (x n / 2, y n / 2) to obtain the final value (Z2) of the vector Z n relating to this FM2 map. According to another embodiment, for the FM2 map, the values ​​located at the coordinates (x n / 2, y n / 2), (x n / 2-1, y n / 2), (x n / 2, y n / 2- 1), (x n / 2-1, y n / 2-1) and filtering is applied to them to obtain the final value (Z2) of the vector Z nrelative to this FM2 map. Oversampling is achieved in this way. Filtering or oversampling can, for example, use bi-cubic filtering or Lanczos filtering, which are known per se. A more general processing can also be used: a neural network can be used to produce the final value (Z2) by weighting the values ​​located at the coordinates (x n / 2, y n / 2), (x n / 2-1, y n / 2), (x n / 2, y n / 2- 1), (x n / 2-1, y n / 2-1), for example an MLP type network.

[0126] As before, the vector Z n is processed by the MLP neural network to generate a second output vector, here a triplet (R, G, B) representative of the sample P n to be coded. The output vector is in this case the triplet (R, G, B) of the pixel P' n encoded then decoded. The triplet (R, G, B) is reinjected into the decoded image at positions (x n, y n ) of the color components (R', G', B') of the image I(p' n ).

[0127] Other types of structure are possible, for example one can use a reduction rate different from half between the maps (a quarter, or a third, etc.) The oversampling step is then adapted to produce the final value of the vector Zn.

[0128] As described in support of Figure 5, other FMEi maps can be introduced, which will be used for the generation of the Z vector n but will not be stored or transmitted.

[0129] Figure 7 is a flowchart representing an example of a decoding method that can be implemented by the decoding device of Figure 2.

[0130] During a step E30, the streams B1 and B2 are extracted from the encoded stream BS. They contain respectively the coded representations of the maps of the first group FMci and the parameters Wc k .

[0131] During a step E31, the M FMdi maps are generated by decoding the FMci values. For this decoding, any known technique similar to that used in the encoder can be used, preferably entropy decoding. In one embodiment, the FMdi maps are decoded in order (FMdi, FMd2,... FMd4), and the variables of each map in a predefined order, for example lexicographic.

[0132] According to an embodiment as described for the encoder, the FMdi cards have the same resolution as the signal I (Pd n ) to be reconstituted, that is to say that they contain N values.

[0133] According to an embodiment as described for the encoder, the FMdi maps are at a resolution less than or equal to that of the signal I (Pd n ) to be reconstituted.

[0134] According to an embodiment as described for the encoder, several FMdi cards have the same resolution, lower than the resolution of the signal.

[0135] During a step E32, according to one embodiment, one or more FME'i maps, forming a second group of L additional feature maps, are generated and complete the first group. They are not decoded but generated by the decoder in a manner identical to the generation of the encoder. They typically comprise data that can assist the MLP' network in the signal reconstruction task. The non-limiting list of possible additional feature maps described in support of FIG. 5 for the encoder applies here.

[0136] During a step E33, values ​​are extracted by the XTR' module from the FMdi and possibly FME'i cards. This extraction is carried out according to the coordinates (x n , y n ) of the input signal of the sample P nof the signal to be reconstructed. This step is identical to step E24 which was described for the encoder in support of Figure 4 and the described embodiments apply.

[0137] The samples to be decoded are for example processed in sequential order, from n=1 to n=N.

[0138] During a step E34, a vector Zd n is constructed by the TT' module from the values ​​extracted from the FMdi and possibly FME'i maps for a sample P n to decode coordinates (x n , y n ). The processing may include inverse quantization of the extracted values ​​or of the vector Zd n constituted, if necessary. The processing may include other operations, such as filtering, scaling, etc. This step is identical to step E25 which was described for the encoder in support of Figure 4 and the embodiments described apply. However, inverse quantization is not mandatory. In this particular case, step E34 and the TT' module are therefore optional.

[0139] In a step E35, the Wdk parameters of the MLP' neural network are generated by decoding the Wck values ​​of the stream B2. Any known decoding technique corresponding to the coding technique used by the encoder can be used for this purpose. The MLP' neural network is similar to the MLP network, i.e. it has the same structure and includes the same parameters, except for the coding, which can be performed with or without losses.

[0140] According to one embodiment, the stream B2 is decoded before the stream B1, in order to have the neural network available before starting to decode the samples.

[0141] During a step E36, the vector Zd nis processed by the MLP' neural network to generate as output a second vector representative of the sample Pd n to decode, according to one embodiment a triplet which is injected into the decoded image I(Pd n ) at positions (x n , y n ) of the color components (Rd, Gd, Bd). This step is identical to step E26 which was described for the encoder in support of figure 4.

[0142] When all the samples of the signal have been processed, the decoded signal corresponding for example to the image I(Pd n ) is available.

[0143] Figure 8 illustrates a decoding method used in one embodiment of the invention to decode an image I(Pd n ).

[0144] In this embodiment, there are 4 LMdi cards. In a preferred embodiment, there are 7.

[0145] In this embodiment, the LMdi cards have the same resolution as the image to be decoded, and therefore include WxH variables, where W represents the width of the image in pixels, and H its height.

[0146] In this embodiment, the vector Zd n is a 4-tuple (Z1...Z4) consisting of the values ​​of the LMdi maps located at the coordinates (x n , y n ) of the current pixel Pd n . The vector Zd n is optionally dequantized then processed by the MLP' neural network to generate a second output vector, according to the example a triplet (R, G, B) representative of the sample Pd n to decode. The triplet (R, G, B) is reinjected into the decoded image I(Pd n ) at coordinates (x n , y n ) in the color components (Rd, Gd, Bd).

[0147] According to a variant presented in dotted lines, there are 5 cards: an additional card FME'o has been introduced. In this embodiment, the vector Zd n is a 5-tuple.

Claims

Claims

1. Method for coding a signal (I(P n )) comprising a plurality of samples (P n ) to be coded comprising the following steps: - a construction stage, comprising the sub-stages of: - construction (E21, E22) of a first group of characteristic maps (FMi); - for at least one sample, called the current sample (P n ), of the signal to be coded, associated with a position (x n , y n ) in the signal to be coded: - construction (E24, E25) of a characteristic vector (Z n ) from said feature maps (FMi) of said first group, as a function of said position (x n , y n ) of said current sample (P n ) ; - processing (E25) of said characteristic vector (Z n) by an artificial neural network (MLP) defined by a set of parameters (Wk), to provide a vector representative of a decoded value (P' n ) of the current sample; - updating (E22, E26) at least one value of one of said characteristic maps of said first group and / or at least one parameter of said network, as a function of a coding performance measurement, - a coding step (E23, E27, E29) of said first group of characteristic maps (FMi) and of said set of parameters (Wk).

2. Method for coding a signal according to claim 1, characterized in that it comprises a step of constructing (E21) a second group of characteristic maps (FMEi), and in that the characteristic vector is further constructed from said characteristic maps of the second group.

3. Method for coding a signal according to one of the preceding claims, characterized in that at least one of said characteristic maps (FMi, FMEi) has a lower resolution than that of the signal to be coded.

4. Method for coding a signal according to one of the preceding claims, characterized in that at least one of said characteristic maps (FMi, FMEi) has a resolution identical to that of the signal to be coded.

5. Method for coding a signal according to one of the preceding claims, characterized in that the construction of said vector (Z n ) characteristic includes a sub-step of extracting a value from said at least one feature map (FMi, FMEi) at an identical position (x n , y n ) to that of the current sample (P n ) in the input signal.

6. Method for coding a signal according to one of the preceding claims, characterized in that the construction of said vector (Z n ) feature includes the following sub-steps: - extraction of a plurality of values ​​from said characteristic maps (FMi, FMEi) as a function of said position (x n , y n ) of said current sample (P n ) ; - processing of said extracted values ​​to obtain the characteristic vector.

7. Method for coding a signal according to one of the preceding claims, characterized in that the coding of said first group of characteristic maps comprises an entropic coding sub-step.

8. A method of decoding a signal comprising a plurality of samples (Pd n ) to be decoded comprising the following steps: - decoding (E31) of a first group of characteristic maps (FMdi); - decoding (E35) of a set of parameters (Wdk) representative of a neural network (MLP'); - for at least one sample, called the current sample (Pd n ), of the signal to be decoded, associated with a position (x n , y n ) in the signal to be decoded: - construction (E33, E34) of a characteristic vector (Zd n ) from the characteristic maps of said first group (FMdi), as a function of said position (x n , y n ) of said current sample, and: - processing (E36) of said characteristic vector (Zd n ) by an artificial neural network (MLP') defined by the decoded parameters (Wdk) to provide a vector representative of a decoded value of the current sample (Pd n ).

9. Decoding method according to claim 8, characterized in that it comprises a step of constructing (E32) a second group of characteristic maps (FME'i) and in that the characteristic vector is further constructed from said characteristic maps of the second group.

10. Decoding method according to claim 8 or 9, characterized in that at least one of said characteristic maps (FMdi, FME'i) has a lower resolution than that of the signal to be decoded.

11. Method for decoding a signal according to one of claims 8 to 10, characterized in that at least one of said characteristic maps (FMdi, FME'i) has a resolution identical to that of the signal to be decoded.

12. Method for decoding a signal according to one of claims 8 to 11, characterized in that the construction of said vector (Zd n) characteristic comprises a sub-step of extracting a value from said at least one characteristic map (FMdi, FME'i) at an identical position (x n , y n ) to that of the current sample (Pd n ) in the signal to be decoded.

13. Method for decoding a signal according to one of claims 8 to 12, characterized in that the construction of said characteristic vector (Zd n ) includes the following sub-steps: - extracting a plurality of values ​​from the feature maps as a function of said position (x n , y n ) of said current sample (Pd n ) ; - processing of said extracted values ​​to obtain the characteristic vector.

14. Method for decoding a signal according to one of claims 8 to 13, characterized in that the decoding (E31) of said first group of characteristic maps comprises an entropic decoding sub-step.

15. Device for coding a signal (I(P n )) comprising a plurality of samples (P n ) to be coded, characterized in that said device is configured to implement: - construction (GEN, MAJ) of a first group of characteristic maps (FM0; - for at least one sample, called the current sample (P n ), of the signal to be coded, associated with a position (x n , y n ) in the signal to be coded: - construction (XTR, TT) of a characteristic vector (Z n ) from said feature maps (FM0 of said first group, as a function of said position (x n , y n ) of said current sample (P n), And ; - processing (MLP) of said characteristic vector (Z n ) by a neural network artificial (MLP) defined by a set of parameters (Wk), to provide a vector representative of a decoded value (P' n ) of the current sample; - updating (MAJ, NNC) at least one value of one of said characteristic maps of said first group and / or at least one parameter of said network, as a function of a coding performance measurement, - coding (FMC, NNC) of said first group of characteristic maps (FMi) and of said set of parameters (Wk).

16. Device for decoding a signal comprising a plurality of samples (Pd n ) to be decoded, characterized in that said device is configured to implement: - decoding (FMD) of a first group of characteristic maps (FMdi); - decoding (NND) of a set of parameters (Wdk) representative of a neural network (MLP'); - for at least one sample, called the current sample (Pd n ), of the signal to be decoded, associated with a position (x n , y n ) in the signal to be decoded: - construction (XTR', TT') of a characteristic vector (Zd n ) from the characteristic maps of said first group (FMdi), as a function of said position (x n , y n ) of said current sample, and: - processing (MLP') of said characteristic vector (Zd n ) by an artificial neural network (MLP') defined by the decoded parameters (Wdk) to provide a vector representative of a decoded value of the current sample (Pd n ).

17. A computer program comprising instructions for carrying out the steps of an encoding or decoding method according to claim 1 or 8 when said program is executed by a computer.

Images