Method and device for encoding and decoding images

Abstract

The invention relates to a method and device for encoding and decoding a signal comprising a plurality of samples. The decoding method comprises the following steps: - decoding a first group of feature maps; - transforming said first group of feature maps to obtain a second group of feature maps in the resolution of the signal to be decoded; - decoding a set of parameters representative of a neural network; - for at least one sample, referred to as the current sample, of the signal to be decoded, associated with a position in the signal to be decoded: - constructing a feature vector from the feature maps of said second group, according to the position of the current sample, and: - processing said feature vector using an artificial neural network defined by the decoded parameters to provide a vector representative of a decoded value of the current sample. Figure for the abstract: Fig. 2.

Description

Description Title of the invention: Method and device for coding and decoding images. Prior art [1] The invention relates to the general field of coding single or multidimensional signals. It relates more particularly to the compression of digital images and video. [2] Digital images are generally subject to source coding aimed at compressing them in order to limit the resources required for their transmission and / or storage. There are many coding standards, such as, for still images, the JPEG family standards, and for animated images, or video, the ITU / MPEG standards (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) as well as their extensions (MVC, SVC, 3D-HEVC, etc.) [3] Encoding an image is generally performed by dividing the image into several rectangular blocks, and encoding these blocks of pixels according to a given processing sequence. In existing video compression techniques, the processing of a block typically includes a prediction of the pixels of the block carried out using previously coded and then decoded pixels present in the image being encoded, in which case we speak of "Intra prediction", or previously coded images, in which case we speak of "Inter prediction". This exploitation of spatial and / or temporal redundancies makes it possible to avoid transmitting or storing the value of the pixels of each block of pixels, by representing at least some of the blocks by a residual representing a difference between the prediction values ​​of the pixels of the block and the actual values ​​of the pixels of the predicted block. [4] As video formats continue to evolve to compress more and more and adapt to the variety of expected formats and communication networks, the possibilities for prediction are becoming ever greater and classic coding and decoding algorithms very complex. [5] In addition to these classic approaches proposed by compression standards (JPEG, MPEG, ITU), approaches based on artificial intelligence, and in particular neural ones, tend to develop. [6] Some of these neural approaches can be seen as a simple extension of the notion of competition of the aforementioned compression techniques, such as the prediction and transformation mode competition in video coding. [7] Other approaches use the concept of "autoencoder". Autoencoders are learning algorithms based on artificial neural networks, which allow the construction of a new representation of a dataset. The architecture of an autoencoder consists of two parts: the encoder and the decoder. The encoder consists of a set of layers of neurons, which process the data in order to construct new representations called "encoded", also called "latent representations". In turn, the layers of neurons in the decoder receive these representations and filter them in order to try to reconstruct the initial data. The differences between the reconstructed data and the initial data make it possible to measure the error made by the autoencoder. Training consists of modifying the parameters of the autoencoder in order to reduce the reconstruction error measured on the different samples of the dataset.The performance of such autoencoder-based systems comes at the cost of a considerable increase in memory footprint and complexity compared to conventional approaches as proposed by compression standards. They can have millions of parameters and may require up to a million MACs (multiply-accumulate) to decode a single pixel. This makes such decoders significantly more complex than conventional decoders, which could hamper the adoption of learning-based compression. [8] More recently, a simple neural network-based encoding technique was described in the article "Compression with Implicit Neural representations" by Emilien Dupont et al. (arXiv:2103.03123). The proposed encoding technique consists of fitting a neural network to an image, quantizing the network weights, and transmitting them. At decoding time, the neural network is evaluated at each pixel position to reconstruct the image. However, such a technique remains inefficient in terms of compression. [9] There is therefore a need for a solution that allows an image or a sequence of images to be coded / compressed in a simple and efficient manner. Statement of the invention

[0010] The invention relates to a coding method according to claim 1 and a decoding method according to claim 7.

[0011] For the purposes of the invention, encoding, or "coding", means the operation which consists of representing a set of samples in a compact form carried for example by a digital binary train. Decoding means the operation which consists of processing a digital binary train to restore decoded samples.

[0012] A signal "sample" means a value taken from the signal. Sampling the signal produces a series of discrete values ​​called samples. In the case of an image signal, the sample is called a pixel, which can be, for example, a color pixel traditionally represented by a triplet of values, for example (R, G, B) or (Y, U, V). Generally speaking, a sample is represented by a vector of one or more components, or data.

[0013] A "signal comprising a plurality of samples" means a signal with one (audio, sound), two (image) or more than two (stereoscopic, multiscopic image, image associated with a depth map, video, etc.) dimensions. Depending on this dimensionality, the sample has one, two or more coordinates in the signal. In the case of an image signal, the position of the sample is identified by its abscissa (x) and ordinate (y) coordinates.

[0014] By "feature maps" we mean an abstract representation of the signal comprising a plurality of variable data, discrete or not, which are also called values, for example real or integer numbers. As is known, these maps are also referred to as the "latent representation" of the input signal.

[0015] By "characteristic map transformation" is meant a mathematical operation that transforms the values ​​of a first map into values ​​of a second map. A first map, called the first group map, intended for coding, is arbitrary. A second map, called the transformed map, or the second group map, has the same resolution as the input signal, i.e. it contains as many values ​​as the input signal contains samples (N). The transformation may include, for example, interpolation, oversampling, filtering, quantization, a Fourier transform, etc.

[0016] By "data feature vector constructed from feature maps as a function of a position" is meant a vector consisting of one or more elements, or data, preferably discrete, the data being constructed from the feature maps of the second group at a position determined by the position of the sample in being processed in the signal. This characteristic vector is the one that is applied to the input of the neural network. For example, in the case of a one-dimensional audio signal, such a vector can be formed from a plurality of values ​​taken from each of the characteristic maps at the same coordinate as the sample to be encoded. In the case of an image, such a vector can be formed from a plurality of values ​​taken from each of the characteristic maps at the same abscissa and ordinate coordinates as the sample to be encoded (resp. to be decoded).

[0017] By "neural network" we mean a neural network such as a convolutional neural network, a multi-layer perceptron, an LSTM (for "Long Short Term Memory" in English), etc. The neural network is defined for example by a plurality of layers of artificial neurons which comprise a set of activation, weighting and addition functions (for example, a layer can calculate y = f(Ax+b), where y and b are vectors of dimension N, x is a vector of dimension M, A is a matrix of dimension MxN, and f is the activation function). Hereinafter we speak of "neural network" or "synthetic neural network".

[0018] By "neural network parameter" we mean one of the values ​​that characterizes the neural network, for example a weight associated with one of the neurons (filter coefficient, weighting, bias, value affecting the operation of non-linearity), etc.

[0019] By "processing by a neural network" is meant the application of a function expressed by a neural network to the input characteristic vector to produce an output vector representative of the sample to be encoded (resp. to be decoded). This output vector may include one or more data representative of the sample.

[0020] By "performance measurement" is meant a measurement between at least one value of a sample to be coded and a decoded value of said sample. The measurement can evaluate, for example, a distortion, or a perceptual error. It can be carried out on a sample or a plurality of samples (for example, the complete image). The measurement can also include a measurement of the flow rate, in particular associated with the coding of the neural network and / or the coding of the feature maps of the first group. The measurement can be a joint measurement between the flow rate and the distortion through their weighting. As is well known in the state of the art, the value of this measurement is generally minimized until a target value is reached.

[0021] A "construction step" means a step that aims to construct the representative parameters of the image, before their actual coding. The construction sub-steps can be repeated as many times as necessary to obtain an acceptable performance measurement.

[0022] Generally speaking, it is considered that the steps of an coding or decoding method should not be interpreted as being linked to a notion of temporal succession. In other words, the steps may be carried out in a different order than that indicated in the independent coding or decoding claim, or even in parallel.

[0023] The coding method according to the invention constructs the coding parameters from the input signal, for example an image, by training a neural network on characteristic vectors associated with a position of a sample to be coded. These characteristic vectors are constructed from characteristic maps at the resolution of the input signal. During the training, or construction, the parameters of the neural network and the values ​​of the characteristic maps are updated according to a performance measurement, for example of the rate-distortion type. When the training is finished, that is to say when the performance measurement obtained is satisfactory, the actual coding of the parameters of the neural network and / or the values ​​of the characteristic maps can be carried out and stored or transmitted to the decoder.

[0024] Advantageously, the training process makes it possible to refine the parameters of the neural network and / or the values ​​of the feature maps until an adequate representation in terms of performance is obtained, for example a desired balance between the generated bit rate and the distortion suffered by the input signal. The training of the values ​​of the feature maps and the parameters of the neural network can be joint. Advantageously, the coding method according to the invention makes it possible to efficiently compress the signal.

[0025] Advantageously, the decoding process is simple since it is sufficient to decode the feature maps of the first group and the neural network to reconstruct a decoded version of the signal, for example an image.

[0026] Such a neural network can advantageously be of very simple structure with few parameters.

[0027] Moreover, decoding can be done sample by sample, in a progressive manner.

[0028] According to embodiments of the coding and / or decoding method:

[0029] - The method comprises a step of constructing a third group of feature maps and the feature vector is further constructed from said feature maps. Advantageously, these additional maps of the third group, constructed identically at the coder and the decoder, are neither stored nor transmitted at the coder, nor decoded at the decoder. They thus make it possible to benefit from additional data to improve compression without degrading the bit rate. They may for example include coordinates, data available in the maps of the first or second group, data concerning other images already processed by the coder or decoder, etc.

[0030] - At least one of the characteristic maps of the first group has a lower resolution than the signal to be coded (or decoded) and the transformation operation involves oversampling. Advantageously, according to this method, the compression of the characteristic maps is more efficient since at least one of the characteristic maps of the first group, to be coded (or decoded), has fewer values ​​than if it were at the resolution of the signal. For example, in the case of a digital image, one of the characteristic maps of the first group may have a resolution of 1 / 2, i.e. it has half as many values ​​on the abscissa and ordinate as the input signal has samples, i.e. a total of 4 times fewer values ​​than a characteristic map at the resolution of the signal. On the other hand, the characteristic map of the second group, which corresponds to a transformation of this map of the first group, has a resolution identical to that of the signal.The transformation therefore includes in this case at least one oversampling operation to obtain the same number of values ​​in this transformed map as the input signal (resp. to be decoded) has samples.

[0031] - At least one of said characteristic maps of the first or third group has a resolution identical to that of the signal to be coded (resp. decoded). Advantageously, according to this mode, at least one of the characteristic maps, at the same resolution as the input signal (resp. to be decoded), allows significant fidelity and respect for the details of the initial resolution of the signal. The transformation in this case preserves the number of values ​​of the transformed characteristic map. It can be reduced to identity (no processing is carried out on the values ​​of the map of the first group) or include a filtering operation, quantization, a Fourier transformation, etc. During coding, quantization is essential for the proper functioning of the system if the characteristic maps include for example, floating point or real values. It is necessary to quantize them before encoding them and / or providing them as input to the neural network. On the other hand, during decoding, inverse quantization is not necessary, depending on the embodiments.

[0032] - The construction of said characteristic vector comprises a sub-step of extracting a value from said at least one characteristic map of the second or third group at a position identical to that of the current sample in the signal.

[0033] Advantageously, it is thus possible to extract a value from a feature map of the second or third group, at the same position as the sample in the signal (input signal for coding, signal to be decoded for decoding), to constitute an element of the feature vector. This process is simple to implement. For example, if we have J feature maps of the same resolution as the signal, a simple extraction of the values ​​of the maps at the coordinates of the current sample (at the same abscissa and the same coordinate in the feature map) makes it possible to directly construct the feature vector of J elements.

[0034] - The coding (resp. decoding) of said first group of feature maps comprises an entropic coding (resp. decoding) sub-step. Advantageously, entropic coding makes it possible to exploit signal redundancies that may persist in the feature maps, the signal thus being compressed more efficiently.

[0035] Correlatively, the invention also relates to a coding device and a decoding device.

[0036] The characteristics and advantages of the coding or decoding method apply in the same way to the coding or decoding device according to the invention and vice versa.

[0037] The invention also relates to a computer program on a recording medium, this program being capable of being implemented in a computer or a control device in accordance with the invention. This program comprises instructions adapted to the implementation of the corresponding method. This program can use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0038] The invention also relates to an information medium or a recording medium readable by a computer, and comprising computer program instructions mentioned above. above. The information or recording media may be any entity or device capable of storing programs. For example, the media may include a storage medium, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording medium, for example a floppy disk or a hard disk, a DNA sequence, or a flash memory. On the other hand, the information or recording media may be transmissible media such as an electrical or optical signal, which may be conveyed via an electrical or optical cable, by radio link, by wireless optical link, or by other means.

[0039] The program according to the invention can in particular be downloaded from an Internet-type network.

[0040] Alternatively, each information or recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of a method according to the invention. Brief description of the drawings

[0041] Other characteristics and advantages of the present invention will emerge from the description given below, with reference to the appended drawings which illustrate exemplary embodiments thereof which are not limiting in nature. [Fig. 1] Figure 1 schematically represents a coding device used in the context of the invention; [Fig. 2] Figure 2 schematically represents a decoding device used in the context of the invention; [Fig. 3] Figure 3 illustrates an example of a synthetic artificial neural network used in the context of the invention for coding and decoding; [Fig. 4] Figure 4 is a flowchart representing an example of an encoding method that can be implemented by the encoding device of Figure 1; [Fig. 5] Figure 5 illustrates an encoding method used in one embodiment of the invention; [Fig. 6] Figure 6 illustrates an oversampling method used in one embodiment of the invention; [Fig. 7] Figure 7 is a flowchart representing an example of a decoding method that can be implemented by the decoding device of Figure 2; [Fig. 8] Figure 8 illustrates a decoding method used in one embodiment of the invention. Description of the embodiments

[0042] Figure 1 schematically represents an ENC coding device.

[0043] This ENC coding device includes a GEN module for generating feature maps, a SE transformation module, an XTR data extraction module, an MLP module corresponding to an artificial neural network, an NNC module for coding a neural network, an FMC module for coding feature maps, an EVAL module for evaluating a performance function, an MAJ update module, and an optional FME module for generating additional feature maps.

[0044] The ENC coding device may be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the above-mentioned modules may then be realized by the cooperation of the processor and computer program instructions stored in the above-mentioned memory and designed to perform the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0045] The ENC coding device in Figure 1 receives as input a succession of samples to be coded, noted P n , for example a temporal succession of sound samples, or a set of image data noted I(P n ). In this second case, the image signal I(P n ) can represent a two-dimensional image, or a plurality of two-dimensional images (video, color components, stereoscopic, multiscopic components, etc.). P ndenotes a sample n of the input signal comprising N samples. In one embodiment, the signal is a color image signal represented by means of at least one two-dimensional representation, such as a matrix of pixels, of width W and height H (with N = WxH) each pixel comprising a red (R), green (G), blue (B), or, alternatively, a luminance (Y) component and at least one chrominance component. The location of each pixel is defined by its abscissa and ordinate (x and y) coordinates in the image. In one embodiment, the image is in grayscale represented by means of a two-dimensional representation, such as a matrix of pixels, each pixel having a grayscale component, or luminance. In this case the vector representing the pixel is reduced to a single component, or data.

[0046] As will be described in more detail later with reference to Figures 3 to 8:

[0047] The GEN module for generating characteristic maps is configured to generate, and in particular initialize, a plurality of M characteristic maps denoted FMi. The optional FME module can generate one or more additional maps, L in number, which will not be coded or transmitted, denoted FMEi.

[0048] The SE module performs a transformation of the first group of feature maps to generate a second group of feature maps at the same resolution as the input signal. The SE module may perform quantization of the values ​​of the maps in the first group by using a quantizer Q to generate an ordered collection of quantized values. Recall that quantization of a value refers to mapping that value to a member of a discrete set of possible code symbols. For example, the set of possible code symbols may consist of integer values, and the quantization system performs simple rounding of a real value to an integer value. In another example, quantization consists of multiplication by a given value and then rounding.Then the SE module performs a transformation of the values ​​of at least one of the feature maps such as oversampling, interpolation, filtering, etc. At the end of the transformation, a transformed feature map of the second group has the same resolution as the input signal.

[0049] The XTR module performs an extraction of values ​​in the FMSi and optionally FMEi characteristic maps, for a sample P n to be encoded, based on its coordinates in the input signal. The extracted values ​​constitute the vector Z n . Z n is a J-tuple, that is, it has J elements, or data Zi. The vector Z n of index n refers to the characteristic vector of the pixel P' n .

[0050] The MLP module is a neural network defined by K parameters Wk, capable of processing the vector Z n, or J-Uplet, as input, to generate as output a second vector representative of the sample P n to code. The neural network is, according to one embodiment, an MLP, or Multi Layer Perceptron, consisting of an input layer adapted to the input format (the J-uplet), optionally one or more hidden layer(s), and an output layer adapted to the output format of the output vector, generally a vector comprising A elements. According to one embodiment, A is equal to 3 and the output vector is the triplet (R, G, B) of the pixel P'n coded then decoded.

[0051] The NNC module performs the coding of the neural network, in particular of its Wk parameters. During the process of training, or building, the coding, that is to say as long as the performance evaluation step is not satisfactory, the NNC module performs a coding simulation, followed by a decoding, intended for the evaluation module. It updates the parameter values ​​according to the results of a performance measurement carried out by the EVAL module. Subsequently, it performs the actual coding of the Wk neural network parameters. The coded parameters are denoted Wck. They are stored or transmitted in the form of a B2 stream. In a known manner, the coding simulation can be identical to the actual coding, or produce an approximation thereof.

[0052] The FMC module performs the coding of the FMi maps, i.e. the quantized values ​​of each map of the first group (excluding the FMEi maps). During the process of training, or building, the coding, i.e. until the performance evaluation step is satisfactory, the FMC module performs a coding simulation, followed by a decoding, intended for the evaluation module. Subsequently, it performs the actual coding of the values ​​of the FMi maps. The coded maps are denoted FMci. They are stored or transmitted in the form of a Bl stream. In a known manner, the coding simulation can be identical to the actual coding, or an approximation thereof. The coding module quantizes, if necessary, the latent representation of the values ​​of the maps of the first group by using a quantizer to generate an ordered collection of quantized values.Then the encoding module compresses the quantized data, for example using entropy-type encoding.

[0053] The EVAL module performs an evaluation and minimization of a coding performance. The evaluation function is for example of the rate-distortion type. The minimization can be carried out by gradient descent, or any other method within the reach of a person skilled in the art.

[0054] The MAJ module updates the values ​​of the FMi cards to be encoded based on the results of the performance function.

[0055] Figure 2 schematically represents a DEC decoding device.

[0056] The DEC coding device of Figure 2 receives as input a first group of encoded data organized into M FMci characteristic maps (also called FM layers) from a stream B1, and the encoded Wck parameters of the neural network from a stream B2.

[0057] This DEC decoding device includes a neural network decoding NND module, a feature map decoding FMD module, a data extraction XTR' module, an inverse transformation SE' module, a neural network corresponding MLP' module, an optional additional feature map generation FME' module.

[0058] The maps, or layers, decoded by the FMD module, numbering M, are denoted FMdi. The parameters decoded by the NND module are denoted Wdk.

[0059] The decoder's FME' module can also generate one or more additional cards, denoted FME'i, L in number, identical to the additional FMEi cards generated by the encoder.

[0060] The SE' module performs a transformation of the first group of decoded feature maps FMdi to generate a second group of feature maps at the same resolution as the input signal, denoted FMS'i. The SE' module optionally performs an inverse quantization corresponding to the quantization performed at the encoder. The inverse quantization is not necessary if the encoder's quantizer Q has performed a simple rounding of the real values ​​submitted to it. The inverse quantization is also not necessary if the neural network is able to take into account a quantization of its input data. Otherwise, the decoder performs the inverse operation of the quantizer Q. The SE' module also performs a transformation of the values ​​of the feature maps, for example oversampling, interpolation, filtering, etc.At the end of the transformation, a transformed feature map of the second group has the same resolution as the signal to be decoded.

[0061] The XTR' module is identical to the XTR module in Figure 1. It performs an extraction of values ​​from the M transformed FMS'i feature maps and possibly from the L additional FME'i maps, to construct the vector Z n taking the form of a J-tuple for a sample P n to be decoded, depending on its coordinates in the signal to be decoded. In one embodiment, J=M. In one embodiment, J=M+L.

[0062] The MLP' module is a neural network defined by K parameters Wdk, capable of processing the vector Zd n , or J-Uplet, as input, to generate as output a second vector representative of the sample P nto decode, generally a vector comprising A elements. According to one embodiment, A=3 and the output vector is the triplet (R, G, B) of the pixel Pd n decoded. Alternatively, the output vector includes a luminance component (Y) and at least one chrominance component. The MLP' module is identical in structure to the MLP module, and its parameters are either identical if the coding of its Wk parameters is lossless, or different if the coding is lossy.

[0063] When all samples P n of the signal have been decoded, we have a reconstructed signal I(Pd n ), according to an example an image I comprising N samples decoded in the form of N vectors Pd n .

[0064] The DEC decoding device may be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the above-mentioned modules may then be realized by the cooperation of the processor and computer program instructions stored in the above-mentioned memory and designed to perform the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0065] Figure 3 illustrates an example of a synthetic artificial neural network used for encoding and decoding in embodiments of the invention.

[0066] The MLP synthetic artificial neural network for coding and the MLP' synthetic artificial neural network for decoding are defined by an identical structure, comprising for example a plurality of layers of artificial neurons, and by a set of weights and activation functions associated respectively with the artificial neurons of the network concerned.

[0067] A vector representation of a current sample (a vector Z n or Zd n from the FMSi and FMEi or FMS'i and FME'i feature maps) is applied as input (i.e. on an input layer) of the MLP or MLP' synthetic artificial neural network. The artificial neural network produces as output a vector P' n or Pd n representative of the decoded sample, according to one embodiment the color components (R, G, B) constituting a color pixel of an image.

[0068] The concatenation of all these reconstructed pixels in an image (2D, 3D) constitutes the decoded, or reconstructed, image.

[0069] At the encoder, the MLP synthesis artificial neural network is trained on the image, so as to minimize the differences between the input representation of the current image I(P n ) and its representation I(P' n ) output, while also minimizing the amount of data to be encoded. The EVAL module performs a performance measurement in this regard.

[0070] Once training is complete, the network parameters are encoded, either lossless, in which case the MLP' neural network is identical to MLP, or lossy, in which case the MLP' network may be slightly different from MLP.

[0071] Figure 4 is a flowchart representing an example of an encoding method that can be implemented by the encoding device of Figure 1.

[0072] According to this embodiment, the signal is a two-dimensional image, each sample to be coded is therefore a pixel P n of coordinates (x n , y n ).

[0073] Encoding takes place in two main phases:

[0074] In a first phase, called the construction phase, learning is carried out in order to determine, for an input signal I(P n), the values ​​of the FMi maps and the Wk parameters to optimize a global cost function. The learning is for example carried out by gradient descent, followed by an update of the parameters of the MLP neural network and the values ​​of the FMi feature maps. As is known in the state of the art, the cost function can be of the rate-distortion type, or rate, or distortion, or perceptual. To measure the rate R, it is necessary to simulate the coding of the FMi maps, then to measure the associated coding rate (the size of the stream B 1). According to one embodiment, the coding of the Wk parameters is not simulated because their influence is less important than that of the feature maps. According to one embodiment, the coding of the Wk parameters is also simulated and the associated rate is measured (the size of the stream B2).To measure the distortion D, it is necessary to simulate the coding then the decoding of at least part of the image I, to obtain at least one pixel P'. n resulting from a simulation of coding then decoding, then measuring the difference between this part of the image I(P n ) as input and a corresponding part of the image I(P' n ) encoded then decoded.

[0075] Then, in a second phase, called the coding phase, the FMi maps and the Wk parameters are encoded to produce the coded values ​​FMci and Wck before transmission or storage. They constitute the compressed representation of the input signal I (P n ).

[0076] We will now describe the steps of a method according to one embodiment of the invention.

[0077] During a step E20, a signal I(P n ) to be coded, comprising a plurality of N samples P n , is provided as input to the process.

[0078] In step E21, the M FMi maps of the first group are initialized by the GEN module. Subsequently, the Wk parameters of the MLP neural network and the values ​​of the FMi maps must be optimized during the construction phase.

[0079] According to one embodiment, the FMi cards are of the same resolution as the input signal I(P n ) and therefore each have the same number of values ​​N as there are samples P n to code.

[0080] According to one embodiment, the FMi cards have a resolution less than or equal to that of the input signal I(P n ) and therefore include, for at least one of them, a number N' of values ​​to be coded less than N.

[0081] According to one embodiment, the first FMi card is at the resolution of the signal and each subsequent card is at half the resolution of the previous one.

[0082] According to one embodiment, several FMi cards have the same resolution, lower than the signal resolution.

[0083] According to one embodiment, the FMi cards are initialized by predefined constant values.

[0084] According to another embodiment, the feature maps are initialized by a set of random real values.

[0085] According to one embodiment, one or more FMEi maps, forming a third group of L additional feature maps, are generated and added to the first group. They are used to construct the feature vector but will not be stored or transmitted.

[0086] These FMi characteristic maps of the first group are subsequently updated, or refined, during a step E22 by the MAJ update module of the encoder during its learning during the construction phase.

[0087] In a step E23, the FMi maps of the first group are coded by the FMC module of the encoder. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is an actual coding and the coded values ​​constitute the stream B 1. The simulation can be identical to the actual coding but it can also be different (for example, simplified). For this coding, any known technique aimed at compressing the values ​​of the maps can be used.

[0088] In one embodiment, the FMi cards are coded in order (FMi, FM2,..., FM4), and the variables of each card in a predefined order, for example lexicographic. Each card undergoes entropy coding. The entropy coding produces a compressed stream B 1 whose bit rate is subsequently measured during a step E28.

[0089] During a step E24, the M maps of the first group FMi are transformed by the module SE to generate maps of the second group FMSi at the resolution of the input signal.

[0090] According to one embodiment, M FMSi maps are generated.

[0091] According to one embodiment, each FMi card is transformed into an FMSi card.

[0092] According to one embodiment, at least one FMi map has a lower resolution than the signal to be coded and the transformation operation includes oversampling so that the transformed FMSi map has the same number of samples as the input signal. Oversampling consists of adding values ​​to the FMSi maps to achieve the resolution of the input signal. It can be simple (by nearest neighbor replication) or include interpolation (linear, polynomial, by filtering, etc.).

[0093] At a step E25, a vector Z nis constructed by the XTR module from the values ​​extracted from the FMSi maps of the second group and possibly FMEi of the third group for each sample P n depending on its coordinates..

[0094] In one embodiment, Z n has as many values ​​as there are FMi (and optionally FMEi) input cards. In this case we have J=M(+L).

[0095] In one embodiment, Z n is a J-uplet (zi, Z2,. . ., zj), consisting of the values ​​of the FMSi (and optionally FMEi) maps located at the coordinates (x n , y n ) of the current pixel P n , as will be illustrated in Figure 5. The value located at the coordinates (x n , y n ) of the FMSi or FMEi map gives the final value (zi) of element i of vector Z n relating to this FMSi or FMEi card.

[0096] The samples to be coded are, for example, processed in sequential order, from n=1 to n=N.

[0097] During a step E26, the vector Z n is processed by the MLP neural network to generate as output a vector representative of the sample P n to code, according to one embodiment, the triplet (R, G, B) of the sample P' n (sample P n coded then decoded).

[0098] The structure and Wk parameters of the neural network are initialized, for example, during the first iteration of this step. These parameters are subsequently updated, or refined, during the construction phase, during subsequent iterations of the process.

[0099] According to one embodiment, the parameters of the neural network are initialized by predefined values ​​known to give a satisfactory result (for example, following training on a corpus of images).

[0100] According to another embodiment, the parameters Wk of the neural network are initialized by a set of random values.

[0101] In step E27, the Wk parameters of the MLP neural network are quantized and coded. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is an actual coding and the coded values ​​constitute the B2 stream. The simulation can be identical to the actual coding but it can also be different (for example, simplified). Any known technique can be used for this purpose, for example the neural network coding standard proposed by the MPEG-7 part 17 standard, also called NNR (Neural Network Representation). Note that in this case, the amount of degradation that the coding brings to the Wk parameters must be chosen.

[0102] During step E28, a performance measure is evaluated.

[0103] For this purpose, the coding simulation rates associated with the feature maps (simulation of the B1 stream by coding and decoding of the FMQ maps and optionally with the parameters of the neural network (simulation of the B2 stream by coding and decoding of the Wk parameters) are measured.

[0104] According to one embodiment, the cost function is of the rate-distortion type, noted (D+L*R), where D is for example the quadratic error measured between the input signal and the decoded signal (or the error measured on a subset of samples of the signal). According to another example, D is calculated from a perceptual function such as the SSIM (for Structural SIMilarity), or MSSSIM (for Multi-scale Structural SIMilarity). In one embodiment, R is the simulated bitrate of stream B 1 ; in another embodiment, R is the total bitrate used to encode this image, i.e., the sum of the simulated bitrates of B 1 and B2. L is a parameter that adjusts the bitrate-distortion tradeoff. Other cost functions are possible.

[0105] As long as the cost function has not reached its minimum, the performance measure is not satisfactory, and the method is repeated from step E22. This minimization can be performed by a mechanism known as gradient descent with parameter updates during step E22 for the feature map values ​​and E26 for the network parameters.

[0106] In a step E29, if the cost function has reached its minimum, the training stops. If an encoded version corresponding to the last simulation of the parameters of the neural network (Wk) and the feature maps (FMi) is available, the streams B1 and B2 can be constituted therefrom. According to another embodiment, the actual encoding of the updated parameters of the neural network (Wk) and the values ​​of the feature maps (FMi) is carried out in this step to produce the encoded parameters Wck and FMd which constitute streams B1 and B2.

[0107] The B 1 and B 2 streams may be concatenated to produce a final stream. According to one embodiment, the B 2 stream of encoded parameters of the neural network is stored or transmitted before the B 1 stream, so that it can be decoded before the B 1 stream.

[0108] Figure 5 illustrates an encoding method used in one embodiment of the invention for encoding an image I(P n ) (not shown).

[0109] In this embodiment, there are 4 FMi maps. In a preferred embodiment, there are 7. The FMi map has the resolution of the image to be encoded. The second FM2 map has half the resolution (in each dimension) of the FMi map. Each additional map has half the resolution of the previous map. This structure makes it possible to reduce the number of variables in the feature maps, which facilitates encoding and learning while minimizing the encoding cost.

[0110] The FM2 map is oversampled by a factor of 2 in each dimension, according to a method illustrated in Figure 6. The FM3 map is oversampled by a factor of 4 in each dimension, and the FM4 map by a factor of 8 in each dimension.

[0111] The resulting FMSi maps are of the same resolution as the I(P) image n ), and therefore each have WxH values, where W represents the width of the image in pixels and H its height (N=WxH).

[0112] Other types of structure are possible, for example one can use a reduction rate different from half between cards (a quarter, or a third, etc.).

[0113] According to a variant presented in dotted lines, the feature maps are 5 in number: an additional FMEo map has been introduced, which will neither be stored nor transmitted. This additional map typically contains data that can assist the MLP network in the task of signal reconstruction. Thus, the added maps can be one or more from the following non-limiting list:

[0114] - A map showing the abscissa of that point at each point.

[0115] - A map showing the ordinate of that point at each point.

[0116] - A map with positional coding at each point (as described for example at the internet address https: / / skosmos.loterre.fr / P66 / fr / page / -K0D65X2X-X ).

[0117] - A map representing an image distinct from the image being processed, capable of providing information on the image to be coded, for example the previously processed image if the current image is part of a series of images to be coded such as a video, a set of medical images, a multi-view representation, etc.

[0118] - A map representing a feature map of an image distinct from the image being processed that may provide information about the image to be coded, for example a previously processed map if the current image is part of a series of images to be coded such as a video, a set of medical images, a multi-view representation, etc.

[0119] - A card containing the value of an already decoded sample from the same card, for example the previous sample in the decoding order.

[0120] In this embodiment, the vector Z n is a 4-tuple (Z1...Z4) consisting of the values ​​extracted from the FMSi maps located at the coordinates (x n , y n ) of the current pixel P n . The extracted values ​​are quantized and the vector Z n consisting of the quantified values ​​is processed by the MLP neural network to generate a second output vector, according to the example a triplet (R, G, B) representative of sample P n to be coded. The output vector is in this case the triplet (R, G, B) of the pixel P'n coded then decoded. The triplet is inserted into the decoded image I(P' n ) at positions ( n, y n ) of the color components (R', G', B').

[0121] In one embodiment, the vector Z nis quantified after its extraction.

[0122] In another embodiment, the 4 FMSi maps are quantized by the SE module, the generated vector in this case includes already quantized values.

[0123] According to the variant shown in dotted lines, the vector Z n is a 5-tuple (Z0...Z4), the value zo being extracted from the additional map FMEo.

[0124] Figure 6 illustrates an oversampling method used in one embodiment of the invention in encoding and decoding.

[0125] According to the example shown in Figure 6, the FM2 card (resp. FNfcb) of the first group is at resolution Yi. The interpolation oversampling process generates the FMS2 image with the same resolution as the input signal.

[0126] According to a simple example, to generate the value of the FMS2 map at the position (x,y) we consider in the FM2 map (resp. FNfcb) the values ​​at the positions (x / 2, y / 2), (x / 2-1, y / 2), (x / 2, y / 2-1) and (x / 2-1, y / 2-1) and an operation is applied to them to obtain the FMS2 value (resp. FMS'2) at the position (x,y). This can be for example averaging, interpolation, etc.

[0127] Naturally, any oversampling algorithm within the reach of a person skilled in the art can be used.

[0128] Figure 7 is a flowchart representing an example of a decoding method that can be implemented by the decoding device of Figure 2.

[0129] During a step E30, the streams B1 and B2 are extracted from the encoded stream. They contain respectively the coded representations of the maps of the first group FMci and the parameters Wc k .

[0130] In a step E31, the M FMdi maps are generated by decoding the FMci values. For this decoding, any known technique similar to that used in the encoder can be used, preferably entropy decoding. In one embodiment, the FMdi maps are decoded in order (FMdi, FMd2,... FMcU), and the variables of each card in a predefined order, for example lexicographic.

[0131] According to an embodiment as described for the encoder, the FMdi cards have the same resolution as the signal I(Pd n ) to be reconstructed and therefore each contain the same number of values ​​N as there are samples Pd n to decode.

[0132] According to an embodiment as described for the encoder, the FMdi maps are at a resolution less than or equal to that of the signal I (Pd n ) to be reconstituted.

[0133] According to an embodiment as described for the encoder, the first FMdi map is at the resolution of the signal and each subsequent map is at half the resolution of the previous one.

[0134] According to an embodiment as described for the encoder, several FMdi cards have the same resolution, lower than the resolution of the signal.

[0135] During a step E32, according to one embodiment, one or more FME'i maps, forming a third group of L additional feature maps, are generated and complete the second group. They are not decoded but generated by the decoder in a manner identical to the generation of the encoder. They typically comprise data that can assist the MLP' network in the signal reconstruction task. The non-limiting list of possible additional feature maps described in support of FIG. 4 for the encoder applies here.

[0136] During a step E33, the M maps of the first group FMdi are transformed by the module SE to generate maps of the second group FMS'i at the resolution of the input signal.

[0137] According to an embodiment as described for the encoder, M FMS'i maps are generated.

[0138] According to an embodiment as described for the encoder, each FMdi card is transformed into an FMSi card.

[0139] According to an embodiment as described for the encoder, at least one FMdi map has a lower resolution than the signal to be encoded and the transformation operation includes oversampling so that the transformed FMS'i map has the same number of samples as the input signal. Oversampling consists of adding values ​​to the FMS'i maps to achieve the resolution of the input signal. It can be simple (by nearest neighbor replication) or include interpolation (linear, polynomial, by filtering, etc.)

[0140] The transformation can optionally include inverse quantization of the extracted values, if necessary. However, inverse quantization is not mandatory.

[0141] During a step E34, a vector Zd n is constructed by the XTR' module from the values ​​extracted from the FMS'i maps of the second group and possibly FME'i of the third group for a sample Pn to decode, according to its coordinates (x n , y n ). This step is identical to step E25 which was described for the encoder in support of Figure 4 and the described embodiments apply.

[0142] In particular, in one embodiment, Zd n is a J-uplet (zi, Z2, . . . , zj), consisting of the values ​​of the FMS'i (and optionally FME'i) maps located at the coordinates (x n , y n ) of the current pixel P n , as will be illustrated in Figure 8. The value located at the coordinates (x n , yn) of the FMS'i map gives the final value (zdi) of element i of the vector Zd n relating to this FMS'i or FME'i card.

[0143] The extraction may involve an inverse quantization of the extracted values ​​or of the vector Zd n constituted, if necessary.

[0144] The samples to be decoded are for example processed in sequential order, from n=1 to n=N.

[0145] In a step E35, the Wdk parameters of the MLP' neural network are generated by decoding the Wck values ​​of the stream B2. Any known decoding technique corresponding to the coding technique used by the encoder can be used for this purpose. The MLP' neural network is similar to the MLP network, i.e. it has the same structure and includes the same parameters, except for the coding, which can be performed with or without losses.

[0146] According to one embodiment, the stream B2 is decoded before the stream B1, in order to have the neural network available before starting to decode the samples.

[0147] During a step E36, the vector Zd n is processed by the MLP' neural network to generate as output a second vector representative of the sample Pd nto decode, according to one embodiment a triplet which is injected into the decoded image I(Pd n ) at positions (x n , y n ) of the color components (Rd, Gd, Bd). This step is identical to step E26 which was described for the encoder in support of figure 4.

[0148] When all the samples of the signal have been processed, the decoded signal corresponding for example to the image I(Pd n ) is available.

[0149] Figure 8 illustrates a decoding method used in one embodiment of the invention.

[0150] In this embodiment, there are 4 FMdi cards. In a preferred embodiment, there are 7.

[0151] In this embodiment, the first FMdi map has the same resolution as the image I, and therefore has WxH variables, where W represents the width of the image in pixels, and H its height. The second FMd2 map has half the resolution (in each dimension) of the FMdi map. Each additional map has half the resolution of the previous map. This structure makes it possible to reduce the number of variables in the feature maps, which facilitates decoding while minimizing the coding cost.

[0152] The FMd2 map is oversampled by a factor of 2 in each dimension, according to a method illustrated in Figure 6. The FMds map is oversampled by a factor of 4 in each dimension, and the FMd4 map by a factor of 8 in each dimension.

[0153] FMS'i cards have the same resolution as the image to be decoded, and therefore have WxH variables, where W represents the width of the image in pixels, and H its height.

[0154] In this embodiment, the vector Zd n is a 4-tuple (zdi...zd4) consisting of the values ​​of the FMS'i maps located at the coordinates (x n , y n ) of the current pixel Pd n . The vector Zd n is optionally dequantized then processed by the MLP' neural network to generate a second output vector, according to the example a triplet (R, G, B) representative of the sample Pd n to decode. The triplet (R, G, B) is reinjected into the decoded image I(Pd n ) at coordinates (x n , yn) in the color components (Rd, Gd, Bd).

[0155] According to a variant presented in dotted lines, there are 5 cards: an additional card FME'o has been introduced. In this embodiment, the vector Zd n is a 5-tuple.

Claims

Claims

1. Method for coding a signal (I(P n )) comprising a plurality of samples (P n ) to be coded comprising the following steps: - a construction stage, comprising the sub-stages of: - construction (E21, E22) of a first group of characteristic maps (FMi); - transformation (E24) of said first group of characteristic maps (FMi) to obtain a second group of characteristic maps (FMSi) at the resolution of the input signal - for at least one sample, called current sample (Pu), of the signal to be coded, associated with a position (x n , y n ) in the signal to be coded: - construction (E25) of a characteristic vector (Z n ) from said characteristic maps (FMSi) of said second group, as a function of said position (x n , y n ) of said current sample (P n ) ; - processing (E26) of said characteristic vector (Z n ) by an artificial neural network (MFP) defined by a set of parameters (Wk), to provide a vector representative of a decoded value (P' n ) of the current sample; - updating (E22, E26) at least one value of one of said characteristic maps of said first group and / or at least one parameter of said network, as a function of a coding performance measurement, - a coding step (E23, E27, E29) of said first group of characteristic maps (FMi) and of said set of parameters (Wk).

2. Method for coding a signal according to claim 1, characterized in that it comprises a step of constructing (E21, E22) a third group of characteristic maps (FMEi), and in that the characteristic vector is further constructed from said characteristic maps of the third group.

3. Method for coding a signal according to claim 1 or 2, characterized in that at least one of said characteristic maps of the first group (FMi) has a lower resolution than that of the signal to be coded and in that the transformation operation comprises oversampling.

4. Method for coding a signal according to one of the preceding claims, characterized in that at least one of said characteristic maps of the first or third group (FMi, FMEi) has a resolution identical to that of the signal to be coded.

5. Method for coding a signal according to one of the preceding claims, characterized in that the construction of said vector (Z n ) characteristic comprises a sub-step of extracting a value from said at least one characteristic map of the second or third group (FMSi, FMEi) at an identical position (x n , y n) to that of the current sample (P n ) in the input signal.

6. Method for coding a signal according to one of the preceding claims, characterized in that the coding of said first group of characteristic maps comprises an entropic coding sub-step.

7. A method of decoding a signal comprising a plurality of samples (Pd n ) to be decoded comprising the following steps: - decoding (E31) of a first group of characteristic maps (FMdi); - transformation (E33) of said first group of characteristic maps (FMdi) to obtain a second group of characteristic maps (FMS'i) at the resolution of the signal to be decoded; - decoding (E35) of a set of parameters (Wck) representative of a neural network (MLP'); - for at least one sample, called the current sample (Pd n), of the signal to be decoded, associated with a position (xn, y n ) in the signal to be decoded: - construction (E34) of a characteristic vector (Zd n ) from the characteristic maps of said second group (FMS'i), as a function of said position (x n , y n ) of said current sample, and: - processing (E36) of said characteristic vector (Zd n ) by an artificial neural network (MLP') defined by the decoded parameters (Wdk) to provide a vector representative of a decoded value of the current sample (Pd n ).

8. Decoding method according to claim 7, characterized in that it comprises a step of constructing (E32) a third group of characteristic maps (FME'i) and in that the characteristic vector is further constructed from said characteristic maps of the third group.

9. Decoding method according to claim 7 or 8, characterized in that at least one of said characteristic maps of the first group (FMdi) has a lower resolution than that of the signal to be decoded and in that the transformation operation comprises oversampling.

10. Method for decoding a signal according to one of claims 7 to 9, characterized in that at least one of said characteristic maps of the first or third group (FMdi, FME'i) has a resolution identical to that of the signal to be decoded.

11. Method for decoding a signal according to one of claims 7 to 10, characterized in that the construction of said vector (Zd n ) characteristic comprises a sub-step of extracting a value from said at least one characteristic map of the second or third group (FMS'i.FME'i) at an identical position (x n , y n ) to that of the current sample (Pdn ) in the signal to be decoded.

12. Method for decoding a signal according to one of claims 7 to 11, characterized in that the decoding (E31) of said first group of characteristic maps comprises an entropic decoding sub-step.

13. Device for coding a signal (I(P n )) comprising a plurality of samples (P n ) to be coded, characterized in that said device is configured to implement: - construction (GEN, MAJ) of a first group of characteristic maps (FM0; - transformation (SE) of said first group of characteristic maps (FM0) to obtain a second group of characteristic maps (FMSi) at the resolution of the input signal; - for at least one sample, called current sample (Pu), of the signal to be coded, associated with a position (xn, y n ) in the signal to be coded: - construction (XTR) of a characteristic vector (Z n ) from said characteristic maps (FMSi) of said second group, as a function of said position (x n , y n ) of said current sample (P n ) ; - processing (MLP) of said characteristic vector (Zn) by an artificial neural network (MLP) defined by a set of parameters (Wk), to provide a vector (S n ) representative of a decoded value (P' n ) of the current sample; - updating (MAJ, NNC) of at least one value of one of said characteristic maps of said first group and / or of at least one parameter of said network, depending of a coding performance measure, - coding (FMC, NNC) of said first group of characteristic maps (FMi) and of said set of parameters (Wk).

14. Device for decoding a signal comprising a plurality of samples (Pd n) to be decoded, characterized in that said device is configured to implement: - decoding (FMD) of a first group of characteristic maps (FMdi); - transformation (SE') of said first group of characteristic maps (FMdi) to obtain a second group of characteristic maps (FMS'i) at the resolution of the signal to be decoded; - decoding (NND) of a set of parameters (Wck) representative of a neural network (MLP'); - for at least one sample, called the current sample (Pd n ), of the signal to be decoded, associated with a position (xn, y n ) in the signal to be decoded: - construction (XTR') of a characteristic vector (Zd n ) from the characteristic maps of said second group (FMS'i), as a function of said position (x n , y n ) of said current sample, and: - processing (MLP') of said characteristic vector (Zd n) by an artificial neural network (MLP') defined by the decoded parameters (Wdk) to provide a vector representative of a decoded value of the current sample (Pd n ).

15. A computer program comprising instructions for carrying out the steps of an encoding or decoding method according to claim 1 or 7 when said program is executed by a computer.

Images