Method and device for coding and decoding images

Abstract

The invention relates to a method and device for coding and decoding a signal comprising a plurality of samples. The decoding method comprises the following steps: - decoding a first group of feature maps representative of the signal; - decoding a set of parameters representative of a neural network, referred to as synthesis neural network, - for at least one sample, referred to as current sample, of the signal to be decoded, associated with a position in the signal to be decoded: - constructing a feature vector from the feature maps of said first group on the basis of said position of said vector by way of the synthesis neural network defined by the decoded parameters so as to deliver a vector representative of a decoded value of the current sample; - decoding / coding said first group of feature maps comprising, for at least one value, referred to as current value, of one of said feature maps, carrying out entropy coding of said value on the basis of a value at least of its neighbourhood.

Description

[0001] Description

[0002] Title of the invention: Method and device for coding and decoding images.

[0003] Prior art

[0004] [1] The invention relates to the general field of coding single or multidimensional signals. It relates more particularly to the compression of digital images.

[0005] [2] Digital images are generally subject to source coding aimed at compressing them in order to limit the resources required for their transmission and / or storage. There are many coding standards, such as, for still images, the JPEG family standards, and for animated images, or video, the ITU / MPEG standards (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) as well as their extensions (MVC, SVC, 3D-HEVC, etc.).

[0006] [3] Encoding an image is generally performed by dividing the image into several rectangular blocks, and encoding these blocks of pixels according to a given processing sequence. In existing video compression techniques, the processing of a block typically includes a prediction of the pixels of the block carried out using previously coded and then decoded pixels present in the image being encoded, in which case we speak of "Intra prediction", or previously coded images, in which case we speak of "Inter prediction". This exploitation of spatial and / or temporal redundancies makes it possible to avoid transmitting or storing the value of the pixels of each block of pixels, by representing at least some of the blocks by a residual representing a difference between the prediction values ​​of the pixels of the block and the actual values ​​of the pixels of the predicted block.

[0007] [4] As video formats continue to evolve to compress more and more and adapt to the variety of expected formats and communication networks, the possibilities for prediction are becoming ever greater and classic coding and decoding algorithms very complex.

[0008] [5] In addition to these classical approaches proposed by compression standards (JPEG, MPEG, ITU), approaches based on artificial intelligence, and in particular neural ones, tend to develop. [6] Some of these neural approaches can be seen as a simple extension of the notion of competition of the aforementioned compression techniques, such as the competition of prediction and transformation mode in video coding.

[0009] [7] Other approaches use the concept of "autoencoder". Autoencoders are learning algorithms based on artificial neural networks, which allow the construction of a new representation of a dataset. The architecture of an autoencoder consists of two parts: the encoder and the decoder. The encoder consists of a set of layers of neurons, which process the data in order to construct new representations called "encoded", also called "latent representations". In turn, the layers of neurons in the decoder receive these representations and filter them in order to try to reconstruct the initial data. The differences between the reconstructed data and the initial data make it possible to measure the error made by the autoencoder. Training consists of modifying the parameters of the autoencoder in order to reduce the reconstruction error measured on the different samples of the dataset.The performance of such autoencoder-based systems comes at the cost of a considerable increase in memory footprint and complexity compared to conventional approaches as proposed by compression standards. They can have millions of parameters and may require up to a million MACs (multiply-accumulate) to decode a single pixel. This makes such decoders significantly more complex than conventional decoders, which could hamper the adoption of learning-based compression.

[0010] [8] More recently, a simple image encoding technique based on a neural network was described in the article "Compression with Implicit Neural representations" by Emilien Dupont et al. (arXiv:2103.03123). The proposed encoding technique consists of fitting a neural network to an image, quantizing the network weights, and transmitting them. At decoding time, the neural network is evaluated at each pixel position to reconstruct the image. However, such a technique remains inefficient in terms of compression.

[0011] [9] There is therefore a need for a solution that allows an image or a sequence of images to be coded / compressed in a simple and efficient manner.

[0012] Statement of the invention

[0013]

[0010] The invention relates to a coding method according to claim 1 and a decoding method according to claim 8.

[0011] For the purposes of the invention, encoding, or "coding", means the operation which consists of representing a set of samples in a compact form carried for example by a digital binary train. Decoding means the operation which consists of processing a digital binary train to restore decoded samples.

[0014]

[0012] By "sample" of the signal is meant a value taken from the signal. Sampling the signal produces a series of discrete values ​​called samples. In the case of an image signal, the sample is called a pixel, which can be, for example, a color pixel traditionally represented by a triplet of values, for example (R, G, B) or (Y, U, V). The position of the sample is identified by its abscissa (x) and ordinate (y) coordinates in the image.

[0015]

[0013] By "signal comprising a plurality of samples" is meant a signal with one (audio, sound), two (image) or more than two (stereoscopic, multiscopic image, image associated with a depth map, video, etc.) dimensions. Depending on this dimensionality, the sample has one, two or more coordinates in the signal. In the case of an image signal, the position of the sample is identified by its abscissa (x) and ordinate (y) coordinates.

[0016]

[0014] By "feature maps" is meant an abstract representation of the signal comprising a plurality of variable data, discrete or not, which are also called values, for example real or integer numbers. In a known manner, these maps are also referred to as "latent representation".

[0017]

[0015] By "transformation of the characteristic maps" is meant a mathematical operation which makes it possible to transform the values ​​of a first map into values ​​of a second map. A first map, called the map of the first group, intended for coding, is arbitrary. A second map, called the transformed map, or map of the second group, has the same resolution as the input signal, i.e. it comprises as many values ​​as the input signal (resp. to be decoded) comprises samples (N). The transformation may comprise, for example, interpolation, oversampling, filtering, quantization, a Fourier transform, etc.

[0018]

[0016] By "data characteristic vector constructed from the characteristic maps as a function of a position" is meant a vector consisting of one or more elements, or data, preferably discrete, the data being constructed from the characteristic maps at a position determined by that of the sample being processed in the signal. This characteristic vector is the one that is applied to the input of the synthesis neural network. For example, in the case of a one-dimensional audio signal, such a vector can be constituted from a plurality of values ​​taken from each of the characteristic maps at the same coordinate as the sample to be coded. In the case of an image, such a vector can be constituted from a plurality of values ​​taken from each of the characteristic maps at the same abscissa and ordinate coordinates as the sample to be coded (resp. to be decoded).Once these values ​​are taken from the feature maps, they can be processed to form the feature vector, before input into the synthetic neural network, for example by quantization, filtering, interpolation, etc.

[0019]

[0017] By "synthetic neural network" is meant a neural network such as a convolutional neural network, a multi-layer perceptron, an LSTM (for "Long Short Term Memory" in English), etc. The neural network is defined for example by a plurality of layers of artificial neurons and by a set of activation, weighting and addition functions (for example, a layer can calculate y = f (Ax+b), where y and b are vectors of dimension N, x a vector of dimension M, A is a matrix of dimension MxN, and f is the activation function).

[0020]

[0018] By "parameter of the neural network" is meant one of the values ​​which characterizes the neural network, for example a weight associated with one of the neurons (filter coefficient, weighting, bias, value affecting the operation of the non-linearity, etc.)

[0021]

[0019] By "processing by a synthetic neural network" is meant the application of a function expressed by a synthetic neural network to the input characteristic vector to produce an output vector representative of the sample to be coded (resp. decoded). This output vector may comprise one or more data representative of the sample.

[0022]

[0020] By "performance measurement" is meant a measurement between at least one value of a sample to be coded and a decoded value of said sample. The measurement can evaluate for example a distortion, or a perceptual error. It can be carried out on a sample or a plurality of samples (for example, a current sample, or the current image, etc.). The measurement can also include a measurement of the flow rate, in particular associated with the coding of the synthesis neural network and / or the coding of the characteristic maps of the first group. The measurement can be a joint measurement between the flow rate and the distortion through their weighting. As is well known in the state of the art, the value of this measurement is generally minimized until a target value is reached.

[0023]

[0021] By "construction step" is meant a step which aims to construct the representative parameters of the image, before their actual coding. The construction sub-steps can be repeated as many times as necessary to obtain an acceptable performance measurement.

[0024]

[0022] Generally speaking, it is considered that the steps of a coding or decoding method should not be interpreted as being linked to a notion of temporal succession. In other words, the steps may be carried out in a different order than that indicated in the independent coding or decoding claim, or even in parallel.

[0025]

[0023] The coding method according to the invention carries out a construction of the coding parameters, from the input signal, for example an image, by training a so-called synthesis neural network on characteristic vectors associated with a position of a sample to be coded. These characteristic vectors are constructed from characteristic maps which can be at the resolution of the input signal, or at a lower resolution. During the training, or construction, the parameters of the neural network and the values ​​of the characteristic maps are updated according to a performance measurement, for example of the rate-distortion type. When the training is finished, that is to say when the performance measurement obtained is satisfactory, the actual coding of the parameters of the synthesis neural network and the values ​​of the characteristic maps can be carried out and stored or transmitted to the decoder.

[0026]

[0024] Advantageously, the training process makes it possible to refine the parameters of the synthetic neural network and / or the values ​​of the characteristic maps until an adequate representation in terms of performance is obtained, for example a desired balance between the generated flow rate and the distortion undergone by the input signal. The training of the values ​​of the characteristic maps and the parameters of the synthetic neural network can be joint. Advantageously, the coding method according to the invention makes it possible to efficiently compress the signal.

[0027]

[0025] Advantageously, the decoding method is simple since it is sufficient to decode the characteristic maps and the synthesis neural network to reconstruct a decoded version of the signal, for example an image.

[0026] Advantageously, the coding of the characteristic maps is made particularly efficient by taking into account the coding neighborhood, which makes it possible to exploit the redundancies present in the maps.

[0028]

[0027] Such a synthetic neural network can advantageously be of very simple structure with few parameters.

[0029]

[0028] Furthermore, decoding can be performed sample by sample, in a progressive manner.

[0030]

[0029] According to embodiments of the coding or decoding method:

[0031]

[0030] - The coding method comprises the following sub-steps of coding said current value of one of said characteristic maps;

[0032] - construction of a neighborhood vector from said characteristic maps of said first group, and;

[0033] - processing of said neighborhood vector by an artificial neural network, called a prediction neural network, defined by a set of parameters, to provide a prediction of said current value;

[0034] - updating at least one parameter of said prediction network, based on the coding performance measurement, coding said set of parameters of the prediction network.

[0035]

[0031] The decoding method comprises the following sub-steps of decoding said current value, of one of said characteristic maps;

[0036] - decoding of a set of parameters representative of a neural network, called a prediction neural network;

[0037] - construction of a neighborhood vector from said characteristic maps of said first group, and;

[0038] - processing of said vector by the prediction neural network, to provide a prediction of said current value;

[0039]

[0032] Advantageously, according to this coding or decoding method, the characteristic maps are efficiently compressed by a prediction neural network capable of predicting a value of the map as a function of its neighborhood. By "neighborhood vector" is meant a vector consisting of one or more elements, or data, constructed from the characteristic maps at a position close to that of the current sample, which is also that of the current value in the characteristic map. The neighboring position may indicate a value in the map currently being processed (for example, the neighboring value at the top left of the one currently being processed) or in another characteristic map (for example, the neighboring value at the same position in the previous map). This neighborhood vector is the one that is applied to the input of the prediction neural network.By "prediction" we mean at least one piece of data that can be used to estimate a current value of the feature map, for example a probability, a statistical value, etc. The prediction neural network, trained on the image maps, is able to represent them efficiently. It is also inexpensive to code.

[0040]

[0033] The method comprises a step of transforming said first group of characteristic maps to obtain a second group of characteristic maps at the resolution of the input signal, the method being characterized in that said characteristic vector is constructed from said characteristic maps of the second group.

[0041]

[0034] Advantageously, according to this mode, the characteristic maps are divided into two groups, one of which is reserved for the extraction of the characteristic vectors and the other for coding. It is thus possible to dissociate the two processes whose purpose is different: the maps of the first group, to be coded (resp. decoded), must be compressed as efficiently as possible, while the maps of the second group must be able to facilitate the process of extraction and construction of the characteristic vector.

[0042]

[0035] - According to a variant, at least one of the characteristic maps of the first group has a lower resolution than that of the signal to be coded (resp. decoded) and the transformation operation includes oversampling. Advantageously, according to this mode, the compression of the characteristic maps is more efficient since at least one of the characteristic maps of the first group, to be coded (resp. decoded), includes fewer values ​​than if it were at the resolution of the signal. For example, in the case of a digital image, one of the characteristic maps of the first group may be at 1 / 2 resolution, that is to say it includes half as many values ​​on the abscissa and ordinate as the input image has samples, i.e. in total 4 times fewer values ​​than a characteristic map at the resolution of the image.On the other hand, the characteristic map of the second group which corresponds to a transformation of this map of the first group has a resolution identical to that of the signal. The transformation therefore comprises in this case at least one oversampling operation to obtain the same number of values ​​in this transformed map as the input signal (resp. to be decoded) has samples.

[0036] - At least one of said characteristic maps of the first group has a resolution identical to that of the signal to be coded (resp. decoded).

[0043]

[0037] Advantageously according to this embodiment, at least one of the characteristic maps, at the same resolution as the input signal to be coded (resp. to be decoded) allows significant fidelity and respect for the details of the initial resolution of the signal. The transformation according to one embodiment retains in this case the number of values ​​of the transformed characteristic map; it can be reduced to identity (no processing is carried out on the values ​​of the map of the first group) or include a filtering operation, quantification, a Fourier transformation, etc.

[0044]

[0038] During the coding step, quantization is essential for the proper functioning of the system if the characteristic maps include, for example, floating point or real values. It is necessary to quantize them before coding them and / or providing them as input to the synthesis and / or prediction neural network. During decoding, however, inverse quantization is not necessary, depending on the embodiments.

[0045]

[0039] - The construction of said characteristic vector comprises a sub-step of extracting a value from said at least one characteristic map at a position identical to that of the current sample in the signal to be coded (resp. decoded).

[0046]

[0040] Advantageously, it is possible to extract a value from a characteristic map of the first or second group, at the same position as the sample in the signal (input signal for coding, signal to be decoded for decoding), to constitute an element of the characteristic vector. This method is simple to implement. For example, if J characteristic maps are available as input, with the same resolution as the signal, a simple extraction of the values ​​of the maps at the coordinates of the current sample (at the same abscissa and the same coordinate in the characteristic map) makes it possible to directly construct the characteristic vector of J elements.

[0047]

[0041] - The construction of said characteristic vector comprises the following sub-steps:

[0048] - extracting a plurality of values ​​from said feature maps of said first group as a function of said position of said current sample;

[0049] - processing of said extracted values ​​to obtain the characteristic vector.

[0050]

[0042] Advantageously according to this mode, the characteristic vector is extracted from the characteristic maps, which can be any, and in particular at a resolution lower than that of the signal to be coded (resp. decoded), before undergoing processing. Such processing can for example correspond to a quantization of the extracted data, or to a scaling, or to a filtering, etc. During coding, a quantization is essential for the proper functioning of the system if the characteristic maps comprise for example floating, or real values. It is necessary to quantize them before coding them and / or providing them as input to the synthesis and / or prediction neural network. During decoding, however, inverse quantization is not necessary, according to the embodiments.

[0051]

[0043] - The method comprises a step of constructing a third group of characteristic maps and the characteristic vector is further constructed from said characteristic maps.

[0052]

[0044] Advantageously, these additional maps of the third group, constructed in an identical manner to the coder and the decoder, are neither stored nor transmitted at the coder level, nor decoded at the decoder level. They thus make it possible to benefit from additional data to improve the compression without degrading the flow rate. They may for example include coordinates, causal data available in the maps of the first or second group, data concerning other images already processed by the coder or decoder, etc.

[0053]

[0045] Correlatively, the invention also relates to a coding device and a decoding device.

[0054]

[0046] The characteristics and advantages of the coding or decoding method apply in the same way to the coding or decoding device according to the invention and vice versa.

[0055]

[0047] The invention also relates to a computer program on a recording medium, this program being capable of being implemented in a computer or a control device in accordance with the invention. This program comprises instructions adapted to the implementation of the corresponding method. This program can use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0056]

[0048] The invention also relates to an information medium or a recording medium readable by a computer, and comprising computer program instructions mentioned above. The information or recording media can be any entity or device capable of storing the programs. For example, the media can comprise a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a floppy disk or a hard disk, a DNA sequence, or a flash memory. Furthermore, the information or recording media can be transmissible media such as an electrical or optical signal, which can be conveyed via an electrical or optical cable, by radio link, by wireless optical link or by other means.

[0057]

[0049] The program according to the invention can in particular be downloaded from an Internet-type network.

[0058]

[0050] Alternatively, each information or recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of a method according to the invention.

[0059] Brief description of the drawings

[0060]

[0051] Other characteristics and advantages of the present invention will emerge from the description given below, with reference to the appended drawings which illustrate exemplary embodiments thereof which are not in any limiting nature.

[0061] [Fig. 1] Figure 1 schematically represents a coding device used in the context of the invention;

[0062] [Fig. 2] Figure 2 schematically represents a decoding device used in the context of the invention;

[0063] [Fig. 3] Figure 3 illustrates an example of a synthetic artificial neural network used in the context of the invention;

[0064] [Fig. 4] Figure 4 illustrates an example of an artificial prediction neural network used in the context of the invention;

[0065] [Fig. 5] Figure 5 is a flowchart representing an example of an encoding method that can be implemented by the encoding device of Figure 1;

[0066] [Fig. 6] Figure 6 illustrates an encoding method used in one embodiment of the invention;

[0067] [Fig. 7] Figure 7 is a flowchart representing an example of a decoding method that can be implemented by the decoding device of Figure 2; [Fig. 8] Figure 8 illustrates a decoding method used in an embodiment of the invention.

[0068] [Fig. 9] Figure 9 is a flowchart representing a method of encoding feature maps which can be implemented by the encoding device of Figure 1 and by the encoding method of Figure 5.

[0069] [Fig. 10] Figure 10 is a flowchart showing a feature map decoding method that can be implemented by the decoding device of Figure 2 and the decoding method of Figure 7.

[0070] [Fig. 11] Figure 11 illustrates a method of encoding or decoding feature maps.

[0071] [Fig. 12] Figure 12 illustrates another method of encoding or decoding feature maps.

[0072] Description of the embodiments

[0073]

[0052] Figure 1 schematically represents an ENC coding device.

[0074]

[0053] This ENC coding device comprises a GEN module for generating feature maps, an FME module for generating additional feature maps, a SE transformation module, an XTR data extraction module, a TT processing and quantification module, an MLP module corresponding to a synthetic artificial neural network, an NNC neural network coding module capable of coding the synthetic neural network and optionally the prediction neural network, an FMC module for coding feature maps, an EVAL module for evaluating coding performance, and an MAJ update module.

[0075]

[0054] The ENC coding device can be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the modules mentioned above can then be realized by the cooperation of the processor and computer program instructions stored in the aforementioned memory and designed to carry out the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0076]

[0055] The ENC coding device of Figure 1 receives as input a succession of samples to be coded, noted P n , for example a temporal succession of sound samples, or a set of image data noted I (P n ). In this second case, the image signal I (P n) may represent a two-dimensional image, or a plurality of two-dimensional images (video, color components, stereoscopic, multiscopic components, etc.). Pn denotes a sample n of the input signal comprising N samples. In one embodiment, the signal is a color image signal represented using at least one two-dimensional representation, such as a matrix of pixels, each pixel comprising a red (R), green (G), blue (B), or alternatively, a luminance (Y) component and at least one chrominance (U,V) component. The location of each pixel is defined by its abscissa and ordinate (x and y) coordinates in the image. In one embodiment, the image is in grayscale represented using a two-dimensional representation, such as a matrix of pixels, each pixel comprising a grayscale, or luminance, component.In this case, the vector representing the pixel is reduced to a single component.

[0077]

[0056] As will be described in more detail later with reference to Figures 3 to 7:

[0078]

[0057] The GEN module for generating feature maps is configured to generate a plurality of M feature maps denoted FMi. The optional FME module can generate one or more additional maps, numbering F, which will not be coded or transmitted, denoted FMEi.

[0079]

[0058] In one embodiment, the SE module performs a transformation of the first group of feature maps FMi to generate a second group of feature maps FMSi at the same resolution as the input signal.

[0080]

[0059] The optional SE module can perform a quantification of the data extracted from this set of M FMi cards, or from the vector Z nconstituted from this data. Recall that quantizing a value refers to matching that value to a member of a discrete set of possible code symbols. For example, the set of possible code symbols may consist of integer values, and the quantization system performs a simple rounding of a real value to an integer value. In another example, quantization consists of multiplication by a given value and then rounding. Then the SE module performs a transformation of the values ​​of at least one of the feature maps, for example, upsampling, interpolation, filtering, etc. At the end of the transformation, a transformed feature map of the second group has the same resolution as the images in the input sequence.Advantageously, according to this mode, the characteristic maps which are coded can be of lower resolution than that of the images to be coded, while the maps of the second group, which are used for the construction of the characteristic vectors, are at the same resolution as the sequence of images, which facilitates the extraction of the values.

[0081]

[0060] In one embodiment, the SE module is absent, in this case the values ​​which will be used to construct the characteristic vector are extracted from the first group of characteristic maps.

[0082]

[0061] The XTR module performs an extraction of values ​​in the FMi characteristic maps (and / or FMSi and / or FMEi, according to one of the embodiments described previously), for a current sample P n to be encoded, depending on its coordinates in the input signal. For example, if we want to encode the sample P n at the coordinates (x n , y n) of an input image, the XTR module performs an extraction of values ​​in the maps at positions imposed by the coordinates (x n , y n ).

[0083]

[0062] In one embodiment, the extracted values ​​constitute the vector Z n . Z n is a J-tuple, that is, it has J elements, or data Zi. The vector Z n of index n refers to the characteristic vector of the pixel P' n .

[0084]

[0063] In one embodiment, the optional TT module performs processing of the extracted values, to generate the vector Z n. The TT module can perform quantization of the data extracted from the set of feature maps. The processing may include other operations, such as filtering, scaling, etc. In particular, if the SE module is not used and if the feature maps of the first group are at lower resolutions than the images in the sequence, the TT module can take into account the coordinates of the values ​​in the lower resolution maps.

[0085]

[0064] It will be noted that at least one of the SE or TT modules must perform a quantification of the characteristic maps.

[0086]

[0065] The MLP module is a synthetic neural network defined by K parameters Wk, capable of processing the vector Z n , or J-Uplet, as input, to generate as output a second vector representative of the sample P nto be coded. The synthetic neural network is, according to one embodiment, an MLP, or Multi Layer Perceptron, consisting of an input layer adapted to the input format (the J-tuple), optionally one or more hidden layer(s), and an output layer adapted to the output format of the output vector, generally a vector comprising A elements. According to one embodiment, A is equal to 3 and the output vector is the triplet (R, G, B) of the pixel P' ncoded then decoded.

[0066] The NNC module performs the coding of the synthesis neural network, in particular of its parameters Wk. Optionally, the NNC module performs the coding of the prediction neural network, ARM, in particular of its parameters Ob. During the process of training, or construction, of the coding, that is to say as long as the step of evaluating a performance is not satisfactory, the NNC module performs a coding simulation, followed by a decoding, intended for the evaluation module. Subsequently, it performs the actual coding of the parameters of the synthesis neural network Wk, and optionally the coding of the prediction neural network, ARM. The coded parameters are noted Wck and Ocb. In a known manner, the coding simulation can be identical to the actual coding, or produce an approximation thereof.

[0087]

[0067] The FMC module performs the coding of the FMi maps, i.e. the values ​​of the characteristic maps of the first group (excluding the additional FMEi maps, and the maps of the second group, optionally resulting from oversampling by the SE module). During the process of training, or constructing, the coding, i.e. as long as the performance evaluation step is not satisfactory, the FMC module performs a coding simulation, followed by decoding, intended for the evaluation module. Subsequently, it performs the actual coding of the values ​​of the FMi maps. The coded maps are denoted FMci. In a known manner, the coding simulation can be identical to the actual coding, or produce an approximation thereof. The coding module quantizes, if necessary, the latent representation of the values ​​of the maps of the first group by using a quantizer to generate an ordered collection of quantized values.Then the encoding module compresses the quantized data, using an encoding that takes into account the neighborhood of a value to be encoded from the feature map. As will be detailed later, the FMC module can include a prediction neural network (ARM).

[0088]

[0068] The EVAL module performs an evaluation and minimization of a coding performance. The evaluation function is for example of the rate-distortion type. The minimization can be carried out by gradient descent, or any other method within the reach of those skilled in the art.

[0089]

[0069] The MAJ module updates the values ​​of the FMi cards to be encoded, based on the results of the performance function.

[0090]

[0070] Figure 2 schematically represents a DEC decoding device.

[0091]

[0071] The DEC coding device of figure 2 receives as input a first group of encoded data organized into M FMci characteristic maps (also called FM layers) and the encoded parameters Wck of the MLP' synthesis neural network, optionally the encoded parameters Ocb of the ARM' synthesis neural network.

[0092]

[0072] This DEC decoding device comprises a neural network(s) decoding module NND capable of decoding the MLP' synthesis neural network and optionally the ARM' prediction neural network, a FMD module for decoding the feature maps, a XTR' data extraction module, an SE' inverse transformation module, a TT' inverse processing and quantization module, an MLP' module corresponding to a synthesis neural network, an FME' module for generating additional feature maps. According to one embodiment, it produces at output a decoded image, denoted I (Pd n), comprising a plurality of decoded samples Pd n .

[0093]

[0073] The maps decoded by the FMD module, numbering M, are denoted FMdi. The parameters of the synthesis neural network (MLP') decoded by the NND module are denoted Wdk. The parameters of the prediction neural network (ARM') decoded by the NND module are denoted Odb.

[0094]

[0074] The FME' module of the decoder can also generate one or more additional cards, denoted FME'i, L in number, identical to the additional FMEi cards generated by the encoder.

[0095]

[0075] In one embodiment, the module SE' performs a transformation of the first group of decoded feature maps FMdi to generate a second group of feature maps at the same resolution as the signal to be decoded, denoted FMS'i. The module SE' optionally performs an inverse quantization corresponding to the quantization performed at the encoder. The inverse quantization is not necessary if the quantizer Q of the encoder has performed a simple rounding of the real values ​​submitted to it. The inverse quantization is also not necessary if the neural network is capable of taking into account a quantization of its input data. Otherwise, the decoder performs the inverse operation of the quantizer Q. Then the module SE' performs a transformation of the values ​​of the feature maps, comprising for example an oversampling, an interpolation, a filtering, etc., similar to that performed by the encoder.At the end of the transformation, a transformed feature map of the second group has the same resolution as the images of the sequence to be decoded.

[0076] In one embodiment, the SE' module is absent, in this case the values ​​which will be used to construct the feature vector are extracted from the first group of feature maps.

[0096]

[0077] The XTR' module is identical to the XTR module of Figure 1. It performs an extraction of values ​​from the M FMdi characteristic maps, (and / or FMS'i and / or FME'i, according to one of the embodiments described previously), for a sample Pd n to be decoded, depending on its coordinates in the signal to be decoded. In one embodiment, J=M. In one embodiment, J=M+L.

[0097]

[0078] In one embodiment, the extracted values ​​constitute the vector Zd n . Zd n is a J-tuple, that is, it has J elements, or zdi data.

[0098]

[0079] In one embodiment, the optional TT' module performs processing of the extracted values, to generate the vector Zd n . The TT' module can perform inverse quantization of the data extracted from the feature map set. The processing may include other operations, such as filtering, scaling, etc., similar to those performed by the encoder.

[0099]

[0080] The MLP' module is a so-called synthesis neural network, defined by K parameters Wdk, capable of processing the vector Zd n , or J-Uplet, as input, to generate as output a second vector representative of the sample P n to be decoded, generally a vector comprising A elements. According to one embodiment, K=3 and the output vector is the triplet (R, G, B) of the pixel Pd ndecoded. The MLP' module is identical in structure to the MLP module, and its parameters are either identical if the coding of its Wk parameters is lossless, or different if the coding is lossy.

[0100]

[0081] When all samples P n of the signal has been decoded, we have a reconstructed signal I (Pd n ), according to an example an image I comprising N samples decoded in the form of N vectors Pd n .

[0101]

[0082] The decoding device DEC can be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the modules mentioned above can then be realized by the cooperation of the processor and computer program instructions stored in the aforementioned memory and designed to carry out the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor

[0083] Figure 3 illustrates an example of a synthetic artificial neural network used for coding and decoding in the context of embodiments of the invention.

[0102]

[0084] The synthetic artificial neural network used for coding, MLP, and the synthetic artificial neural network used for decoding, MLP', are defined by an identical structure, comprising for example a plurality of layers of artificial neurons, and by a set of weights and activation functions associated respectively with the artificial neurons of the network concerned.

[0103]

[0085] A vector representation of a current sample (a vector Z n or Zd n from the FMi / FMSi and FMEi feature maps to the encoder or FMdi / FMS'i and FME'i to the decoder) is applied as input (i.e. on an input layer) of the MLP or MLP' synthesis artificial neural network. The synthesis artificial neural network produces as output a vector, according to one embodiment the color components (R, G, B) constituting a color pixel of an image.

[0104]

[0086] The concatenation of all these reconstructed pixels in an image (2D, 3D) constitutes the decoded, or reconstructed, image.

[0105]

[0087] At the encoder, the MLP synthesis artificial neural network is trained on the image, so as to minimize the differences between the input representation of the current image I (P n ) and its representation I (P' n ) output, while also minimizing the amount of data to be encoded. The EVAL module performs a performance measurement in this regard.

[0106]

[0088] Once the encoder training is complete, the network parameters are encoded, either lossless, in which case the MLP' neural network is identical to MLP, or lossy, in which case the MLP' network may be slightly different from MLP.

[0107]

[0089] Figure 4 illustrates an example of a prediction artificial neural network used for encoding (ARM) and decoding (ARM') for the feature maps, within the framework of embodiments of the invention.

[0108]

[0090] The artificial prediction neural network used for coding, ARM, and the artificial prediction neural network used for decoding, ARM', are defined by an identical structure, comprising for example a plurality of layers of artificial neurons, and by a set of weights and activation functions associated respectively with the artificial neurons of the network concerned.

[0109]

[0091] A vector representation of a current neighborhood (a vector C n or CD nfrom the FMi / FMSi and FMEi feature maps at the encoder or FMdi / FMS'i and FME'i at the decoder) is given as input (i.e. on an input layer) of the ARM (at coding) or ARM' (at decoding) prediction artificial neural network.

[0110]

[0092] The prediction artificial neural network behaves as a function that produces as output a prediction of the current value of the feature map being processed, which can be in the form of a prediction value or probability data.

[0111]

[0093] According to one embodiment, at the encoder, the network implements a function f\ which provides an expected mean and / or variance (pc) for the current value of the current value V nof the FMi map to be encoded. These statistical values ​​are used to perform the entropy coding of this value. For example, if the function produces a mean, this mean is subtracted from the current value and only the difference is entropy encoded, this mean being considered as a prediction of the current value. Alternatively, if the function produces a mean and a variance, the mean is subtracted from the current value, and the difference is entropy encoded using an entropy coding adapted to the variance, for example by quantizing said variance into a predetermined set of variances and associating a type of entropy coding with each quantized variance value. At the decoder, the network implements a function f\ which provides an expected mean and / or variance for the current value of the current value Vd nof the FMdi map to be decoded. These statistical values ​​are used to perform the entropy decoding of this value. For example, if the function produces a mean, the current value is decoded by the entropy decoder and this mean is added to the current value. Alternatively, if the function produces a mean and a variance, the current value is decoded by the decoder using an entropy decoding adapted to the variance, for example by quantizing said variance into a predetermined set of variances and associating an entropy decoding type with each quantized variance value.

[0112]

[0094] According to another embodiment, the neural network can produce the expected probability (pr) of each possible value of the current sample. In this case, the entropy coding or decoding will be adapted to this probability (as is known for Huffman or arithmetic entropy coding).

[0113]

[0095] At the encoder, the ARM prediction artificial neural network is trained on the image, so as to minimize the amount of data to be encoded. The EVAL module performs a performance measurement in this sense. It is recalled that the overall performance measurement relates to minimizing the distortion between coded and then decoded images I (P' n ) and the input images I (P n ), while minimizing the coding rate. In one embodiment, the feature maps are losslessly encoded, using entropy coding. In this case, the coding of the feature maps affects the rate, but not the distortion of the coded images. In another embodiment, if the feature maps are lossy encoded, the coding of the feature maps affects the rate, but also the distortion.

[0114]

[0096] Once training is complete, the B Ocb parameters of the network are encoded, either losslessly, in which case the ARM' neural network is identical to ARM, or lossily, in which case the ARM' network may be slightly different from ARM.

[0115]

[0097] Figure 5 is a flowchart representing an example of an encoding method that can be implemented by the encoding device of Figure 1.

[0116]

[0098] According to this embodiment, the signal is a two-dimensional image, each sample to be coded is therefore a pixel P n of coordinates (x n , y n ).

[0117]

[0099] Encoding takes place in two main phases:

[0118]

[0100] In a first phase, called the construction phase, learning is carried out in order to determine, for an input signal I (P n), the values ​​of the FMi maps and the parameters Wk and optionally Ob to optimize a global cost function. The learning is for example carried out by a gradient descent, followed by an update of the parameters of the MLP synthesis neural network, the values ​​of the FMi feature maps, and optionally the ARM prediction neural network. As is known in the state of the art, the cost function can be of the rate-distortion, or rate, or distortion, or perceptual type. To measure the rate R, it is necessary to simulate the coding of the FMi maps, then to measure the associated coding rate (the size of the stream Bl). According to one embodiment, the coding of the parameters Wk and / or Ob is not simulated because their influence is less important than that of the feature maps. According to one embodiment, the coding of the parameters Wk and / or Ob is also simulated and the associated rate is measured (the size of the stream B2).To measure the distortion D, it is necessary to simulate the coding then the decoding of at least part of the image I, to obtain at least one pixel P'. n resulting from a simulation of coding then decoding, then measuring the difference between this part of the image I (P n ) as input and a corresponding part of image I (P' n ) encoded then decoded.

[0119]

[0101] Then during a second phase, called the coding phase, the FMi maps and the parameters Wk and possibly Ob are encoded to produce the coded values ​​FMci and Wck (and possibly Ocb) before transmission or storage. They constitute the compressed representation of the input signal I (P n ).

[0120]

[0102] We will now describe the steps of a method according to one embodiment of the invention.

[0121]

[0103] During a step E20, a signal I (P n ) to be coded, comprising a plurality of N samples Pn , is provided as input to the process.

[0122]

[0104] During a step E21, the M FMi maps of the first group are initialized. Subsequently, the parameters Wk of the MLP synthesis neural network and the values ​​of the FMi maps, and possibly the parameters Ob of the prediction neural network must be optimized during the construction phase.

[0123]

[0105] According to one embodiment, the FMi cards have the same resolution as the input signal I (P n ) and therefore each have the same number of values ​​N as there are samples P n to code.

[0124]

[0106] According to one embodiment, the FMi cards have a resolution lower than or equal to that of the input signal I (P n) and therefore include, for at least one of them, a number N' of values ​​to be coded less than N. According to a variant, the first FMi card is at the resolution of the images and each following card is at a resolution half that of the previous one.

[0125]

[0107] According to one embodiment, several FMi cards have the same resolution, lower than that of the input signal I (P n ).

[0126]

[0108] According to one embodiment, the FMi maps are transformed to provide a second group of transformed feature maps FMSi. In this mode, the feature vectors are preferably extracted from the transformed maps of the second group, and not directly from the maps of the first group. In this mode, the feature vectors are therefore extracted indirectly from the maps of the first group. The maps of the second group are not coded, they are used only for the construction of the feature vectors.

[0127]

[0109] According to one embodiment, the FMi cards are initialized by predefined constant values.

[0128]

[0110] According to another embodiment, the characteristic maps are initialized by a set of random real values.

[0129]

[0111] According to one embodiment, one or more FMEi maps, forming an additional group of L additional characteristic maps, are generated and added to the first group. They are used to construct the characteristic vector but are neither stored nor transmitted.

[0130]

[0112] The FMi characteristic maps of the first group are subsequently updated, or refined, during a step E22, by the update module MAJ of the encoder during its learning.

[0131]

[0113] During a step E23, the FMi maps of the first group are coded by the FMC module of the encoder. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is an actual coding and the coded values ​​constitute the stream Bl. The simulation may be identical to the actual coding but it may also be different (for example, simplified). For this coding, a technique for predicting a feature map value by its neighborhood is used, as will be described for example with reference to FIG. 9. In one embodiment, the structure and the parameters Ob of the prediction neural network are initialized for example during the first iteration of this step. These parameters are subsequently updated, or refined, during the construction phase, during subsequent iterations of the method.

[0132]

[0114] In one embodiment, the FMi cards are coded in order (FMi, FM2,..., FM4), and the variables of each card in a predefined order, for example lexicographic. Each card undergoes entropic coding. The entropic coding produces a compressed stream B1 whose flow rate is subsequently measured during a step E29.

[0133]

[0115] During a step E24, according to one embodiment, the M maps of the first group FMi are transformed by the module SE to generate maps of the second group FMSi at the resolution of the images of the input sequence.

[0134]

[0116] According to one embodiment, M FMSi maps are generated.

[0117] According to one embodiment, each FMi map is transformed into an FMSi map.

[0135]

[0118] According to one embodiment, at least one FMi map has a lower resolution than the images of the sequence to be coded and the transformation operation comprises oversampling so that the transformed FMSi map has the same number of samples as the images of the sequence. Oversampling consists of adding values ​​to the FMSi maps to achieve the resolution of the images of the input sequence. It can be simple (by nearest neighbor replication) or include interpolation (linear, polynomial, by filtering, etc.).

[0136]

[0119] During a step E25, values ​​are extracted by the XTR module from the transformed FMi or possibly FMSi maps, and optionally additional FMEi. This extraction is carried out according to the coordinates (x n , y n ) of sample P n of the input signal. It can also be performed according to the resolution of the card considered.

[0137]

[0120] According to one embodiment, the vector Z n characteristic results directly from this extraction.

[0138]

[0121] The samples to be coded are for example processed in sequential order, from n=1 to n=N.

[0139]

[0122] According to one embodiment, during a step E26, the vector Z n characteristic is constructed by the TT module from the values ​​extracted from the FMi or FMSi maps and optionally FMEi for each sample P n of coordinates (x n , y n ) of the input signal. The processing may include quantization of the values ​​extracted from the FMi maps or the Z vector n constituted, if necessary. The processing may include other operations, such as filtering, scaling, applying any function, preferably monotonic, etc.

[0140]

[0123] In one embodiment, Z nhas as many values ​​as there are FMi or FMSi (and optionally FMEi) input cards. In this case we have J=M (+L).

[0141]

[0124] In one embodiment, Z n is a J-uplet (zi, Z2, . . . , zj), consisting of the values ​​of the maps FMi or FMSi (and optionally FMEi) located at the coordinates (x n , y n ) of the current pixel P n , as will be illustrated in support of Figure 6.

[0142]

[0125] In one embodiment, Z n is a J-tuple constructed from values ​​taken from FMi (and optionally FMEi) maps at coordinates that may be different between maps. For example, if the FMi (and / or FMEi) maps are at different resolutions because they have been downsampled, the coordinates are adapted (by scaling) to match the resolution of each map.

[0143]

[0126] In one embodiment, Z nis a J-tuple constructed from values ​​taken from the FMi (and optionally FMEi) maps by applying the processing to one or more values ​​of the maps, for example a filtering of the values ​​close to the targeted value in a map. For example, for a current sample P n in an FMi map which is at the same resolution as the input signal, we can extract the values ​​located at the coordinates (x n , y n ), (x n -l, y n ), (x n , y n -l) and (x n -l, y n -l) and apply processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (z0 of element i of vector Z n relative to this FMi or FMEi card. According to another example, in an FMi card which is at half resolution of the input signal, we can consider the values ​​located at the coordinates (x n / 2, y n / 2), (xn / 2-1, y n / 2), (xn / 2, y n / 2-l) and (x n / 2-l, y n / 2-l) and apply processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (zi) of element i of vector Z n relating to this FMi or FMEi card.

[0144]

[0127] During a step E27, the vector Z n is processed by the MLP synthesis neural network to generate as output a vector representative of the sample P n to code, according to one embodiment, the triplet (R, G, B) of the sample P' n (sample P n coded then decoded).

[0145]

[0128] The structure and parameters Wk of the synthetic neural network are initialized for example during the first iteration of this step. These parameters are subsequently updated, or refined, during the construction phase, during subsequent iterations of the method.

[0146]

[0129] According to one embodiment, the parameters of the synthesis neural network and / or the prediction neural network are initialized by predefined values ​​known to give a satisfactory result (for example, following training on a corpus of images).

[0147]

[0130] According to another embodiment, the parameters of the synthesis neural network and / or the prediction neural network are initialized by a set of random values.

[0148]

[0131] During a step E28, the parameters Wk of the MLP synthesis neural network and the parameters Ob of the ARM prediction neural network, if it exists, are quantified and coded. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is an actual coding and the coded values ​​constitute the stream B2. The simulation can be identical to the actual coding but it can also be different (for example, simplified). Any known technique can be used for this purpose, for example the neural network coding standard proposed by the MPEG-7 part 17 standard, also called NNR (Neural Network Representation). It should be noted that in this case, the amount of degradation that the coding brings to the weights Wket, optionally Ob, must be chosen.

[0149]

[0132] During a step E29, a performance measurement is evaluated.

[0150]

[0133] For this purpose, the coding simulation rates associated with the characteristic maps of the first group (simulation of the B1 flow by coding the FMi maps) and optionally with the parameters of the neural network(s) (simulation of the B2 flow by coding the Wket parameters, optionally Ob) are measured.

[0151]

[0134] According to one embodiment, the cost function is of the rate-distortion type, noted (D+L*R), where D for example the quadratic error measured between the input signal and the decoded signal (or the error measured on a subset of samples of the signal). According to another example, D is calculated from a perceptual function such as the SSIM (for Structural SIMilarity), or MSSSIM (for Multi-scale Structural SIMilarity). According to one embodiment, R is the simulated rate of the stream B1; according to another embodiment, R is the total rate used to code this image, that is to say the sum of the simulated rates of B1 and B2. L is a parameter which regulates the rate-distortion trade-off. Other cost functions are possible.

[0152]

[0135] As long as the cost function has not reached its minimum, the performance measurement is not satisfactory, and the method is repeated from step E22. This minimization can be carried out by a mechanism known as gradient descent with updating of the parameters during step E22 for the values ​​of the characteristic maps and E23, E27 for the parameters of the network(s).

[0153]

[0136] During a step EF, if the cost function has reached its minimum, the training stops. If an encoded version corresponding to the last simulation of the parameters of the synthesis neural network (Wk) and the feature maps (FMi) is available, the streams B1 and B2 can be constituted therefrom. According to another embodiment, the actual encoding of the updated parameters of the synthesis neural network (Wk) and the values ​​of the feature maps (FMi) and optionally of the prediction neural network (Ob) is carried out at this step to produce the encoded parameters Wck (optionally Ob) and FMci which constitute streams B1 and B2. The streams B1 and B2 can be concatenated to produce a final stream. According to one embodiment, the stream B2 of the coded parameters of the neural network(s) is (are) stored or transmitted before the stream B1, in order to be able to be decoded before the stream B1.Figure 6 illustrates an encoding method used in one embodiment of the invention.

[0154]

[0137] In this embodiment, there are 4 FMi maps generated. In a preferred embodiment, there are 7.

[0155]

[0138] The first FMi map has the same resolution as image I (P n ), and therefore has WxH variables, where W represents the width of the image in pixels, and H its height. The second FM2 map has half the resolution (in each dimension) of the FMi map. Each additional map has half the resolution of the previous map. This structure allows to reduce the number of variables of the feature maps which facilitates coding and training while minimizing the coding cost.

[0156]

[0139] The FM2 map is oversampled by the SE module by a factor of 2 in each dimension, according to a method illustrated in support of Figure 6. The FM3 map is oversampled by a factor of 4 in each dimension, and the FM4 map by a factor of 8 in each dimension.

[0157]

[0140] The resulting FMSi maps are of the same resolution as the image I (P n ), and therefore each have WxH values, where W represents the width of the image in pixels and H its height (N=WxH).

[0158]

[0141] According to this embodiment, the FMSi layers are quantified by the SE module.

[0159]

[0142] Other types of structure are possible, for example one can use a reduction rate different from half between the cards (a quarter, or a third, etc.).

[0160]

[0143] According to a variant presented in dotted lines, the characteristic maps are 5 in number: an additional FMEo map has been introduced, which will neither be coded nor transmitted. This additional map typically contains data which can assist the MLP network in the task of signal reconstruction. Thus, the added maps can be one or more from the following non-limiting list:

[0161]

[0144] - A map showing at each point the abscissa of that point.

[0162]

[0145] - A map showing at each point the ordinate of that point.

[0163]

[0146] - A map comprising at each point a positional coding (as described for example at the Internet address https: / / skosmos.loterre.fr / P66 / fr / page / -K0D65X2X-X).

[0147] - A map representing an image distinct from the image currently being processed, capable of providing information on the image to be coded, for example the previously processed image if the current image is part of a series of images to be coded such as a video, a set of medical images, a multi-view representation, etc.

[0164]

[0148] - A map representing a map of characteristics of an image distinct from the image being processed capable of providing information on the image to be coded, for example a previously processed map if the current image is part of a series of images to be coded such as a video, a set of medical images, a multi-view representation, etc.

[0165]

[0149] - A card containing the value of an already decoded sample of the same card, for example the previous sample in the decoding order.

[0166]

[0150] In this embodiment, the vector Z n is a 4-tuple (Z1...Z4) consisting of the values ​​extracted from the FMSi maps located at the coordinates (x n , y n ) of the current pixel P n . The vector Z n consisting of the extracted (quantized) values ​​of the FMSi maps is processed by the MLP synthesis neural network to generate a second vector as output, according to the example the output vector is the triplet (R, G, B) of the pixel P' n encoded and then decoded. The triplet is inserted into the decoded image I (P'n) at positions (x n , y n ) of the color components (R', G', B').

[0167]

[0151] In another embodiment, not shown, the vector Z nis extracted directly from the FMi layers, at positions recalculated according to the size of the maps, then the extracted values ​​are optionally processed and quantified after extraction.

[0168]

[0152] According to the variant shown in dotted lines, the vector Z n is a 5-tuple (Z0...Z4), the value zo being extracted from the additional map FMEo.

[0169]

[0153] Figure 6 is a flowchart representing an example of a decoding method that can be implemented by the decoding device of Figure 2.

[0170]

[0154] During a step E30, the streams B1 and B2 are extracted from the encoded stream. They respectively contain the coded representations of the maps of the first group FMci and the parameters Wck, optionally the parameters Ocb.

[0171]

[0155] During a step E31, the M FMdi maps are generated by decoding the FMci values. For this decoding, a technique for predicting a feature map value by its neighborhood is used, as will be described for example with reference to FIG. 10. T1

[0172] . In one embodiment, the FMdi maps are decoded in order (FMdi, FMd2,...

[0173] FMc), and the values ​​of each card in a predefined order, for example lexicographic.

[0174]

[0156] According to embodiments as described for the encoder:

[0175]

[0157] - The FMdi cards have the same resolution as the I signal (Pd n ) to be reconstituted, that is to say that they contain N values.

[0176]

[0158] - FMdi maps are at a resolution lower than or equal to that of the I signal (Pd n ) to be reconstituted.

[0177]

[0159] - Several FMdi cards are of the same resolution, lower than the signal resolution.

[0178]

[0160] During a step E32, according to one embodiment, one or more FME'i maps, forming an additional group of L additional feature maps, are generated and complete the first group. They are not decoded but generated by the decoder in a manner identical to the generation of the encoder. They typically comprise data which can assist the MLP' network in the signal reconstruction task. The non-limiting list of possible additional feature maps described in support of FIG. 6 for the encoder applies here.

[0179]

[0161] During a step E33, according to one embodiment, the M maps of the first group FMdi are transformed by the module SE' to generate maps of the second group FMS'i at the resolution of the input images. This step is similar to step E24 which was described for the encoder with reference to FIG. 5, and the embodiments apply. In particular:

[0180]

[0162] According to one embodiment, M FMS'i maps are generated.

[0181]

[0163] According to one embodiment, each FMdi card is transformed into an FMSi card.

[0182]

[0164] According to one embodiment, at least one FMdi map has a lower resolution than the images of the image to be coded and the transformation operation includes oversampling so that the transformed FMS'i map includes the same number of samples as the input image. Oversampling consists of adding values ​​to the FMS'i maps to achieve the resolution of the input image. It can be simple (by nearest neighbor replication) or include interpolation (linear, polynomial, by filtering, etc.)

[0183]

[0165] The transformation may optionally include an inverse quantization of the extracted values, if necessary. However, the inverse quantization is not mandatory.

[0166] During a step E34, values ​​are extracted by the XTR' module from the transformed FMdi or possibly FMS'i maps, and optionally additional FME'i. This extraction is carried out according to the coordinates (x n, y n ) of a sample P n of the input signal. It can also be carried out according to the resolution of the card considered. This step is similar to step E25 which was described for the encoder in support of Figure 5, and the embodiments apply. In particular:

[0184]

[0167] According to one embodiment, the vector Zd n characteristic results directly from this extraction.

[0185]

[0168] In one embodiment, Zd n is a J-uplet (zi, Z2, . . . , zj), consisting of the values ​​of the maps FMdi or FMS'i (and optionally FME'i) located at the coordinates (x n , y n ) of a current pixel Pd n , as will be illustrated in support of Figure 8.

[0186]

[0169] The samples to be decoded are for example processed in sequential order, from n=1 to n=N.

[0187]

[0170] According to one embodiment, during a step E35, a vector Zdn is constructed by the TT' module from the values ​​extracted from the FMdi maps of the first group or from the FMS'i maps of the second group and possibly FME'i of the additional group, for each sample Pd n of coordinates (x n , y n ) to decode, based on the coordinates (x n , y n ). This step is similar to step E26 which was described for the encoder in support of Figure 5 and the described embodiments apply. The extraction may include inverse quantization of the extracted values ​​or of the vector Zd n constituted, if necessary.

[0188]

[0171] During a step E36, the Wdk parameters of the MLP' synthesis neural network and possibly the Odb parameters of the prediction neural network are generated by decoding the Wck and Ocb values ​​of the stream B2. Any known decoding technique corresponding to the coding technique used by the encoder can be used for this purpose. The MLP' synthesis neural network is similar to the MLP synthesis network, i.e. it has the same structure and includes the same parameters, except for the coding, which can be performed with or without losses. Similarly, the ARM' prediction neural network, if it is used to decode the feature maps, is similar to the ARM prediction network, i.e. it has the same structure and includes the same parameters, except for the coding, which can be performed with or without losses.

[0189]

[0172] According to one embodiment, the stream B2 is decoded before the stream B1, in order to be able to have the synthesis neural network MLP' and possibly the prediction neural network ARM' before starting to decode the samples.

[0173] During a step E37, the vector Zd n is processed by the MLP' synthesis neural network to generate as output a second vector representative of the sample Pd n to decode, according to one embodiment a triplet which is injected into the decoded image I (Pd n ) at positions (x n , y n ) of the color components (Rd, Gd, Bd). This step is similar to step E27 which was described for the encoder in support of Figure 5.

[0190]

[0174] When all the samples of the signal have been processed, the decoded signal corresponding for example to image I (Pd n ) is available.

[0191]

[0175] Figure 8 illustrates a decoding method used in one embodiment of the invention.

[0192]

[0176] In this embodiment, there are 4 FMdi decoded cards. In a preferred embodiment, there are 7.

[0193]

[0177] In this embodiment, the first FMdi map has the same resolution as the image I, and therefore comprises WxH variables, where W represents the width of the image in pixels, and H its height. The second FMd2 map has half the resolution (in each dimension) of the FMdi map. Each additional map has half the resolution of the previous map. This structure makes it possible to reduce the number of variables in the feature maps, which facilitates decoding while minimizing the coding cost.

[0194]

[0178] The FMd2 map is oversampled by a factor of 2 in each dimension, according to any oversampling method within the reach of those skilled in the art. The FMdi map is oversampled by a factor of 4 in each dimension, and the FMcL map by a factor of 8 in each dimension.

[0195]

[0179] The FMS'i cards have the same resolution as the image to be decoded, and therefore have WxH values, where W represents the width of the image in pixels, and H its height.

[0196]

[0180] In this embodiment, the vector Zd n is a 4-tuple (Z1...Z4) consisting of the values ​​of the FMS'i maps located at the coordinates (x n , y n ) of the current pixel Pd n . The vector Zd n is optionally dequantized then processed by the MLP' synthesis neural network to generate as output the triplet (R, G, B) representative of the sample Pd n to be decoded. The triplet (R, G, B) is inserted into the decoded image I (Pdn ) at coordinates (x n , y n ) in the color components (Rd, Gd, Bd).

[0181] According to a variant presented in dotted lines, there are 5 cards: an additional card FME'o has been introduced. In this embodiment, the vector Zd n is a 5-tuple.

[0197]

[0182] Figure 9 is a flowchart representing a method of encoding feature maps that can be implemented by the encoding device of Figure 1 and by the encoding method of Figure 5.

[0198]

[0183] These steps constitute sub-steps of step E23 described previously with reference to Figure 5. Their purpose is to code a current value V n of an FMi feature map of the first group being processed using neighborhood values.

[0199]

[0184] During a sub-step E231, a neighborhood vector (C n) is established, including values ​​close to the value V n . As will be illustrated later in support of Figures 11 and 12, these neighboring values ​​can be located in the same map or / and in a different map of the plurality M of FMi maps. This neighborhood vector is made up of a number C of values, or data, corresponding to neighborhood values ​​(for example, C=10). These values ​​must be known to the encoder and the decoder, they must therefore be located in a causal neighborhood of the value V n .

[0200]

[0185] According to a first embodiment, these values ​​are used to determine the context of an entropy coder to code the current value during a step E234. This coder can be a CAB AC (Context-adaptive binary arithmetic coding) type coder. This type of coder is well known to those skilled in the art. It is notably used in the H.265 / HEVC video compression standard. It is an arithmetic coder whose compression is lossless. It decomposes all non-binary symbols into binary symbols. Then, for each bit, the coder selects the most suitable probability model and uses a context to optimize the estimation of the probability. This context can be defined by information from the neighboring elements. Arithmetic coding is then applied to compress the resulting data. As is known to those skilled in the art, there are several ways of using the neighborhood vector to produce context information.For example, one can count the number of non-zero neighboring values, and associate a context with each number. Alternatively, one can perform comparisons between several neighboring values, and associate a given context with an ordering configuration between the neighboring values, for example by ranking the neighboring values ​​in ascending order, and associating a context with each possible order.

[0201]

[0186] In a second embodiment, the neighborhood is used to predict, during a step E232, the current value from an autoregressive model. It is recalled that an autoregressive model predicts a sample of a series by its past values. In this mode, the past values ​​are constituted by the context, and the difference between the predicted variable and the actual value is quantified then entropically coded during step E234.

[0202]

[0187] In a third embodiment, as illustrated in support of Figure 12, an ARM prediction neural network is used during step E233 to predict the statistical characteristics of the variable to be coded. The neighborhood vector is given as input to the ARM network, to provide as output a prediction of the current value. According to one embodiment, the ARM network behaves like a function f\ as described in support of Figure 4, which provides a set of statistical parameters (mean, variance, median, etc.) used to entropically code the current value. The role of this ARM module is to best predict the current value, for the set of values ​​V nto encode FMi maps, in order to reduce the throughput required for encoding feature maps. In another embodiment, the ARM prediction neural network is used to produce the expected probability (pr) of the possible value of the current sample. Entropy coding is adapted to this probability (as is known for Huffman or arithmetic entropy coding).

[0203]

[0188] In a fourth embodiment, each feature map is divided into blocks of predetermined size, and the coding of each block comprises a transform (e.g., a DCT, Discrete, Cosine Transform, a Haar transform, etc.), and the transformed values ​​are encoded by an entropy coding.

[0204]

[0189] In a fifth embodiment, each feature map is divided into blocks of predetermined size, and each block is represented by a product code of the Lattice Vector Quantization type.

[0205]

[0190] At the end of the process, the current coded value Vc n of the FMi card being processed is encoded.

[0191] Figure 10 is a flowchart representing a method of decoding feature maps which can be implemented by the decoding device of Figure 2 and by the decoding method of Figure 7

[0206]

[0192] These steps constitute sub-steps of step E31 described previously with reference to FIG. 7. Their purpose is to decode a current value Vd n of an FMdi feature map of the first group being processed using neighborhood values.

[0207]

[0193] During a sub-step E311, a neighborhood vector (Cd n ) is established, including values ​​close to the value Vd n. This step is similar to step E231 previously described and the same embodiments apply. This neighborhood vector consists of a number C of values, or data, corresponding to neighborhood values ​​(for example, C=10) located in the same map and / or in a different map of the plurality M of FMdi maps. These values ​​are located in a causal neighborhood of the value Vd n are known to the decoder.

[0208]

[0194] According to a first embodiment, these values ​​are used to determine the context of an entropy decoder to decode the current value during a step E314. This decoding is similar to that which was used in the encoder, for example CABAC. The use of the neighborhood to produce context information is similar to that which was chosen in the encoder. For example, the number of neighboring values ​​other than zero can be counted, and a context can be associated with each number. Alternatively, comparisons can be made between several neighboring values, and a given context can be associated with an order configuration between the neighboring values, for example by classifying the neighboring values ​​in ascending order, and by associating the context with each possible order.

[0209]

[0195] In a second embodiment, the neighborhood is used to predict during a step E312 the current value from an autoregressive model. In this mode, the past values ​​are constituted by the context, and the difference between the predicted variable and the actual value is quantified then entropically coded during step E314.

[0210]

[0196] In a third embodiment, as illustrated in support of Figure 12, a prediction neural network ARM' is used in step E313 to predict the statistical characteristics of the variable to be decoded. The neighborhood vector is applied as input to the network ARM', to provide as output a prediction of the current value. According to one embodiment, the network ARM behaves like a function f\ as described in support of Figure 4, defined by a set of statistical parameters (mean, variance, median, etc.) used to entropically code the current value. According to another embodiment, the prediction neural network ARM' is used to produce the expected probability (pr) of the possible value of the current sample. Entropic decoding is adapted to this probability (as is known for Huffman or arithmetic entropic coding).The ARM network is identical to the ARM network if the coding was done losslessly.

[0211]

[0197] In a fourth embodiment, each feature map is divided into blocks of predetermined size, and the decoding of each block comprises the entropy decoding of values ​​which then undergo an inverse transform (e.g. an inverse DCT, Discrete, Cosine Transform, an inverse Haar transform, etc.).

[0212]

[0198] In a fifth embodiment, each feature map is divided into blocks of predetermined size, and each block is decoded by a Lattice Vector Quantization product code to produce the decoded block.

[0213]

[0199] At the end of the process, the current decoded value Vd n of the FMdi card being processed is decoded.

[0214]

[0200] Figure 11 illustrates a method of encoding or decoding feature maps according to one of the embodiments.

[0215]

[0201] In this illustration, the coding (resp. decoding) of the current value V n (resp. Vd n ) located in coordinates (x n , y n ) in the current feature map FMi (i.e. the one being encoded or simulated) (resp. FMdi - i.e. the one being decoded) uses the contextual information of its own map and the previous map FM2 (resp. FMd2). The values ​​located at the coordinates (x n , y n -l), (x n , y n -2), (x n -l, y n -l), (x n -l, y n ) ,(x n -l, yn+l) ,(x n -2, y n ) in the FMi layer (resp. FMdi), and the variables located at the coordinates (x n -l, y n -l), (xn -l, y n ), ( x nl, y n +l), (x n , y n -L), ( x n, y n ), ( x n, y n +l), (x n +l, y n - 1), (xn+l, y n ), (xn+l, y n +l) in the FM2 (resp. FMd2) layer are used to determine the neighborhood for encoding (resp. decoding) the current value. These values, which are all available to the encoder and decoder, constitute the neighborhood vector C n (resp. Cd n ) of the variable V n (resp. Vd n ) which can be used in one of the embodiments described previously in support of Figure 9 (resp. Figure 10).

[0202] Figure 12 illustrates another method of encoding or decoding feature maps according to one of the embodiments

[0216]

[0203] In this illustration, the coding (rep. decoding) of the current value V n (resp. Vdn ) located in coordinates (x n , y n ) in the current feature map FMi (resp. FMdi) uses the contextual information from its own map. The values ​​shown in gray are used to determine the neighborhood for encoding (resp. decoding) the current value. These values, which are all available to the encoder and decoder, constitute the neighborhood vector C n (resp. Cd n ) of the value V n (resp. Vd n ) which can be used in one of the embodiments described previously in support of Figure 9 (resp. Figure 10).

[0204] In the illustrated embodiment, the neighborhood vector is extracted by a module

[0217] CTX (resp. CTX') of the FMC (resp. FMD) coding module, then it is applied to the input of the ARM (resp. ARM') prediction neural network, used to predict the statistical characteristics (p,o) or the probability (pr) of the value to be encoded (resp. decoded) by the CE (resp. DE) entropy coder.

Claims

Claims

1. Method for coding a signal (I (P n )) comprising a plurality of samples (P n ) to be coded comprising the following steps: - a construction stage, comprising the sub-stages of: - construction (E21, E22) of a first group of characteristic maps (FMi) representative of the signal (FMi); - for at least one sample, called the current sample (P n ), of the signal to be coded, associated with a position (x n , y n ) in the signal to be coded: - construction (E25) of a characteristic vector (Z n ) from said feature maps (FMi) of said first group, as a function of said position (x n , y n ) of said current sample (P n ) ; - processing (E27) of said characteristic vector (Z n) by an artificial neural network, called a synthetic neural network, (MLP) defined by a set of parameters (Wk), to provide a vector (P' n ) representative of a decoded value of the current sample; - updating (E22, E27) at least one value of one of said characteristic maps of said first group and / or at least one parameter of said network, as a function of a coding performance measurement, - a coding step (E23, E27, EF) of said first group of characteristic maps (FMi) comprising, for at least one value, called current value (V n ), of one of said characteristic maps, an entropic coding of said value as a function of at least one value of its neighborhood; - a step of coding said set of parameters (Wk) of said synthetic neural network.

2. Coding method according to claim 1, characterized in that the step of coding said current value (V n ) of one of said feature maps comprises the following sub-steps: - construction (E231) of a neighborhood vector (C n ) from said characteristic maps (FMi) of said first group; - processing (E233) of said neighborhood vector (C n ) by an artificial neural network, called a prediction neural network (ARM), defined by a set of parameters (Ob), to provide a prediction of said current value (V n ); - updating (E22, E23, E28) of at least one parameter of said prediction network, depending on the coding performance measure, - a step of coding said set of parameters (Ob) of said prediction network.

3. Method for coding a sequence of images according to claim 1 or 2, characterized in that it comprises a step of transforming (E24) said first group of characteristic maps (FMi) to obtain a second group of characteristic maps (FMSi) at the resolution of the images of the input sequence; and in that said characteristic vector (Z n ) is constructed from said transformed feature maps (FMSi) of the second group obtained from said feature maps (FMi) of said first group.

4. Coding method according to claim 3, characterized in that at least one of said characteristic maps of the first group (FMi) has a lower resolution than that of the signal to be coded and in that the transformation operation comprises oversampling.

5. Coding method according to claim 1, characterized in that the construction of said characteristic vector (Z n ) includes the following sub-steps: - extracting a plurality of values ​​from said feature maps of said first group (FMi) as a function of said position (x n , y n ) of said current sample (P n ) ; - processing of said extracted values ​​to obtain the characteristic vector.

6. Coding method according to one of the preceding claims, characterized in that it comprises a step of constructing (E21, E22) an additional group of characteristic maps (FMEi), and in that the characteristic vector is further constructed from said characteristic maps of the additional group.

7. A method of decoding a signal comprising a plurality of samples (Pd n) to be decoded comprising the following steps: - decoding (E31) of a first group of characteristic maps (FMdi) representative of the signal comprising, for at least one value, called current value (V n ), of one of said characteristic maps, an entropic decoding of said value as a function of at least one value of its neighborhood; - decoding (E35) of a set of parameters (Wdk) representative of a neural network (MLP'), called a synthetic neural network; - for at least one sample, called the current sample (Pd n ), of the signal to be decoded, associated with a position (x n , y n ) in the signal to be decoded: - construction (E34) of a characteristic vector (Zd n ) from the characteristic maps of said first group (FMdi), as a function of said position (x n , y n ) of said current sample, and: - processing (E37) of said vector (Zd n ) by the synthesis neural network (MLP') defined by the decoded parameters (Wdk) to provide a vector representative of a decoded value of the current sample (Pd n ).

8. Decoding method according to claim 7, characterized in that the step of decoding said current value (V n ), of one of said feature maps comprises the following sub-steps; - decoding (E36) of a set of parameters (Odk) representative of a neural network, called a prediction neural network (ARM'); - construction (E311) of a neighborhood vector (C n ) from said characteristic maps (FM0 of said first group; - processing (E313) of said neighborhood vector (C n ) by the prediction neural network (ARM'), to provide a prediction of said current value (V n ).

9. Decoding method according to claim 7 or 8, characterized in that it comprises a step of transforming (E33) said first group of decoded characteristic maps (FMdi) to obtain a second group of characteristic maps (FMS'i) at the resolution of the input signal; and in that said characteristic vector (Zd n ) is constructed from said transformed feature maps (FMS'i) of the second group obtained from said decoded feature maps (FMdi) of said first group.

10. Decoding method according to claim 9, characterized in that at least one of said characteristic maps of the first group (FMdi) has a lower resolution than that of the signal to be decoded and in that the transformation operation comprises oversampling.

11. Decoding method according to one of claims 7 to 10, characterized in that the construction of said vector (Zd n ) characteristic comprises a sub-step of extracting (E34) a value from said at least one characteristic map (FMdi, FME'i) at an identical position (x n , y n ) to that of the current sample (Pd n ) in the signal to be decoded.

12. Decoding method according to claim 7, characterized in that the construction of said characteristic vector (Zd n ) includes the following sub-steps: - extracting a plurality of values ​​from said feature maps of said first group (FMdi) as a function of said position (x n , y n ) of said current sample (P n ) ; - processing (E35, TT') of said extracted values ​​to obtain the characteristic vector.

13. Decoding method according to one of claims 7 to 12, characterized in that it comprises a step of constructing (E32) an additional group of characteristic maps (FME'i) and in that the characteristic vector is further constructed from said characteristic maps of the additional group.

14. Device for coding a signal (I (P n )) comprising a plurality of samples (P n ) to be coded, characterized in that said device is configured to implement: - construction (GEN, MAJ) of a first group of characteristic maps (FMi) representative of the signal (FMi); - for at least one sample, called the current sample (P n ), of the signal to be coded, associated with a position (x n , y n ) in the signal to be coded: - construction (XTR) of a characteristic vector (Z n) from said feature maps (FMi) of said first group, as a function of said position (x n , y n ) of said current sample (P n ; - processing (MLP) of said characteristic vector (Z n ) by an artificial neural network, called a synthetic neural network, (MLP) defined by a set of parameters (Wk), to provide a vector (S n ) representative of a decoded value (P' n ) of the current sample; - updating (MAJ, NNC) at least one value of one of said characteristic maps of said first group and / or at least one parameter of said synthesis network, as a function of a coding performance measurement, - a coding step (FMC) of said first group of characteristic maps (FMi) comprising, for at least one value, called current value (V n), of one of said characteristic maps, an entropic coding of said value as a function of at least one value of its neighborhood; - a coding step (NNC) of said set of parameters (Wk) of said synthetic neural network.

15. Device for decoding a signal comprising a plurality of samples (Pd n ) to be decoded, characterized in that said device is configured to implement: - decoding (FMD) of a first group of characteristic maps (FMdi) representative of the signal comprising, for at least one value, called current value (V n ), of one of said characteristic maps, an entropic decoding of said value as a function of at least one value of its neighborhood; - decoding (NND) of a set of parameters (Wdk) representative of a neural network (MLP'), called a synthetic neural network; - for at least one sample, called the current sample (Pd n ), of the signal to be decoded, associated with a position (x n , y n ) in the signal to be decoded: - construction (XTR') of a characteristic vector (Zd n ) from the characteristic maps of said first group (FMdi), as a function of said position (x n , y n ) of said current sample; - processing (MLP') of said characteristic vector (Zd n ) by the synthesis neural network (MLP') defined by the decoded parameters (Wdk) to provide a vector representative of a decoded value of the current sample (Pd n ).

16. A computer program comprising instructions for carrying out the steps of an encoding or decoding method according to claim 1 or 7 when said program is executed by a computer.

Images