Method and device for encoding and decoding image sequences.

Abstract

Method and device for encoding and decoding images. The invention relates to a method and device for encoding and decoding a sequence of images. The decoding method comprises the following steps for a current image: - obtaining at least one feature map from a first group of decoded feature maps representative of at least one image distinct from the current image.- Decoding a second group of feature maps representative of the current image, comprising, for at least one value, an entropy decoding based on a neighborhood including at least one decoded value extracted from at least one reference map; - Decoding by a synthetic neural network; - For at least one sample of the current image: - Construction of a feature vector from the feature maps of the second group, based on the position of said current sample, and: - Processing of the vector by said synthetic neural network to provide a decoded value of the sample. Figure for the abstract: Fig. 2.

Description

Title of the invention: Method and device for coding and decoding image sequences. Prior art

[0001] The invention relates to the general field of coding and decoding of digital image sequences. It relates more particularly to the compression of digital videos.

[0002] Digital videos are generally subject to source coding aimed at compressing them in order to limit the resources required for their transmission and / or storage. There are many coding standards, such as the ITU / MPEG standards (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) as well as their extensions (MVC, SVC, 3D-HEVC, etc.).

[0003] As video formats continue to evolve to compress more and more and adapt to the variety of expected formats and communication networks, the possibilities for prediction are becoming ever greater and the classic coding and decoding algorithms very complex.

[0004] In addition to these classic approaches proposed by compression standards (MPEG, ITU), approaches based on artificial intelligence, and in particular neural ones, tend to develop.

[0005] Some of these neural approaches can be seen as a simple extension of the notion of competition of the aforementioned compression techniques, such as the competition of prediction and transformation mode in video coding.

[0006] Other approaches use the concept of "autoencoder". Autoencoders are learning algorithms based on artificial neural networks, which allow the construction of a new representation of a data set. The architecture of an autoencoder consists of two parts: the encoder and the decoder. The encoder is made up of a set of layers of neurons, which process the data in order to construct new representations called "encoded", also called "latent representations". In turn, the layers of neurons in the decoder receive these representations and filter them in order to try to reconstruct the initial data. The differences between the reconstructed data and the initial data make it possible to measure the error made by the autoencoder. Training consists of modifying the parameters of the autoencoder in order to reduce the reconstruction error measured on the different samples of the data set.The performance of such autoencoder-based systems comes at the cost of a considerable increase in memory footprint and complexity compared to . conventional approaches as proposed by compression standards. They can have millions of parameters and may require on the order of a million MACs (multiply-accumulate) to decode a single pixel. This makes such decoders much more complex than conventional decoders, which could hamper the adoption of learning-based compression.

[0007] More recently, a simple image encoding technique based on a neural network has been described in the article "Compression with Implicit Neural representations" by Emilien Dupont et al. (arXiv:2103.03123). The proposed encoding technique consists of fitting a neural network to an image, quantizing the network weights and transmitting them. At decoding time, the neural network is evaluated at each pixel position to reconstruct the image. However, such a technique remains inefficient in terms of compression and requires coding the video frames independently.

[0008] There is therefore a need for a solution allowing a sequence of images to be coded / compressed in a simple and efficient manner. Statement of the invention

[0009] The invention relates to a method for coding a sequence of images comprising a plurality of samples to be coded, comprising the following steps: - a construction step, comprising the following sub-steps for at least one current image of the sequence: - construction of a first group of characteristic maps representative of at least one image distinct from the current image, called the reference image; - construction of a second group of characteristic maps representative of the current image; - for at least one sample, called the current sample, of the current image, associated with a position in the current image: - construction of a characteristic vector from said characteristic maps of said second group, as a function of said position of said current sample; - processing of said characteristic vector by an artificial neural network, called a synthetic neural network, defined by a set of parameters, to provide a vector representative of a decoded value of the current sample; - updating at least one value of one of said feature maps of said second group and / or at least one parameter of said network, based on a coding performance measurement, - a step of coding said second group of characteristic maps comprising, for at least one value, called current value, of one of said characteristic maps, called current map, an entropic coding of said value as a function of a neighborhood comprising at least one value extracted from at least one map, called reference map, of the first group of characteristic maps, - a step of coding said set of parameters of said synthetic neural network.

[0010] The invention also relates to a method of decoding from a binary stream of a sequence of images comprising a plurality of samples to be decoded, comprising the following steps for at least one current image of the sequence: - obtaining at least one feature map, called a reference map, from a first group of decoded feature maps representative of at least one image distinct from the current image, called a reference image. - decoding of a second group of characteristic maps representative of the current image comprising, for at least one value, called current value, of one of said characteristic maps, called current map, an entropic decoding of said value as a function of a neighborhood comprising at least one decoded value extracted from said at least one reference map; - decoding of a set of parameters representative of a synthetic neural network; - for at least one sample, called the current sample, of the current image, associated with a position in the current image: - construction of a characteristic vector from the characteristic maps of said second group, as a function of said position of said current sample, and: - processing said vector by said synthesis neural network to provide a vector representative of a decoded value of the current sample.

[0011] For the purposes of the invention, encoding, or “coding”, means the operation which consists of representing a set of samples, or pixels, in a compact form carried for example by a digital binary train. Decoding means the operation which consists of processing a digital binary train to restore decoded samples.

[0012] By "sequence of images" is meant a plurality of ordered two-dimensional images, for example temporally in the case of a video. According to another example, the images may be views of the same scene represented in multi-views. According to another example, the images may be a plurality of temporal and multi-view images (immersive video).

[0013] By "sample" is meant a value taken from an image of the sequence. Sampling a signal produces a series of discrete values ​​called samples. In the case of an image signal, the sample is called a pixel, which can be, for example, a grayscale pixel represented by a single component, or a color pixel traditionally represented by a triplet of values, for example (R,G,B) or (Y,U,V). The position of the sample is identified by its coordinates in abscissa (x) and ordinate (y) in the image.

[0014] By "feature map" is meant a part of an abstract representation of an image, comprising a plurality of variable scalar data, which are also called characteristic values, for example real numbers, or integers, signed or unsigned. The abstract representation of an image can therefore comprise several feature maps. The abstract representation of an image can also correspond to a single feature map comprising a set of values ​​associated with a position.

[0015] By "feature vector constructed from the feature maps as a function of a position" is meant a vector consisting of one or more elements, or data, preferably discrete, the data being constructed from the feature maps at a position determined by that of the samples being processed in the images. This feature vector is the one that is applied to the input of the synthesis neural network. In the case of an image, such a vector can be constituted for example from a plurality of values ​​taken from each of the feature maps at the same abscissa and ordinate coordinates as the samples to be coded (resp. to be decoded).

[0016] By "synthetic neural network" is meant a neural network such as a convolutional neural network, a multi-layer perceptron, an LSTM (for "Long Short Term Memory" in English), etc. The neural network is defined for example by a plurality of layers of artificial neurons and by a set of activation, weighting and addition functions (for example, a layer can calculate y = f(Ax+b), where y and b are vectors of dimension N, x a vector of dimension M, A is a matrix of dimension MxN, and f is the activation function).

[0017] By "parameter of the neural network" is meant one of the values ​​which characterizes the neural network, for example a weight associated with one of the neurons (filter coefficient, weighting, bias, value affecting the operation of the non-linearity, etc.)

[0018] By "processing by a synthetic neural network" is meant the application of a function expressed by a neural network to the input characteristic vector to produce an output vector representative of the sample to be coded (resp. decoded). This output vector may comprise one or more data representative of the sample.

[0019] By “performance measurement” is meant a measurement between at least one value of a sample to be coded and a decoded value of said sample. The measurement can evaluate for example a distortion, or a perceptual error. It can be carried out on a sample or a plurality of samples (for example, the current samples, or the current images, etc.). The measurement can also include a measurement of the flow rate, in particular associated with the coding of the neural network and / or the coding of the feature maps of the second group. The measurement can be a joint measurement between the flow rate and the distortion through their weighting. As is well known from the state of the art, the value of this measurement is generally minimized until a target value is reached.

[0020] By "construction step" is meant a step which aims to construct the representative parameters of the image, before their actual coding. The construction sub-steps can be repeated as many times as necessary to obtain an acceptable performance measurement.

[0021] Generally speaking, it is considered that the steps of a coding or decoding method should not be interpreted as being linked to a notion of temporal succession. In other words, the steps can be carried out in a different order than that indicated, or even in parallel.

[0022] The coding method according to the invention carries out a construction of the coding parameters, from a sequence of input images, by training a neural network on characteristic vectors associated with a position of a sample to be coded in the input image. These characteristic vectors are constructed from characteristic maps which can be at the resolution of the input images, or at a lower resolution. During the training, or construction, the parameters of the neural network and the values ​​of the characteristic maps are updated according to a performance measurement, for example of the rate-distortion type. When the training is finished, that is to say that the performance measurement obtained is satisfactory, the actual coding of the parameters of the neural network and the values ​​of the characteristic maps can be carried out and stored or transmitted to the decoder.

[0023] Advantageously, the training process makes it possible to refine the parameters of the synthetic neural network and / or the values ​​of the feature maps until an adequate representation in terms of performance is obtained, for example a desired balance between the generated bit rate and the distortion undergone by the input image being encoded. The training of the values ​​of the feature maps and the parameters of the synthetic neural network can be joint. Advantageously, the encoding method according to the invention makes it possible to efficiently compress the signal.

[0024] Advantageously, the decoding method is simple since it suffices to decode the synthesis neural network and the characteristic maps of the current image as a function of other decoded characteristic maps of other images in the sequence, for example previous images, to reconstitute a decoded version of the current image.

[0025] Advantageously, the coding of the characteristic maps is made particularly efficient by taking into account a coding neighborhood in characteristic maps of other images in the sequence, which makes it possible to exploit the redundancies. present in the cards associated with the different images.

[0026] Such a synthetic neural network can advantageously be of very simple structure with few parameters.

[0027] Furthermore, the decoding can be carried out sample by sample, in a progressive manner.

[0028] According to embodiments of the coding or decoding method:

[0029] - The coding method comprises the following sub-steps of coding said current value: - construction of a neighborhood vector comprising said at least one neighborhood value extracted from said at least one reference map, and; - processing of said neighborhood vector by an artificial neural network, called a prediction neural network, defined by a set of parameters, to provide a prediction of said current value; - updating at least one parameter of said prediction network, based on the coding performance measurement, - coding of said set of parameters of the prediction network.

[0030] - The decoding method comprises the following sub-steps of decoding said current value of one of said characteristic maps; - decoding of a set of parameters representative of a neural network, called a prediction neural network; - construction of a neighborhood vector comprising said at least one decoded value extracted from said at least one reference map, and; - processing of said vector by the prediction neural network, to provide a prediction of said current value;

[0031] Advantageously, according to such a coding or decoding mode, the characteristic maps are efficiently compressed by a prediction neural network capable of predicting a value of the map as a function of its temporal neighborhood. By "neighborhood vector" is meant a vector consisting of one or more elements, or data, constructed from the characteristic maps of other images of the sequence, or reference maps, preferably extracted at a position close to that of the current sample, which is also that of the current value in the current characteristic map. The neighboring position may indicate a value in any map of the set of characteristic maps associated with another image of the sequence (for example, the neighboring value at the top left of that currently being processed in a characteristic map of the previous image).This neighborhood vector is the one that is applied to the input of the prediction neural network. By "prediction" we mean at least one piece of data allowing us to estimate a current value of the feature map, for example a probability, a sta value. tistics, etc. The prediction neural network, trained on the image maps of the sequence, is able to represent them efficiently. It is also inexpensive to code.

[0032] According to variants:

[0033] - at least one of said characteristic maps of the second group is of re solution lower than that of the image to be decoded and the method includes oversampling. Advantageously, according to this mode, the compression of the characteristic maps is more efficient since at least one of the characteristic maps of the second group, to be coded (resp. decoded), includes fewer values ​​than if it were at the resolution of the signal. For example, in the case of a digital image, one of the characteristic maps may be at 1 / 2 resolution, that is to say that it includes half as many values ​​on the abscissa and ordinate as the input image includes samples, i.e. in total 4 times fewer values ​​than a characteristic map at the resolution of the image. On the other hand, the characteristic map used for the synthesis may have the same resolution as that of the image. The method therefore includes in this case at least one oversampling operation to obtain the same number of values ​​in this map as the input image (resp.to decode) has samples.

[0034] - At least one of said characteristic maps of the second group is of re solution identical to that of the signal to be coded (resp. decoded). Advantageously, according to this mode, at least one of the characteristic maps, at the same resolution as the image to be coded (resp. decoded) allows significant fidelity and respect for the details of the initial resolution of the image signal.

[0035] - the construction of the characteristic vector includes an extraction sub-step of a value of a feature map at a position identical to that of the current sample in the image to be encoded or decoded. Advantageously, it is possible to extract a value of a feature map at the same position as the sample in the image, to constitute an element of the feature vector. This process is simple to implement. For example, if we have J feature maps as input, with the same resolution as the image, a simple extraction of the values ​​of the maps at the coordinates of the current sample (at the same abscissa and the same coordinate in the feature map) makes it possible to directly construct the feature vector of J elements.

[0036] - The construction of said characteristic vector comprises the following sub-steps: - extracting a plurality of values ​​from said feature maps as a function of said position of said current sample; - processing of said extracted values ​​to obtain the characteristic vector.

[0037] Advantageously according to this mode, the characteristic vector is extracted from the characteristic maps, which can be any, and in particular at a resolution in lower than that of the image to be encoded (resp. decoded), before undergoing processing. Such processing may for example correspond to a quantization of the extracted data, or to scaling, or to filtering, etc. During encoding, quantization is essential for the proper functioning of the system if the feature maps contain, for example, floating point or real values. It is necessary to quantize them before encoding them and / or providing them as input to the synthesis and / or prediction neural network. During decoding, however, inverse quantization is not necessary, depending on the embodiments.

[0038] - said first and second groups contain the same number (M) of cards of features; advantageously, according to this mode, the different images of the sequence are associated with the same number of feature maps, which can also be of the same structures. For example, the images of the sequence are associated with five feature maps presenting a hierarchy of decreasing resolutions. This facilitates the generation and extraction of the values ​​of the feature maps to produce the neighborhood vector.

[0039] - said at least one reference map is at the same resolution as the map Advantageously, in this mode, the construction of the neighborhood vector is facilitated since the reference map is at the same resolution as the current map, it is very simple to extract a value at the same position or at a position close to that of the value of the current map.

[0040] - said at least one reference image is indicated in the bitstream. Avanta Conveniently, according to this mode, a simple indicator coded in the stream allows the decoder to directly access the reference image(s) to be used for decoding the current map of the current image, for example via an indicator of the reference image number.

[0041] - said at least one reference image is predetermined. Advantage Conveniently, according to this mode, the reference image(s) to be used for decoding the current map of the current image are predetermined. For example, the reference image is the previous image in the sequence. According to another example, the first image in the sequence is the reference image. According to another example, the first image in the sequence and the next image in the current image are two reference images.

[0042] - said at least one reference card is indicated in the bit stream. Avanta Conveniently, according to this mode, a simple indicator coded in the stream allows the decoder to directly access the reference map(s) to be used for decoding the current map, for example via an indicator of the number of the reference map in the group of feature maps associated with a reference image.

[0043] - said at least one reference card is predetermined. Advantageously- Conveniently, according to this mode, the feature map(s) to be used for decoding the current map are predetermined. For example, the reference feature map is the one that has the same order number as the current map in the feature map groups to which they respectively belong. According to another example, all feature maps of the target reference image serve as reference maps for the current map.

[0044] - said at least one decoded value extracted from said at least one map of reference is at the same position in said at least one reference map as the current value in the current feature map. Advantageously, according to this mode, once the reference map is known, the value which is at the same position as that being encoded (resp. decoded) in the current map is used to compose the neighborhood vector. As these values ​​correspond to the same location, they are very likely to be correlated, and thus, the extracted value provides relevant information on the current value.

[0045] - the neighborhood vector further comprises at least one second decoded value (Vdn>>) extracted from another reference map. Advantageously, according to this mode, at least a second value can be extracted from another reference map. This makes it possible to take into account values ​​at different time instants (for example, at two previous instants, or, depending on the order of coding / decoding of the images, at a future instant and a past instant) and thus to better take into account the evolution of the signal for the coding / decoding of the current value.

[0046] Correlatively, the invention also relates to a coding device and a decoding device.

[0047] The characteristics and advantages of the coding or decoding method apply in the same way to the coding or decoding device according to the invention and vice versa.

[0048] The invention also relates to a computer program on a recording medium, this program being capable of being implemented in a computer or a coding or decoding device in accordance with the invention. This program comprises instructions adapted to the implementation of the corresponding method. This program can use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0049] The invention also relates to an information medium or a recording medium readable by a computer, and comprising computer program instructions mentioned above. The information or recording media can be any entity or device capable of storing the programs. For example, the media can comprise a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording medium, for example a floppy disk or a hard disk, a DNA sequence, or a flash memory. On the other hand, the information or recording media may be transmissible media such as an electrical or optical signal, which may be conveyed via an electrical or optical cable, by radio link, by wireless optical link or by other means.

[0050] The program according to the invention can in particular be downloaded from an Internet-type network.

[0051] Alternatively, each information or recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of a method according to the invention. Brief description of the drawings

[0052] Other characteristics and advantages of the present invention will emerge from the description given below, with reference to the appended drawings which illustrate exemplary embodiments thereof which are not in any limiting nature.

[0053] [Fig-1] [Fig.l] schematically represents a coding device used in the scope of the invention;

[0054] [Fig.2] [Fig.2] schematically represents a decoding device used in the scope of the invention;

[0055] [Fig.3] [Fig.3] illustrates an example of a synthetic artificial neural network used in the context of the invention;

[0056] [Fig.4] [Fig.4] illustrates an example of a prediction artificial neural network used in the context of the invention;

[0057] [Fig.5] [Fig.5] is a flowchart representing an example of a coding method which can be implemented by the coding device of [Fig.l];

[0058] [Fig.6] [Fig.6] is a flowchart representing an example of a decoding method which can be implemented by the decoding device of [Fig.2];

[0059] [Fig.7] [Fig.7] illustrates a decoding method used in an embodiment of the invention.

[0060] [Fig.8] [Fig.8] is a flowchart representing a method of encoding cards of characteristics which can be implemented by the coding device of [Fig.l] and by the coding method of [Fig.5].

[0061] [Fig.9] [Fig.9] is a flowchart representing a method of decoding cards of characteristics which can be implemented by the decoding device of [Fig.2] and by the decoding method of [Fig.6].

[0062] [Fig. 10] [Fig. 10] illustrates a method of encoding or decoding feature maps.

[0063] [Fig. 11] [Fig. 11] illustrates another method of coding or of decoding feature maps. Description of the embodiments

[0064] [Fig.l] schematically represents a device for coding digital image sequences ENC.

[0065] This ENC coding device comprises as many ENCV coding sub-modules as there are images to be coded. A sub-module comprises, for an image v of the sequence to be coded denoted Iv or I(PVn), called the current image, a GEN module for generating feature maps, a SE transformation module, an XTR data extraction module, an MLP module corresponding to a synthetic artificial neural network, an NNC neural network coding module capable of coding the synthetic neural network and optionally a prediction neural network, an FMC module for coding and decoding feature maps, an EVAL module for evaluating coding performance, and an MAJ update module.

[0066] The ENC coding device can be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the above-mentioned modules can then be realized by the cooperation of the processor and computer program instructions stored in the above-mentioned memory and designed to carry out the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0067] The coding device ENC of [Fig.l] receives as input a sequence of at least two images to be coded, denoted I(PVn), each comprising respectively a plurality of samples PVn- The index v indicates the image number in the sequence Sv of V images. The index n indicates the pixel number in an image of N pixels. The image I(P Vn) can represent a two-dimensional image and the sequence a plurality of two-dimensional images (video sequence, stereoscopic, multiscopic components, series of medical images, etc.). In one embodiment, an image of the sequence is represented by means of at least one two-dimensional representation, such as a matrix of pixels, each pixel comprising a vector of red (R), green (G), blue (B) components, or, alternatively, a luminance component and at least one chrominance component. The location of each pixel is defined by its abscissa and ordinate coordinates (xn and yn) in the image.In one embodiment, the sequence is a sequence of grayscale images represented using a two-dimensional representation, such as a matrix of pixels, each pixel having a grayscale, or luminance, component. In this case, the vector representing the pixel is reduced to a single component.

[0068] As will be described in more detail later with reference to Figures 3 to 11:

[0069] The module GEN for generating feature maps is configured to generate a plurality of M feature maps denoted FMv; for the current image v. An optional module, not shown, can generate one or more additional maps, L in number, which will not be coded or transmitted.

[0070] In one embodiment, the SE module performs a transformation of the group of feature maps FMv; to generate a group of feature maps FMSvi at the same resolution as the input signal.

[0071] The optional SE module can perform a quantization of the data extracted from this set of M FMv maps. It is recalled that the quantization of a value refers to the mapping of this value with a member of a discrete set of possible code symbols. For example, the set of possible code symbols can be made up of integer values, and the quantization system performs a simple rounding of a real value to an integer value. According to another example, the quantization consists of a multiplication by a given value then a rounding. Then the SE module can perform a transformation of the values ​​of at least one of the characteristic maps, for example an oversampling, an interpolation, a filtering, etc. At the end of the transformation, a transformed characteristic map is according to one embodiment of the same resolution as the images of the input sequence.Advantageously, according to this mode, the characteristic maps which are coded can be of lower resolution than that of the images to be coded, while the transformed maps, which are used for the construction of the characteristic vectors, are at the same resolution as the sequence of images, which facilitates the extraction of the values.

[0072] In one embodiment, the SE module is absent, in this case the values ​​which will be used to construct the characteristic vector are extracted directly from the second group of characteristic maps.

[0073] The XTR module performs an extraction of values ​​in FMvi (and / or FMSvi according to one of the embodiments described previously) characteristic maps, for a current sample Pvn to be coded, as a function of its coordinates in the input signal. For example, if one seeks to code the sample Pvn at the coordinates (xn, yn) of an input image, the XTR module performs an extraction of values ​​in maps at positions imposed by the coordinates (xn, yn).

[0074] In one embodiment, the extracted values ​​constitute the vector Zn. Zn is a J-tuple, that is to say it comprises J elements, or data z;. The vector Zn of index n refers to the characteristic vector of the pixel Pvn.

[0075] The XTR module can also perform a quantization of the data extracted from the set of feature maps. In particular, if the SE module is not used and if the feature maps are at resolutions lower than that of the images of the sequence, the XTR module can take into account the coordinates of the values ​​in the lower resolution maps.

[0076] It will be noted that at least one of the SE or XTR modules must perform a quantification of the values ​​extracted from the characteristic maps of the current image.

[0077] The MLP module is a synthesis neural network defined by K parameters Wvk, capable of processing the vector Zn, or J-Uplet, as input, to generate as output a vector representative of the sample Pvn to be coded of the current image. The synthesis neural network is according to one embodiment an MLP, or Multi Layer Perceptron, consisting of an input layer adapted to the input format (the J-uplet), optionally one or more hidden layer(s), and an output layer adapted to the output format of the output vector, generally a vector comprising NC elements corresponding to the NC components of the image to be decoded. According to one embodiment, NC is equal to 3 and the output vector is the triplet (R, G, B) of the pixel P'vn coded then decoded of the current image.

[0078] The NNC module performs the coding of the synthesis neural network of the current image, in particular of its parameters Wvk. Optionally, the NNC module performs the coding of the prediction neural network, ARM, in particular of its parameters Ov b. During the process of training, or construction, of the coding, that is to say as long as the step of evaluating a performance is not satisfactory, the NNC module performs a coding simulation, followed by a decoding, intended for the evaluation module. Subsequently, it performs the actual coding of the parameters of the synthesis neural network Wvk and optionally the coding of the prediction neural network, ARM. The coded parameters of the current image are denoted Wcvk (and Ocvb). In a known manner, the coding simulation can be identical to the actual coding, or produce an approximation thereof.

[0079] The FMC module performs the coding of the FMv; maps, i.e. the values ​​of the characteristic maps of the current image, called the second group (excluding the maps optionally resulting from oversampling by the SE module, and the additional maps). During the process of training, or construction, of the coding, i.e. as long as the step of evaluating a performance is not satisfactory, the FMC module performs a coding simulation, followed by decoding, intended for the evaluation module. Subsequently, it performs the actual coding of the values ​​of the FMv; maps. The coded maps of the current image are denoted FMcv;. In a known manner, the coding simulation can be identical to the actual coding, or produce an approximation thereof. The coding module quantifies, if necessary, the latent representation of the values ​​of the maps of the second group by using a quantizer to generate an ordered collection of quantized values.Then the encoding module compresses the quantized data, using an encoding that takes into account the . temporal neighborhood of a value to be coded from the feature map. To construct the neighborhood, the FMC module uses coded and decoded feature maps corresponding to images v' of the sequence distinct from the current image v, previously coded then decoded, for example those of the previous image in the sequence, or of two past reference images of the sequence, or of a past and a future reference image, to the extent that the decoding order of the sequence allows it (in a known manner, the decoding order can be different from the chronological order of the sequence), etc. They are called feature maps of the first group, or reference maps, denoted FMdv';. As will be detailed later, the FMC module can include a prediction neural network (ARM) of the current image.

[0080] The EVAL module performs an evaluation and minimization of a coding performance. The evaluation function is for example of the rate-distortion type. The minimization can be carried out by gradient descent, or any other method within the reach of those skilled in the art.

[0081] The MAJ module updates the values ​​of the FMv maps to be encoded in the current image, based on the results of the performance function.

[0082] [Fig.2] schematically represents a DEC decoding device.

[0083] The DEC coding device of [Fig.2] comprises as many DECV decoding sub-modules as there are images to be decoded in the sequence. A sub-module receives as input, for a current image v to be decoded denoted I(Pdvn): - encoded data organized into M FMcv characteristic maps; (also called second group characteristic maps, or current maps); - the Wvck encoded parameters of the synthesis neural network corresponding to the current image; - optionally the Ocvb encoded parameters of the prediction neural network corresponding to the current image; - data organized into feature maps FMdv'iCorresponding to images v' of the sequence distinct from the current image v, previously decoded, for example those of the previous image in the sequence, or of two reference images of the sequence, etc. They are called feature maps of the first group, or reference maps.

[0084] The DECV sub-module decodes the N samples Pdvnd of the image v of the sequence. The set of sub-modules decodes the V images of the sequence to be decoded.

[0085] This DECV decoding device comprises, for a current image, a neural network(s) decoding module NND capable of decoding the MLP' synthesis neural network and optionally the ARM' prediction neural network, a FMD module for decoding the feature maps, an XTR' extraction module of data, an inverse transformation SE' module, an MLP' module corresponding to a synthesis neural network. According to one embodiment, it produces as output a current decoded image, noted I (Pdvn), comprising a plurality of decoded samples Pdvn

[0086] The maps decoded by the FMD module, numbering M, are denoted FMdv^characteristic maps of the current image v). The parameters of the synthesis neural network (MLP') decoded by the NND module are denoted Wdvk. The parameters of the prediction neural network (ARM") optionally decoded by the NND module are denoted Odvb. The FMD module uses to decode a current map FMdv; one or more characteristic maps, called reference maps, of images v' of the sequence distinct from the current image, called reference images, previously decoded by another decoding sub-module, and denoted FMdv';

[0087] The NND module corresponds to the NNC module. It can use, for example, the neural network coding standard proposed by the MPEG-7 part 17 standard, also called NNR (Neural Network Representation). According to one embodiment, the NND module decodes the neural network associated with the image v using the previously decoded parameters of the other neural networks in the sequence.

[0088] The decoder can also generate one or more additional maps, not shown.

[0089] In one embodiment, the module SE' performs a transformation of the first group of decoded feature maps FMdv; to generate a group of transformed feature maps, denoted FMS'v;. The module SE' optionally performs an inverse quantization corresponding to the quantization performed at the encoder. The inverse quantization is not necessary if the quantizer Q of the encoder has performed a simple rounding of the real values ​​submitted to it. The inverse quantization is also not necessary if the neural network is capable of taking into account a quantization of its input data. Otherwise, the decoder performs the inverse operation of the quantizer Q. Then the module SE' optionally performs a transformation of the values ​​of the feature maps, comprising for example an oversampling, an interpolation, a filtering, etc., similar to that performed by the encoder.At the end of the transformation, a transformed feature map FMS'Vi is preferably of the same resolution as the images of the sequence to be decoded.

[0090] In one embodiment, the SE' module is absent, in this case the values ​​which will be used to construct the characteristic vector are extracted from the second group of characteristic maps.

[0091] In one embodiment, the SE' module uses an oversampling neural network.

[0092] The XTR' module is identical to the XTR module in [Fig. 1]. It performs an extraction of values ​​of the M FMdv characteristic maps; of the current image (and / or FMS'v; according to one of the embodiments described previously), for a sample Pdvn to be decoded, as a function of its coordinates in the image to be decoded.

[0093] In one embodiment, the extracted values ​​constitute the vector Zdn. Zdn is a J-tuple, i.e. it comprises J elements, or data zd; In one embodiment, J=M.

[0094] In one embodiment, the XTR' module also performs processing of the extracted values, to generate the vector Zdn. The XTR' module can perform inverse quantization of the data extracted from the set of feature maps, in particular if the SE' module is absent. The processing can include other operations, such as filtering, scaling, etc., similar to those performed by the encoder.

[0095] The MLP' module is a so-called synthesis neural network, defined by K parameters Wdvk, capable of processing the vector Zdn, or J-Uplet, as input, to generate as output a vector representative of the sample Pvn to be decoded, generally a vector comprising NC elements corresponding to the NC components of the image to be decoded. According to one embodiment, K=3 and the output vector is the triplet (R, G, B) of the decoded pixel Pdvn. The MLP' module has an identical structure to the MLP module, and its parameters are either identical if the coding of its parameters Wvk is lossless, or different if the coding is carried out with losses.

[0096] When all the samples Pvn of the signal have been decoded, we have a reconstructed signal I (Pdvn), according to one example an image Iv comprising N samples decoded in the form of N vectors Pdvn.

[0097] The decoding device DEC can be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the modules mentioned above can then be realized by the cooperation of the processor and computer program instructions stored in the aforementioned memory and designed to carry out the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor

[0098] [Fig.3] illustrates an example of a synthetic artificial neural network used for encoding and decoding in the context of embodiments of the invention.

[0099] The synthetic artificial neural network used for coding, MLP, and the synthetic artificial neural network used for decoding, MLP, are defined by an identical structure, comprising for example a plurality of layers of artificial neurons, and by a set of weights and activation functions associated respectively with the artificial neurons of the network concerned.

[0100] A vector representation of a current sample (a vector Zn or Zdn from the feature maps FMv; / FMSvi at the encoder or FMdvi / FMSdv; at the decoder) is applied as input (i.e. on an input layer) of the MLP or MLP artificial synthesis neural network. The artificial synthesis neural network produces as output a vector, according to one embodiment the color components (R, G, B) constituting a color pixel of an image.

[0101] The concatenation of all these reconstructed pixels in an image (2D, 3D) constitutes the decoded, or reconstructed, image.

[0102] At the encoder, the MLP synthesis artificial neural network is trained on the image, so as to minimize the differences between the input representation of the current image I(Pvn) and its output representation I(P'vn), while also minimizing the amount of data to be encoded. The EVAL module performs a performance measurement in this sense.

[0103] Once the encoder training is complete, the network parameters are encoded, either lossless, in which case the MLP' neural network is identical to MLP, or lossy, in which case the MLP' network may be slightly different from MLP.

[0104] [Fig.4] illustrates an example of a prediction artificial neural network used to encoding (ARM) and decoding (ARM”) for the feature maps, in the context of embodiments of the invention.

[0105] The artificial prediction neural network used for coding, ARM, and the artificial prediction neural network used for decoding, ARM, are defined by an identical structure, comprising for example a plurality of layers of artificial neurons, and by a set of weights and activation functions associated respectively with the artificial neurons of the network concerned.

[0106] A vector representation of a current neighborhood (a vector Cvn or Cdvn from the feature maps FMdv; or FMdv'; of another image of the sequence, to the encoder or to the decoder) is given as input (i.e. on an input layer) of the artificial neural network of prediction ARM (at coding) or ARM » (at decoding).

[0107] The prediction artificial neural network behaves as a function that produces as output a prediction of the current value of the feature map being processed, which can be presented in the form of a prediction value or probability data.

[0108] According to one embodiment, at the encoder, the network implements a function fv which provides an expected mean and / or variance (p,o) for the current value of the current value Vvn of the map FMv; to be encoded. These statistical values ​​are used to perform the entropy coding of this value. For example, if the function produces a mean, this mean is subtracted from the current value and only the difference is entropically encoded, this mean being considered as a prediction of the current value. Alternatively, if the function produces a mean and a variance, the mean is subtracted from the current value, and the difference is encoded using an entropy coding adapted to the variance, for example by quantizing said variance into a predetermined set of variances and associating a type of entropy coding with each quantized variance value. At the decoder, the network implements a function fv which provides an expected mean and / or variance for the current value of the current value Vdvn of the map FMdv; to be decoded. These statistical values ​​are used to perform the entropy decoding of this value. For example, if the function produces a mean, the current value is decoded by the entropy decoder and this mean is added to the current value.Alternatively, if the function produces a mean and a variance, the current value is decoded by the decoder using variance-adapted entropy decoding, for example by quantizing said variance into a predetermined set of variances and associating an entropy decoding type with each quantized variance value.

[0109] According to another embodiment, the neural network can produce the expected probability (pr) of each possible value of the current sample. In this case, the entropy coding or decoding will be adapted to this probability (as is known for Huffman or arithmetic entropy coding).

[0110] At the encoder, the ARM prediction artificial neural network is trained on the current image and the reference images, so as to minimize the amount of data to be encoded. The EVAL module performs a performance measurement in this sense. It is recalled that the overall performance measurement relates to minimizing the distortion between coded and then decoded images I (P'vn) and the input images I (Pvn), while minimizing the coding rate. According to one embodiment, the feature maps are encoded without losses, via entropy coding. In this case, the coding of the feature maps influences the rate, but not the distortion of the coded images. According to another embodiment, if the feature maps are encoded with losses, the coding of the feature maps influences the rate, but also the distortion.

[0111] Once training is complete, the B Ocvb parameters of the network are encoded, either losslessly, in which case the ARM' neural network is identical to ARM, or lossily, in which case the ARM' network may be slightly different from ARM.

[0112] [Fig.5] is a flowchart representing an example of a coding method that can be implemented by the coding device of [Fig.l].

[0113] According to this embodiment, the input signal is a two-dimensional image of the image sequence, each sample to be coded is therefore a pixel Pvn with coordinates (xn, yn) of the current image v.

[0114] Encoding takes place in two main phases:

[0115] In a first phase, called the construction phase, learning is carried out, in order to determine, for an input image I(Pvn), the values ​​of the FMv; maps and the parameters Wvk and optionally Ovb to optimize a global cost function. The learning is for example carried out by a gradient descent, followed by an update of the parameters of the MLP synthesis neural network, the values ​​of the FMv; feature maps, and optionally the ARM prediction neural network. As is known in the state of the art, the cost function can be of the rate-distortion type, or rate, or distortion, or perceptual. To measure the rate R, it is necessary to simulate the coding of the FMv; maps, then to measure the associated coding rate (the size of the flow Bl). According to one embodiment, the coding of the Wvk and / or Ovb parameters is not simulated because their influence is less important than that of the feature maps.According to one embodiment, the coding of the parameters Wvk and / or Ovb is also simulated and the associated flow rate (the size of the flow B2) is measured. To measure the distortion D, it is necessary to simulate the coding then the decoding of at least part of the image Iv, to obtain at least one pixel P'vn resulting from a simulation of coding then decoding, then to measure the difference between this part of the image I (Pvn) at input and a corresponding part of the image I (P'vn) coded then decoded.

[0116] Then during a second phase, called the coding phase, the FMv maps; and the Wvk and possibly Ovb parameters are encoded to produce the coded values ​​FMcvi and Wcvk (and possibly Ocvb) before transmission or storage. They constitute the compressed representation of the image (Pvn).

[0117] We will now describe the steps of a method according to one embodiment of the invention.

[0118] During a step E20, an image I(Pvn) to be coded with index v in a sequence of images, comprising a plurality of N samples Pvn, and a number NC of components, is provided as input to the method.

[0119] During a step E21, the M FMv maps; of the second group, or feature maps of the current image v, are initialized. Subsequently, the values ​​of the FMv maps i, and possibly the parameters of the neural networks must be optimized during the construction phase.

[0120] According to one embodiment, the FMv; cards have the same resolution as the input image I(Pvn) and therefore each comprise the same number of values ​​N as there are samples Pvn to be coded. According to one embodiment, the FMv; cards have a resolution less than or equal to that of the input signal I(Pvn) and therefore comprise, for at least one of them, a number N' of values ​​to be coded less than N. According to one variant, the first FMvk card has the resolution of the images and each subsequent card has a resolution half that of the previous one. According to one embodiment, several FMv; cards have the same resolution, less than that of the input signal I(Pvn).

[0121] According to one embodiment, the maps FMv; are transformed to provide a group of transformed feature maps FMSv;. In this mode, the feature vectors are preferably extracted from the transformed maps, and not directly from the maps of the second group. In this mode, the feature vectors are therefore extracted indirectly from the maps of the second group. The transformed maps are not coded, they are used only for the construction of the feature vectors.

[0122] According to one embodiment, the FMv; maps of the current image are initialized by predefined constant values. According to another embodiment, the feature maps are initialized by a set of random real values. According to one embodiment, one or more maps, forming an additional group of additional feature maps, are generated, and added to the second group. They are used for the construction of the feature vector but are neither stored nor transmitted.

[0123] The FMv; characteristic maps of the second group are subsequently updated, or refined, during a step E22, by the update module MAJ of the encoder during its learning.

[0124] During a step E23, the FMv; maps of the second group are coded by the FMC module of the encoder. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is an actual coding and the coded values ​​constitute the stream B1. The simulation can be identical to the actual coding but it can also be different (for example, simplified). For this coding, a technique is used for predicting a feature map value by its temporal neighborhood, that is to say from feature maps of a first group, consisting of feature maps of other images of the sequence than the current image, called reference maps of reference images, as will be described in more detail in support of [Fig.8].In one embodiment, the structure and parameters of a synthetic neural network (optionally prediction) are initialized for example during the first iteration of this step. These parameters are subsequently updated, or refined, during the construction phase, during subsequent iterations of the method.

[0125] In one embodiment, the FMv; cards are coded in order (FMvb FMv2,..., FMv4), and the variables of each card in a predefined order, for example lexicographic. Each card undergoes entropic coding. The entropic coding produces a compressed stream B1 whose flow rate can be measured subsequently during a step E29.

[0126] During a step E24, according to one embodiment, the M maps of the first group FM; are transformed by the module SE to generate transformed maps FMSv; at the resolution of the images of the input sequence.

[0127] According to one embodiment, M FMSv; maps are generated. According to one embodiment, each FMv; map is transformed into an FMSv; map. According to one embodiment, at least one FMv; map has a lower resolution than the images of the sequence to be coded and the transformation operation includes oversampling so that the transformed FMSvi map includes the same number of samples as the current image. Oversampling consists of adding values ​​to the FMSv; maps to achieve the resolution of the images of the input sequence. It can be simple (by nearest neighbor replication) or include interpolation (linear, polynomial, by filtering, etc.). It can also be performed by an oversampling neural network.

[0128] During a step E25, values ​​are extracted by the XTR module from the transformed FMv maps; or possibly FMSv maps; and optionally from additional maps. This extraction is carried out as a function of the coordinates (xn, yn) of the sample Pvn of the input signal. It can also be carried out as a function of the resolution of the map considered.

[0129] According to one embodiment, the characteristic Zn vector results directly from this extraction.

[0130] The samples to be coded are for example processed in sequential order, from n=1 to n=N.

[0131] According to one embodiment, during step E25, the characteristic vector Zn is constructed by the module XTR from the values ​​extracted from the maps FMv; or FMSv; for each sample Pvn of coordinates (xn, yn) of the input image. The processing may include a quantification of the values ​​extracted from the maps FMv; or of the vector Z nconstituted, if necessary. The processing may include other operations, such as filtering, scaling, the application of any function, preferably monotonic, etc.

[0132] In one embodiment, Zn has as many values ​​as there are FMvi or FMSvi input cards. In this case we have J=M.

[0133] In one embodiment, Zn is a J-tuple (zb z2,..., Zj), consisting of the values ​​of the maps FMv; or FMSv; located at the coordinates (xn, yn) of the current pixel Pvn, as will be illustrated in support of [Fig.7].

[0134] In one embodiment, Zn is a J-tuple constructed from values ​​taken from the FMvi maps at coordinates that may be different depending on the maps. For example, if the FMv maps are at different resolutions because they have been downsampled, the coordinates are adapted (by scaling) to match the resolution of each map.

[0135] In one embodiment, Zn is a J-tuple constructed from values ​​taken from the FMv maps; by applying the processing to one or more values ​​of the maps, for example a filtering of the values ​​neighboring the targeted value in a map. example, for a current sample Pvn in an FMvi map which is at the same resolution as the input signal, we can extract the values ​​located at the coordinates (xn, yn), (xn-l, yn), (xn, yn-l) and (xn-l, yn-l) and apply a processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (zO of the element i of the vector Zn relative to this FMv map;. According to another example, in an FMv map; which is at half the resolution of the input signal, we can consider the values ​​located at the coordinates (xn / 2, yn / 2), (xn / 2-l, yn / 2), (xn / 2, yn / 2-l) and (xn / 2-l, yn / 2-l) and apply a processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (zO of the element i of the vector Zn relative to this FMv map;. vector Zn relative to this FMv map;.

[0136] During a step E26, the vector Zn is processed by the MLP synthesis neural network to generate as output a vector representative of the sample Pvn to be coded, according to one embodiment the triplet (R, G, B) of the sample P'vn (the sample Pvn coded then decoded).

[0137] The structure and the Wvk parameters of the synthetic neural network (and optionally those of the synthetic neural network, and those of the oversampling neural network if it exists) are initialized for example during the first iteration of this step. These parameters are subsequently updated, or refined, during the construction phase, during subsequent iterations of the method.

[0138] According to one embodiment, the parameters of the synthesis neural network and / or the prediction neural network are initialized by predefined values ​​known to give a satisfactory result (for example, following training on a corpus of images). According to another embodiment, the parameters of the synthesis neural network and / or the prediction neural network are initialized by a set of random values.

[0139] During a step E27, the Wvk parameters of the MLP synthesis neural network and the Ovb parameters of the ARM prediction neural network, if it exists, are quantized and coded. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is an actual coding and the coded values ​​constitute the stream B2. The simulation can be identical to the actual coding but it can also be different (for example, simplified). Any known technique can be used for this purpose, for example the neural network coding standard proposed by the MPEG-7 part 17 standard, also called NNR (Neural Network Representation). It should be noted that in this case, it is necessary to choose the amount of degradation that the coding brings to the Wvk weights, optionally Ovb.

[0140] During a step E28, a performance measurement is evaluated.

[0141] For this purpose, the coding simulation rates associated with the characteristic maps of the first group (simulation of the B1 flow by coding the FMv maps;) and optionally with the parameters of the neural network(s) (simulation of the B2 flow by coding of the Wvket parameters (optionally Ovb) are measured.

[0142] According to one embodiment, the cost function is of the rate-distortion type, noted (D+L*R), where D for example the quadratic error measured between the input signal and the decoded signal (or the error measured on a subset of samples of the signal). According to another example, D is calculated from a perceptual function such as the SSIM (for Structural SIMilarity), or MSSSIM (for Multi-scale Structural SI-Milarity). According to one embodiment, R is the simulated rate of the stream B1; according to another embodiment, R is the total rate used to code this image, that is to say the sum of the simulated rates of B1 and B2. L is a parameter which regulates the rate-distortion trade-off. Other cost functions are possible.

[0143] As long as the cost function has not reached its minimum, the performance measurement is not satisfactory, and the method is repeated from step E22. Alternatively, the method can be interrupted after a predefined time or a predefined number of iterations, so as to control its complexity or duration. This minimization can be carried out by a mechanism known as gradient descent with updating of the parameters during step E22 for the values ​​of the characteristic maps and E26, E27 for the parameters of the network(s).

[0144] During a step E29, if the cost function has reached its minimum, the training stops. If a coded version corresponding to the last simulation of the parameters of the synthesis neural network (Wvk) and the feature maps (FMv;) is available, the streams B1 and B2 can be constituted therefrom. According to another embodiment, the actual coding of the updated parameters of the synthesis neural network (Wvk) and the values ​​of the feature maps (FMvj) and optionally of the prediction neural network (Ovb) is carried out at this step to produce the encoded parameters Wcvk (optionally Ovb) and FMcv; which constitute streams B1 and B2. The streams B1 and B2 can be concatenated to produce a final stream. According to one embodiment, the stream B2 of the coded parameters of the neural network(s) is (are) stored or transmitted before the stream B1, in order to be able to be decoded before the stream B1.

[0145] [Fig.6] is a flowchart representing an example of a decoding method that can be implemented by the decoding device of [Fig.2].

[0146] During a step E30, the streams B1 and B2 are extracted from the encoded stream. They contain respectively, for a current image v of the sequence, the coded representations of the maps of the second group FMcv; and parameters Wcvk, optionally parameters Ocvb, of the images of the sequence.

[0147] During a step E31, the M maps FMdv; of a current image v of the sequence are generated by decoding the values ​​FMcv; of the current image as a function of the values ​​of characteristic maps already decoded FMdv'; for other images of the sequence, distinct from the current image. For example, if the current image is the image 3 (v=3), the feature maps of one or more previous images (v'= 1 and / or v'=2) can be used for decoding. In another example, the feature maps of one or more previous images (v'= 1 and / or v'=2) and of one or more future images in the sequence (v'= 4,....) can be used for decoding, provided that these have already been decoded. Naturally, if the image is the first in the sequence, only the current FMdn maps are available. In this specific case, decoding is not a function of previous image maps, but simply of values ​​present in the map associated with the current image.

[0148] For decoding, a technique is used for predicting a feature map value by its temporal neighborhood, as will be described for example in support of [Fig.9]. In one embodiment, the FMdv maps are decoded in order (FMdvB FMdv2,... FMdv4), and the values ​​of each map in a predefined order, for example lexicographic.

[0149] According to embodiments as described for the encoder:

[0150] - The FMdvi cards have the same resolution as the I(Pdvn) signal to be reconstructed, that is, they contain N values.

[0151] - FMdvi cards are at a resolution lower than or equal to that of the I(Pdvn) signal at to reconstitute.

[0152] - Several FMdvi cards have the same resolution, lower than the signal resolution.

[0153] - The reference images v', already decoded, whose characteristic maps are used to decode the current feature map, are indicated in the bitstream Bl, for example as an index indicating the image number in the sequence.

[0154] - The reference images v', already decoded, whose characteristic maps are used to decode the current feature map are at predefined positions. For example, this can be the previous image in the sequence, the next one (provided it is already decoded), the first one, etc.

[0155] - The feature maps used to decode the feature map current are indicated in the binary stream Bl, for example in the form of an index indicating the position of the feature map (between 1 and M) in the group of feature maps associated with a reference image.

[0156] - The feature maps used to decode the feature map current are at predefined positions. For example, this could be the feature card that is at the same position as the current feature card in the group of feature cards associated with a reference image, or the previous one, or the previous and next one, etc.

[0157] - the feature cards of the second group, associated with the current image, are also used for decoding the current feature map. For example, a spatial neighborhood of the current feature map value can be extracted into the current feature map (causal neighborhood) or into an already decoded feature map of the same (second) group.

[0158] According to one embodiment, one or more maps forming an additional group of additional feature maps are generated and complete the second group. They are not decoded but generated by the decoder in a manner identical to the generation of the encoder. They typically comprise data which can assist the MLP network in the signal reconstruction task.

[0159] During a step E32, according to one embodiment, the M maps of the second group FMdVi are transformed by the module SE' to generate transformed maps FMS'v; at the resolution of the input images. This step is similar to step E24 which was described for the encoder in support of [Fig.5], and the embodiments apply. In particular:

[0160] According to one embodiment, M FMS'v maps are generated.

[0161] According to one embodiment, each FMdvi card is transformed into an FMS'v card i*

[0162] According to one embodiment, at least one FMdvi map has a lower resolution than the images of the image to be coded and the transformation operation includes oversampling so that the transformed FMS'v; map includes the same number of samples as the input image. Oversampling consists of adding values ​​to the FMS'v; maps to achieve the resolution of the input image. It can be simple (by nearest neighbor replication) or include interpolation (linear, polynomial, by filtering, etc.). It can also be performed by an oversampling neural network.

[0163] The transformation may optionally include inverse quantization of the extracted values, if necessary. However, inverse quantization is not mandatory.

[0164] During a step E33, values ​​are extracted by the XTR' module from the FMdvi or possibly FMS'v maps; transformed. This extraction is carried out as a function of the coordinates (xn, yn) of a sample Pvn of the current image. It can also be carried out as a function of the resolution of the map considered. This step is similar to step E25 which was described for the encoder in support of [Fig.5], and the embodiments apply. In particular:

[0165] According to one embodiment, the characteristic vector Zdn results directly from this extraction.

[0166] In one embodiment, Zdn is a J-tuple (zb z2,..., Zj), consisting of the values ​​of the maps FMdv; or FMS'v; located at the coordinates (xn, yn) of a current pixel Pdvn, as will be illustrated in support of [Fig.7].

[0167] The samples to be decoded are for example processed in sequential order, from n=1 to n=N.

[0168] According to one embodiment, during a step E33, a vector Zdn is constructed by the module XTR' from the values ​​extracted from the maps FMdv; and FMdv'i of the second group or from the transformed maps FMS'v;, and optionally from maps of an additional group, for each sample Pdvn of coordinates (xn, yn) to be decoded, as a function of the coordinates (xn, yn). This step is similar to step E25 which was described for the coder in support of [Fig.5] and the described embodiments apply. The extraction may comprise an inverse quantization of the extracted values ​​or of the constituted vector Zdn, if necessary.

[0169] During a step E34, the parameters Wdvk of the synthesis neural network MLP' and possibly the parameters Odvb of the prediction neural network (and optionally the parameters of an oversampling network of the SE module) are generated by decoding the values ​​Wcvk and Ocvb of the stream B2. Any known decoding technique corresponding to the coding technique used by the encoder can be used for this purpose. The synthesis neural network MLP' is similar to the synthesis network MLP, i.e. it has the same structure and includes the same parameters, except for the coding, which can be performed with or without losses. Similarly, the prediction neural network ARM", if it is used to decode the feature maps, is similar to the prediction network ARM, i.e. it has the same structure and includes the same parameters, except for the coding, which can be performed with or without losses.

[0170] According to one embodiment, the stream B2 is decoded before the stream B1, in order to be able to have the synthesis neural network MLP' and possibly the prediction neural network ARM' before starting to decode the samples.

[0171] During a step E35, the vector Zdn is processed by the synthesis neural network MLP' to generate as output a second vector representative of the sample Pdvn to be decoded, according to one embodiment a triplet which is injected into the decoded image I (Pdvn) at the positions (xn, yn) of the color components (Rd, Gd, Bd) of the image v. This step is similar to step E26 which was described for the coder with the support of [Fig.5].

[0172] When all the samples have been processed, the current decoded image corresponding for example to the image I(Pdvn) is available.

[0173] [Fig.7] illustrates a decoding method used in one embodiment of the invention.

[0174] In this embodiment, the decoded cards FMdv; associated with the current image v of the sequence are 4 in number. In a preferred mode, they are 7 in number.

[0175] In this embodiment, the first FMdvi card has the same resolution as the image Iv, and therefore has WxH variables, where W represents the width of the image in pixels, and H its height. The second FMdv2 map has half the resolution (in each dimension) of the FMdvi map. Each additional map has half the resolution of the previous map. This structure makes it possible to reduce the number of variables in the feature maps, which facilitates decoding while minimizing the coding cost.

[0176] In this embodiment, a group of reference maps associated with an image v' of the sequence, already decoded (for example the previous image in the sequence, i.e. v'=vl), is available at the decoder. The group of maps associated with the image v' has the same structure as the group of maps currently being decoded associated with the current image v: the first map FMdv'i has the same resolution as the image Iv', the second map FMdv'2 has half the resolution (in each dimension) of the map FMdv'i, each additional map has half the resolution of the previous map.

[0177] When decoding a current map, for example, FMdvb at least one reference map is used, for example FMdv'i and FMdv'2As will be illustrated in support of Figures 10 and 11, the reference maps are used to construct the neighborhood of the values ​​to be decoded in the current map.

[0178] When the FMdv; maps are decoded, the FMdv2 map is oversampled by a factor of 2 in each dimension, according to any oversampling method within the reach of those skilled in the art, the FMdv3 map is oversampled by a factor of 4 in each dimension, and the FMdv4 map by a factor of 8 in each dimension.

[0179] The resulting maps FMS'v; are of the same resolution as the image to be decoded, and therefore include WxH values, where W represents the width of the image in pixels, and H its height.

[0180] In this embodiment, the vector Zdn is a 4-tuple (zi... z4) consisting of the values ​​of the FMS'v maps; located at the coordinates (xn, yn) of the current pixel Pdvn. The vector Zdn is optionally dequantized then processed by the synthesis neural network MLP' to generate as output the triplet (R, G, B) representative of the sample Pdvn to be decoded. The triplet (R, G, B) is inserted into the decoded image I (Pdvn) at the coordinates (xn, yn) in the color components (Rd, Gd, Bd).

[0181] According to a variant presented in dotted lines, there are 5 cards: an additional card FME'O has been introduced. In this embodiment, the vector Zdn is a 5-tuple. Such an additional card can be, for example, in a non-limiting manner: - A map showing the abscissa of that point at each point. - A map showing the ordinate of that point at each point. - A map with positional coding at each point (as described for example at the internet address https: / / skosmos.loterre.fr / P66 / fr / page / -K0D65X2X-X). - A map representing an image distinct from the image being processed, capable of providing information on the image to be coded, for example the previously processed image if the current image is part of a series of images to be coded such as a video, a set of medical images, a multi-view representation, etc. - A map representing a feature map of an image distinct from the image being processed, capable of providing information on the image to be coded, for example a previously processed map if the current image is part of a series of images to be coded such as a video, a set of medical images, a multi-view representation, etc. - A card containing the value of an already decoded sample from the same card, for example the previous sample in the decoding order.

[0182] [Fig.8] is a flowchart representing a method of coding feature maps which can be implemented by the coding device of [Fig.l] and by the coding method of [Fig.5].

[0183] These steps constitute sub-steps of step E23 described previously in support of [Fig.5]. Their purpose is to code a current value Vvn of a current feature map FMv; of the second group of the current image being processed using neighborhood values ​​extracted from the reference maps FMv'i associated with a reference image v".

[0184] During a sub-step E231, a neighborhood vector (Cvn) is established, comprising values ​​neighboring the value Vvn. As will be illustrated later with reference to FIGS. 10 and 11, these neighboring values ​​may be located in the equivalent map (i.e. of the same order number in the group) and / or in one or more different maps of the plurality M of maps FMv'i. This neighborhood vector is made up of a number C of values, or data, corresponding to neighborhood values ​​(for example, C=10).

[0185] According to a variant, values ​​can also be extracted from the FMv maps; associated with the current image, already decoded. In this case, these values ​​must be known to the encoder and the decoder, they must therefore be located in a causal neighborhood of the value Vvn.

[0186] According to a first embodiment, these values ​​are used to determine the context of an entropy coder to code the current value during a step E234. This coder may be a CAB AC (Context-adaptive binary arithmetic coding) type coder. This type of coder is well known to those skilled in the art. It is notably used in the H.265 / HEVC video compression standard. It is an arithmetic coder whose compression is lossless. It decomposes all the non-binary symbols into binary symbols. Then, for each bit, the coder selects the most suitable probability model and uses a context to optimize the estimation of the pro enabled. This context can be defined by information from neighboring elements. Arithmetic coding is then applied to compress the resulting data. As is known to those skilled in the art, there are several ways to use the neighborhood vector to produce context information. For example, one can count the number of non-zero neighboring values, and associate a context with each number. Alternatively, one can perform comparisons between several neighboring values, and associate a given context with an ordering configuration between the neighboring values, for example by ranking the neighboring values ​​in ascending order, and associating a context with each possible order.

[0187] In a second embodiment, the neighborhood is used to predict, during a step E232, the current value from an autoregressive model. It is recalled that an autoregressive model predicts a sample of a series by its past values. In this mode, the past values ​​are constituted by the context, and the difference between the predicted variable and the real value is quantified then en-tropically coded during step E234.

[0188] In a third embodiment, as illustrated in support of [Fig. 11], an ARM prediction neural network is used during step E233 to predict the statistical characteristics of the variable to be coded. The neighborhood vector is given as input to the ARM network, to provide as output a prediction of the current value. According to one embodiment, the ARM network behaves like a function fv as described in support of [Fig.4], which provides a set of statistical parameters (mean, variance, median, etc.) used to entropically code the current value. The role of this ARM module is to best predict the current value, for all the values ​​Vvn to be coded from the FMv maps, in order to reduce the throughput required for encoding the characteristic maps.According to another embodiment, the ARM prediction neural network is used to produce the expected probability (pr) of the possible value of the current sample. Entropy coding is adapted to this probability (as is known for Huffman or arithmetic entropy coding).

[0189] In a fourth embodiment, each feature map is divided into blocks of predetermined size, and the coding of each block comprises a transform (e.g., a DCT, Discrete, Cosine Transform, a Haar transform, etc.), and the transformed values ​​are encoded by an entropy coding.

[0190] At the end of the process, the current coded value Vvnd of the current map FMv; being processed is coded in the form of a variable Vcvn.

[0191] [Fig.9] is a flowchart representing a method of decoding cards of ca characteristics which can be implemented by the decoding device of [Fig.2] and by the decoding method of [Fig.6].

[0192] These steps constitute sub-steps of step E31 described previously in support of [Fig.6]. Their purpose is to decode a current value Vdvn of a current feature map FMdv; of the second group of the current image being processed using neighborhood values ​​extracted from the reference maps FMdv'; associated with already decoded reference images.

[0193] During a sub-step E311, a neighborhood vector (Cdvn) is established, comprising values ​​neighboring the value Vdvn. This step is similar to step E231 previously described and the same embodiments apply. In particular, these neighboring values ​​may be located in the equivalent map (i.e. of the same order number in the group of characteristic maps) and / or in one or more different maps of the plurality M of FMdv' maps; This neighborhood vector is made up of a number C of values, or data, corresponding to neighborhood values ​​(for example, C=10) located in the same map and / or in a different map of the plurality M of FMdv' maps;.

[0194] According to a variant, values ​​can also be extracted from the FMdv maps; of the current image. In this case, these values ​​must be known to the encoder and the decoder, they must therefore be located in a causal neighborhood of the Vdvn value.

[0195] According to a first embodiment, these values ​​are used to determine the context of an entropy decoder to decode the current value during a step E314. This decoding is of a type similar to that which was used in the encoder, for example CAB AC. The use of the neighborhood to produce context information is similar to that which was chosen in the encoder. For example, the number of neighboring values ​​other than zero can be counted, and a context can be associated with each number. Alternatively, comparisons can be made between several neighboring values, and a given context can be associated with an order configuration between the neighboring values, for example by classifying the neighboring values ​​in ascending order, and by associating the context with each possible order.

[0196] In a second embodiment, the neighborhood is used to predict during a step E312 the current value from an autoregressive model. In this mode, the past values ​​are constituted by the context, and the difference between the predicted variable and the actual value is quantified then entropically coded during step E314.

[0197] In a third embodiment, as illustrated in support of [Fig. 11], a prediction neural network ARM' is used in step E313 to predict the statistical characteristics of the variable to be decoded. The neighborhood vector is applied as input to the ARM network", to provide as output a prediction of the current value. According to one embodiment, the ARM network behaves like a function fv as described in support of [Fig.4], defined by a set of statistical parameters (mean, variance, median, etc.) used to entropically decode the current value. In another embodiment, the prediction neural network ARM' is used to produce the expected probability (pr) of the possible value of the current sample. Entropy decoding is adapted to this probability (as is known for Huffman or arithmetic entropy coding). The ARM' network is identical to the ARM network if the coding was performed losslessly.

[0198] In a fourth embodiment, each feature map is divided into blocks of predetermined size, and the decoding of each block comprises the entropy decoding of values ​​which then undergo an inverse transform (e.g. an inverse DCT, Discrete, Cosine Transform, an inverse Haar transform, etc.).

[0199] At the end of the process, the current decoded value Vdvnde of the current map FMdv; being processed is decoded.

[0200] [Fig. 10] illustrates a method of encoding or decoding feature maps according to one of the embodiments.

[0201] In this illustration, the coding (resp. decoding) of the current value Vvn (resp. Vdvn) located in coordinates (xn, yn) in the current feature map FMvi (i.e. the one being coded or simulated) (resp. FMdvi - i.e. the one being decoded) uses the contextual information of two maps of a reference image v' that has already been decoded: the map FMv'i that is located at the same position as the current map in the group of reference maps, and the previous map FMv'2 (resp. FMdv'2). The values ​​located at coordinates (xn-l, yn -1), (xn-l, yn), (xn-l, yn+l), (xn, yn-l), (xn, yn), (xn, yn+l), (xn+l, yn-l), (xn+l, yn), (xn+l, y n+l) in the FMv'i (resp. FMdv'i) map, and the variables located at coordinates (xn / 2-1, yn / 2-1), (xn / 2-l, yn / 2), (xn / 2, yn / 2-l), (xn / 2, yn / 2) in the FMv'2 (resp. FMdv2 ) layer are used to determine the neighborhood to encode (resp. decode) the current value.These values, which are all available to the encoder and the decoder, constitute the neighborhood vector Cvn (resp. Cdvn) of the value to be encoded Vvn (resp. Vdvn) which can be used in one of the embodiments described previously in support of [Fig.8] (resp. [Fig.9]).

[0202] [Fig. 11] illustrates another method of encoding or decoding feature maps according to one of the embodiments

[0203] In this illustration, the coding (resp. decoding) of the current value Vvn (resp. Vdvn) located in coordinates (xn, yn) in the current feature map FMvi (resp. FMdvO) uses the contextual information of the reference map located at the same position in a reference image v ». The values ​​represented in gray are used to determine the neighborhood for coding (resp. decoding) the current value. These values, which are all available to the encoder and the decoder, constitute the neighborhood vector Cvn (resp. Cdvn) of the value Vvn (resp. Vdvn) which can be used in one of the embodiments described previously in support of [Fig.8] (resp. [Fig.9]).

[0204] In the illustrated embodiment, the neighborhood vector is extracted by a CTX module (resp. CTX ") of the FMC coding module (resp. FMD), then it is applied to the input of the ARM prediction neural network (resp. ARM "), used to predict the statistical characteristics (p,o) or the probability (pr) of the value to be coded (resp. decoded) represented by a cross. The value is coded (resp. decoded) by the CE entropy coder (resp. DE).

Claims

Claims

1. Method for coding a sequence of images (Sv) comprising the following steps: - a construction step, comprising the following sub-steps for at least one current image (v, I(Pvn)) of the sequence: - construction (E21, E22) of a first group of characteristic maps (FMv'i) representative of at least one image distinct from the current image (v1,1(Pv'n)), called the reference image; - construction (E21, E22) of a second group of feature maps (FMvj) representative of the current image (v, I(Pvn)); - for at least one sample, called current sample (Pvn), of the current image, associated with a position (xn, yn) in the current image: - construction (E25) of a characteristic vector (Zn) from said characteristic maps (FMv;) of said second group, as a function of said position (xn, yn) of said current sample (Pvn); - processing (E26) of said characteristic vector (Zn) by an artificial neural network (MLP), called a synthetic neural network, defined by a set of parameters (Wvk), to provide a vector (P'vn) representative of a decoded value of said at least one current sample; - updating (E22, E27) of at least one value of one of said characteristic maps of said second group and / or of at least one parameter of said network, as a function of a coding performance measurement, - a coding step (E23) of said second group of characteristic maps (FMv;) comprising, for at least one value, called current value (Vvn), of one of said characteristic maps, called current map, an entropic coding of said value as a function of a neighborhood comprising at least one value (Vv'n) extracted from at least one map, called reference map, of the first group of characteristic maps, - a coding step (E27) of said set of parameters (Wvk) of said synthetic neural network.

2. Coding method according to claim 1, characterized in that the step of coding said current value (Vvn) comprises the following sub-steps: - construction (E231) of a neighborhood vector (Cv'n) comprising said at least one neighborhood value extracted from said at least one reference map (FMv'j);

3.

4. - processing (E233) of said neighborhood vector (Cv'n) by an artificial neural network, called a prediction neural network (ARM), defined by a set of parameters (Ovb), to provide a prediction of said current value (Vvn); - updating (E22, E23, E28) at least one parameter of said prediction network, as a function of the coding performance measurement, - a step of coding said set of parameters (Ovb) of said prediction network. Method for decoding from a binary stream of a sequence of images (Sv), comprising the following steps for at least one current image (v, I(Pdvn)) of the sequence: - obtaining (E31) at least one characteristic map (FMdv';), called reference map, from a first group of decoded characteristic maps representative of at least one image distinct from the current image (v1,1(Pdv'n)), called reference image. - decoding (E31) of a second group of characteristic maps (FMdvi) representative of the current image comprising, for at least one value, called current value (Vdvn), of one of said characteristic maps, called current map, an entropic decoding (DE) of said value as a function of a neighborhood comprising at least one decoded value (Vdv'n) extracted from said at least one reference map; - decoding (E34) of a set of parameters (Wdvk) representative of a synthetic neural network (MLP'); - for at least one sample, called current sample (Pdvn), of the current image (v), associated with a position (xn, yn) in the current image: - construction (E33) of a characteristic vector (Zdn) from the characteristic maps of said second group (FMdv;), as a function of said position (xn, yn) of said current sample, and: - processing (E35) of said vector (Zdn) by said synthesis neural network (MLP') to provide a vector representative of a decoded value of said at least one current sample (Pdvn). Decoding method according to claim 3, characterized in that the step of decoding said current value (Vdvn) comprises the following sub-steps: - decoding (E34) of a set of parameters (Odvk) representative of a neural network, called a prediction neural network (ARM1); - construction (E311) of a neighborhood vector (Cdvn) comprising said at least one decoded value (Vv'n) extracted from said at least one reference map (FMdv'i); - processing (E313) of said neighborhood vector (Cdvn) by the prediction neural network (ARM1), to provide a prediction of said current value (Vdvn).

5. Coding or decoding method according to one of the preceding claims, characterized in that said first and second groups (FMdv'i, FMdVi) contain the same number (M) of characteristic maps.

6. Encoding or decoding method according to one of the preceding claims, characterized in that said at least one reference map (FMdv'i) is at the same resolution as the current map (FMdv;).

7. Coding or decoding method according to one of the preceding claims, characterized in that said at least one reference image (v') is indicated in the binary stream.

8. Coding or decoding method according to one of the preceding claims, characterized in that said at least one reference image (v1) is predetermined.

9. Encoding or decoding method according to one of the preceding claims, characterized in that said at least one reference map is indicated in the bit stream.

10. Encoding or decoding method according to one of the preceding claims, characterized in that said at least one reference map is predetermined.

11. Coding method according to claim 2 or decoding method according to one of claims 4 to 10, characterized in that said at least one decoded value (Vdv'n) extracted from said at least one reference map is at the same position (xn,yn) in said at least one reference map (FMdv'i) as the current value (Vdvn) in the current characteristics map (FMdv;).

12. Coding method according to claim 2 or decoding method according to one of claims 4 to 11, characterized in that the neighborhood vector further comprises at least one second decoded value (Vdn') extracted from another reference map.

13. Device for coding a sequence of images (Sv), said device being configured to implement: - a construction step, comprising the following sub-steps for at least one current image (v, I(Pvn)) of the sequence: - construction (GEN, MAJ) of a first group of characteristic maps (FMv'j) representative of at least one image distinct from the current image (v1,1(Pv'n)), called the reference image; - construction (GEN, MAJ) of a second group of feature maps (FMvj) representative of the current image (v, I(Pvn)); - for at least one sample, called current sample (Pvn), of the current image, associated with a position (xn, yn) in the current image: - construction (XTR) of a characteristic vector (Zn) from said characteristic maps (FMv;) of said second group, as a function of said position (xn, yn) of said current sample (Pvn); - processing (MLP) of said characteristic vector (Zn) by an artificial neural network, called a synthetic neural network, defined by a set of parameters (Wvk), to provide a vector (P'vn) representative of a decoded value of said at least one current sample; - updating (MAJ) of at least one value of one of said characteristic maps of said second group and / or of at least one parameter of said network, as a function of a coding performance measurement, - a coding step (FMC, ARM) of said second group of characteristic maps (FMv;) comprising, for at least one value, called current value (Vvn), of one of said characteristic maps, called current map, an entropic coding of said value as a function of a neighborhood comprising at least one value (Vv'n) extracted from at least one map, called reference map, of the first group of characteristic maps, - a coding step (NNC) of said set of parameters (Wvk) of said synthetic neural network.

14. Device for decoding from a binary stream of a sequence of images (Sv), said device being configured to implement, for at least one current image (v, I(Pdvn)) of the sequence: - obtaining (FMD) at least one feature map (FMdv';), called reference map, from a first group of decoded feature maps representative of at least one image distinct from the current image (v1,1(Pdv'n)), called reference image. - decoding (FMD) of a second group of characteristic maps (FMdvi) representative of the current image comprising, for at least one value, called current value (Vdvn), of one of said characteristic maps, called current map, an entropic decoding (DE) of said value as a function of a neighborhood comprising at least one value decoded (Vdv'n) extracted from said at least one reference map; - decoding (NND) of a set of parameters (Wdvk) representative of a synthetic neural network (MLP'); - for at least one sample, called current sample (Pdvn), of the current image (v), associated with a position (xn, yn) in the current image: - construction (XTR») of a characteristic vector (Zdn) from the characteristic maps of said second group (FMdv;), as a function of said position (xn, yn) of said current sample, and: - processing (MLP1) of said vector (Zdn) by said synthesis neural network (MLP') to provide a vector representative of a decoded value of said at least one current sample (Pdvn).

15. A computer program comprising instructions for carrying out the steps of an encoding or decoding method according to claim 1 or 3 when said program is executed by a computer.

Images