Method and device for coding and decoding image sequences

Abstract

The invention relates to a method and device for coding and decoding an image sequence. The decoding method comprises the following steps for a current image: - obtaining at least one feature map of a first group of decoded feature maps representative of at least one image separate from the current image. - decoding a second group of feature maps representative of the current image comprising, for at least one value, entropy decoding according to a neighbourhood comprising at least one decoded value extracted from at least one reference map: - decoding an artificial neural network; and - for at least one sample of the current image: - constructing a feature vector from the feature maps of the second group, according to the position of the current sample; and - processing the vector by the artificial neural network in order to provide a decoded value of the sample.

Description

Method and device for coding and decoding image sequences.

[0001] The invention relates to the general field of coding and decoding digital image sequences. It relates more particularly to the compression of digital videos.

[0002] Digital videos are generally source-encoded to compress them in order to reduce the resources required for transmission and / or storage. There are many encoding standards, such as the ITU / MPEG standards (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) and their extensions (MVC, SVC, 3D-HEVC, etc.).

[0003] As video formats continue to evolve to compress more and more and adapt to the variety of expected formats and communication networks, the possibilities for prediction are becoming ever greater and traditional coding and decoding algorithms are becoming very complex.

[0004] In addition to these classic approaches proposed by compression standards (MPEG, ITU), approaches based on artificial intelligence, and in particular neural ones, tend to develop.

[0005] Some of these neural approaches can be seen as a simple extension of the competition notion of the aforementioned compression techniques, such as the prediction and transformation mode competition in video coding.

[0006] Other approaches use the concept of "autoencoder." Autoencoders are learning algorithms based on artificial neural networks, which allow the construction of a new representation of a dataset. The architecture of an autoencoder consists of two parts: the encoder and the decoder. The encoder consists of a set of layers of neurons, which process the data in order to construct new representations called "encoded," also called "latent representations." In turn, the layers of neurons in the decoder receive these representations and filter them in order to try to reconstruct the initial data. The differences between the reconstructed data and the initial data make it possible to measure the error made by the autoencoder. Training consists of modifying the parameters of the autoencoder in order to reduce the reconstruction error measured on the different samples of the dataset.The performance of such autoencoder-based systems comes at the cost of a considerable increase in memory footprint and complexity compared to conventional approaches as proposed by compression standards. They can have millions of parameters and may require on the order of a million MACs (multiply-accumulate) to decode a single pixel. This makes such decoders significantly more complex than conventional decoders, which could hamper the adoption of learning-based compression.

[0007] More recently, a simple neural network-based image encoding technique was described in the article "COmpression with Implicit Neural Representations" by Emilien Dupont et al. (arXiv:2103.03123). The proposed encoding technique consists of fitting a neural network to an image, quantizing the network weights, and transmitting them. At decoding time, the neural network is evaluated at each pixel position to reconstruct the image. However, such a technique remains inefficient in terms of compression and requires coding the video frames independently.

[0008] There is therefore a need for a solution that allows a sequence of images to be encoded / compressed in a simple and efficient manner.

[0009] The invention relates to a method for decoding a sequence of images comprising a plurality of samples to be coded comprising the following steps: - a construction step, comprising the following sub-steps for at least one current image of the sequence: - construction of a first group of characteristic maps representative of at least one image distinct from the current image, called the reference image; - construction of a second group of characteristic maps representative of the current image; - for at least one sample, called the current sample, of the current image, associated with a position in the current image: - construction of a characteristic vector from said characteristic maps of said second group, as a function of said position of said current sample;- processing said characteristic vector by an artificial neural network, called a synthetic neural network, defined by a set of parameters, to provide a vector representative of a decoded value of the current sample; - updating at least one value of one of said characteristic maps of said second group and / or at least one parameter of said network, as a function of a coding performance measurement, - a step of coding said second group of characteristic maps comprising, for at least one value, called the current value, of one of said characteristic maps, called the current map, an entropic coding of said value as a function of a neighborhood comprising at least one value extracted from at least one map, called the reference map, of the first group of characteristic maps, - a step of coding said set of parameters of said synthetic neural network.;

[0010] The invention also relates to a method of decoding from a binary stream of a sequence of images comprising a plurality of samples to be decoded, comprising the following steps for at least one current image of the sequence: - obtaining at least one characteristic map, called reference map, of a first group of decoded characteristic maps representative of at least one image distinct from the current image, called reference image.- decoding a second group of characteristic maps representative of the current image comprising, for at least one value, called current value, of one of said characteristic maps, called current map, an entropic decoding of said value as a function of a neighborhood comprising at least one decoded value extracted from said at least one reference map; - decoding a set of parameters representative of a synthetic neural network; - for at least one sample, called current sample, of the current image, associated with a position in the current image: - construction of a characteristic vector from the characteristic maps of said second group, as a function of said position of said current sample, and: - processing of said vector by said synthetic neural network to provide a vector representative of a decoded value of the current sample.

[0011] For the purposes of the invention, encoding, or "coding", means the operation which consists of representing a set of samples, or pixels, in a compact form carried for example by a digital binary train. Decoding means the operation which consists of processing a digital binary train to restore decoded samples.

[0012] By "image sequence" is meant a plurality of ordered two-dimensional images, for example temporally in the case of a video. According to another example, the images can be views of the same scene represented in multi-views. According to another example, the images can be a plurality of temporal and multi-view images (immersive video).

[0013] A "sample" is a value taken from an image in the sequence. Sampling a signal produces a series of discrete values ​​called samples. In the case of an image signal, the sample is called a pixel, which can be, for example, a grayscale pixel represented by a single component, or a color pixel traditionally represented by a triplet of values, for example (R,G,B) or (Y,U,V). The position of the sample is identified by its abscissa (x) and ordinate (y) coordinates in the image.

[0014] A "feature map" means a part of an abstract representation of an image, comprising a plurality of variable scalar data, also called characteristic values, for example real numbers, or integers, signed or unsigned. The abstract representation of an image may therefore comprise several feature maps. The abstract representation of an image may also correspond to a single feature map comprising a set of values ​​associated with a position.

[0015] By "feature vector constructed from feature maps as a function of a position" is meant a vector consisting of one or more elements, or data, preferably discrete, the data being constructed from the feature maps at a position determined by that of the samples being processed in the images. This feature vector is the one that is applied to the input of the synthetic neural network. In the case of an image, such a vector can be constituted for example from a plurality of values ​​taken from each of the feature maps at the same abscissa and ordinate coordinates as the samples to be coded (resp. to be decoded).

[0016] By "synthetic neural network" we mean a neural network such as a convolutional neural network, a multi-layer perceptron, an LSTM (for "Long Short Term Memory"), etc. The neural network is defined for example by a plurality of layers of artificial neurons and by a set of activation, weighting and addition functions (for example, a layer can calculate y = f(Ax+b), where y and b are vectors of dimension N, x is a vector of dimension M, A is a matrix of dimension MxN, and f is the activation function).

[0017] By "neural network parameter" we mean one of the values ​​that characterizes the neural network, for example a weight associated with one of the neurons (filter coefficient, weighting, bias, value affecting the operation of non-linearity, etc.)

[0018] By "processing by a synthetic neural network" is meant the application of a function expressed by a neural network to the input characteristic vector to produce an output vector representative of the sample to be encoded (resp. decoded). This output vector may include one or more data representative of the sample.

[0019] By "performance measurement" is meant a measurement between at least one value of a sample to be coded and a decoded value of said sample. The measurement can evaluate, for example, a distortion, or a perceptual error. It can be carried out on a sample or a plurality of samples (for example, the current samples, or the current images, etc.). The measurement can also include a measurement of the flow rate, in particular associated with the coding of the neural network and / or the coding of the feature maps of the second group. The measurement can be a joint measurement between the flow rate and the distortion through their weighting. As is well known in the state of the art, the value of this measurement is generally minimized until a target value is reached.

[0020] A "construction step" means a step that aims to construct the representative parameters of the image, before their actual coding. The construction sub-steps can be repeated as many times as necessary to obtain an acceptable performance measure.

[0021] Generally speaking, it is considered that the steps of an coding or decoding method should not be interpreted as being linked to a notion of temporal succession. In other words, the steps may be carried out in a different order than that indicated in the independent coding or decoding claim, or even in parallel.

[0022] The coding method according to the invention constructs the coding parameters, from a sequence of input images, by training a neural network on characteristic vectors associated with a position of a sample to be coded in the input image. These characteristic vectors are constructed from characteristic maps which may be at the resolution of the input images, or at a lower resolution. During the training, or construction, the parameters of the neural network and the values ​​of the characteristic maps are updated according to a performance measurement, for example of the rate-distortion type. When the training is finished, that is to say when the performance measurement obtained is satisfactory, the actual coding of the parameters of the neural network and the values ​​of the characteristic maps can be carried out and stored or transmitted to the decoder.

[0023] Advantageously, the training process makes it possible to refine the parameters of the synthetic neural network and / or the values ​​of the feature maps until an adequate representation in terms of performance is obtained, for example a desired balance between the generated bit rate and the distortion undergone by the input image being encoded. The training of the values ​​of the feature maps and the parameters of the synthetic neural network can be joint. Advantageously, the coding method according to the invention makes it possible to efficiently compress the signal.

[0024] Advantageously, the decoding method is simple since it suffices to decode the synthesis neural network and the characteristic maps of the current image as a function of other characteristic maps decoded from other images in the sequence, for example previous images, to reconstruct a decoded version of the current image.

[0025] Advantageously, the coding of the feature maps is made particularly efficient by taking into account a coding neighborhood in feature maps of other images in the sequence, which makes it possible to exploit the redundancies present in the maps associated with the different images.

[0026] Such a synthetic neural network can advantageously be of very simple structure with few parameters.

[0027] Moreover, decoding can be done sample by sample, in a progressive manner.

[0028] According to embodiments of the coding or decoding method:

[0029] - The coding method comprises the following sub-steps of coding said current value:- construction of a neighborhood vector comprising said at least one neighborhood value extracted from said at least one reference map, and;- processing of said neighborhood vector by an artificial neural network, called a prediction neural network, defined by a set of parameters, to provide a prediction of said current value;- updating of at least one parameter of said prediction network, as a function of the coding performance measurement,- coding of said set of parameters of the prediction network.

[0030] - The decoding method comprises the following sub-steps of decoding said current value, of one of said characteristic maps;- decoding a set of parameters representative of a neural network, called a prediction neural network;- construction of a neighborhood vector comprising said at least one decoded value extracted from said at least one reference map, and;- processing of said vector by the prediction neural network, to provide a prediction of said current value;

[0031] Advantageously, according to such a coding or decoding mode, the feature maps are efficiently compressed by a prediction neural network capable of predicting a value of the map as a function of its temporal neighborhood. By "neighborhood vector" is meant a vector consisting of one or more elements, or data, constructed from the feature maps of other images in the sequence, or reference maps, preferably extracted at a position close to that of the current sample, which is also that of the current value in the current feature map. The neighboring position may indicate a value in any map of the set of feature maps associated with another image in the sequence (for example, the neighboring value at the top left of that currently being processed in a feature map of the previous image).This neighborhood vector is the one that is applied to the input of the prediction neural network. By "prediction" we mean at least one data item that can estimate a current value of the feature map, for example a probability, a statistical value, etc. The prediction neural network, trained on the image maps of the sequence, is able to represent them efficiently. It is also inexpensive to code.

[0032] According to variants:

[0033] - at least one of said characteristic maps of the second group has a lower resolution than that of the image to be decoded and the method includes oversampling. Advantageously, according to this method, the compression of the characteristic maps is more efficient since at least one of the characteristic maps of the second group, to be coded (resp. decoded), has fewer values ​​than if it were at the resolution of the signal. For example, in the case of a digital image, one of the characteristic maps may be at 1 / 2 resolution, i.e. it has half as many values ​​on the abscissa and ordinate as the input image has samples, i.e. a total of 4 times fewer values ​​than a characteristic map at the resolution of the image. On the other hand, the characteristic map used for the synthesis may have a resolution identical to that of the image.The method therefore comprises in this case at least one oversampling operation to obtain the same number of values ​​in this map as the input image (resp. to be decoded) has samples.

[0034] – At least one of said characteristic maps of the second group has a resolution identical to that of the signal to be coded (resp. decoded). Advantageously, according to this mode, at least one of the characteristic maps, at the same resolution as the image to be coded (resp. decoded) allows significant fidelity and respect for the details of the initial resolution of the image signal.

[0035] - the construction of the characteristic vector includes a sub-step of extracting a value from a characteristic map at a position identical to that of the current sample in the image to be coded or decoded. Advantageously, it is possible to extract a value from a characteristic map at the same position as the sample in the image, to constitute an element of the characteristic vector. This process is simple to implement. For example, if we have J characteristic maps as input, with the same resolution as the image, a simple extraction of the values ​​of the maps at the coordinates of the current sample (at the same abscissa and the same coordinate in the characteristic map) makes it possible to directly construct the characteristic vector of J elements.

[0036] - The construction of said characteristic vector comprises the following sub-steps: - extraction of a plurality of values ​​from said characteristic maps as a function of said position of said current sample; - processing of said extracted values ​​to obtain the characteristic vector.

[0037] Advantageously, according to this mode, the characteristic vector is extracted from the characteristic maps, which may be of any kind, and in particular at a resolution lower than that of the image to be coded (resp. decoded), before undergoing processing. Such processing may, for example, correspond to a quantification of the extracted data, or to scaling, or to filtering, etc. During coding, quantification is essential for the proper functioning of the system if the characteristic maps comprise, for example, floating point or real values. It is necessary to quantify them before coding them and / or providing them as input to the synthesis and / or prediction neural network. During decoding, however, inverse quantification is not necessary, depending on the embodiments.

[0038] - said first and second groups contain the same number (M) of feature maps; advantageously according to this mode, the different images of the sequence are associated with the same number of feature maps, which can moreover be of the same structures. For example, the images of the sequence are associated with five feature maps having a hierarchy of decreasing resolutions. This facilitates the generation and extraction of the values ​​of the feature maps to produce the neighborhood vector.

[0039] - said at least one reference map is at the same resolution as the current map. Advantageously, according to this mode, the construction of the neighborhood vector is facilitated since the reference map is at the same resolution as the current map, it is very simple to extract a value at the same position or at a position close to that of the value of the current map.

[0040] - said at least one reference image is indicated in the bit stream. Advantageously, according to this mode, a simple indicator coded in the stream allows the decoder to directly access the reference image(s) to be used for decoding the current map of the current image, for example via an indicator of the reference image number.

[0041] - said at least one reference image is predetermined. Advantageously, according to this embodiment, the reference image(s) to be used for decoding the current map of the current image are predetermined. For example, the reference image is the previous image of the sequence. According to another example, the first image of the sequence is the reference image. According to another example, the first image of the sequence and the following image of the current image are two reference images.

[0042] - said at least one reference map is indicated in the bit stream. Advantageously, according to this mode, a simple indicator coded in the stream allows the decoder to directly access the reference map(s) to be used for decoding the current map, for example via an indicator of the number of the reference map in the group of feature maps associated with a reference image.

[0043] - said at least one reference map is predetermined. Advantageously, according to this embodiment, the feature map(s) to be used for decoding the current map are predetermined. For example, the reference feature map is the one which has the same order number as the current map in the groups of feature maps to which they respectively belong. According to another example, all the feature maps of the targeted reference image serve as reference maps for the current map.

[0044] - said at least one decoded value extracted from said at least one reference map is at the same position in said at least one reference map as the current value in the current feature map. Advantageously, according to this mode, once the reference map is known, the value which is at the same position as that being coded (resp. decoded) in the current map is used to compose the neighborhood vector. As these values ​​correspond to the same location, they have a high chance of being correlated, and thus, the extracted value provides relevant information on the current value.

[0045] - the neighborhood vector further comprises at least a second decoded value (Vd n ») extracted from another reference map. Advantageously, according to this mode, at least a second value can be extracted from another reference map. This makes it possible to take into account values ​​at different time instants (for example, at two previous instants, or, depending on the order of coding / decoding of the images, at a future instant and a past instant) and thus to better take into account the evolution of the signal for the coding / decoding of the current value.

[0046] Correlatively, the invention also relates to a coding device and a decoding device.

[0047] The characteristics and advantages of the coding or decoding method apply in the same way to the coding or decoding device according to the invention and vice versa.

[0048] The invention also relates to a computer program on a recording medium, this program being capable of being implemented in a computer or an encoding or decoding device in accordance with the invention. This program comprises instructions adapted to the implementation of the corresponding method. This program can use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0049] The invention also relates to a computer-readable information carrier or recording medium comprising computer program instructions mentioned above. The information or recording media may be any entity or device capable of storing the programs. For example, the media may comprise a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a floppy disk or a hard disk, a DNA sequence, or a flash memory. Furthermore, the information or recording media may be transmissible media such as an electrical or optical signal, which may be conveyed via an electrical or optical cable, by radio link, by wireless optical link or by other means.

[0050] The program according to the invention can in particular be downloaded from an Internet-type network.

[0051] Alternatively, each information or recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of a method according to the invention.

[0052] Other characteristics and advantages of the present invention will emerge from the description given below, with reference to the appended drawings which illustrate exemplary embodiments thereof which are not limiting in nature.

[0053] schematically represents a coding device used in the context of the invention;

[0054] schematically represents a decoding device used in the context of the invention;

[0055] illustrates an example of a synthetic artificial neural network used in the context of the invention;

[0056] Illustrates an example of an artificial prediction neural network used in the context of the invention;

[0057] is a flowchart representing an example of a coding method that can be implemented by the coding device of the;

[0058] is a flowchart representing an example of a decoding method that can be implemented by the decoding device of the;

[0059] illustrates a decoding method used in one embodiment of the invention.

[0060] is a flowchart representing a method of coding feature maps that can be implemented by the coding device of the and by the coding method of the.

[0061] is a flowchart representing a method of decoding feature maps that can be implemented by the decoding device of the and by the decoding method of the.

[0062] illustrates a method of encoding or decoding feature maps.

[0063] illustrates another method of encoding or decoding feature maps.

[0064] It schematically represents a device for coding digital image sequences ENC.

[0065] This ENC encoding device includes as many ENC encoding sub-modules v than images to be coded. A sub-module includes, for an image v of the sequence to be coded denoted Iv or I(P Vn), called current image, a GEN module for generating feature maps, a SE transformation module, an XTR data extraction module, an MLP module corresponding to a synthetic artificial neural network, an NNC neural network coding module capable of coding the synthetic neural network and optionally a prediction neural network, an FMC module for coding and decoding feature maps, an EVAL module for evaluating coding performance, an MAJ update module.

[0066] The ENC coding device may be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the above-mentioned modules may then be realized by the cooperation of the processor and computer program instructions stored in the above-mentioned memory and designed to perform the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0067] The ENC coding device receives as input a sequence of at least two images to be coded, denoted I(P Vn ), each comprising respectively a plurality of samples P Vn . The index v indicates the image number in the sequence S V of V images. The index n indicates the pixel number in an image of N pixels. The image I(P Vn) may represent a two-dimensional image and the sequence a plurality of two-dimensional images (video sequence, stereoscopic, multiscopic components, series of medical images, etc.). In one embodiment, an image of the sequence is represented using at least one two-dimensional representation, such as a matrix of pixels, each pixel comprising a vector of red (R), green (G), blue (B) components, or, alternatively, a luminance component and at least one chrominance component. The location of each pixel is defined by its abscissa and ordinate coordinates (x n and there n ) in the image. In one embodiment, the sequence is a sequence of grayscale images represented using a two-dimensional representation, such as a pixel matrix, each pixel having a grayscale, or luminance, component. In this case the vector representing the pixel is reduced to a single component.

[0068] As will be described in more detail later with reference to Figures 3 to 11:

[0069] The GEN feature map generation module is configured to generate a plurality of M feature maps denoted FMv i for the current image v. An optional module, not shown, can generate one or more additional maps, L in number, which will not be coded or transmitted.

[0070] In one embodiment, the SE module performs a transformation of the FMv feature map group i to generate a group of FMSv feature maps i at the same resolution as the input signal.

[0071] The optional SE module can perform a quantification of the data extracted from this set of M FMv maps i. It is recalled that quantizing a value refers to matching that value to a member of a discrete set of possible code symbols. For example, the set of possible code symbols may consist of integer values, and the quantization system performs a simple rounding of a real value to an integer value. According to another example, the quantization consists of multiplying by a given value and then rounding. Then the SE module may perform a transformation of the values ​​of at least one of the feature maps, for example, upsampling, interpolation, filtering, etc. At the end of the transformation, a transformed feature map is, according to one embodiment, of the same resolution as the images of the input sequence.Advantageously, according to this mode, the characteristic maps which are coded can be of lower resolution than that of the images to be coded, while the transformed maps, which are used for the construction of the characteristic vectors, are at the same resolution as the sequence of images, which facilitates the extraction of the values.

[0072] In one embodiment, the SE module is absent, in this case the values ​​which will be used to construct the characteristic vector are extracted directly from the second group of characteristic maps.

[0073] The XTR module performs value extraction in FMv feature maps i (and / or FMSv i, according to one of the embodiments described previously), for a current sample Pv n to be encoded, depending on its coordinates in the input signal. For example, if we want to encode the sample Pv n at the coordinates (x n , y n) of an input image, the XTR module performs an extraction of values ​​in maps at positions imposed by the coordinates (x n , y n ).

[0074] In one embodiment, the extracted values ​​constitute the vector Z n . Z n is a J-tuple, that is, it has J elements, or data z i . The vector Z n of index n refers to the characteristic vector of the pixel Pv n .

[0075] The XTR module can also perform quantization of the data extracted from the feature map set. In particular, if the SE module is not used and the feature maps are at lower resolutions than the sequence images, the XTR module can take into account the coordinates of the values ​​in the lower resolution maps.

[0076] Note that at least one of the SE or XTR modules must perform a quantization of the values ​​extracted from the feature maps of the current image.

[0077] The MLP module is a synthetic neural network defined by K parameters Wv k , able to process the vector Z n , or J-Uplet, as input, to generate as output a vector representative of the sample Pv n to encode the current image. The synthesis neural network is, according to one embodiment, an MLP, or Multi Layer Perceptron, consisting of an input layer adapted to the input format (the J-uplet), optionally one or more hidden layer(s), and an output layer adapted to the output format of the output vector, generally a vector comprising NC elements corresponding to the NC components of the image to be decoded. According to one embodiment, NC is equal to 3 and the output vector is the triplet (R, G, B) of the pixel P'v nencoded then decoded from the current image.

[0078] The NNC module performs the coding of the neural network for synthesizing the current image, in particular its Wv parameters. k. Optionally, the NNC module performs the coding of the prediction neural network, ARM, in particular its Ov parameters b.Durant le processus d’entraînement, ou de construction, du codage, c’est-à-dire tant que l’étape d’évaluation d’une performance n’est pas satisfaisante, le module NNC réalise une simulation de codage, suivi d’un décodage, à destination du module d’évaluation. Par la suite, il effectue le codage effectif des paramètres du réseau de neurones de synthèse Wvk,et optionnellementle codage du réseau de neurones de prédiction, ARM. Les paramètres codés de l’image courante sont notés Wcvk(et Ocvb). De manière connue, la simulation de codage peut être identique au codage effectif, ou en réaliser une approximation.

[0079] The FMC module performs the coding of FMv cards i , that is to say the values ​​of the feature maps of the current image, called the second group (excluding the maps optionally resulting from oversampling by the SE module, and the additional maps). During the process of training, or construction, of the coding, that is to say as long as the stage of evaluation of a performance is not satisfactory, the FMC module carries out a coding simulation, followed by a decoding, intended for the evaluation module. Subsequently, it carries out the effective coding of the values ​​of the FMv maps i. The coded maps of the current image are denoted FMcv i. As is known, the coding simulation can be identical to the actual coding, or approximate it. The coding module quantizes, if necessary, the latent representation of the values ​​of the maps of the second group by using a quantizer to generate an ordered collection of quantized values. Then the coding module compresses the quantized data, using a coding taking into account the temporal neighborhood of a value to be coded from the feature map.To construct the neighborhood, the FMC module uses coded and decoded feature maps corresponding to images v' of the sequence distinct from the current image v, previously coded then decoded, for example those of the previous image in the sequence, or of two past reference images of the sequence, or of a past and a future reference image, to the extent that the decoding order of the sequence allows it (as is known, the decoding order can be different from the chronological order of the sequence), etc. They are called feature maps of the first group, or reference maps, denoted FMdv'. i . As will be detailed later, the FMC module can include a prediction neural network (ARM) of the current image.

[0080] The EVAL module performs an evaluation and minimization of a coding performance. The evaluation function is for example of the rate-distortion type. The minimization can be carried out by gradient descent, or any other method within the reach of a person skilled in the art.

[0081] The MAJ module updates the values ​​of the FMv maps i to encode from the current image, depending on the results of the performance function.

[0082] It schematically represents a DEC decoding device.

[0083] The DEC encoding device of thecomprises as many DEC decoding sub-modules v than images to decode in the sequence. A sub-module receives as input, for a current image v to decode noted I(Pdv n ) :- encoded data organized into M FMcv feature maps i (also called second group feature maps, or current maps); - the Wvc encoded parametersk of the synthesis neural network corresponding to the current image; - optionally the encoded Ocv parameters b of the prediction neural network corresponding to the current image; - data organized into FMdv' feature maps i corresponding to images v' of the sequence distinct from the current image v, previously decoded, for example those of the previous image in the sequence, or of two reference images of the sequence, etc. They are called feature maps of the first group, or reference maps.

[0084] The DEC submodule v decodes the N Pd samples vn of image v of the sequence. The set of sub-modules decodes the V images of the sequence to be decoded.

[0085] This DEC decoding device vcomprises, for a current image, a neural network(s) decoding module NND capable of decoding the MLP' synthesis neural network and optionally the ARM' prediction neural network, a FMD module for decoding feature maps, a XTR' data extraction module, an SE' inverse transformation module, an MLP' module corresponding to a synthesis neural network. According to one embodiment, it produces as output a current decoded image, denoted I (Pdv n ), comprising a plurality of decoded samples Pdv n.

[0086] The cards decoded by the FMD module, numbering M, are noted FMdv i (feature maps of the current image v). The parameters of the synthesis neural network (MLP') decoded by the NND module are denoted Wdv k. Les paramètres du réseau de neurones de prédiction (ARM ») optionnellement décodés par le module NND sont notés Odvb. Le module FMD utilise pour décoder une carte courante FMdviune ou plusieurs cartesde caractéristiques, dites cartes de référence, d’images v' de la séquence distinctes de l’image courante, dites images de référence, décodées précédemment par un autre sous-module de décodage, et notées FMdv’i.

[0087] The NND module corresponds to the NNC module. It can use, for example, the neural network coding standard proposed by the MPEG-7 part 17 standard, also called NNR (Neural Network Representation). According to one embodiment, the NND module decodes the neural network associated with the image v using the previously decoded parameters of the other neural networks in the sequence.

[0088] The decoder can also generate one or more additional maps, not shown.

[0089] In one embodiment, the SE' module performs a transformation of the first group of FMdv feature maps i decoded to generate a group of transformed feature maps, denoted FMS'v i. The SE' module optionally performs an inverse quantization corresponding to the quantization performed at the encoder. The inverse quantization is not necessary if the encoder's quantizer Q has performed a simple rounding of the real values ​​submitted to it. The inverse quantization is also not necessary if the neural network is able to take into account a quantization of its input data. Otherwise, the decoder performs the inverse operation of the quantizer Q. Then the SE' module optionally performs a transformation of the values ​​of the feature maps, including for example oversampling, interpolation, filtering, etc., similar to that performed by the encoder. At the end of the transformation, a transformed feature map FMS'v i is preferably of the same resolution as the images of the sequence to be decoded.

[0090] In one embodiment, the SE' module is absent, in this case the values ​​which will be used to construct the characteristic vector are extracted from the second group of characteristic maps.

[0091] In one embodiment, the SE' module uses an oversampling neural network.

[0092] The XTR' module is identical to the XTR module of the. It performs an extraction of values ​​from the M FMdv characteristic maps i of the current image (and / or FMS'v i according to one of the embodiments described previously), for a Pdv sample n to decode, depending on its coordinates in the image to be decoded.

[0093] In one embodiment, the extracted values ​​constitute the vector Zd n . Zd n is a J-tuple, that is, it has J elements, or data zd i. In one embodiment, J=M.

[0094] In one embodiment, the XTR' module also performs processing of the extracted values, to generate the vector Zd n . The XTR' module can perform inverse quantization of the data extracted from the feature map set, especially if the SE' module is absent. The processing may include other operations, such as filtering, scaling, etc., similar to those performed by the encoder.

[0095] The MLP' module is a so-called synthetic neural network, defined by K parameters Wdv k , capable of processing the vector Zd n , or J-Uplet, as input, to generate as output a vector representative of the sample Pv n to be decoded, generally a vector comprising NC elements corresponding to the NC components of the image to be decoded. According to one embodiment, K=3 and the output vector is the triplet (R, G, B) of the pixel Pdv ndecoded. The MLP' module is identical in structure to the MLP module, and its parameters are either identical if the coding of its parameters Wv k is lossless, or different if the coding is done with losses.

[0096] When all Pv samples n of the signal have been decoded, we have a reconstructed signal I (Pdv n), selon un exemple une image Iv comportant N échantillons décodés sous la forme de N vecteurs Pdvn.

[0097] The decoding device DEC can be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the modules mentioned above can then be realized by the cooperation of the processor and computer program instructions stored in the aforementioned memory and designed to carry out the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor

[0098] Illustrates an example of a synthetic artificial neural network used for encoding and decoding in embodiments of the invention.

[0099] The synthetic artificial neural network used for coding, MLP, and the synthetic artificial neural network used for decoding, MLP, are defined by an identical structure, comprising for example a plurality of layers of artificial neurons, and by a set of weights and activation functions associated respectively with the artificial neurons of the network concerned.

[0100] A vector representation of a current sample (a vector Z n or Zd n from FMv feature maps i / FMSv i to the encoder or FMdv i / FMSdv ito the decoder) is applied as input (i.e. on an input layer) of the MLP or MLP artificial synthesis neural network. The artificial synthesis neural network produces as output a vector, according to one embodiment the color components (R, G, B) constituting a color pixel of an image.

[0101] The concatenation of all these reconstructed pixels in an image (2D, 3D) constitutes the decoded, or reconstructed, image.

[0102] At the encoder, the MLP synthesis artificial neural network is trained on the image, so as to minimize the differences between the input representation of the current image I(Pv n ) and its representation I(P'v n ) output, while also minimizing the amount of data to be encoded. The EVAL module performs a performance measurement in this regard.

[0103] Once the encoder training is complete, the network parameters are encoded, either losslessly, in which case the MLP' neural network is identical to MLP, or lossily, in which case the MLP' network may be slightly different from MLP.

[0104] Illustrates an example of a prediction artificial neural network used in encoding (ARM) and decoding (ARM") for feature maps, in accordance with embodiments of the invention.

[0105] The artificial neural network for prediction used in coding, ARM, and the artificial neural network for prediction used in decoding, ARM, are defined by an identical structure, comprising for example a plurality of layers of artificial neurons, and by a set of weights and activation functions associated respectively with the artificial neurons of the network concerned.

[0106] A vector representation of a current neighborhood (a vector Cv nor Cdv n from FMdv feature maps i or FMdv' i of another image of the sequence, to the encoder or to the decoder) is given as input (i.e. on an input layer) of the artificial neural network of prediction ARM (to the encoding) or ARM » (to the decoding).

[0107] The prediction artificial neural network behaves as a function that produces as output a prediction of the current value of the feature map being processed, which can be presented as a prediction value or probability data.

[0108] According to one embodiment, at the encoder, the network implements a function f ѱ which provides an expected mean and / or variance (µ,σ) for the current value of the current value Vv n of the FMv card ito be encoded. These statistical values ​​are used to perform the entropy coding of this value. For example, if the function produces a mean, this mean is subtracted from the current value and only the difference is entropy encoded, this mean being considered as a prediction of the current value. Alternatively, if the function produces a mean and a variance, the mean is subtracted from the current value, and the difference is entropy encoded using an entropy coding adapted to the variance, for example by quantizing said variance into a predetermined set of variances and associating a type of entropy coding with each quantized variance value. At the decoder, the network implements a function f ѱ which provides an expected mean and / or variance for the current value of the current value Vdv n of the FMdv card ito be decoded. These statistical values ​​are used to perform the entropy decoding of this value. For example, if the function produces a mean, the current value is decoded by the entropy decoder and this mean is added to the current value. Alternatively, if the function produces a mean and a variance, the current value is decoded by the decoder using entropy decoding adapted to the variance, for example by quantizing said variance into a predetermined set of variances and associating an entropy decoding type with each quantized variance value.

[0109] In another embodiment, the neural network can output the expected probability (pr) of each possible value of the current sample. In this case, the entropy coding or decoding will be adapted to this probability (as is known for Huffman or arithmetic entropy coding).

[0110] At the encoder, the ARM prediction artificial neural network is trained on the current image and the reference images, so as to minimize the amount of data to be encoded. The EVAL module performs a performance measurement in this sense. We recall that the overall performance measurement concerns a minimization of the distortion between encoded and then decoded images I (P'v n ) and the input images I (Pv n ), while minimizing the coding rate. In one embodiment, the feature maps are losslessly encoded, using entropy coding. In this case, the coding of the feature maps affects the rate, but not the distortion of the coded images. In another embodiment, if the feature maps are lossy encoded, the coding of the feature maps affects the rate, but also the distortion.

[0111] Once training is complete, the B Ocv parameters bof the network are encoded, either lossless, in which case the ARM' neural network is identical to ARM, or lossy, in which case the ARM' network may be slightly different from ARM.

[0112] This is a flowchart representing an example of a coding process that can be implemented by the coding device of the.

[0113] According to this embodiment, the input signal is a two-dimensional image of the image sequence, each sample to be coded is therefore a pixel Pv n of coordinates (x n , y n ) of the current image v.

[0114] Encoding takes place in two main phases:

[0115] In a first phase, called the construction phase, learning is carried out in order to determine, for an input image I(Pv n ), the values ​​of the FMv cards i and Wv parameters k and optionally Ov bto optimize a global cost function. Learning is for example carried out by gradient descent, followed by an update of the parameters of the MLP synthesis neural network, the values ​​of the FMv feature maps i , and optionally the ARM prediction neural network. As known in the state of the art, the cost function can be of the rate-distortion, or rate, or distortion, or perceptual type. To measure the rate R, it is necessary to simulate the coding of the FMv maps i , then measure the associated coding rate (the size of the stream B1). According to one embodiment, the coding of the parameters Wv is not simulated k and / or Ov b because their influence is less important than that of the feature maps. According to one embodiment, the coding of the Wv parameters is also simulated k and / or Ov band the associated flow rate (the size of the flow B2) is measured. To measure the distortion D, it is necessary to simulate the coding and then the decoding of at least part of the image Iv, to obtain at least one pixel P'v n resulting from a simulation of coding then decoding, then measuring the difference between this part of the image I (Pv n ) as input and a corresponding part of image I (P'v n ) encoded then decoded.

[0116] Then during a second phase, called the coding phase, the FMv cards i and the Wv parameters k and possibly Ov b are encoded to produce the FMcv encoded values i and WCV k (and possibly Ocv b ) before transmission or storage. They constitute the compressed representation of the image (Pv n ).

[0117] We will now describe the steps of a method according to one embodiment of the invention.

[0118] During a step E20, an image I(Pv n ) to be coded with index v in a sequence of images, comprising a plurality of N samples Pv n , and a number NC of components, is provided as input to the process.

[0119] During a step E21, the M FMv cards i of the second group, or feature maps of the current image v, are initialized. Subsequently, the values ​​of the FMv maps i , and possibly the parameters of the neural networks must be optimized during the construction phase.

[0120] According to one embodiment, the FMv cards i are of the same resolution as the input image I(Pv n ) and therefore each have the same number of values ​​N as there are samples Pv n to be coded. According to one embodiment, the FMv cards i are of resolution less than or equal to that of the input signal I(Pv n) and therefore comprise, for at least one of them, a number N' of values ​​to be coded less than N. According to a variant, the first FMv1 card is at the resolution of the images and each following card is at a resolution half that of the previous one. According to one embodiment, several FMv cards i are of the same resolution, lower than that of the input signal I(Pv n ).

[0121] According to one embodiment, the FMv cards i are transformed to provide a group of FMSv transformed feature maps i . In this mode, the feature vectors are preferably extracted from the transformed maps, and not directly from the maps of the second group. In this mode, the feature vectors are therefore extracted indirectly from the maps of the second group. The transformed maps are not coded, they are only used for the construction of the feature vectors.

[0122] According to one embodiment, the FMv cards i of the current image are initialized by predefined constant values. According to another embodiment, the feature maps are initialized by a set of random real values. According to one embodiment, one or more maps, forming an additional group of additional feature maps, are generated, and added to the second group. They are used to construct the feature vector but are neither stored nor transmitted.

[0123] FMv Feature Maps i of the second group are subsequently updated, or refined, during a step E22, by the MAJ update module of the encoder during its learning.

[0124] During a step E23, the FMv cards iof the second group are coded by the FMC module of the encoder. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is an actual coding and the coded values ​​constitute the stream B1. The simulation can be identical to the actual coding but it can also be different (for example, simplified). For this coding, a technique is used for predicting a feature map value by its temporal neighborhood, that is to say from feature maps of a first group, consisting of feature maps of other images in the sequence than the current image, called reference maps of reference images, as will be described in more detail in support of the. In one embodiment, the structure and parameters of a synthetic neural network (optionally prediction) are initialized for example during the first iteration of this step.These parameters are subsequently updated, or refined, during the construction phase, during subsequent iterations of the process.

[0125] In one embodiment, the FMv cards i are encoded in order (FMv1, FMv2,…, FMv4), and the variables of each map in a predefined order, for example lexicographic. Each map undergoes entropy coding. Entropy coding produces a compressed stream B1 whose throughput can be measured later during a step E29.

[0126] During a step E24, according to one embodiment, the M cards of the first group FM i are transformed by the SE module to generate FMSv transformed maps i to the resolution of the images in the input sequence.

[0127] According to one embodiment, M FMSv cards i are generated. According to one embodiment, each FMv card i is transformed into an FMSv map i. According to one embodiment, at least one FMv card i is of lower resolution than the images in the sequence to be encoded and the transformation operation involves oversampling so that the transformed FMSvi map has the same number of samples as the current image. Oversampling consists of adding values ​​to the FMSv maps i to achieve the resolution of the images in the input sequence. It can be simple (by nearest neighbor replication) or include interpolation (linear, polynomial, filtering, etc.). It can also be achieved by an oversampling neural network.

[0128] During step E25, values ​​are extracted by the XTR module from the FMv cards i or possibly FMSv i transformed, and optionally additional cards .Cetteextraction est réalisée en fonction des coordonnées (xn, yn) de l’échantillon Pvndu signal d’entrée. Elle peut aussi être réalisée en fonction de la résolution de la carte considérée.

[0129] According to one embodiment, the vector Z ncharacteristic results directly from this extraction.

[0130] The samples to be coded are, for example, processed in sequential order, from n=1 to n=N.

[0131] According to one embodiment, during step E25, the vector Z n characteristic is constructed by the XTR module from the values ​​extracted from the FMv maps i or FMSv i for each sample Pv n of coordinates (x n , y n ) of the input image. The processing may include quantization of the values ​​extracted from the FMv maps i or the vector Z n constituted, if necessary. The processing may include other operations, such as filtering, scaling, applying any function, preferably monotonic, etc.

[0132] In one embodiment, Z n has as many values ​​as FMvi or FMSv cards i as input. In this case we have J=M.

[0133] In one embodiment, Z n is a J-tuple (z1, z2,…, z J ), consisting of the values ​​of the FMv cards i or FMSv i located at the coordinates (x n , y n ) of the current pixel Pv n ,as will be illustrated in support of the.

[0134] In one embodiment, Z n is a J-tuple constructed from values ​​taken from FMvi maps at coordinates that may be different depending on the maps. For example, if FMv maps i are at different resolutions because they have been downsampled, the coordinates are adapted (by scaling) to match the resolution of each map.

[0135] In one embodiment, Z n is a J-tuple constructed from values ​​taken from the FMv maps iby applying the processing to one or more values ​​of the maps, for example filtering the values ​​close to the targeted value in a map. For example, for a current sample Pv n in an FMvi map which is at the same resolution as the input signal, we can extract the values ​​located at the coordinates (x n , y n ), (x n -1, y n ), (x n , y n -1) and (x n -1, y n -1) and apply processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (z i ) of element i of vector Z n relating to this FMv card i . According to another example, in an FMv card i which is at half the resolution of the input signal, we can consider the values ​​located at the coordinates (x n / 2, y n / 2), (x n / 2-1, y n / 2), (x n / 2, y n / 2-1) and (x n / 2-1, y n / 2-1) and apply processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (z i ) of element i of vector Z n relating to this FMv card i .

[0136] At step E26, the vector Z n is processed by the MLP synthesis neural network to generate as output a vector representative of the sample Pv n to code, according to one embodiment, the triplet (R, G, B) of the sample P'v n (the Pv sample n coded then decoded).

[0137] The Wv structure and parameters kof the synthesis neural network (and optionally those of the synthesis neural network, and those of the upsampling neural network if it exists) are initialized for example during the first iteration of this step. These parameters are subsequently updated, or refined, during the construction phase, during subsequent iterations of the process.

[0138] According to one embodiment, the parameters of the synthetic neural network and / or the prediction neural network are initialized by predefined values ​​known to give a satisfactory result (for example, following training on a corpus of images). According to another embodiment, the parameters of the synthetic neural network and / or the prediction neural network are initialized by a set of random values.

[0139] During a step E27, the parameters Wv k of the MLP synthesis neural network and the Ov parametersb of the ARM prediction neural network, if any, are quantized and encoded. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is an actual coding and the encoded values ​​constitute the B2 stream. The simulation can be identical to the actual coding but it can also be different (for example, simplified). Any known technique can be used for this purpose, for example the neural network coding standard proposed by the MPEG-7 part 17 standard, also called NNR (Neural Network Representation). Note that in this case, it is necessary to choose the amount of degradation that the coding brings to the weights Wv k and optionally Ov b .

[0140] In step E28, a performance measure is evaluated.

[0141] For this purpose, the coding simulation rates associated with the feature maps of the first group (simulation of the B1 flow by coding the FMv maps i ) and optionally to the parameters of the neural network(s) (simulation of the B2 flow by coding the Wv parameters k and optionally Ov b ) are measured.

[0142] According to one embodiment, the cost function is of the rate-distortion type, denoted (D+L*R), where D for example the squared error measured between the input signal and the decoded signal (or the error measured on a subset of samples of the signal). According to another example, D is calculated from a perceptual function such as the SSIM (for Structural SIMilarity), or MSSSIM (for Multi-scale Structural SIMilarity). According to one embodiment, R is the simulated rate of the stream B1; according to another embodiment, R is the total rate used to code this image, that is to say the sum of the simulated rates of B1 and B2. L is a parameter which adjusts the rate-distortion trade-off. Other cost functions are possible.

[0143] As long as the cost function has not reached its minimum, the performance measurement is not satisfactory, and the method is repeated from step E22. Alternatively, the method can be interrupted after a predefined time or a predefined number of iterations, so as to control its complexity or duration. This minimization can be carried out by a mechanism known as gradient descent with updating of the parameters during step E22 for the values ​​of the feature maps and E26, E27 for the parameters of the network(s).

[0144] At step E29, if the cost function has reached its minimum, training stops. If a coded version corresponding to the last simulation of the parameters of the synthetic neural network (Wv k ) and feature maps (FMv i) is available, streams B1 and B2 can be made up of them. According to another embodiment, the effective coding of the updated parameters of the synthesis neural network (Wv k ) and values ​​of the characteristic maps (FMv i ) and optionally the prediction neural network (Ov b ) is performed at this step to produce the Wcv encoded parameters k (optionally Ov b ) and FMcv iqui constituent des flux B1 etB2. Les flux B1 et B2 peuvent être concaténés pour produire un flux final. Selon un mode de réalisation, le flux B2 des paramètres codés du (des) réseau(x) de neurones est (sont) stocké(s) ou transmis avant le flux B1, afin de pouvoir être décodé(s) avant le flux B1.

[0145] This is a flowchart representing an example of a decoding method that can be implemented by the decoding device of the.

[0146] In a step E30, the streams B1 and B2 are extracted from the encoded stream. They contain respectively, for a current image v of the sequence, the coded representations of the maps of the second group FMcv i and Wcv parameters k , optionally Ocv parameters b ,images from the sequence.

[0147] During a step E31, the M FMdv cards i of a current image v of the sequence are generated by decoding the FMcv values i of the current image based on the values ​​of already decoded feature maps FMdv' i for other images in the sequence, distinct from the current image. For example, if the current image is image 3 (v=3), the feature maps of one or more previous images (v'= 1 and / or v'=2) can be used for decoding. According to another example, the feature maps of one or more previous images (v'= 1 and / or v'=2) and of one or more future images in the sequence (v'= 4,....) can be used for decoding, provided that these have already been decoded. Naturally, if the image is the first in the sequence, only the current maps FMd 1iare available. In this specific case, decoding is not a function of previous image maps, but simply of values ​​present in the map associated with the current image.

[0148] For decoding, a technique for predicting a feature map value by its temporal neighborhood is used, as will be described for example in support of the. In one embodiment, the FMdv maps i are decoded in order (FMdv1, FMdv2,… FMdv4), and the values ​​of each card in a predefined order, for example lexicographic.

[0149] According to embodiments as described for the encoder:

[0150] - FMd cards vi are of the same resolution as the signal I(Pdv n ) to be reconstituted, that is to say that they contain N values.

[0151] - FMd cards vi are at a resolution lower than or equal to that of the signal I(Pdv n ) to be reconstituted.

[0152] - Several FMd cards vi are of the same resolution, lower than the signal resolution.

[0153] – The already decoded reference v' frames, whose feature maps are used to decode the current feature map, are indicated in the B1 bitstream, for example in the form of an index indicating the frame number in the sequence.

[0154] - The reference v' frames, already decoded, whose feature maps are used to decode the current feature map are at predefined positions. For example, this can be the previous frame in the sequence, the next one (provided it is already decoded), the first one, etc.

[0155] - The feature maps used to decode the current feature map are indicated in the B1 bitstream, for example as an index indicating the position of the feature map (between 1 and M) in the feature map group associated with a reference image.

[0156] - The feature maps used to decode the current feature map are at predefined positions. For example, this may be the feature map that is at the same position as the current feature map in the group of feature maps associated with a reference image, or the previous one, or the previous and the next one, etc.

[0157] – the feature maps of the second group, associated with the current image, are also used for decoding the current feature map. For example, a spatial neighborhood of the value of the current feature map can be extracted into the current feature map (causal neighborhood) or into an already decoded feature map of the same (second) group.

[0158] According to one embodiment, one or more cards , forming an additional group of additional feature maps, are generated and complement the second group. They are not decoded but generated by the decoder in a manner identical to the encoder generation. They typically include data that can assist the MLP network in the task of signal reconstruction.

[0159] During a step E32, according to one embodiment, the M cards of the second group FMdv iare transformed by the SE' module to generate FMS'v transformed maps i to the resolution of the input images. This step is similar to step E24 which was described for the encoder in support of the, and the embodiments apply. In particular:

[0160] According to one embodiment, M FMS'v cards i are generated.

[0161] According to one embodiment, each FMd card vi is transformed into an FMS'v card i .

[0162] According to one embodiment, at least one FMd card vi is of lower resolution than the images of the image to be encoded and the transformation operation involves oversampling so that the FMS'v card i transform has the same number of samples as the input image. Oversampling consists of adding values ​​to the FMS'v maps ito achieve the resolution of the input image. It can be simple (by nearest neighbor replication) or include interpolation (linear, polynomial, filtering, etc.). It can also be achieved by an upsampling neural network.

[0163] The transformation can optionally include inverse quantization of the extracted values, if necessary. However, inverse quantization is not mandatory.

[0164] During a step E33, values ​​are extracted by the XTR' module from the FMdv cards i or possibly FMS'v i transformed . This extraction is carried out according to the coordinates (x n, yn) d’un échantillon Pvnde l’image courante. Elle peut aussi être réalisée en fonction de la résolution de la carte considérée. Cette étape est similaire à l’étape E25 qui a été décrite pour le codeur à l’appui de la, et les modes de réalisation s’appliquent. Notamment :

[0165] According to one embodiment, the vector Zd n characteristic results directly from this extraction.

[0166] In one embodiment, Zd n is a J-tuple (z1, z2,…, z J ), consisting of the values ​​of the FMdv cardsi or FMS'v i located at the coordinates (x n , y n ) of a current pixel Pdv n ,as will be illustrated in support of the.

[0167] The samples to be decoded are for example processed in sequential order, from n=1 to n=N.

[0168] According to one embodiment, during a step E33, a vector Zd n is constructed by the XTR' module from the values ​​extracted from the FMdv maps i and FMdv'i of the second group or FMS'v cards i transformed, and optionally maps of an additional group, for each Pdv sample n of coordinates (x n , y n ) to decode, based on the coordinates (x n , y n ). This step is similar to step E25 which was described for the encoder in support of the and the described embodiments apply. The extraction may include an inverse quantization of the extracted values ​​or of the vector Zdn constituted, if necessary.

[0169] During a step E34, the Wdv parameters k of the MLP synthesis neural network and possibly the Odv parameters b of the prediction neural network (and optionally the parameters of an oversampling network of the SE module) are generated by decoding the Wcv values k and Ocv bof the B2 stream. Any known decoding technique corresponding to the coding technique used by the encoder can be used for this purpose. The MLP synthesis neural network is similar to the MLP synthesis network, i.e. it has the same structure and the same parameters, except for the coding, which can be done with or without losses. Similarly, the ARM prediction neural network, if used to decode the feature maps, is similar to the ARM prediction network, i.e. it has the same structure and the same parameters, except for the coding, which can be done with or without losses.

[0170] According to one embodiment, the stream B2 is decoded before the stream B1, in order to be able to have the synthesis neural network MLP' and possibly the prediction neural network ARM' before starting to decode the samples.

[0171] During a step E35, the vector Zd nis processed by the MLP' synthesis neural network to generate as output a second vector representative of the Pdv sample n to decode, according to one embodiment a triplet which is injected into the decoded image I (Pdv n ) at positions (x n , y n ) of the color components (Rd, Gd, Bd) of the image v. This step is similar to step E26 which was described for the encoder in support of the.

[0172] When all samples have been processed, the current decoded image corresponding for example to image I(Pdv n ) is available.

[0173] Illustratively represents a decoding method used in one embodiment of the invention.

[0174] In this embodiment, the FMdv decoded cards i associated with the current image v of the sequence are 4 in number. In a preferred mode, there are 7 of them.

[0175] In this embodiment, the first map FMdv1 has the same resolution as image Iv, and therefore has WxH variables, where W represents the width of the image in pixels, and H its height. The second map FMdv2 has half the resolution (in each dimension) of map FMdv1. Each additional map has half the resolution of the previous map. This structure makes it possible to reduce the number of variables in the feature maps, which facilitates decoding while minimizing the coding cost.

[0176] In this embodiment, a group of reference maps associated with an image v' of the sequence, already decoded (for example the previous image in the sequence, i.e. v'=v-1), is available at the decoder. The group of maps associated with image v' has the same structure as the group of maps being decoded associated with the current image v: the first map FMdv'1 has the same resolution as image Iv', the second map FMdv'2 has half the resolution (in each dimension) of map FMdv'1, each additional map has half the resolution of the previous map.

[0177] When decoding a current map, e.g., FMdv1, at least one reference map is used, e.g., FMdv'1andFMdv' 2.Comme il sera illustré à l'appui des figures 10 et 11, les cartes de référence sont utilisées pour construire le voisinage des valeurs à décoder dans la carte courante.

[0178] When FMdv cards iare decoded, the FMdv2 card is oversampled by a factor of 2 in each dimension, according to any oversampling method within the reach of those skilled in the art, the FMdv3 card is oversampled by a factor of 4 in each dimension, and the FMdv4 card by a factor of 8 in each dimension.

[0179] The resulting FMS'v maps i are of the same resolution as the image to be decoded, and therefore have WxH values, where W represents the width of the image in pixels, and H its height.

[0180] In this embodiment, the vector Zd n is a 4-tuple (z1…z4) consisting of the values ​​of the FMS'v cards i located at the coordinates (x n , y n ) of the current pixel Pdv n . The vector Zd n is optionally dequantized then processed by the MLP' synthesis neural network to generate as output the triplet (R, G, B) representative of the Pdv sample nto decode. The triplet (R, G, B) is inserted into the decoded image I (Pdv n ) at coordinates (x n , y n ) in the color components (Rd, Gd, Bd).

[0181] According to a variant presented in dotted lines, there are 5 cards: an additional card FME'0 has been introduced. In this embodiment, the vector Zd nis a 5-tuple. Such an additional map may be, for example, but not limited to:– A map containing at each point the abscissa of this point.– A map containing at each point the ordinate of this point.– A map containing at each point a positional coding (as described for example at the internet address https: / skosmos.loterre.fr / P66 / fr / page / -K0D65X2X-X).– A map representing an image distinct from the image currently being processed, capable of providing information on the image to be coded, for example the previously processed image if the current image is part of a series of images to be coded such as a video, a set of medical images, a multi-view representation, etc.– A map representing a feature map of an image distinct from the image being processed that may provide information about the image to be encoded, for example a previously processed map if the current image is part of a series of images to be encoded such as a video, a set of medical images, a multi-view representation, etc.– A map containing the value of an already decoded sample of the same map, for example the previous sample in the decoding order.

[0182] Laest is a flowchart representing a method of coding feature maps that can be implemented by the laet coding device and by the laet coding method.

[0183] These steps constitute sub-steps of step E23 described previously in support of the. Their purpose is to code a current value Vv n of a current FMv feature map i from the second group of the current image en cours de traitementen utilisant des valeurs de voisinage extraites des cartes de référence FMv'iassociées à une image de référence v ».

[0184] In a sub-step E231, a neighborhood vector (Cv n ) is established, including values ​​close to the value Vv n. As will be illustrated later in support of figures 10 and 11, these neighboring values ​​can be located in the equivalent card (i.e. of the same order number in the group) and / or in one or more different cards of the plurality M of FMv' cards i.Ce vecteur de voisinage est constituéd’un nombre C de valeurs, ou données, correspondant à des valeurs de voisinage (par exemple, C=10).

[0185] Alternatively, values ​​can also be extracted from FMv maps i associated with the current image, already decoded. In this case, these values ​​must be known to the encoder and the decoder, so they must be located in a causal neighborhood of the value Vv n .

[0186] According to a first embodiment, these values ​​are used to determine the context of an entropy coder to code the current value during a step E234. This coder can be a CABAC (Context-adaptive binary arithmetic coding) type coder. This type of coder is well known to those skilled in the art. It is notably used in the H.265 / HEVC video compression standard. It is an arithmetic coder whose compression is lossless. It decomposes all non-binary symbols into binary symbols. Then, for each bit, the coder selects the most suitable probability model and uses a context to optimize the probability estimation. This context can be defined by information from the neighboring elements. Arithmetic coding is then applied to compress the resulting data. As is known to those skilled in the art, there are several ways to use the neighborhood vector to produce context information.For example, one can count the number of non-zero neighboring values, and associate a context with each number. Alternatively, one can perform comparisons between several neighboring values, and associate a given context with an ordering configuration between the neighboring values, for example by ranking the neighboring values ​​in ascending order, and associating a context with each possible order.

[0187] In a second embodiment, the neighborhood is used to predict, during a step E232, the current value from an autoregressive model. It is recalled that an autoregressive model predicts a sample of a series by its past values. In this mode, the past values ​​are constituted by the context, and the difference between the predicted variable and the actual value is quantified then entropically coded during step E234.

[0188] In a third embodiment, as illustrated in support of the, an ARM prediction neural network is used during step E233 to predict the statistical characteristics of the variable to be coded. The neighborhood vector is given as input to the ARM network, to provide as output a prediction of the current value. According to one embodiment, the ARM network behaves like a function f ѱ as described in support of the, which provides a set of statistical parameters (mean, variance, median, etc.) used to encode the current value entropically. The role of this ARM module is to best predict the current value, for the set of values ​​Vv n to encode FMv cards i, in order to reduce the throughput required for encoding feature maps. In another embodiment, the ARM prediction neural network is used to produce the expected probability (pr) of the possible value of the current sample. Entropy coding is adapted to this probability (as is known for Huffman or arithmetic entropy coding).

[0189] In a fourth embodiment, each feature map is divided into blocks of predetermined size, and the encoding of each block comprises a transform (e.g., a DCT, Discrete, Cosine Transform, a Haar transform, etc.), and the transformed values ​​are encoded by an entropy coding.

[0190] At the end of the process, the current coded value Vv n of the current FMv card i being processed is coded in the form of a variable Vcv n.

[0191] The is a flowchart representing a method of decoding feature maps that can be implemented by the decoding device of the and by the decoding method of the.

[0192] These steps constitute sub-steps of step E31 described previously in support of the. Their purpose is to decode a current value Vdv n of a current FMdv feature map i from the second group of the current image en cours de traitement,en utilisant des valeurs du voisinage extrait des cartes de référence FMdv'iassociées à des images de référence déjà décodées.

[0193] During a sub-step E311, a neighborhood vector (Cdv n ) is established, including values ​​close to the value Vdv n. This step is similar to step E231 previously described and the same embodiments apply. In particular, these neighboring values ​​may be located in the equivalent map (i.e. of the same order number in the group of characteristic maps) and / or in one or more different maps of the plurality M of FMdv' mapsi.. This neighborhood vector is made up of a number C of values, or data, corresponding to neighborhood values ​​(for example, C=10) located in the same map or / and in a different map of the plurality M of FMdv maps i.

[0194] Alternatively, values ​​can also be extracted from FMdv maps i of the current image. In this case, these values ​​must be known to the encoder and the decoder, so they must be located in a causal neighborhood of the value Vdv n.

[0195] According to a first embodiment, these values ​​are used to determine the context of an entropy decoder to decode the current value during a step E314. This decoding is of a type similar to that which was used in the encoder, for example CABAC. The use of the neighborhood to produce context information is similar to that which was chosen in the encoder. For example, the number of neighboring values ​​other than zero can be counted, and a context can be associated with each number. Alternatively, comparisons can be made between several neighboring values, and a given context can be associated with an order configuration between the neighboring values, for example by classifying the neighboring values ​​in ascending order, and by associating the context with each possible order.

[0196] In a second embodiment, the neighborhood is used to predict during a step E312 the current value from an autoregressive model. In this mode, the past values ​​are constituted by the context, and the difference between the predicted variable and the actual value is quantified then entropically coded during step E314.

[0197] In a third embodiment, as illustrated in support of the, an ARM prediction neural network is used in step E313 to predict the statistical characteristics of the variable to be decoded. The neighborhood vector is applied as input to the ARM network to provide as output a prediction of the current value. According to one embodiment, the ARM network behaves like a function f ѱas described in support of the, defined by a set of statistical parameters (mean, variance, median, etc.) used to entropically decode the current value. According to another embodiment, the prediction neural network ARM' is used to produce the expected probability (pr) of the possible value of the current sample. Entropy decoding is adapted to this probability (as is known for Huffman or arithmetic entropy coding). The network ARM' is identical to the network ARM if the coding was performed without loss.

[0198] In a fourth embodiment, each feature map is divided into blocks of predetermined size, and decoding each block involves entropy decoding values ​​which then undergo an inverse transform (e.g., an inverse DCT, Discrete, Cosine Transform, an inverse Haar transform, etc.).

[0199] At the end of the process, the current decoded value Vdv n of the current FMdv card i being processed is decoded.

[0200] Illustratively depicts a method of encoding or decoding feature maps according to one of the embodiments.

[0201] In this illustration, the coding (resp. decoding) of the current value Vv n (resp. Vdv n ) located in coordinates (x n , y n ) in the current feature map FMv1 (i.e. the one being encoded or simulated) (resp. FMdv1 - i.e. the one being decoded) uses the contextual information of two maps of a reference image v' that has already been decoded: the map FMv'1 which is located at the same position as the current map in the reference map group, and the previous map FMv'2 (resp. FMdv'2). The values ​​located at the coordinates (x n -1, yn -1), (x n -1, y n ), (x n -1, y n +1), (x n , y n -1), (x n , y n ), (x n , y n +1), (x n +1, y n -1), (x n +1, y n ), (x n +1, y n+ 1) in the map FMv'1 (resp. FMdv'1), and the variables located at the coordinates (x n / 2-1, y n / 2-1), (x n / 2-1, y n / 2), (x n / 2, y n / 2-1), (x n / 2, y n / 2) in the FMv'2 (resp. FMdv2) layer are used to determine the neighborhood to encode (resp. decode) the current value. These values, which are all available to the encoder and decoder, constitute the neighborhood vector Cv n (resp. Cdv n ) of the value to be coded Vv n (resp. Vdv n ) which can be used in one of the embodiments described above in support of the(resp.).

[0202] Illustratively depicts another method of encoding or decoding feature maps according to one of the embodiments

[0203] In this illustration, the coding (resp. decoding) of the current value Vv n (resp. Vdv n ) located in coordinates (x n , y n ) in the current feature map FMv1 (resp. FMdv1) uses the contextual information of the reference map located at the same position in a reference image v ». The values ​​represented in gray are used to determine the neighborhood to encode (resp. decode) the current value. These values, which are all available to the encoder and decoder, constitute the neighborhood vector Cv n (resp. Cdv n ) of the value Vv n (resp. Vdv n ) which can be used in one of the embodiments described above in support of the(resp.).

[0204] In the illustrated embodiment, the neighborhood vector is extracted by a CTX module (resp. CTX ") of the FMC coding module (resp. FMD), then it is applied to the input of the ARM prediction neural network (resp. ARM "), used to predict the statistical characteristics (µ,σ) or the probability (pr) of the value to be coded (resp. decoded) represented by a cross. The value is coded (resp. decoded) by the CE entropy coder (resp. DE).

Claims

Method of decoding a sequence of images (S v ) comprising the following steps: - a construction step, comprising the following sub-steps for at least one current image (v, I(Pv n )) of the sequence: - construction (E21, E22) of a first group of characteristic maps (FMv' i ) representative of at least one image distinct from the current image (v', I(Pv' n )), called reference image; - construction (E21, E22) of a second group of characteristic maps (FMv i ) representative of the current image (v, I(Pv n ));- for at least one sample, called the current sample (Pv n ), of the current image, associated with a position (x n , y n ) in the current image: - construction (E25) of a characteristic vector (Z n ) from the said characteristic maps (FMv i ) of said second group, depending on said position (x n , y n) of said current sample (Pv n ) ;- processing (E26) of said characteristic vector (Z n ) by an artificial neural network (MLP), called a synthetic neural network, defined by a set of parameters (Wv k ), to provide a vector (P'v n ) representative of a decoded value of said at least one current sample;- updating (E22, E27) of at least one value of one of said characteristic maps of said second group and / or of at least one parameter of said network, as a function of a coding performance measurement,- a coding step (E23) of said second group of characteristic maps (FMv i ) including, for at least one value, called current value (Vv n ), of one of said characteristic maps, called current map, an entropic coding of said value as a function of a neighborhood comprising at least one value (Vv' n) extracted from at least one map, called the reference map, of the first group of characteristic maps, - a coding step (E27) of said set of parameters (Wv k ) of said synthetic neural network. Coding method according to claim 1, characterized in that the step of coding said current value (Vv n ) includes the following sub-steps:- construction (E231) of a neighborhood vector (Cv' n ) comprising said at least one neighborhood value extracted from said at least one reference map (FMv' i ) ;- processing (E233) of said neighborhood vector (Cv' n ) by an artificial neural network, called a prediction neural network (ARM), defined by a set of parameters (Ov b ), to provide a prediction of said current value (Vv n);- updating (E22, E23, E28) of at least one parameter of said prediction network, as a function of the coding performance measurement,- a step of coding said set of parameters (Ov b ) of said prediction network. Method of decoding from a binary stream of a sequence of images (S v ), comprising the following steps for at least one current image (v, I(Pdv n )) of the sequence: - obtaining (E31) at least one characteristic map (FMdv' i ), called reference map, of a first group of decoded feature maps representative of at least one image distinct from the current image (v', I(Pdv' n )), called reference image.- decoding (E31) of a second group of feature maps (FMdv i ) representative of the current image including, for at least one value, called current value (Vdv n), of one of said characteristic maps, called current map, an entropic decoding (DE) of said value as a function of a neighborhood comprising at least one decoded value (Vdv' n ) extracted from said at least one reference map;- decoding (E34) of a set of parameters (Wdv k ) representative of a synthetic neural network (MLP');- for at least one sample, called current sample (Pdv n ), of the current image (v), associated with a position (x n , y n ) in the current image:- construction (E33) of a characteristic vector (Zd n ) from the characteristic maps of the said second group (FMdv i ), depending on said position (x n , y n ) of said current sample, and:- processing (E35) of said vector (Zd n ) by said synthesis neural network (MLP') to provide a vector representative of a decoded value of said at least one current sample (Pdvn ). Decoding method according to claim 3, characterized in that the step of decoding said current value (Vdv n ) includes the following sub-steps:- decoding (E34) of a set of parameters (Odv k ) representative of a neural network, called a prediction neural network (ARM'); - construction (E311) of a neighborhood vector (Cdv n ) comprising said at least one decoded value (Vv' n ) extracted from said at least one reference map (FMdv' i ) ;- processing (E313) of said neighborhood vector (Cdv n ) by the prediction neural network (ARM'), to provide a prediction of said current value (Vdv n ). Coding or decoding method according to one of the preceding claims, characterized in that said first and second groups (FMdv' i , FMdv i ) contain the same number (M) of feature cards. Coding or decoding method according to one of the preceding claims, characterized in that said at least one reference map (FMdv' i ) is at the same resolution as the current map (FMdv i ). Coding or decoding method according to one of the preceding claims, characterized in that said at least one reference image (v') is indicated in the binary stream. Coding or decoding method according to one of the preceding claims, characterized in that said at least one reference image (v') is predetermined. Encoding or decoding method according to one of the preceding claims, characterized in that said at least one reference map is indicated in the binary stream. Coding or decoding method according to one of the preceding claims, characterized in that said at least one reference map is predetermined. Coding method according to claim 2 or decoding method according to one of claims 4 to 10, characterized in that said at least one decoded value (Vdv' n ) extracted from said at least one reference map is at the same position (x n ,y n ) in said at least one reference map (FMdv' i ) than the current value (Vdv n ) in the current feature map (FMdv i ). Coding method according to claim 2 or decoding method according to one of claims 4 to 11, characterized in that the neighborhood vector further comprises at least one second decoded value (Vd n ') taken from another reference map. Device for coding a sequence of images (S v ), said device being configured to implement:- a construction step, comprising the following sub-steps for at least one current image (v, I(Pv n)) of the sequence: - construction (GEN, MAJ) of a first group of characteristic maps (FMv' i ) representative of at least one image distinct from the current image (v', I(Pv' n )), called reference image; - construction (GEN, MAJ) of a second group of characteristic maps (FMv i ) representative of the current image (v, I(Pv n ));- for at least one sample, called the current sample (Pv n ), of the current image, associated with a position (x n , y n ) in the current image: - construction (XTR) of a characteristic vector (Z n ) from the said characteristic maps (FMv i ) of said second group, depending on said position (x n , y n ) of said current sample (Pv n ) ;- processing (MLP) of said characteristic vector (Z n) by an artificial neural network, called a synthetic neural network, defined by a set of parameters (Wv k ), to provide a vector (P'v n ) representative of a decoded value of said at least one current sample;- updating (MAJ) of at least one value of one of said characteristic maps of said second group and / or of at least one parameter of said network, as a function of a coding performance measurement,- a coding step (FMC, ARM) of said second group of characteristic maps (FMv i ) including, for at least one value, called current value (Vv n ), of one of said characteristic maps, called current map, an entropic coding of said value as a function of a neighborhood comprising at least one value (Vv' n ) extracted from at least one map, called the reference map, of the first group of characteristic maps, - a coding step (NNC) of said set of parameters (Wv k) of said synthetic neural network. Device for decoding from a binary stream of a sequence of images (S v ), said device being configured to implement, for at least one current image (v, I(Pdv n )) of the sequence: - obtaining (FMD) at least one characteristic map (FMdv' i ), called reference map, of a first group of decoded feature maps representative of at least one image distinct from the current image (v', I(Pdv' n )), called reference image.- decoding (FMD) of a second group of feature maps (FMdv i ) representative of the current image including, for at least one value, called current value (Vdv n ), of one of said characteristic maps, called current map, an entropic decoding (DE) of said value as a function of a neighborhood comprising at least one decoded value (Vdv' n) extracted from said at least one reference map;- decoding (NND) of a set of parameters (Wdv k ) representative of a synthetic neural network (MLP');- for at least one sample, called current sample (Pdv n ), of the current image (v), associated with a position (x n , y n ) in the current image:- construction (XTR”) of a characteristic vector (Zd n ) from the characteristic maps of the said second group (FMdv i ), depending on said position (x n , y n ) of said current sample, and:- processing (MLP') of said vector (Zd n ) by said synthesis neural network (MLP') to provide a vector representative of a decoded value of said at least one current sample (Pdv n ). Computer program comprising instructions for carrying out the steps of an encoding or decoding method according to claim 1 or 3 when said program is executed by a computer.

Images