Method and device for encoding and decoding image sequences.

Abstract

Method and device for encoding and decoding images The invention relates to a method and device for encoding and decoding a sequence of images.The decoding process comprises the following steps: - decoding (E35) a set of parameters representative of a synthetic neural network; - decoding (E31) a set of current feature maps including at least one current feature map representative of a current image; - obtaining (E32) a set of reference feature maps, including at least one decoded map; - for at least one current sample of said at least one current image, associated with a position in the image: - construction (E34) of a feature vector from said reference feature map and said current feature map, as a function of said position of said current sample; - processing (E37) of said vector by said synthetic neural network to provide a vector representative of a decoded value of the current sample. Figure for the abstract: Fig. 2.

Description

Title of the invention: Method and device for coding and decoding image sequences. Prior art

[0001] The invention relates to the general field of coding digital image sequences. It relates more particularly to the compression of digital videos.

[0002] Digital videos are generally subject to source coding aimed at compressing them in order to limit the resources required for their transmission and / or storage. There are many coding standards, such as the ITU / MPEG standards (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) as well as their extensions (MVC, SVC, 3D-HEVC, etc.).

[0003] The encoding of an image is generally carried out by dividing the image into several rectangular blocks, and by encoding these blocks of pixels according to a given processing sequence. In existing video compression techniques, the processing of a block typically comprises a prediction of the pixels of the block carried out using previously coded then decoded pixels present in the image being encoded, in which case we speak of “Intra prediction”, or of previously coded images, in which case we speak of “Inter prediction”. This exploitation of spatial and / or temporal redundancies makes it possible to avoid transmitting or storing the value of the pixels of each block of pixels, by representing at least some of the blocks by a residual representing a difference between the prediction values ​​of the pixels of the block and the actual values ​​of the pixels of the predicted block.

[0004] As video formats continue to evolve to compress more and more and adapt to the variety of expected formats and communication networks, the possibilities for prediction are becoming ever greater and the classic coding and decoding algorithms very complex.

[0005] In addition to these classic approaches proposed by compression standards (MPEG, ITU), approaches based on artificial intelligence, and in particular neural ones, tend to develop.

[0006] Some of these neural approaches can be seen as a simple extension of the notion of competition of the aforementioned compression techniques, such as the competition of prediction and transformation mode in video coding.

[0007] Other approaches use the concept of "auto-encoder". Auto-encoders are learning algorithms based on artificial neural networks, which allow a new representation of a data set to be constructed. The architecture of an auto-encoder consists of two parts: the encoder and the decoder. The encoder consists of a set of neural layers, which process the data in order to construct new representations called "encoded", also called "latent representations". In turn, the decoder's neural layers receive these representations and filter them in order to try to reconstruct the initial data. The differences between the reconstructed data and the initial data make it possible to measure the error made by the autoencoder. Training consists of modifying the autoencoder's parameters in order to reduce the reconstruction error measured on the different samples of the data set. The performance of such autoencoder-based systems comes at the cost of a considerable increase in memory footprint and complexity compared to conventional approaches such as those proposed by compression standards.They can have millions of parameters and may require up to a million MACs (multiply-accumulate) to decode a single pixel. This makes such decoders far more complex than conventional decoders, which could hamper the adoption of learning-based compression.

[0008] More recently, a simple image encoding technique based on a neural network has been described in the article "Compression with Implicit Neural representations" by Emilien Dupont et al. (arXiv:2103.03123). The proposed encoding technique consists of fitting a neural network to an image, quantizing the network weights and transmitting them. At decoding time, the neural network is evaluated at each pixel position to reconstruct the image. However, such a technique remains inefficient in terms of compression and requires coding the video frames independently.

[0009] In the video domain, an equivalent technique was presented in the article “Scalable Neural Video Representations with Learnable Positional Features” (NeurIPS 2022) by Subin Kim et al. The temporal dimension is taken into account. The coding algorithm generates a set of three two-dimensional latent keyframes, on each spatio-temporal axis, to learn the common frames of the video. It also generates a latent 3D representation grid to capture the local details of the video. This representation is used to modulate a neural network trained to process the three keyframes. However, such an approach lacks flexibility in the choice of representation of the keyframes (the number of which is fixed at 3) and the 3D grid (fixed for the entire sequence). Moreover, it requires the processing of a transmitted 3D grid, which is both heavy in terms of the amount of transmitted data and the memory required for its processing.

[0010] There is therefore a need for a solution allowing a sequence of images to be coded / compressed in a simple and efficient manner. Statement of the invention

[0011] The invention relates to a method for coding a plurality of images of an image sequence comprising the following steps for at least one current image to be coded: - a construction step, comprising the following sub-steps: - construction of a set of current feature maps, representative of said at least one current image, comprising at least one current feature map; - construction of a set of reference characteristic maps, comprising at least one coded and then decoded reference characteristic map; - for at least one sample, called the current sample, of said at least one current image, associated with a position in the current image: - construction of a characteristic vector from said at least one reference characteristic map and said at least one current characteristic map, as a function of said position of said current sample; - processing of said characteristic vector by an artificial neural network, called a synthetic neural network, defined by a set of parameters, to provide a vector representative of a decoded value of the current sample; - updating at least one value of said at least one current feature map and / or at least one parameter of said network, based on a coding performance measurement; - a step of coding said set of current feature maps; - a step of coding said set of parameters of said synthetic neural network.

[0012] The invention also relates to a method for decoding a sequence of images from a binary stream, comprising the following steps for at least one current image to be decoded: - decoding a set of parameters representative of a synthetic neural network; - decoding a set of current feature maps comprising at least one current feature map representative of said at least one current image; - obtaining a set of reference feature maps, comprising at least one decoded reference feature map; - for at least one sample, called current sample, of said at least one current image to be decoded, associated with a position in the image: - construction of a characteristic vector from said at least one reference characteristic map and said at least one current characteristic map, as a function of said position of said current sample, and: - processing of said vector by said synthesis neural network to provide a vector representative of a decoded value of said at least one current sample.

[0013] For the purposes of the invention, encoding, or “coding”, means the operation which consists of representing a set of samples, or pixels, in a compact form carried for example by a digital binary train. Decoding means the operation which consists of processing a digital binary train to restore decoded samples.

[0014] By "sequence of images" is meant a plurality of ordered two-dimensional images, for example temporally in the case of a video. According to one example, the sequence corresponds to a scene. According to one example, the sequence corresponds to a set of predefined images, for example a fixed number, or, in the sense of the MPEG standards, a GOP (Group Of Pictures) comprising the images located between two images of the Intra-image type, also called "intra period". According to another example, the images can be views of the same scene represented in multi-views. According to another example, the images can be a plurality of temporal and multi-view images (immersive video).

[0015] By “at least one current image” we mean a sub-sequence of at least one image of the sequence.

[0016] By "sample" is meant a value taken from an image of the sequence. Sampling a signal produces a series of discrete values ​​called samples. In the case of an image signal, the sample is called a pixel, which can be, for example, a color pixel traditionally represented by a triplet of values, for example (R,G,B) or (Y,U,V). The position of the sample is identified by its abscissa (x) and ordinate (y) coordinates in the image.

[0017] By "feature map" is meant a part of an abstract representation of an image, comprising a plurality of variable scalar data, which are also called characteristic values, for example real numbers, or integers, signed or unsigned. The abstract representation of an image can therefore comprise several feature maps. The abstract representation of an image can also correspond to a single feature map comprising a set of values ​​associated with a position.

[0018] By reference feature map is meant such a feature map obtained for all the images in the sequence. On the contrary, a current feature map only concerns the current subsequence.

[0019] By "feature vector constructed from feature maps as a function of a position" is meant a vector consisting of one or more elements, or data, preferably discrete, the data being constructed from the feature maps at a position determined by that of the samples being processed in the images. This feature vector is the one that is applied to the input of the synthetic neural network. In the case of an image, such a vector can be constituted for example from a plurality of values ​​taken from each of the feature maps at the same abscissa and ordinate coordinates as the subsequence of samples to be coded (resp. to be decoded).

[0020] By "synthetic neural network" is meant a neural network such as a convolutional neural network, a multi-layer perceptron, an LSTM (for "Long Short Term Memory" in English), etc. The neural network is defined for example by a plurality of layers of artificial neurons and by a set of activation, weighting and addition functions (for example, a layer can calculate y = f(Ax+b), where y and b are vectors of dimension N, x a vector of dimension M, A is a matrix of dimension MxN, and f is the activation function).

[0021] By "parameter of the neural network" is meant one of the values ​​which characterizes the neural network, for example a weight associated with one of the neurons (filter coefficient, weighting, bias, value affecting the operation of the non-linearity, etc.)

[0022] By "processing by a synthetic neural network" is meant the application of a function expressed by a neural network to the input characteristic vector to produce an output vector representative of the sub-sequence of samples to be coded (resp. decoded). This output vector may comprise one or more data representative of the sub-sequence of samples.

[0023] By "performance measurement" is meant a measurement between at least one value of a sample to be coded and a decoded value of said sample. The measurement can evaluate for example a distortion, or a perceptual error. It can be carried out on a sample or a plurality of samples (for example, the current samples, or the current images, etc.). The measurement can also include a measurement of the flow rate, in particular associated with the coding of the neural network and / or the coding of the characteristic maps. The measurement can be a joint measurement between the flow rate and the distortion through their weighting. As is well known in the state of the art, the value of this measurement is generally minimized until a target value is reached.

[0024] By "construction step" is meant a step which aims to construct the representative parameters of the image, before their actual coding. The construction sub-steps can be repeated as many times as necessary to obtain an acceptable performance measurement.

[0025] Generally speaking, it is considered that the steps of a coding or decoding method should not be interpreted as being linked to a notion of temporal succession. In other words, the steps may be carried out in a different order than that indicated, or even in parallel.

[0026] The coding method according to the invention carries out a construction of the parameters of coding, from a sequence of input images, by training a neural network on feature vectors associated with a position of a sample to be coded in one of the input images of a sub-sequence of images being processed. The sub-sequence comprises at least one image. These feature vectors are constructed from feature maps of the image itself, and reference feature maps of the sequence, representative of the images in the sequence. The feature maps may be at the resolution of the input images, or at a lower resolution. During training, or construction, the parameters of the neural network and the values ​​of the feature maps are updated according to a performance measure, for example of the rate-distortion type.When the training is finished, that is to say that the performance measurement obtained is satisfactory, the actual coding of the parameters of the neural network and the values ​​of the current characteristic maps can be carried out and stored or transmitted to the decoder. According to one embodiment, when the entire sequence has been processed, the reference maps are stabilized and can be encoded and stored or transmitted to the decoder.

[0027] Advantageously, the training process makes it possible to refine the parameters of the synthetic neural network and / or the values ​​of the characteristic maps until an adequate representation in terms of performance is obtained, for example a desired balance between the generated bit rate and the distortion undergone by the image or the sub-sequence of input images being encoded. The training of the values ​​of the characteristic maps and the parameters of the synthetic neural network can be joint. Advantageously, the encoding method according to the invention makes it possible to efficiently compress the signal.

[0028] Advantageously, the decoding method is simple since it suffices to decode the characteristic maps of the current sub-sequence of images, the reference characteristic maps and the synthesis neural network to reconstruct a decoded version of the current sub-sequence.

[0029] Advantageously, the current maps dynamically capture the local characteristics of the current sub-sequence, since they are coded (resp. decoded) as the sequence is processed. The reference maps, coded (resp. decoded) only once for the sequence, capture the global characteristics of the sequence. It is thus possible to benefit from both types of characteristics. Furthermore, once the reference maps have been transmitted and are therefore available on the decoder side, the current maps make it possible to generate the current image, and it is therefore possible to design a transmission system that works image by image with low latency, each current image being decoded upon receipt of the associated current maps.

[0030] Such a synthetic neural network can advantageously be of very simple structure with few parameters.

[0031] Furthermore, the decoding can be carried out by sequence of samples, in a progressive manner.

[0032] According to embodiments of the coding or decoding method:

[0033] - The coding method comprises a step of constructing and coding a group of reference feature maps representative of the sequence comprising said set of reference feature maps.

[0034] - The decoding method further comprises: - a step of decoding a group of reference characteristic maps representative of the sequence and - obtaining said set of reference feature maps from said group.

[0035] Advantageously, the group of reference maps is constructed and coded by the encoder, and decoded by the decoder for the entire sequence. Then for each sub-sequence being processed, a part of the group of reference maps is considered to constitute the reference set used for the sub-sequence. It is thus possible to simplify the decoding by keeping a limited number of reference maps. According to one embodiment, the set is smaller than the group. According to one embodiment, the set is equal to the group.

[0036] - The method comprises a step of transforming said set of cards of ca common and / or reference characteristics to obtain a set of transformed characteristic maps at the resolution of the input image, the method being characterized in that said characteristic vector is constructed from said transformed characteristic maps. Advantageously according to this mode, the characteristic maps are divided into two sets, one of which is reserved for the extraction of the characteristic vectors and the other for coding. It is thus possible to dissociate the two processes whose purpose is different: the maps of the first group, to be coded (resp. decoded), must be compressed as efficiently as possible, while the maps of the second group, or transformed maps, must be able to facilitate the process of extraction and construction of the characteristic vector.

[0037] - According to a variant, at least one of the current characteristic cards and / or reference is of lower resolution than that of the subsequence to be coded (resp. decoded) and the transformation operation involves oversampling. Advantageously, in this mode, the compression of the characteristic maps is more efficient since at least one of the current and / or reference characteristic maps, to be coded (resp. decoded), contains fewer values ​​than if it were at the resolution of the signal. For example, in the case of a digital image, one of the maps of current and / or reference characteristics can be at 1 / 2 resolution, that is to say that it has half as many values ​​on the abscissa and ordinate as the input image has samples, or in total 4 times fewer values ​​than a characteristic map at the resolution of the image. On the other hand, the transformed characteristic map, which corresponds to a transformation of this current and / or reference map, has the same resolution as that of the image. The transformation therefore in this case includes at least one oversampling operation to obtain the same number of values ​​in this transformed map as the input image (resp. to be decoded) has samples.

[0038] - At least one of said current and / or reference characteristic maps is resolution identical to that of the subsequence to be coded (resp. decoded). Advantageously, according to this mode, at least one of the characteristic maps, at the same resolution as the input signal to be coded (resp. decoded) allows significant fidelity and respect for the details of the initial resolution of the image signal. The transformation according to one embodiment retains in this case the number of values ​​of the transformed characteristic map; it can be reduced to identity (no processing is carried out on the values ​​of the current map) or include a filtering operation, quantization, a Fourier transformation, etc. During the coding step, quantization is essential for the proper functioning of the system if the characteristic maps include, for example, floating point or real values. It is necessary to quantize them before coding them and / or providing them as input to the synthesis and / or prediction neural network.In decoding, however, inverse quantization is not necessary, depending on the embodiments.

[0039] - The construction of the characteristic vector includes an extraction sub-step of a value from a current and / or reference feature map at a position identical to that of the current sample in the current image to be encoded or decoded. Advantageously, it is possible to extract a value from a feature map at the same position as the sample in the image, to constitute an element of the feature vector. This method is simple to implement. For example, if we have NF feature maps as input (NF = NFM + NFR, where NFM represents the number of current feature maps, and NFR the number of reference feature maps), with the same resolution as the image, a simple extraction of the values ​​from the maps at the coordinates of the current sample (at the same abscissa and the same coordinate in the feature map) makes it possible to directly construct the feature vector of J elements.

[0040] - The construction of said characteristic vector comprises the following sub-steps: - extracting a plurality of values ​​from said current and / or reference characteristic maps as a function of said position of said current sample; - processing of said extracted values ​​to obtain the characteristic vector.

[0041] Advantageously according to this mode, the characteristic vector is extracted from the characteristic maps, which can be any, and in particular at a resolution lower than that of the image to be coded (resp. decoded), before undergoing processing. Such processing can for example correspond to a quantification of the extracted data, or to a scaling, or to a filtering, etc. During coding, a quantification is essential for the proper functioning of the system if the characteristic maps comprise for example floating, or real values. It is necessary to quantify them before coding them and / or providing them as input to the synthesis and / or prediction neural network. During decoding, however, inverse quantification is not necessary, according to the embodiments.

[0042] - The method comprises a step of constructing a group of character maps additional characteristics and the characteristic vector is further constructed from said characteristic maps. Advantageously, these additional maps, constructed identically at the encoder and the decoder, are neither stored nor transmitted at the encoder level, nor decoded at the decoder level. They thus make it possible to benefit from additional data to improve compression without degrading the flow rate. They may, for example, include coordinates, causal data available in other maps, data concerning other images already processed by the encoder or decoder, etc.

[0043] - Said sets of reference and current characteristic maps contain the same number of feature maps; advantageously, in this mode, the sequence and the different subsequences of images in the sequence are associated with the same number of feature maps, which may also have the same structure. For example, the sequence and the subsequences are associated with five feature maps with a hierarchy of decreasing resolutions. This facilitates the generation and extraction of values ​​from the feature maps to produce the feature vector.

[0044] - Said at least one reference map is at the same resolution as said at least a current map. Advantageously, according to this mode, the construction of the characteristic vector is facilitated since the reference map is at the same resolution as the current map, it is very simple to carry out the extraction of a value at the same position or at a position close to that of the value of the current map being encoded.

[0045] - Said at least one reference card is indicated in the bitstream. Avanta Conveniently, according to this mode, a simple indicator coded in the stream allows the decoder to directly access the reference card(s) to be used for decoding the current image or subsequence, for example via an indicator of the number of the reference card in the group of reference cards.

[0046] - Said at least one reference card is predetermined. Advantage Conveniently, according to this mode, the feature map(s) to be used for decoding the current image or subsequence are predetermined. For example, the reference feature map is the one that has the same order number as the current map in the group of feature maps to which they respectively belong. According to another example, all feature maps in the reference group serve as reference maps for the current map.

[0047] - Said at least one current map is decoded according to said at least one reference map. Advantageously, according to this mode, the (at least one) reference map is used not only for the construction of the characteristic vector, but also for the decoding (for example entropic) of a current map.

[0048] - Said at least one current map is decoded according to at least one map Advantageously, according to this mode, a current map already at least partially decoded can be used to more efficiently encode the current map being processed.

[0049] Correlatively, the invention also relates to a coding device and a decoding device.

[0050] The characteristics and advantages of the coding or decoding method apply in the same way to the coding or decoding device according to the invention and vice versa.

[0051] The invention also relates to a computer program on a recording medium, this program being capable of being implemented in a computer or a coding or decoding device in accordance with the invention. This program comprises instructions adapted to the implementation of the corresponding method. This program can use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0052] The invention also relates to an information carrier or a recording medium readable by a computer, and comprising computer program instructions mentioned above. The information or recording carriers can be any entity or device capable of storing the programs. For example, the carriers can comprise a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a floppy disk or a hard disk, a DNA sequence, or a flash memory. Furthermore, the information or recording carriers can be transmissible media such as an electrical or optical signal, which can be conveyed via an electrical or optical cable, by radio link, by wireless optical link or by other means.

[0053] The program according to the invention can in particular be downloaded from an Internet-type network.

[0054] Alternatively, each information or recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of a method according to the invention. Brief description of the drawings

[0055] Other characteristics and advantages of the present invention will emerge from the description given below, with reference to the appended drawings which illustrate exemplary embodiments thereof which are not in any limiting nature.

[0056] [Fig-1] [Fig.l] schematically represents a coding device used in the scope of the invention;

[0057] [Fig.2] [Fig.2] schematically represents a decoding device used in the context of the invention;

[0058] [Fig.3] [Fig.3] illustrates an example of a synthetic artificial neural network used in an embodiment of the invention;

[0059] [Fig.4] [Fig.4] illustrates an example of a synthetic artificial neural network used in another embodiment of the invention;

[0060] [Fig.5] [Fig.5] illustrates an example selection of reference cards used in the context of the invention;

[0061] [Fig.6] [Fig.6] is a flowchart representing an example of a coding method which can be implemented by the coding device of [Fig.l];

[0062] [Fig.7] [Fig.7] is a flowchart representing an example of a decoding method which can be implemented by the decoding device of [Fig.2];

[0063] [Fig.8] [Fig.8] illustrates a decoding method used in one embodiment of the invention. Description of the embodiments

[0064] [Fig.l] schematically represents a device for coding digital image sequences ENC.

[0065] The coding device ENC comprises an initialization module INITc, responsible in particular for constructing and coding the group of reference maps of the sequence.

[0066] An image may represent a two-dimensional image and the sequence a plurality of two-dimensional images (video sequence, stereoscopic, multi-scopic components, series of medical images, etc.). The sequence is any succession of images, for example a GOP.

[0067] For a current sub-sequence to be coded v, comprising at least one current image, a coding module denoted ENCv comprises a GEN module for generating current characteristic maps, a REF module for obtaining characteristic maps reference statistics, an SE transformation module, an XTR data extraction module, an MLP synthesis module corresponding to a synthetic artificial neural network, an NNC neural network coding module capable of coding the synthetic neural network, an FMC feature map coding module (which can also be used by the INITc module to code the reference maps), an EVAL coding performance evaluation module, an MAJ update module.

[0068] The ENC coding device can be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the modules mentioned above can then be realized by the cooperation of the processor and computer program instructions stored in the aforementioned memory and designed to carry out the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0069] The ENC coding device of [Fig.l] receives as input a sub-sequence of at least one image to be coded, denoted I(PVn), comprising a plurality of samples PVn-The index v indicates the image number(s) of the sub-sequence in the sequence of V images (for example, v=3, or v={2,3,4}). The index n indicates the pixel number in an image of N pixels. In one embodiment, an image of the sequence is represented by means of at least one two-dimensional representation, such as a matrix of pixels, each pixel comprising a vector of red (R), green (G), blue (B), or, alternatively, a luminance component and at least one chrominance component. The location of each pixel is defined by its abscissa and ordinate coordinates (xn and yn) in the image.In one embodiment, the sequence is a sequence of grayscale images represented by means of a two-dimensional representation, such as a pixel matrix, each pixel comprising a grayscale component, or luminance. In this case the vector representing the pixel is reduced to a single component. In one embodiment, a subsequence is reduced to a single image. In this case the ENC coder successively processes the current samples Pvn to be coded of the image. In one embodiment, a subsequence comprises at least two successive images. In this case the ENC coder simultaneously processes the current samples Pvn to be coded of the successive images, considered as a subsequence of current samples. In one embodiment, a subsequence comprises a predefined number of successive images, for example 4. .

[0070] As will be described in more detail later with reference to Figures 3 to 7:

[0071] The GEN module for generating feature maps is configured to generate a plurality of NFM current feature maps, denoted FMv; for the current subsequence v. These maps are called current feature maps. According to one embodiment, an optional module not shown may generate one or more additional maps, which will not be encoded or transmitted.

[0072] The REF module is configured to generate a plurality of NFR reference feature maps, denoted FRvj, for the sequence v, from a group of reference feature maps, corresponding to the sequence, to the number of NFRMAX. We therefore have NFR <= NFRMAX. For example, the reference feature maps considered for the current subsequence are all those of the reference group, or are selected as a subset of the reference group. In the latter case, they can be predetermined or their reference can be coded in the stream in the form of an indicator.

[0073] The set of current NFM maps and reference NFR maps is used by the current coding module ENCv to code the subsequence Iv. The number of feature maps used is therefore NFM + NFR.

[0074] In one embodiment, the transformation module SE performs a transformation of the set of current and / or reference feature maps (FMvi, FRvj) to generate a set of transformed feature maps (FMSvi, FRSvj) at the same resolution as the input signal.

[0075] The optional SE module can perform a quantization of the data extracted from this set of NFM+NFR feature maps, or of the feature vector formed from this data. It is recalled that the quantization of a value refers to the mapping of this value to a member of a discrete set of possible code symbols. For example, the set of possible code symbols can be made up of integer values, and the quantization system performs a simple rounding of a real value to an integer value. According to another example, the quantization consists of a multiplication by a given value and then a rounding. Then the SE module can perform a transformation of the values ​​of at least one of the feature maps, for example an oversampling, an interpolation, a filtering, etc.At the end of the transformation, a transformed feature map has the same resolution as the images of the input subsequence. Advantageously, according to this mode, the feature maps that are coded can have a lower resolution than the images to be coded, while the transformed maps, which are used to construct the feature vectors, have the same resolution as the image sequence, which facilitates the extraction of the values.

[0076] In one embodiment, the SE module is absent, in this case the values ​​which will be used to construct the characteristic vector are extracted from the untransformed current and reference characteristic maps.

[0077] The XTR module performs an extraction of values ​​in feature maps current and reference values, optionally transformed, for a current sample or a subsequence of current samples Pvn to be coded, according to its (their) coordinates in the input signal. For example, if we want to code the sample or subsequence of samples Pvn at the coordinates (xn, yn) of an input image, the XTR module performs an extraction of values ​​in maps at positions imposed by the coordinates (xn, yn). In one embodiment, the extracted values ​​constitute the characteristic vector Zn. Zn is a J-tuple, that is to say it comprises J elements, or data z;. Zn comprises two sets of values: one (Zvn ) is extracted from the current characteristic maps of the subsequence v, and the other (Zr n) is extracted from the reference characteristic maps. The vector Zn of index n refers to the characteristic vector of the pixel or sequence of pixels P'vn. According to one embodiment, J= NFM+NFR.

[0078] In one embodiment, the XTR module performs processing of the extracted values, to generate the vector Zn. The XTR module can perform quantification of the data extracted from the set of feature maps. The processing can include other operations, such as filtering, scaling, etc. In particular, if the SE module is not used and if the feature maps are at resolutions lower than that of the images of the sequence, the XTR module can take into account the coordinates of the values ​​in the lower resolution maps.

[0079] It will be noted that at least one of the SE or XTR modules must perform a quantification of the values ​​extracted from the current characteristic maps of the current subsequence.

[0080] The MLP module contains a synthesis neural network defined by K parameters Wk, capable of processing the vector Zn, or J-Uplet, as input, to generate as output a vector representative of the sample or sub-sequence of samples Pvn to be coded of the current sub-sequence. The synthesis neural network contains according to one embodiment an MLP, or Multi Layer Perceptron, consisting of an input layer adapted to the input format (the J-uplet), optionally one or more hidden layer(s), and an output layer adapted to the output format of the output vector, generally a vector comprising NC (number of components) elements for each sample. According to one embodiment, NC is equal to 3 and the output vector is the triplet (R, G, B) of the pixel or pixels P'vn coded then decoded of the current sub-sequence. According to another embodiment, the MLP module contains a convolutional neural network.

[0081] The NNC module carries out the coding of the neural network for synthesizing the current image or sub-sequence, in particular its parameters WvkDuring the training process, or construction, of the coding, that is to say as long as the step of evaluating a performance is not satisfactory, the NNC module carries out a si encoding simulation, followed by decoding, to the evaluation module. Subsequently, it performs the actual encoding of the parameters of the Wvk synthesis neural network. The encoded parameters of the current image or subsequence are denoted Wvc k. As is known, the encoding simulation can be identical to the actual encoding, or an approximation thereof.

[0082] The INITc module generates and encodes the FRj maps, i.e. the values ​​of the characteristic maps of the sequence. The reference coded maps are denoted FRCj. For example, during the initialization of the encoder, all the maps (NFRMAX) of the sequence are constructed, then coded and decoded so that they can be used as a reference for a sub-sequence to be coded. According to another embodiment, the reference maps are generated dynamically as the sub-sequences are coded, in the same way as the current maps. According to one embodiment, the encoding takes into account the neighborhood of a value to be coded of the reference characteristic map, in the map itself and / or in another reference map.

[0083] When coding the subsequences, the FMC module performs the coding of the FMv i maps, i.e. the values ​​of the characteristic maps of the current subsequence (excluding the transformed maps, optionally resulting from oversampling or other transformation by the SE module). During the process of training, or construction, of the coding, i.e. as long as the step of evaluating a performance is not satisfactory, the FMC module performs a coding simulation, followed by decoding, intended for the evaluation module. Subsequently, it performs the actual coding of the values ​​of the FMv maps; and according to one embodiment, as explained above, of the reference maps. The coded maps of the current subsequence are denoted FMcv;. In a known manner, the coding simulation can be identical to the actual coding, or produce an approximation thereof.The encoding module quantizes the latent representation of the map values ​​if necessary by using a quantizer to generate an ordered collection of quantized values. Then the encoding module compresses the quantized data, for example by using an entropy encoder.

[0084] According to one embodiment, it implements a coding taking into account the neighborhood of a value to be coded from the feature map. According to one embodiment, the neighboring position is selected in the map being coded (for example, the neighboring value at the top left of the one being coded). According to one embodiment, this neighborhood is selected in another current map, at a neighboring position. According to another embodiment, the neighborhood is selected in a reference map, at a neighboring position. According to yet another embodiment, the neighborhood is selected in a coded and decoded feature map corresponding to one or more sub-sequences v' of the sequence distinct from the current subsequence v, at a neighboring position, for example the previous subsequence in the sequence, or two past subsequences of the sequence, or one past and one future subsequence, to the extent that the decoding order of the sequence allows it (as is known, the decoding order may be different from the chronological order of the sequence). All these embodiments can be combined, that is to say that the neighborhood can consist of values ​​extracted from the current map, the current maps, the reference maps, and / or current maps of another subsequence.The FMC module may comprise for this purpose a feature map prediction neural network whose parameters may be encoded and transmitted to the decoder, which takes as input a neighborhood vector consisting of the neighborhood values ​​and produces as output a prediction of the current value of the feature map being encoded, which may be presented, in a known manner, in the form of a prediction value or probability data.

[0085] The EVAL module performs an evaluation and minimization of a coding performance. The evaluation function is for example of the rate-distortion type. The minimization can be carried out by gradient descent, or any other method within the reach of those skilled in the art.

[0086] The MAJ module updates the values ​​of the FMv maps to be encoded in the current subsequence, based on the results of the performance function. According to one embodiment, it also updates the values ​​of the reference maps to be encoded in the sequence, based on the results of the performance function.

[0087] Once the current feature maps are stabilized, they can be encoded. According to one embodiment, they are encoded using the reference maps.

[0088] [Fig.2] schematically represents a DEC decoding device.

[0089] The coding device DEC of [Fig.2] decodes the N sequences of samples Pdv n of the sub-sequence v, proceeding by sub-sequence. At the end of the decoding, the V images of the sequence are decoded.

[0090] The coding device DEC comprises an initialization module INITd, responsible in particular for decoding the group of reference maps of the sequence SV, denoted FRdj. For example, during initialization of the decoder, all the maps (NFRMAX) of the sequence are decoded so that they can be used as a reference for a sub-sequence to be decoded by the module DECv.

[0091] A current subsequence to be decoded v comprises at least one current image to be decoded denoted I(PdVn), each image comprising respectively a plurality of samples. The decoder DECv dedicated to the subsequence receives as input the data necessary to decode the subsequence. As for the coder, the index v indicates the image number(s) of the subsequence in the sequence of V images (for example, v=3, or v={2,3,4}). The index n indicates the pixel number in an image of N pixels or a subsequence of images of N pixels. The decoding submodule receives as input at least: - encoded data organized into NFM FMcv feature maps; (the encoded current feature maps); - the Wvck encoded parameters of the synthesis neural network corresponding to the current subsequence, optionally the parameters of other neural networks such as an oversampling neural network, feature map prediction, etc.;

[0092] Furthermore, the DECv module obtains (for example, reads from the decoder memory) a set of reference characteristic maps, denoted FRdvj, from the group of reference maps FRdj

[0093] The DECV decoding module comprises, for a current sub-sequence, a neural network(s) decoding module NND capable of decoding the synthesis neural network MLP', a FMD module for decoding the current feature maps (which can also be used by the INITd module to decode the reference maps), a REF' module for obtaining reference feature maps, a data extraction module XTR', an inverse transformation module SE', a MLP' module corresponding to a synthesis neural network, optionally a module for generating additional feature maps. According to one embodiment, it produces as output a current decoded sub-sequence, comprising at least one image denoted I (Pdvn), comprising a plurality of samples or sequences of decoded samples Pdvn

[0094] The FMD decoding module decompresses the quantized data, for example using an entropy coder. According to one embodiment, it implements a decoding taking into account the neighborhood of a value to be decoded from the feature map. The embodiments proposed for the coder apply here by replacing the coding with a decoding of the values. In particular, the neighborhood can be selected from a current map (including the one being decoded), a reference map, or a current map of a sequence distinct from the current subsequence. The FMD module can comprise for this purpose a feature map prediction neural network, as described for the FMC module. The maps decoded by the FMD module, numbering NFM, are denoted FMdvi (feature maps of the current image).

[0095] The parameters of the synthesis neural network (MLP') decoded by the NND module are noted Wdvk.

[0096] The REF' module constitutes a set of NFR characteristic maps, called reference feature maps, denoted FRdvj, from a group of previously encoded and then decoded reference feature maps, corresponding to the sequence, to the number of NFRMAX. We therefore have NFR <= NFRMAX. The REF' module is identical to that of the encoder and the same examples apply. For example, the reference feature maps considered for the current subsequence are all those of the reference group, or are selected as a subset of the reference group. In the latter case, they can be predetermined or their reference can be decoded in the stream in the form of an indicator.

[0097] The set of current NFM maps and reference NFR maps is used by the current decoding module to decode the current subsequence v. The number of feature maps used is therefore NFM + NFR.

[0098] The decoder can also generate one or more additional maps, identical to the additional maps generated by the encoder for the current subsequence.

[0099] In one embodiment, the transformation module SE' performs a transformation of the set of current and / or reference feature maps (FMdvi, FRdvj) to generate a set of transformed feature maps (FMS'Vi, FRS'vj) at the same resolution as the input signal.

[0100] The SE' module is identical to the SE module of [Fig.l]. The optional SE' module can perform an inverse quantization of the data extracted from this set of NFM+NFR characteristic maps. Then the SE' module can perform a transformation of the values ​​of at least one of the characteristic maps, for example an oversampling, an interpolation, a filtering, etc. At the end of the transformation, a transformed characteristic map has the same resolution as the images of the sub-sequence to be decoded. Advantageously, according to this mode, the characteristic maps that are decoded can have a lower resolution than that of the images to be decoded, while the transformed maps, which are used to construct the characteristic vectors, have the same resolution as the sequence of images, which facilitates the extraction of the values.

[0101] In one embodiment, the SE' module is absent, in this case the values ​​which will be used to construct the characteristic vector are extracted from the untransformed current and reference characteristic maps.

[0102] The XTR' module is identical to the XTR module of [Fig.l]. It performs an extraction of values ​​in current and reference feature maps, for a current sample or a subsequence of current samples Pdvn to be decoded, according to its (their) coordinates in the input signal. For example, if one seeks to decode the sample or the subsequence of samples Pvn at the coordinates (xn, yn) of an input image, the XTR' module performs an extraction of values ​​in maps at positions imposed by the coordinates (xn, yn). In a embodiment, the extracted values ​​constitute the characteristic vector Zdn. Zdn comprises two sets of values: one (Zdvn) is extracted from the current characteristic maps of the subsequence v, and the other (Zdrn) is extracted from the reference characteristic maps. Zdn is a J-tuple, that is, it comprises J elements, or data zd;. The vector Zdn of index n refers to the characteristic vector of the pixel or subsequence of pixels Pdvn. according to one embodiment, J= NFM+NFR.

[0103] In one embodiment, the XTR' module performs processing of the extracted values, to generate the vector Zdn. The XTR' module can perform inverse quantization of the data extracted from the set of feature maps. The processing can include other operations, such as filtering, scaling, etc. In particular, if the SE' module is not used and if the feature maps are at lower resolutions than that of the images of the sequence, the XTR' module can take into account the coordinates of the values ​​in the lower resolution maps.

[0104] It will be noted that at least one of the modules SE' or XTR' must perform an inverse quantization of the values ​​extracted from the current characteristic maps of the current sub-sequence, if a quantization has been performed at the encoder.

[0105] The MLP' module contains a so-called synthesis neural network, defined by K parameters Wdk, capable of processing the vector Zdn, or J-Uplet, as input, to generate as output a vector representative of the sample or sub-sequence of samples Pdvn to be decoded. The synthesis neural network contains, according to one embodiment, an MLP, or Multi Layer Perceptron, consisting of an input layer adapted to the input format (the J-uplet), optionally one or more hidden layer(s), and an output layer adapted to the output format of the output vector, generally a vector comprising NC (number of components) elements for each sample. It can also comprise a convolution neural network. According to one embodiment, K=3 and the output vector is the triplet (R, G, B) or (Y, U, V) of each decoded pixel Pdvn.The MLP' module is identical in structure to the MLP module, and its parameters are either identical if the coding of its Wk parameters is lossless, or different if the coding is lossy.

[0106] When all the samples Pdvn of the sub-sequence v have been decoded, a reconstructed signal I(Pdvn) is available, according to one embodiment an image Iv comprising N samples decoded in the form of N vectors Pdvn, according to another embodiment a sub-sequence of images Iv each comprising N samples decoded in the form of N vectors Pdvn.

[0107] The decoding device DEC can be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the modules mentioned above can then be realized by the cooperation of the processor and computer program instructions stored in the aforementioned memory and designed to carry out the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor

[0108] [Fig.3] illustrates an example of a synthetic artificial neural network used for coding and decoding according to an embodiment of the invention.

[0109] The synthetic artificial neural network used for coding, MLP, and the synthetic artificial neural network used for decoding, MLP', are defined by an identical structure, comprising for example a plurality of layers of artificial neurons, and by a set of weights and activation functions associated respectively with the artificial neurons of the network concerned.

[0110] A vector representation of a current sample (a vector Zn or Zdn from the feature maps FMv / FMSvi and FRvj / FRSvj at the encoder or FMdv. / FMS'v, and FRdvj / FRS'vj at the decoder) is applied as input (i.e. to an input layer) of the artificial neural network of synthesis MLP or MLP'. The input vector comprises values ​​from the current maps (Zvn,Zdvn) and values ​​from the reference maps (Zrn,Zdrn).

[0111] Generally, the MLP / MLP' artificial neural network produces as output a plurality of vectors P'vn or Pdvn representative of the reconstructed samples (at coding) or decoded samples (at decoding), according to one embodiment the color components (R, G, B or Y, U, V) constituting the color pixels of the sub-sequence of images.

[0112] In [Fig.3], the sub-sequence of images corresponds to an image, each sub-sequence of samples therefore comprises a single sample and the corresponding vector is noted P' ln (at the encoder) or Pdin (at the decoder). According to the example represented in [Fig.3], 4 values ​​are extracted from each input set. The artificial synthesis neural network produces as output the constituent components of a pixel of the image.

[0113] The concatenation of all these reconstructed pixels in the image constitutes the decoded, or reconstructed, image.

[0114] At the encoder, the MLP synthesis artificial neural network is trained on the image, so as to minimize the differences between the input representation of the current subsequence I (Pvn) and its output representation I (P'vn), while also minimizing the amount of data to be encoded. The EVAL module performs a performance measurement in this sense. Once the encoder training is complete, the network parameters are encoded, either lossless, in which case the MLP' neural network is identical to MLP, or lossy, in which case the MLP' network may be slightly different from MLP.

[0115] [Fig.4] illustrates an example of a synthetic artificial neural network used in coding and decoding according to another embodiment of the invention.

[0116] [Fig.4] is identical to [Fig.3] with respect to the input of the neural network.

[0117] Generally, the MLP / MLP' artificial neural network produces as output a plurality of vectors P'vn or Pdvn representative of the reconstructed samples (at coding) or decoded samples (at decoding), according to one embodiment the color components (R, G, B or Y, U, V) constituting the color pixels of the sub-sequence of images.

[0118] In [Fig.4], the sub-sequence of images corresponds to two images, a sub-sequence of samples therefore comprises two samples and the two corresponding vectors (R, G, B) or (Y, U, V) are noted P' ln / Pdln. and P'2n / Pd2n. The concatenation of all these pixels reconstructed in the two images constitutes the decoded, or reconstructed, sub-sequence.

[0119] [Fig.5] illustrates an example of selection of reference cards in a video sequence.

[0120] The sequence shown in [Fig.5] comprises 4 images, numbered from 1 to 4. For example, it is an Intra period of a standard video coder, or a GOP, etc. It is made up of 3 sub-sequences v=l, v={2,3] and v=4. The reference maps FRj (resp. FRdj) of the sequence are 3 in number (NFR = 3) and of different resolutions.

[0121] The subsequence v={2,3] is coded (resp. decoded) using 3 current maps FM{2,3 (,-1.2.1 (resp. FMd{2,3 (,-,.2.0 and a reference map FR{2,3 (,-,(resp. FRd{2,3 (ji) of the sequence. In total, 4 feature maps are presented (after having been possibly transformed by the SE / SE' module, then their feature vectors extracted by the XTR / XTR' module) at the input of the MLP synthesis network (resp. MLP') which synthesizes the samples of the coded (resp. decoded) images 2 and 3.

[0122] The subsequence v={4] is coded (resp. decoded) using 3 current maps FM4,-i 2.i (resp. FMd4,-, 2.i) and three reference maps FR4J-,2i(rcsp. FRd4,-i 2.i) of the sequence. In total, 6 feature maps are presented (after having been possibly transformed by the SE / SE' module, then their feature vectors extracted by the XTR / XTR' module) at the input of the MLP synthesis network (resp. MLP') which synthesizes the samples of the coded (resp. decoded) image number 4.

[0123] The numerical examples chosen are indicative.

[0124] [Fig.6] is a flowchart representing an example of a coding method that can be implemented by the coding device of [Fig.l].

[0125] According to one embodiment, the current sub-sequence is a sequence of two-dimensional images, each sample sub-sequence to be coded is therefore a set of pixels Pvn of coordinates (xn, yn) in the respective images I(Pvn) of the sequence to be coded. According to one embodiment, the current subsequence comprises a single image.

[0126] Encoding takes place in two main phases:

[0127] In a first phase, called the construction phase, a first learning process is carried out in the initialization step E0 to construct, code and decode a group of reference images FRCj of the sequence SV. According to one embodiment, SV comprises a series of images, for example temporal images of a video sequence. According to one embodiment, the images are multi-view or 3D components of a multi-view or 3D image or sequence of images. According to one embodiment, the images correspond to an Intra period or a GOP of a conventional video coder.

[0128] These maps can be inserted into the B1 stream after coding. According to one embodiment, these maps are generated before the processing of the subsequences. According to another embodiment, they are generated during the processing of the subsequences, by optimization, in the same way as the current characteristic maps.

[0129] For an input subsequence v comprising at least one image I(Pvn), learning is performed in order to determine the values ​​of the maps FMv; and of the parameters Wk to optimize a global cost function. The learning is for example performed by gradient descent, followed by an update of the parameters of the MLP neural network and the values ​​of the current and / or reference feature maps. As is known in the state of the art, the cost function can be of the rate-distortion, or rate, or distortion, or perceptual type. To measure the rate R, it is necessary to simulate the coding of the maps FMv;, optionally of the maps FRvj, then to measure the associated coding rate (the size of the flow Bl). According to one embodiment, the coding of the parameters Wk is not simulated because their influence is less important than that of the feature maps.According to one embodiment, the coding of the parameters Wk is also simulated and the associated flow rate (the size of the flow B2) is measured. To measure the distortion D, it is necessary to simulate the coding then the decoding of at least part of the sequence of images, to obtain at least one sequence of pixels P'vn resulting from a simulation of coding then decoding of the samples of index n, then to measure the difference between this sub-sequence of images I(Pv n) at input and a corresponding part of the sequence I(P'vn) coded then decoded.

[0130] Then during a second phase, called the coding phase, the maps FMv; and the parameters Wk are encoded to produce the coded values ​​FMcv; and Wcvk before transmission or storage. They constitute the compressed representation of the input sequence I(Pvn).

[0131] We will now describe the steps of a method according to one embodiment of the invention.

[0132] During a step E20, an input sequence I(Pvn) to be coded, comprising at least an image comprising a plurality of N samples Pvn is provided as input to the method.

[0133] During a step E21, NFM current and optionally reference feature maps are initialized. Subsequently, the Wk parameters of the MLP neural network and the values ​​of the maps must be optimized during the construction phase.

[0134] According to one embodiment, the characteristic maps are initialized by predefined constant values. According to another embodiment, the characteristic maps are initialized by a set of random real values. The characteristic maps are subsequently updated, or refined, during a step E22, by the update module MAJ of the encoder during its learning.

[0135] During a step E23, NFR reference maps are obtained by the REF module, from the group of feature maps associated with the sequence. The REF module constitutes a set of NFR reference feature maps, denoted FRvj, from a group of previously decoded reference feature maps FRj, numbered NFRMAX (NFR <= NFRMAX). For example, the reference feature maps considered for the sub-sequence being encoded are all those of the reference group (FRvj= FRj) or are selected as a subset of the reference group. In the latter case, they may be predetermined. For example, reference feature maps of the same size as the current feature maps may be used. Their reference may alternatively be encoded in the stream in the form of an indicator. For example, the reference map used for encoding / decoding the images v={2,3] of [Fig.5] is image number 1 in the reference card group. This indication can be encoded in the stream. .

[0136] The set of current NFM maps and reference NFR maps constitutes a set of feature maps for the MLP synthesis network.

[0137] According to one embodiment, the current and / or reference characteristic maps have the same resolution as the images of the input sequence I(Pvn) and therefore each comprise the same number of values ​​N as there are samples Pvn to be coded in each image. According to one embodiment, the current and / or reference characteristic maps have a resolution less than or equal to that of the images of the input sequence I(Pvn) and therefore comprise, for at least one of them, a number N' of values ​​to be coded less than N. According to one variant, the first current map (resp. reference map) is at the resolution of the images and each subsequent map is at a resolution half that of the previous one. According to one embodiment, several maps have the same resolution, less than that of the images of the input sequence I(Pvn).

[0138] During a step E24, according to one embodiment, the NFM maps of the set of current maps FMv; and / or NFR reference maps FRvj are transformed by the SE module to generate transformed maps FMSv; / FRSvj at the resolution of the input images. In this mode, the feature vectors are preferably extracted from the transformed maps, and not directly from the current maps to be coded and / or the reference maps. The transformed maps are neither stored nor transmitted, they are only used for the construction of the feature vectors.

[0139] According to one embodiment, one or more additional maps, forming an additional group of additional feature maps, are generated, and complete the set of feature maps constituted. They are used for the construction of the feature vector but are neither stored nor transmitted.

[0140] According to one embodiment, NFM + NFR FMSv; / FRSvj maps are generated. According to one embodiment, each FMv; map is transformed into an FMSv; map. According to one embodiment, each reference FRvj map is transformed into an FRSvj map.

[0141] According to one embodiment, at least one FMvi / FRvj map has a lower resolution than the images of the sequence to be coded and the transformation operation comprises oversampling so that the transformed map has the same number of samples as the images of the sequence. Oversampling consists of adding values ​​to the FMSv; and / or FRSvj maps to achieve the resolution of the images of the input sequence. It can be simple (by nearest neighbor replication) or include interpolation (linear, polynomial, by filtering, etc.). The transformation can include an oversampling neural network.

[0142] During a step E25, values ​​are extracted by the XTR module: - FMv current feature maps; (or possibly FMSv; transformed, and optionally additional) - FRvj reference feature maps. or possibly transformed FRSvj.

[0143] This extraction is carried out according to the coordinates (xn, yn) of the current subsequence of samples Pvn of the current subsequence. It can also be carried out according to the resolution of the map considered. The sequences of samples to be coded are for example processed in sequential order, from n=1 to n=N.

[0144] According to one embodiment, the characteristic Zn vector results directly from this extraction.

[0145] According to one embodiment, the characteristic vector Zn is constructed by processing the extracted values. The processing may include quantization of the extracted values ​​or of the constituted vector Zn, if necessary. The processing may include other operations, such as filtering, scaling, application of any function, preferably monotonic, etc. In one embodiment, Zn is a J-uplet constructed from values ​​taken from the current and reference (and optionally additional) feature maps by applying processing to one or more values ​​of the maps, for example filtering the values ​​close to the targeted value in a map. For example, in a map that is at the same resolution as the input signal, one can extract the values ​​located at the coordinates (xn, yn), (xn-l, yn), (xn, yn-l) and (xn-l, yn-l) and apply processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (¾) of the element i of the vector Zn relative to this map. According to another example, in a map which is at half the resolution of the input signal, we can consider the values ​​located at the coordinates (xn / 2, yn / 2), (xn / 2-l, yn / 2), (xn / 2, yn / 2-l) and (xn / 2-l, yn / 2-l) and apply a processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (¾) of element i of the vector Zvn or Zrn relative to this current, reference (or additional) characteristic map.

[0146] In one embodiment, Zn comprises as many values ​​as there are input cards.

[0147] In one embodiment, Zn is a J-tuple comprising values ​​located at the co ordinates (xn, yn) of a current pixel Pvn, as will be illustrated in support of [Fig.8]. The vector Zn of index n refers to the characteristic vector of the pixel or sequence of pixels Pvn. According to one embodiment, J = NFM + NFR. The vector Zn (zvi... zvNFM, zr i... zrNFR), comprises two sets of values: one (Zvn) is extracted from the current characteristic maps of the subsequence v, and the other (Zrn) is extracted from the reference characteristic maps. Optionally, it can comprise values ​​from the additional maps.

[0148] In one embodiment, Zn is a J-tuple constructed from values ​​taken from the current and reference (and optionally additional) feature maps at coordinates that may be different across maps. For example, if the maps are at different resolutions, the coordinates are adapted (by scaling) to match the resolution of each map.

[0149] During a step E26, the vector Zn is processed by the MLP neural network to generate as output the sub-sequence of samples Pvn to be coded, according to one embodiment the triplets (R, G, B) of the samples P'vn (the coded then decoded Pvn samples) of the images of the sub-sequence.

[0150] During a step E27, the current FMv; maps are coded by the FMC module of the encoder. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is an actual coding and the coded values ​​constitute the stream B1. The simulation can be identical to the actual coding but it can also be different (for example, simplified). For this coding, any known technique can be used for compressing the values ​​of the maps. In one embodiment, the reference maps are also coded at this step.

[0151] In one embodiment, the maps are coded in order (FMv1, FMv2,..., FMv4), and the variables of each map in a predefined order, for example lexico-graphic. In one embodiment, the reference maps are also coded in order. Each map undergoes entropy coding. The entropy coding produces a compressed stream B1 whose bit rate is subsequently measured during a step E29.

[0152] During a step E28, the Wk parameters of the MLP neural network are quantized and coded. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is an actual coding and the coded values ​​constitute the stream B2. The simulation can be identical to the actual coding but it can also be different (for example, simplified). Any known technique can be used for this purpose, for example the neural network coding standard proposed by the MPEG-7 part 17 standard, also called NNR (Neural Network Representation). It should be noted that in this case, the amount of degradation that the coding brings to the Wk parameters must be chosen. The structure and the Wk parameters of the neural network are initialized for example during the first iteration of this step. These parameters are subsequently updated, or refined, during the construction phase, during subsequent iterations of the method.

[0153] According to one embodiment, the parameters of the neural network are initialized by predefined values ​​known to give a satisfactory result (for example, following training on a corpus of image sequences). According to another embodiment, the parameters Wk of the neural network are initialized by a set of random values.

[0154] During a step E29, a performance measurement is evaluated.

[0155] For this purpose, the coding simulation rates associated with the characteristic maps (simulation of the B1 flow by coding the FMv; / FRvj maps) and optionally with the parameters of the neural network (simulation of the B2 flow by coding the Wk parameters) are measured.

[0156] According to one embodiment, the cost function is of the rate-distortion type, noted (D+L*R), where D for example the quadratic error measured between the input images and the decoded images (or the error measured on a subset of samples of the images). According to another example, D is calculated from a perceptual function such as the SSIM (for Structural SIMilarityy or MSSSIM (for Multi-scale Structural Sl-Milarity). According to one embodiment, R is the simulated rate of the stream B1; according to another embodiment, R is the total rate used to code this image, that is to say the sum of the simulated rates of B1 and B2. L is a parameter which regulates the rate-distortion trade-off. Other cost functions are possible.

[0157] As long as the cost function has not reached its minimum, the performance measurement is not satisfactory, and the method is repeated from step E22. Alternatively, the process can be interrupted after a predefined time or a predefined number of iterations, so as to control its complexity or duration. This minimization can be carried out by a mechanism known as gradient descent with updating of the parameters during step E22 for the values ​​of the feature maps and E27 for the network parameters.

[0158] During a step EF, if the cost function has reached its minimum, the training stops. If a coded version corresponding to the last simulation of the parameters of the neural network and the feature maps is available, the streams B1 and B2 can be constituted from them. According to another embodiment, the actual coding of the updated parameters of the neural network and the values ​​of the feature maps (FMvj) is carried out at this step to produce the encoded parameters Wck and FMc; which constitute streams B1 and B2. According to one embodiment, the maps FRvj are coded at the end of the training, when the entire sequence has been processed.

[0159] The streams B1 and B2 can be concatenated to produce a final stream. According to one embodiment, the stream B2 of the coded parameters of the neural network is stored or transmitted before the stream B1, so that it can be decoded before the stream B1.

[0160] [Fig.6] is a flowchart representing an example of a decoding method that can be implemented by the decoding device of [Fig.2].

[0161] The decoding described relates to a sub-sequence of images comprising at least one image of the sequence to be decoded, it is assumed that the group of reference maps of the sequence FRCj has been decoded by the initialization module. According to one embodiment, a technique for predicting a feature map value by its neighborhood is used, as described in support of [Fig.l]. In one embodiment, the reference maps are decoded in order, and the values ​​of each map in a predefined order, for example lexicographic.

[0162] During a step E30, the streams B1 and B2 are extracted from the encoded stream. They contain respectively, for a current subsequence of the sequence, the coded representations of the current maps FMcv;, and of the parameters Wcvk of the neural network.

[0163] During a step E31, the NFM maps FMdv; of the current sub-sequence are generated by decoding the values ​​FMcv;. For this decoding, entropic decoding is used, for example. According to one embodiment, a technique for predicting a feature map value by its neighborhood is used, as described in support of [Fig.l]. In one embodiment, the maps FMdv; are decoded in the order (FMdvi, FMdv2,... FMdv4), and the values ​​of each map in a predefined order, for example lexicographic.

[0164] During a step E32, NFR reference maps are obtained by the module REF', from among the group of characteristic maps associated with the sequence. The module REF' constitutes a set of NFR reference characteristic maps, denoted FRdvj, from the group of previously decoded reference feature maps FRdj, to the number NFRMAX (NFR <= NFRMAX). For example, the reference feature maps considered for the current subsequence are all those of the reference group (FRvj= FRj) or are selected as a subset of the reference group. In the latter case, they can be predetermined. For example, reference feature maps of the same size as the current feature maps can be used. Their reference can alternatively be encoded in the stream as an indicator. For example, the reference map used for decoding images v={2,3} in [Fig.5] is image number 1 in the group of reference maps. This indication can be read and decoded in the stream. On the contrary, all reference maps are used for decoding image v= 4 in [Fig.5].So there is no need to read this indication in the stream.

[0165] The set of current NFM maps and reference NFR maps constitutes a set of feature maps for the synthesis network.

[0166] According to embodiments as described for the encoder:

[0167] - The current and / or reference feature maps are of the same resolution than the sequence I (Pdvn) to be reconstructed, that is to say that they contain N values.

[0168] - Current and / or reference feature maps are at an in resolution less than or equal to that of the subsequence I (Pdvn) to be reconstructed. According to a variant, the first current map (resp. reference map) is at the resolution of the images and each following map is at a resolution half that of the previous one.

[0169] - Several common and / or reference characteristic cards are also re solution, less than the resolution of the subsequence.

[0170] During a step E33, according to one embodiment, the NFM maps of the set of current maps FMdv; and / or the NFR reference maps FRdvj are transformed by the module SE' to generate transformed maps FMS'v; / FRS'Vj at the resolution of the images to be decoded. This step is similar to step E24 which was described for the coder with the support of [Fig.6], and the embodiments apply. In particular:

[0171] According to one embodiment, NFM + NFR maps FMS'v; / FRS'Vj are generated.

[0172] According to one embodiment, each FMdv card; is transformed into a card FMS'Vi. According to one embodiment, each FRdvj card is transformed into a FRS'Vj card.

[0173] According to one embodiment, at least one FMdv; / FRdvj map has a lower resolution than that of the images to be decoded and the transformation operation comprises oversampling so that the transformed map has the same number of samples as the input image. Oversampling consists of adding values ​​to the FMS'Vi and / or FRS'Vj maps to achieve the resolution of the image to be decoded. decode. It can be simple (by nearest neighbor replication) or include interpolation (linear, polynomial, filtering, etc.). The transformation can optionally include inverse quantization of the extracted values, if necessary. However, inverse quantization is not mandatory. The transformation can include an upsampling neural network.

[0174] During a step E34, values ​​are extracted by the XTR' module: - FMdv current feature maps; (or possibly FMS'v; transformed, and optionally additional) - reference characteristic maps FRdvj or possibly FRS'Vj transformed.

[0175] This extraction is carried out as a function of the coordinates (xn, yn) of the subsequence of current samples Pvn of the current subsequence. It can also be carried out as a function of the resolution of the map considered. This step is similar to step E25 which was described for the encoder in support of [Fig.6], and the embodiments apply. In particular:

[0176] In one embodiment, Zdn comprises as many values ​​as there are input cards.

[0177] According to one embodiment, the characteristic vector Zdn is constructed by processing the extracted values. The processing may include quantification of the extracted values ​​or of the constituted vector Zdn, if necessary. The processing may include other operations, such as filtering, scaling, application of any function, preferably monotonic, etc. The examples cited for the encoder apply.

[0178] In one embodiment, Zdn is a J-tuple comprising values ​​located at the coordinates (xn, yn) of a current pixel Pdvn, as will be illustrated in support of [Fig.8]. The vector Zdn of index n refers to the characteristic vector of the pixel or of the sequence of pixels Pdvn. According to one embodiment, J = NFM + NFR. The vector Zdn (zdvi...zdvNFM, zdri...zdrNFR), comprises two sets of values: one (Zdvn) is extracted from the current characteristic maps of the sub-sequence v, and the other (Zdrn) is extracted from the reference characteristic maps. Optionally, it can comprise values ​​from the additional maps.

[0179] In one embodiment, Zdn is a J-tuple constructed from values ​​taken from the current and reference (and optionally additional) feature maps at coordinates that may be different across maps. For example, if the maps are at different resolutions, the coordinates are adapted (by scaling) to match the resolution of each map.

[0180] The samples to be decoded are for example processed in sequential order, from n=1 to n=N.

[0181] During a step E35, the Wdvk parameters of the MLP' synthesis neural network du are generated by decoding the Wcvk values ​​of the B2 stream. Any known decoding technique corresponding to the coding technique used by the encoder can be used for this purpose. The MLP synthesis neural network is similar to the MLP synthesis network, i.e. it has the same structure and the same parameters, except for the coding, which can be done with or without losses.

[0182] During a step E36, the characteristic vector Zdn is processed by the synthesis neural network MLP' to generate as output a vector representative of the sample or subsequence of samples Pdvn to be decoded, according to one embodiment one or more triplets which are injected into the decoded image I (Pdvn) at the positions (xn, yn) of the color components (Rd, Gd, Bd). This step is similar to step E26 which was described for the coder with the support of [Fig.6].

[0183] When all the samples of the signal have been processed, the current decoded sub-sequence corresponding for example to image I (Pdvn) is available.

[0184] According to one embodiment, the stream B2 is decoded before the stream B1 (with the exception of the reference maps), in order to be able to have the MLP' synthesis neural network before starting to decode the samples.

[0185] [Fig.8] illustrates a decoding method used in one embodiment of the invention.

[0186] In this embodiment, the current decoded cards FMdv; are 3 in number. In a preferred embodiment, they are 7 in number.

[0187] In this embodiment, the first FMdvi map has the same resolution as the image I(Pdvn) to be decoded, and therefore comprises WxH variables, where W represents the width of the image in pixels, and H its height. The second FMdv2 map has half the resolution (in each dimension) of the FMdvb map. Each additional map has half the resolution of the previous map. This structure makes it possible to reduce the number of variables in the feature maps, which facilitates decoding while minimizing the coding cost.

[0188] The FMdv2 map is oversampled by a factor of 2 by the SE' module in each dimension, according to any oversampling method within the reach of those skilled in the art. The FMdv3 map is oversampled by a factor of 4 in each dimension.

[0189] In this embodiment, the decoded reference maps FRdj are 3 in number. In a preferred embodiment, there are 7 of them. In this embodiment, the first map FRdi has the same resolution as the image, and therefore comprises WxH variables, where W represents the width of the image in pixels, and H its height. The second map FMd2 has half the resolution (in each dimension) of the map FMdi. Each additional map has half the resolution of the previous map. This structure makes it possible to reduce the number of variables in the characteristic maps, which facilitates decoding while minimizing the coding cost.

[0190] The three cards of the reference group are selected by the decoder to constitute the reference set. (FRdvi=FRdi; FRdv2 =FRd2; FRdv3=FRd3).

[0191] The map FRd2 is oversampled by a factor of 2 in each dimension by the module SE', according to any oversampling method within the reach of those skilled in the art. The map FRd3 is oversampled by a factor of 4 in each dimension.

[0192] The transformed maps FMS'v; and FRS'Vj are of the same resolution as the image to be decoded, and therefore comprise WxH values, where W represents the width of the image in pixels, and H its height.

[0193] In this embodiment, the vector Zdn is a 6-tuple (zv3> zv2zvi zr3, zr2, zrj consisting of the 3 values ​​of the FMS'v maps; located at the coordinates (xn, yn) of the current pixel Pdvn. and the 3 values ​​of the FRS'Vj maps located at the coordinates (xn, yn) of the current pixel Pdvn. The vector Zdn is optionally dequantized then processed by the MLP' synthesis neural network to generate as output the triplets (R, G, B) representative of the samples Pdvn to be decoded. The triplet (R, G, B) is inserted into each decoded image I (Pdvn) of the sub-sequence v at the coordinates (xn, yn) in the color components (Rd, Gd, Bd).

[0194] According to a variant presented in dotted lines, there are 7 cards: an additional card denoted FME'o has been introduced. In this embodiment, the vector Zdn is a 7-tuple.

[0195] This additional map typically includes data that can assist the MLP network in the task of reconstructing images. Thus, the added maps can be one or more of the following non-limiting list:

[0196] - A map showing at each point the abscissa of that point.

[0197] - A map showing at each point the ordinate of that point.

[0198] - A map comprising a positional coding at each point.

[0199] - A map representing an image distinct from the images being processed, above capable of providing information about the images to be coded, for example a previously processed image or sequence of images.

[0200] - A map containing data representative of the time difference between the frames of the video being encoded. For example, if the first and last frames of the video are 8 frames apart, all samples in the map contain the value 8.

[0201] - A map representing a feature map of a distinct image of the images being processed, which may provide information about the images to be coded, for example a previously processed map.

[0202] - A card containing the value of an already decoded sample of the same card, for example example the previous sample in the decoding order.

Claims

Claims

1. Method for coding a sequence of images (Sv) comprising the following steps for at least one current image to be coded: - a construction step, comprising the following sub-steps: - construction (E21, E22) of a set of current characteristic maps (FMv;) representative of said at least one current image (v, I(Pvn)), comprising at least one current characteristic map; - construction (E23) of a set of reference characteristic maps (FRvj), comprising at least one coded then decoded reference characteristic map;- for at least one sample, called current sample (Pvn), of said at least one current image, associated with a position (xn, yn) in said at least one current image: - construction (E25) of a characteristic vector (Zn) from said at least one reference characteristic map (Zrn) and said at least one current characteristic map (Zvn), as a function of said position (xn, yn) of said at least one current sample (Pvn); - processing (E26) of said characteristic vector (Zn) by an artificial neural network (MLP), called a synthetic neural network, defined by a set of parameters (Wvk), to provide a vector (P'vn) representative of a decoded value of said at least one current sample; - updating (E22, E29) of at least one value of said at least one current characteristic map and / or of at least one parameter of said network, as a function of a coding performance measurement;- a coding step (E24) of said set of current feature maps (FMv;); - a coding step (E28) of said set of parameters (Wvk) of said synthetic neural network.;

2. Method for coding a sequence of images according to claim 1, characterized in that it further comprises: - a step of construction (FRj) and coding (FRCj) of a group of reference characteristic maps representative of the sequence comprising said set of reference characteristic maps (FRvj).

3. Method for decoding a sequence of images (Sv) of a binary stream, comprising the following steps for at least one current image (v): - decoding (E35) of a set of parameters (Wdvk) representative of a synthesis neural network (MLP'); - decoding (E31) of a set of current characteristic maps (FMdVi) comprising at least one current characteristic map representative of said at least one current image; - obtaining (E32) of a set of reference characteristic maps (FRdvj), comprising at least one decoded reference characteristic map;- for at least one sample, called current sample (Pdvn), of said at least one current image (v) to be decoded, associated with a position (xn, yn) in the image: - construction (E34) of a characteristic vector (Zdn) from said at least one reference characteristic map (Zdrn) and said at least one current characteristic map (Zdvn), as a function of said position (xn, yn) of said at least one current sample, and: - processing (E37) of said vector (Zdn,ZRdn) by said synthesis neural network (MLP') to provide a vector representative of a decoded value of said at least one current sample (Pdvn).;

4. Decoding method according to claim 3, characterized in that it further comprises: - a step of decoding (E31) a group of reference characteristic maps (FRdj) representative of the sequence and - obtaining (E32) said set of reference characteristic maps (FRdvj) from said group (FRdj).

5. Coding or decoding method according to one of the preceding claims, characterized in that the set of reference maps and the set of current maps (FRdvj, FMdv;) contain the same number (NFM, NFR) of characteristic maps.

6. Encoding or decoding method according to one of the preceding claims, characterized in that said at least one reference map is at the same resolution as said at least one current map.

7. Encoding or decoding method according to one of the preceding claims, characterized in that said at least one reference map is indicated in the binary stream.

8. Encoding or decoding method according to one of the preceding claims, characterized in that said at least one reference map is predetermined.

9. Coding or decoding method according to one of the preceding claims, characterized in that it further comprises a step of trans- forming said set of current and / or reference feature maps to obtain a set of transformed feature maps at the resolution of the current image, the method being characterized in that said feature vector is constructed from said transformed feature maps.

10. Decoding method according to one of claims 3 to 9, characterized in that said at least one current map is decoded as a function of said at least one reference map and / or a current characteristic map.

11. Decoding method according to claim 10 characterized in that the decoding is carried out by a prediction neural network.

12. Device for coding a sequence of images (Sv), said device being configured to implement: - a construction step, comprising the following sub-steps: - construction (GEN, MAJ) of a set of current characteristic maps, (FMv;) representative of said at least one current image (v, I(Pvn)), comprising at least one current characteristic map; - construction (INITc, REF) of a set of reference characteristic maps (FRvj), comprising at least one coded then decoded reference characteristic map;- for at least one sample, called current sample (Pvn), of said at least one current image, associated with a position (xn, yn) in said at least one current image: - construction (XTR) of a characteristic vector (Zn) from said characteristic map, at least one reference characteristic map (Zrn) and said at least one current characteristic map (Zvn), as a function of said position (xn, yn) of said at least one current sample (Pvn); - processing (MLP) of said characteristic vector (Zn) by an artificial neural network, called a synthetic neural network, defined by a set of parameters (Wvk), to provide a vector (P'vn) representative of a decoded value of said at least one current sample;- updating (E22, E29) at least one value of said at least one current characteristic map and / or at least one parameter of said network, as a function of a coding performance measurement, - a coding step (E24) of said set of current characteristic maps (FMv;); - a coding step (E28) of said set of parameters (Wvk) of said; synthetic neural network.

13. Device for decoding from a binary stream of a sequence of images (Sv), said device being configured to implement, for at least one current image (v, I(Pdvn)) of the sequence: - decoding (NND) of a set of parameters (Wdvk) representative of a synthesis neural network (MLP'); - decoding (FMD) of a set of current feature maps comprising at least one current feature map (FMdv;) representative of said at least one current image; - obtaining (INITd, REF') a set of reference characteristic maps (FRdVj), comprising at least one decoded reference characteristic map; - for at least one sample, called current sample (Pdvn), of said at least one current image (v) to be decoded, associated with a position (xn, yn) in the image: - construction (E34) of a characteristic vector (Zdn) from said at least one reference characteristic map (Zdrn) and said at least one current characteristic map (Zdvn), as a function of said position (xn, yn) of said current sample, and: - processing (E37) of said vector (Zdn) by said synthesis neural network (MLP') to provide a vector representative of a decoded value of said at least one current sample (Pdvn).

14. A computer program comprising instructions for carrying out the steps of an encoding or decoding method according to claim 1 or 3 when said program is executed by a computer.

Images