Method and device for encoding and decoding image sequences

Abstract

The invention relates to a method and device for encoding and decoding an image sequence. The decoding method comprises the following steps: - decoding (E35) a set of parameters representing an artificial neural network; - decoding (E31) a set of current feature maps including at least one current feature map representing a current image; - obtaining (E32) a set of reference feature maps, including at least one decoded map; and - for at least one current sample of the at least one current image, associated with a position in the image: - constructing (E34) a feature vector from the reference feature map and from the current feature map, according to said position of the current sample; and - processing (E37) of the vector by the artificial neural network in order to provide a vector representing a decoded value of the current sample.

Description

Method and device for coding and decoding image sequences.

[0001] The invention relates to the general field of coding digital image sequences. It relates more particularly to the compression of digital videos.

[0002] Digital videos are generally source-encoded to compress them in order to reduce the resources required for transmission and / or storage. There are many encoding standards, such as the ITU / MPEG standards (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) and their extensions (MVC, SVC, 3D-HEVC, etc.).

[0003] Encoding an image is generally performed by dividing the image into several rectangular blocks, and encoding these blocks of pixels according to a given processing sequence. In existing video compression techniques, the processing of a block typically includes a prediction of the pixels of the block performed using previously coded and then decoded pixels present in the image being encoded, in which case we speak of "Intra prediction", or previously coded images, in which case we speak of "Inter prediction". This exploitation of spatial and / or temporal redundancies makes it possible to avoid transmitting or storing the value of the pixels of each block of pixels, by representing at least some of the blocks by a residual representing a difference between the prediction values ​​of the pixels of the block and the actual values ​​of the pixels of the predicted block.

[0004] As video formats continue to evolve to compress more and more and adapt to the variety of expected formats and communication networks, the possibilities for prediction are becoming ever greater and traditional coding and decoding algorithms are becoming very complex.

[0005] In addition to these classic approaches proposed by compression standards (MPEG, ITU), approaches based on artificial intelligence, and in particular neural ones, tend to develop.

[0006] Some of these neural approaches can be seen as a simple extension of the competition notion of the aforementioned compression techniques, such as the prediction and transformation mode competition in video coding.

[0007] Other approaches use the concept of "autoencoder." Autoencoders are learning algorithms based on artificial neural networks, which allow the construction of a new representation of a dataset. The architecture of an autoencoder consists of two parts: the encoder and the decoder. The encoder consists of a set of layers of neurons, which process the data in order to construct new representations called "encoded," also called "latent representations." In turn, the layers of neurons in the decoder receive these representations and filter them in order to try to reconstruct the initial data. The differences between the reconstructed data and the initial data make it possible to measure the error made by the autoencoder. Training consists of modifying the parameters of the autoencoder in order to reduce the reconstruction error measured on the different samples of the dataset.The performance of such autoencoder-based systems comes at the cost of a considerable increase in memory footprint and complexity compared to conventional approaches as proposed by compression standards. They can have millions of parameters and may require up to a million MACs (multiply-accumulate) to decode a single pixel. This makes such decoders significantly more complex than conventional decoders, which could hamper the adoption of learning-based compression.

[0008] More recently, a simple image encoding technique based on a neural network was described in the article "COmpression with Implicit Neural representations" by Émilien Dupont et al. (arXiv:2103.03123). The proposed encoding technique consists of fitting a neural network to an image, quantizing the network weights, and transmitting them. At decoding time, the neural network is evaluated at each pixel position to reconstruct the image. However, such a technique remains inefficient in terms of compression and requires coding the video frames independently.

[0009] In the video domain, an equivalent technique was presented in the article “Scalable Neural Video Representations with Learnable Positional Features” (NeurIPS 2022) by Subin Kim et al. The temporal dimension is taken into account. The coding algorithm generates a set of three two-dimensional latent keyframes, on each spatio-temporal axis, to learn the common frames of the video. It also generates a latent 3D representation grid to capture the local details of the video. This representation is used to modulate a neural network trained to process the three keyframes. However, such an approach lacks flexibility in the choice of representation of the keyframes (the number of which is fixed at 3) and the 3D grid (fixed for the entire sequence). Moreover, it requires the processing of a transmitted 3D grid, which is both heavy in terms of the amount of transmitted data and the memory required for its processing.

[0010] There is therefore a need for a solution that allows a sequence of images to be encoded / compressed in a simple and efficient manner.

[0011] The invention relates to a method for coding a plurality of images of an image sequence comprising the following steps for at least one current image to be coded: - a construction step, comprising the following sub-steps: - construction of a set of current characteristic maps, representative of said at least one current image, comprising at least one current characteristic map; - construction of a set of reference characteristic maps, comprising at least one coded and then decoded reference characteristic map; - for at least one sample, called current sample, of said at least one current image, associated with a position in the current image: - construction of a characteristic vector from said at least one reference characteristic map and said at least one current characteristic map, as a function of said position of said current sample;- processing of said characteristic vector by an artificial neural network, called a synthetic neural network, defined by a set of parameters, to provide a vector representative of a decoded value of the current sample; - updating of at least one value of said at least one current characteristic map and / or of at least one parameter of said network, as a function of a coding performance measurement; - a step of coding of said set of current characteristic maps; - a step of coding of said set of parameters of said synthetic neural network.;

[0012] The invention also relates to a method for decoding a sequence of images of a binary stream, comprising the following steps for at least one current image to be decoded: - decoding a set of parameters representative of a synthetic neural network; - decoding a set of current characteristic maps comprising at least one current characteristic map representative of said at least one current image; - obtaining a set of reference characteristic maps, comprising at least one decoded reference characteristic map;- for at least one sample, called current sample, of said at least one current image to be decoded, associated with a position in the image: - construction of a characteristic vector from said at least one reference characteristic map and said at least one current characteristic map, as a function of said position of said current sample, and: - processing of said vector by said synthesis neural network to provide a vector representative of a decoded value of said at least one current sample.;

[0013] For the purposes of the invention, encoding, or "coding", means the operation which consists of representing a set of samples, or pixels, in a compact form carried for example by a digital binary train. Decoding means the operation which consists of processing a digital binary train to restore decoded samples.

[0014] By "sequence of images" is meant a plurality of ordered two-dimensional images, for example temporally in the case of a video. According to one example, the sequence corresponds to a scene. According to one example, the sequence corresponds to a set of predefined images, for example a fixed number, or, within the meaning of the MPEG standards, a GOP (Group Of Pictures) comprising the images located between two images of the Intra-image type, also called "intra period". According to another example, the images can be views of the same scene represented in multi-views. According to another example, the images can be a plurality of temporal and multi-view images (immersive video).

[0015] By "at least one current image" we mean a subsequence of at least one image of the sequence.

[0016] A "sample" is a value taken from an image in the sequence. Sampling a signal produces a series of discrete values ​​called samples. In the case of an image signal, the sample is called a pixel, which can be, for example, a color pixel traditionally represented by a triplet of values, for example (R,G,B) or (Y,U,V). The position of the sample is identified by its abscissa (x) and ordinate (y) coordinates in the image.

[0017] A "feature map" means a part of an abstract representation of an image, comprising a plurality of variable scalar data, also called characteristic values, for example real numbers, or integers, signed or unsigned. The abstract representation of an image may therefore comprise several feature maps. The abstract representation of an image may also correspond to a single feature map comprising a set of values ​​associated with a position.

[0018] A reference feature map is a feature map obtained for all images in the sequence. In contrast, a current feature map concerns only the current subsequence.

[0019] By "feature vector constructed from feature maps as a function of a position" is meant a vector consisting of one or more elements, or data, preferably discrete, the data being constructed from the feature maps at a position determined by that of the samples being processed in the images. This feature vector is the one that is applied to the input of the synthetic neural network. In the case of an image, such a vector can be constituted for example from a plurality of values ​​taken from each of the feature maps at the same abscissa and ordinate coordinates as the subsequence of samples to be coded (resp. to be decoded).

[0020] By "synthetic neural network" we mean a neural network such as a convolutional neural network, a multi-layer perceptron, an LSTM (for "Long Short Term Memory"), etc. The neural network is defined for example by a plurality of layers of artificial neurons and by a set of activation, weighting and addition functions (for example, a layer can calculate y = f(Ax+b), where y and b are vectors of dimension N, x is a vector of dimension M, A is a matrix of dimension MxN, and f is the activation function).

[0021] By "neural network parameter" we mean one of the values ​​that characterizes the neural network, for example a weight associated with one of the neurons (filter coefficient, weighting, bias, value affecting the operation of non-linearity, etc.)

[0022] By "processing by a synthetic neural network" is meant the application of a function expressed by a neural network to the input characteristic vector to produce an output vector representative of the subsequence of samples to be coded (resp. decoded). This output vector may include one or more data representative of the subsequence of samples.

[0023] By "performance measurement" is meant a measurement between at least one value of a sample to be coded and a decoded value of said sample. The measurement can evaluate, for example, a distortion, or a perceptual error. It can be carried out on a sample or a plurality of samples (for example, the current samples, or the current images, etc.). The measurement can also include a measurement of the flow rate, in particular associated with the coding of the neural network and / or the coding of the feature maps. The measurement can be a joint measurement between the flow rate and the distortion through their weighting. As is well known in the state of the art, the value of this measurement is generally minimized until a target value is reached.

[0024] A "construction step" means a step that aims to construct the representative parameters of the image, before their actual coding. The construction sub-steps can be repeated as many times as necessary to obtain an acceptable performance measure.

[0025] Generally speaking, it is considered that the steps of an coding or decoding method should not be interpreted as being linked to a notion of temporal succession. In other words, the steps may be carried out in a different order than that indicated in the independent coding or decoding claim, or even in parallel.

[0026] The coding method according to the invention constructs the coding parameters, from a sequence of input images, by training a neural network on characteristic vectors associated with a position of a sample to be coded in one of the input images of a sub-sequence of images being processed. The sub-sequence comprises at least one image. These characteristic vectors are constructed from characteristic maps of the image itself, and reference characteristic maps of the sequence, representative of the images of the sequence. The characteristic maps may be at the resolution of the input images, or at a lower resolution. During the training, or construction, the parameters of the neural network and the values ​​of the characteristic maps are updated according to a performance measure, for example of the rate-distortion type.When the training is complete, i.e. the performance measurement obtained is satisfactory, the actual coding of the parameters of the neural network and the values ​​of the current feature maps can be performed and stored or transmitted to the decoder. According to one embodiment, when the entire sequence has been processed, the reference maps are stabilized and can be encoded and stored or transmitted to the decoder.

[0027] Advantageously, the training process makes it possible to refine the parameters of the synthetic neural network and / or the values ​​of the feature maps until an adequate representation in terms of performance is obtained, for example a desired balance between the generated bit rate and the distortion undergone by the image or sub-sequence of input images being encoded. The training of the values ​​of the feature maps and the parameters of the synthetic neural network can be joint. Advantageously, the coding method according to the invention makes it possible to efficiently compress the signal.

[0028] Advantageously, the decoding method is simple since it suffices to decode the feature maps of the current image sub-sequence, the reference feature maps and the synthesis neural network to reconstruct a decoded version of the current sub-sequence.

[0029] Advantageously, the current maps dynamically capture the local characteristics of the current subsequence, since they are coded (resp. decoded) as the sequence is processed. The reference maps, coded (resp. decoded) only once for the sequence, capture the global characteristics of the sequence. We can thus benefit from both types of characteristics. Furthermore, once the reference maps are transmitted and therefore available on the decoder side, the current maps allow the current image to be generated, and we can therefore design a transmission system that works image by image with low latency, each current image being decoded upon receipt of the associated current maps.

[0030] Such a synthetic neural network can advantageously be of very simple structure with few parameters.

[0031] Furthermore, decoding can be done by sequence of samples, in a progressive manner.

[0032] According to embodiments of the coding or decoding method:

[0033] - The coding method comprises a step of constructing and coding a group of reference characteristic maps representative of the sequence comprising said set of reference characteristic maps.

[0034] - The decoding method further comprises: - a step of decoding a group of reference characteristic maps representative of the sequence and - obtaining said set of reference characteristic maps from said group.

[0035] Advantageously, the group of reference maps is constructed and encoded by the encoder, and decoded by the decoder for the entire sequence. Then for each sub-sequence being processed, a part of the group of reference maps is considered to constitute the reference set used for the sub-sequence. It is thus possible to simplify the decoding by keeping a limited number of reference maps. According to one embodiment, the set is smaller than the group. According to one embodiment, the set is equal to the group.

[0036] - The method comprises a step of transforming said set of current and / or reference feature maps to obtain a set of feature maps transformed to the resolution of the input image, the method being characterized in that said feature vector is constructed from said transformed feature maps. Advantageously according to this mode, the feature maps are divided into two sets, one of which is reserved for the extraction of feature vectors and the other for coding. It is thus possible to dissociate the two processes whose purpose is different: the maps of the first group, to be coded (resp. decoded), must be compressed as efficiently as possible, while the maps of the second group, or transformed maps, must be able to facilitate the process of extracting and constructing the feature vector.

[0037] – According to a variant, at least one of the current and / or reference feature maps has a lower resolution than that of the sub-sequence to be coded (resp. decoded) and the transformation operation includes oversampling. Advantageously, according to this mode, the compression of the feature maps is more efficient since at least one of the current and / or reference feature maps, to be coded (resp. decoded), includes fewer values ​​than if it were at the resolution of the signal. For example, in the case of a digital image, one of the current and / or reference feature maps may be at 1 / 2 resolution, that is to say it includes half as many values ​​on the abscissa and ordinate as the input image has samples, i.e. in total 4 times fewer values ​​than a feature map at the resolution of the image.On the other hand, the transformed feature map, which corresponds to a transformation of this current and / or reference map, has the same resolution as the image. The transformation therefore includes in this case at least one oversampling operation to obtain the same number of values ​​in this transformed map as the input image (resp. to be decoded) has samples.

[0038] – At least one of said current and / or reference characteristic maps has a resolution identical to that of the sub-sequence to be coded (resp. decoded). Advantageously, according to this embodiment, at least one of the characteristic maps, at the same resolution as the input signal to be coded (resp. decoded), allows significant fidelity and respect for the details of the initial resolution of the image signal. The transformation according to one embodiment in this case preserves the number of values ​​of the transformed characteristic map; it can be reduced to identity (no processing is carried out on the values ​​of the current map) or include a filtering operation, quantization, a Fourier transformation, etc. During the coding step, quantization is essential for the proper functioning of the system if the characteristic maps include, for example, floating point or real values.It is necessary to quantize them before coding them and / or providing them as input to the synthesis and / or prediction neural network. On the other hand, during decoding, inverse quantization is not necessary, depending on the embodiments.

[0039] - The construction of the characteristic vector includes a sub-step of extracting a value from a current and / or reference characteristic map at a position identical to that of the current sample in the current image to be coded or decoded. Advantageously, it is possible to extract a value from a characteristic map at the same position as the sample in the image, to constitute an element of the characteristic vector. This method is simple to implement. For example, if we have NF characteristic maps as input (NF = NFM + NFR, where NFM represents the number of current characteristic maps, and NFR the number of reference characteristic maps), of the same resolution as the image, a simple extraction of the values ​​of the maps at the coordinates of the current sample (at the same abscissa and the same coordinate in the characteristic map) makes it possible to directly construct the characteristic vector of J elements.

[0040] - The construction of said characteristic vector comprises the following sub-steps: - extraction of a plurality of values ​​from said current and / or reference characteristic maps as a function of said position of said current sample; - processing of said extracted values ​​to obtain the characteristic vector.

[0041] Advantageously, according to this mode, the characteristic vector is extracted from the characteristic maps, which may be of any kind, and in particular at a resolution lower than that of the image to be coded (resp. decoded), before undergoing processing. Such processing may, for example, correspond to a quantification of the extracted data, or to scaling, or to filtering, etc. During coding, quantification is essential for the proper functioning of the system if the characteristic maps comprise, for example, floating point or real values. It is necessary to quantify them before coding them and / or providing them as input to the synthesis and / or prediction neural network. During decoding, however, inverse quantification is not necessary, depending on the embodiments.

[0042] - The method comprises a step of constructing a group of additional feature maps and the feature vector is further constructed from said feature maps. Advantageously, these additional maps, constructed identically at the coder and the decoder, are neither stored nor transmitted at the coder level, nor decoded at the decoder level. They thus make it possible to benefit from additional data to improve compression without degrading the bit rate. They may for example include coordinates, causal data available in other maps, data concerning other images already processed by the coder or decoder, etc.

[0043] - Said sets of reference and current feature maps contain the same number of feature maps; advantageously according to this mode, the sequence and the different sub-sequences of images of the sequence are associated with the same number of feature maps, which may moreover be of the same structure. For example, the sequence and the sub-sequences are associated with five feature maps having a hierarchy of decreasing resolutions. This facilitates the generation and extraction of the values ​​of the feature maps to produce the feature vector.

[0044] - Said at least one reference map is at the same resolution as said at least one current map. Advantageously, according to this mode, the construction of the characteristic vector is facilitated since the reference map is at the same resolution as the current map, it is very simple to carry out the extraction of a value at the same position or at a position close to that of the value of the current map being encoded.

[0045] - Said at least one reference card is indicated in the bitstream. Advantageously, according to this mode, a simple indicator coded in the stream allows the decoder to directly access the reference card(s) to be used for decoding the current image or subsequence, for example via an indicator of the number of the reference card in the group of reference cards.

[0046] - Said at least one reference map is predetermined. Advantageously, according to this mode, the feature map(s) to be used for decoding the current image or sub-sequence are predetermined. For example, the reference feature map is the one which has the same order number as the current map in the group of feature maps to which they respectively belong. According to another example, all the feature maps of the reference group serve as reference maps for the current map.

[0047] - Said at least one current map is decoded as a function of said at least one reference map. Advantageously, according to this mode, the (at least one) reference map is used not only for the construction of the characteristic vector, but also for the decoding (for example entropic) of a current map.

[0048] - Said at least one current map is decoded according to at least one current map. Advantageously, according to this mode, a current map already at least partially decoded can be used to more efficiently encode the current map being processed.

[0049] Correlatively, the invention also relates to a coding device and a decoding device.

[0050] The characteristics and advantages of the coding or decoding method apply in the same way to the coding or decoding device according to the invention and vice versa.

[0051] The invention also relates to a computer program on a recording medium, this program being capable of being implemented in a computer or an encoding or decoding device in accordance with the invention. This program comprises instructions adapted to the implementation of the corresponding method. This program can use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0052] The invention also relates to a computer-readable information carrier or recording medium comprising computer program instructions mentioned above. The information or recording media may be any entity or device capable of storing the programs. For example, the media may comprise a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a floppy disk or a hard disk, a DNA sequence, or a flash memory. Furthermore, the information or recording media may be transmissible media such as an electrical or optical signal, which may be conveyed via an electrical or optical cable, by radio link, by wireless optical link or by other means.

[0053] The program according to the invention can in particular be downloaded from an Internet-type network.

[0054] Alternatively, each information or recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of a method according to the invention.

[0055] Other characteristics and advantages of the present invention will emerge from the description given below, with reference to the appended drawings which illustrate exemplary embodiments thereof which are not limiting in nature.

[0056] schematically represents a coding device used in the context of the invention;

[0057] schematically represents a decoding device used in the context of the invention;

[0058] illustrates an example of a synthetic artificial neural network used in one embodiment of the invention;

[0059] illustrates an example of a synthetic artificial neural network used in another embodiment of the invention;

[0060] Illustrates an example selection of reference cards used in the context of the invention;

[0061] is a flowchart representing an example of a coding method that can be implemented by the coding device of the;

[0062] is a flowchart representing an example of a decoding method that can be implemented by the decoding device of the;

[0063] illustrates a decoding method used in one embodiment of the invention.

[0064] It schematically represents a device for coding digital image sequences ENC.

[0065] The ENC coding device includes an INITc initialization module, responsible in particular for constructing and coding the group of reference maps of the sequence.

[0066] An image can represent a two-dimensional image and the sequence a plurality of two-dimensional images (video sequence, stereoscopic, multiscopic components, series of medical images, etc.). The sequence is any succession of images, for example a GOP.

[0067] For a current sub-sequence to be coded v, comprising at least one current image, a coding module denoted ENCv comprises a GEN module for generating current feature maps, a REF module for obtaining reference feature maps, a SE transformation module, an XTR data extraction module, an MLP synthesis module corresponding to a synthetic artificial neural network, an NNC neural network coding module capable of coding the synthetic neural network, an FMC module for coding feature maps (which can also be used by the INITc module to code the reference maps), an EVAL module for evaluating coding performance, and an MAJ update module.

[0068] The ENC coding device may be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the above-mentioned modules may then be realized by the cooperation of the processor and computer program instructions stored in the above-mentioned memory and designed to perform the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0069] The ENC coding device receives as input a sub-sequence of at least one image to be coded, denoted I(P Vn ), comprising a plurality of samples P Vn. The subscript v indicates the image number(s) of the subsequence in the sequence of V images (e.g., v=3, or v={2,3,4}). The subscript n indicates the pixel number in an image of N pixels. In one embodiment, an image of the sequence is represented using at least one two-dimensional representation, such as a pixel matrix, each pixel having a vector of red (R), green (G), blue (B), or alternatively, a luminance component and at least one chrominance component. The location of each pixel is defined by its abscissa and ordinate (x n and there n) in the image. In one embodiment, the sequence is a sequence of grayscale images represented using a two-dimensional representation, such as a pixel matrix, each pixel having a grayscale component, or luminance. In this case the vector representing the pixel is reduced to a single component. In one embodiment, a subsequence is reduced to a single image. In this case the ENC encoder successively processes the current samples Pv n to encode the image. In one embodiment, a subsequence comprises at least two successive images. In this case the ENC coder simultaneously processes the current samples Pv n to encode successive images, considered as a subsequence of current samples. In one embodiment, a subsequence comprises a predefined number of successive images, for example 4.

[0070] As will be described in more detail later with reference to Figures 3 to 7:

[0071] The GEN feature map generation module is configured to generate a plurality of NFM common feature maps, denoted FMv i for the current subsequence v. These maps are called current feature maps. According to one embodiment, an optional module not shown may generate one or more additional maps, which will not be encoded or transmitted.

[0072] The REF module is configured to generate a plurality of NFR reference feature maps, denoted FRv j, for sequence v, from a group of reference feature maps, corresponding to the sequence, to the number of NFRMAX. We therefore have NFR <= NFRMAX. For example, the reference feature maps considered for the current subsequence are all those of the reference group, or are selected as a subset of the reference group. In the latter case, they can be predetermined or their reference can be encoded in the stream as an indicator.

[0073] The set of current NFM maps and reference NFR maps is used by the current coding module ENCv to code the subsequence Iv. The number of feature maps used is therefore NFM + NFR.

[0074] In one embodiment, the transformation module SE performs a transformation of the set of current and / or reference feature maps (FMv i , FRv j) to generate a set of transformed feature maps (FMSv i , FRSv j ) at the same resolution as the input signal.

[0075] The optional SE module can perform a quantization of the data extracted from this set of NFM+NFR feature maps, or of the feature vector formed from these data. Recall that the quantization of a value refers to the mapping of this value to a member of a discrete set of possible code symbols. For example, the set of possible code symbols can consist of integer values, and the quantization system performs a simple rounding of a real value to an integer value. According to another example, the quantization consists of a multiplication by a given value and then a rounding. Then the SE module can perform a transformation of the values ​​of at least one of the feature maps, for example an oversampling, an interpolation, a filtering, etc.At the end of the transformation, a transformed feature map has the same resolution as the images in the input subsequence. Advantageously, in this mode, the feature maps that are encoded can have a lower resolution than the images to be encoded, while the transformed maps, which are used to construct the feature vectors, have the same resolution as the image sequence, which facilitates the extraction of values.

[0076] In one embodiment, the SE module is absent, in this case the values ​​which will be used to construct the characteristic vector are extracted from the untransformed current and reference characteristic maps.

[0077] The XTR module performs value extraction in current and reference feature maps, optionally transformed, for a current sample or a subsequence of current samples Pv nto be coded, depending on its (their) coordinates in the input signal. For example, if we want to code the sample or subsequence of samples Pv n at the coordinates (x n , y n ) of an input image, the XTR module performs an extraction of values ​​in maps at positions imposed by the coordinates (x n , y n ). In one embodiment, the extracted values ​​constitute the characteristic vector Z n . Z n is a J-tuple, that is, it has J elements, or data z i . Z n has two sets of values: one (Zv n ) is extracted from the current feature maps of subsequence v, and the other (Zr n ) is extracted from the reference feature maps. The vector Z n of index n refers to the characteristic vector of the pixel or sequence of pixels P'v n . According to one embodiment, J = NFM + NFR.

[0078] In one embodiment, the XTR module performs processing of the extracted values, to generate the vector Z n . The XTR module can perform quantization of the data extracted from the set of feature maps. The processing may include other operations, such as filtering, scaling, etc. In particular, if the SE module is not used and if the feature maps are at lower resolutions than the images in the sequence, the XTR module can take into account the coordinates of the values ​​in the lower resolution maps.

[0079] Note that at least one of the SE or XTR modules must perform a quantization of the values ​​extracted from the current feature maps of the current subsequence.

[0080] The MLP module contains a synthetic neural network defined by K parameters W k , able to process the vector Z n, or J-Uplet, as input, to generate as output a vector representative of the sample or subsequence of samples Pv n to be coded from the current subsequence. The synthetic neural network contains, according to one embodiment, an MLP, or Multi Layer Perceptron, consisting of an input layer adapted to the input format (the J-tuple), optionally one or more hidden layer(s), and an output layer adapted to the output format of the output vector, generally a vector comprising NC (number of components) elements for each sample. According to one embodiment, NC is equal to 3 and the output vector is the triplet (R, G, B) of the pixel or pixels P'v n coded then decoded from the current subsequence. According to another embodiment, the MLP module contains a convolutional neural network.

[0081] The NNC module performs the coding of the neural network for synthesizing the current image or sub-sequence, in particular its Wv parameters. k. During the process of training, or building, the coding, that is to say as long as the stage of evaluating a performance is not satisfactory, the NNC module carries out a coding simulation, followed by a decoding, intended for the evaluation module. Subsequently, it carries out the actual coding of the parameters of the synthesis neural network Wv k . The coded parameters of the current image or subsequence are denoted Wvc k . As is known, the coding simulation can be identical to the actual coding, or an approximation of it.

[0082] The INITc module generates and encodes FR cards j , that is, values ​​of the sequence feature maps. The reference coded maps are denoted FRc j. For example, during encoder initialization, all (NFRMAX) maps of the sequence are constructed, then encoded and decoded so that they can be used as a reference for a subsequence to be encoded. According to another embodiment, the reference maps are generated dynamically as the subsequences are encoded, in the same way as the current maps. According to one embodiment, the encoding takes into account the neighborhood of a value to be encoded of the reference feature map, in the map itself and / or in another reference map.

[0083] When coding subsequences, the FMC module performs the coding of FMv maps i, i.e. the values ​​of the feature maps of the current subsequence (excluding the transformed maps, optionally resulting from oversampling or other transformation by the SE module). During the process of training, or construction, of the coding, i.e. as long as the performance evaluation step is not satisfactory, the FMC module performs a coding simulation, followed by decoding, for the evaluation module. Subsequently, it performs the actual coding of the values ​​of the FMv maps i and according to one embodiment, as explained above, reference maps. The coded maps of the current subsequence are noted FMcv i. As is known, the coding simulation can be identical to the actual coding, or an approximation of it. The coding module quantizes the latent representation of the map values ​​if necessary by using a quantizer to generate an ordered collection of quantized values. Then the coding module compresses the quantized data, for example by using an entropy coder.

[0084] According to one embodiment, it implements a coding taking into account the neighborhood of a value to be coded from the feature map. According to one embodiment, the neighboring position is selected in the map being coded (for example, the neighboring value at the top left of the one being coded). According to one embodiment, this neighborhood is selected in another current map, at a neighboring position. According to another embodiment, the neighborhood is selected in a reference map, at a neighboring position.According to yet another embodiment, the neighborhood is selected from a coded and decoded feature map corresponding to one or more subsequences v' of the sequence distinct from the current subsequence v, at a neighboring position, for example the previous subsequence in the sequence, or two past subsequences of the sequence, or a past and a future subsequence, to the extent that the decoding order of the sequence allows it (in known manner, the decoding order may be different from the chronological order of the sequence). All these embodiments may be combined, i.e. the neighborhood may consist of values ​​extracted from the current map, the current maps, the reference maps, and / or current maps of another subsequence.The FMC module may comprise for this purpose a feature map prediction neural network whose parameters may be encoded and transmitted to the decoder, which takes as input a neighborhood vector consisting of the neighborhood values ​​and produces as output a prediction of the current value of the feature map being encoded, which may be presented, in a known manner, in the form of a prediction value or probability data.

[0085] The EVAL module performs an evaluation and minimization of a coding performance. The evaluation function is for example of the rate-distortion type. The minimization can be carried out by gradient descent, or any other method within the reach of a person skilled in the art.

[0086] The MAJ module updates the values ​​of the FMv maps ito encode of the current subsequence, depending on the results of the performance function. According to one embodiment, it also performs an update of the values ​​of the reference maps to be encoded of the sequence, depending on the results of the performance function.

[0087] Once the current feature maps are stabilized, they can be encoded. In one embodiment, they are encoded using the reference maps.

[0088] It schematically represents a DEC decoding device.

[0089] The DEC coding device decodes the N Pdv sample sequences n of the subsequence v, proceeding by subsequence. At the end of the decoding, the V images of the sequence are decoded.

[0090] The DEC coding device includes an initialization module INITd, responsible in particular for decoding the group of reference cards of the SV sequence, noted FRdj . For example, during decoder initialization, all (NFRMAX) maps of the sequence are decoded so that they can be used as a reference for a subsequence to be decoded by the DECv module.

[0091] A current subsequence to be decoded v includes at least one current image to be decoded noted I(Pd Vn ), each image comprising a plurality of samples respectively. The decoder DECv dedicated to the subsequence receives as input the data necessary to decode the subsequence. As for the encoder, the index v indicates the image number(s) of the subsequence in the sequence of V images (for example, v=3, or v= {2,3,4}). The index n indicates the pixel number in an image of N pixels or a subsequence of images of N pixels. The decoding sub-module receives as input at least:- encoded data organized in NFM feature maps FMcv i(the encoded current feature maps);- the encoded Wvc parameters k of the synthesis neural network corresponding to the current subsequence, optionally the parameters of other neural networks such as an oversampling neural network, feature map prediction, etc.;

[0092] Furthermore, the DECv module obtains (e.g., reads from the decoder's memory) a set of reference characteristic maps, denoted FRdv j , from the FRd reference card group j.

[0093] The DEC decoding module vcomprises, for a current subsequence, a neural network(s) decoding module NND capable of decoding the MLP' synthesis neural network, a FMD module for decoding the current feature maps (which can also be used by the INITd module to decode the reference maps), a REF' module for obtaining reference feature maps, a XTR' data extraction module, an SE' inverse transformation module, an MLP' module corresponding to a synthesis neural network, optionally a module for generating additional feature maps. According to one embodiment, it produces as output a current decoded subsequence, comprising at least one image denoted I (Pdv n ), comprising a plurality of samples or sequences of samples decoded Pdv n.

[0094] The FMD decoding module decompresses the quantized data, for example using an entropy coder. According to one embodiment, it implements a decoding taking into account the neighborhood of a value to be decoded from the feature map. The embodiments proposed for the coder apply here by replacing the coding with a decoding of the values. In particular, the neighborhood can be selected from a current map (including the one being decoded), a reference map, or a current map of a sequence distinct from the current subsequence. The FMD module can comprise for this purpose a feature map prediction neural network, as described for the FMC module. The maps decoded by the FMD module, numbering NFM, are denoted FMdv i (feature maps of the current image).

[0095] The parameters of the synthetic neural network (MLP') decoded by the NND module are denoted Wdv k .

[0096] The REF' module constitutes a set of NFR characteristic maps, called reference characteristic maps, noted FRdv j , from a group of previously encoded and then decoded reference feature maps, corresponding to the sequence, to the number of NFRMAX. We therefore have NFR <= NFRMAX. The REF' module is identical to that of the encoder and the same examples apply. For example, the reference feature maps considered for the current subsequence are all those of the reference group, or are selected as a subset of the reference group. In the latter case, they can be predetermined or their reference can be decoded in the stream in the form of an indicator.

[0097] The set of current NFM maps and reference NFR maps is used by the current decoding module to decode the current subsequence v. The number of feature maps used is therefore NFM + NFR.

[0098] The decoder can also generate one or more additional maps, identical to the additional maps generated by the encoder for the current subsequence.

[0099] In one embodiment, the transformation module SE' performs a transformation of the set of current and / or reference feature maps (FMdv i , FRdv j ) to generate a set of transformed feature maps (FMS'v i , FRS'v j ) at the same resolution as the input signal.

[0100] The SE' module is identical to the SE module of the. The optional SE' module can perform an inverse quantization of the data extracted from this set of NFM+NFR feature maps. Then the SE' module can perform a transformation of the values ​​of at least one of the feature maps, for example an oversampling, an interpolation, a filtering, etc. At the end of the transformation, a transformed feature map has the same resolution as the images of the subsequence to be decoded. Advantageously, according to this mode, the feature maps that are decoded can have a lower resolution than that of the images to be decoded, while the transformed maps, which are used to construct the feature vectors, have the same resolution as the image sequence, which facilitates the extraction of the values.

[0101] In one embodiment, the SE' module is absent, in this case the values ​​which will be used to construct the characteristic vector are extracted from the untransformed current and reference characteristic maps.

[0102] The XTR' module is identical to the XTR module of the. It performs an extraction of values ​​in current and reference feature maps, for a current sample or a subsequence of current samples Pdv n to decode, depending on its (their) coordinates in the input signal. For example, if we want to decode the sample or subsequence of samples Pv n at the coordinates (x n , y n ) of an input image, the XTR' module performs an extraction of values ​​in maps at positions imposed by the coordinates (x n , y n ). In one embodiment, the extracted values ​​constitute the characteristic vector Zd n . Zd nhas two sets of values: one (Zdv n ) is extracted from the current feature maps of subsequence v, and the other (Zdr n ) is extracted from the reference feature maps. Zd n is a J-tuple, that is, it has J elements, or data zd i . The vector Zd n of index n refers to the characteristic vector of the pixel or subsequence of pixels Pdv n . according to one embodiment, J = NFM + NFR.

[0103] In one embodiment, the XTR' module performs processing of the extracted values, to generate the vector Zd n. The XTR' module can perform an inverse quantization of the data extracted from the set of feature maps. The processing may include other operations, such as filtering, scaling, etc. In particular, if the SE' module is not used and if the feature maps are at lower resolutions than the images in the sequence, the XTR' module can take into account the coordinates of the values ​​in the lower resolution maps.

[0104] Note that at least one of the SE' or XTR' modules must perform an inverse quantization of the values ​​extracted from the current feature maps of the current subsequence, if a quantization has been performed at the encoder.

[0105] The MLP' module contains a so-called synthesis neural network, defined by K parameters Wd k , capable of processing the vector Zd n, or J-Uplet, as input, to generate as output a vector representative of the sample or subsequence of samples Pdv n to be decoded. The synthetic neural network contains, according to one embodiment, an MLP, or Multi Layer Perceptron, consisting of an input layer adapted to the input format (the J-tuple), optionally one or more hidden layer(s), and an output layer adapted to the output format of the output vector, generally a vector comprising NC (number of components) elements for each sample. It can also comprise a convolutional neural network. According to one embodiment, K=3 and the output vector is the triplet (R, G, B) or (Y, U, V) of each pixel Pd vn decoded. The MLP' module is identical in structure to the MLP module, and its parameters are either identical if the coding of its parameters W k is lossless, or different if the coding is done with losses.

[0106] When all Pdv samples n of the subsequence v have been decoded, we have a reconstructed signal I(Pdv n ), according to one embodiment an image Iv comprising N samples decoded in the form of N vectors Pdv n , according to another mode a sub-sequence of images Iv each comprising N samples decoded in the form of N vectors Pdv n .

[0107] The decoding device DEC can be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the modules mentioned above can then be realized by the cooperation of the processor and computer program instructions stored in the aforementioned memory and designed to carry out the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor

[0108] Illustrates an example of a synthetic artificial neural network used for encoding and decoding according to one embodiment of the invention.

[0109] The synthetic artificial neural network used for coding, MLP, and the synthetic artificial neural network used for decoding, MLP', are defined by an identical structure, comprising for example a plurality of layers of artificial neurons, and by a set of weights and activation functions associated respectively with the artificial neurons of the network concerned.

[0110] A vector representation of a current sample (a vector Z n or Zd n from FMv feature maps i / FMSv i and FRv j / FRSv j to the encoder or FMdv i / FMS'v i and FRdv j / FRS'v jto the decoder) is applied as input (i.e. on an input layer) of the MLP or MLP' artificial synthesis neural network. The input vector includes values ​​from the current maps (Zv n ,Zdv n ) and values ​​from the reference maps (Zr n ,Zdr n ).

[0111] Generally speaking, the MLP / MLP' artificial neural network produces as output a plurality of P' vectors vn or Pd vn representative of the reconstructed (on coding) or decoded (on decoding) samples, according to one embodiment the color components (R, G, B or Y, U, V) constituting the color pixels of the sub-sequence of images.

[0112] On the, the sub-sequence of images corresponds to an image, each sub-sequence of samples therefore comprises a single sample and the corresponding vector is noted P'1 n (to the encoder) or Pd 1n(to the decoder). According to the example shown in, 4 values ​​are extracted from each input set. The synthetic artificial neural network produces as output the constituent components of a pixel of the image.

[0113] The concatenation of all these reconstructed pixels in the image constitutes the decoded, or reconstructed, image.

[0114] At the encoder, the MLP synthesis artificial neural network is trained on the image, so as to minimize the differences between the input representation of the current subsequence I (Pv n ) and its representation I (P'v n) at the output, while also minimizing the amount of data to be encoded. The EVAL module performs a performance measurement in this sense. Once the encoder training is complete, the network parameters are encoded, either lossless, in which case the MLP' neural network is identical to MLP, or lossy, in which case the MLP' network may be slightly different from MLP.

[0115] Illustrates an example of a synthetic artificial neural network used for encoding and decoding according to another embodiment of the invention.

[0116] The is identical to the as far as the input to the neural network is concerned.

[0117] Generally speaking, the MLP / MLP' artificial neural network produces as output a plurality of P' vectors vn or Pd vnrepresentative of the reconstructed (on coding) or decoded (on decoding) samples, according to one embodiment the color components (R, G, B or Y, U, V) constituting the color pixels of the sub-sequence of images.

[0118] On the, the sub-sequence of images corresponds to two images, a sub-sequence of samples therefore comprises two samples and the two corresponding vectors (R, G, B) or (Y, U, V) are noted P'1 n / Pd1 n .and P'2 n / Pd2 n. The concatenation of all these reconstructed pixels in the two images constitutes the decoded, or reconstructed, subsequence.

[0119] Illustrates an example of selecting reference cards in a video sequence.

[0120] The sequence represented on thecomprises 4 images, numbered from 1 to 4. For example, it is an Intra period of a standard video encoder, or a GOP, etc. It is made up of 3 sub-sequences v=1, v={2,3} and v=4. The FR reference maps j (resp. FRdj) of the sequence are 3 in number (NFR = 3) and of different resolutions.

[0121] The subsequence v={2,3} is encoded (resp. decoded) using 3 common maps FM{2,3} i=1,2,3 (resp. FMd{2,3} i=1,2,3 ) and a reference map FR{2,3} j=1 (resp. FRd{2,3} j=1 ) of the sequence. In total, 4 cards of caractéristiques sont présentées (après avoir été éventuellement transformées par le module SE / SE', puis leurs vecteurs caractéristiques extraits par le module XTR / XTR') à l'entrée du réseau de synthèse MLP (resp. MLP') qui synthétise les échantillons des images codées (resp. décodées) 2 et 3.

[0122] The subsequence v={4} is encoded (resp. decoded) using 3 common FM4 cards i=1,2,3 (resp. FMd4 i=1,2,3 ) and three FR4 reference cards j=1,2,3 (resp. FRd4 j=1,2,3 ) of the sequence. In total, 6 cards of caractéristiques sont présentées (après avoir été éventuellement transformées par le module SE / SE', puis leurs vecteurs caractéristiques extraits par le module XTR / XTR') à l'entrée du réseau de synthèse MLP (resp. MLP') qui synthétise les échantillons de l'image codée (resp. décodée) numéro 4.

[0123] The numerical examples chosen are indicative.

[0124] This is a flowchart representing an example of a coding process that can be implemented by the coding device of the.

[0125] According to one embodiment, the current subsequence is a sequence of two-dimensional images, each sample subsequence to be coded is therefore a set of pixels Pv n of coordinates (x n , y n ) in the respective images I(Pv n ) of the sequence to be coded. According to one embodiment, the current subsequence comprises a single image.

[0126] Encoding takes place in two main phases:

[0127] In a first phase, called the construction phase, a first learning is carried out at the initialization step E0 to construct, code and decode a group of reference images FRc jof the SV sequence. According to one embodiment, SV comprises a series of images, for example temporal images of a video sequence. According to one embodiment, the images are multi-view or 3D components of a multi-view or 3D image or sequence of images. According to one embodiment, the images correspond to an Intra period or a GOP of a conventional video encoder.

[0128] These maps can be inserted into the B1 stream after coding. According to one embodiment, these maps are generated before the processing of the subsequences. According to another embodiment, they are generated during the processing of the subsequences, by optimization, in the same way as the current feature maps.

[0129] For an input subsequence v comprising at least one image I(Pv n ), a learning process is carried out in order to determine the values ​​of the FMv cards i and W parameters kto optimize a global cost function. Learning is for example performed by gradient descent, followed by updating the parameters of the MLP neural network and the values ​​of the current and / or reference feature maps. As is known in the state of the art, the cost function can be of the rate-distortion, or rate, or distortion, or perceptual type. To measure the rate R, it is necessary to simulate the coding of the FMv maps i , optionally FRv cards j , then measure the associated coding rate (the size of the stream B1). According to one embodiment, the coding of the parameters W is not simulated k because their influence is less important than that of the feature maps. According to one embodiment, the coding of the parameters W is also simulated kand the associated flow rate (the size of the flow B2) is measured. To measure the distortion D, it is necessary to simulate the coding and then the decoding of at least part of the image sequence, to obtain at least one sequence of pixels P'v n resulting from a simulation of coding then decoding of the samples of index n, then measuring the difference between this sub-sequence of images I(Pv n ) as input and a corresponding part of the sequence I(P'v n ) encoded then decoded.

[0130] Then during a second phase, called the coding phase, the FMv cards i and the W parameters k are encoded to produce the FMcv encoded values i and WCV k before transmission or storage. They constitute the compressed representation of the input sequence I(Pv n ).

[0131] We will now describe the steps of a method according to one embodiment of the invention.

[0132] During a step E20, an input sequence I(Pv n ) to be coded, comprising at least one image comprising a plurality of N samples Pv n , is provided as input to the process.

[0133] In a step E21, NFM current and optionally reference feature maps are initialized. Subsequently, the parameters W k of the MLP neural network and the values ​​of the maps must be optimized during the construction phase.

[0134] According to one embodiment, the feature maps are initialized by predefined constant values. According to another embodiment, the feature maps are initialized by a set of random real values. The feature maps are subsequently updated, or refined, during a step E22, by the update module MAJ of the encoder during its learning.

[0135] In a step E23, NFR reference maps are obtained by the REF module, from the group of feature maps associated with the sequence. The REF module constitutes a set of NFR reference feature maps, denoted FRv j, issues d'un groupe de cartes de caractéristiques de référence FRj,précédemment décodées, au nombre de NFRMAX (NFR <= NFRMAX). Par exemple, les cartes de caractéristiques de référence considérées pour la sous-séquence en cours de codage sont toutes celles du groupe de référence (FRvj= FRj) ou sont sélectionnées comme un sous-ensemble du groupe de référence. Dans ce dernier cas, elles peuvent être prédéterminées. Par exemple, on peut utiliser les cartes de caractéristiques de référence de même taille que les cartes de caractéristiques courantes. Leur référence peut alternativement être codée dans le flux sous forme d'un indicateur. Par exemple, la carte de référence utilisée pour le codage / décodage des images v={2,3} de laest l'image numéro 1 du groupe de cartes de référence. Cette indication peut être codée dans le flux.

[0136] The set of current NFM maps and reference NFR maps constitutes a set of feature maps for the MLP synthesis network.

[0137] According to one embodiment, the current and / or reference feature maps are of the same resolution as the images of the input sequence I(Pv n ) and therefore each have the same number of values ​​N as there are samples Pv n to be coded in each image. According to one embodiment, the current and / or reference feature maps are of resolution less than or equal to that of the images of the input sequence I(Pv n) and therefore comprise, for at least one of them, a number N' of values ​​to be coded less than N. According to a variant, the first current map (resp. reference map) is at the resolution of the images and each following map is at a resolution half that of the previous one. According to one embodiment, several maps are of the same resolution, lower than that of the images of the input sequence I(Pv n ).

[0138] During a step E24, according to one embodiment, the NFM cards of the set of current cards FMv i and / or the NFR reference maps FRv j are transformed by the SE module to generate FMSv transformed maps i / FRSv jto the resolution of the input images. In this mode, the feature vectors are preferably extracted from the transformed maps, and not directly from the current maps to be encoded and / or the reference maps. The transformed maps are neither stored nor transmitted, they are only used for the construction of the feature vectors.

[0139] According to one embodiment, one or more additional maps, forming an additional group of additional feature maps, are generated, and complete the set of feature maps constituted. They are used for the construction of the feature vector but are neither stored nor transmitted.

[0140] According to one embodiment, NFM + NFR FMSv maps i / FRSv j are generated. According to one embodiment, each FMv card i is transformed into an FMSv map i . According to one embodiment, each FRv card jreference is transformed into an FRSv map j.

[0141] According to one embodiment, at least one FMv card i / FRv j is of lower resolution than the images in the sequence to be encoded and the transformation operation involves oversampling so that the transformed map has the same number of samples as the images in the sequence. Oversampling involves adding values ​​to the FMSv maps i and / or FRSv j to achieve the resolution of the images in the input sequence. It can be simple (by nearest neighbor replication) or include interpolation (linear, polynomial, filtering, etc.). The transformation can include an upsampling neural network.

[0142] During a step E25, values ​​are extracted by the XTR module: - current FMv characteristic maps i (or possibly FMSv itransformed, and optionally additional) . - FRv reference feature maps j . or possibly FRSv j transformed.

[0143] This extraction is carried out according to the coordinates (x n, yn) de la sous-séquence d’échantillons Pvncourants de la sous-séquence courante. Elle peut aussi être réalisée en fonction de la résolution de la carte considérée. Les séquences d’échantillons à coder sont par exemple traitées par ordre séquentiel, de n=1 à n=N.

[0144] According to one embodiment, the vector Z n characteristic results directly from this extraction.

[0145] According to one embodiment, the vector Z n feature is constructed by processing the extracted values. The processing may include quantification of the extracted values ​​or of the Z vector n constituted, if necessary. The processing may include other operations, such as filtering, scaling, applying any function, preferably monotonic, etc. In one embodiment, Z nis a J-tuple constructed from values ​​taken from the current and reference (and optionally additional) feature maps by applying processing to one or more values ​​in the maps, for example, filtering values ​​close to the target value in a map. For example, in a map that is at the same resolution as the input signal, one can extract the values ​​located at the coordinates (x n , y n ), (x n -1, y n ), (x n , y n -1) and (x n -1, y n -1) and apply processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (z i ) of element i of vector Z n relative to this map. As another example, in a map that is at half the resolution of the input signal, we can consider the values ​​located at the coordinates (x n / 2, y n / 2), (x n / 2-1, yn / 2), (x n / 2, y n / 2-1) and (x n / 2-1, y n / 2-1) and apply processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (z i ) of element i of vector Zv n or Zr n relating to this current, reference (or additional) feature map.

[0146] In one embodiment, Z n has as many values ​​as there are input cards.

[0147] In one embodiment, Z n is a J-tuple with values ​​located at the coordinates (x n , y n ) of a current pixel Pv n ,as will be illustrated in support of the. The vector Z nd’indice n réfère au vecteur caractéristique du pixel ou de la séquence de pixels Pvn. Selon un mode de réalisation, J= NFM+NFR. Le vecteur Zn(zv1… zvNFM, zr1… zrNFR), comporte deux ensembles de valeurs : l'une (Zvn) est extraite des cartes de caractéristiques courantes de la sous-séquence v, et l'autre (Zrn) est extraite des cartes de caractéristiques de référence. Optionnellement il peut comporter des valeurs issues des cartes supplémentaires.

[0148] In one embodiment, Z nis a J-tuple constructed from values ​​taken from the current and reference (and optionally additional) feature maps at coordinates that may be different between maps. For example, if the maps are at different resolutions, the coordinates are adapted (by scaling) to match the resolution of each map.

[0149] At step E26, the vector Z n is processed by the MLP neural network to generate the output subsequence of samples Pv n to code, according to one embodiment, the triplets (R, G, B) of the samples P'v n (Pv samples n coded then decoded) of the images of the sub-sequence.

[0150] During a step E27, the FMv cards iThe current values ​​are encoded by the encoder's FMC module. During the construction phase, this operation is an encoding simulation. During the encoding phase, this operation is an actual encoding and the encoded values ​​constitute the B1 stream. The simulation can be identical to the actual encoding but it can also be different (for example, simplified). For this encoding, any known technique for compressing the values ​​of the maps can be used. In one embodiment, the reference maps are also encoded at this stage.

[0151] In one embodiment, the cards are encoded in order (FMv1, FMv2,…, FMv4), and the variables of each card in a predefined order, for example lexicographic. In one embodiment, the reference cards are also encoded in order. Each card undergoes entropy coding. The entropy coding produces a compressed stream B1 whose bit rate is subsequently measured during a step E29.

[0152] During a step E28, the parameters W k of the MLP neural network are quantized and encoded. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is an actual coding and the encoded values ​​constitute the B2 stream. The simulation can be identical to the actual coding but it can also be different (for example, simplified). Any known technique can be used for this purpose, for example the neural network coding standard proposed by the MPEG-7 part 17 standard, also called NNR (Neural Network Representation). Note that in this case, it is necessary to choose the amount of degradation that the coding brings to the parameters W k . The structure and parameters W kof the neural network are initialized for example during the first iteration of this step. These parameters are subsequently updated, or refined, during the construction phase, during subsequent iterations of the process.

[0153] According to one embodiment, the parameters of the neural network are initialized by predefined values ​​known to give a satisfactory result (for example, following training on a corpus of image sequences). According to another embodiment, the parameters W k of the neural network are initialized by a set of random values.

[0154] In step E29, a performance measure is evaluated.

[0155] For this purpose, the coding simulation rates associated with the feature maps (simulation of the B1 flow by coding the FMv maps i / FRv j) and optionally to the parameters of the neural network (simulation of the B2 flow by coding the W parameters k ) are measured.

[0156] According to one embodiment, the cost function is of the rate-distortion type, denoted (D+L*R), where D for example the quadratic error measured between the input images and the decoded images (or the error measured on a sample subset of the images). According to another example, D is calculated from a perceptual function such as the SSIM (for Structural SIMilarity), or MSSSIM (for Multi-scale Structural SIMilarity). According to one embodiment, R is the simulated rate of the stream B1; according to another embodiment, R is the total rate used to code this image, that is to say the sum of the simulated rates of B1 and B2. L is a parameter which regulates the rate-distortion trade-off. Other cost functions are possible.

[0157] As long as the cost function has not reached its minimum, the performance measurement is not satisfactory, and the method is repeated from step E22. Alternatively, the method can be interrupted after a predefined time or a predefined number of iterations, so as to control its complexity or duration. This minimization can be carried out by a mechanism known as gradient descent with parameter updates during step E22 for the feature map values ​​and E27 for the network parameters.

[0158] In an EF step, if the cost function has reached its minimum, training stops. If an encoded version corresponding to the last simulation of the neural network parameters and feature maps is available, streams B1 and B2 can be made up of them. According to another embodiment, the encoding effectively the updated parameters of the neural network and the values ​​of the feature maps (FMv i ) is performed at this step to produce the encoded parameters Wc k and FMc i which constitute flows B1 and B2. According to one embodiment, the FRv cards jsont codées à la fin de l'apprentissage, lorsque toute la séquence a été traitée.

[0159] The B1 and B2 streams may be concatenated to produce a final stream. In one embodiment, the B2 stream of encoded neural network parameters is stored or transmitted before the B1 stream, so that it can be decoded before the B1 stream.

[0160] This is a flowchart representing an example of a decoding method that can be implemented by the decoding device of the.

[0161] The decoding described concerns a subsequence of images comprising at least one image of the sequence to be decoded. It is assumed that the group of reference maps of the sequence FRc j , has been decoded by the initialization module. According to one embodiment, a technique for predicting a feature map value by its neighborhood is used, as described in support of the. In one embodiment, the reference maps are decoded in order, and the values ​​of each map in a predefined order, for example lexicographic.

[0162] In a step E30, the streams B1 and B2 are extracted from the encoded stream. They contain respectively, for a current subsequence of the sequence, the coded representations of the current maps FMcv i , and Wcv parameters kof the neural network.

[0163] During a step E31, the NFM FMdv cards i of the current subsequence are generated by decoding the FMcv values i . For this decoding, for example, entropy decoding is used. According to one embodiment, a technique for predicting a feature map value by its neighborhood is used, as described in support of the. In one embodiment, the FMdv maps i are decoded in order (FMdv1, FMdv2,… FMdv4), and the values ​​of each card in a predefined order, for example lexicographic.

[0164] In a step E32, NFR reference maps are obtained by the REF' module, from the group of feature maps associated with the sequence. The REF' module constitutes a set of NFR reference feature maps, denoted FRdv j, issues du groupe de cartes de caractéristiques de référence FRdj,précédemment décodées, au nombre de NFRMAX (NFR <= NFRMAX). Par exemple, les cartes de caractéristiques de référence considérées pour la sous-séquence en cours sont toutes celles du groupe de référence (FRvj= FRj) ou sont sélectionnées comme un sous-ensemble du groupe de référence. Dans ce dernier cas, elles peuvent être prédéterminées. Par exemple, on peut utiliser les cartes de caractéristiques de référence de même taille que les cartes de caractéristiques courantes. Leur référence peut alternativement être codée dans le flux sous forme d'un indicateur. Par exemple, la carte de référence utilisée pour le décodage des images v={2,3} de laest l'image numéro 1 du groupe de cartes de référence. Cette indication peut être lue et décodée dans le flux. Au contraire, toutes les cartes de référence sont utilisées pour le décodage de l'image v= 4 de la. Il n'est donc pas nécessaire de lire cette indication dans le flux.

[0165] The set of current NFM maps and reference NFR maps constitutes a set of feature maps for the synthesis network.

[0166] According to embodiments as described for the encoder:

[0167] - The current and / or reference feature maps are of the same resolution as sequence I (Pdv n ) to be reconstituted, that is to say that they contain N values.

[0168] - The current and / or reference feature maps are at a resolution lower than or equal to that of subsequence I (Pdv n ) to be reconstructed. According to a variant, the first current map (resp. reference map) is at the resolution of the images and each following map is at half the resolution of the previous one.

[0169] - Several common and / or reference feature maps are of the same resolution, lower than the resolution of the subsequence.

[0170] During a step E33, according to one embodiment, the NFM cards of the set of current cards FMdv i and / or the NFR FRdv reference maps j are transformed by the SE' module to generate FMS'v transformed maps i / FRS'v j to the resolution of the images to be decoded. This step is similar to step E24 which was described for the encoder in support of the, and the embodiments apply. In particular:

[0171] According to one embodiment, NFM + NFR FMS'v cards i / FRS'v j are generated.

[0172] According to one embodiment, each FMdv card i is transformed into an FMS'v card i . According to one embodiment, each FRdv card j is transformed into a FRS'v card j .

[0173] According to one embodiment, at least one FMdv card i / FRdv jis of lower resolution than the images to be decoded and the transformation operation involves oversampling so that the transformed map has the same number of samples as the input image. Oversampling involves adding values ​​to the FMS'v maps i and / or FRS'v j to achieve the resolution of the image to be decoded. It can be simple (by nearest neighbor replication) or include interpolation (linear, polynomial, by filtering, etc.). The transformation can optionally include inverse quantization of the extracted values, if necessary. However, inverse quantization is not mandatory. The transformation can include an upsampling neural network.

[0174] During a step E34, values ​​are extracted by the XTR' module: - current characteristic maps FMdv i (or possibly FMS'v i transformed, and optionally additional). - FRdv reference feature maps j or possibly FRS'v j transformed.

[0175] This extraction is carried out according to the coordinates (x n, yn) de la sous-séquence d’échantillons Pvncourants de la sous-séquence courante. Elle peut aussi être réalisée en fonction de la résolution de la carte considérée. Cette étape est similaire à l’étapeE25qui a été décrite pour le codeur à l’appui de la, et les modes de réalisation s’appliquent. Notamment :

[0176] In one embodiment, Zd n has as many values ​​as there are input cards.

[0177] According to one embodiment, the vector Zd n characteristic is constructed by processing the extracted values. The processing may include quantification of the extracted values ​​or of the vector Zd n constituted, if necessary. The processing may include other operations, such as filtering, scaling, applying any function, preferably monotonic, etc. The examples cited for the encoder apply.

[0178] In one embodiment, Zd n is a J-tuple with values ​​located at the coordinates (x n , y n) of a current pixel Pdv n ,as will be illustrated in support of the. The vector Zd nd’indice n réfère au vecteur caractéristique du pixel ou de la séquence de pixels Pdvn. Selon un mode de réalisation, J= NFM+NFR. Le vecteur Zdn(zdv1...zdvNFM, zdr1...zdrNFR), comporte deux ensembles de valeurs : l'une (Zdvn) est extraite des cartes de caractéristiques courantes de la sous-séquence v, et l'autre (Zdrn) est extraite des cartes de caractéristiques de référence. Optionnellement il peut comporter des valeurs issues des cartes supplémentaires.

[0179] In one embodiment, Zd n is a J-tuple constructed from values ​​taken from the current and reference (and optionally additional) feature maps at coordinates that may be different between maps. For example, if the maps are at different resolutions, the coordinates are adapted (by scaling) to match the resolution of each map.

[0180] The samples to be decoded are for example processed in sequential order, from n=1 to n=N.

[0181] During a step E35, the Wdv parameters k of the MLP synthesis neural network' are generated by decoding the Wcv values kof the B2 stream. Any known decoding technique corresponding to the coding technique used by the encoder can be used for this purpose. The MLP synthesis neural network is similar to the MLP synthesis network, i.e. it has the same structure and the same parameters, except for the coding, which can be done with or without losses.

[0182] During a step E36, the vector Zd n feature is processed by the MLP' synthesis neural network to generate as output a vector representative of the sample or subsequence of samples Pdv n to decode, according to one embodiment, one or more triplets which are injected into the decoded image I (Pdv n ) at positions (x n , y n ) of the color components (Rd, Gd, Bd). This step is similar to step E26 which was described for the encoder in support of the.

[0183] When all the signal samples have been processed, the current decoded subsequence corresponding for example to image I (Pdv n ) is available.

[0184] According to one embodiment, the B2 stream is decoded before the B1 stream (except for the reference maps), in order to have the MLP' synthesis neural network available before starting to decode the samples.

[0185] Illustratively represents a decoding method used in one embodiment of the invention.

[0186] In this embodiment, the current FMdv decoded maps i are 3 in number. In a preferred mode, there are 7.

[0187] In this embodiment, the first FMdv1 map has the same resolution as the image I(Pdvn) to be decoded, and therefore comprises WxH variables, where W represents the width of the image in pixels, and H its height. The second FMdv2 map has half the resolution (in each dimension) of the FMdv1 map. Each additional map has half the resolution of the previous map. This structure makes it possible to reduce the number of variables in the feature maps, which facilitates decoding while minimizing the coding cost.

[0188] The FMdv2 map is oversampled by a factor of 2 by the SE' module in each dimension, according to any oversampling method within the reach of those skilled in the art. The FMdv3 map is oversampled by a factor of 4 in each dimension.

[0189] In this embodiment, the decoded reference maps FRd jare 3 in number. In a preferred embodiment, there are 7. In this embodiment, the first map FRd1 has the same resolution as the image, and therefore has WxH variables, where W represents the width of the image in pixels, and H its height. The second map FMd2 has half the resolution (in each dimension) of map FMd1. Each additional map has half the resolution of the previous map. This structure makes it possible to reduce the number of variables in the feature maps, which facilitates decoding while minimizing the coding cost.

[0190] The three cards in the reference group are selected by the decoder to form the reference set. (FRdv1=FRd 1 ;FRdv2 =FRd2 ;FRdv3=FRd3).

[0191] The FRd2 map is oversampled by a factor of 2 in each dimension by the SE' module, according to any oversampling method within the reach of those skilled in the art. The FRd3 map is oversampled by a factor of 4 in each dimension.

[0192] FMS'v Transformed Maps i and FRS'v j are of the same resolution as the image to be decoded, and therefore have WxH values, where W represents the width of the image in pixels, and H its height.

[0193] In this embodiment, the vector Zd n is a 6-tuple (zv 3, zv 2, zv 1, zr3, zr2, zr1) made up of the 3 values ​​of the FMS'v cards isituées aux coordonnées (xn, yn) du pixel courant Pdvn.et des 3 valeurs des cartes FRS’vjsituées aux coordonnées (xn, yn) du pixel courant Pdvn. Le vecteur Zdnest optionnellement déquantifié puis traité par le réseau de neurones de synthèse MLP’ pour générer en sortie les triplets (R, G, B) représentatifs des échantillons Pdvnà décoder. Le triplet (R, G, B) est inséré dans chaque image décodée I (Pdvn) de la sous-séquence v aux coordonnées (xn, yn) dans les composantes couleur (Rd, Gd, Bd).

[0194] According to a variant presented in dotted lines, there are 7 cards: an additional card noted FME'0 has been introduced. In this embodiment, the vector Zd n is a 7-tuple.

[0195] This additional map typically contains data that can assist the MLP network in the task of image reconstruction. Thus, the added maps can be one or more of the following, non-limiting, list:

[0196] – A map showing the abscissa of that point at each point.

[0197] – A map showing the ordinate of that point at each point.

[0198] – A map with positional coding at each point.

[0199] – A map representing an image distinct from the images being processed, capable of providing information on the images to be coded, for example a previously processed image or sequence of images.

[0200] – A map containing data representing the time difference between frames in the video being encoded. For example, if the first and last frames of the video are 8 frames apart, all samples in the map contain the value 8.

[0201] – A map representing a feature map of an image distinct from the images being processed, likely to provide information on the images to be coded, for example a previously processed map.

[0202] – A card containing the value of an already decoded sample from the same card, for example the previous sample in the decoding order.

Claims

Method of decoding a sequence of images (S v ) comprising the following steps for at least one current image to be coded:- a construction step, comprising the following sub-steps:- construction (E21, E22) of a set of current feature maps (FMv i ) representative of said at least one current image (v, I(Pv n )), comprising at least one current feature map;- construction (E23) of a set of feature maps (FRv j ) of reference, comprising at least one coded then decoded reference characteristic map;- for at least one sample, called current sample (Pv n ), of said at least one current image, associated with a position (x n , y n ) in said at least one current image: - construction (E25) of a characteristic vector (Z n ) from said at least one reference characteristic map (Zr n) and said at least one current feature map (Zv n ), depending on said position (x n , y n ) of said at least one current sample (Pv n ) ;- processing (E26) of said characteristic vector (Z n ) by an artificial neural network (MLP), called a synthetic neural network, defined by a set of parameters (Wv k ), to provide a vector (P'v n ) representative of a decoded value of said at least one current sample;- updating (E22, E29) of at least one value of said at least one current characteristic map and / or of at least one parameter of said network, as a function of a coding performance measurement;- a coding step (E24) of said set of current characteristic maps (FMv i ) ;- a coding step (E28) of said set of parameters (Wv k ) of said synthetic neural network. Method for coding a sequence of images according to claim 1, characterized in that it further comprises:- a construction step (FR j ) and coding (FRc j ) of a group of reference feature maps representative of the sequence comprising said set of reference feature maps (FRv j ). Method for decoding a sequence of images (Sv) of a binary stream, comprising the following steps for at least one current image (v): - decoding (E35) of a set of parameters (Wdv k ) representative of a synthetic neural network (MLP'); - decoding (E31) of a set of current feature maps (FMdv i ) comprising at least one current feature map representative of said at least one current image;- obtaining (E32) a set of feature maps (FRdv j) of reference, comprising at least one decoded reference characteristic map;- for at least one sample, called current sample (Pdv n ), of said at least one current image (v) to be decoded, associated with a position (x n , y n ) in the image: - construction (E34) of a characteristic vector (Zd n ) from said at least one reference characteristic map (Zdr n ) and said at least one current feature map (Zdv n ), depending on said position (x n , y n ) of said at least one current sample, and:- processing (E37) of said vector (Zd n , ZRdn) par ledit réseau de neurones de synthèse (MLP’) pour fournir un vecteur représentatif d’une valeur décodée dudit au moins un échantillon courant (Pdvn). Decoding method according to claim 3, characterized in that it further comprises:- a step of decoding (E31) a group of characteristic maps (FRd j) of reference representative of the sequence and- obtaining (E32) said set of reference characteristic maps (FRdv j ) from said group (FRd j ). Coding or decoding method according to one of the preceding claims, characterized in that the set of reference maps and the set of current maps (FRdv j , FMdv i ) contain the same number (NFM, NFR) of feature maps. Coding or decoding method according to one of the preceding claims, characterized in that said at least one reference map is at the same resolution as said at least one current map. Encoding or decoding method according to one of the preceding claims, characterized in that said at least one reference map is indicated in the binary stream. Coding or decoding method according to one of the preceding claims, characterized in that said at least one reference map is predetermined. Coding or decoding method according to one of the preceding claims, characterized in that it further comprises a step of transforming said set of current and / or reference characteristic maps to obtain a set of transformed characteristic maps at the resolution of the current image, the method being characterized in that said characteristic vector is constructed from said transformed characteristic maps. Decoding method according to one of claims 3 to 9, characterized in that said at least one current map is decoded as a function of said at least one reference map and / or a current characteristic map. Decoding method according to claim 10 characterized in that the decoding is carried out by a prediction neural network. Device for coding a sequence of images (S v ), said device being configured to implement:- a construction step, comprising the following sub-steps:- construction (GEN, MAJ) of a set of current characteristic maps, (FMv i ) representative of said at least one current image (v, I(Pv n )), including at least one current characteristic map;- construction (INITc, REF) of a set of characteristic maps (FRv j ) of reference, comprising at least one coded then decoded reference characteristic map;- for at least one sample, called current sample (Pv n ), of said at least one current image, associated with a position (x n , y n) in said at least one current image: - construction (XTR) of a characteristic vector (Z n ) from said characteristic map at least one reference characteristic map (Zr n ) and said at least one current feature map (Zv n ), depending on said position (x n , y n ) of said at least one current sample (Pv n ) ;- processing (MLP) of said characteristic vector (Z n ) by an artificial neural network, called a synthetic neural network, defined by a set of parameters (Wv k ), to provide a vector (P'v n) representative of a decoded value of said at least one current sample;- updating (E22, E29) of at least one value of said at least one current characteristic map and / or of at least one parameter of said network, as a function of a coding performance measurement,- a coding step (E24) of said set of current characteristic maps (FMv i ) ;- a coding step (E28) of said set of parameters (Wv k ) of said synthetic neural network. Device for decoding from a binary stream of a sequence of images (S V ), said device being configured to implement, for at least one current image (v, I(Pdv n )) of the sequence: - decoding (NND) of a set of parameters (Wdv k ) representative of a synthetic neural network (MLP'); - decoding (FMD) of a set of current feature maps including at least one feature map (FMdv i) current representative of said at least one current image;- obtaining (INITd, REF') a set of characteristic maps (FRdv j ) of reference, comprising at least one decoded reference characteristic map;- for at least one sample, called current sample (Pdv n ), of said at least one current image (v) to be decoded, associated with a position (x n , y n ) in the image: - construction (E34) of a characteristic vector (Zd n ) from said at least one reference characteristic map (Zdr n ) and said at least one current feature map (Zdv n ), depending on said position (x n , y n ) of said current sample, and:- processing (E37) of said vector (Zd n ) by said synthesis neural network (MLP') to provide a vector representative of a decoded value of said at least one current sample (Pdv n ). Computer program comprising instructions for carrying out the steps of an encoding or decoding method according to claim 1 or 3 when said program is executed by a computer.

Images