Method and device for encoding and decoding image sequences.

Abstract

Method and device for encoding and decoding images. The invention relates to a method for encoding and decoding a sequence of images. The decoding method comprises, for at least one current image of the sequence: - decoding a set of so-called correction images, including at least one correction image representing correction data for said at least one current image; - decoding a set of so-called prediction feature maps, including at least one feature map representing a movement of said at least one current image; - decoding a set of parameters representing a prediction neural network; - obtaining a set of reference images, including at least one decoded reference image;- processing of said prediction feature maps and said reference images by a synthesis module comprising said at least one prediction neural network to the input of which said prediction feature maps are applied, to produce at least one prediction image; - reconstruction of the current image from said at least one prediction image and said at least one correction image. Figure for the abstract: Fig. 1.;

Description

Title of the invention: Method and device for coding and decoding image sequences. Prior art

[0001] The invention relates to the general field of coding digital image sequences. It relates more particularly to the compression of digital videos.

[0002] Digital videos are generally subject to source coding aimed at compressing them in order to limit the resources required for their transmission and / or storage. There are many coding standards, such as the ITU / MPEG standards (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) as well as their extensions (MVC, SVC, 3D-HEVC, etc.)

[0003] The encoding of an image is generally carried out by dividing the image into several rectangular blocks, and by encoding these blocks of pixels according to a given processing sequence. In existing video compression techniques, the processing of a block typically comprises a prediction of the pixels of the block carried out using previously coded then decoded pixels present in the image being encoded, in which case we speak of “Intra prediction”, or of previously coded images, in which case we speak of “Inter prediction”. This exploitation of spatial and / or temporal redundancies makes it possible to avoid transmitting or storing the value of the pixels of each block of pixels, by representing at least some of the blocks by a residual representing a difference between the prediction values ​​of the pixels of the block and the actual values ​​of the pixels of the predicted block.

[0004] As video formats continue to evolve to compress more and more and adapt to the variety of expected formats and communication networks, the possibilities for prediction are becoming ever greater and the classic coding and decoding algorithms very complex.

[0005] In addition to these classic approaches proposed by compression standards (MPEG, ITU), approaches based on artificial intelligence, and in particular neural ones, tend to develop.

[0006] Some of these neural approaches can be seen as a simple extension of the notion of competition of the aforementioned compression techniques, such as the competition of prediction and transformation mode in video coding.

[0007] Other approaches use the concept of "auto-encoder". Auto-encoders are learning algorithms based on artificial neural networks, which allow a new representation of a data set to be constructed. The architecture of an auto-encoder consists of two parts: the encoder and the decoder. The encoder consists of a set of neural layers, which process the data in order to construct new representations called "encoded", also called "latent representations". In turn, the decoder's neural layers receive these representations and filter them in order to try to reconstruct the initial data. The differences between the reconstructed data and the initial data make it possible to measure the error made by the autoencoder. Training consists of modifying the autoencoder's parameters in order to reduce the reconstruction error measured on the different samples of the data set. The performance of such autoencoder-based systems comes at the cost of a considerable increase in memory footprint and complexity compared to conventional approaches such as those proposed by compression standards.They can have millions of parameters and may require up to a million MACs (multiply-accumulate) to decode a single pixel. This makes such decoders far more complex than conventional decoders, which could hamper the adoption of learning-based compression.

[0008] More recently, a simple image encoding technique based on a neural network has been described in the article "Compression with Implicit Neural representations" by Emilien Dupont et al. (arXiv:2103.03123). The proposed encoding technique consists of fitting a neural network to an image, quantizing the network weights and transmitting them. At decoding time, the neural network is evaluated at each pixel position to reconstruct the image. However, such a technique remains inefficient in terms of compression and requires coding the video frames independently.

[0009] In the video domain, an equivalent technique was presented in the article “Scalable Neural Video Representations with Learnable Positional Features” (NeurIPS 2022) by Subin Kim et al. The temporal dimension is taken into account. The coding algorithm generates a set of three two-dimensional latent keyframes, one on each spatio-temporal axis, to learn the common frames of the video. It also generates a latent 3D representation grid to capture the local details of the video. This representation is used to modulate a neural network trained to process the three keyframes. However, such an approach lacks flexibility in the choice of representation of the keyframes (the number of which is fixed at 3) and the 3D grid (fixed for the entire sequence). Moreover, it requires the processing of a transmitted 3D grid, which is both heavy in terms of the amount of transmitted data and the memory required for its processing..

[0010] Patent document published under number US 2023 / 0145525 A1 describes a method for decoding images comprising obtaining a current optical flow and correction data generated on the basis of a predicted image and an image current. The optical flow is encoded and decoded by a neural network. Feature data representative of the correction data are obtained via a neural network-based image encoder; and the current reconstructed image is obtained by combining the correction data and the predicted image. However, such a method requires storing neural networks that are very large.

[0011] There is therefore a need for a solution allowing a sequence of images to be coded / compressed in a simple and efficient manner. Statement of the invention

[0012] The invention relates to a method for decoding a sequence of images from a binary stream, comprising the following steps for at least one current image of the sequence to be decoded: - decoding a set of so-called correction images comprising at least one correction image representative of correction data of said at least one current image; - decoding a set of so-called prediction feature maps comprising at least one feature map representative of a movement of said at least one current image; - decoding of a set of parameters representative of a prediction neural network; - obtaining a set of reference images, comprising at least one decoded reference image; - processing said prediction feature maps and said reference images by a synthesis module comprising said at least one prediction neural network to the input of which said prediction feature maps are applied, to produce at least one prediction image; - reconstruction of the current image from said at least one prediction image and said at least one correction image.

[0013] The invention also relates to a method for coding a sequence of images comprising the following steps for at least one current image of the sequence: - initialization of a set of so-called correction images; - initialization of a set of parameters representative of a prediction neural network; - initialization of a set of so-called prediction feature maps; - obtaining a set of reference images, comprising at least one coded then decoded reference image; - processing of said prediction feature maps and said reference images by a synthesis module comprising said at least one neural network prediction to the input of which said prediction feature maps are applied, to produce at least one prediction image; - reconstruction of the current image from said at least one prediction image and said at least one correction image; - updating at least one value of said at least one feature map and / or at least one parameter of said network, based on a coding performance measurement. - a coding step in the binary stream comprising: - coding of said at least one correction image; - coding of the set of representative parameters of the prediction neural network; - coding of the set of prediction feature maps.

[0014] For the purposes of the invention, encoding, or "coding", means the operation which consists of representing a set of samples, or pixels, in a compact form carried for example by a digital binary train. "Decoding" means the operation which consists of processing a digital binary train to restore decoded samples or pixels.

[0015] By "sequence of images" is meant a plurality of ordered two-dimensional images Iv, for example temporally in the case of a video. According to one example, the sequence corresponds to a scene. The images may comprise several components, for example to represent the colors, in RGB or composite YUV form for example. According to one example, the sequence corresponds to a set of predefined images, for example a fixed number, or, in the sense of the MPEG standards, a GOP (Group Of Pictures) comprising the images located between two images of the Intra-image type, also called "intra period". According to another example, the images may be views of the same scene represented in multi-views. According to another example, the images may be a plurality of temporal and multi-view images (immersive video si ve).

[0016] By “at least one current image” is meant a sub-sequence of at least one image of the sequence, currently being encoded or decoded, comprising a plurality of N samples.

[0017] By "sample" is meant a value taken from an image of the sequence. Sampling a signal produces a series of discrete values ​​called samples. In the case of an image signal, the sample is called a pixel which can be a color pixel traditionally represented by a triplet of values, for example (R,G,B) or (Y,U,V), each value belonging to a map of components which may or may not be the same size. The position of the sample is identified by its abscissa (x) and ordinate (y) coordinates in the image, or in one of the components.

[0018] By "correction image" is meant a concrete or abstract representation of an image, comprising a plurality of variable scalar data, which are also called correction values, for example real numbers, or integers, signed or unsigned. The set of correction data of an image can correspond to a single correction image, or several correction images, the dimension of which can be arbitrary. A correction image or map corrects a prediction image or map to produce an approximation of the current image to be coded or decoded.

[0019] By "neural network" is meant a neural network such as a convolutional neural network, a multilayer perceptron, an LS TM (for "Long Short Terni Memory" in English), etc. The neural network is defined for example by a plurality of layers of artificial neurons and by a set of preferably non-linear activation, weighting and addition functions (for example, a layer can calculate y = f(Ax+b), where y and b are vectors of dimension N, x a vector of dimension M, A is a matrix of dimension MxN, and f is the activation function). The input of the neural network consists of data extracted from the feature maps, for example in the form of vectors.

[0020] By "parameter of the neural network" is meant one of the values ​​which characterizes the neural network, for example a weight associated with one of the neurons (filter coefficient, weighting, bias, value affecting the operation of the non-linearity, etc.)

[0021] By "feature map" is meant all or part of an abstract representation of an image in a latent domain, comprising a plurality of variable scalar data, which are also called characteristic values, for example real numbers, or integers, signed or unsigned. The abstract representation of an image can therefore comprise several feature maps. The abstract representation of an image can also correspond to a single feature map comprising a set of values ​​associated with a position.

[0022] A "latent domain" is a representation composed of a set of feature maps in 2D form, the size of which is proportional to the input image, or of 1D vectors or scalar values.

[0023] By "processing by a synthesis module" is meant the synthesis of one or more motion-compensated images. Such a synthesis module comprises at least one prediction neural network, to the input of which are applied at least the prediction characteristic maps, in the form of input vectors called characteristic vectors extracted from the characteristic maps. Such a module may comprise other modules, such as for example a motion compensation module. According to embodiments, reference maps or images are also applied to the input of the prediction neural network. The application of a function expressed by the neural network to an input characteristic vector produces an output vector representing a prediction of the samples to be coded (resp. decoded). This output vector corresponds to motion information relating to the current image or sub-sequence of the sequence, or to a sample of the motion-compensated image.

[0024] By "reference image" is meant a decoded image resulting from a prior decoding of an image in the sequence. For example, it may be a previous image in the sequence, or a key image (Intra image, first image in the sequence, etc.) or a following image in the sequence if the decoding order allows it. It may be represented in the image domain (in the form of a reference image) or in the latent domain (in the form of latent value feature maps).

[0025] By "reconstruction of the current image" is meant a combination of the obtained prediction image(s) and the obtained correction image(s). This combination may include, for example, an addition or subtraction of the two types of images. It may also include more complex processing, for example a combination by neural network.

[0026] By "performance measurement" is meant a measurement between at least one value of a sample to be coded and a decoded value of said sample. The measurement can evaluate for example a distortion, or a perceptual error. It can be carried out on a sample or a plurality of samples (for example, current samples of the current image, or of a current sub-sequence, etc.) The measurement can also include a measurement of the flow rate, in particular associated with the coding of the neural network and / or the coding of the characteristic maps. The measurement can be a joint measurement between the flow rate and the distortion through their weighting. As is well known in the state of the art, a minimization of the value of this measurement is generally carried out.

[0027] Generally speaking, it is considered that the steps of a coding or decoding method should not be interpreted as being linked to a notion of temporal succession. In other words, the steps may be carried out in a different order from that indicated in the independent coding or decoding claim, or even in parallel.

[0028] When the training is finished, that is to say that the performance measurement obtained is satisfactory, or that a predetermined training time has been reached, the effective coding of the parameters of the neural network and of the values ​​of the characteristic maps associated with the current image or sub-sequence can be carried out and stored or transmitted to the decoder. Advantageously, at the level of the encoder, the training process makes it possible to refine the parameters of the prediction neural network and / or the values ​​of the prediction characteristic maps. as well as the data of the correction images until an adequate representation is obtained in terms of performance, for example a desired balance between the generated flow rate and the distortion undergone by the image or sub-sequence of images being coded. Advantageously, the coding method according to the invention makes it possible to efficiently compress the signal.

[0029] Advantageously, the decoding method is simple since it is sufficient to decode the prediction characteristic maps relating to the current image or sub-sequence and the parameters of the neural network to reconstruct a decoded version of the prediction image. Such a neural network can advantageously be of very simple structure with few parameters.

[0030] Advantageously, the feature maps dynamically capture the local features of the current image or sub-sequence, since they are coded (resp. decoded) as the sequence is processed. It is therefore possible to design a transmission system that works image by image with low latency, each current image or sub-sequence being decoded upon receipt of the current maps and the associated neural network.

[0031] According to one embodiment of the invention, such a coding or decoding method is further characterized in that the processing step comprises the following sub-steps: - applying said prediction feature maps to the input of said at least one prediction neural network to produce movement information; - compensating said at least one reference image using said motion information to produce said at least one prediction image.

[0032] By "motion information" is meant a set of data of any nature to describe the movement of a sample or part of an image, for example a field of motion vectors per sample, called a dense motion field, or optical flow, or a block motion field, or a set of equations representing an affine movement of one or more samples of the image, or even an angle and an amplitude of a motion vector per sample or per block, etc.

[0033] By "compensating a reference image" is meant the application of motion information to the selected reference image(s) or map(s), to generate a motion-compensated image, called a prediction image.

[0034] Advantageously, according to this mode, the generation of the prediction image takes place according to two traditional steps well known to those skilled in the art, namely the generation or decoding of the movement, synthesized here by a prediction neural network from the prediction maps, followed by compensation of reference images by this movement. The compensation operation, well known, consists of apply the resulting movement to a pixel or set of pixels in the reference image.

[0035] According to a variant of this embodiment of the invention, the step of processing said prediction characteristic maps by said at least one prediction neural network comprises, for at least one sample, called current sample, of said at least one current image, associated with a position in the image: - construction of a characteristic vector from said at least one prediction characteristic map, as a function of said position of said at least one current sample, and: - applying said characteristic vector to the input of said prediction neural network to provide a vector representative of the prediction of said at least one current sample.

[0036] By "feature vector constructed from the feature maps as a function of a position" is meant a vector consisting of one or more elements, or data, preferably discrete, the data being constructed from the feature maps at a position determined by that of the samples being processed in the images. This feature vector is the one that is applied, or presented, to the input of the prediction neural network. Such a vector can be constituted for example from a plurality of values ​​taken from each of the feature maps at the same abscissa and ordinate coordinates as those of the sample that one seeks to predict or reconstruct. Such a vector can also be constituted from a plurality of values ​​close to those of the sample, taken from each of the feature maps.In the case where we seek to reconstruct a movement field via the prediction neural network, we obtain, for example, at the output of the network a movement vector for this sample.

[0037] The coding method according to this mode constructs the prediction and / or correction parameters, from a sequence of input images, by training a prediction and / or correction neural network on characteristic vectors associated with a position of a sample to be coded in the current image or sub-sequence. These characteristic vectors are constructed from prediction and / or correction characteristic maps. The characteristic maps may be at the resolution of the input images, or at a lower resolution. A coding or decoding method according to this mode uses the parameters of the neural network and the characteristic maps transmitted to it to generate characteristic vectors similar (apart from coding errors) to those of the encoder, present them as input to the appropriate neural network, and obtain motion information as output.

[0038] According to another embodiment of the invention, such a coding or decoding method is characterized in that the processing step comprises a sub-step applying said prediction feature maps and said reference images to the input of said at least one prediction neural network to produce at least one prediction image.

[0039] Advantageously according to this mode, the generation of the prediction image combines the generation of the motion field and the compensation. Thus, a pooling of operations is ensured, ensuring reduced complexity. Preferably in this case, the reference images are in the latent domain, and represented in the form of feature maps.

[0040] According to a variant of this embodiment of the invention, the step of applying said prediction characteristic maps and said at least one reference image to the input of said at least one prediction network comprises, for at least one sample, called current sample, of said at least one current image, associated with a position in the image: - construction of a characteristic vector from said at least one prediction characteristic map and said at least one reference image, as a function of said position of said at least one current sample, and: - applying said vector to the input of said prediction neural network to provide a vector representative of the prediction of said at least one current sample.

[0041] In the case where one seeks to reconstruct the prediction image via the prediction neural network, such an output vector can consist of a vector (eg (Y,U,V)) representative of the prediction of the current pixel.

[0042] According to one embodiment of the invention, the step of decoding said set of correction images further comprises the following sub-steps: - decoding of a set of parameters representative of a correction neural network; - decoding of a set of so-called correction feature maps; - applying said decoded correction feature maps to the input of said at least one correction neural network to produce at least one prediction image.

[0043] Advantageously, according to this embodiment, a neural network is used to encode and decode the correction images. The neural network trained on the encoder is transmitted to the decoder, which can, from the correction feature maps and the correction network alone, reconstruct a correction image. The correction neural network can be trained jointly with the prediction neural network to provide a better compromise between the throughput and the distortion of the decoded images.

[0044] According to a variant of this embodiment of the invention, the step of processing said correction characteristic maps by said at least one network of correction neurons comprises, for at least one sample, called current sample, of said at least one current image, associated with a position in the image: - construction of a characteristic vector from said at least one correction characteristic map, as a function of said position of said at least one current sample, and: - applying said characteristic vector to the input of said correction neural network to provide a vector representative of the correction of said at least one current sample.

[0045] In the case where one seeks to reconstruct the correction image via the correction neural network, such an output vector can consist of a vector (eg (Y,U,V)) representative of the correction of the current pixel.

[0046] According to another embodiment of the invention, the correction neural network and / or the prediction neural network comprise an MLP and / or a convolutional network.

[0047] MLP means a Multi-Layer Perceptron. Convolutional network means a network that includes at least one convolution layer. Advantageously, this type of network is very simple in nature and its parameters can be easily learned and transmitted from the encoder to the decoder.

[0048] According to embodiments of the coding or decoding method:

[0049] - the reconstruction step includes an addition or subtraction applied to the correction and prediction images; advantageously in this mode, the correction data are simply added to the prediction data, for example for each pixel of the image, to reconstruct the current image.

[0050] - the reconstruction step uses a convolutional neural network applied to the correction and prediction images; advantageously in this mode, the correction data can be combined more finely with the prediction data to reconstruct the current image

[0051] - the motion information is a dense field of bidi motion vectors monthly;

[0052] - the compensation step comprises a displacement of at least one sample of the reference image.

[0053] - The method comprises a step of transforming said set of cards of ca prediction and / or correction characteristics to obtain a set of transformed characteristic maps at the resolution of the current image, the method being characterized in that said characteristic vector is constructed from said transformed characteristic maps. Advantageously according to this mode, the characteristic maps are divided into two sets, one of which is reserved for the extraction of the characteristic vectors and the other for coding / decoding. It is thus possible to dissociate the two processes with different purposes: the maps of the group to be coded (resp. decoded) must be compressed as efficiently as possible, while the transformed maps must be able to facilitate the process of extraction and construction of the characteristic vector.

[0054] - According to a variant, at least one of the prediction characteristic maps and / or correction is of lower resolution than that of the sub-sequence to be coded (resp. decoded) and the transformation operation involves oversampling. Advantageously, according to this mode, the compression of the feature maps is more efficient since at least one of the feature maps to be coded (resp. decoded) has fewer values ​​than if it were at the resolution of the signal. For example, one of the feature maps may be at 1 / 2 resolution, that is to say it has half as many values ​​on the abscissa and ordinate as the input image has samples, i.e. in total 4 times fewer values ​​than a feature map at the resolution of the image. On the other hand, the transformed feature map, which corresponds to a transformation of this map, has the same resolution as the image.The transformation therefore includes in this case at least one oversampling operation to obtain the same number of values ​​in this transformed map as the current image has samples.

[0055] - At least one of said prediction and / or correction characteristic maps is of identical resolution to that of the image to be coded (resp. decoded). Advantageously, according to this mode, at least one of the characteristic maps, at the same resolution as the input signal to be coded (resp. decoded) allows significant fidelity and respect for the details of the initial resolution of the image signal. The transformation according to one embodiment in this case retains the number of values ​​of the transformed characteristic map; it can be reduced to identity (no processing is carried out on the values ​​of the current map) or include a filtering operation, quantization, a Fourier transformation, etc.

[0056] - The construction of the characteristic vector includes an extraction sub-step of a value from a prediction and / or correction feature map at a position identical to that of the current sample in the current image to be coded or decoded. Advantageously, it is possible to extract a value from a feature map at the same position as the sample in the image, to constitute an element of the feature vector. This method is simple to implement. A simple extraction of the values ​​from the maps at the coordinates of the current sample (at the same abscissa and the same coordinate in the feature map) makes it possible to directly construct the feature vector.

[0057] - The construction of said characteristic vector comprises the following sub-steps: - extracting a plurality of values ​​from said prediction feature maps and / or correction according to said position of said current sample; - processing of said extracted values ​​to obtain the characteristic vector.

[0058] Advantageously according to this mode, the characteristic vector is extracted from the characteristic maps, which can be any, and in particular at a resolution lower than that of the image to be coded (resp. decoded), before undergoing processing. Such processing can for example correspond to a quantification of the extracted data, or to a scaling, or to a filtering, etc. During coding, a quantification is essential for the proper functioning of the system if the characteristic maps comprise for example floating, or real values. It is necessary to quantify them before coding them and / or providing them as input to the prediction and / or correction neural network. During decoding, however, inverse quantification is not necessary, according to the embodiments.

[0059] - The method comprises a step of constructing a group of character maps additional characteristics and the motion and / or correction characteristic vector is further constructed from said characteristic maps. Advantageously, these additional maps, constructed identically to the encoder and the decoder, are neither stored nor transmitted at the encoder level, nor decoded at the decoder level. They thus make it possible to benefit from additional data to improve compression without degrading the flow rate. They may, for example, include coordinates, causal data available in other maps, data concerning other images already processed by the encoder or decoder, etc.

[0060] - Said at least one reference image is indicated in the bitstream. Avanta Conveniently, according to this mode, a simple indicator coded in the stream allows the decoder to directly access the reference image(s) to be used for decoding the current image or subsequence, for example via an indicator indicating the absolute or relative position of the number of the reference image(s) in the sequence.

[0061] - Said at least one reference image is predetermined. Advantage Conveniently, according to this mode, the reference image(s) to be used for decoding the current image or subsequence are predetermined. For example, it is the previous one, or the first one in the sequence, etc.

[0062] Correlatively, the invention also relates to a coding device and a decoding device.

[0063] The characteristics and advantages of the coding or decoding method apply in the same way to the coding or decoding device according to the invention and vice versa.

[0064] The invention also relates to a computer program on a recording medium, this program being capable of being implemented in a computer or a coding or decoding device in accordance with the invention. This program includes instructions suitable for implementing the corresponding method. This program may use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0065] The invention also relates to an information medium or a recording medium readable by a computer, and comprising computer program instructions mentioned above. The information or recording media can be any entity or device capable of storing the programs. For example, the media can comprise a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a floppy disk or a hard disk, a DNA sequence, or a flash memory. Furthermore, the information or recording media can be transmissible media such as an electrical or optical signal, which can be conveyed via an electrical or optical cable, by radio link, by wireless optical link or by other means.

[0066] The program according to the invention can in particular be downloaded from an Internet-type network.

[0067] Alternatively, each information or recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of a method according to the invention. Brief description of the drawings

[0068] Other characteristics and advantages of the present invention will emerge from the description given below, with reference to the appended drawings which illustrate exemplary embodiments thereof which are not in any limiting nature.

[0069] [Fig-1] [Fig.l] schematically represents a decoding device used in the scope of the invention;

[0070] [Fig.2] [Fig.2] schematically represents a coding device used in the context of the invention;

[0071] [Fig.3] [Fig.3] illustrates a synthesis module used in two embodiments of the invention;

[0072] [Fig.4] [Fig.4] is a flowchart representing an example of a decoding method which can be implemented by the decoding device of [Fig.l];

[0073] [Fig.5] [Fig.5] is a flowchart representing an example of a coding method which can be implemented by the coding device of [Fig.2];

[0074] [Fig.6] [Fig.6] illustrates a correction image decoding method used in one embodiment of the invention.

[0075] [Fig.7] [Fig.7] illustrates a field decoding method of movement used in one embodiment of the invention. Description of the embodiments

[0076] [Fig.l] schematically represents a DEC decoding device.

[0077] The decoding device DECv of [Fig.l] decodes an image or sub-sequence of index v of the sequence SV, proceeding image by image. At the end of the decoding, the V images of the sequence are decoded.

[0078] A current subsequence to be decoded V comprises at least one current image to be decoded denoted Iv, each image comprising respectively a plurality of N samples. Subsequently, it is considered, without loss of generality, that the current subsequence comprises a single current image. The decoder DECv dedicated to the image receives as input the data necessary to decode it:

[0079] - FPc encoded data organized into NFP feature maps of prediction;

[0080] - WPc encoded parameters of a prediction neural network;

[0081] - one or more IC correction images; according to one embodiment, the images of correction are decoded by entropy decoding their values; according to one embodiment, these correction images can be decoded using a correction neural network WCc applied to NFC correction feature maps FCc which are presented (applied) to its input. In this case, the associated encoded data includes the parameters of the correction neural network and the encoded values ​​of the correction feature maps.

[0082] - optionally, the coded parameters of other neural networks, in particular:

[0083] - predictive decoding of prediction and / or correction feature maps - oversampling and / or processing of prediction and / or correction feature maps;

[0084] - mixing of compensated images and correction images;

[0085] - post-processing of reconstructed data after mixing;

[0086] - etc.

[0087] The DECV decoding module comprises, for a current image, an NND module for decoding neural network(s), an MSP synthesis module comprising at least one prediction neural network, optionally an MLPC correction neural network, an FPD module for decoding prediction characteristic maps FPc, an FCD module for decoding correction images IC, optionally responsible for decoding correction characteristic maps FCc, an XTRM module, optionally XTRC, for processing and extracting data from the feature maps, a REF module for obtaining images or reference maps, a MIX module for combining the ICO compensated images and the IC correction data, optionally a module for generating additional feature maps, a post-processing module, not shown. According to one embodiment, it produces as output a current decoded image, noted Idv, comprising a plurality of N decoded samples. According to another embodiment, not shown, it produces as output a sub-sequence of decoded images comprising at least the current image.

[0088] The FPD decoding module decompresses the compressed data, for example using an entropy decoder, from one or more prediction feature maps. According to one embodiment, it implements a decoding taking into account the neighborhood of a value to be decoded from the feature map. The neighborhood can be selected from a current motion map (including the one being decoded), or a map of an image distinct from the current image. The FPD module can comprise for this purpose a feature map prediction neural network. The prediction maps decoded by the FPD module, numbering NFP, are denoted FP and are called motion maps of the current image, or current motion maps.

[0089] The FCD decoding module decompresses the IC correction images. According to one embodiment, it is a conventional decoder, for example of the JPEG, or MPEG, etc. type, responsible for decoding a current IC correction image. According to another embodiment, it decodes quantized data, for example using an entropy decoder, of one or more correction characteristic maps. According to one embodiment, it implements a decoding taking into account the neighborhood of a value to be decoded from the characteristic map. The neighborhood can be selected from a current map (including the one being decoded), or a map of an image distinct from the current image. The FCD module can comprise for this purpose a characteristic map prediction neural network. The maps decoded by the FCD module, to the number of NFCs, are denoted FC and are called correction maps of the current image, or current correction maps.

[0090] The NND module decodes neural networks. Any known neural network decoding technique can be used for this purpose, for example entropy decoding, or the neural network coding standard proposed by the MPEG-7 part 17 standard, also called NNR. The parameters of the prediction neural network (MLPP) of the synthesis module decoded by the NND module are denoted WP. The parameters of the correction neural network (MLPC) decoded by the NND module, if they exist, are denoted WC. Their coded values ​​are respectively denoted WPc and WCc.

[0091] The REF module constitutes a set of NFR reference images, denoted FREF, corresponding to one or more previously coded and then decoded images of the sequence. They can be predetermined or their reference can be decoded in the stream in the form of an indicator. In one embodiment, a single reference image is selected from the set. In one embodiment, several reference feature maps, in the latent domain, represent the reference image.

[0092] The MSP module contains at least one so-called MLPP prediction neural network, defined by the K WPc parameters decoded by the NND module. The module takes as input the reference image(s), the decoded prediction characteristic maps, and the decoded parameters of the prediction neural network. It generates as output at least one motion-compensated image(s) denoted ICO. Two embodiments will be presented in support of [Fig.3].

[0093] The XTRC module, if present, performs processing and extraction of values ​​in feature maps, in order to present them to the neural network. It will be presented in support of [Fig.7].

[0094] The MIX module uses as input one or more predicted images ICO and the decoded correction information IC and performs a combination of the two pieces of information to obtain the reconstructed image. The combination can be, in its simplest form, an addition of the values ​​of a prediction image and a correction image. According to another embodiment, the MIX module uses a neural network.

[0095] The decoding device DEC can be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the modules mentioned above can then be realized by the cooperation of the processor and computer program instructions stored in the aforementioned memory and designed to carry out the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor

[0096] [Fig.2] schematically represents a device for coding digital image sequences ENC.

[0097] The coding device ENCv of [Fig.2] codes a current image or sub-sequence v of the sequence SV, proceeding by sub-sequences of images. A sub-sequence can comprise a single image.

[0098] Subsequently, it is considered, without loss of generality, that the subsequence comprises a single image Iv comprising respectively a plurality of N samples. The ENCv coder produces coded parameters as output. These parameters include:

[0099] - encoded prediction features organized into NFP feature maps- FP characteristics;

[0100] - the encoded parameters WPc of the prediction neural network;

[0101] - the coded correction image data IC; according to one embodiment, the correction images are encoded by entropy encoding their values; according to one embodiment, these correction images can be encoded using a correction neural network applied to correction feature maps. In this case, the associated encoded data includes the parameters of the WCc neural network and the encoded values ​​of the FCc correction feature maps.

[0102] - optionally, the coded parameters of other neural networks, such as mentioned for the decoder.

[0103] The ENCV coding module comprises, for a current image, an NNC module for coding neural network parameters, an FPC module for coding prediction characteristic maps FPc and optionally an FCC module for coding images or correction characteristic maps FCc, an RD-OPT evaluation module, an INIT / MAJ initialization and update module.

[0104] Furthermore, in a conventional manner, the coding module comprises a decoder similar to the DECv decoder which was previously described, in order to reconstruct the current image l'v coded then decoded.

[0105] The FPC coding module compresses the quantized data, for example using an entropy coder, of one or more prediction feature maps. During the process of training, or building, the coding, i.e. as long as the performance evaluation step is not satisfactory, the FPC module performs a coding simulation, followed by decoding, for the RD-OPT evaluation module. Subsequently, it performs the actual coding of the values ​​of the FPc maps. In a known manner, the coding simulation can be identical to the actual coding, or produce an approximation thereof. The coding module quantizes, if necessary, the latent representation of the values ​​of the maps by using a quantizer to generate an ordered collection of quantized values. Then the coding module compresses the quantized data, for example using an entropy coder.As described in the decoder, according to one embodiment, it implements a coding taking into account the neighborhood of a value to be decoded from the feature map. The FPC module may comprise for this purpose a feature map prediction neural network, identical to that of the decoder. The motion maps coded by the FPC module, numbering NFP, are noted FPc (prediction maps of the current image).

[0106] The FCC coding module compresses the quantized data, for example using an entropy coder, of one or more images or correction maps. During the process of training, or building, the coding, that is to say as long as the step of evaluating a performance is not satisfactory, the FCC module performs a coding simulation, followed by a decoding, intended for the RD-OPT evaluation module. Subsequently, it performs the effective coding of the values ​​of the images or FCc maps. In a known manner, the coding simulation can be identical to the effective coding, or produce an approximation thereof. The coding module quantizes if necessary the latent representation of the values ​​of the maps by using a quantizer to generate an ordered collection of quantized values. Then the coding module compresses the quantized data, for example by using an entropy coder. As described to the decoder, according to one embodiment, it implements a coding taking into account the neighborhood of a value to be decoded from the feature map.The FCC module may include for this purpose a feature map prediction neural network, identical to that of the decoder. The correction maps coded by the FCC module, if they exist, to the number of NFC, are noted FCc (correction maps of the current image).

[0107] The INIT / MAJ module is responsible for initializing the values ​​of the characteristic maps and the parameters of the neural network(s), noted respectively FPo, WPo, FCo, WCo.

[0108] The RD-OPT module performs an evaluation and minimization of a coding performance. The evaluation function is for example of the rate-distortion type. The minimization can be carried out by gradient descent, or any other optimization method within the reach of those skilled in the art.

[0109] The INIT / MAJ module updates the values ​​to be encoded in the current image (FPo, WPo, FCo, WCo), based on the results of the performance function. Once the current feature maps and the neural network(s) are stabilized, they can be encoded.

[0110] The ENC coding device can be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the above-mentioned modules can then be realized by the cooperation of the processor and computer program instructions stored in the above-mentioned memory and designed to carry out the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0111] [Fig.3] illustrates an MSP synthesis module used in two embodiments of the invention (a) and (b). The module takes as input the reference image(s), the decoded prediction feature maps, and the decoded parameters of the prediction neural network. It generates as output at least one motion-compensated image(s) denoted ICO.

[0112] The description of the XTRM module below applies respectively to the XTRC module of [Fig.l] if it is present.

[0113] According to both embodiments (a) and (b), an XTRM module is used to format the data to be presented to the prediction neural network.

[0114] In embodiments, XTRM (resp. XTRC) uses an optional inverse quantization module to perform inverse quantization of the data extracted from the feature maps. This inverse quantization is mandatory if quantization has been performed at the encoder.

[0115] In embodiments, XTRM (resp. XTRC) performs a transformation of the values ​​of at least one of the characteristic maps, for example oversampling, interpolation, filtering, etc. At the end of the transformation, a transformed characteristic map is of the same resolution as the current image or sub-sequence to be decoded. Advantageously according to this mode, the characteristic maps which are decoded can be of lower resolution than that of the images to be decoded, while the transformed maps, which are used for the construction of the characteristic vectors, are at the same resolution as the sequence of images, which facilitates the extraction of the values.

[0116] In embodiments, a prediction sample (resp. a correction sample) is generated for each sample of the input image. The extraction of the values ​​is done according to the position of a sample being processed. For example, if one seeks to decode the sample Pn at the coordinates (xn, yn) of a current image, the XTRM module (resp. XTRC) performs an extraction of values ​​in the prediction maps (resp. correction) at positions imposed by the coordinates (xn, yn). In one embodiment, the extracted values ​​constitute the characteristic vector Zn. Zn is a J-tuple, that is to say that it comprises J elements, or data z;. The vector Zn of index n refers to the characteristic vector of the pixel Pn. according to one embodiment, J = NFP (resp. NFC).

[0117] In embodiments, XTRM (resp. XTRC) does not perform any transformation, in this case the values ​​that will be used to construct the characteristic vector are extracted from the untransformed prediction (resp. correction) characteristic maps. XTRM (resp. XTRC) can perform a processing of the extracted values, to generate the vector Zdn. XTRM (resp. XTRC) can perform an inverse quantization of the data extracted from the set of characteristic maps. The processing can include other operations, such as filtering, scaling, etc. In particular, if the characteristic maps are at resolutions lower than that of the images of the sequence, the XTRM (resp. XTRC) module can take into account the coordinates of the values ​​in the lower resolution maps.

[0118] The MSP module contains at least one so-called MLPP prediction neural network, defined by K WP parameters, capable of processing a vector Zn, or J-Uplet, as input, to generate as output a vector representative of the sample Pn being processed. The prediction neural network contains, according to one embodiment, an MLP, or Multi Layer Perceptron, consisting of an input layer adapted to the input format (the J-uplet), optionally one or more hidden layer(s), and an output layer adapted to the output format of the output vector, generally a vector comprising NCP (number of components of the prediction output vector) elements, or values, for each sample. It can also include a convolution neural network. The MLPP prediction neural network has the same structure as the encoder and the decoder, and its parameters are either identical if the coding of its WP parameters is lossless, or different if the coding is carried out with losses.When all the samples Pn of the image v have been processed, we have predicted information comprising as many decoded output vectors as there are samples in the image (N samples, N output vectors).

[0119] If the correction neural network is present, its structure can be identical to that of the motion synthesis module and when all the samples Pn of the image v have been processed, there is correction information comprising NCC decoded correction vectors, each comprising for example a triplet of correction values ​​for the components (R,G,B) or (Y,U,V) of the image. The correction information can for example correspond to a correction image IC of the same size as the image Iv.

[0120] According to the first mode, (a), represented on the left of the figure:

[0121] - MLPP performs motion synthesis from the prediction maps; the output vector is a motion information MV, for example the pair (mvx, mvy) of the motion displacement of each pixel Pn being processed

[0122] - the MSP module also includes a WARP compensation module motion. It performs motion compensation on one or more reference images using the generated motion field, an operation well known to a person skilled in the art. The images resulting from the compensation are denoted ICO. Warping can be a type of geometric transformation allowing the position of samples in an image to be modified. For example, each pixel in the reference image can be moved according to a motion vector (mvx, mvy) to obtain a predicted sample in an ICO image.

[0123] According to the second mode shown on the right, noted (b):

[0124] - MLPP performs prediction image synthesis from prediction maps and one or more reference images or maps; the output vector is a predicted image sample, for example a pixel (Y,U,V). The images resulting from the prediction are denoted ICO. The MLPP performs the field generation operations of movement and warping in a combined manner.

[0125] [Fig. 4] is a flowchart representing an example of a decoding method that can be implemented by the decoding device of [Fig. 1].

[0126] The decoding described relates to a current sub-sequence comprising at least one current image Iv of the sequence to be decoded.

[0127] During a step E30, the streams A1, A2, B1 and optionally B2 are extracted from the encoded stream. They respectively contain the coded representations of the current prediction maps FPc, the parameters WPc of the prediction neural network, the coded data FCc of the correction images or maps, the parameters WCc of the correction network if it exists.

[0128] During a step E31, the correction image(s) IC are decoded.

[0129] According to one embodiment, they are decoded by a conventional technique within the reach of those skilled in the art, for example a prediction followed by decoding by a contextual entropic coder then dequantization.

[0130] According to one embodiment, they are synthesized by a correction neural network (MLPC) which takes as input a set of correction feature maps FCc. In this case, the embodiments and variants presented below for the prediction feature maps and the prediction neural network apply mutatis mutandis.

[0131] During a step E32, the NFP maps FP of prediction characteristics are generated by decoding the FPc values. For this decoding, a technique is used, for example, for predicting a characteristic map value by its neighborhood followed by entropic decoding. In one embodiment, the FP maps are decoded in order (FPi, FP2,... FP4), and the values ​​of each map in a predefined order, for example lexicographic.

[0132] According to embodiments:

[0133] - The FP maps have the same resolution as the Iv signal to be reconstructed, i.e. that they contain as many values ​​as samples (N) in image Iv.

[0134] - FP maps are at a resolution lower than or equal to that of the Iv signal to re constitute and therefore include, for at least one of them, a number of values ​​to be decoded less than N. According to a variant, the first map is at the resolution of the images and each following map is at a resolution half that of the previous one.

[0135] - Several FP cards are of the same resolution, lower than the signal resolution.

[0136] During a step E33, according to embodiments, the M FP cards are de quantified, transformed, and data are extracted from them by an XTRM module as presented in support of [Fig.3].

[0137] According to one embodiment, at least one FP card has a lower resolution than that of the decoded images and the transformation operation includes an oversampling Upsampling is done so that the transformed map has the same number of samples as the images in the sequence. Upsampling consists of adding values ​​to the maps to achieve the resolution of the images in the input sequence. It can be simple (by nearest neighbor replication) or involve interpolation (linear, polynomial, filtering, etc.). It can be performed by an upsampling neural network.

[0138] According to one embodiment, the XTRM module performs an extraction of values ​​in the transformed characteristic maps, for a sample Pn to be decoded, according to its coordinates in the signal. The extracted values ​​constitute the vector Zn. Zn is a J-tuple, that is to say that it comprises J elements, or data z;. The vector Zn of index n refers to the characteristic vector of the pixel Pn.

[0139] In one embodiment, Zn is a J-tuple (zb z2,..., Zj), consisting of the values ​​of the maps (possibly transformed) located at the coordinates (xn, yn) of the current pixel Pn. In one embodiment, Zn comprises as many values ​​as there are input maps. In one embodiment, Zn is a J-tuple constructed from values ​​taken from the maps at coordinates that may be different depending on the maps. For example, if the maps are at different resolutions, the coordinates are adapted (by scaling) to correspond to the resolution of each map. In one embodiment, Zn is a J-tuple constructed from values ​​taken from the maps by applying the processing to one or more values ​​of the maps, for example a filtering of the values ​​neighboring the targeted value in a map.For example, for a current sample Pn in an FP card which is at the same resolution as the image signal to be decoded, we can extract the values ​​located at the coordinates (xn, yn), (xn-l, yn), (xn, yn-l) and (xn-l, yn-l) and apply a processing to these values ​​(filtering, averaging, interpolation, application of any function, preferably monotonic, etc.) to obtain the final value (z;) of the element i of the vector Zn relative to this card. According to another example, in a map which is at half the resolution of the signal, we can consider the values ​​located at the coordinates (xn / 2, yn / 2), (xn / 2-1, yn / 2), (xn / 2, yn / 2-1) and (xn / 2-l, yn / 2-l) and apply a processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (¾) of the element i of the vector Zn relative to this FP map;. The processing can include a quantification of the extracted values ​​or of the vector Znconstituted, if necessary.The samples to be decoded are for example processed in sequential order, from n=1 to n=N.

[0140] During a step E34, the WP parameters of the MLPP prediction neural network are generated by decoding the WPc values ​​of the stream A2. For this purpose, any known neural network decoding technique corresponding to the coding technique used by the coder can be used, for example the neural network coding standard proposed by the MPEG-7 part 17, NNR standard.

[0141] During a step E35, FREF reference maps are obtained by the REF module, from among the already decoded images associated with the sequence. The REF module constitutes a set of NFRMAX reference images. Then a number NFR of reference images is selected from this set. According to one embodiment, a single image is selected. According to one embodiment, a reference image is represented by feature maps in the latent domain. For example, the reference image considered for the current image is the previous image of the sequence, or the first, or the last, or an Intra image, etc. The reference image can thus be predetermined. Its (their) reference can alternatively be coded in the stream in the form of an indicator.

[0142] During a step E36, the values ​​extracted from the feature maps are processed by the MLPP prediction neural network to generate prediction values. According to an embodiment as shown in [Fig.3], case (b), the selected reference images or maps are also presented as input to the neural network. According to one embodiment, a vector Zn is processed by an MLPP prediction neural network to generate an output vector containing NCP values ​​for the sample Pn to be decoded.

[0143] According to one embodiment, as shown in [Fig.3](a), the output vector is a motion vector MVn. According to one example, NCP is equal to 2 and the two-dimensional output vector (mvx, mvy) comprises the values ​​of the motion vector horizontally and vertically. When all the samples of the signal have been processed, a dense motion field (i.e., one vector per pixel) is available. For example, this field is available in the form of two images, the first image comprising the horizontal coordinate (x) of the motion vector, and the second image comprising the vertical coordinate (y) of the motion vector.

[0144] According to this embodiment, during a step E37, the reference image (or images) are compensated, that is to say that the previously determined motion field is applied to them by the WARP module to produce one or more ICO compensated images. According to one embodiment, the compensation may be a geometric transformation making it possible to modify the position of the samples in an image. For example, the pixels of the reference image may be moved according to the associated motion vector (mvx, mvy) to obtain the predicted samples in the ICO image. The result may be continuously interpolated, spatially sampled at the sampling rate of the image to be reconstructed. Other more complex transformations, carried out on sets of pixels, may be used (application of a polynomial, homographic, affine model, etc.) depending on the number of parameters delivered by the prediction neural network.It is also possible to compensate several reference images to obtain a current image. In this case, the MV motion field has four values, or components, for a pixel (2 for each image). The compensated images are then merged (e.g. by averaging or linear combination, or weighted prediction, etc.) to produce at least one ICO compensated image.

[0145] According to another embodiment, as shown in [Fig.3](b), the reference images are also applied, or presented, to the input of the MLPP network. The reference images and the prediction feature maps may be combined before entering the MLPP network. The output vector is a predicted image sample, for example a pixel (Y,U,V). When all the signal samples have been processed, a prediction image is available, for example in the form of three components (YUV). In this embodiment, the WARP module is not used.

[0146] During a step E38, a combination of the compensated ICO images and the IC correction images is performed by the MIX module. The combination can be, in its simplest form, an addition of the values ​​of the prediction image and those of the correction image. According to another embodiment, the MIX module uses a neural network which takes the ICO and IC images as input and produces, for example using a series of convolutional layers, the image Iv.

[0147] [Fig.5] is a flowchart representing an example of a coding method that can be implemented by the coding device of [Fig.2]

[0148] It is recalled that during coding, the entire system is optimized during a learning phase which precedes the actual coding phase: the values ​​of the FPi and optionally FCi maps as well as the parameters of the MLPP and optionally MLPC networks are learned (in the sense of machine learning, for example by gradient descent) so as to optimize a performance measure, for example a cost function of the rate-distortion type noted D+ LR, where D is the quadratic error of the reconstructed image Iv with respect to the original image, R is the sum of the coding rates of the feature maps and optionally of the neural networks, and L a parameter set by the user for the RD-OPT module. According to another example, D is calculated from a perceptual function such as SSIM (for Structural SIMilarity), or MSSSIM (for Multi-Scale Structural SIMilarity).

[0149] During a step E20, an input sequence SV to be coded, comprising at least one current image Iv comprising a plurality of N samples Pn, is provided as input to the method.

[0150] During a step E21, the prediction feature maps and the correction images or maps, as well as the parameters of the neural networks, are initialized. Subsequently, these data are optimized during the construction phase. According to a embodiment, the feature maps are initialized by predefined constant values. According to another embodiment, the feature maps are initialized by a set of random real values. According to one embodiment, the parameters of the neural network are initialized by predefined values ​​known to give a satisfactory result (for example, following training on a corpus of image sequences). According to another embodiment, the parameters of the neural network are initialized by a set of random values. The prediction feature maps, images or correction maps and parameters of the neural networks are subsequently updated, or refined, during a step E22, by the update module MAJ of the encoder during its training.

[0151] During a step E23, a coding simulation is carried out for all the coding parameters: the prediction characteristic maps are encoded by the FPC module, the correction characteristic images or maps by the FDC module, the neural network(s) by the NNC module. The simulation may be identical to the actual coding but it may also be different (for example, simplified).

[0152] During a step E24, a decoding simulation is performed. To measure the distortion D, it is indeed necessary to simulate the coding then the decoding of at least part of the sequence of images, to obtain at least one pixel P'nd of at least one image l'v resulting from a simulation of coding then decoding of the samples of index n, then to measure the difference between these coded then decoded pixels and the pixels of the corresponding uncoded sequence. For this purpose, steps E31 to E38 of the decoding method are performed in order to reconstruct the current image or sub-sequence l'v decoded.

[0153] During a step E25, the performance measurement is evaluated.

[0154] For this purpose, the coding simulation rates associated with the feature maps (simulation of the flow A1 by coding the feature maps and of the flow B1 by coding the correction image or its feature maps). According to one embodiment, the coding simulation rates associated with the parameters of the neural network (simulation of the flow A2 by coding the WP parameters and optionally of the flow B2 by coding the WC parameters) are also measured. According to one embodiment, the coding of the parameters of the neural networks is not simulated, because their influence is less important than that of the feature maps.

[0155] As long as the cost function has not reached its minimum, the performance measurement is not satisfactory, and the method is repeated from step E22. Alternatively, the method can be interrupted after a predefined time or a predefined number of iterations, so as to control its complexity or duration.

[0156] During a step E26, if the cost function has reached its minimum or the time allocated to the encoding is deemed sufficient, the training stops. If an encoded version corresponding to the last simulation of the neural network parameters and feature maps is available, the streams A1, A2, B1 and B2 can be made up of them. According to another embodiment, the actual encoding of the updated parameters of the neural network(s) and the feature map values ​​is performed at this stage to produce the encoded parameters.

[0157] The streams A1, A2, B1 and B2 can be concatenated to produce a final stream. According to one embodiment, the stream A2 / B2 of the coded parameters of the neural network is stored or transmitted before the stream A1 / B1, so that it can be decoded before.

[0158] [Fig.6] illustrates an image decoding method of correction used in one embodiment of the invention.

[0159] In this embodiment, the correction images are synthesized (at encoding and decoding) by an MLPC neural network applied to FC correction feature maps processed by an XTRC module.

[0160] During decoding, the FC maps are obtained as described previously by a conventional decoding (for example an entropic decoding) from the stream Bl, by the FCD module.

[0161] In this embodiment, there are 4 current decoded cards FCi. In a preferred embodiment, there are 7. In this embodiment, the first card FCi has the same resolution as the image Iv to be decoded, and therefore comprises WxH variables, where W represents the width of the image in pixels, and H its height. The second card FC2 has half the resolution (in each dimension) of the card FCp. Each additional card has half the resolution of the previous card. This structure makes it possible to reduce the number of variables in the characteristic cards, which facilitates decoding while minimizing the coding cost.

[0162] The FCi cards are then processed by the XTR module, which can include all of the following steps:

[0163] - a step noted DQi (DQ1... DQ4) of dequantization of each card, the dequantization tification consisting for example of multiplication by a predefined or transmitted quantization step;

[0164] - a step noted SEi (SEL.. SE4) of oversampling likely to put the FCi card at the same resolution as the signal to be coded; according to the example shown, the FC2 card is oversampled by a factor of 2 by the SE2 module in each dimension, according to any oversampling method within the reach of those skilled in the art. The FC3 card is oversampled by a factor of 4 in each dimension, etc.

[0165] - a step noted Ti (Tl... T4) of improvement processing, this step being able include convolutional filtering, a linear function, or even a neural network whose W1 parameters have been decoded in a classical way (decoding classic or NNR) from the B2 stream.

[0166] The steps thus described can be "mixed", in the sense that the oversampling can be based on a neural network. They can also be joint, that is to say applied to a set of maps and not to a single map as shown in [Fig.6].

[0167] According to this embodiment, these steps produce the FCSi maps (FCS1... FCS4), which are at the same resolution as the signal to be coded, and therefore comprise WxH values, where W represents the width of the image in pixels, and H its height. The transformed maps are processed by an MLPC neural network, typically a Multi-Layer Perceptron, which associates with an input vector Zn consisting of the values ​​of the FCSi layers at the coordinates (x,y) of a pixel being processed an output vector (R,G,B) at the coordinates (x,y) in a series of layers DR,DG,DB. These layers are for example of the same number as the number of components of the output image (in figure 3). In this embodiment, the vector Zn is a 4-tuple (zi, z2,z3, z4) consisting of the 4 values ​​of the FCSi maps located at the coordinates (x,y) of the current pixel Pn. The triplet (R, G, B) is inserted into each decoded image Iv of the subsequence v at the coordinates (x, y) in the color components (DR,DG,DB).

[0168] The WC parameters of the MLPC network were decoded from the B2 stream by conventional or NNR decoding.

[0169] Finally, a last PPC post-processing network, whose W2 parameters have also been decoded from the stream, can process the DR, DG, DB layers to produce the decoded image IC. The PPC network may comprise a convolutional network, an MLP or any other neural network or conventional processing.

[0170] According to a variant not shown, additional correction feature maps are introduced. In such a variant, the vector Zn is of dimension greater than 4. Such an additional map typically comprises data which can assist the MLPC network in the task of reconstructing the images. Thus, the added maps can be, in a non-limiting manner, one or more of:

[0171] - a map including at each point the abscissa and / or the ordinate of this point;

[0172] - a map comprising at each point a positional coding;

[0173] - a map representing an image distinct from the images being processed, above capable of providing information about the images to be coded, for example a previously processed image or sequence of images;

[0174] - a map containing data representative of the time difference between the frames of the video being encoded. For example, if the first and last frames of the video are 8 frames apart, all samples in the map contain the value 8;

[0175] - a map representing a feature map of a distinct image of the images being processed, which may provide information on the images to be coded, for example a previously processed map;

[0176] - a card containing the value of an already decoded sample of the same card, for example example the previous sample in the decoding order.

[0177] [Fig.7] illustrates a motion field decoding method used in an embodiment of the invention according to [Fig.3], case (a).

[0178] In this embodiment, the motion field is synthesized (at encoding and decoding) by an MLPP prediction neural network applied to FP prediction feature maps processed by an XTRM module.

[0179] During decoding, the FP maps are obtained as described previously by a conventional decoding (for example an entropic decoding) from the flow Al, by the FPD module.

[0180] In this embodiment, the current decoded cards FPi are 4 in number. In a preferred embodiment, there are 7 of them. In this embodiment, the first card FPi has the same resolution as the image Iv to be decoded, and therefore comprises WxH variables, where W represents the width of the image in pixels, and H its height. The second card FP2 has half the resolution (in each dimension) of the card FPi. Each additional card has half the resolution of the previous card. This structure makes it possible to reduce the number of variables of the characteristic cards, which facilitates decoding while minimizing the coding cost.

[0181] The FPi cards are then processed by the XTRM module, which can include all of the following steps:

[0182] - a step noted DQi (DQ1... DQ4) of dequantization of each card, the dequantization tification consisting for example of multiplication by a predefined or transmitted quantization step;

[0183] - a step noted SEi (SEL.. SE4) of oversampling likely to put the FPi card at the same resolution as the signal to be coded; according to the example shown, the FP2 card is oversampled by a factor of 2 by the SE2 module in each dimension, according to any oversampling method within the reach of those skilled in the art. The FP3 card is oversampled by a factor of 4 in each dimension, etc.

[0184] - a step noted Ti (Tl... T4) of improvement processing, this step being able to include convolutional filtering, a linear function, or even a neural network whose W3 parameters have been decoded in a classical manner (classical decoding or NNR) from the A2 stream.

[0185] The steps thus described can be nested, in the sense that the oversampling can be based on a neural network. They can also be joint, that is to say applied to a set of maps and not to a single map as re- presented in [Fig.7].

[0186] According to one embodiment, these steps produce the FPSi maps (FPS1... FPS4), which are at the same resolution as the signal to be coded, and therefore comprise WxH values, where W represents the width of the image in pixels, and H its height. The transformed maps are processed by an MLPP prediction neural network, typically a Multi-Layer Perceptron, which associates with an input vector Zn consisting of the values ​​of the FPSi layers at the coordinates (x,y) of a pixel being processed an output vector (mvx, mvy) at the coordinates (x,y) in a series of motion layers. These layers are for example of the same number as the number of components of the output motion vector (in the figure, 7: MVX comprises the horizontal component of the motion vectors, and MVY the vertical component). In this embodiment, the vector Zn is a 4-tuple (zt z2z3, z4) consisting of the 4 values ​​of the FPSi maps located at the coordinates (x, y) of the current pixel Pn.The pair (mvx, mvy) is inserted into each motion layer MVX, MVY of the image or the current subsequence v at the coordinates (x, y). The concatenation of all these vectors constitutes a dense motion field.

[0187] According to other embodiments: - the output motion vector is representative of a group of pixels (for example a block); - different parameters are obtained at the output: affine motion parameters, etc. - the current subsequence corresponds to several images and the movement is synthesized for the subsequence; - etc.

[0188] The WP parameters of the MLPP prediction neural network were decoded from the A2 stream by conventional or NNR decoding. The prediction neural network used for encoding, and the MLPP prediction neural network used for decoding, are defined by an identical structure, comprising for example a plurality of layers of artificial neurons, and by a set of weights and activation functions associated respectively with the artificial neurons of the network concerned. Finally, a last PPM post-processing network, whose W4 parameters were also decoded from the stream, can process the MVX, MVY layers to produce the final motion information MV for the current image. The PPM network can be a convolutional network, an MLP or any other neural network or conventional processing.The PPM network can also integrate the WARP compensation module, in which case it takes as input not only motion information but also reference images or maps.

[0189] According to a variant not shown, maps of sup prediction characteristics additional dimensions are introduced. In such a variant, the vector Zn is of dimension greater than 4.

[0190] Such an additional map typically comprises data that can assist the MLPP network in the task of motion reconstruction. Thus, the added maps may be, but are not limited to, one or more of:

[0191] - a map including at each point the abscissa and / or the ordinate of this point;

[0192] - a map comprising at each point a positional coding;

[0193] - a map representing an image distinct from the images being processed, above capable of providing information about the motion of the images to be encoded, for example an image, a sequence of images or previously processed motion layers;

[0194] - a map containing data representative of the time difference between the frames of the video being encoded. For example, if the first and last frames of the video are 8 frames apart, all samples in the map contain the value 8;

[0195] - a map representing a feature map of a distinct image of the images being processed, which may provide information on the images to be coded, for example a previously processed motion map;

[0196] - a card comprising the value of a motion vector already decoded from the same map, for example that of the previous sample in the decoding order.

[0197] - a map comprising the quantification precision of the characteristic values.

[0198] etc.

Claims

Claims

1. Method for coding a sequence of images (SV) comprising the following steps for at least one current image (Iv) of the sequence: - initialization (E21) of at least one correction image (IC); - initialization (E21) of a set of parameters (WP) representative of a prediction neural network (MLPP); - initialization (E21) of a set of so-called prediction feature maps (FP); - obtaining (E24) a set of reference images (FREF), comprising at least one coded then decoded reference image; - processing (E24) of said prediction characteristic maps (FP) and of said reference images by a synthesis module comprising said at least one prediction neural network to the input of which said prediction characteristic maps are applied, to produce at least one prediction image (ICO); - reconstruction (E24) of the current image (Iv) from said at least one prediction image (ICO) and said at least one correction image (IC); - updating (E22, E25) at least one value of said at least one characteristic map and / or at least one parameter of said network, as a function of a coding performance measurement. - coding (E26) of a binary stream comprising: - coding of said at least one correction image (IC); - coding of the set of parameters (WPc) representative of the prediction neural network (MLPP); - encoding of the set of prediction feature maps (FPc);

2. Method for decoding a sequence of images (SV) from a binary stream, comprising the following steps for at least one current image (Iv) of the sequence: - decoding (E31) of a set of so-called correction images (IC) comprising at least one correction image representative of correction data of said at least one current image (Iv); - decoding (E32) of a set of so-called prediction feature maps (FPc) comprising at least one feature map representative of a movement of said at least one current image; - decoding (E34) of a set of parameters (WPc) representative of a prediction neural network (MLPP); - obtaining (E35) a set of reference images (FREF), comprising at least one decoded reference image; - processing (E36) said prediction characteristic maps (FP) and said reference images (FREF) by a synthesis module (MPS) comprising said at least one prediction neural network to the input of which said prediction characteristic maps are applied, to produce at least one prediction image (ICO); - reconstruction (E38) of the current image (Iv) from said at least one prediction image (ICO) and said at least one correction image (IC).

3. Coding or decoding method according to one of the preceding claims, characterized in that the processing step comprises the following sub-steps: - application (E33, E34) of said prediction characteristic maps (FP) to the input of said at least one prediction neural network to produce motion information (MV); - compensation (E37) of said at least one reference image (FREF) using said motion information to produce said at least one prediction image (ICO).

4. Coding or decoding method according to the preceding claim, characterized in that the step of applying said prediction characteristic maps to the input of said at least one prediction network comprises, for at least one sample, called current sample (Pn), of said at least one current image (Iv), associated with a position (xn, yn) in the image: - construction (E33) of a characteristic vector (Zn) from said at least one prediction characteristic map, as a function of said position (xn, yn) of said at least one current sample, and: - application (E34) of said vector (Zn) to the input of said prediction neural network (MLPP) to provide a vector representative of the prediction of said at least one current sample (Pn).

5. Coding or decoding method according to one of claims 1 or 2, characterized in that the processing step comprises a step of applying (E33, E34, E35) said prediction characteristic maps (FP) and said at least one reference image (FREF) to the input of said at least one prediction neural network to produce said at least one prediction image (ICO).

6. Encoding or decoding method according to the preceding claim, characterized in that the step of applying said prediction characteristic maps and said at least one reference image to the input of said at least one prediction network comprises, for at least one sample, called current sample (Pn), of said at least one current image (Iv), associated with a position (xn, yn) in the image: - construction (E33) of a characteristic vector (Zn) from said at least one prediction characteristic map and said at least one reference image as a function of said position (xn, yn) of said at least one current sample, and: - application (E34) of said vector (Zn) to the input of said prediction neural network (MLPP) to provide a vector representative of the prediction of said at least one current sample (Pn).

7. Decoding method according to one of claims 2 to 6, characterized in that the step of decoding said set of correction images further comprises the following sub-steps: - decoding (E31) of a set of parameters (WCc) representative of a correction neural network (MLPC); - decoding (E31) of a set of so-called correction characteristic maps (FCc); - application (E31) of said decoded correction characteristic maps (FC) to the input of said at least one correction neural network to produce at least one correction image (IC).

8. Decoding method according to the preceding claim, characterized in that the correction neural network (MLPC) comprises an MLP and / or a convolutional network.

9. Encoding or decoding method according to one of the preceding claims, characterized in that the prediction neural network (MLPP) comprises an MLP and / or a convolutional network.

10. Coding or decoding method according to one of the preceding claims, characterized in that the reconstruction step (E38) comprises an addition or a subtraction applied to the correction and prediction images.

11. Coding or decoding method according to one of the preceding claims, characterized in that the reconstruction step (E38) uses a convolutional neural network applied to the correction and prediction images.

12. Encoding or decoding method according to claim 3 or 4, characterized in that said motion information is a dense field of two-dimensional motion vectors.

13. Device for decoding a sequence of images (SV) from a binary stream, said device being configured to implement, for at least one current image (Iv) of the sequence: - decoding (FCD) of a set of so-called correction images (IC) comprising at least one correction image representative of correction data of said at least one current image (Iv); - decoding (FPD) of a set of so-called prediction feature maps (FPc) comprising at least one feature map representative of a movement of said at least one current image; - decoding (NND) of a set of parameters (WPc) representative of a prediction neural network (MLPP); - obtaining (REF) a set of reference images (FREF), comprising at least one decoded reference image; - processing said prediction feature maps and said reference images by a synthesis module comprising said at least one prediction neural network to the input of which said prediction feature maps are applied, to produce at least one prediction image; - reconstruction (MIX) of the current image (Iv) from said at least one prediction image (ICO) and said at least one correction image (IC).

14. Device for coding a sequence of images (SV), said device being configured to implement for at least one current image (Iv) of the sequence: - initialization of a set of so-called correction images (IC); - initialization of a set of parameters (WPc) representative of a prediction neural network (MLPP); - initialization of a set of so-called prediction feature maps (FPc); - obtaining a set of reference images (FREF), comprising at least one coded then decoded reference image; - processing said prediction feature maps and said reference images by a synthesis module comprising said at least one prediction neural network to the input of which said prediction feature maps are applied to produce at least one prediction image; - reconstruction of the current image (Iv) from said at least one prediction image (ICO) and said at least one correction image (IC); - updating at least one value of said at least one feature map and / or at least one parameter of said network, based on a coding performance measurement. - coding of a binary stream comprising: - coding (FCC) of said at least one correction image (IC); - coding (NNC) of the set of parameters (WPc) representative of the prediction neural network (MLPP); - coding (FPC) of the set of prediction feature maps (FPc).

15. A computer program comprising instructions for carrying out the steps of an encoding or decoding method according to claim 1 or 2 when said program is executed by a computer.

Images