Method and device for encoding and decoding image sequences

Abstract

The invention relates to a method for encoding and decoding an image sequence. The decoding method comprises, for at least one current image of the sequence: - decoding a set of images, referred to as correction images, comprising at least one correction image representative of correction data for said at least one current image; - decoding a set of feature maps, referred to as prediction feature maps, comprising at least one feature map representative of a movement of said at least one current image; - decoding a set of parameters representative of a prediction neural network; - obtaining a set of reference images, comprising at least one decoded reference image; - processing said prediction feature maps and said reference images using a synthesis module comprising said at least one prediction neural network at the input of which said prediction feature maps are applied, so as to produce at least one prediction image; - reconstructing the current image from said at least one prediction image and said at least one correction image.

Description

Method and device for coding and decoding image sequences.

[0001] The invention relates to the general field of coding digital image sequences. It relates more particularly to the compression of digital videos.

[0002] Digital videos are generally source-encoded to compress them in order to limit the resources required for their transmission and / or storage. There are many encoding standards, such as the ITU / MPEG standards (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) and their extensions (MVC, SVC, 3D-HEVC, etc.).

[0003] Encoding an image is generally performed by dividing the image into several rectangular blocks, and encoding these blocks of pixels according to a given processing sequence. In existing video compression techniques, the processing of a block typically includes a prediction of the pixels of the block performed using previously coded and then decoded pixels present in the image being encoded, in which case we speak of "Intra prediction", or previously coded images, in which case we speak of "Inter prediction". This exploitation of spatial and / or temporal redundancies makes it possible to avoid transmitting or storing the value of the pixels of each block of pixels, by representing at least some of the blocks by a residual representing a difference between the prediction values ​​of the pixels of the block and the actual values ​​of the pixels of the predicted block.

[0004] As video formats continue to evolve to compress more and more and adapt to the variety of expected formats and communication networks, the possibilities for prediction are becoming ever greater and traditional coding and decoding algorithms are becoming very complex.

[0005] In addition to these classic approaches proposed by compression standards (MPEG, ITU), approaches based on artificial intelligence, and in particular neural ones, tend to develop.

[0006] Some of these neural approaches can be seen as a simple extension of the competition notion of the aforementioned compression techniques, such as the prediction and transformation mode competition in video coding.

[0007] Other approaches use the concept of "autoencoder." Autoencoders are learning algorithms based on artificial neural networks, which allow the construction of a new representation of a dataset. The architecture of an autoencoder consists of two parts: the encoder and the decoder. The encoder consists of a set of layers of neurons, which process the data in order to construct new representations called "encoded," also called "latent representations." In turn, the layers of neurons in the decoder receive these representations and filter them in order to try to reconstruct the initial data. The differences between the reconstructed data and the initial data make it possible to measure the error made by the autoencoder. Training consists of modifying the parameters of the autoencoder in order to reduce the reconstruction error measured on the different samples of the dataset.The performance of such autoencoder-based systems comes at the cost of a considerable increase in memory footprint and complexity compared to conventional approaches as proposed by compression standards. They can have millions of parameters and may require up to a million MACs (multiply-accumulate) to decode a single pixel. This makes such decoders significantly more complex than conventional decoders, which could hamper the adoption of learning-based compression.

[0008] More recently, a simple image encoding technique based on a neural network was described in the article "COmpression with Implicit Neural representations" by Émilien Dupont et al. (arXiv:2103.03123). The proposed encoding technique consists of fitting a neural network to an image, quantizing the network weights, and transmitting them. At decoding time, the neural network is evaluated at each pixel position to reconstruct the image. However, such a technique remains inefficient in terms of compression and requires coding the video frames independently.

[0009] In the video domain, an equivalent technique was presented in the article “Scalable Neural Video Representations with Learnable Positional Features” (NeurIPS 2022) by Subin Kim et al. The temporal dimension is taken into account. The coding algorithm generates a set of three two-dimensional latent keyframes, one on each spatio-temporal axis, to learn the common frames of the video. It also generates a latent 3D representation grid to capture the local details of the video. This representation is used to modulate a neural network trained to process the three keyframes. However, such an approach lacks flexibility in the choice of representation of the keyframes (the number of which is fixed at 3) and the 3D grid (fixed for the entire sequence). Moreover, it requires the processing of a transmitted 3D grid, which is both heavy in terms of the amount of transmitted data and the memory required for its processing.

[0010] Patent document published under number US 2023 / 0145525 A1 describes an image decoding method comprising obtaining a current optical flow and correction data generated on the basis of a predicted image and a current image. The optical flow is encoded and decoded by a neural network. Feature data representative of the correction data are obtained via a neural network-based image encoder; and the current reconstructed image is obtained by combining the correction data and the predicted image. However, such a method requires storing neural networks that are very large.

[0011] There is therefore a need for a solution that allows a sequence of images to be encoded / compressed in a simple and efficient manner.

[0012] The invention relates to a method for decoding a sequence of images from a binary stream, comprising the following steps for at least one current image of the sequence to be decoded: - decoding a set of so-called correction images comprising at least one correction image representative of correction data of said at least one current image; - decoding a set of so-called prediction characteristic maps comprising at least one characteristic map representative of a movement of said at least one current image; - decoding a set of parameters representative of a prediction neural network; - obtaining a set of reference images, comprising at least one decoded reference image;- processing of said prediction characteristic maps and said reference images by a synthesis module comprising said at least one prediction neural network to the input of which said prediction characteristic maps are applied, to produce at least one prediction image; - reconstruction of the current image from said at least one prediction image and said at least one correction image.;

[0013] The invention also relates to a method for coding a sequence of images comprising the following steps for at least one current image of the sequence: - initialization of a set of so-called correction images; - initialization of a set of parameters representative of a prediction neural network; - initialization of a set of so-called prediction characteristic maps; - obtaining a set of reference images, comprising at least one coded then decoded reference image; - processing of said prediction characteristic maps and said reference images by a synthesis module comprising said at least one prediction neural network to the input of which said prediction characteristic maps are applied, to produce at least one prediction image; - reconstruction of the current image from said at least one prediction image and said at least one correction image;- updating at least one value of said at least one feature map and / or at least one parameter of said network, as a function of a coding performance measurement.- a coding step in the binary stream comprising:- coding of said at least one correction image;- coding of the set of parameters representative of the prediction neural network; - coding of the set of prediction feature maps.;

[0014] For the purposes of the invention, encoding, or "coding", means the operation which consists of representing a set of samples, or pixels, in a compact form carried for example by a digital binary train. "Decoding" means the operation which consists of processing a digital binary train to restore decoded samples or pixels.

[0015] By "sequence of images" is meant a plurality of ordered two-dimensional images Iv, for example temporally in the case of a video. According to one example, the sequence corresponds to a scene. The images may comprise several components, for example to represent the colors, in RGB or YUV composite form for example. According to one example, the sequence corresponds to a set of predefined images, for example a fixed number, or, within the meaning of the MPEG standards, a GOP (Group Of Pictures) comprising the images located between two images of the Intra-image type, also called "intra period". According to another example, the images may be views of the same scene represented in multi-views. According to another example, the images may be a plurality of temporal and multi-view images (immersive video).

[0016] By "at least one current image" is meant a sub-sequence of at least one image of the sequence, currently being encoded or decoded, comprising a plurality of N samples.

[0017] A "sample" is a value taken from an image in the sequence. Sampling a signal produces a series of discrete values ​​called samples. In the case of an image signal, the sample is called a pixel, which can be a color pixel traditionally represented by a triplet of values, for example (R,G,B) or (Y,U,V), each value belonging to a map of components that may or may not be the same size. The position of the sample is identified by its abscissa (x) and ordinate (y) coordinates in the image, or in one of the components.

[0018] By "correction image" is meant a concrete or abstract representation of an image, comprising a plurality of variable scalar data, also called correction values, for example real numbers, or integers, signed or unsigned. The set of correction data of an image can correspond to a single correction image, or several correction images, the dimension of which can be arbitrary. A correction image or map corrects a prediction image or map to produce an approximation of the current image to be encoded or decoded.

[0019] By "neural network" is meant a neural network such as a convolutional neural network, a multi-layer perceptron, an LSTM (for "Long Short Term Memory"), etc. The neural network is defined for example by a plurality of layers of artificial neurons and by a set of preferably non-linear activation, weighting and addition functions (for example, a layer can calculate y = f(Ax+b), where y and b are vectors of dimension N, x is a vector of dimension M, A is a matrix of dimension MxN, and f is the activation function). The input of the neural network consists of data extracted from the feature maps, for example in the form of vectors.

[0020] By "neural network parameter" we mean one of the values ​​that characterizes the neural network, for example a weight associated with one of the neurons (filter coefficient, weighting, bias, value affecting the operation of non-linearity, etc.)

[0021] A "feature map" means all or part of an abstract representation of an image in a latent domain, comprising a plurality of variable scalar data, also called characteristic values, for example real numbers, or integers, signed or unsigned. The abstract representation of an image may therefore comprise several feature maps. The abstract representation of an image may also correspond to a single feature map comprising a set of values ​​associated with a position.

[0022] A "latent domain" is a representation composed of a set of 2D feature maps, the size of which is proportional to the input image, or 1D vectors or scalar values.

[0023] By "processing by a synthesis module" is meant the synthesis of one or more motion-compensated images. Such a synthesis module comprises at least one prediction neural network, to the input of which are applied at least the prediction characteristic maps, in the form of input vectors called characteristic vectors extracted from the characteristic maps. Such a module may comprise other modules, such as for example a motion compensation module. According to embodiments, reference maps or images are also applied to the input of the prediction neural network. The application of a function expressed by the neural network to an input characteristic vector produces an output vector representative of a prediction of the samples to be coded (resp. decoded).This output vector corresponds to motion information relating to the current image or sub-sequence of the sequence, or to a sample of the motion-compensated image.

[0024] A "reference image" means a decoded image resulting from a prior decoding of an image in the sequence. For example, it can be a previous image in the sequence, or a key image (Intra image, first image of the sequence, etc.) or a following image in the sequence if the decoding order allows it. It can be represented in the image domain (as a reference image) or in the latent domain (as feature maps of latent values).

[0025] By "reconstruction of the current image" we mean a combination of the obtained prediction image(s) and the obtained correction image(s). This combination may involve, for example, an addition or subtraction of the two types of images. It may also involve more complex processing, for example a combination by neural network.

[0026] By "performance measurement" is meant a measurement between at least one value of a sample to be coded and a decoded value of said sample. The measurement can evaluate, for example, a distortion, or a perceptual error. It can be carried out on a sample or a plurality of samples (for example, current samples of the current image, or of a current subsequence, etc.) The measurement can also include a measurement of the bit rate, in particular associated with the coding of the neural network and / or the coding of the feature maps. The measurement can be a joint measurement between the bit rate and the distortion through their weighting. As is well known in the state of the art, the value of this measurement is generally minimized.

[0027] Generally speaking, it is considered that the steps of an coding or decoding method should not be interpreted as being linked to a notion of temporal succession. In other words, the steps may be carried out in a different order than that indicated in the independent coding or decoding claim, or even in parallel.

[0028] When the training is complete, i.e. the performance measurement obtained is satisfactory, or a predetermined training time has been reached, the actual coding of the parameters of the neural network and the values ​​of the feature maps associated with the current image or sub-sequence can be performed and stored or transmitted to the decoder. Advantageously, at the encoder, the training process makes it possible to refine the parameters of the prediction neural network and / or the values ​​of the prediction feature maps as well as the data of the correction images until an adequate representation in terms of performance is obtained, for example a desired balance between the generated bit rate and the distortion undergone by the image or sub-sequence of images being coded. Advantageously, the coding method according to the invention makes it possible to efficiently compress the signal.

[0029] Advantageously, the decoding method is simple since it is sufficient to decode the prediction feature maps relating to the current image or subsequence and the parameters of the neural network to reconstruct a decoded version of the prediction image. Such a neural network can advantageously be of very simple structure with few parameters.

[0030] Advantageously, feature maps dynamically capture the local features of the current image or subsequence, since they are encoded (resp. decoded) as the sequence is processed. It is therefore possible to design a transmission system that works image by image with low latency, with each current image or subsequence being decoded upon receipt of the current maps and the associated neural network.

[0031] According to one embodiment of the invention, such a coding or decoding method is further characterized in that the processing step comprises the following sub-steps: - applying said prediction feature maps to the input of said at least one prediction neural network to produce motion information; - compensating said at least one reference image using said motion information to produce said at least one prediction image.

[0032] By "motion information" is meant a set of data of any nature to describe the movement of a sample or part of an image, for example a field of motion vectors per sample, called a dense motion field, or optical flow, or a block motion field, or a set of equations representing an affine movement of one or more samples of the image, or an angle and an amplitude of a motion vector per sample or per block, etc.

[0033] By "reference image compensation" is meant the application of motion information to the selected reference image(s) or map(s) to generate a motion-compensated image, called a prediction image.

[0034] Advantageously, according to this mode, the generation of the prediction image takes place according to two traditional steps well known to those skilled in the art, namely the generation or decoding of the movement, synthesized here by a prediction neural network from the prediction maps, followed by compensation of reference images by this movement. The compensation operation, well known, consists of applying the movement obtained to a pixel or a set of pixels of the reference image.

[0035] According to a variant of this embodiment of the invention, the step of processing said prediction characteristic maps by said at least one prediction neural network comprises, for at least one sample, called current sample, of said at least one current image, associated with a position in the image: - construction of a characteristic vector from said at least one prediction characteristic map, as a function of said position of said at least one current sample, and: - application of said characteristic vector to the input of said prediction neural network to provide a vector representative of the prediction of said at least one current sample.

[0036] By "feature vector constructed from feature maps as a function of a position" is meant a vector consisting of one or more elements, or data, preferably discrete, the data being constructed from the feature maps at a position determined by that of the samples being processed in the images. This feature vector is the one that is applied, or presented, to the input of the prediction neural network. Such a vector can be constituted for example from a plurality of values ​​taken from each of the feature maps at the same abscissa and ordinate coordinates as those of the sample that one seeks to predict or reconstruct. Such a vector can also be constituted from a plurality of values ​​close to those of the sample, taken from each of the feature maps.In the case where we seek to reconstruct a movement field via the prediction neural network, we obtain, for example, at the output of the network a movement vector for this sample.

[0037] The coding method according to this mode constructs the prediction and / or correction parameters, from a sequence of input images, by training a prediction and / or correction neural network on characteristic vectors associated with a position of a sample to be coded in the current image or sub-sequence. These characteristic vectors are constructed from prediction and / or correction characteristic maps. The characteristic maps may be at the resolution of the input images, or at a lower resolution. A coding or decoding method according to this mode uses the parameters of the neural network and the characteristic maps transmitted to it to generate characteristic vectors similar (apart from coding errors) to those of the encoder, present them as input to the appropriate neural network, and obtain motion information as output.

[0038] According to another embodiment of the invention, such a coding or decoding method is characterized in that the processing step comprises a sub-step of applying said prediction feature maps and said reference images to the input of said at least one prediction neural network to produce at least one prediction image.

[0039] Advantageously, according to this mode, the generation of the prediction image combines the generation of the motion field and the compensation. Thus, a pooling of operations is ensured, ensuring reduced complexity. Preferably in this case, the reference images are in the latent domain, and represented in the form of feature maps.

[0040] According to a variant of this embodiment of the invention, the step of applying said prediction characteristic maps and said at least one reference image to the input of said at least one prediction network comprises, for at least one sample, called current sample, of said at least one current image, associated with a position in the image: - construction of a characteristic vector from said at least one prediction characteristic map and said at least one reference image, as a function of said position of said at least one current sample, and: - application of said vector to the input of said prediction neural network to provide a vector representative of the prediction of said at least one current sample.

[0041] In the case where we seek to reconstruct the prediction image via the prediction neural network, such an output vector can consist of a vector (eg (Y,U,V)) representative of the prediction of the current pixel.

[0042] According to one embodiment of the invention, the step of decoding said set of correction images further comprises the following sub-steps:- decoding a set of parameters representative of a correction neural network;- decoding a set of so-called correction characteristic maps;- applying said decoded correction characteristic maps to the input of said at least one correction neural network to produce at least one prediction image.

[0043] Advantageously, in this mode, a neural network is used to encode and decode the correction images. The neural network trained on the encoder is transmitted to the decoder, which can, from the correction feature maps and the correction network alone, reconstruct a correction image. The correction neural network can be trained jointly with the prediction neural network to provide a better compromise between the throughput and the distortion of the decoded images.

[0044] According to a variant of this embodiment of the invention, the step of processing said correction characteristic maps by said at least one correction neural network comprises, for at least one sample, called current sample, of said at least one current image, associated with a position in the image: - construction of a characteristic vector from said at least one correction characteristic map, as a function of said position of said at least one current sample, and: - application of said characteristic vector to the input of said correction neural network to provide a vector representative of the correction of said at least one current sample.

[0045] In the case where we seek to reconstruct the correction image via the correction neural network, such an output vector can be made up of a vector (eg (Y,U,V)) representative of the correction of the current pixel.

[0046] According to another embodiment of the invention, the correction neural network and / or the prediction neural network comprise an MLP and / or a convolutional network.

[0047] MLP stands for Multi-Layer Perceptron. Convolutional network stands for a network that has at least one convolutional layer. Advantageously, this type of network is very simple in nature and its parameters can be easily learned and transmitted from the encoder to the decoder.

[0048] According to embodiments of the coding or decoding method:

[0049] - the reconstruction step comprises an addition or subtraction applied to the correction and prediction images; advantageously according to this mode, the correction data are simply added to the prediction data, for example for each pixel of the image, to reconstruct the current image.

[0050] - the reconstruction step uses a convolutional neural network applied to the correction and prediction images; advantageously, in this mode, the correction data can be combined more precisely with the prediction data to reconstruct the current image

[0051] – motion information is a dense field of two-dimensional motion vectors;

[0052] - the compensation step involves a displacement of at least one sample of the reference image.

[0053] - The method comprises a step of transforming said set of prediction and / or correction feature maps to obtain a set of transformed feature maps at the resolution of the current image, the method being characterized in that said feature vector is constructed from said transformed feature maps. Advantageously according to this mode, the feature maps are divided into two sets, one of which is reserved for the extraction of feature vectors and the other for coding / decoding. It is thus possible to dissociate the two processes whose purpose is different: the maps of the group to be coded (resp. decoded) must be compressed as efficiently as possible, while the transformed maps must be able to facilitate the process of extracting and constructing the feature vector.

[0054] – According to a variant, at least one of the prediction and / or correction feature maps has a lower resolution than that of the sub-sequence to be coded (resp. decoded) and the transformation operation includes oversampling. Advantageously, according to this mode, the compression of the feature maps is more efficient since at least one of the feature maps to be coded (resp. decoded) has fewer values ​​than if it were at the resolution of the signal. For example, one of the feature maps may be at 1 / 2 resolution, that is to say it has half as many values ​​on the abscissa and ordinate as the input image has samples, i.e. in total 4 times fewer values ​​than a feature map at the resolution of the image. On the other hand, the transformed feature map, which corresponds to a transformation of this map, has a resolution identical to that of the image.The transformation therefore includes in this case at least one oversampling operation to obtain the same number of values ​​in this transformed map as the current image has samples.

[0055] – At least one of said prediction and / or correction characteristic maps has a resolution identical to that of the image to be coded (resp. decoded). Advantageously, according to this embodiment, at least one of the characteristic maps, at the same resolution as the input signal to be coded (resp. decoded), allows significant fidelity and respect for the details of the initial resolution of the image signal. The transformation according to one embodiment in this case retains the number of values ​​of the transformed characteristic map; it can be reduced to identity (no processing is carried out on the values ​​of the current map) or include a filtering operation, quantification, a Fourier transformation, etc.

[0056] - The construction of the characteristic vector includes a sub-step of extracting a value from a prediction and / or correction characteristic map at a position identical to that of the current sample in the current image to be coded or decoded. Advantageously, it is possible to extract a value from a characteristic map at the same position as the sample in the image, to constitute an element of the characteristic vector. This method is simple to implement. A simple extraction of the values ​​from the maps at the coordinates of the current sample (at the same abscissa and the same coordinate in the characteristic map) makes it possible to directly construct the characteristic vector.

[0057] - The construction of said characteristic vector comprises the following sub-steps: - extraction of a plurality of values ​​from said prediction and / or correction characteristic maps as a function of said position of said current sample; - processing of said extracted values ​​to obtain the characteristic vector.

[0058] Advantageously, according to this mode, the characteristic vector is extracted from the characteristic maps, which may be of any kind, and in particular at a resolution lower than that of the image to be coded (resp. decoded), before undergoing processing. Such processing may, for example, correspond to a quantification of the extracted data, or to scaling, or to filtering, etc. During coding, quantification is essential for the proper functioning of the system if the characteristic maps contain, for example, floating point or real values. It is necessary to quantify them before coding them and / or providing them as input to the prediction and / or correction neural network. During decoding, however, inverse quantification is not necessary, depending on the embodiments.

[0059] - The method comprises a step of constructing a group of additional feature maps and the motion and / or correction feature vector is further constructed from said feature maps. Advantageously, these additional maps, constructed identically at the coder and the decoder, are neither stored nor transmitted at the coder level, nor decoded at the decoder level. They thus make it possible to benefit from additional data to improve compression without degrading the bit rate. They may for example include coordinates, causal data available in other maps, data concerning other images already processed by the coder or decoder, etc.

[0060] - Said at least one reference image is indicated in the binary stream. Advantageously, according to this mode, a simple indicator coded in the stream allows the decoder to directly access the reference image(s) to be used for decoding the current image or sub-sequence, for example via an indicator indicating the absolute or relative position of the number of the reference image(s) in the sequence.

[0061] - Said at least one reference image is predetermined. Advantageously, according to this mode, the reference image(s) to be used for decoding the current image or sub-sequence are predetermined. For example, it is the previous one, or the first one in the sequence, etc.

[0062] Correlatively, the invention also relates to a coding device and a decoding device.

[0063] The characteristics and advantages of the coding or decoding method apply in the same way to the coding or decoding device according to the invention and vice versa.

[0064] The invention also relates to a computer program on a recording medium, this program being capable of being implemented in a computer or an encoding or decoding device in accordance with the invention. This program comprises instructions adapted to the implementation of the corresponding method. This program can use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0065] The invention also relates to a computer-readable information carrier or recording medium comprising computer program instructions mentioned above. The information or recording media may be any entity or device capable of storing the programs. For example, the media may comprise a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a floppy disk or a hard disk, a DNA sequence, or a flash memory. Furthermore, the information or recording media may be transmissible media such as an electrical or optical signal, which may be conveyed via an electrical or optical cable, by radio link, by wireless optical link or by other means.

[0066] The program according to the invention can in particular be downloaded from an Internet-type network.

[0067] Alternatively, each information or recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of a method according to the invention.

[0068] Other characteristics and advantages of the present invention will emerge from the description given below, with reference to the appended drawings which illustrate exemplary embodiments thereof which are not limiting in nature.

[0069] schematically represents a decoding device used in the context of the invention;

[0070] schematically represents a coding device used in the context of the invention;

[0071] illustrates a synthesis module used in two embodiments of the invention;

[0072] is a flowchart representing an example of a decoding method that can be implemented by the decoding device of the;

[0073] is a flowchart representing an example of a coding method that can be implemented by the coding device of the;

[0074] illustrates a correction image decoding method used in one embodiment of the invention.

[0075] illustrates a motion field decoding method used in one embodiment of the invention.

[0076] It schematically represents a DEC decoding device.

[0077] The decoding device DECv decodes an image or sub-sequence of index v of the sequence SV, proceeding image by image. At the end of the decoding, the V images of the sequence are decoded.

[0078] A current subsequence to be decoded V includes at least one current image to be decoded noted I V , each image comprising respectively a plurality of N samples. Subsequently, we consider, without loss of generality, that the current subsequence comprises a single current image. The DECv decoder dedicated to the image receives as input the data necessary to decode it:

[0079] - FPc encoded data organized into NFP prediction feature maps;

[0080] - WPc encoded parameters of a prediction neural network;

[0081] – one or more correction images IC; according to one embodiment, the correction images are decoded by entropy decoding their values; according to one embodiment, these correction images can be decoded using a correction neural network WCc applied to NFC correction feature maps FCc which are presented (applied) to its input. In this case, the associated coded data comprises the parameters of the correction neural network and the coded values ​​of the correction feature maps.

[0082] - optionally, the coded parameters of other neural networks, including:

[0083] - predictive decoding of prediction and / or correction feature maps;- oversampling and / or processing of prediction and / or correction feature maps;

[0084] - mixing of compensated images and correction images;

[0085] - post-processing of reconstructed data after mixing;

[0086] - etc.

[0087] The DEC decoding module vcomprises, for a current image, a neural network(s) decoding module NND, a synthesis module MSP comprising at least one prediction neural network, optionally a correction neural network MLPC, a FPD module for decoding prediction characteristic maps FPc, a FCD module for decoding IC correction images, optionally responsible for decoding FCc correction characteristic maps, an XTRM module, optionally XTRC, for processing and extracting data from the characteristic maps, a REF module for obtaining reference images or maps, a MIX module for combining ICO compensated images and IC correction data, optionally a module for generating additional characteristic maps, a post-processing module, not shown. According to one embodiment, it produces as output a current decoded image, denoted Idv, comprising a plurality of N decoded samples.According to another embodiment, not shown, it produces as output a sub-sequence of decoded images comprising at least the current image.

[0088] The FPD decoding module decompresses the compressed data, for example using an entropy decoder, from one or more prediction feature maps. According to one embodiment, it implements a decoding taking into account the neighborhood of a value to be decoded from the feature map. The neighborhood can be selected from a current motion map (including the one being decoded), or a map of an image distinct from the current image. The FPD module can comprise for this purpose a feature map prediction neural network. The prediction maps decoded by the FPD module, numbering NFP, are denoted FP and are called maps de mouvement de l’image courante, ou cartes de mouvement courantes.

[0089] The FCD decoding module decompresses the IC correction images. According to one embodiment, it is a conventional decoder, for example of the JPEG, or MPEG, etc. type, responsible for decoding a current IC correction image. According to another embodiment, it decodes quantized data, for example using an entropy decoder, of one or more correction characteristic maps. According to one embodiment, it implements a decoding taking into account the neighborhood of a value to be decoded from the characteristic map. The neighborhood can be selected from a current map (including the one being decoded), or a map of an image distinct from the current image. The FCD module can comprise for this purpose a characteristic map prediction neural network. The maps decoded by the FCD module, to the number of NFC, are denoted FC and are called maps de correction de l’image courante, ou cartes de correction courantes.

[0090] The NND module decodes neural networks. Any known neural network decoding technique can be used for this purpose, for example, entropy decoding, or the neural network coding standard proposed by MPEG-7 Part 17, also called NNR. The parameters of the prediction neural network (MLPP) of the synthesis module decoded by the NND module are denoted WP. The parameters of the correction neural network (MLPC) decoded by the NND module, if any, are denoted WC. Their encoded values ​​are denoted WPc and WCc, respectively.

[0091] The REF module constitutes a set of NFR reference images, denoted FREF, corresponding to one or more previously encoded and then decoded images of the sequence. They can be predetermined or their reference can be decoded in the stream in the form of an indicator. In one embodiment, a single reference image is selected from the set. In one embodiment, several reference feature maps, in the latent domain, represent the reference image.

[0092] The MSP module contains at least one so-called MLPP prediction neural network, defined by the K WPc parameters decoded by the NND module. The module takes as input the reference image(s), the decoded prediction feature maps, and the decoded parameters of the prediction neural network. It generates as output at least one motion-compensated image(s) denoted ICO. Two embodiments will be presented in support of the.

[0093] The XTRC module, if present, performs processing and extraction of values ​​in feature maps, in order to present them to the neural network. It will be presented in support of the.

[0094] The MIX module uses as input one or more predicted images ICO and the decoded correction information IC and performs a combination of the two pieces of information to obtain the reconstructed image. The combination can be, in its simplest form, an addition of the values ​​of a prediction image and a correction image. According to another embodiment, the MIX module uses a neural network.

[0095] The decoding device DEC can be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the modules mentioned above can then be realized by the cooperation of the processor and computer program instructions stored in the aforementioned memory and designed to carry out the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor

[0096] It schematically represents a device for coding digital image sequences ENC.

[0097] The ENCv coding device encodes a current image or subsequence v of the SV sequence, proceeding by subsequences of images. A subsequence can consist of a single image.

[0098] Subsequently, we consider, without loss of generality, that the subsequence comprises a single image Iv comprising respectively a plurality of N samples. The ENCv coder produces coded parameters as output. These parameters include:

[0099] - encoded prediction features organized into NFP FP feature maps;

[0100] - the encoded WPc parameters of the prediction neural network;

[0101] - the coded correction image data IC; according to one embodiment, the correction images are coded by entropy coding their values; according to one embodiment, these correction images can be coded using a correction neural network applied to correction feature maps. In this case, the associated coded data comprise the parameters of the WCc neural network and the coded values ​​of the FCc correction feature maps.

[0102] - optionally, the encoded parameters of other neural networks, as mentioned for the decoder.

[0103] The ENC coding module v includes, for a current image, an NNC module for coding neural network parameters, an FPC module for coding FPc prediction characteristic maps and optionally an FCC module for coding FCc correction characteristic images or maps, an RD-OPT evaluation module, an INIT / MAJ initialization and update module.

[0104] Furthermore, in a conventional manner, the coding module includes a decoder similar to the DECv decoder which was previously described, in order to reconstruct the current image I'v coded then decoded.

[0105] The FPC coding module compresses the quantized data, for example using an entropy coder, of one or more prediction feature maps. During the process of training, or building, the coding, i.e. until the performance evaluation step is satisfactory, the FPC module performs a coding simulation, followed by a decoding, intended for the RD-OPT evaluation module. Subsequently, it performs the actual coding of the values ​​of the FPc maps. In a known manner, the coding simulation can be identical to the actual coding, or perform an approximation thereof. The coding module quantizes, if necessary, the latent representation of the values ​​of the maps by using a quantizer to generate an ordered collection of quantized values. Then the coding module compresses the quantized data, for example using an entropy coder.As described in the decoder, according to one embodiment, it implements a coding taking into account the neighborhood of a value to be decoded from the feature map. The FPC module may comprise for this purpose a feature map prediction neural network, identical to that of the decoder. The motion maps coded by the FPC module, numbering NFP, are denoted FPc(maps. de prédiction de l’image courante).

[0106] The FCC coding module compresses the quantized data, for example using an entropy coder, of one or more correction images or maps. During the process of training, or building, the coding, i.e. until the performance evaluation step is satisfactory, the FCC module performs a coding simulation, followed by decoding, for the RD-OPT evaluation module. Subsequently, it performs the actual coding of the values ​​of the FCc images or maps. In a known manner, the coding simulation can be identical to the actual coding, or produce an approximation thereof. The coding module quantizes, if necessary, the latent representation of the values ​​of the maps by using a quantizer to generate an ordered collection of quantized values. Then the coding module compresses the quantized data, for example using an entropy coder.As described in the decoder, according to one embodiment, it implements a coding taking into account the neighborhood of a value to be decoded from the feature map. The FCC module may comprise for this purpose a feature map prediction neural network, identical to that of the decoder. The correction maps coded by the FCC module, if they exist, in the number of NFCs, are noted FCc(maps. de correction de l’image courante).

[0107] The INIT / MAJ module is responsible for initializing the values ​​of the feature maps and the parameters of the neural network(s), noted respectively FPo, WPo, FCo, WCo.

[0108] The RD-OPT module performs an evaluation and minimization of a coding performance. The evaluation function is for example of the rate-distortion type. The minimization can be carried out by gradient descent, or any other optimization method within the reach of a person skilled in the art.

[0109] The INIT / MAJ module updates the values ​​to be encoded in the current image (FPo, WPo, FCo, WCo), based on the results of the performance function. Once the current feature maps and the neural network(s) are stabilized, they can be encoded.

[0110] The ENC coding device may be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the above-mentioned modules may then be realized by the cooperation of the processor and computer program instructions stored in the above-mentioned memory and designed to perform the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0111] Illustrates an MSP synthesis module used in two embodiments of the invention (a) and (b). The module takes as input the reference image(s), the decoded prediction feature maps, and the decoded parameters of the prediction neural network. It generates as output at least one motion-compensated image(s) denoted ICO.

[0112] The description of the XTRM module below applies respectively to the XTRC module if present.

[0113] According to both embodiments (a) and (b), an XTRM module is used to format the data to be presented to the prediction neural network.

[0114] In embodiments, XTRM (resp. XTRC) uses an optional inverse quantization module to perform inverse quantization of the data extracted from the feature maps. This inverse quantization is mandatory if quantization has been performed at the encoder.

[0115] In embodiments, XTRM (resp. XTRC) performs a transformation of the values ​​of at least one of the feature maps, for example oversampling, interpolation, filtering, etc. At the end of the transformation, a transformed feature map is of the same resolution as the current image or subsequence to be decoded. Advantageously according to this mode, the feature maps which are decoded can be of lower resolution than that of the images to be decoded, while the transformed maps, which are used for the construction of the feature vectors, are at the same resolution as the image sequence, which facilitates the extraction of the values.

[0116] In embodiments, a prediction sample (resp. a correction sample) is generated for each sample of the input image. The extraction of the values ​​is done according to the position of a sample being processed. For example, if one seeks to decode the sample P n at the coordinates (x n , y n ) of a current image, the XTRM (resp. XTRC) module performs an extraction of values ​​in the prediction (resp. correction) maps at positions imposed by the coordinates (x n , y n ). In one embodiment, the extracted values ​​constitute the characteristic vector Z n . Z n is a J-tuple, that is, it has J elements, or data z i . The vector Z n of index n refers to the characteristic vector of the pixel P n . according to one embodiment, J= NFP (resp. NFC).

[0117] In embodiments, XTRM (resp. XTRC) does not perform any transformation, in this case the values ​​which will be used to construct the characteristic vector are extracted from the untransformed prediction (resp. correction) characteristic maps. XTRM (resp. XTRC) can perform a processing of the extracted values, to generate the vector Zd n . XTRM (resp. XTRC) can perform an inverse quantization of the data extracted from the set of feature maps. The processing can include other operations, such as filtering, scaling, etc. In particular, if the feature maps are at lower resolutions than the images in the sequence, the XTRM (resp. XTRC) module can take into account the coordinates of the values ​​in the lower resolution maps.

[0118] The MSP module contains at least one neural network called MLPP prediction, defined by K WP parameters, capable of processing a vector Zn , or J-Uplet, as input, to generate as output a vector representative of the sample P n being processed. The prediction neural network contains, according to one embodiment, an MLP, or Multi Layer Perceptron, consisting of an input layer adapted to the input format (the J-tuple), optionally one or more hidden layers, and an output layer adapted to the output format of the output vector, generally a vector comprising NCP (number of components of the prediction output vector) elements, or values, for each sample. It may also comprise a convolutional neural network. The MLPP prediction neural network has the same structure as the encoder and the decoder, and its parameters are either identical if the coding of its WP parameters is lossless, or different if the coding is carried out with losses. When all the samples P nof the image v have been processed, we have predicted information comprising as many decoded output vectors as there are samples in the image (N samples, N output vectors).

[0119] If the correction neural network is present, its structure can be identical to that of the motion synthesis module and when all the samples P n of image v have been processed, we have correction information comprising NCC decoded correction vectors, each comprising for example a triplet of correction values ​​for the components (R,G,B) or (Y,U,V) of the image. The correction information can for example correspond to a correction image IC of the same size as image Iv.

[0120] According to the first mode, (a), represented on the left of the figure:

[0121] - MLPP performs motion synthesis from the prediction maps; the output vector is motion information MV, for example the pair (mvx, mvy) of the motion displacement of each pixel Pn being processed

[0122] - the MSP module further comprises a WARP motion compensation module. It performs motion compensation on one or more reference images using the generated motion field, an operation well known to a person skilled in the art. The images resulting from the compensation are denoted ICO. Warping may be a type of geometric transformation allowing the position of samples in an image to be modified. For example, each pixel of the reference image may be moved according to a motion vector (mvx, mvy) to obtain a predicted sample in an ICO image.

[0123] According to the second mode represented on the right, noted (b):

[0124] - MLPP performs a prediction image synthesis from the prediction maps and one or more reference images or maps; the output vector is a predicted image sample, for example a pixel (Y,U,V). The images resulting from the prediction are denoted ICO. MLPP performs the operations of motion field generation and dewarping in a combined manner.

[0125] This is a flowchart representing an example of a decoding method that can be implemented by the decoding device of the.

[0126] The decoding described concerns a current subsequence comprising at least one current image Iv of the sequence to be decoded.

[0127] During a step E30, the streams A1, A2, B1 and optionally B2 are extracted from the encoded stream. They contain respectively the coded representations of the current prediction maps FPc, the parameters WPc of the prediction neural network, the coded data FCc of the correction images or maps, the parameters WCc du réseau de correction s'il existe.

[0128] During a step E31, the correction IC image(s) are decoded.

[0129] According to one embodiment, they are decoded by a conventional technique within the reach of those skilled in the art, for example a prediction followed by decoding by a contextual entropic coder then dequantization.

[0130] According to one embodiment, they are synthesized by a correction neural network (MLPC) which takes as input a set of FCc correction feature maps. In this case, the embodiments and variants presented below for the prediction feature maps and the prediction neural network apply mutatis mutandis.

[0131] During a step E32, the NFP prediction feature maps FP are generated by decoding the FPc values. For this decoding, a technique is used, for example, for predicting a feature map value by its neighborhood followed by entropic decoding. In one embodiment, the FP maps are decoded in order (FP1, FP2,… FP4), and the values ​​of each map in a predefined order, for example lexicographic.

[0132] According to embodiments:

[0133] - The FP maps have the same resolution as the Iv signal to be reconstructed, i.e. they contain as many values ​​as there are samples (N) in the Iv image.

[0134] - The FP maps have a resolution lower than or equal to that of the Iv signal to be reconstructed and therefore include, for at least one of them, a number of values ​​to be decoded lower than N. According to a variant, the first map has the resolution of the images and each subsequent map has a resolution half that of the previous one.

[0135] - Several FP cards are of the same resolution, lower than the signal resolution.

[0136] During a step E33, according to embodiments, the M FP cards are dequantized, transformed, and data are extracted therefrom by an XTRM module as presented in support of the.

[0137] According to one embodiment, at least one FP map has a lower resolution than the decoded images and the transformation operation includes oversampling so that the transformed map has the same number of samples as the images in the sequence. Oversampling consists of adding values ​​to the maps to achieve the resolution of the images in the input sequence. It can be simple (by nearest neighbor replication) or include interpolation (linear, polynomial, by filtering, etc.). It can be performed by an oversampling neural network.

[0138] According to one embodiment, the XTRM module performs an extraction of values ​​in the transformed characteristic maps, for a sample P n to decode, based on its coordinates in the signal. The extracted values ​​constitute the vector Z n . Z nis a J-tuple, that is, it has J elements, or data z i . The vector Z n of index n refers to the characteristic vector of the pixel P n .

[0139] In one embodiment, Z n is a J-tuple (z1, z2,…, z J ), consisting of the values ​​of the maps (possibly transformed) located at the coordinates (x n , y n ) of the current pixel P n . In one embodiment, Z n has as many values ​​as there are input cards. In one embodiment, Z n is a J-tuple constructed from values ​​taken from the maps at coordinates that may be different across maps. For example, if the maps are at different resolutions, the coordinates are adapted (by scaling) to match the resolution of each map. In one embodiment, Z nis a J-tuple constructed from values ​​taken from the maps by applying processing to one or more values ​​of the maps, for example filtering values ​​close to the target value in a map. For example, for a current sample P n in an FP map which is at the same resolution as the image signal to be decoded, we can extract the values ​​located at the coordinates (x n , y n ), (x n -1, y n ), (x n , y n -1) and (x n -1, y n -1) and apply a treatment to these values ​​(filtering, averaging, interpolation, application of any function, preferably monotonic, etc.) to obtain the final value (z i ) of element i of vector Z n relative to this map. According to another example, in a map which is at half the resolution of the signal, we can consider the values ​​located at the coordinates (x n / 2, y n / 2), (xn / 2-1, y n / 2), (x n / 2, y n / 2-1) and (x n / 2-1, y n / 2-1) and apply processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (z i ) of element i of vector Z n relating to this FP card i . The processing may include quantification of the extracted values ​​or of the Z vector. n constituted, if necessary. The samples to be decoded are for example processed in sequential order, from n=1 to n=N.

[0140] In a step E34, the WP parameters of the MLPP prediction neural network are generated by decoding the WPc values ​​of the stream A2. For this purpose, any known neural network decoding technique corresponding to the coding technique used by the encoder can be used, for example the neural network coding standard proposed by the MPEG-7 part 17, NNR standard.

[0141] During a step E35, FREF reference maps are obtained by the REF module, from among the already decoded images associated with the sequence. The REF module constitutes a set of NFRMAX reference images. Then a number NFR of reference images is selected from this set. According to one embodiment, a single image is selected. According to one embodiment, a reference image is represented by feature maps in the latent domain. For example, the reference image considered for the current image is the previous image of the sequence, or the first, or the last, or an Intra image, etc. The reference image can thus be predetermined. Its (their) reference can alternatively be coded in the stream in the form of an indicator.

[0142] In a step E36, the values ​​extracted from the feature maps are processed by the MLPP prediction neural network to generate prediction values. According to an embodiment as shown in case (b), the selected reference images or maps are also presented as input to the neural network. According to one embodiment, a vector Z n is processed by an MLPP prediction neural network to generate an output vector containing NCP values ​​for sample P n to decode.

[0143] According to one embodiment, as shown in (a), the output vector is a motion vector MVn .In an example, NCP is equal to 2 and the two-dimensional output vector (mvx, mvy) contains the values ​​of the motion vector horizontally and vertically. When all the signal samples have been processed, a dense motion field (i.e., one vector per pixel) is available. For example, this field is available as two images, the first image containing the horizontal (x) coordinate of the motion vector, and the second image containing the vertical (y) coordinate of the motion vector.

[0144] According to this embodiment, during a step E37, the reference image (or images) are compensated, that is to say that the previously determined motion field is applied to them by the WARP module to produce one or more ICO compensated images. According to one embodiment, the compensation may be a geometric transformation making it possible to modify the position of the samples in an image. For example, the pixels of the reference image may be moved according to the associated motion vector (mvx, mvy) to obtain the predicted samples in the ICO image. The result may be continuously interpolated, spatially sampled at the sampling rate of the image to be reconstructed. Other more complex transformations, operated on sets of pixels, may be used (application of a polynomial, homographic, affine model, etc.) depending on the number of parameters delivered by the prediction neural network.It is also possible to compensate several reference images to obtain a current image. In this case, the MV motion field has four values, or components, for a pixel (2 for each image). The compensated images are then merged (for example by averaging or linear combination, or weighted prediction, etc.) so as to produce at least one compensated ICO image.

[0145] According to another embodiment, as shown in (b), the reference images are also applied, or presented, to the input of the MLPP network. The reference images and the prediction feature maps may be combined before entering the MLPP network. The output vector is a predicted image sample, e.g., a pixel (Y,U,V). When all the signal samples have been processed, a prediction image is available, e.g., in the form of three components (YUV). In this embodiment, the WARP module is not used.

[0146] In a step E38, a combination of the compensated ICO images and the IC correction images is performed by the MIX module. The combination can be, in its simplest form, an addition of the values ​​of the prediction image and those of the correction image. According to another embodiment, the MIX module uses a neural network which takes the ICO and IC images as input and produces, for example using a series of convolutional layers, the image Iv.

[0147] This is a flowchart representing an example of a coding process that can be implemented by the coding device of the

[0148] It is recalled that during coding, the entire system is optimized during a learning phase which precedes the actual coding phase: the values ​​of the FPi and optionally FCi maps as well as the parameters of the MLPP and optionally MLPC networks are learned (in the sense of machine learning, for example by gradient descent) in order to optimize a performance measure, for example a rate-distortion type cost function noted D+ LR, where D is the quadratic error of the reconstructed image Iv with respect to the original image, R is the sum of the coding rates of the feature maps and optionally of the neural networks, and L a parameter set by the user for the RD-OPT module. According to another example, D is calculated from a perceptual function such as the SSIM (for Structural SIMilarity), or MSSSIM (for Multi-Scale Structural SIMilarity).

[0149] During a step E20, an input sequence SV to be coded, comprising at least one current image Iv comprising a plurality of N samples P n , is provided as input to the process.

[0150] In a step E21, the prediction feature maps and the correction images or maps, as well as the parameters of the neural networks, are initialized. Subsequently, these data are optimized during the construction phase. According to one embodiment, the feature maps are initialized by predefined constant values. According to another embodiment, the feature maps are initialized by a set of random real values. According to one embodiment, the parameters of the neural network are initialized by predefined values ​​known to give a satisfactory result (for example, following training on a corpus of image sequences). According to another embodiment, the parameters of the neural network are initialized by a set of random values.The prediction feature maps, images or correction maps and parameters of the neural networks are subsequently updated, or refined, during a step E22, by the encoder's MAJ update module during its learning.

[0151] In a step E23, a coding simulation is performed for all the coding parameters: the prediction feature maps are encoded by the FPC module, the correction feature images or maps by the FDC module, the neural network(s) by the NNC module. The simulation can be identical to the actual coding but it can also be different (for example, simplified).

[0152] During a step E24, a decoding simulation is carried out. To measure the distortion D, it is in fact necessary to simulate the coding then the decoding of at least part of the image sequence, to obtain at least one pixel P' nof at least one image I'v resulting from a simulation of coding then decoding of the samples of index n, then measuring the difference between these coded then decoded pixels and the pixels of the corresponding uncoded sequence. For this purpose, steps E31 to E38 of the decoding method are carried out in order to reconstruct the decoded current image or sub-sequence I'v.

[0153] In step E25, the performance measurement is evaluated.

[0154] For this purpose, the coding simulation rates associated with the feature maps (simulation of the A1 stream by coding the feature maps and of the B1 stream by coding the correction image or its feature maps). According to one embodiment, the coding simulation rates associated with the parameters of the neural network (simulation of the A2 stream by coding the WP parameters and optionally of the B2 stream by coding the WC parameters) are also measured. According to one embodiment, the coding of the parameters of the neural networks is not simulated, because their influence is less important than that of the feature maps.

[0155] As long as the cost function has not reached its minimum, the performance measurement is not satisfactory, and the process is repeated from step E22. Alternatively, the process can be interrupted after a predefined time or a predefined number of iterations, so as to control its complexity or duration.

[0156] In a step E26, if the cost function has reached its minimum or the time allowed for encoding is deemed sufficient, the training stops. If an encoded version corresponding to the last simulation of the parameters of the neural network and the feature maps is available, the streams A1, A2, B1 and B2 can be formed from it. According to another embodiment, the actual encoding of the updated parameters of the neural network(s) and the values ​​of the feature maps is carried out at this step to produce the encoded parameters.

[0157] The streams A1, A2, B1 and B2 may be concatenated to produce a final stream. According to one embodiment, the stream A2 / B2 of encoded parameters of the neural network is stored or transmitted before the stream A1 / B1, so that it can be decoded before.

[0158] Illustratively depicts a correction image decoding method used in one embodiment of the invention.

[0159] In this embodiment, the correction images are synthesized (at encoding and decoding) by an MLPC neural network applied to correction FC feature maps processed by an XTRC module.

[0160] During decoding, the FC maps are obtained as described before by a classical decoding (for example an entropy decoding) from the B1 stream, by the FCD module.

[0161] In this embodiment, there are 4 current decoded cards FCi. In a preferred embodiment, there are 7. In this embodiment, the first card FC1 has the same resolution as the image Iv to be decoded, and therefore comprises WxH variables, where W represents the width of the image in pixels, and H its height. The second card FC2 has half the resolution (in each dimension) of the card FC1. Each additional card has half the resolution of the previous card. This structure makes it possible to reduce the number of variables in the characteristic cards, which facilitates decoding while minimizing the coding cost.

[0162] The FCi cards are then processed by the XTR module, which can include all of the following steps:

[0163] – a step noted DQi (DQ1... DQ4) of dequantization of each card, the dequantization consisting for example of multiplication by a predefined or transmitted quantization step;

[0164] – a step denoted SEi (SE1... SE4) of oversampling capable of putting the FCi card at the same resolution as the signal to be coded; according to the example shown, the FC2 card is oversampled by a factor of 2 by the SE2 module in each dimension, according to any oversampling method within the reach of those skilled in the art. The FC3 card is oversampled by a factor of 4 in each dimension, etc.

[0165] – a step noted Ti (T1... T4) of improvement processing, this step being able to include convolutional filtering, a linear function, or even a neural network whose W1 parameters have been decoded in a classical manner (classical decoding or NNR) from the B2 flow.

[0166] The steps described can be "mixed", in the sense that the oversampling can be based on a neural network. They can also be joint, that is, applied to a set of maps and not to a single map as shown in the figure.

[0167] According to this embodiment, these steps produce the FCSi maps (FCS1... FCS4), which are at the same resolution as the signal to be coded, and therefore comprise WxH values, where W represents the width of the image in pixels, and H its height. The transformed maps are processed by an MLPC neural network, typically a Multi-Layer Perceptron, which associates with a vector Z ninput consisting of the values ​​of the FCSi layers at the coordinates (x,y) of a pixel being processed an output vector (R,G,B) at the coordinates (x,y) in a series of layers DR,DG,DB. These layers are for example the same number as the number of components of the output image (in figure, 3). In this embodiment, the vector Z n is a 4-tuple (z 1, z 2, z3, z4) made up of the 4 values ​​of the FCS cards isituées aux coordonnées (x, y) du pixel courant Pn. Le triplet (R, G, B) est inséré dans chaque image décodée Iv de la sous-séquence v aux coordonnées (x, y) dans les composantes couleur (DR,DG,DB).

[0168] The WC parameters of the MLPC network were decoded from the B2 stream by conventional or NNR decoding.

[0169] Finally, a last PPC post-processing network, whose W2 parameters have also been decoded from the stream, can process the DR, DG, DB layers to produce the decoded image IC. The PPC network can include a convolutional network, an MLP or any other neural network or conventional processing.

[0170] According to a variant not shown, additional correction feature maps are introduced. In such a variant, the vector Z n is of dimension greater than 4. Such an additional map typically contains data that can assist the MLPC network in the task of image reconstruction. Thus, the added maps can be, but are not limited to, one or more of:

[0171] – a map showing at each point the abscissa and / or the ordinate of this point;

[0172] – a map with positional coding at each point;

[0173] – a map representing an image distinct from the images being processed, capable of providing information on the images to be coded, for example a previously processed image or sequence of images;

[0174] – a map containing data representing the time difference between the frames of the video being encoded. For example, if the first and last frames of the video are 8 frames apart, all samples in the map contain the value 8;

[0175] – a map representing a map of characteristics of an image distinct from the images being processed, capable of providing information on the images to be coded, for example a previously processed map;

[0176] – a card containing the value of an already decoded sample from the same card, for example the previous sample in the decoding order.

[0177] Illustratively represents a motion field decoding method used in an embodiment of the invention according to case (a).

[0178] In this embodiment, the motion field is synthesized (at encoding and decoding) by an MLPP prediction neural network applied to FP prediction feature maps processed by an XTRM module.

[0179] During decoding, the FP maps are obtained as described before by a classical decoding (for example an entropy decoding) from the A1 stream, by the FPD module.

[0180] In this embodiment, there are 4 current decoded cards FPi. In a preferred embodiment, there are 7. In this embodiment, the first card FP1 has the same resolution as the image Iv to be decoded, and therefore has WxH variables, where W represents the width of the image in pixels, and H its height. The second card FP2 has half the resolution (in each dimension) of the card FP1. Each additional card has half the resolution of the previous card. This structure makes it possible to reduce the number of variables in the feature cards, which facilitates decoding while minimizing the coding cost.

[0181] The FPi cards are then processed by the XTRM module, which can include all of the following steps:

[0182] – a step noted DQi (DQ1... DQ4) of dequantization of each card, the dequantization consisting for example of multiplication by a predefined or transmitted quantization step;

[0183] – a step denoted SEi (SE1... SE4) of oversampling capable of putting the FPi card at the same resolution as the signal to be coded; according to the example shown, the FP2 card is oversampled by a factor of 2 by the SE2 module in each dimension, according to any oversampling method within the reach of those skilled in the art. The FP3 card is oversampled by a factor of 4 in each dimension, etc.

[0184] – a step noted Ti (T1... T4) of improvement processing, this step being able to include convolutional filtering, a linear function, or even a neural network whose W3 parameters have been decoded in a classical manner (classical decoding or NNR) from the A2 flow.

[0185] The steps described can be nested, in the sense that the oversampling can be based on a neural network. They can also be joint, that is, applied to a set of maps and not to a single map as shown in the figure.

[0186] According to one embodiment, these steps produce the FPSi maps (FPS1... FPS4), which are at the same resolution as the signal to be coded, and therefore comprise WxH values, where W represents the width of the image in pixels, and H its height. The transformed maps are processed by an MLPP prediction neural network, typically a Multi-Layer Perceptron, which associates with a vector Z ninput consisting of the values ​​of the FPSi layers at the coordinates (x,y) of a pixel being processed an output vector (mvx, mvy) at the coordinates (x,y) in a series of motion layers. These layers are for example the same number as the number of components of the output motion vector (in figure 7: MVX comprises the horizontal component of the motion vectors, and MVY the vertical component). In this embodiment, the vector Z n is a 4-tuple (z 1, z 2, z3, z4) made up of the 4 values ​​of the FPS cards isituées aux coordonnées (x, y) du pixel courant Pn. Le couple (mvx, mvy) est inséré dans chaque couche de mouvement MVX, MVY de l'image ou la sous-séquence courante v aux coordonnées (x, y). La concaténation de tous ces vecteurs constitue un champ de mouvement dense.

[0187] According to other embodiments: - the output motion vector is representative of a group of pixels (for example a block); - different parameters are obtained at the output: affine motion parameters, etc. - the current sub-sequence corresponds to several images and the motion is synthesized for the sub-sequence; - etc.

[0188] The WP parameters of the MLPP prediction neural network were decoded from the A2 stream by conventional or NNR decoding. The prediction neural network used for encoding, and the MLPP prediction neural network used for decoding, are defined by an identical structure, comprising for example a plurality of layers of artificial neurons, and by a set of weights and activation functions associated respectively with the artificial neurons of the network concerned. Finally, a last PPM post-processing network, whose W4 parameters were also decoded from the stream, can process the MVX, MVY layers to produce the final motion information MV for the current image. The PPM network can be a convolutional network, an MLP or any other neural network or conventional processing. The PPM network can also integrate the WARP compensation module, in which case it takes as input not only the motion information but also the reference images or maps.

[0189] According to a variant not shown, additional prediction feature maps are introduced. In such a variant, the vector Z n is of dimension greater than 4.

[0190] Such an additional map typically includes data that can assist the MLPP network in the task of motion reconstruction. Thus, the added maps may be, but are not limited to, one or more of:

[0191] – a map showing at each point the abscissa and / or the ordinate of this point;

[0192] – a map with positional coding at each point;

[0193] – a map representing an image distinct from the images being processed, capable of providing information on the movement of the images to be coded, for example an image, a sequence of images or layers of motion previously processed;

[0194] – a map containing data representing the time difference between the frames of the video being encoded. For example, if the first and last frames of the video are 8 frames apart, all samples in the map contain the value 8;

[0195] – a map representing a map of characteristics of an image distinct from the images being processed, capable of providing information on the images to be coded, for example a previously processed motion map;

[0196] – a map containing the value of a motion vector already decoded from the same map, for example that of the previous sample in the decoding order.

[0197] – a map including the quantification precision of characteristic values.

[0198] etc.

Claims

Method for coding a sequence of images (SV) comprising the following steps for at least one current image (Iv) of the sequence:- initialization (E21) of at least one correction image (IC);- initialization (E21) of a set of parameters (WP) representative of a prediction neural network (MLPP);- initialization (E21) of a set of so-called prediction characteristic maps (FP);- obtaining (E24) a set of reference images (FREF), comprising at least one coded and then decoded reference image;- processing (E24) of said prediction characteristic maps (FP) and of said reference images by a synthesis module comprising said at least one prediction neural network to the input of which said prediction characteristic maps are applied, to produce at least one prediction image (ICO);- reconstruction (E24) of the current image (Iv) from said at least one prediction image (ICO) and said at least one correction image (IC);- updating (E22, E25) of at least one value of said at least one feature map and / or of at least one parameter of said network, as a function of a coding performance measurement.- coding (E26) of a binary stream comprising:- coding of said at least one correction image (IC);- coding of the set of parameters (WPc) representative of the prediction neural network (MLPP);- coding of the set of prediction feature maps (FPc); Method for decoding a sequence of images (SV) from a binary stream, comprising the following steps for at least one current image (Iv) of the sequence:- decoding (E31) of a set of so-called correction images (IC) comprising at least one correction image representative of correction data of said at least one current image (Iv);- decoding (E32) of a set of so-called prediction feature maps (FPc) comprising at least one feature map representative of a movement of said at least one current image;- decoding (E34) of a set of parameters (WPc) representative of a prediction neural network (MLPP);- obtaining (E35) of a set of reference images (FREF), comprising at least one decoded reference image;- processing (E36) of said prediction characteristic maps (FP) and said reference images (FREF) by a synthesis module (MPS) comprising said at least one prediction neural network to the input of which said prediction characteristic maps are applied, to produce at least one prediction image (ICO); - reconstruction (E38) of the current image (Iv) from said at least one prediction image (ICO) and said at least one correction image (IC).; Coding or decoding method according to one of the preceding claims, characterized in that the processing step comprises the following sub-steps: - application (E33, E34) of said prediction characteristic maps (FP) to the input of said at least one prediction neural network to produce motion information (MV); - compensation (E37) of said at least one reference image (FREF) using said motion information to produce said at least one prediction image (ICO). Coding or decoding method according to the preceding claim, characterized in that the step of applying said prediction characteristic maps to the input of said at least one prediction network comprises, for at least one sample, called current sample (P n ), of said at least one current image (Iv), associated with a position (x n , y n ) in the image: - construction (E33) of a characteristic vector (Zn ) from said at least one prediction feature map, as a function of said position (x n , y n ) of said at least one current sample, and:- application (E34) of said vector (Z n ) at the input of said prediction neural network (MLPP) to provide a vector representative of the prediction of said at least one current sample (P n ). Coding or decoding method according to one of claims 1 or 2, characterized in that the processing step comprises a step of applying (E33, E34, E35) said prediction characteristic maps (FP) and said at least one reference image (FREF) to the input of said at least one prediction neural network to produce said at least one prediction image (ICO). Coding or decoding method according to the preceding claim, characterized in that the step of applying said prediction characteristic maps and said at least one reference image to the input of said at least one prediction network comprises, for at least one sample, called the current sample (P n ), of said at least one current image (Iv), associated with a position (x n , y n ) in the image: - construction (E33) of a characteristic vector (Z n ) from said at least one prediction feature map and said at least one reference image as a function of said position (x n , y n ) of said at least one current sample, and:- application (E34) of said vector (Z n ) at the input of said prediction neural network (MLPP) to provide a vector representative of the prediction of said at least one current sample (P n ). Decoding method according to one of claims 2 to 6, characterized in that the step of decoding said set of correction images further comprises the following sub-steps:- decoding (E31) of a set of parameters (WCc) representative of a correction neural network (MLPC);- decoding (E31) of a set of so-called correction characteristic maps (FCc);- application (E31) of said decoded correction characteristic maps (FC) to the input of said at least one correction neural network to produce at least one correction image (IC). Decoding method according to the preceding claim, characterized in that the correction neural network (MLPC) comprises an MLP and / or a convolutional network. Coding or decoding method according to one of the preceding claims, characterized in that the prediction neural network (MLPP) comprises an MLP and / or a convolutional network. Coding or decoding method according to one of the preceding claims, characterized in that the reconstruction step (E38) comprises an addition or subtraction applied to the correction and prediction images. Coding or decoding method according to one of the preceding claims, characterized in that the reconstruction step (E38) uses a convolutional neural network applied to the correction and prediction images. Coding or decoding method according to claim 3 or 4, characterized in that said motion information is a dense field of two-dimensional motion vectors. Device for decoding a sequence of images (SV) from a binary stream, said device being configured to implement, for at least one current image (Iv) of the sequence:- decoding (FCD) of a set of so-called correction images (IC) comprising at least one correction image representative of correction data of said at least one current image (Iv);- decoding (FPD) of a set of so-called prediction characteristic maps (FPc) comprising at least one characteristic map representative of a movement of said at least one current image;- decoding (NND) of a set of parameters (WPc) representative of a prediction neural network (MLPP);- obtaining (REF) of a set of reference images (FREF), comprising at least one decoded reference image;- processing of said prediction characteristic maps and said reference images by a synthesis module comprising said at least one prediction neural network to the input of which said prediction characteristic maps are applied, to produce at least one prediction image; - reconstruction (MIX) of the current image (Iv) from said at least one prediction image (ICO) and said at least one correction image (IC).; Device for coding a sequence of images (SV), said device being configured to implement for at least one current image (Iv) of the sequence: - initialization of a set of so-called correction images (IC); - initialization of a set of parameters (WPc) representative of a prediction neural network (MLPP); - initialization of a set of so-called prediction characteristic maps (FPc); - obtaining a set of reference images (FREF), comprising at least one coded then decoded reference image; - processing of said prediction characteristic maps and said reference images by a synthesis module comprising said at least one prediction neural network to the input of which said prediction characteristic maps are applied to produce at least one prediction image;- reconstruction of the current image (Iv) from said at least one prediction image (ICO) and said at least one correction image (IC);- updating of at least one value of said at least one feature map and / or of at least one parameter of said network, as a function of a coding performance measurement.- coding of a binary stream comprising:- coding (FCC) of said at least one correction image (IC);- coding (NNC) of the set of parameters (WPc) representative of the prediction neural network (MLPP);- coding (FPC) of the set of prediction feature maps (FPc).; Computer program comprising instructions for carrying out the steps of an encoding or decoding method according to claim 1 or 2 when said program is executed by a computer.

Images