Method and device for encoding and decoding images

Abstract

The invention relates to a method and device for encoding and decoding a signal of a plurality of samples (Pdn) comprising motion information. The decoding method comprises the following steps: - decoding (E31, FMD) a first group of feature maps (FMdi); - decoding (E35, NND) a set of parameters (Wdk) representative of a neural network (MLP'); - for at least one sample, called the current sample (Pdn), of the signal to be decoded, associated with a position (xn, yn) in the signal to be decoded: - constructing (E33, E34, XTR', TT') a feature vector (Zd n ) from the feature maps of said first group (FMdi), and: - processing (E36, MLP') said feature vector (Zdn) using an artificial neural network (MLP') defined by the decoded parameters (Wdk) so as to provide a vector representative of a decoded value of the current sample (Pdn).

Description

Method and device for coding and decoding images.

[0001] The invention relates to the general field of coding single or multidimensional signals. It relates more particularly to the compression of digital images and video.

[0002] Digital images are generally subject to source coding aimed at compressing them in order to limit the resources required for their transmission and / or storage. There are many coding standards, such as, for still images, the JPEG family standards, and for animated images, or video, the ITU / MPEG standards (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) as well as their extensions (MVC, SVC, 3D-HEVC, etc.)

[0003] Encoding an image is generally performed by dividing the image into several rectangular blocks, and encoding these blocks of pixels according to a given processing sequence. In existing video compression techniques, the processing of a block typically includes a prediction of the pixels of the block performed using previously coded and then decoded pixels present in the image being encoded, in which case we speak of "Intra prediction", or previously coded images, in which case we speak of "Inter prediction". This exploitation of spatial and / or temporal redundancies makes it possible to avoid transmitting or storing the value of the pixels of each block of pixels, by representing at least some of the blocks by a residual representing a difference between the prediction values ​​of the pixels of the block and the actual values ​​of the pixels of the predicted block.

[0004] Inter prediction is usually done using a motion field. In conventional schemes, this motion field takes the form of a motion vector estimated per block of the image to be coded, and transmitted to the decoder.

[0005] In addition to these classic approaches proposed by compression standards (JPEG, MPEG, ITU), approaches based on artificial intelligence, and in particular neural ones, tend to develop.

[0006] Some of these neural approaches can be seen as a simple extension of the competition notion of the aforementioned compression techniques, such as the prediction and transformation mode competition in video coding.

[0007] Other approaches use the concept of "autoencoder." Autoencoders are learning algorithms based on artificial neural networks, which allow the construction of a new representation of a dataset. The architecture of an autoencoder consists of two parts: the encoder and the decoder. The encoder consists of a set of layers of neurons, which process the data in order to construct new representations called "encoded," also called "latent representations." In turn, the layers of neurons in the decoder receive these representations and filter them in order to try to reconstruct the initial data. The differences between the reconstructed data and the initial data make it possible to measure the error made by the autoencoder. Training consists of modifying the parameters of the autoencoder in order to reduce the reconstruction error measured on the different samples of the dataset.The performance of such autoencoder-based systems comes at the cost of a considerable increase in memory footprint and complexity compared to conventional approaches as proposed by compression standards. They can have millions of parameters and may require up to a million MACs (multiply-accumulate) to decode a pixel. This makes such decoders significantly more complex than conventional decoders, which could hamper the adoption of learning-based compression.

[0008] More recently, a simple image encoding technique based on a neural network was described in the article "COmpression with Implicit Neural representations" by Emilien Dupont et al. (arXiv:2103.03123). The proposed encoding technique consists of fitting a neural network to this image, quantifying the network weights and transmitting them. At the time of decoding, the neural network represents the image on its own. It is evaluated at each pixel position to reconstruct the image. The R, G, B color pixels for each of the image coordinates (i,j) are obtained by a function F, modeled by the neural network: R,G,B=f(i,j). However, such a technique remains inefficient in terms of compression and only applies to still images.

[0009] There are methods to estimate the motion field using a neural network. For example, the article "AnyFlow: Arbitrary Scale Optical Flow with Implicit Neural Representation, Hyunyoung Jung, Zhuo Hui, Lei Luo, Haitao Yang, Feng Liu, Sungjoo Yoo, Rakesh Ranjan, Denis Demandolx (https: / arxiv.org / abs / 2303.16493) proposes a neural method for estimating dense motion fields, or optical flow. This learning-based technique requires a large number of parameters (about 5 million) and is complex in terms of the number of operations per pixel, as is the autoencoder approach. In addition, the optical flows obtained are not suitable for their transmission at low bitrates. These two aspects do not allow its application to low-complexity video coding.

[0010] In the field of coding, the article "AIVC: Artificial Intelligence based VideoCodec, Théo Ladune, Pierrick Philippe", ICIP 2022, Oct 2022) proposes a video coding method that includes two auto-encoders, one of which is used to encode information representative of the movement. However, this auto-encoder is very complex and includes several tens of millions of parameters.

[0011] There is therefore a need for a solution to generate a dense motion field and associated parameters in order to encode or decode an image in a simple and efficient manner.

[0012] The invention relates to a method for coding a signal comprising a plurality of samples comprising motion information, the method comprising the following steps: - construction of a first group of characteristic maps; - for at least one sample, called the current sample, of the signal, associated with a position in the signal to be coded: - construction of a characteristic vector from said characteristic maps of said first group; - processing of said characteristic vector by an artificial neural network defined by a set of parameters, to provide a vector representative of a decoded value of the current sample; - updating of at least one value of one of said characteristic maps of said first group and / or of at least one parameter, as a function of a coding performance measurement, - coding of said first group of characteristic maps and of said set of parameters.

[0013] The invention also relates to a method for decoding a signal comprising a plurality of samples comprising motion information, the method comprising the following steps: - decoding a first group of characteristic maps; - decoding a set of parameters representative of a neural network; - for at least one sample, called the current sample, of the signal: - construction of a characteristic vector from the characteristic maps of said first group, and: - processing of said characteristic vector by an artificial neural network defined by the decoded parameters to provide a vector representative of a decoded value of the current sample.

[0014] For the purposes of the invention, encoding, or "coding", means the operation which consists of representing a set of samples in a compact form carried for example by a digital binary train. Decoding means the operation which consists of processing a digital binary train to restore decoded samples.

[0015] A signal "sample" means a value taken from the signal. Sampling the signal produces a sequence of discrete values ​​called samples. A sample has at least one associated motion information. It may have other information. It is represented by a vector of dimension A, i.e., comprising A values. A sample can be associated with a position in an image (pixel, rectangle, etc.).

[0016] By "motion information" we mean any representation of motion that can be associated with a sample, for example a motion vector, or several motion vectors, or a set of equations defining a motion.

[0017] A "signal comprising a plurality of samples" means a signal with two (image, composite image) or more than two (stereoscopic, multiscopic, image associated with a depth map, video, etc.) dimensions. Depending on this dimensionality, the sample has two or more coordinates in the signal. In the case of a signal represented by a two-dimensional image, the position of the sample is commonly identified by its abscissa (x) and ordinate (y) coordinates. In the case of a signal represented by a three-dimensional image, the position of the sample is usually identified by its abscissa (x), ordinate (y), and depth (z) coordinates, etc.

[0018] By "feature maps" is meant an abstract representation of the signal comprising a plurality of variable scalar data, discrete or not, which are also called characteristic values, for example real or integer numbers, signed or unsigned. They can also include vectors or scalar values. As is known, these maps are also referred to as the "latent representation" of the input signal.

[0019] By "data characteristic vector constructed from the characteristic maps" is meant a vector consisting of one or more elements, or data, preferably discrete, the data being constructed from the characteristic maps. This characteristic vector is the one that is applied to the input of the neural network. In the case of an image, such a vector can be constituted from a plurality of values ​​taken from each of the characteristic maps at the same abscissa and ordinate coordinates as the sample to be coded. The vector can also comprise values ​​taken from each of the characteristic maps at coordinates close to the sample to be coded (for example immediately above, to the right, etc.).It can also be formed from a plurality of values ​​taken from each of the feature maps at a position that depends not only on the position of the sample but also on the size of the feature map. For example, in a feature map with a resolution lower than a factor of two of the image to be reconstructed, the corresponding feature value can be taken at the abscissa and ordinate coordinates divided by two. Once these values ​​are taken from the feature maps, they can be processed to form the feature vector, before input to a neural network, a quantization function, filtering, interpolation, etc.

[0020] By "neural network" we mean a neural network such as a convolutional neural network, a multi-layer perceptron, an LSTM (for "Long Short Term Memory"), etc. The neural network is defined for example by a plurality of layers of artificial neurons which comprise a set of activation, weighting and addition functions (for example, a layer can calculate y = f(Ax+b), where y and b are vectors of dimension N, x is a vector of dimension M, A is a matrix of dimension MxN, and f is the activation function). Hereinafter we speak of "neural network" or "synthetic neural network".

[0021] By "neural network parameter" we mean one of the values ​​that characterizes the neural network, for example a weight associated with one of the neurons (filter coefficient, weighting, bias, value affecting the operation of non-linearity), etc.

[0022] By "processing by a neural network" is meant the application of a function expressed by a neural network to the input characteristic vector to produce an output vector representative of the sample to be encoded (resp. to be decoded). This output vector comprises one or more data representative of the sample. It can for example represent a motion vector. It can also represent a motion vector and associated composition information.

[0023] By "performance measurement" is meant a measurement between at least one value of a sample to be encoded and a decoded value of said sample. The measurement can evaluate, for example, a distortion, or a perceptual error. It can be carried out on a sample or a plurality of samples (for example, the complete image). The measurement can also include a measurement of the flow rate, in particular associated with the coding of the neural network and / or the coding of the characteristic maps. The measurement can be a joint measurement between the flow rate and the distortion through their weighting. As is well known in the state of the art, the value of this measurement is generally minimized until a target value is reached.

[0024] Generally speaking, it is considered that the steps of an coding or decoding method should not be interpreted as being linked to a notion of temporal succession. In other words, the steps may be carried out in a different order than that indicated in the independent coding or decoding claim, or even in parallel.

[0025] The coding method according to the invention constructs the coding parameters from the input signal, for example a motion image, or composite comprising motion and composition information, by training a neural network on characteristic vectors associated with the position of a sample to be coded. These characteristic vectors are constructed from characteristic maps. During the training, or construction, the parameters of the neural network and the values ​​of the characteristic maps are updated according to a performance measurement, for example of the rate-distortion type. When the training is finished, that is to say when the performance measurement obtained is satisfactory, the actual coding of the parameters of the neural network and / or the values ​​of the characteristic maps can be carried out and stored or transmitted to the decoder.

[0026] Advantageously, the training process makes it possible to refine the parameters of the neural network and / or the values ​​of the feature maps until an adequate representation in terms of performance is obtained, for example a desired balance between the generated bit rate and the distortion undergone by the input signal. The training of the values ​​of the feature maps and the parameters of the neural network can be joint. The training aims to construct the parameters representative of the image, before their actual coding. Advantageously, the coding method according to the invention makes it possible to efficiently compress the signal.

[0027] Advantageously, the decoding process is simple since it is sufficient to decode the feature maps and the neural network to reconstruct a decoded version of the signal, for example a motion or composition map.

[0028] Such a neural network can advantageously be of very simple structure with few parameters. They can thus be conveyed at low speed and adapted to the particular signal.

[0029] The decoded signal containing motion information per pixel can be used in itself, for example to analyze a motion field, for a sports medicine application for example, or for a trajectory tracking application, etc. It can also be used, as is well known, in a video decoder, in order to exploit the temporal redundancy between the images of a sequence.

[0030] Standard video coders use motion estimation (at the encoder) and motion compensation (at the decoder) for inter-frame prediction. Motion vectors indicate how the pixels in the reference frame should be shifted in order to obtain a good prediction for the pixels in the frame being encoded / decoded. Motion compensation is often performed on blocks, allowing a reduced number of motion vectors to be transmitted. However, such compensation is inaccurate and produces block artifacts. Using per-pixel information improves this prediction.

[0031] According to embodiments of the coding and / or decoding method:

[0032] - The method comprises a step of constructing a second group of feature maps and the feature vector is further constructed from said feature maps. Advantageously, these additional maps of the second group, constructed identically to the coder and the decoder, are neither stored nor transmitted at the coder level, nor decoded at the decoder level. They thus make it possible to benefit from additional data to improve compression without degrading the bit rate. They may for example comprise coordinates, data available in the maps of the first group, data concerning other images already processed by the coder or decoder, etc. Unlike the maps of the first group, they are not transmitted by the coder, and are not decoded by the decoder. These maps of the second group are directly deduced from the maps of the first group.

[0033] – At least one of the feature maps has a lower resolution than the signal to be coded (or decoded). Advantageously, in this mode, the compression of the feature maps is more efficient since at least one of the feature maps has fewer values ​​than if it were at the resolution of the signal. For example, in the case of a digital optical flow image, one of the feature maps may have a resolution of 1 / 2, i.e. it has half as many values ​​on the abscissa and ordinate as the input signal has samples, i.e. a total of 4 times fewer values ​​than a feature map at the resolution of the signal.

[0034] – At least one of said characteristic maps has a resolution identical to that of the signal to be coded (resp. decoded). Advantageously, according to this mode, at least one of the characteristic maps, at the same resolution as the input signal (resp. to be decoded), allows significant fidelity and respect for the details of the initial resolution of the signal.

[0035] - The characteristic vector is constructed as a function of the said position of the said current sample.

[0036] – The construction of said characteristic vector comprises a sub-step of extracting a value from said at least one characteristic map at a position identical to that of the current sample in the signal.

[0037] Advantageously, it is thus possible to extract a value from a feature map of the first or second group, at a position identical to that of the sample in the signal (input signal for coding, signal to be decoded for decoding), to constitute an element of the feature vector. This method is simple to implement, in particular via an MLP type network. For example, if we have J feature maps as input, with the same resolution as the signal, a simple extraction of the values ​​of the maps at the coordinates of the current sample (at the same abscissa and the same coordinate in the feature map) makes it possible to directly construct the feature vector of J elements. Moreover, in this case, the decoding can be carried out sample by sample, in a progressive manner.

[0038] – The construction of said characteristic vector comprises a sub-step of extracting at least one value from said at least one characteristic map at a position close to that of the current sample in the signal.

[0039] Advantageously, it is thus possible to extract values ​​from a feature map of the first or second group, at a position close to that of the sample in the signal (input signal for coding, signal to be decoded for decoding), to constitute one or more elements of the feature vector. This method is simple to implement, and is particularly suitable in the case of a network using a convolution layer. By neighbor, we mean in a close neighborhood in the sense of a distance measurement (for example, a neighborhood of 3 samples, for example a neighborhood covering the surface of a convolution kernel, etc.)

[0040] – The construction of said characteristic vector includes a sub-step of processing said extracted values ​​to obtain the characteristic vector.

[0041] Advantageously, it is possible to process the values ​​extracted from the feature maps to construct the feature vector. Such processing may, for example, correspond to a quantization of the extracted data, or to scaling, or to filtering, etc. During coding, quantization is essential for the proper functioning of the system if the feature maps contain, for example, floating point or real values. It is necessary to quantize them before coding them and / or providing them as input to the neural network. During decoding, however, inverse quantization is not necessary, depending on the embodiments. This is the case in particular when the quantization step is reduced to the simple approximation of a value.

[0042] - The coding (resp. decoding) of said first group of feature maps comprises an entropic coding (resp. decoding) sub-step. Advantageously, the prediction and entropic coding make it possible to exploit the statistical redundancies of the signal which may persist in the feature maps, the signal thus being compressed more efficiently.

[0043] – A sample of the signal includes at least one composition information

[0044] Composition information includes one or more data items for indicating how to use the motion information, which may include, but is not limited to, merging a compensated signal with another signal, masking areas of the signal, or weighting the importance of the signal. Composition information may be used to advantage to indicate how to use the motion information, particularly when several reference images are used to perform a prediction by motion compensation. "Motion compensation" means the generation of a predicted, or motion-compensated, image using a reference image and motion information. Such composition information may indicate a blending factor for weighting the different reference images used, for example after their motion compensation.The composition information contained in the decoded samples can thus be used to improve the compensation of the image by taking into account its content, in particular occlusion zones, or static zones, or high motion zones of the image, as is well known to those skilled in the art, for specific processing of these zones. For example, if an area is occluded, that is to say it does not exist in the reference images, it can be ignored, or even predicted using an Intra type prediction. According to another example, a combination of several motion-compensated images can be carried out, using a weighting of the different zones of the compensated images. For example, an occluded zone will only be visible on a reference image and the weighting indicates that the decoding will be carried out solely based on the reference image which contains the occluded zone. This improves the prediction of the image.Thus, the composition information may include data indicating a weighting factor, a masking factor, a reference image, a type of reference to use, etc. A composite is a sample or image containing composition information.

[0045] – Motion information includes at least one motion vector.

[0046] In the case of a signal comprising motion information, the sample can traditionally be represented by a motion vector, for example a pair of values ​​indicating horizontal and vertical movement of a pixel or group of pixels in a digital image. In the case of motion information per sample, we speak of a dense motion field, or optical flow. Generally, a sample is represented by a vector of one or more components, or data. The motion vector can also be represented by an angle and an amplitude, etc. Motion values ​​can represent movements on integers, or floating point numbers to signify precise displacements.

[0047] - The coding (resp. decoding) of the signal is followed by a step of compensating at least one reference image using the motion information, to obtain a motion-compensated image. Advantageously, the motion information associated with the decoded samples can be used, as is well known to those skilled in the art, in a video decoder, to compensate a reference image.

[0048] - The coding (resp. decoding) method is followed by a step of combination from at least one reference image using said motion-compensated image and said composition information, as set out above and described in the remainder of the description.

[0049] Correlatively, the invention also relates to a coding device and a decoding device.

[0050] The characteristics and advantages of the coding or decoding method apply in the same way to the coding or decoding device according to the invention and vice versa.

[0051] The invention also relates to a computer program on a recording medium, this program being capable of being implemented in a computer or an encoding or decoding device in accordance with the invention. This program comprises instructions adapted to the implementation of the corresponding method. This program can use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0052] The invention also relates to a computer-readable information carrier or recording medium comprising computer program instructions mentioned above. The information or recording media may be any entity or device capable of storing the programs. For example, the media may comprise a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a floppy disk or a hard disk, a DNA sequence, or a flash memory. Furthermore, the information or recording media may be transmissible media such as an electrical or optical signal, which may be conveyed via an electrical or optical cable, by radio link, by wireless optical link or by other means.

[0053] The program according to the invention can in particular be downloaded from an Internet-type network.

[0054] Alternatively, each information or recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of a method according to the invention.

[0055] Other characteristics and advantages of the present invention will emerge from the description given below, with reference to the appended drawings which illustrate exemplary embodiments thereof which are not limiting in nature.

[0056] Schematically represents a coding device used in the context of the invention;

[0057] schematically represents a decoding device used in the context of the invention;

[0058] illustrates an example of a synthetic artificial neural network used in the context of the invention for coding and decoding;

[0059] is a flowchart representing an example of a coding method that can be implemented by the coding device of the;

[0060] illustrates an encoding method used in one embodiment of the invention;

[0061] illustrates an encoding method used in another embodiment of the invention;

[0062] is a flowchart representing an example of a decoding method that can be implemented by the decoding device of the;

[0063] illustrates a decoding method used in one embodiment of the invention;

[0064] illustrates a decoding method used in another embodiment of the invention;

[0065] illustrates a method for decoding video sequences using a motion image generated according to one embodiment of the invention;

[0066] illustrates a method of decoding a video sequence using a composite image generated according to one embodiment of the invention;

[0067] illustrates a method of decoding a video sequence using a composite image generated according to another embodiment of the invention.

[0068] Schematically represents an ENC coding device.

[0069] This ENC coding device includes a GEN module for generating feature maps, a TT processing and quantification module, an XTR data extraction module, an MLP module corresponding to an artificial neural network, an NNC neural network coding module, an FMC module for coding feature maps, an EVAL module for evaluating a performance function, an MAJ update module, and an optional FME module for generating additional feature maps.

[0070] The ENC coding device may be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the above-mentioned modules may then be realized by the cooperation of the processor and computer program instructions stored in the above-mentioned memory and designed to perform the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0071] The ENC coding device receives as input a succession of samples to be coded, noted P n , for example a temporal succession of data corresponding to an image containing motion data, noted I(P n ). In this case, the image signal I(P n ) can represent a two-dimensional motion image, or a three-dimensional one corresponding to a plurality of motion images. P ndenotes a sample n of the input signal comprising N samples. According to another example, the coding device ENC receives as input a succession of samples to be coded P n , in the form of a temporal succession of data corresponding to a composite image I(P n ) including movement and composition information.

[0072] In one embodiment, the signal is an image comprising motion information represented by means of at least one two-dimensional representation, such as a matrix of pixels, of width W and height H (with N = WxH), each sample of the image conventionally corresponding to a motion vector comprising a horizontal (MVx) and vertical (MVy) component, or, alternatively, an amplitude component and an angle component. The location of each motion vector is defined by its abscissa and ordinate (x and y) coordinates in the motion image. According to other embodiments, the output motion vector may comprise more than two values ​​(for example 12 for a polynomial model, 4 for an affine model, 8 for a bilinear or homographic model, etc.).If the motion information concerns several images, for example if the motion information applies to several images then each sample of the image can refer to the relative movements of the different images. In the case where the motion information applies to two images then the motion samples can comprise two motion vectors.

[0073] According to other embodiments, a sample further comprises composition information C. Such composition information may consist of K data (c1,...cK), where K is a natural integer, such as for example weighting, masking, fusion coefficients, or more generally data of any type making it possible to indicate how to apply the motion information (for example, the type of reference image to use, the weighting to be carried out on a reference image, or the reference image number to use, or a weighting to be applied to the motion vector, a motion correction, etc.).

[0074] In a particular embodiment, the signal is an image comprising motion information comprising:- a displacement relative to a first reference image with two components (mv1x, mv1y);- a displacement relative to a second reference image with two components (mv2x, mv2y), in a particular case mv2x is of opposite sign to mv1x and mv2y of opposite sign to mv1y;- weighting information (c1) to be applied between the reference images after the application of the relative displacement, this weighting being able to take values ​​c1 between 0 and 1 and applied by multiplication to the product of the first displacement and by multiplication by (1- c1) to the second product.- masking or mode information (c2), the value of which is between 0 and 1, which is intended to eliminate by multiplication undesirable zones resulting from the weighting operation or to indicate a particular processing mode for the composite image.

[0075] As will be described in more detail later with reference to Figures 3 to 12:

[0076] The GEN module for generating characteristic maps is configured to generate, and in particular initialize, a plurality of M characteristic maps denoted FM i The optional FME module can generate one or more additional cards, numbering L, which will not be coded or transmitted, noted FME l .

[0077] The XTR module performs value extraction in FM feature maps i and optionally FME l , for a sample P n to code.

[0078] The TT module performs processing of the extracted values, to generate the vector Z n . Z n is a J-tuple, that is, it has J elements, or data z i . The vector Z n of index n refers to the characteristic vector of the pixel P' n .

[0079] The TT module performs a quantification of the data extracted from this set of M FM maps i , or the vector Z n constituted from this data. To prepare for subsequent coding (e.g., entropy-based), the coding system quantizes the latent representation of the input data by using a quantizer Q to generate an ordered collection of quantized values. Recall that quantizing a value refers to mapping that value to a member of a discrete set of possible code symbols. For example, the set of possible code symbols may consist of integer values, and the quantization system performs simple rounding from a real value to an integer value. In another example, quantization consists of multiplying by a given value and then rounding.

[0080] The MLP module is a neural network defined by K parameters W k, able to process the vector Z n , or J-Uplet, as input, to generate as output a vector representative of the sample P n to be coded. The neural network may comprise, according to one embodiment, an MLP, or Multi Layer Perceptron, consisting of an input layer adapted to the input format (the J-uplet), optionally one or more hidden layer(s), and an output layer adapted to the output format of the output vector, generally a vector comprising A elements. According to one embodiment, the neural network may comprise a convolutional network, a convolutional network comprising, in a known manner, at least one convolution layer, which may be followed by non-linearities. According to one embodiment, A is equal to 2 and the output vector is the pair (mvx, mvy) of the motion vector P' nencoded then decoded. According to one embodiment, as shown in, A is equal to 3 and the output vector is the triplet (mvx, mvy, c1) of the sample P' n encoded then decoded, comprising a motion vector and composition information. According to one embodiment, A is equal to 5 and the output vector is the triplet (mv1x, mv1y, mv2x, mv2y, c1) of the sample P' n coded then decoded, comprising two motion vectors and a composition data item c1. According to one embodiment, as shown in, A is equal to 6 and the output vector is the triplet (mv1x, mv1y, mv2x, mv2y, c1, c2) of the sample P'n coded then decoded, comprising two motion vectors, and a composition information item C comprising a first composition data item c1 indicating for example a weighting and a second composition data item c2 indicating for example a masking.

[0081] The NNC module performs the coding of the neural network, in particular its W parameters k. During the process of training, or building, the coding, that is to say as long as the performance evaluation step is not satisfactory, the NNC module performs a coding simulation, followed by a decoding, intended for the evaluation module. It updates the parameter values ​​according to the results of a performance measurement carried out by the EVAL module. Subsequently, it performs the actual coding of the parameters of the neural network W k . The coded parameters are denoted Wc k . As is known, the coding simulation can be identical to the actual coding, or an approximation of it.

[0082] The FMC module performs the coding of the FMi cards, i.e. the quantified values ​​of each card of the first group (excluding the FME cards l). During the process of training, or building, the coding, that is to say as long as the stage of evaluating a performance is not satisfactory, the FMC module carries out a coding simulation, followed by a decoding, intended for the evaluation module. Subsequently, it carries out the actual coding of the values ​​of the FM maps i . Coded cards are denoted FMc i . As is known, the coding simulation can be identical to the actual coding, or an approximation of it. The coding module compresses the quantized data using, for example, entropy-type coding.

[0083] The EVAL module performs an evaluation and minimization of a coding performance. The evaluation function is for example of the rate-distortion type. The minimization can be carried out by gradient descent, or any other method within the reach of a person skilled in the art.

[0084] The MAJ module updates the values ​​of the FMi cards to be encoded based on the results of the performance function. Optionally, it can update the values ​​of the additional FME cards. l .

[0085] It schematically represents a DEC decoding device.

[0086] The DEC coding device receives as input a first group of encoded data organized into M FMc characteristic maps i (also called FM layers) and the Wc encoded parameters k of the neural network.

[0087] This DEC decoding device includes a neural network decoding NND module, a feature map decoding FMD module, a data extraction XTR' module, an inverse processing TT' module, a neural network corresponding MLP' module, an optional additional feature map generation FME' module.

[0088] The cards decoded by the FMD module, numbering M, are noted FMd i . The parameters decoded by the NND module are denoted Wd k .

[0089] The decoder's FME' module can also generate one or more additional cards, denoted FME' l , numbered L, identical to the FME additional cards l generated by the encoder.

[0090] The XTR' module is identical to the XTR module of the. It performs an extraction of values ​​from the M FMd characteristic maps i decoded, and possibly additional FME cards' l , to construct the vector Z n taking the form of a J-tuple for a sample P n to be decoded, depending on its coordinates in the signal to be decoded. In one embodiment J comprises a number of samples depending on M, L and the topology of a convolutional neural network.

[0091] The TT' module optionally performs an inverse quantization corresponding to the quantization performed at the encoder. The inverse quantization is not necessary if the encoder's quantizer Q has performed a simple rounding of the real values ​​submitted to it. The inverse quantization is not necessary if the neural network is able to take into account a quantization of its input data. Otherwise, the decoder performs the inverse operation of the quantizer Q. Furthermore, like the encoder's TT module, the TT' module optionally performs a processing of the values ​​extracted from the feature maps, to generate the decoded vector Zd n .

[0092] The MLP' module is a neural network defined by K parameters Wd k , capable of processing the vector Zd n , or J-Uplet, as input, to generate as output a vector representative of the sample P nto decode, generally a vector comprising A elements. According to one embodiment, A=2 and the output vector is the pair (mvx, mvy) of the motion vector Pd n decoded. According to one embodiment, A is equal to 3 and the output vector is the triplet (mvx, mvy, c1) of the sample P' n encoded then decoded, comprising a motion vector and composition information C comprising a data item c1. As described for the encoder, different embodiments are possible. According to one embodiment, A is equal to 5 and the output vector is the quintuplet (mv1x, mv1y, mv2x, mv2y, c1) of the sample P' ncoded then decoded, comprising two motion vectors and a data c1 indicating a weighting. According to one embodiment, A is equal to 6 and the output vector is the triplet (mv1x, mv1y, mv2x, mv2y, c1, c2) of the sample P'n coded then decoded, comprising two motion vectors, composition information comprising a weighting data c1 and a masking data c2. The MLP' module has an identical structure to the MLP module, and its parameters are either identical if the coding of its parameters W k is lossless, or different if the coding is done with losses.

[0093] When all samples P n of the signal have been decoded, we have a reconstructed signal I(Pd n), selon un mode de réalisation une image comportant N échantillons décodés sous la forme de N vecteurs Pdnde dimension A. De manière équivalente, on peut dire que l'image comporte A composantes, chaque composante comprenant une valeur du vecteur de sortie, par exemple l'image reconstruite comporte une composante pour les vecteurs de mouvement horizontaux (MVx), une composante pour les vecteurs de mouvement verticaux (MVy), K composantes (C1...CK) pour l'information de composition C(= c1...cK)

[0094] The DEC decoding device may be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the above-mentioned modules may then be realized by the cooperation of the processor and computer program instructions stored in the above-mentioned memory and designed to perform the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0095] illustrates an example of a synthetic artificial neural network used for encoding and decoding in the context of embodiments of the invention applied to motion or composite images consisting of motion vectors and optionally composition information.

[0096] The MLP synthetic artificial neural network for coding and the MLP' synthetic artificial neural network for decoding are defined by an identical structure, comprising for example a plurality of layers of artificial neurons, and by a set of weights and activation functions associated respectively with the artificial neurons of the network concerned.

[0097] A vector representation of a current sample and optionally its neighborhood (a vector Z n or Zd n from FM feature maps i and FME l or FMd i and FME' l ) is applied as input (i.e. on an input layer) of the MLP or MLP' synthetic artificial neural network. The artificial neural network produces as output a vector P' n or Pd nrepresentative of the decoded sample, according to one embodiment the motion components (mvx, mvy) constituting a motion vector of a motion image. According to a variant represented in dotted lines, the output vector also includes a composition data c1 (C={c1})

[0098] The concatenation of all these reconstructed pixels into a motion or composite image constitutes the decoded, or reconstructed, image.

[0099] At the encoder, the MLP synthesis artificial neural network is trained on the image, so as to minimize the differences between the input representation of the current motion or composite image (P n ) and its representation I(P' n ) output, while also minimizing the amount of data to be encoded. The EVAL module performs a performance measurement in this regard.

[0100] Once training is complete, the network parameters are encoded, either lossless, in which case the MLP' neural network is identical to MLP, or lossy, in which case the MLP' network may be slightly different from MLP.

[0101] This is a flowchart representing an example of a coding process that can be implemented by the coding device of the.

[0102] According to this embodiment, the signal is a composite two-dimensional image, each sample to be encoded P n of coordinates (x n , y n ) includes a motion vector and composition information C.

[0103] Encoding takes place in two main phases:

[0104] In a first phase, called the training phase, learning is carried out in order to determine, for an input signal I(P n ), the values ​​of FM cards i and W parameters kto optimize a global cost function. Learning is for example achieved by gradient descent, followed by updating the parameters of the MLP neural network and the values ​​of the FM feature maps i . As known in the state of the art, the cost function can be of rate-distortion type, or rate, or distortion, or perceptual. To measure the rate R, it is necessary to simulate the coding of FM cards i , then measure the associated coding rate (the size of the stream B1). According to one embodiment, the coding of the parameters W is not simulated k because their influence is less important than that of the feature maps. According to one embodiment, the coding of the parameters W is also simulated kand the associated flow rate (the size of the flow B2) is measured. To measure the distortion D, it is necessary to simulate the coding and then the decoding of at least part of the image I, to obtain at least one motion and composition vector P' n resulting from a simulation of coding then decoding, then measuring the difference between this part of the composite image I(P n ) as input and a corresponding part of the composite image I(P' n ) encoded then decoded.

[0105] Then during a second phase, called the coding phase, the FM cards i and the W parameters k are encoded to produce the FMc encoded values i and Wc k before transmission or storage. They constitute the compressed representation of the input signal I (P n ).

[0106] We will now describe the steps of a method according to one embodiment of the invention.

[0107] During a step E20, a signal I(Pn ) to be coded, comprising a plurality of N samples P n , is provided as input to the process.

[0108] During an E21 step, the M FM cards i of the first group are initialized by the GEN module. Subsequently, the W parameters k of the MLP neural network and the values ​​of the FM maps i must be optimized during the construction phase.

[0109] According to one embodiment, the FM cards i are of the same resolution as the input signal I(P n ) and therefore each have the same number of values ​​N as there are samples P n to code.

[0110] According to one embodiment, the FM cards i are of resolution less than or equal to that of the input signal I(P n ) and therefore include, for at least one of them, a number N' of values ​​to be coded less than N.

[0111] According to one embodiment, the first FM cardi is at the signal resolution and each subsequent map is at half the resolution of the previous one.

[0112] According to one embodiment, several FM cards i are of the same resolution, lower than the signal resolution.

[0113] According to one embodiment, the FM cards i are initialized by predefined constant values.

[0114] According to another embodiment, the feature maps are initialized by a set of random real values.

[0115] According to one embodiment, one or more FME cards l , forming a second groupe de L cartes de caractéristiques supplémentaires, sont générées et ajoutées au premier groupe. Elles servent à la construction du vecteur caractéristique mais ne seront ni stockées ni transmises.

[0116] These FM feature cards i of the first group are subsequently updated, or refined, during a step E22 by the MAJ update module of the encoder during its learning during the construction phase.

[0117] During an E23 step, the FM cards i of the first group are encoded by the FMC module of the encoder. During the construction phase, this operation is an encoding simulation. During the encoding phase, this operation is an actual encoding and the encoded values ​​constitute the B1 stream. The simulation can be identical to the actual encoding but it can also be different (for example, simplified). For this encoding, any known technique aimed at compressing the values ​​of the maps can be used.

[0118] In one embodiment, the FM cards i are encoded in order (FM1, FM2,…, FM4), and the variables of each map in a predefined order, for example lexicographic. Each map undergoes entropy coding. The entropy coding produces a compressed stream B1 whose flow rate is subsequently measured during a step E28.

[0119] During an E24 step, values ​​are extracted by the XTR module from the FM cards iand possibly FME l. This extraction can be performed based on the coordinates (x n, yn) de l’échantillon Pndu signal d’entrée.

[0120] According to one embodiment, the coordinate values ​​(x n , y n ) of each card are extracted.

[0121] According to one embodiment, the values ​​adjacent to the coordinates (x n , y n ) of each card are also extracted.

[0122] According to one embodiment, the coordinate values ​​are scaled to extract the lower resolution layers.

[0123] According to one embodiment, values ​​close to those previously cited are also extracted for the subsequent processing of step E25, for example for a convolution kernel.

[0124] At a step E25, a vector Z n is constructed by the TT module from the values ​​extracted from the FM maps i and possibly FMEl for each sample P n of coordinates (x n , y n ) of the input signal. The processing includes a quantization of the values ​​extracted from the FM maps i , if necessary. The processing may include other operations, such as filtering, scaling, applying any function, preferably monotonic, etc.

[0125] In one embodiment, Z n includes as many values ​​as there are cards or FMi cards (and optionally FME l ) as input.

[0126] In one embodiment, Z n is a J-tuple (z1, z2,…, z J ), consisting of the values ​​of the FM cards i (and optionally FME l ) located at the coordinates (x n , y n ) of the current pixel P n ,as will be illustrated in support of the. The value located at the coordinates (x n, yn) de la carte FMidonne la valeur finale (zi) de l’élément i du vecteur Znrelative à cette carte FMiou FMEl. Dans ce cas le module TT n’applique aucun traitement autre que la quantification.

[0127] In one embodiment, Z nis a J-uplet constructed from values ​​taken from the FMi (and optionally FME) maps l ) to coordinates that may be different depending on the maps. For example, if the FM maps i (and / or FME l ) are at different resolutions because they have been downsampled, the coordinates are adapted (by scaling) to match the resolution of each map, as will be illustrated in support of the.

[0128] In one embodiment, Z n is a J-tuple constructed from values ​​taken from FM maps i (and FME l ) by applying the processing to one or more values ​​of the maps, for example filtering the values ​​close to the target value in a map. For example, in an FMi map which is at the same resolution as the input signal, we can extract the values ​​located at the coordinates (x n , y n ), (x n -1, yn ), (x n , y n -1) and (x n -1, y n -1) and apply processing to these values ​​(filtering, averaging, interpolation, application of a convolution network, etc.) to obtain the final value (z i ) of element i of vector Z n relating to this FM card i or FME l . According to another example, in an FM card i which is at half the resolution of the input signal, we can consider the values ​​located at the coordinates (x n / 2, y n / 2), (x n / 2-1, y n / 2), (x n / 2, y n / 2-1) and (x n / 2-1, y n / 2-1) and apply processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (z i ) of element i of vector Z n relating to this FM card i or FME l .

[0129] The samples to be coded are, for example, processed in sequential order, from n=1 to n=N.

[0130] During a step E26, the vector Z n is processed by the MLP neural network to generate as output a vector representative of the sample P n to code, according to one embodiment, the triplet (mvx, mvy,c1) of the sample P' n (sample P n encoded then decoded), according to other embodiments an N-tuple consisting of one or more motion vectors and one or more composition data.

[0131] The W structure and parameters k of the neural network are initialized for example during the first iteration of this step. These parameters are subsequently updated, or refined, during the construction phase, during subsequent iterations of the process.

[0132] According to one embodiment, the parameters of the neural network are initialized by predefined values ​​known to give a satisfactory result (for example, following training on a corpus of images).

[0133] According to another embodiment, the parameters W k of the neural network are initialized by a set of random values.

[0134] During a step E27, the parameters W kof the MLP neural network are quantized and encoded. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is an actual coding and the encoded values ​​constitute the B2 stream. The simulation can be identical to the actual coding but it can also be different (for example, simplified). Any known technique can be used for this purpose, for example the neural network coding standard proposed by the MPEG-7 part 17 standard, also called NNR (Neural Network Representation). Note that in this case, it is necessary to choose the amount of degradation that the coding brings to the parameters W k .

[0135] During step E28, a performance measure is evaluated.

[0136] For this purpose, the coding simulation rates associated with the characteristic maps (simulation of the B1 flow by coding and decoding of the FM maps i) and optionally to the parameters of the neural network (simulation of the B2 flow by coding and decoding of the W parameters k ) are measured.

[0137] According to one embodiment, the cost function is of the rate-distortion type, noted (D+L*R), where D is for example the quadratic error measured between the input signal and the decoded signal (or the error measured on a subset of samples of the signal). According to another example, D is calculated from the maximum deviation in absolute value between the coded and original values. According to one embodiment, R is the simulated rate of the stream B1; according to another embodiment, R is the total rate used to code this image, that is to say the sum of the simulated rates of B1 and B2. L is a parameter which regulates the rate-distortion trade-off. Other cost functions are possible.

[0138] According to another embodiment, the distortion D is measured not on the motion image itself, but on the result of the use of the motion vector. For example, if the motion field is used to perform motion compensation of an image, then the distortion will be measured on the compensated image. In this case the distortion measurement may comprise a perceptual measurement of the MS-SSIM, VMAF, etc. type as is well known to those skilled in the art.

[0139] As long as the cost function has not reached its minimum, the performance measure is not satisfactory, and the method is repeated from step E22. Alternatively, a maximum number of iterations, or a maximum learning time can be defined. This minimization can be performed by a mechanism known as gradient descent with parameter updates during step E22 for the feature map values ​​and E26 for the network parameters.

[0140] At step E29, if the cost function has reached its minimum, training stops. If a coded version corresponding to the last simulation of the neural network parameters (W k ) and feature maps (FM i ) is available, streams B1 and B2 can be made up of them. According to another embodiment, the effective coding of the updated parameters of the neural network (W k ) and values ​​of the characteristic maps (FM i ) is performed at this step to produce the encoded parameters Wc k and FMc i which constitute B1 flows and B2.

[0141] The streams B1 and B2 may be concatenated to produce a final stream BS. According to one embodiment, the stream B2 of encoded parameters of the neural network is stored or transmitted before the stream B1, so that it can be decoded before the stream B1.

[0142] The following illustrates an encoding method used in one embodiment of the invention for encoding an image I(P n ) (not shown as input) of N samples corresponding to motion vectors (in which case we speak of a motion image) and optionally composition information (in which case we speak of a composite image).

[0143] In this embodiment, the FM cards i are 4 in number. In a preferred mode, there are 7. FM cards i are of the same resolution as the image I(P n ), and therefore each have WxH values, where W represents the width of the image in pixels and H its height (N=WxH).

[0144] According to a variant presented in dotted lines, the feature maps are 5 in number: an additional map FME0 has been introduced, which will neither be stored nor transmitted. This additional map typically contains data that can assist the MLP network in the task of signal reconstruction. Thus, the added maps can be one or more from the following non-exhaustive list:

[0145] – A map showing the abscissa of that point at each point.

[0146] – A map showing the ordinate of that point at each point.

[0147] – A map with positional coding at each point (as described for example at the internet address https: / skosmos.loterre.fr / P66 / fr / page / -K0D65X2X-X ).

[0148] – A map representing an image distinct from the image being processed, capable of providing information about the image to be coded, for example the previously processed image if the current image is part of a series of motion or composite images to be coded such as a video, a set of medical images, a multi-view representation, etc.

[0149] – A map representing a feature map of an image distinct from the image being processed that may provide information about the image to be encoded, for example, a previously processed map if the current image is part of a series of motion or composite images to be encoded such as a video, a set of medical images, a multi-view representation, etc.

[0150] – A card containing the value of an already decoded sample from the same card, for example the previous sample in the decoding order.

[0151] In this embodiment, the vector Zn is a 4-tuple (z1...z4) consisting of the values ​​extracted from the FM maps i located at the coordinates (x n , y n ) of the current sample P n The extracted values ​​are quantified by the modulus TT and the vector Z n consisting of the quantized values ​​is processed by the MLP neural network to generate an output vector.

[0152] In one embodiment, the 4 FM cards i are quantified.

[0153] According to the variant shown in dotted lines, the vector Z n is a 5-tuple (z0...z4), the value z0 being extracted from the additional map FME0.

[0154] In this embodiment, the output vector is a pair (mvx, mvy) representative of the sample P n to be coded. The output vector is in this case the pair (mvx, mvy) of the sample P' n coded then decoded. The pair is inserted into the decoded image I(P' n) at positions (x n , y n ) of the horizontal and vertical motion components (MVX', MVY'). In the embodiment shown in dotted-dotted lines, the output vector is a triplet (mvx, mvy, c1) representative of the sample P n to be encoded. The output vector is inserted into the decoded image I(P' n ) at positions (x n , y n ) of its horizontal and vertical movement components (MVX', MVY') and its composition component (C1').

[0155] The following illustrates an encoding method used in another embodiment of the invention.

[0156] In this embodiment, the FM cards iare 4 in number. As in the previous mode, the first map FM1 has the same resolution as image I, and therefore has WxH values, where W represents the width of the image in pixels, and H its height. The second map FM2 has half the resolution (in each dimension) of map FM1. Each additional map has half the resolution of the previous map. This structure makes it possible to reduce the number of variables in the feature maps, which facilitates coding and learning while minimizing the coding cost, particularly in terms of throughput.

[0157] In this embodiment, a 4-tuple Z n is built from the values ​​of the FM maps i located respectively at the coordinates (x n , y n ), (x n / 2, y n / 2), (x n / 4, y n / 4), (x n / 8, y n / 8) of the current pixel P n (x n , y n). These extracted values ​​are processed and quantized to generate the 4-tuple Z n . According to one embodiment, for the FM2 map, the value located at the coordinates (x n / 2, y n / 2) to obtain the final value (z2) of the vector Z n relating to this FM2 map. According to another embodiment, for the FM2 map, the values ​​located at the coordinates (x n / 2, y n / 2), (x n / 2-1, y n / 2), (x n / 2, y n / 2-1), (x n / 2-1, y n / 2-1) and we apply a filter to them to obtain the final value (z2) of the vector Z nrelative to this FM2 map. Oversampling is achieved in this way. Filtering or oversampling can, for example, use bicubic filtering or Lanczos filtering, which are known per se. A more general treatment can also be used: a neural network can be used to produce the final value (z2) by weighting the values ​​located at the coordinates (x n / 2, y n / 2), (x n / 2-1, y n / 2), (x n / 2, y n / 2-1), (x n / 2-1, y n / 2-1), for example an MLP or convolutional network.

[0158] As before, the vector Z n is processed by the MLP neural network to generate an output vector. According to one embodiment, the output vector is a pair (mvx, mvy) representative of the sample P n to be coded. The output vector is in this case the pair (mvx, mvy) of the sample P' ncoded then decoded. The pair is inserted into the decoded image I(P' n ) at positions (x n , y n ) of the horizontal and vertical motion components (MVX', MVY'). In the embodiment shown in dotted-dotted lines, the output vector is a triplet (mvx, mvy,c1) representative of the sample P n to be encoded. The output vector is inserted into the decoded image I(P' n ) at positions (x n , y n ) of its horizontal and vertical movement components (MVX', MVY') and its composition component (C1').

[0159] Other types of structure are possible, for example one can use a reduction rate different from half between the maps (a quarter, or a third, etc.) The oversampling step is then adapted to produce the final value of the vector Zn.

[0160] As described in support of the, other FME cards lcan be introduced, which will be used to generate the vector Z n but will not be stored or transmitted.

[0161] This is a flowchart representing an example of a decoding method that can be implemented by the decoding device of the.

[0162] During a step E30, the streams B1 and B2 are extracted from the encoded stream BS. They respectively contain the coded representations of the maps of the first group FMc i and Wc parameters k.

[0163] During a step E31, the M FMd cards i are generated by decoding FMc values i . For this decoding, any known technique similar to that used in the encoder can be used, preferably entropy decoding. In one embodiment, the FMd cards i are decoded in order (FMd1, FMd2,… FMd4), and the variables of each card in a predefined order, for example lexicographic.

[0164] According to an embodiment as described for the encoder, the FMd cards i are of the same resolution as the signal I (Pd n ) to be reconstituted, that is to say that they contain N values.

[0165] According to an embodiment as described for the encoder, the FMd cards i are at a resolution lower than or equal to that of the signal I (Pd n ) to be reconstituted.

[0166] According to an embodiment as described for the encoder, several FMd cards i are of the same resolution, lower than the signal resolution.

[0167] During a step E32, according to one embodiment, one or more FME' cards l,forming a second group of L additional feature maps, are generated and complement the first group. They are not decoded but generated by the decoder in a manner identical to the encoder generation. They typically include data that can assist the MLP network in the signal reconstruction task. The non-exhaustive list of possible additional feature maps described in support of the for the encoder applies here.

[0168] During a step E33, values ​​are extracted by the XTR' module from the FMd cards i and possibly FME' l. This extraction can be performed based on the coordinates (x n, yn) du signal d’entrée de l’échantillon Pndu signal à reconstruire. Cette étape est identique à l’étape E24 qui a été décrite pour le codeur à l’appui de laet les modes de réalisation décrits s’appliquent.

[0169] The samples to be decoded are for example processed in sequential order, from n=1 to n=N.

[0170] During a step E34, a vector Zd n is constructed by the TT' module from the values ​​extracted from the FMd maps iand possibly FME' l for a sample P n to decode coordinates (x n , y n ). The processing may include inverse quantization of the extracted values ​​or of the vector Zd n constituted, if necessary. The processing may include other operations, such as filtering, scaling, convolution, etc. This step is identical to step E25 which was described for the encoder in support of the and the described embodiments apply. However, inverse quantization is not mandatory. In this particular case, step E34 and the TT' module are therefore optional.

[0171] During a step E35, the Wd parameters k of the MLP neural network' are generated by decoding the Wc values kof the B2 stream. Any known decoding technique corresponding to the coding technique used by the encoder can be used for this purpose. The MLP' neural network is similar to the MLP network, i.e. it has the same structure and has the same parameters, except for the coding, which can be done with or without losses.

[0172] According to one embodiment, the B2 stream is decoded before the B1 stream, in order to have the neural network available before starting to decode the samples.

[0173] During a step E36, the vector Zd n is processed by the MLP' neural network to generate an output vector representative of the sample Pd n to decode, according to one embodiment, a pair inserted into the decoded image I(Pd n ) at positions (x n , y n) of its horizontal and vertical motion components. In another embodiment, the output vector is a triplet (mvx, mvy, c1) inserted into the composite decoded image I(Pd n ) at positions (x n , y n ) of its horizontal and vertical motion components and its composition component C1. This step is identical to step E26 which was described for the encoder in support of the.

[0174] When all the samples of the signal have been processed, the decoded signal corresponding for example to the image I(Pd n ) is available.

[0175] The illustrative representation shows a decoding method used in one embodiment of the invention for decoding an image I(Pd n ).

[0176] In this embodiment, the FMd cards i there are 4 of them. In a preferred mode there are 7 of them.

[0177] In this embodiment, the FMd cards i are of the same resolution as the image to be decoded, and therefore include WxH variables, where W represents the width of the image in pixels, and H its height.

[0178] In this embodiment, the vector Zd n is a 4-tuple (z1...z4) consisting of the values ​​of the FMd maps i located at the coordinates (x n , y n ) of the current pixel Pd n . The vector Zd n is optionally dequantized and then processed by the MLP' neural network to generate a second vector as output. According to a decoding embodiment, corresponding to the coding embodiment of the, the output vector is a pair (mvx, mvy) representative of the motion vector Pd n to decode. The pair is inserted into the decoded image I(Pd n ) at positions (x n , y n) of the horizontal and vertical motion components (MVX, MVY). In the embodiment shown in dotted-dotted lines, the output vector is a triplet (mvx, mvy, c1) representative of the sample P n to be decoded. The output vector is inserted into the decoded composite image I(Pd n ) at positions (x n , y n, a) of its horizontal and vertical motion components (MVX, MVY) and its composition component (C1). In a video coding scheme such as that shown in, the composition data c1 can be used to make a weighted sum of several reference images, one of which is compensated by the MV vector (mvx, mvy).

[0179] According to a variant presented in dotted lines, there are 5 cards: an additional card FME'0 has been introduced. In this embodiment, the vector Zd n is a 5-tuple.

[0180] Naturally, the mode represented in for coding can be transposed mutatis mutandis for decoding.

[0181] Illustratively represents a decoding method used in another embodiment of the invention for decoding an image I(Pd n ).

[0182] The inputs to the MLP neural network are identical to those of the and the same comments apply

[0183] According to this decoding embodiment, the output vector Pd n is a 5-tuple (mv1x, mv1y, mv2x, mv2y, c2) representing two motion vectors to be decoded and a composition information C consisting of a data c2. The 5-tuple is inserted into the decoded image I(Pd n ) at positions (x n , y n) of its horizontal and vertical motion components (MV1X, MV1Y, MV2X, MV2Y) and its composition component (B). In a video coding scheme such as that shown in, the composition data c2 can be used to make a weighted sum of two reference images compensated respectively by the vectors MV1 (mv1x, mv1y) and MV2 (mv2x, mv2y).

[0184] According to a variant represented in mixed dotted lines, the output vector Pd nis a 6-tuple (mv1x, mv1y, mv2x, mv2y, c1, c2) representative of two motion vectors to be decoded and a composition information C consisting of two data (c1, c2). In a video coding scheme such as that shown in Figures 11 and 12, the weighting type composition data c2 can make it possible to make a weighted sum of two reference images compensated respectively by the vectors MV1 (mv1x, mv1y) and MV2 (mv2x, mv2y), the masking type composition data c1 can make it possible to use an Intra type reference image in the occlusion zones.

[0185] The invention illustrates a method for decoding video sequences using a motion image generated according to the method of the invention.

[0186] A motion image as described above, comprising for each position (xn, yn) of the motion image I(Pd n) a motion vector (mvx, mvy) is presented at the input of a motion compensation module, or warping (WARP). The WARP module uses as input one or more IREF reference images and the decoded motion information to perform motion compensation, or warping, on the reference image(s) in order to obtain one or more prediction image(s), or compensated image(s), noted ICOMP. According to the embodiment presented, warping conventionally makes it possible to modify the position of the samples in the reference image to obtain the displaced samples in the compensated image. For example, each pixel of the reference image can be displaced according to the motion vector (mvx, mvy) to obtain a compensated sample in the ICOMP image. Other more complex motion representations, as mentioned previously, can of course be used.It is assumed here that the digital image of the video sequence is of the same size as the composite image, i.e. it has N pixels. Naturally, depending on the variants, it can be of a different size, and in particular larger than N. In this case, a motion vector can be used for groups of pixels, or blocks, of the image.

[0187] The illustrated method represents a method for decoding video sequences using a composite image (of motion and composition) generated according to the method of the invention.

[0188] The composite image I(Pd n) as previously described, in particular in support of the, comprises for each position (xn, yn) a motion vector (mvx, mvy). As previously described in support of the, the motion vector is presented at the input of a motion compensation, or warping (WARP) module. The WARP module uses as input one or more IREF_INTER reference images and the decoded motion information to perform motion compensation, or warping, on the reference image(s) in order to obtain one or more prediction image(s), or compensated image(s), noted ICOMP. The WARP module also uses as input one or more IREF_INTRA reference images which will be used for an intra-image prediction (i.e. they are not intended to undergo motion compensation). The composite image I(Pd n) as described previously, in particular in support of the, also includes for each position (xn, yn) a composition information in the map C1, for example a real number c1 between 0 and 1. The compensated image, the intra image and this composition data c1 are presented as input to a mixer module noted BLEND. The BLEND module generates as output a reconstructed image IREC of which each of the N pixels PIX(n) can be obtained by a calculation of the type: PIX(n) = c1 * ICOMP(n) + (1-c1)*IREF_INTRA(n). By using this mechanism, it is possible to simply define compensated image zones in inter-image mode (when c1 is 1), and compensated image zones in intra-image mode (when c1 is 0) or any combination of the two modes (when c1 is strictly between 0 and 1). This mechanism can be very useful for managing, for example, occlusion zones in an image, which cannot be compensated by movement.

[0189] It is assumed here that the digital image of the video sequence is of the same size as the composite image, i.e. it has N pixels. Naturally, according to variants, it can be of a different size, and in particular larger than N. In this case, a motion vector and a composition coefficient are used for groups of pixels, or blocks, of the image. According to a variant, the composition parameter can also contain data relating to the type or number of image to be used.

[0190] The invention illustrates a method for decoding video sequences using a composite image (of motion and composition) generated according to the method of the invention.

[0191] The composite image I(Pd n) as previously described, in particular in support of the, comprises for each position (xn, yn) two motion vectors MV1 (mv1x, mv1y) and MV2 (mv2x, mv2y). As previously described in support of figures 10 and 11, each motion vector is presented at the input of a motion compensation module, or warping (WARP1, WARP2). The module WARP1 (resp. WARP2) uses as input one or more reference images IREF1 (resp. IREF2) and the decoded motion information MV1 (resp. MV2) to perform motion compensation, or warping, on the reference image(s) in order to obtain one or more prediction image(s), or compensated image(s), denoted ICOMP1 (resp. ICOMP2). The composite image I(Pd n) as described previously, in particular in support of the, also includes for each position (xn, yn) composition information in a map C2, including for example a real number c2 between 0 and 1. The compensated images and this composition data c2 are presented as input to a mixer module noted MERGE. The MERGE module generates as output a reconstructed image IREC' of which each of the N pixels PIX'(n) can be obtained by a calculation of the type: PIX'(n) = c2 * ICOMP1(n) + (1- c2)* ICOMP2(n). By using this mechanism, it is possible to simply define compensated image zones benefiting from a weighted prediction of the images ICOMP1 and ICOMP2. This mechanism can prove very useful for managing for example transition zones or a change in brightness between two images in a sequence.

[0192] It is assumed here that the digital image of the video sequence is of the same size as the composite image, i.e. it has N pixels. Naturally, depending on the variants, it can be of a different size, and in particular larger than N. In this case, the motion vectors and the composition coefficient are used for groups of pixels, or blocks, of the image.

[0193] Alternatively, the composition information may further contain data relating to the type or number of reference image to be used.

[0194] Alternatively, this mode can be combined with that of the. We can then obtain a prediction of the type: PIX'(n) = IREF_INTRA / 2+ c2 / 2 * ICOMP1(n) + (1- c2) / 2* ICOMP2(n), or PIX'(n) = c1* IREF_INTRA+ (1- c1)* c2 * ICOMP1(n) + (1- c1)*(1- c2) ICOMP2(n), etc.

Claims

Method of coding a signal (I(P n )) comprising a plurality of samples (P n ) comprising movement information, the method comprising the following steps:- construction (E21, E22) of a first group of characteristic maps (FM i );- for at least one sample, called the current sample (P n ), of the signal, associated with a position (x n , y n ) in the signal to be coded: - construction (E24, E25) of a characteristic vector (Z n ) from said feature maps (FM i ) of said first group;- processing (E25) of said characteristic vector (Z n ) by an artificial neural network (MLP) defined by a set of parameters (W k ), to provide a vector representative of a value (P' n) of the current sample;- updating (E22, E28) of at least one value of one of said characteristic maps of said first group and / or of at least one parameter, as a function of a coding performance measurement,- coding (E23, E27, E29) of said first group of characteristic maps (FM i ) and said set of parameters (W k ). Method for decoding a signal comprising a plurality of samples (Pd n ) comprising movement information, the method comprising the following steps:- decoding (E31) of a first group of characteristic maps (FMd i ) ;- decoding (E35) of a set of parameters (Wd k ) representative of a neural network (MLP');- for at least one sample, called the current sample (Pd n ), of the signal, associated with a position (x n , y n ) in the signal to be decoded: - construction (E33, E34) of a characteristic vector (Zd n) from the characteristic maps of said first group (FMd i ), and:- processing (E36) of said characteristic vector (Zd n ) by an artificial neural network (MLP') defined by the decoded parameters (Wd k ) to provide a vector representative of a decoded value of the current sample (Pd n ). Coding or decoding method according to one of the preceding claims, characterized in that it comprises a step of constructing (E32) a second group of characteristic maps (FME' l ) and in that the characteristic vector is further constructed from said characteristic maps of the second group. Coding or decoding method according to one of the preceding claims, characterized in that at least one of said characteristic maps (FMd i , FME’l) est de résolution inférieure à celle du signal. Coding or decoding method according to one of the preceding claims, characterized in that at least one of said characteristic maps (FMd i , FME' l ) has the same resolution as the signal. Coding or decoding method according to one of the preceding claims, characterized in that the characteristic vector (Z n ) is constructed as a function of said position (x n , y n ) of said current sample (P n ). Coding or decoding method according to one of the preceding claims, characterized in that the construction of said vector (Zd n ) characteristic comprises a sub-step of extracting a value from said at least one characteristic map (FMd i, FME' i ) at an identical position (x n , y n ) to that of the current sample (Pd n ) in the signal. Coding or decoding method according to one of the preceding claims, characterized in that the construction of said vector (Zd n ) characteristic comprises a sub-step of extracting at least one value from said at least one characteristic map (FMd i, FME' i ) at a neighboring position (x n , y n ) of that of the current sample (Pd n ) in the signal. Coding or decoding method according to one of the preceding claims, characterized in that the construction of said characteristic vector (Zd n ) comprises a sub-step of processing (E33, E34) said extracted values ​​to obtain the characteristic vector. Coding or decoding method according to one of the preceding claims, characterized in that the coding or decoding (E31) of said first group of characteristic maps comprises an entropic coding or decoding sub-step. Coding or decoding method according to one of the preceding claims, characterized in that the motion information is a motion vector (mvx, mvy). Coding or decoding method according to one of the preceding claims, characterized in that a sample comprises at least one composition data (C, c1, c2). Device for coding a signal (I(P n )) of a plurality of samples (P n ) to be coded comprising movement information characterized in that said device is configured to implement:- construction (GEN, MAJ) of a first group of characteristic maps (FM i ) ;- for at least one sample, called the current sample (P n ), of the signal to be coded, associated with a position (x n , y n ) in the signal to be coded: - construction (XTR, TT) of a characteristic vector (Z n ) from said feature maps (FM i) of said first group, and;- processing (MLP) of said characteristic vector (Z n ) by an artificial neural network (MLP) defined by a set of parameters (W k ), to provide a vector representative of a decoded value (P' n ) of the current sample;- updating (MAJ, NNC) of at least one value of one of said characteristic maps of said first group and / or of at least one parameter of said network, as a function of a coding performance measurement,- coding (FMC, NNC) of said first group of characteristic maps (FM i ) and said set of parameters (W k ). Device for decoding a signal of a plurality of samples (Pd n ) to be decoded comprising motion information characterized in that said device is configured to implement:- decoding (FMD) of a first group of characteristic maps (FMd i ) ;- decoding (NND) of a set of parameters (Wdk ) representative of a neural network (MLP');- for at least one sample, called the current sample (Pd n ), of the signal to be decoded, associated with a position (x n , y n ) in the signal to be decoded: - construction (XTR', TT') of a characteristic vector (Zd n ) from the characteristic maps of said first group (FMd i ), and:- processing (MLP') of said characteristic vector (Zd n ) by an artificial neural network (MLP') defined by the decoded parameters (Wd k ) to provide a vector representative of a decoded value of the current sample (Pd n ). Computer program comprising instructions for carrying out the steps of an encoding or decoding method according to claim 1 or 2 when said program is executed by a computer.

Images