Method and device for encoding and decoding images.

Abstract

Method and device for encoding and decoding images. The invention relates to a method and device for encoding and decoding a signal from a plurality of samples (Pdn) comprising motion information. The decoding method comprises the following steps: - decoding (E31, FMD) of a first group of feature maps (FMdi); - decoding (E35, NND) of a set of parameters (Wdk) representative of a neural network (MLP'); - for at least one sample, called current sample (Pdn), of the signal to be decoded, associated with a position (xn, yn) in the signal to be decoded: - construction (E33, E34, XTR', TT') of a characteristic vector (Zdn) from the characteristic maps of said first group (FMdi), and: - processing (E36, MLP') of said characteristic vector (Zdn) by an artificial neural network (MLP') defined by the decoded parameters (Wdk) to provide a vector representative of a decoded value of the current sample (Pdn).Figure for the summary: Fig. 2.

Description

Title of the invention: Method and device for coding and decoding images. Prior art

[0001] The invention relates to the general field of coding single or multidimensional signals. It relates more particularly to the compression of digital images and video.

[0002] Digital images are generally subject to source coding aimed at compressing them in order to limit the resources required for their transmission and / or storage. There are numerous coding standards, such as, for still images, the JPEG family standards, and for animated or video images, the ITU / MPEG standards (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) as well as their extensions (MVC, SVC, 3D-HEVC, etc.)

[0003] The encoding of an image is generally carried out by dividing the image into several rectangular blocks, and by encoding these blocks of pixels according to a given processing sequence. In existing video compression techniques, the processing of a block typically comprises a prediction of the pixels of the block carried out using previously coded then decoded pixels present in the image being encoded, in which case we speak of “Intra prediction”, or of previously coded images, in which case we speak of “Inter prediction”. This exploitation of spatial and / or temporal redundancies makes it possible to avoid transmitting or storing the value of the pixels of each block of pixels, by representing at least some of the blocks by a residual representing a difference between the prediction values ​​of the pixels of the block and the actual values ​​of the pixels of the predicted block.

[0004] Inter prediction is generally done using a motion field. In conventional schemes, this motion field takes the form of a motion vector estimated per block of the image to be coded, and transmitted to the decoder.

[0005] In addition to these classic approaches proposed by compression standards (JPEG, MPEG, ITU), approaches based on artificial intelligence, and in particular neural ones, tend to develop.

[0006] Some of these neural approaches can be seen as a simple extension of the notion of competition of the aforementioned compression techniques, such as the prediction and transformation mode competition in video coding.

[0007] Other approaches use the concept of "auto-encoder". Auto-encoders are learning algorithms based on artificial neural networks, which allow a new representation of a data set to be constructed. The architecture of an autoencoder consists of two parts: the encoder and the decoder. The encoder consists of a set of neural layers, which process the data in order to construct new representations called “encoded”, also called “latent representations”. In turn, the decoder’s neural layers receive these representations and filter them in order to try to reconstruct the initial data. The differences between the reconstructed data and the initial data make it possible to measure the error made by the autoencoder. Training consists of modifying the autoencoder’s parameters in order to reduce the reconstruction error measured on the different samples of the dataset. The performance of such autoencoder-based systems comes at the cost of a considerable increase in memory footprint and complexity compared to conventional approaches such as those proposed by compression standards.They can have millions of parameters and may require up to a million MACs (multiply-accumulate) to decode a pixel. This makes such decoders far more complex than conventional decoders, which could hamper the adoption of learning-based compression.

[0008] More recently, a simple technique for encoding an image based on a neural network has been described in the article "Compression with Implicit Neural representations" by Emilien Dupont et al. (arXiv:2103.03123). The proposed coding technique consists of fitting a neural network to this image, quantifying the network weights and transmitting them. At the time of decoding, the neural network represents the image on its own. It is evaluated at each pixel position to reconstruct the image. The R, G, B color pixels for each of the image coordinates (i,j) are obtained by a function F, modeled by the neural network: R,G,B=f(i,j). However, such a technique remains inefficient in terms of compression and only applies to still images.

[0009] There are methods to evaluate the motion field using a neural network. For example, the article "AnyFlow: Arbitrary Scale Optical Flow with Implicit Neural Representation, Hyunyoung Jung, Zhuo Hui, Lei Luo, Haitao Yang, Feng Liu, Sungjoo Yoo, Rakesh Ranjan, Denis Demandolx (https: / / arxiv.org / abs / 2303.16493) proposes a neural method for estimating dense motion fields, or optical flow. This learning-based technique requires a large number of parameters (about 5 million) and is complex in number of operations per pixel, as is the auto-encoder approach. In addition, the optical flows obtained are not suitable for their low-bitrate transmission. These two aspects do not allow its application to low-complexity video coding.

[0010] In the field of coding, the article "AIVC: Artificial Intelligence based Vi-deoCodec, Théo Ladune, Pierrick Philippe", ICIP 2022, Oct 2022) proposes a video coding method that involves two autoencoders, one of which is used to encode information representative of the movement. However, this autoencoder is very complex and involves several tens of millions of parameters.

[0011] There is therefore a need for a solution making it possible to generate a dense motion field and associated parameters in order to encode or decode an image in a simple and efficient manner. Statement of the invention

[0012] The invention relates to a method for coding a signal comprising a plurality of samples comprising motion information, the method comprising the following steps: - construction of a first group of characteristic maps; - for at least one sample, called current sample, of the signal, associated with a position in the signal to be coded: - construction of a characteristic vector from said characteristic maps of said first group; - processing said characteristic vector by an artificial neural network defined by a set of parameters, to provide a vector representative of a decoded value of the current sample; - updating at least one value of one of said feature maps of said first group and / or at least one parameter, as a function of a coding performance measurement, - coding said first group of feature maps and said set of parameters.

[0013] The invention also relates to a method for decoding a signal comprising a plurality of samples comprising motion information, the method comprising the following steps: - decoding of a first group of characteristic maps; - decoding of a set of parameters representative of a neural network; - for at least one sample, called current sample, of the sign: - construction of a characteristic vector from the characteristic maps of said first group, and: - processing of said characteristic vector by an artificial neural network defined by the decoded parameters to provide a vector representative of a decoded value of the current sample.

[0014] For the purposes of the invention, encoding, or “coding”, means the operation which consists of representing a set of samples in a compact form carried for example by a digital binary train. Decoding means the operation which consists of processing a digital bitstream to restore decoded samples.

[0015] By "sample" of the signal, we mean a value taken from the signal. Sampling the signal produces a sequence of discrete values ​​called samples. A sample has at least one associated motion information. It can have other information. It is represented by a vector of dimension A, that is, comprising A values. A sample can be associated with a position in an image (pixel, rectangle, etc.).

[0016] By "motion information" is meant any representation of the motion that can be associated with a sample, for example a motion vector, or several motion vectors, or a set of equations defining a motion.

[0017] By "signal comprising a plurality of samples" is meant a signal with two (image, composite image) or more than two (stereoscopic, multiscopic image, image associated with a depth map, video, etc.) dimensions. Depending on this dimensionality, the sample has two or more coordinates in the signal. In the case of a signal represented by a two-dimensional image, the position of the sample is commonly identified by its abscissa (x) and ordinate (y) coordinates. In the case of a signal represented by a three-dimensional image, the position of the sample is usually identified by its abscissa (x), ordinate (y), and depth (z) coordinates, etc.

[0018] By "feature maps" is meant an abstract representation of the signal comprising a plurality of variable scalar data, discrete or not, which are also called characteristic values, for example real or integer numbers, signed or not. They can also comprise vectors or scalar values. In a known manner, these maps are also referred to as "latent representation" of the input signal.

[0019] By "data characteristic vector constructed from the characteristic maps" is meant a vector consisting of one or more elements, or data, preferably discrete, the data being constructed from the characteristic maps. This characteristic vector is the one that is applied to the input of the neural network. In the case of an image, such a vector can be constituted from a plurality of values ​​taken from each of the characteristic maps at the same abscissa and ordinate coordinates as the sample to be coded. The vector can also comprise values ​​taken from each of the characteristic maps at coordinates close to the sample to be coded (for example immediately above, to the right, etc.).It can also be made from a plurality of values ​​taken from each of the feature maps at a position that depends not only on the sample position but also on the size of the feature map. For example, in a lower resolution feature map of a . factor two of the image to be reconstructed, the corresponding characteristic value can be taken from the abscissa and ordinate coordinates divided by two. Once these values ​​are taken from the feature maps, they can be processed to form the feature vector, before input into a neural network, a quantization function, filtering, interpolation, etc.

[0020] By "neural network" is meant a neural network such as a convolutional neural network, a multilayer perceptron, an LS TM (for "Long Short Terni Memory" in English), etc. The neural network is defined for example by a plurality of layers of artificial neurons which comprise a set of activation, weighting and addition functions (for example, a layer can calculate y = f(Ax+b), where y and b are vectors of dimension N, x is a vector of dimension M, A is a matrix of dimension MxN, and f is the activation function). Hereinafter we speak of "neural network" or "synthetic neural network".

[0021] By "parameter of the neural network" is meant one of the values ​​which characterizes the neural network, for example a weight associated with one of the neurons (filter coefficient, weighting, bias, value affecting the operation of the non-linearity), etc.

[0022] By "processing by a neural network" is meant the application of a function expressed by a neural network to the input characteristic vector to produce an output vector representative of the sample to be coded (resp. to be decoded). This output vector comprises one or more data representative of the sample. It can for example represent a motion vector. It can also represent a motion vector and associated composition information.

[0023] By "performance measurement" is meant a measurement between at least one value of a sample to be coded and a decoded value of said sample. The measurement can evaluate for example a distortion, or a perceptual error. It can be carried out on a sample or a plurality of samples (for example, the complete image). The measurement can also include a measurement of the flow rate, in particular associated with the coding of the neural network and / or the coding of the characteristic maps. The measurement can be a joint measurement between the flow rate and the distortion through their weighting. As is well known in the state of the art, the value of this measurement is generally minimized until a target value is reached.

[0024] Generally speaking, it is considered that the steps of a coding or decoding method should not be interpreted as being linked to a notion of temporal succession. In other words, the steps may be carried out in a different order than that indicated in the independent coding or decoding claim, or even in parallel.

[0025] The coding method according to the invention carries out a construction of the coding parameters, from the input signal, for example a motion image, or composite comprising motion and composition information, by training a neural network on feature vectors associated with the position of a sample to be encoded. These feature vectors are constructed from feature maps. During training, or construction, the neural network parameters and the feature map values ​​are updated based on a performance measure, for example of the rate-distortion type. When training is complete, i.e. the performance measure obtained is satisfactory, the actual encoding of the neural network parameters and / or the feature map values ​​can be carried out and stored or transmitted to the decoder.

[0026] Advantageously, the training process makes it possible to refine the parameters of the neural network and / or the values ​​of the feature maps until an adequate representation in terms of performance is obtained, for example a desired balance between the generated flow rate and the distortion undergone by the input signal. The training of the values ​​of the feature maps and the parameters of the neural network can be joint. The training aims to construct the parameters representative of the image, before their actual coding. Advantageously, the coding method according to the invention makes it possible to efficiently compress the signal.

[0027] Advantageously, the decoding method is simple since it is sufficient to decode the characteristic maps and the neural network to reconstruct a decoded version of the signal, for example a motion or composition map.

[0028] Such a neural network can advantageously be of very simple structure with few parameters. They can thus be conveyed at low speed and adapted to the particular signal.

[0029] The decoded signal comprising motion information per pixel can be used in itself, for example to perform an analysis of a motion field, for a sports medicine application for example, or for a trajectory tracking application, etc. It can also be used, as is well known, in a video decoder, in order to exploit the temporal redundancy between the images of a sequence.

[0030] Standard video coders use motion estimation (at the encoder) and motion compensation (at the decoder) for inter-frame prediction. Motion vectors indicate how the pixels of the reference frame should be shifted in order to obtain a good prediction for the pixels of the frame being encoded / decoded. Motion compensation is often performed on blocks, which allows a reduced number of motion vectors to be transmitted. However, such compensation is inaccurate and produces block artifacts. Using per-pixel information improves this prediction.

[0031] According to embodiments of the coding and / or decoding method:

[0032] - The method comprises a step of constructing a second group of cards of ca characteristics and the characteristic vector is further constructed from said characteristic maps. Advantageously, these additional maps of the second group, constructed identically to the coder and the decoder, are neither stored nor transmitted at the coder level, nor decoded at the decoder level. They thus make it possible to benefit from additional data to improve compression without degrading the flow rate. They may for example include coordinates, data available in the maps of the first group, data concerning other images already processed by the coder or decoder, etc. Unlike the maps of the first group, they are not transmitted by the coder, and are not decoded by the decoder. These maps of the second group are directly deduced from the maps of the first group.

[0033] - At least one of the feature maps is of lower resolution than that of the signal to be encoded (resp. decoded). Advantageously, in this mode, the compression of the feature maps is more efficient since at least one of the feature maps contains fewer values ​​than if it were at the resolution of the signal. For example, in the case of a digital optical flow image, one of the feature maps can be at 1 / 2 resolution, that is to say it contains half as many values ​​on the abscissa and ordinate as the input signal contains samples, i.e. in total 4 times fewer values ​​than a feature map at the resolution of the signal.

[0034] - At least one of said characteristic maps is of identical resolution to that of the signal to be coded (resp. decoded). Advantageously, according to this mode, at least one of the characteristic maps, at the same resolution as the input signal (resp. to be decoded), allows significant fidelity and respect for the details of the initial resolution of the signal.

[0035] - The characteristic vector is constructed as a function of said position of said current sample.

[0036] - The construction of said characteristic vector comprises an extraction sub-step of a value of said at least one feature map at a position identical to that of the current sample in the signal.

[0037] Advantageously, it is thus possible to extract a value from a characteristic map of the first or second group, at a position identical to that of the sample in the signal (input signal for coding, signal to be decoded for decoding), to constitute an element of the characteristic vector. This method is simple to implement, in particular via an MLP type network. For example, if J characteristic maps are available as input, with the same resolution as the signal, a simple extraction of the values ​​of the maps at the coordinates of the current sample (at the same abscissa and the same coordinate in the feature map) allows to directly construct the feature vector of J elements. Moreover, in this case, the decoding can be carried out sample by sample, in a progressive manner.

[0038] - The construction of said characteristic vector comprises an extraction sub-step of at least one value of said at least one feature map at a position close to that of the current sample in the signal.

[0039] Advantageously, it is thus possible to extract values ​​from a characteristic map of the first or second group, at a position close to that of the sample in the signal (input signal for coding, signal to be decoded for decoding), to constitute one or more elements of the characteristic vector. This method is simple to implement, and is particularly suitable in the case of a network using a convolution layer. By neighbor, we mean in a close neighborhood in the sense of a distance measurement (for example, a neighborhood of 3 samples, for example a neighborhood covering the surface of a convolution kernel, etc.)

[0040] - The construction of said characteristic vector comprises a processing sub-step of said extracted values ​​to obtain the characteristic vector.

[0041] Advantageously, it is possible to process the values ​​extracted from the characteristic maps to construct the characteristic vector. Such processing may for example correspond to a quantification of the extracted data, or to a scaling, or to a filtering, etc. During coding, quantification is essential for the proper functioning of the system if the characteristic maps include, for example, floating point or real values. It is necessary to quantify them before coding them and / or providing them as input to the neural network. During decoding, however, inverse quantization is not necessary, depending on the embodiments. This is the case in particular when the quantization step is reduced to the simple approximation of a value.

[0042] - The coding (resp. decoding) of said first group of characteristic maps includes an entropy coding (resp. decoding) sub-step. Advantageously, prediction and entropy coding make it possible to exploit statistical redundancies in the signal that may persist in the feature maps, thus compressing the signal more efficiently.

[0043] - A sample of the signal comprises at least one composition information

[0044] Composition information comprises one or more data making it possible to indicate how to use the motion information, this way being able to comprise, in a non-limiting manner: a fusion of a compensated signal with another signal, or a masking of the areas of the signal, or even a weighting of the importance of the signal. Composition information can be used profitably to indicate how to use the motion information, in particular when several reference images are used to carry out a prediction by motion compensation.By "motion compensation" is meant the generation of a predicted, or motion-compensated, image using a reference image and motion information. Such composition information may indicate a mixing factor allowing weighting between the different reference images used, after their motion compensation for example. The composition information contained in the decoded samples may thus be used to improve the compensation of the image by taking into account its content, in particular occlusion zones, or static zones, or high-motion zones of the image, as is well known to those skilled in the art, for specific processing of these zones. For example, if an area is occluded, i.e. it does not exist in the reference images, it may be ignored, or even predicted using an Intra-type prediction.In another example, a combination of several motion-compensated images can be achieved by using a weighting of the different areas of the compensated images. For example, an occluded area will only be visible in a reference image and the weighting indicates that decoding will be performed based only on the reference image that contains the occluded area. This improves image prediction. Thus, the composition information can include data indicating a weighting factor, a masking factor, a reference image, a type of reference to be used, etc. A sample or image containing composition information is called a composite.

[0045] - The motion information comprises at least one motion vector.

[0046] In the case of a signal comprising motion information, the sample can be traditionally represented by a motion vector, for example a pair of values ​​indicating a horizontal and vertical movement of a pixel or a group of pixels of a digital image. In the case of motion information per sample, we speak of a dense motion field, or optical flow. Generally, a sample is represented by a vector of one or more components, or data. The motion vector can also be represented by an angle and an amplitude, etc. The motion values ​​can represent movements on integers, or floating points to signify precise displacements.

[0047] - The coding (resp. decoding) of the signal is followed by a compensation step of at least least one reference image using the motion information, to obtain a motion-compensated image. Advantageously, the motion information associated with the decoded samples can be used, as is well known to those skilled in the art, in a video decoder, to compensate for a reference image.

[0048] - The coding (resp. decoding) process is followed by a combination step from of at least one reference image using said motion-compensated image and said composition information, as set forth above and described in the remainder of the description.

[0049] Correlatively, the invention also relates to a coding device and a decoding device.

[0050] The characteristics and advantages of the coding or decoding method apply in the same way to the coding or decoding device according to the invention and vice versa.

[0051] The invention also relates to a computer program on a recording medium, this program being capable of being implemented in a computer or a coding or decoding device in accordance with the invention. This program comprises instructions adapted to the implementation of the corresponding method. This program can use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0052] The invention also relates to an information carrier or a recording medium readable by a computer, and comprising computer program instructions mentioned above. The information or recording carriers can be any entity or device capable of storing the programs. For example, the carriers can comprise a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a floppy disk or a hard disk, a DNA sequence, or a flash memory. Furthermore, the information or recording carriers can be transmissible media such as an electrical or optical signal, which can be conveyed via an electrical or optical cable, by radio link, by wireless optical link or by other means.

[0053] The program according to the invention can in particular be downloaded from an Internet-type network.

[0054] Alternatively, each information or recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of a method according to the invention. Brief description of the drawings

[0055] Other characteristics and advantages of the present invention will emerge from the description given below, with reference to the appended drawings which illustrate exemplary embodiments thereof which are not in any limiting nature.

[0056] [Fig-1] [Fig.l] schematically represents a coding device used in the scope of the invention;

[0057] [Fig.2] [Fig.2] schematically represents a decoding device used in the context of the invention;

[0058] [Fig.3] [Fig.3] illustrates an example of a synthetic artificial neural network used in the context of the invention for coding and decoding;

[0059] [Fig.4] [Fig.4] is a flowchart representing an example of a coding method which can be implemented by the coding device of [Fig.l];

[0060] [Fig.5] [Fig.5] illustrates an encoding method used in one embodiment of the invention;

[0061] [Fig.6] [Fig.6] illustrates an encoding method used in another embodiment of the invention;

[0062] [Fig.7] [Fig.7] is a flowchart representing an example of a decoding method which can be implemented by the decoding device of [Fig.2];

[0063] [Fig.8] [Fig.8] illustrates a decoding method used in one embodiment of the invention;

[0064] [Fig.9] [Fig.9] illustrates a decoding method used in another embodiment of the invention;

[0065] [Fig. 10] [Fig. 10] illustrates a method for decoding video sequences using a motion image generated according to one embodiment of the invention;

[0066] [Fig. 11] [Fig. 11] illustrates a method for decoding a video sequence using a composite image generated according to one embodiment of the invention;

[0067] [Fig. 12] [Fig. 12] illustrates a method for decoding a video sequence using a composite image generated according to another embodiment of the invention. Description of the embodiments

[0068] [Fig.l] schematically represents an ENC coding device.

[0069] This ENC coding device comprises a GEN module for generating characteristic maps, a TT processing and quantification module, an XTR data extraction module, an MLP module corresponding to an artificial neural network, an NNC neural network coding module, an FMC module for coding characteristic maps, an EVAL module for evaluating a performance function, an MAJ updating module, an optional FME module for generating additional characteristic maps.

[0070] The ENC coding device can be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the above-mentioned modules can then be realized by the cooperation of the processor and computer program instructions stored in the above-mentioned memory and designed to carry out the functionalities of the module concerned, including as described below, when such instructions are executed by the processor.

[0071] The coding device ENC of [Fig.l] receives as input a succession of samples to be coded, denoted Pn, for example a temporal succession of data corresponding to an image comprising motion data, denoted I(Pn). In this case, the image signal I(Pn) can represent a two-dimensional motion image, or a three-dimensional one corresponding to a plurality of motion images. Pn designates a sample n of the input signal comprising N samples. According to another example, the coding device ENC receives as input a succession of samples to be coded Pn, in the form of a temporal succession of data corresponding to a composite image I(Pn) comprising motion and composition information.

[0072] In one embodiment, the signal is an image comprising motion information represented by means of at least one two-dimensional representation, such as a matrix of pixels, of width W and height H (with N = WxH), each sample of the image conventionally corresponding to a motion vector comprising a horizontal (MVx) and vertical (MVy) component, or, alternatively, an amplitude component and an angle component. The location of each motion vector is defined by its abscissa and ordinate coordinates (x and y) in the motion image. According to other embodiments, the output motion vector may comprise more than two values ​​(for example 12 for a polynomial model, 4 for an affine model, 8 for a bilinear or homographic model, etc.).If the motion information concerns several images, for example if the motion information applies to several images then each sample of the image can refer to the relative movements of the different images. In the case where the motion information applies to two images, then the motion samples can comprise two motion vectors.

[0073] According to other embodiments, a sample further comprises composition information C. Such composition information may consist of K data (cl,...cK), where K is a natural integer, such as for example weighting, masking, fusion coefficients, or more generally data of any type making it possible to indicate how to apply the motion information (for example, the type of reference image to be used, the weighting to be carried out on a reference image, or the reference image number to be used, or a weighting to be applied to the motion vector, a motion correction, etc.).

[0074] In a particular embodiment, the signal is an image comprising an in- movement training including: - a displacement relative to a first two-component reference image (mvlx, mvly); - a displacement relative to a second reference image with two components (mv2x, mv2y), in a particular case mv2x is of opposite sign to mvlx and mv2y of opposite sign to mvly; - weighting information (cl) to be applied between the reference images after the application of the relative displacement, this weighting being able to take values ​​cl between 0 and 1 and applied by multiplication to the product of the first displacement and by multiplication by (1- cl) to the second product. - masking or mode information (c2), the value of which is between 0 and 1, which is intended to eliminate by multiplication unwanted areas resulting from the weighting operation or to indicate a particular processing mode for the composite image.

[0075] As will be described in more detail later with reference to Figures 3 to 12:

[0076] The GEN module for generating characteristic maps is configured to generate, and in particular initialize, a plurality of M characteristic maps denoted FM;. The optional FME module can generate one or more additional maps, L in number, which will be neither coded nor transmitted, denoted FMEi.

[0077] The XTR module performs an extraction of values ​​in the FM characteristic maps; and optionally FMEi, for a sample Pn to be coded.

[0078] The TT module performs a processing of the extracted values, to generate the vector Zn. Zn is a J-tuple, that is to say that it comprises J elements, or data z;. The vector Zn of index n refers to the characteristic vector of the pixel P'n.

[0079] The module TT performs a quantization of the data extracted from this set of M FM cards, or of the vector Zn formed from this data. To prepare for subsequent coding (for example of the entropic type), the coding system quantizes the latent representation of the input data using a quantizer Q to generate an ordered collection of quantized values. It is recalled that the quantization of a value refers to the mapping of this value to a member of a discrete set of possible code symbols. For example, the set of possible code symbols may consist of integer values, and the quantization system performs a simple rounding of a real value to an integer value. According to another example, the quantization consists of a multiplication by a given value and then a rounding.

[0080] The MLP module is a neural network defined by K parameters Wk, capable of processing the vector Zn, or J-Uplet, as input, to generate as output a vector representative of the sample Pn to be coded. The neural network can comprise according to a mode of rea lisation an MLP, or Multi Layer Perceptron, consisting of an input layer adapted to the input format (the J-uplet), optionally one or more hidden layer(s), and an output layer adapted to the output format of the output vector, generally a vector comprising A elements. According to one embodiment, the neural network may comprise a convolutional network, a convolutional network comprising, in a known manner, at least one convolution layer, which may be followed by non-linearities. According to one embodiment, A is equal to 2 and the output vector is the pair (mvx, mvy) of the motion vector P'n coded then decoded. According to one embodiment, as shown in [Fig.8], A is equal to 3 and the output vector is the triplet (mvx, mvy, cl) of the sample P'n coded then decoded, comprising a motion vector and composition information.According to one embodiment, A is equal to 5 and the output vector is the triplet (mvlx, mvly, mv2x, mv2y, cl) of the sample P'n coded then decoded, comprising two motion vectors and a composition data item cl. According to one embodiment, as shown in [Fig.9], A is equal to 6 and the output vector is the triplet (mvlx, mvly, mv2x, mv2y, cl, c2) of the sample P'n coded then decoded, comprising two motion vectors, and a composition information item C comprising a first composition data item cl indicating for example a weighting and a second composition data item c2 indicating for example a masking.

[0081] The NNC module performs the coding of the neural network, in particular of its parameters Wk. During the process of training, or construction, of the coding, that is to say as long as the step of evaluating a performance is not satisfactory, the NNC module performs a coding simulation, followed by a decoding, intended for the evaluation module. It updates the values ​​of the parameters according to the results of a performance measurement carried out by the EVAL module. Subsequently, it performs the actual coding of the parameters of the neural network Wk. The coded parameters are denoted Wck. In a known manner, the coding simulation can be identical to the actual coding, or produce an approximation thereof.

[0082] The FMC module performs the coding of the FMi maps, i.e. the quantified values ​​of each map of the first group (excluding the FMEi maps). During the process of training, or constructing, the coding, i.e. as long as the step of evaluating a performance is not satisfactory, the FMC module performs a coding simulation, followed by decoding, intended for the evaluation module. Subsequently, it performs the actual coding of the values ​​of the FM maps;. The coded maps are denoted FMc;. In a known manner, the coding simulation can be identical to the actual coding, or produce an approximation thereof. The coding module compresses the quantized data using, for example, entropic coding.

[0083] The EVAL module performs an evaluation and minimization of a performance of coding. The evaluation function is for example of the rate-distortion type. The minimization can be carried out by gradient descent, or any other method within the reach of those skilled in the art.

[0084] The MAJ module updates the values ​​of the FMi cards to be encoded based on the results of the performance function. Optionally, it can update the values ​​of the additional FMEi cards.

[0085] [Fig.2] schematically represents a DEC decoding device.

[0086] The DEC coding device of [Fig.2] receives as input a first group of encoded data organized into M FMc characteristic maps; (also called FM layers) and the encoded Wck parameters of the neural network.

[0087] This DEC decoding device comprises a neural network decoding module NND, a feature map decoding module FMD, a data extraction module XTR', an inverse processing module TT', a neural network corresponding MLP' module, an optional FME' module for generating additional feature maps.

[0088] The maps decoded by the FMD module, numbering M, are denoted FMd;. The parameters decoded by the NND module are denoted Wdk.

[0089] The FME' module of the decoder can also generate one or more additional cards, denoted FME'i, L in number, identical to the additional FMEi cards generated by the encoder.

[0090] The XTR' module is identical to the XTR module of [Fig. 1]. It performs an extraction of values ​​from the M decoded feature maps FMd;, and possibly from the L additional maps FME'b to construct the vector Zn taking the form of a J-tuple for a sample Pn to be decoded, as a function of its coordinates in the signal to be decoded. In one embodiment, J comprises a number of samples depending on M, L and the topology of a convolutional neural network.

[0091] The TT' module optionally performs an inverse quantization corresponding to the quantization performed at the encoder. The inverse quantization is not necessary if the quantizer Q of the encoder has performed a simple rounding of the real values ​​submitted to it. The inverse quantization is not necessary if the neural network is capable of taking into account a quantization of its input data. Otherwise, the decoder performs the inverse operation of the quantizer Q. Furthermore, like the TT module of the encoder, the TT' module optionally performs a processing of the values ​​extracted from the feature maps, to generate the decoded vector Zdn.

[0092] The MLP' module is a neural network defined by K parameters Wdk, capable of processing the vector Zdn, or J-Uplet, as input, to generate as output a vector representative of the sample Pn to be decoded, generally a vector comprising A elements. According to one embodiment, A=2 and the output vector is the pair (mvx, mvy) of the decoded motion vector Pdn. According to one embodiment, A is equal to 3 and the output vector is the triplet (mvx, mvy, cl) of the coded then decoded sample P'n, comprising a motion vector and composition information C comprising a data item cl. As described for the encoder, different embodiments are possible. According to one embodiment, A is equal to 5 and the output vector is the quintuplet (mvlx, mvly, mv2x, mv2y, cl) of the coded then decoded sample P'n, comprising two motion vectors and a data item c 1 indicating a weighting. According to one embodiment, A is equal to 6 and the output vector is the triplet (mvlx, mvly, mv2x, mv2y, cl, c2) of the sample P'n coded then decoded, comprising two motion vectors, composition information comprising a weighting data item cl and a masking data item c2.The MLP' module is identical in structure to the MLP module, and its parameters are either identical if the coding of its Wk parameters is lossless, or different if the coding is lossy.

[0093] When all the samples Pndu signal have been decoded, a reconstructed signal I(Pdn) is available, according to one embodiment an image comprising N samples decoded in the form of N vectors Pdnd of dimension A. Equivalently, it can be said that the image comprises A components, each component comprising a value of the output vector, for example the reconstructed image comprises a component for the horizontal motion vectors (MVx), a component for the vertical motion vectors (MVy), K components (CL..CK) for the composition information C(= cl...cK)

[0094] The decoding device DEC can be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the modules mentioned above can then be realized by the cooperation of the processor and computer program instructions stored in the aforementioned memory and designed to carry out the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0095] [Fig.3] illustrates an example of a synthetic artificial neural network used for coding and decoding in the context of embodiments of the invention applied to motion or composite images consisting of motion vectors and optionally composition information.

[0096] The MLP synthesis artificial neural network for coding and the MLP' synthesis artificial neural network for decoding are defined by an identical structure, comprising for example a plurality of layers of artificial neurons, and by a set of weights and activation functions associated respectively with the artificial neurons of the network concerned.

[0097] A vector representation of a current sample and optionally of its neighborhood (a vector Zn or Zdn from the FM feature maps; and FMEi or FMd; and FME'i) is applied as input (i.e. on an input layer) of the MLP or MLP' synthesis artificial neural network. The artificial neural network produces as output a vector P'n or Pdn representative of the decoded sample, according to one embodiment the motion components (mvx, mvy) constituting a motion vector of a motion image. According to a variant shown in dotted lines, the output vector also includes a composition data cl (C={cl})

[0098] The concatenation of all these reconstructed pixels in a motion or composite image constitutes the decoded, or reconstructed, image.

[0099] At the encoder, the MLP synthesis artificial neural network is trained on the image, so as to minimize the differences between the input representation of the current motion or composite image (Pn) and its output representation I(P'n), while also minimizing the amount of data to be encoded. The EVAL module performs a performance measurement in this sense.

[0100] Once training is complete, the network parameters are encoded, either lossless, in which case the MLP' neural network is identical to MLP, or lossy, in which case the MLP' network may be slightly different from MLP.

[0101] [Fig.4] is a flowchart representing an example of a coding method that can be implemented by the coding device of [Fig.l].

[0102] According to this embodiment, the signal is a composite two-dimensional image, each sample to be coded Pn of coordinates (xn, yn) comprises a motion vector and composition information C.

[0103] Encoding takes place in two main phases:

[0104] In a first phase, called the training phase, learning is carried out in order to determine, for an input signal I(Pn), the values ​​of the FM maps; and the parameters Wk to optimize a global cost function. The learning is for example carried out by a gradient descent, followed by an update of the parameters of the MLP neural network and the values ​​of the FM characteristic maps;. As is known in the state of the art, the cost function can be of the rate-distortion, or rate, or distortion, or perceptual type. To measure the rate R, it is necessary to simulate the coding of the FM maps;, then to measure the associated coding rate (the size of the flow B1). According to one embodiment, the coding of the parameters Wk is not simulated because their influence is less important than that of the characteristic maps. According to one embodiment, the coding of the parameters Wk is also simulated and the associated rate is measured (the size of the flow B2).To measure the distortion D, it is necessary to simulate the coding then the decoding of at least part of the image I, to obtain at least one motion and composition vector P'n resulting from a simulation of coding then decoding, then to measure the difference between this part of . the composite image I(Pn) as input and a corresponding part of the composite image I(P'n) encoded then decoded.

[0105] Then during a second phase, called the coding phase, the FM maps; and the Wk parameters are encoded to produce the coded values ​​FMc; and Wck before transmission or storage. They constitute the compressed representation of the input signal I (Pn).

[0106] We will now describe the steps of a method according to one embodiment of the invention.

[0107] During a step E20, a signal I(Pn) to be coded, comprising a plurality of N samples Pn, is provided as input to the method.

[0108] During a step E21, the M FM cards; of the first group are initialized by the GEN module. Subsequently, the parameters Wk of the MLP neural network and the values ​​of the FM cards; must be optimized during the construction phase.

[0109] According to one embodiment, the FM cards; have the same resolution as the input signal I(Pn) and therefore each comprise the same number of values ​​N as there are samples Pn to be coded.

[0110] According to one embodiment, the FM cards; have a resolution less than or equal to that of the input signal I(Pn) and therefore comprise, for at least one of them, a number N' of values ​​to be coded less than N.

[0111] According to one embodiment, the first FM map is at the resolution of the signal and each subsequent map is at a resolution half that of the previous one.

[0112] According to one embodiment, several FM cards have the same resolution, lower than the resolution of the signal.

[0113] According to one embodiment, the FM cards are initialized by predefined constant values.

[0114] According to another embodiment, the characteristic maps are initialized by a set of random real values.

[0115] According to one embodiment, one or more FMEi maps, forming a second group of L additional characteristic maps, are generated and added to the first group. They are used to construct the characteristic vector but will not be stored or transmitted.

[0116] These FM characteristic maps; of the first group are subsequently updated, or refined, during a step E22 by the update module MAJ of the encoder during its learning during the construction phase.

[0117] During a step E23, the FM cards of the first group are coded by the FMC module of the encoder. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is an actual coding and the coded values ​​constitute the flow B1. The simulation can be identical to the actual coding but It can also be different (for example, simplified). For this encoding, any known technique for compressing card values ​​can be used.

[0118] In one embodiment, the FM cards are coded in the order (FMb FM2,..., FM4), and the variables of each card in a predefined order, for example lexico-graphic. Each card undergoes entropic coding. The entropic coding produces a compressed stream B1 whose flow rate is subsequently measured during a step E28.

[0119] During a step E24, values ​​are extracted by the XTR module from the FM cards; and possibly FMEi. This extraction can be carried out as a function of the coordinates (xn, yn) of the sample Pn of the input signal.

[0120] According to one embodiment, the coordinate values ​​(xn, yn) of each map are extracted.

[0121] According to one embodiment, the values ​​adjacent to the coordinates (xn, yn) of each map are also extracted.

[0122] According to one embodiment, the coordinate values ​​are scaled to extract the lower resolution layers.

[0123] According to one embodiment, values ​​close to those previously cited are also extracted for the subsequent processing of step E25, for example for a convolution kernel.

[0124] During a step E25, a vector Zn is constructed by the module TT from the values ​​extracted from the FM maps; and possibly FMEi for each sample Pn of coordinates (xn, yn) of the input signal. The processing includes a quantification of the values ​​extracted from the FM maps;, if necessary. The processing may include other operations, such as filtering, scaling, the application of any function, preferably monotonic, etc.

[0125] In one embodiment, Zn comprises as many values ​​as there are input cards or FMi cards (and optionally FMEi).

[0126] In one embodiment, Zn is a J-uplet (zb z2,..., Zj), consisting of the values ​​of the FM maps; (and optionally FMEi) located at the coordinates (xn, yn) of the current pixel Pn, as will be illustrated in support of [Fig.5]. The value located at the coordinates (xn, yn) of the FM map; gives the final value (z;) of the element i of the vector Zn relative to this FM map; or FMEb In this case the TT module does not apply any processing other than quantization.

[0127] In one embodiment, Zn is a J-tuple constructed from values ​​taken from the FMi (and optionally FMEi) maps at coordinates that may be different depending on the maps. For example, if the FM; (and / or FMEi) maps are at different resolutions because they have been downsampled, the coordinates are adapted (by scaling) to match the resolution of each map, as will be illustrated in support of [Fig.6].

[0128] In one embodiment, Zn is a J-uplet constructed from values ​​taken from the FM maps; (and FMEi) by applying processing to one or more values ​​of the maps, for example filtering the values ​​close to the targeted value in a map. For example, in an FMi map which is at the same resolution as the input signal, it is possible to extract the values ​​located at the coordinates (xn, yn), (xn-l, yn), (xn, yn -1) and (xn-l, yn-l) and apply processing to these values ​​(filtering, averaging, interpolation, application of a convolution network, etc.) to obtain the final value (z;) of the element i of the vector Zn relating to this FM map; or FMEi. According to another example, in an FM map; which is at half the resolution of the input signal, we can consider the values ​​located at the coordinates (xn / 2, yn / 2), (xn / 2-l, yn / 2), (xn / 2, yn / 2-l) and (xn / 2-l, yn / 2-1) and apply processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (¾) of element i of the vector Zn relative to this FM map; or FMEi. .

[0129] The samples to be coded are for example processed in sequential order, from n=1 to n=N.

[0130] During a step E26, the vector Zn is processed by the MLP neural network for generate as output a vector representative of the sample Pn to be coded, according to one embodiment the triplet (mvx, mvy,cl) of the sample P'n (the sample Pncoded then decoded), according to other embodiments an N-uplet consisting of one or more motion vectors and one or more composition data.

[0131] The structure and parameters Wk of the neural network are initialized for example during the first iteration of this step. These parameters are subsequently updated, or refined, during the construction phase, during subsequent iterations of the method.

[0132] According to one embodiment, the parameters of the neural network are initialized by predefined values ​​known to give a satisfactory result (for example, following training on a corpus of images).

[0133] According to another embodiment, the parameters Wk of the neural network are initialized by a set of random values.

[0134] During a step E27, the Wk parameters of the MLP neural network are quantified and coded. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is an actual coding and the coded values ​​constitute the stream B2. The simulation can be identical to the actual coding but it can also be different (for example, simplified). Any known technique can be used for this purpose, for example the neural network coding standard proposed by the MPEG-7 part 17 standard, also called NNR (Neural Network Representation). It should be noted that in this case, the amount of degradation that the coding brings to the Wk parameters must be chosen.

[0135] During a step E28, a performance measurement is evaluated.

[0136] For this purpose, the coding simulation rates associated with the characteristic maps (simulation of the B1 flow by coding and decoding of the FM maps;) and optionally with the parameters of the neural network (simulation of the B2 flow by coding and decoding of the Wk parameters) are measured.

[0137] According to one embodiment, the cost function is of the rate-distortion type, noted (D+L*R), where D is for example the quadratic error measured between the input signal and the decoded signal (or the error measured on a subset of samples of the signal). According to another example, D is calculated from the maximum deviation in absolute value between the coded and original values. According to one embodiment, R is the simulated rate of the stream B1; according to another embodiment, R is the total rate used to code this image, that is to say the sum of the simulated rates of B1 and B2. L is a parameter which regulates the rate-distortion trade-off. Other cost functions are possible.

[0138] According to another embodiment, the distortion D is measured not on the motion image itself, but on the result of the use of the motion vector. For example, if the motion field is used to perform motion compensation of an image, then the distortion will be measured on the compensated image. In this case the distortion measurement may comprise a perceptual measurement of the MS-SSIM, VMAF, etc. type as is well known to those skilled in the art.

[0139] As long as the cost function has not reached its minimum, the performance measurement is not satisfactory, and the method is repeated from step E22. Alternatively, a maximum number of iterations, or a maximum learning time can be defined. This minimization can be carried out by a mechanism known as gradient descent with updating of the parameters during step E22 for the values ​​of the feature maps and E26 for the network parameters.

[0140] During a step E29, if the cost function has reached its minimum, the training stops. If a coded version corresponding to the last simulation of the parameters of the neural network (Wk) and the characteristic maps (FM;) is available, the streams B1 and B2 can be constituted therefrom. According to another embodiment, the actual coding of the updated parameters of the neural network (Wk) and the values ​​of the characteristic maps (FM;) is carried out at this step to produce the encoded parameters Wck and FMc; which constitute streams B1 and B2.

[0141] The streams B1 and B2 can be concatenated to produce a final stream BS. According to one embodiment, the stream B2 of the coded parameters of the neural network is stored or transmitted before the stream B1, so that it can be decoded before the stream B1.

[0142] [Fig.5] illustrates a coding method used in an embodiment of the invention to code an image I(Pn) (not shown as input) of N samples corresponding to motion vectors (in which case we speak motion image) and optionally composition information (in which case we speak of a composite image).

[0143] In this embodiment, there are 4 FM cards. In a preferred embodiment, there are 7. The FM cards have the same resolution as the image I(Pn), and therefore each have WxH values, where W represents the width of the image in pixels and H its height (N=WxH).

[0144] According to a variant presented in dotted lines, the characteristic maps are 5 in number: an additional map FME0 has been introduced, which will neither be stored nor transmitted. This additional map typically contains data which can assist the MLP network in the task of signal reconstruction. Thus, the added maps can be one or more from the following non-limiting list:

[0145] - A map showing at each point the abscissa of that point.

[0146] - A map showing at each point the ordinate of that point.

[0147] - A map comprising at each point a positional coding (as described by example at the internet address https: / / skosmos.loterre.fr / P66 / fr / page / -K0D65X2X-X ).

[0148] - A card representing an image distinct from the image being processed, above capable of providing information about the image to be encoded, for example the previously processed image if the current image is part of a series of motion or composite images to be encoded such as a video, a set of medical images, a multi-view representation, etc.

[0149] - A map representing a feature map of a distinct image of the image being processed that may provide information about the image to be encoded, for example a previously processed map if the current image is part of a series of motion or composite images to be encoded such as a video, a set of medical images, a multi-view representation, etc.

[0150] - A card containing the value of an already decoded sample of the same card, for example example the previous sample in the decoding order.

[0151] In this embodiment, the vector Zn is a 4-tuple (zp..z4) consisting of the values ​​extracted from the FM maps; located at the coordinates (xn, yn) of the current sample Pn. The extracted values ​​are quantized by the TT module and the vector Zn consisting of the quantized values ​​is processed by the MLP neural network to generate an output vector as output.

[0152] In one embodiment, the 4 FM maps are quantized.

[0153] According to the variant presented in dotted lines, the vector Zn is a 5-tuple (z0...z4), the value z0 being extracted from the additional map FME0.

[0154] In this embodiment, the output vector is a pair (mvx, mvy) representative of the sample Pn to be coded. The output vector is in this case the pair (mvx, mvy) of the sample P'n coded then decoded. The pair is inserted into the decoded image I(P'n) at the positions (xn, yn) of the horizontal and vertical motion components (MVX1, MVY'). In the embodiment shown in dotted-dotted lines, the output vector is a triplet (mvx, mvy, cl) representative of the sample Pn to be encoded. The output vector is inserted into the decoded image I(P'n) at the positions (xn, yn) of its horizontal and vertical motion components (MVX1, MVY') and its composition component (Cl1).

[0155] [Fig.6] illustrates an encoding method used in another embodiment of the invention.

[0156] In this embodiment, there are 4 FM maps. As in the previous embodiment, the first FM map has the same resolution as the image I, and therefore has WxH values, where W represents the width of the image in pixels, and H its height. The second FM map has half the resolution (in each dimension) of the FM map. Each additional map has half the resolution of the previous map. This structure makes it possible to reduce the number of variables in the characteristic maps, which facilitates coding and learning while minimizing the coding cost, particularly in terms of throughput.

[0157] In this embodiment, a 4-tuple Zn is constructed from the values ​​of the FM maps; located respectively at the coordinates (xn, yn), (xn / 2, yn / 2), (xn / 4, yn / 4), (xn / 8, yn / 8) of the current pixel Pn(xn, yn). These extracted values ​​are processed and quantized to generate the 4-tuple Zn. According to one embodiment, for the FM2 map, the value located at the coordinates (xn / 2, yn / 2) is extracted to obtain the final value (z2) of the vector Zn relating to this FM2 map. According to another embodiment, for the FM2 map, the values ​​located at the coordinates (xn / 2, yn / 2), (xn / 2-1, yn / 2), (xn / 2, yn / 2-1), (xn / 2-1, yn / 2-1) are extracted and filtered to obtain the final value (z2) of the vector Zn relating to this FM2 map. Oversampling is carried out in this way. The filtering or oversampling can, for example, use bicubic filtering or Lanczos filtering, known per se.A more general processing can also be used: a neural network can be used to produce the final value (z2) by weighting the values ​​located at the coordinates (xn / 2, yn / 2), (xn / 2-1, yn / 2), (xn / 2, yn / 2-1), (xn / 2-1, yn / 2-1), for example an MLP or convolutional type network.

[0158] As previously, the vector Zn is processed by the MLP neural network to generate an output vector. According to one embodiment, the output vector is a pair (mvx, mvy) representative of the sample Pn to be coded. The output vector is in this case the pair (mvx, mvy) of the sample P'n coded then decoded. The pair is inserted into the decoded image I(P'n) at the positions (xn, yn) of the horizontal and vertical motion components (MVX1, MVY'). In the embodiment shown in dotted-dotted lines, the output vector is a triplet (mvx, mvy,cl) representative of the sample Pn to be encoded. The output vector is inserted into the decoded image I(P'n) at the positions (xn, yn) of its horizontal and vertical motion components (MVX1, MVY') and its composition component (Cl1).

[0159] Other types of structure are possible, for example one can use a reduction rate different from half between the maps (a quarter, or a third, etc.) The oversampling step is then adapted to produce the final value of the vector Zn.

[0160] As described in support of [Fig.5], other FMEi cards can be introduced, which will be used for the generation of the Zn vector but will not be stored or transmitted.

[0161] [Fig.7] is a flowchart representing an example of a decoding method that can be implemented by the decoding device of [Fig.2].

[0162] During a step E30, the streams B1 and B2 are extracted from the encoded stream BS. They contain respectively the coded representations of the maps of the first group FMCi and the parameters Wck.

[0163] During a step E31, the M FMd; cards are generated by decoding the FMCi values. For this decoding, any known technique similar to that used in the encoder can be used, preferably entropic decoding. In one embodiment, the FMd; cards are decoded in the order (FMdi, FMd2,... FMd4), and the variables of each card in a predefined order, for example lexicographic.

[0164] According to an embodiment as described for the encoder, the FMd; maps have the same resolution as the signal I (Pdn) to be reconstructed, that is to say they comprise N values.

[0165] According to an embodiment as described for the encoder, the FMd maps; are at a resolution less than or equal to that of the signal I (Pdn) to be reconstructed.

[0166] According to an embodiment as described for the encoder, several FMd cards have the same resolution, lower than the resolution of the signal.

[0167] During a step E32, according to one embodiment, one or more FME maps forming a second group of L additional feature maps are generated and complete the first group. They are not decoded but generated by the decoder in a manner identical to the generation of the encoder. They typically comprise data which can assist the MLP' network in the signal reconstruction task. The non-limiting list of possible additional feature maps described in support of [Fig.5] for the encoder applies here.

[0168] During a step E33, values ​​are extracted by the module XTR' from the cards FMd; and possibly FME'i. This extraction can be carried out as a function of the coordinates (xn, yn) of the input signal of the sample Pn of the signal to be reconstructed. This step is identical to step E24 which was described for the encoder with the support of [Fig.4] and the described embodiments apply.

[0169] The samples to be decoded are for example processed in sequential order, from n=1 to n=N.

[0170] During a step E34, a vector Zdn is constructed by the module TT' from the values ​​extracted from the maps FMd; and possibly FME'i for a sample Pn to be decoded with coordinates (xn, yn). The processing may include an inverse quantization of the extracted values ​​or of the vector Zdnconstituted, if necessary. The processing may include other operations, such as filtering, scaling, convolution, etc. This step is identical to step E25 which was described for the coder in support of [Fig.4] and the described embodiments apply. However, inverse quantization is not obligatory. In this particular case, step E34 and the module TT' are therefore optional.

[0171] During a step E35, the Wdk parameters of the MLP' neural network are generated by decoding the Wck values ​​of the stream B2. Any known decoding technique corresponding to the coding technique used by the encoder can be used for this purpose. The MLP' neural network is similar to the MLP network, i.e. it has the same structure and includes the same parameters, except for the coding, which can be carried out with or without losses.

[0172] According to one embodiment, the stream B2 is decoded before the stream B1, in order to be able to have the neural network available before starting to decode the samples.

[0173] During a step E36, the vector Zdn is processed by the neural network MLP' to generate as output an output vector representative of the sample Pdn to be decoded, according to one embodiment a pair inserted into the decoded image I(Pdn) at the positions (xn, yn) of its horizontal and vertical motion components. In another embodiment, the output vector is a triplet (mvx, mvy, cl) inserted into the composite decoded image I(Pdn) at the positions (xn, yn) of its horizontal and vertical motion components and its composition component CL. This step is identical to step E26 which was described for the coder with the support of [Fig.4].

[0174] When all the samples of the signal have been processed, the decoded signal corresponding for example to the image I(Pdn) is available.

[0175] [Fig.8] illustrates a decoding method used in one embodiment of the invention for decoding an image I(Pdn).

[0176] In this embodiment, there are 4 FMd cards. In a preferred embodiment, there are 7.

[0177] In this embodiment, the FMd; cards have the same resolution as the image to be decoded, and therefore comprise WxH variables, where W represents the width of the image in pixels, and H its height.

[0178] In this embodiment, the vector Zdn is a 4-tuple (zi...z4) consisting of the values ​​of the FMd maps; located at the coordinates (xn, yn) of the current pixel Pdn. The vector Zdn is optionally dequantized and then processed by the MLP' neural network to generate a second vector as output. According to a decoding embodiment, corresponding to the coding embodiment of [Fig.5], the output vector is a pair (mvx, mvy) representative of the motion vector Pdn to be decoded. The pair is inserted into the decoded image I(Pdn) at the positions (xn, yn) of the horizontal and vertical motion components (MVX, MVY). In the embodiment shown in dotted and dashed lines, the output vector is a triplet (mvx, mvy, cl) representative of the sample Pn to be decoded. The output vector is inserted into the decoded composite image I(Pdn) at the positions (xn, yn>a) of its horizontal and vertical motion components (MVX, MVY) and its composition component (Cl). In a video coding scheme such as that shown in [Fig.l 1], the composition data cl can be used to make a weighted sum of several reference images, one of which is compensated by the vector MV (mvx, mvy).

[0179] According to a variant shown in dotted lines, there are 5 cards: an additional card FME'o has been introduced. In this embodiment, the vector Zdn is a 5-tuple.

[0180] Naturally, the mode represented in [Fig.6] for coding can be transposed mutatis mutandis for decoding.

[0181] [Fig.9] illustrates a decoding method used in another embodiment of the invention for decoding an image I(Pdn).

[0182] The inputs to the MLP' neural network are identical to those in [Fig.8] and the same comments apply

[0183] According to this decoding embodiment, the output vector Pdn is a 5-tuple (mvlx, mvly, mv2x, mv2y, c2) representative of two motion vectors to be decoded and of composition information C consisting of a data item c2. The 5-tuple is inserted into the decoded image I(Pdn) at the positions (xn, yn) of its horizontal and vertical motion components (MV1X, MV1Y, MV2X, MV2Y) and of its composition component (B). In a video coding scheme such as that represented in [Fig. 12], the composition data item c2 can be used to make a weighted sum of two reference images compensated respectively by the vectors MV1 (mvlx, mvly) and MV2 (mv2x, mv2y).

[0184] According to a variant represented in mixed dotted lines, the output vector Pdn is a 6-tuple (mvlx, mvly, mv2x, mv2y, cl, c2) representative of two motion vectors to be decoded and of a composition information C consisting of two data (cl, c2). In a video coding scheme such as that represented in figures 11 and 12, the composition data c2 of weighting type can make it possible to make a weighted sum of two reference images compensated respectively by the vectors MV1 (mvlx, mvly) and MV2 (mv2x, mv2y), the composition data cl of masking type can allow to use an Intra type reference image in the occlusion zones.

[0185] [Fig. 10] illustrates a method for decoding video sequences using a motion image generated according to the method of the invention.

[0186] A motion image as described previously, comprising for each position (xn, yn) of the motion image I(Pdn) a motion vector (mvx, mvy) is presented at the input of a motion compensation module, or warping (WARP). The WARP module uses as input one or more IREF reference images and the decoded motion information to perform motion compensation, or warping, on the reference image(s) in order to obtain one or more prediction image(s), or compensated image(s), noted ICOMP. According to the embodiment presented, warping conventionally makes it possible to modify the position of the samples in the reference image to obtain the displaced samples in the compensated image. For example, each pixel of the reference image can be displaced according to the motion vector (mvx, mvy) to obtain a compensated sample in the ICOMP image.Other, more complex motion representations, as mentioned before, can of course be used. It has been assumed here that the digital image of the video sequence is of the same size as the composite image, i.e., it has N pixels. Naturally, depending on the variant, it can be of a different size, and in particular larger than N. In this case, a motion vector can be used for groups of pixels, or blocks, of the image.

[0187] [Fig. 11] illustrates a method for decoding video sequences using a composite image (of motion and composition) generated according to the method of the invention.

[0188] The composite image I(Pdn) as described previously, in particular in support of [Fig. 8], comprises for each position (xn, yn) a motion vector (mvx, mvy). As previously described in support of [Fig. 10], the motion vector is presented at the input of a motion compensation module, or warping (WARP). The WARP module uses as input one or more IREF_INTER reference images and the decoded motion information to perform motion compensation, or warping, on the reference image(s) in order to obtain one or more prediction frames, or compensated frames, noted ICOMP. The WARP module also uses as input one or more IREF_INTRA reference images which will be used for an intra-image type prediction (i.e. they are not intended to undergo motion compensation). The composite image I(Pdn) as described previously, in particular in support of [Fig.8], also includes for each position (xn, yn) a composition information in the . Cl map, for example a real number cl between 0 and 1. The compensated image, the intra image and this composition data cl are presented as input to a mixer module noted BLEND. The BLEND module generates as output a reconstructed image IREC of which each of the N pixels PIX(n) can be obtained by a calculation of the type: PIX(n) = cl * ICOMP(n) + (l-cl)*IREF_INTRA(n). Using this mechanism, it is possible to simply define compensated image zones in inter-image mode (when cl is 1), and compensated image zones in intra-image mode (when cl is 0) or any combination of the two modes (when cl is strictly between 0 and 1). This mechanism can be very useful for managing, for example, occlusion zones in an image, which cannot be compensated by movement.

[0189] It has been assumed here that the digital image of the video sequence is of the same size as the composite image, i.e. that it comprises N pixels. Naturally, according to variants, it may be of a different size, and in particular larger than N. In this case, a motion vector and a composition coefficient are used for groups of pixels, or blocks, of the image. According to a variant, the composition parameter may further contain data relating to the type or number of image to be used.

[0190] [Fig. 12] illustrates a method for decoding video sequences using a composite image (of motion and composition) generated according to the method of the invention.

[0191] The composite image I(Pdn) as described previously, in particular in support of [Fig.9], comprises for each position (xn, yn) two motion vectors MV1 (mvlx, mvly) and MV2 (mv2x, mv2y). As previously described in support of Figures 10 and 11, each motion vector is presented at the input of a motion compensation module, or warping (WARP1, WARP2). The WARP1 (resp. WARP2) module uses as input one or more reference images IREF1 (resp. IREF2) and the decoded motion information MV1 (resp. MV2) to perform motion compensation, or warping, on the reference image(s) in order to obtain one or more prediction image(s), or compensated image(s), noted ICOMP1 (resp. ICOMP2). The composite image I(Pdn) as described previously, in particular with the support of [Fig.9], also includes for each position (xn, yn) composition information in a map C2, comprising for example a real number c2 between 0 and 1.The compensated images and this composition data c2 are presented as input to a mixer module denoted MERGE. The MERGE module generates as output a reconstructed image IREC of which each of the N pixels PIX'(n) can be obtained by a calculation of the type: PIX'(n) = c2 * ICOMPl(n) + (1- c2)* ICOMP2(n). Using this mechanism, it is possible to simply define compensated image zones benefiting from a weighted prediction of the images ICOMP1 and ICOMP2. This mechanism can be very useful for managing, for example, transition zones or a change in brightness between two images in a sequence.

[0192] It has been assumed here that the digital image of the video sequence is of the same size as the composite image, i.e. that it comprises N pixels. Naturally, according to variants, it may be of a different size, and in particular larger than N. In this case, the motion vectors and the composition coefficient are used for groups of pixels, or blocks, of the image.

[0193] According to a variant, the composition information may further contain data relating to the type or number of reference image to be used.

[0194] According to a variant, this mode can be combined with that of [Fig.l 1]. We can then obtain a prediction of the type: PIX'(n) = IREF_INTRA / 2+ c2 / 2 * ICOMPl(n) + (1- c2) / 2* ICOMP2(n), or even PIX'(n) = cl* IREF_INTRA+ (1- cl)* c2 * ICOMPl(n) + (1- cl)*(l- c2) ICOMP2(n), etc.

Claims

Claims

1. Method for coding a signal (I(Pn)) comprising a plurality of samples (Pn) comprising motion information, the method comprising the following steps: - construction (E21, E22) of a first group of characteristic maps (FM;); - for at least one sample, called the current sample (Pn), of the signal, associated with a position (xn, yn) in the signal to be coded: - construction (E24, E25) of a characteristic vector (Zn) from said characteristic maps (FM;) of said first group; - processing (E25) of said characteristic vector (Zn) by an artificial neural network (MLP) defined by a set of parameters (Wk), to provide a vector representative of a value (P'n) of the current sample;- updating (E22, E28) at least one value of one of said characteristic maps of said first group and / or at least one parameter, as a function of a coding performance measurement, - coding (E23, E27, E29) of said first group of characteristic maps (FM;) and of said set of parameters (Wk).;

2. Method for decoding a signal comprising a plurality of samples (Pdn) comprising motion information, the method comprising the following steps: - decoding (E31) a first group of characteristic maps (FMdi); - decoding (E35) a set of parameters (Wdk) representative of a neural network (MLP'); - for at least one sample, called current sample (Pdn), of the signal, associated with a position (xn, yn) in the signal to be decoded: - construction (E33, E34) of a characteristic vector (Zdn) from the characteristic maps of said first group (FMd;), and: - processing (E36) of said characteristic vector (Zdn) by an artificial neural network (MLP') defined by the decoded parameters (Wdk) to provide a vector representative of a decoded value of the current sample (Pdn).

3. Coding or decoding method according to one of the preceding claims, characterized in that it comprises a step of constructing (E32) a second group of characteristic maps (FME'i) and in that the characteristic vector is further constructed from said characteristic maps of the second group.

4. Coding or decoding method according to one of the preceding claims, characterized in that at least one of said characteristic maps (FMdi, FME'i) has a resolution lower than that of the signal.

5. Coding or decoding method according to one of the preceding claims, characterized in that at least one of said characteristic maps (FMdi, FME'i) has a resolution identical to that of the signal.

6. Coding or decoding method according to one of the preceding claims, characterized in that the characteristic vector (Zn) is constructed as a function of said position (xn, yn) of said current sample (Pn)-

7. Coding or decoding method according to one of the preceding claims, characterized in that the construction of said characteristic vector (Zdn) comprises a sub-step of extracting a value from said at least one characteristic map (FMdjFME'j) at a position (xn, yn) identical to that of the current sample (Pdn) in the signal.

8. Method of coding or decoding coding or decoding according to one of the preceding claims, characterized in that the construction of said characteristic vector (Zdn) comprises a sub-step of extracting at least one value from said at least one characteristic map (FMdjFME'i) at a position close (xn, yn) to that of the current sample (Pdn) in the signal.

9. Coding or decoding method according to one of the preceding claims, characterized in that the construction of said characteristic vector (Zdn) comprises a sub-step of processing (E33, E34) said extracted values ​​to obtain the characteristic vector.

10. Coding or decoding method according to one of the preceding claims, characterized in that the coding or decoding (E31) of said first group of characteristic maps comprises an entropic coding or decoding sub-step.

11. Coding or decoding method according to one of the preceding claims, characterized in that the motion information is a motion vector (mvx, mvy).

12. Coding or decoding method according to one of the preceding claims, characterized in that a sample comprises at least one composition data (C, cl, c2).

13. Device for coding a signal (I(Pn)) of a plurality of samples (Pn ) to be coded comprising movement information characterized in that said device is configured to implement: - construction (GEN, MAJ) of a first group of characteristic maps (FM;); - for at least one sample, called current sample (Pn), of the signal to be coded, associated with a position (xn, yn) in the signal to be coded: - construction (XTR, TT) of a characteristic vector (Zn) from said characteristic maps (FM;) of said first group, and; - processing (MLP) of said characteristic vector (Zn) by an artificial neural network (MLP) defined by a set of parameters (Wk), to provide a vector representative of a decoded value (P'n) of the current sample; - updating (MAJ, NNC) at least one value of one of said characteristic maps of said first group and / or at least one parameter of said network, as a function of a coding performance measurement, - coding (FMC, NNC) of said first group of characteristic maps (FM;) and of said set of parameters (Wk).

14. Device for decoding a signal of a plurality of samples (Pdn) to be decoded comprising movement information characterized in that said device is configured to implement: - decoding (FMD) of a first group of characteristic maps (FMdi); - decoding (NND) of a set of parameters (Wdk) representative of a neural network (MLP'); - for at least one sample, called current sample (Pdn), of the signal to be decoded, associated with a position (xn, yn) in the signal to be decoded: - construction (XTR', TT') of a characteristic vector (Zdn) from the characteristic maps of said first group (FMd;), and: - processing (MLP') of said characteristic vector (Zdn) by an artificial neural network (MLP') defined by the decoded parameters (Wdk) to provide a vector representative of a decoded value of the current sample (Pdn).

15. A computer program comprising instructions for carrying out the steps of an encoding or decoding method according to claim 1 or 2 when said program is executed by a computer.

Images