Method and device for encoding and decoding images

Abstract

The invention relates to a method and device for encoding and decoding images including a plurality of samples to be encoded or decoded. The decoding method comprises the following steps: - decoding (E31) a group of (M) feature maps (FMdi); - obtaining (E32) a set of synthesis parameters (Wck) representative of a synthesis module (MLP'); - obtaining (E32) a set of convolution parameters (Vck) representative of a convolutional neural network; - processing (E33, E34) the group of feature maps by the synthetic neural network (MLP') to provide a group of (NE) latent value maps (L'j), the number of latent value maps being less than or equal to the number of feature maps; - processing (E35) the group of latent value maps by the convolutional neural network (MPP') in order to provide the at least one decoded image (I(pdn)).

Description

Method and device for coding and decoding images.

[0001] The invention relates to the general field of coding digital image sequences.

[0002] Digital images or videos are generally subject to source encoding aimed at compressing them in order to limit the resources required for their transmission and / or storage. There are many encoding standards, such as the ITU / MPEG standards (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) as well as their extensions (MVC, SVC, 3D-HEVC, etc.).

[0003] As video formats continue to evolve to compress more and more and adapt to the variety of expected formats and communication networks, the possibilities for prediction are becoming ever greater and traditional coding and decoding algorithms are becoming very complex.

[0004] In addition to these classic approaches proposed by compression standards (MPEG, ITU), approaches based on artificial intelligence, and in particular neural ones, tend to develop.

[0005] Some of these neural approaches can be seen as a simple extension of the competition notion of the aforementioned compression techniques, such as the prediction and transformation mode competition in video coding.

[0006] Other approaches use the concept of "autoencoder." Autoencoders are learning algorithms based on artificial neural networks, which allow the construction of a new representation of a dataset. The architecture of an autoencoder consists of two parts: the encoder and the decoder. The encoder consists of a set of layers of neurons, which process the data in order to construct new representations called "encoded," also called "latent representations." In turn, the layers of neurons in the decoder receive these representations and filter them in order to try to reconstruct the initial data. The differences between the reconstructed data and the initial data make it possible to measure the error made by the autoencoder. Training consists of modifying the parameters of the autoencoder in order to reduce the reconstruction error measured on the different samples of the dataset.The performance of such autoencoder-based systems comes at the cost of a considerable increase in memory footprint and complexity compared to conventional approaches as proposed by compression standards. They can have millions of parameters and may require on the order of a million MACs (multiply-accumulate) to decode a single pixel. This makes such decoders significantly more complex than conventional decoders, which could hamper the adoption of learning-based compression. In addition, the neural networks used at the encoder and decoder are different. US 2022 / 0385907 describes such an autoencoder system.

[0007] More recently, a simple neural network-based image encoding technique was described in the article "COmpression with Implicit Neural Representations" by Emilien Dupont et al. (arXiv:2103.03123). The proposed encoding technique consists of fitting a neural network to an image, quantizing the network weights, and transmitting them. At decoding time, the neural network is evaluated at each pixel position to reconstruct the image. However, such a technique remains inefficient in terms of compression.

[0008] Document US 2022 / 0385907 also describes a coding and decoding system based on the use of such implicit neural representations.

[0009] However, there is a need for a solution that can encode / compress a sequence of images in a simple and efficient way.

[0010] The invention relates to a method for coding at least one image comprising the following steps: - a step of constructing said at least one coded and then decoded image, comprising the sub-steps of: - constructing a group of feature maps; - processing said group of feature maps by a synthesis module, defined by synthesis parameters, to provide a group of latent value maps, the number of latent value maps being less than or equal to the number of feature maps; - processing said group of latent value maps by at least one convolution neural network, defined by convolution parameters, to provide said at least one coded and then decoded image; - updating at least one value of one of said feature maps and / or at least one synthesis parameter and / or at least one convolution parameter, as a function of a coding performance measurement;- a step of coding said group of feature maps and said set of synthesis and convolution parameters;

[0011] The invention also relates to a method for decoding an image at least from a binary stream, comprising the following steps: - decoding a group of characteristic maps; - obtaining a set of synthesis parameters representative of a synthesis module; - obtaining a set of convolution parameters representative of at least one convolution neural network; - processing said group of characteristic maps by said synthesis module to provide a group of latent value maps, the number of latent value maps being less than or equal to the number of characteristic maps; - processing said group of latent value maps by said at least one convolution neural network to provide said at least one decoded image.

[0012] For the purposes of the invention, encoding, or "coding", means the operation which consists of representing a set of samples, or pixels, of the image or of the plurality of images, in a compact form carried for example by a digital binary train. Decoding means the operation which consists of processing a digital binary train to restore decoded samples of the image or of the plurality of images.

[0013] By "image" we mean a two-dimensional digital image comprising one or more components (color, depth, views, etc.)

[0014] By "at least one image" is meant a plurality of ordered two-dimensional images, for example temporally in the case of a video. According to another example, the images can be views of the same scene represented in multi-views. According to another example, the images can be a plurality of temporal and multi-view images (immersive video).

[0015] A "sample" or "pixel" refers to a value or set of values ​​taken from an image. Sampling a signal produces a sequence of discrete values ​​called samples. In the case of an image signal, the sample is called a pixel, which can be, for example, a color pixel traditionally represented by a triplet of values, each value belonging to a component, for example (R,G,B) or (Y,U,V). The position of the sample can be identified by its abscissa (x) and ordinate (y) coordinates in the image.

[0016] A "feature map" means a part of an abstract representation of an image or a sequence of images, comprising a plurality of variable scalar data, also called characteristic values, for example real numbers, or integers, signed or unsigned. The abstract representation of an image or a sequence may therefore comprise several feature maps. The abstract representation of one or more images may also correspond to a single feature map.

[0017] A "latent value map" means a part of an abstract representation of an image or a sequence of images, comprising a plurality of variable scalar data, also called latent values, for example real numbers, or integers, signed or unsigned. The abstract representation of an image or a sequence may therefore comprise several latent value maps. The abstract representation of one or more images may also correspond to a single latent value map. These maps result from the processing of the feature maps by a synthesis module, for example a synthesis neural network.

[0018] By "synthesis module" we mean for example a neural network, or a filter bank, or a linear inverse transformation (wavelet, DCT, etc.) defined by its parameters W k, capable of processing the transformed characteristic maps as input, to generate as output a set of NL latent value maps.

[0019] By "synthetic neural network" and "convolutional neural network" is meant a neural network such as a convolutional neural network, a multi-layer perceptron, etc. The neural network is defined for example by a plurality of layers of artificial neurons and by a set of activation, weighting and addition functions (for example, a layer can calculate y = f(Ax+b), where y and b are vectors of dimension N, x is a vector of dimension M, A is a matrix of dimension MxN, and f is the activation function).

[0020] By "neural network parameter" is meant one of the values ​​that characterizes the neural network, or the plurality of neural networks, for example a weight associated with one of the neurons (filter coefficient, values ​​of a matrix, weighting, bias, value affecting the operation of the non-linearity, etc.)

[0021] By "generation of said at least one image" is meant a step which aims to reconstruct the decoded samples of the image or of the plurality of images.

[0022] By "performance measurement" is meant a measurement between at least one value of a sample to be coded and a decoded value of said sample. The measurement can evaluate, for example, a distortion, or a perceptual error. It can be carried out on a sample or a plurality of samples (for example, the current samples, or the current images, etc.). The measurement can also include a measurement of the flow rate, in particular associated with the coding of the neural network and / or the coding of the group's characteristic maps. The measurement can be a joint measurement between the flow rate and the distortion through their weighting. As is well known in the state of the art, the value of this measurement is generally minimized until a target value is reached.

[0023] By "construction step" is meant a step which aims to construct the encoded and then decoded values ​​of the pixels of the image or of the plurality of images, and the representative parameters of the image, in particular the values ​​of the characteristic maps and the parameters of the neural networks, before their actual encoding. The construction sub-steps can be repeated as many times as necessary to obtain an acceptable performance measure.

[0024] Generally speaking, it is considered that the steps of an coding or decoding method should not be interpreted as being linked to a notion of temporal succession. In other words, the steps may be carried out in a different order than that indicated in the independent coding or decoding claim, or even in parallel.

[0025] The coding method according to the invention constructs the coding parameters, from an input image or sequence of images, by training neural networks on feature maps and then latent value maps which may be at the resolution of the input images, or at a lower resolution. During the training, or construction, the parameters of the neural network and the values ​​of the feature maps and the latent value maps are updated according to a performance measure, for example of the rate-distortion type. When the training is finished, that is to say when the performance measure obtained is satisfactory, the actual coding of the parameters of the neural networks and the values ​​of the feature maps can be carried out and stored or transmitted to the decoder. The latent value maps are neither coded nor transmitted.

[0026] Advantageously, the training process makes it possible to refine the parameters of the neural networks and / or the values ​​of the feature maps until an adequate representation in terms of performance is obtained, for example a desired balance between the generated bit rate and the distortion undergone by the input image or sequence of images being encoded. The training of the values ​​of the feature maps and the parameters of the neural networks can be joint. Advantageously, the coding method according to the invention makes it possible to efficiently compress the signal.

[0027] Advantageously, the compression of characteristic maps is efficient since it allows them to be transported over networks or on storage media with reduced capacities.

[0028] Furthermore, the use of two sets of networks, synthesis then convolution, is advantageous in itself, because it allows the two networks to be trained separately to ensure better convergence. According to certain embodiments, it is possible to freeze one of the two networks and learn only one, so that only the parameters of the learned network will be transmitted to the decoder.

[0029] Furthermore, since the number of latent value maps is less than or equal to the number of feature maps, the number of operations to be performed by the convolution network is reduced. This results in reduced complexity, as less data is processed. Advantageously, the decoding process is simple since it is sufficient to decode the feature maps associated with an image or a group of images, and to obtain the synthesis and / or convolution neural networks, to reconstruct a decoded version of the current image or group of current images.

[0030] Such a synthetic or convolutional neural network can advantageously be of very simple structure with few parameters and few operations per sample.

[0031] According to embodiments of the coding or decoding method:

[0032] - Said at least one image comprises a plurality of samples to be decoded, and the step of processing said group of characteristic maps to provide a group of latent value maps comprises the following sub-steps: - for at least one sample, called current sample, of said at least one image to be decoded, associated with a position: - construction of a characteristic vector from the characteristic maps, as a function of said position of said current sample, and: - processing of said characteristic vector by said synthesis module to provide a latent vector, representative of a latent value of the current sample; - construction of the group of latent value maps from said at least one latent vector.

[0033] Advantageously, according to this mode, the synthesis module is of the positional type; the chaining of a positional type synthesis network and a convolution network makes it possible to carry out local processing at the level of the synthesis network, then to correct it by using the convolution network.

[0034] - The synthesis module includes a multi-layer perceptron, or MLP. Such a neural network is advantageously very simple in structure with few parameters. In addition, decoding can be performed sample by sample, progressively and on a particular block of the image.

[0035] - The convolution network has at least one convolution layer. Thus, non-localized processing of the image can be performed, which improves the consistency of the generated image by limiting noise and improving the contours present in the image.

[0036] - The number of latent value maps is equal to the number of components in the image. Advantageously, the complexity is limited by processing exactly the number of components in the output pixels (for example, three in the case of an RGB or YUV image). Alternatively, each of the latent value maps (corresponding, for example, to the Y, U, and V components) can be processed independently. This further limits the processing complexity because there are fewer multiplication / addition operations in the convolution network, and therefore a limitation of the memory space required by the encoder and decoder, as well as a lower cost of the convolution parameters.

[0037] – At least part of the set of synthesis and / or convolution parameters is obtained (resp. coded) from data of said binary stream (resp. coded in the binary stream); advantageously according to this mode, the parameters of the neural networks are transmitted in the stream. The synthesis and / or convolution module can be transmitted in a quantized form and coded in a compact form using any quantizer and entropy coder accessible to those skilled in the art. A format analogous to that of the MPEG-7 NNR standard can be used.

[0038] – At least part of the set of synthesis and / or convolution parameters is obtained (resp. coded) from predetermined parameters. Advantageously, according to this mode, the parameters of the neural networks are accessible to the decoder, for example they can be stored in a storage memory (resp. coded in a storage memory). In particular, part of the synthesis and / or convolution module can be stored in a quantized form and coded in a compact form using any quantizer and entropy coder accessible to those skilled in the art. A format similar to that of the MPEG-7 NNR standard can be used.

[0039] – At least part of the set of synthesis and / or convolution parameters is obtained (resp. coded) as a function of reference parameters. Advantageously, according to this mode, the parameters of the neural network are decoded by taking into account reference parameters which can be stored in a memory (storage memory, network, etc.) accessible to the decoder. Thus, the storage space or the transmission rate on the network can be effectively reduced: certain parameters can be coded / decoded in a complementary manner, others not be coded / decoded at all, because they are available in the reference parameters. According to alternative embodiments, which can be combined with each other: the reference parameters can constitute a set of variants of a synthesis and / or convolution network: type of convolution, attention modules, etc.Thus, only these parameters need to be encoded, inserted into the stream and decoded by the decoder; certain synthesis and / or convolution parameters are coded in a complementary manner to the parameters of the reference network. Thus, the decoder will only need to decode these residual parameters and then add them, multiply them (or combine them in any other known way) with those of the reference neural network; part of the parameters of the reference network is reused for the synthesis and / or convolution network. For example, a complete layer of the network can be copied from the reference network, the parameters of this layer are therefore neither encoded, nor transmitted, nor decoded; a reference network indicator is transmitted in the stream. It is thus possible to indicate a network to be used for decoding from among a plurality of possible networks known to the decoder.

[0040] - Said at least one convolution network comprises at least two convolution networks. Advantageously, the use of several cascaded networks makes it possible in particular to enhance the filtering and improve the consistency of the generated image by limiting its noise and improving its contours. In addition, each network can be learned individually or fixed independently. According to alternative embodiments, which can be combined with each other: the binary stream comprises an indicator indicating the use of at least one of said networks. Thus, depending on the quality and complexity required, one or more networks can be used for decoding. The indicator indicates, for example, that network 1 is mandatory, but that networks 2 and 3, which are cascaded behind network 1, are optional. The decoder can then choose whether or not to use them: if it uses them, it will obtain better quality at the expense of complexity, and vice versa.At least two of the convolution networks have common parameters. It is thus advantageous to transmit the characteristic parameters of one network, which can be used for the others, which reduces the transmission cost. The same network is thus shared. At least two of the networks have an identical structure and only their weights differ partially or totally. The same computational elements are thus shared.

[0041] – At least one image has at least two dimensions, and said convolution layer uses a kernel separable according to said at least two dimensions. Advantageously, according to this mode, the transmission cost and complexity are reduced by the use of operations (for example, convolution, upsampling, etc.) that are separable, which require fewer operations than non-separable operations.

[0042] Correlatively, the invention also relates to a coding device and a decoding device.

[0043] The characteristics and advantages of the coding or decoding method apply in the same way to the coding or decoding device according to the invention and vice versa.

[0044] The invention also relates to a computer program on a recording medium, this program being capable of being implemented in a computer or a control device in accordance with the invention. This program comprises instructions adapted to the implementation of the corresponding method. This program can use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0045] The invention also relates to a computer-readable information carrier or recording medium comprising computer program instructions mentioned above. The information or recording media may be any entity or device capable of storing the programs. For example, the media may comprise a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a floppy disk or a hard disk, a DNA sequence, or a flash memory. Furthermore, the information or recording media may be transmissible media such as an electrical or optical signal, which may be conveyed via an electrical or optical cable, by radio link, by wireless optical link or by other means.

[0046] A program according to the invention can in particular be downloaded from an Internet-type network.

[0047] Alternatively, each information or recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of a method according to the invention.

[0048] Other characteristics and advantages of the present invention will emerge from the description given below, with reference to the appended drawings which illustrate exemplary embodiments thereof which are not limiting in nature.

[0049] Schematically represents a coding device used in the context of the invention;

[0050] schematically represents a decoding device used in the context of the invention;

[0051] schematically represents artificial neural networks of synthesis and convolution used in an embodiment of the invention for coding and decoding.

[0052] schematically represents artificial neural networks of synthesis and convolution used in another embodiment of the invention for coding and decoding.

[0053] Schematically represents convolution networks according to embodiments of the invention.

[0054] Schematically represents convolution networks according to embodiments of the invention.

[0055] is a flowchart representing an example of a coding process that can be implemented by the coding device of the.

[0056] is a flowchart representing an example of a decoding method that can be implemented by the decoding device of the.

[0057] illustrates a decoding method used in one embodiment of the invention.

[0058] It schematically represents a digital image coding device ENC.

[0059] The ENC coding device receives as input a succession of samples to be coded, noted P n , for example an image data set denoted I(P n ). The image signal I(P n ) can represent a two-dimensional image, or a plurality of two-dimensional images (video, color components, stereoscopic, multiscopic components, etc.). P n denotes a sample n of the input signal comprising N samples. In one embodiment, the signal is a color image signal represented by means of at least one two-dimensional representation, such as a matrix of pixels, of width W and height H (with N = WxH) each pixel comprising a red (R), green (G), blue (B) component, or, alternatively, a luminance (Y) component and at least one chrominance (U and / or V) component of width W u / v and height H u / vThe location of each pixel is defined by its abscissa and ordinate (x and y) coordinates in the image. In one embodiment, the image is grayscale, represented using a two-dimensional representation, such as a pixel matrix, with each pixel having a grayscale component, or luminance. Generally, the number of components is denoted NC.

[0060] This ENC coding device comprises an INIT module for generating feature maps, an XTR module for processing and extracting data, an MLP synthesis module, an MPP convolution module corresponding to at least one artificial convolution neural network, an FMC module for coding feature maps, an NNC module for coding neural networks, an EVAL module for evaluating a performance function, and an MAJ update module.

[0061] The synthesis module used for coding, MLP, and the synthesis module used for decoding, MLP', are defined by an identical structure, comprising for example a set of filter banks, or at least one artificial neural network and a set of weights and activation functions associated respectively with the artificial neurons of the network concerned.

[0062] The (at least one) artificial convolutional neural network used for coding, MPP, and the artificial convolutional neural network used for decoding, MPP', are defined by an identical structure, comprising for example a plurality of layers of artificial neurons, and by a set of weights and activation functions associated respectively with the artificial neurons of the network concerned.

[0063] During the process of training, or construction, of the coding, that is to say as long as the stage of evaluation of a performance is not satisfactory, the coding modules (FMC, NNC) carry out a coding simulation, followed by a decoding, intended for the evaluation module. Subsequently, they carry out the effective coding of the values ​​of the M FM characteristic maps i , with 1 k and V k . As is known, the coding simulation can be identical to the actual coding, or an approximation of it.

[0064] ​Each feature map contains scalar data, according to the definition of a feature map given above, and there are M feature maps. However, it is possible to organize the feature values ​​differently, for example by forming a single signal comprising, for each sample, a vector of M scalar values, without loss of generality. Even if the data are organized differently, we consider in the rest of the document that there are M feature maps. The maps coded by the FMC module for a signal, for example a current image (or a plurality of images) are denoted FMc i The FMC encoding module quantizes the feature map values ​​if necessary using a quantizer to generate an ordered collection of quantized values. Then the encoding module compresses the quantized data, for example using an entropy encoder.

[0065] The XTR module performs data processing and extraction from feature maps. The XTR module can perform quantization of feature map values ​​by using a quantizer Q to generate an ordered collection of quantized values. The XTR module can alternatively obtain the quantized data by the FMC module. Recall that quantization of a value refers to mapping that value to a member of a discrete set of possible code symbols. For example, the set of possible code symbols may consist of integer values, and the quantization system performs simple rounding of a real value to an integer value. In another example, quantization consists of multiplication by a given value and then rounding.Then, depending on the embodiments, the XTR module can perform a transformation of the values ​​of at least one of the characteristic maps such as for example oversampling, interpolation, filtering, etc. In one embodiment, at the end of the transformation, a transformed characteristic map has the same resolution as the input signal. Then the XTR module performs a data extraction from the characteristic maps thus quantized and optionally transformed, denoted FMt. i , to present the data to the MLP module.

[0066] The MLP module is a synthesis module, for example a neural network, or a filter bank, or an inverse linear transformation (wavelet, DCT, etc.) defined by its parameters W k , capable of processing the transformed characteristic maps as input, to generate as output a set of NL latent maps L j with 1 <j <= NL.

[0067] According to one embodiment:

[0068] - The XTR module performs an extraction of values ​​in the transformed FMt characteristic maps i , for a sample P n to be encoded, based on its coordinates in the input signal. The extracted values ​​constitute the vector Z n . Z n is a J-tuple, that is, it has J elements, or data z i . The vector Z n of index n refers to the characteristic vector of the pixel P n .

[0069] - The MLP module includes a neural network defined by K parameters W k , with 1 <k <=K, apte à traiter le vecteur Z n , or J-Uplet, as input, to generate as output a second vector containing latent values ​​representative of the sample P nto be coded. The synthetic neural network comprises, according to one embodiment, an MLP, or Multi Layer Perceptron, consisting of an input layer adapted to the input format (the J-tuple), optionally one or more hidden layer(s), and an output layer adapted to the output format of the output vector, generally a vector comprising NL elements, each element being intended for a latent value map. According to one embodiment, NL is equal to 3 and the output vector is a triplet to be inserted into the three maps L1, L2 and L3 (for example, Y, U and V) of latent values ​​at the pixel coordinates. According to another embodiment, NL is equal to 4 and the output vector is a quadruplet to be inserted into four maps L1–L4 of latent values ​​at the pixel coordinates. According to another embodiment, NL is between the number M of FM maps i and the number of components of the image, M≤ NL≤ 3 for the case of three components.

[0070] According to one embodiment, the NL latent value maps comprise any real values.

[0071] Latent value maps represent an intermediate signal before convolutional processing.

[0072] The (at least one) MPP neural network is of the convolutional type, that is, it includes at least one convolutional layer. It may include convolutional elements, with or without a residual structure and / or attention modules, as will be described later.

[0073] The W parameters k of the synthesis module coded by the NNC module are noted Wc k .

[0074] The V parameters k of the convolutional neural network(s) coded by the NNC module are noted Vc k.

[0075] The MPP network is applied at the output of MLP on the latent value maps (L j ) to obtain decoded samples P' n. When all samples P n of the signal have been coded then decoded, we have a reconstructed signal I(P' n), c'est-à-dire au moins une image I comportant N échantillons décodés.

[0076] The EVAL module performs an evaluation and minimization of a coding performance. The evaluation function is for example of the rate-distortion type. It aims to minimize the differences between the input representation of the current image I(P n ) and its representation I(P' n ) at the output, while optimizing the amount of data to be encoded, namely the encoded feature maps, the parameters of the Wc synthesis module k , and the parameters of the convolutional neural network(s) Vc k . The minimization can be carried out by gradient descent, or any other method within the reach of those skilled in the art.

[0077] The MAJ module updates the values ​​of the FM maps ito encode from the current image, depending on the results of the performance function.

[0078] Once the encoder training is complete, the neural network parameters are encoded, either lossless, in which case the decoder's MLP' / MPP' neural network is identical to MLP / MPP, or lossy, in which case the MLP' / MPP' network may be slightly different from MLP / MPP. The values ​​of the feature maps, FM i , are also encoded as FMc coded data i, par exemple via un codeur entropique.

[0079] It schematically represents a DEC decoding device.

[0080] The decoding device DEC receives as input a first group of encoded data organized into M FMc characteristic maps i , the Wc parameters k of a synthesis module and the Vc parameters k of at least one convolutional neural network.

[0081] This DEC decoding device comprises a neural network decoding NND module, a feature map decoding FMD module, an inverse transformation and data extraction XTR' module, a synthesis MLP' module, a convolution MPP' module corresponding to at least one convolution neural network.

[0082] The cards decoded by the FMD module, numbering M, are noted FMd i .

[0083] The parameters decoded by the NND module are denoted Wd k and Vd k . In case the synthesis module does not include a neural network, a different decoder can be used to decode the Wd parameters k. The Vd parameters k And Wdksont identiques aux paramètresVket Wkdu codeur si le codage a été effectué sans pertes.

[0084] The XTR' module performs processing and extraction of data from the decoded FMd feature maps i. The XTR' module optionally performs an inverse quantization corresponding to the quantization performed at the encoder. The inverse quantization is not necessary if the quantizer Q of the encoder has performed a simple rounding of the real values ​​submitted to it. The inverse quantization is also not necessary if the neural network is capable of taking into account a quantization of its input data. Otherwise, the decoder performs the inverse operation of the quantizer Q. Then, depending on the embodiments, the XTR' module can perform a transformation of the values ​​of at least one of the characteristic maps such as oversampling, interpolation, filtering, etc. In one embodiment, at the end of the transformation, a transformed characteristic map has the same resolution as the input signal.Then the XTR' module performs a data extraction from the characteristic maps thus quantified and optionally transformed, noted FMt'. i , to present the data to the MLP module”.

[0085] As previously described in support of the, the MLP' module is a synthesis module, for example a neural network or an inverse linear transformation (wavelet, DCT, etc.), defined by its parameters Wd k , capable of processing the transformed characteristic maps as input, to generate as output a set of NL latent maps L' j with 1 <j ≤ NL.

[0086] In a preferred embodiment:

[0087] - The XTR' module is identical to the XTR module of the. It performs an extraction of values ​​from the M FMt' characteristic maps i transformed, to construct the vector Zd n taking the form of a J-tuple for a sample P nto be decoded, depending on its coordinates in the signal to be decoded. In one embodiment, J=M.

[0088] - The MLP' module includes a neural network defined by K parameters Wd k , capable of processing the vector Zd n , or J-Uplet, as input, extracts feature maps, to generate as output a second vector ZLd n of latent values ​​representative of the sample P nto be decoded. The synthetic neural network comprises, according to one embodiment, an MLP, or Multi Layer Perceptron, consisting of an input layer adapted to the input format (the J-tuple), optionally one or more hidden layers, and an output layer adapted to the output format of the output vector, generally a vector comprising NL elements, each element being intended for a latent value map. According to one embodiment, NL is equal to 3 and the output vector is a triplet to be inserted into the three maps L'1, L'2 and L'3 of latent values ​​at the pixel coordinates. According to another embodiment, R is equal to 4 and the output vector is a quadruplet to be inserted into four maps L'1– L'4 of latent values ​​at the pixel coordinates. According to another embodiment, R is between the number M of FMd maps i and the number of components of the image or images to be decoded M≤ R≤ 3 for the case of three components.

[0089] Latent value maps represent an intermediate signal before convolutional processing.

[0090] The (at least one) MPP' neural network is of the convolutional type, that is to say that it comprises at least one convolutional layer. It may include convolutional elements, comprising a residual structure or not and / or attention modules, as will be described later. At the output of the MPP' module, we have the reconstructed image or plurality of images, I(Pd n ).

[0091] The DEC decoding device may be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the above-mentioned modules may then be realized by the cooperation of the processor and computer program instructions stored in the above-mentioned memory and designed to perform the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0092] Schematically represents synthetic and convolutional artificial neural networks used in one embodiment of the invention for encoding and decoding.

[0093] According to this embodiment, the synthesis module comprises an MLP / MLP' neural network and the convolution module comprises an MPP / MPP' neural network. The MLP / MPP synthetic artificial neural network for coding and the MLP / MPP' synthetic artificial neural network for decoding are defined by an identical structure, comprising for example a plurality of layers of artificial neurons, and by a set of weights and activation functions associated respectively with the artificial neurons of the network concerned.

[0094] A vector representation of a current sample (a vector Z n or Zd n from FMt feature maps i or FMt' i ) is applied as input (i.e. on an input layer) of the MLP or MLP synthetic neural network. The synthetic neural network produces as output a ZL vector n or ZLd nof latent values ​​representative of the decoded sample, which are reinserted into the NL latent value maps L j / L' j . On the, we have represented a layer of the synthetic neural network which has 40 outputs noted u1...u 40 . Each output is followed by a RELU type activation function. The RELU function (for Rectified Linear Unit) is well known to those skilled in the art. The output layer has three components denoted v1...v3, similarly followed by a RELU function. At the output of the MLP / MLP' module, the resulting values ​​of each RELU function, composing the vector ZL n / ZLd n , are injected into the maps Lj at the position defined by that of the pixel n being processed. The concatenation of all these values ​​constitutes the decoded latent value maps Lj / L'j. On the, there are therefore 3 latent value maps.

[0095] the decoded latent value maps L j / L' j are then processed by the MPP / MPP' convolution module, which produces as output a decoded signal, according to one embodiment the color components (R, G, B) constituting the image I.

[0096] According to the example in the, the convolution network consists of a residual convolution layer (CR1) as input, a set of RELU functions, and a residual convolution layer (CR2) as output. The CR modules are residual convolution elements, without limiting aspect. They can have kernels of any size. They can be of simple convolutional types or include attention elements. They can also include strides (shifting of the convolution kernel at each iteration), dilation (spacing of the input values ​​provided to the kernel), etc. A convolution element is potentially followed by a bias as is well known to those skilled in the art. Any activation function within the reach of those skilled in the art can be used (ReLU, sigmoid, Softmax, Softplus, Softsign, etc.)

[0097] The convolution elements can be composed of 2D convolutions, that is, the convolution kernel is applied jointly to the rows and columns of the latent value map. The convolution elements can be composed of 1D convolutions, in a separable form. In this case, rather than performing the convolution on the two-dimensional latent value map, a first one-dimensional (1D) convolution operator of length Lh is performed on the rows, then a second convolution of length Lv on the columns of the map obtained after the first operator. The row-column processing can be reversed to perform column-wise and then row-wise processing. This implementation has the effect of reducing the storage from Lh*Lv for a given non-separable kernel to Lv+Lh, and limits the complexity to Lh+Lv multiplications per map value versus Lh*Lv in the 2D case.

[0098] Naturally, the convolution module can include several networks, in cascade. Each network can include several convolution layers with or without bias, with activation functions such as relu or other, etc.

[0099] Schematically represents synthetic and convolutional artificial neural networks used in another embodiment of the invention for encoding and decoding.

[0100] These networks are capable of processing images whose components have different resolutions.

[0101] Similarly to the, according to this embodiment, the synthesis module comprises an MLP / MLP' neural network and the convolution module comprises an MPP / MPP' neural network. A vector representation of a current sample (a vector Z n or Zd n from FMt feature maps i or FMt' i) is applied as input to the MLP or MLP synthesis artificial neural network. The synthesis neural network produces a ZL vector as output n or ZLd n of latent values ​​representative of the decoded sample, which are reinserted into the NL latent value maps Lj / L'j. In the figure, a layer of the synthetic neural network is shown which has 40 neuron outputs denoted u1...u40. Each output is followed by a RELU type activation function. The output layer has six components denoted v1...v6, similarly followed by a RELU function. At the output of the MLP / MLP' module, the resulting values ​​of each RELU function are injected into the maps Lj at the position defined by that of the pixel being processed. The concatenation of all these values ​​constitutes the decoded latent value maps Lj / L'j. In the figure, there are therefore 3 latent value maps.

[0102] Unlike the, however, in this mode the latent value maps L j do not all three have the same number of samples. The MLP / MLP' module generates more latent values ​​for the first map than for the following ones: 4 values ​​are generated for the first map noted Ly against 1 for the L maps u and L v . This advantageously results in latent value maps corresponding to chrominance components that contain fewer samples than latent value maps corresponding to luminances. Such a size adaptation makes it possible to reduce the number of operations performed and the number of network parameters to be coded and transmitted (resp. decoded). This number can be adjusted to the dimension of the components.

[0103] The decoded latent value maps L j / L' jare then processed by the MPP / MPP' convolution module, which produces a decoded signal as output, according to this embodiment the color components (I Y , I u , I v ) constituting the image I, the U and V components typically comprising four times fewer samples than the Y component.

[0104] The architecture of the convolution network is identical to that of the, and the same comments apply. The convolution elements are denoted CR1, CR2, CR3, and CR4.

[0105] Schematically represents convolution networks according to embodiments of the invention.

[0106] Cases (a) to (d) respectively illustrate convolution layers used in the context of the invention for processing three input latent value maps (NL=3), and a YUV three-component image (NC=3) at the output of the convolution module. In the figure, the oblique lines surmounted by a number indicate the number of input latent value maps and output image components.

[0107] Naturally, other embodiments, not shown, are conceivable, with a greater or lesser number of latent value maps and a variable number of output components (for example, NL = 4 latent value maps for an image with NC = 3 YUV output components), without loss of generality.

[0108] Furthermore, in the embodiment presented, each latent value map contains only scalar data, in accordance with the definition of a latent value map given previously, and there are three latent value maps each corresponding to one of the three coordinates of the samples of said signal. It is however possible to organize the latent values ​​differently, for example by forming a single signal comprising, for each sample, a vector of three values, without loss of generality.

[0109] These convolution layers, which use convolution elements denoted Ci, and attention elements (SGM: sigmoid function), can be combined by cascade. The complexity increases with the cases presented, both from the point of view of memory occupation (number of filter coefficients) and the number of arithmetic operations they involve.

[0110] Case (a) presents a simple convolution that processes the NL latent value maps Li / L'i as input. The NL (according to the example, NL=3) maps undergo a simple convolution, denoted C. A filtering is performed using the convolution kernel, of size kxk (e.g., k=3). The (3x3) kernel is applied to the different positions of the maps to obtain the NC (according to the example, NC=3) components of the image (e.g., YUV) as output. The number of operations required is therefore 3*3*3*3 (= 81). Generally, the number of operations is NL*NC*k*k. NL*NC kernels of size k*k are used and must be stored and / or transmitted. In addition, NC biases must also be stored and / or transmitted.

[0111] Case (b) has the same complexity, but a residual structure: unlike case (a), the signal from the convolution is combined (added, subtracted, multiplied) with the original signal composed of the three intermingled latents. The memory footprint (number of operations) is identical to that of case (a), i.e. NL*NC*k*k operations, but this structure can facilitate the training of the neural network.

[0112] Case (c) presents a residual structure which treats in a differentiated way the NL (according to the example, NL=3) latent value maps: a map (L' o ) is used to generate the Y component, noted I Y , of the image. Its convolution is noted C1, its kernel is of size k1*k1. The other two maps (L'1 and L'2) are used to generate the U and V components of the image, noted I U and I V. Their convolution is denoted C2, its kernel is of size k2*k2. The memory footprint is smaller than when processing the 3 maps together, since the number of operations required is 1*1*k1*k1 for C1 and 2*2*k2*k2 for C2. In this case, it is necessary to store 3 biases. As explained in support of the, this structure is well suited to processing images of different resolutions.

[0113] Case (d) presents a structure comprising an SGM attention module, jointly processing the NL latent value maps. The memory footprint is larger since the number of operations required is NL*NC*k3*k3 for C3 and NL*1*k4*k4 for C4. In this case, it is necessary to store NC+1 bias. However, this structure improves the local processing of images, particularly of the contours present in the images. For C2: 2*2*k2*k2

[0114] Schematically represents convolution networks according to embodiments of the invention.

[0115] The left part of the figure shows a low-complexity differentiated processing that independently processes the three latent value maps. It is capable of processing maps of different resolutions, so each component of the image can have its own resolution, for example to manage a YUV 420 format for which the U and V components are of reduced size compared to the Y component. The convolution elements are denoted C5, C6 and C7.

[0116] The right part of the figure presents an alternative view of the differentiated processing of case (c). The convolution network is able to process YUV formats separately; the Y component is processed separately while the processing of the U and V components is linked. The convolution elements are denoted C8, C9, C10, C11 and C12.

[0117] The table below summarizes the number of operations required (MAC) and the number of values ​​stored (storage) for each of the cases presented in Figures 5 and 6. In this example, we consider that there are three latent value maps (NL=3), three outputs (NC=3) and all convolution kernels are of size 3x3 (k=k1=k2=...=3).

[0118] figureMACstorage5-a81845-b81845-c45485-d1081126 right45486 left2730

[0119] This is a flowchart representing an example of a coding process that can be implemented by the coding device of the.

[0120] According to this embodiment, the signal is a two-dimensional image, each sample to be coded is therefore a pixel P n of coordinates (x n , y n ).

[0121] Encoding takes place in two main phases:

[0122] In a first phase, called the construction phase, learning is carried out in order to determine, for an input signal I(P n ), the values ​​of FM cards i and W parameters k of the synthesis module and V k of the convolutional neural network(s) to optimize a cost function. Learning is for example carried out by gradient descent, followed by an update of the values ​​of the FM feature maps i , and optionally synthesis and convolution neural networks. As is known in the state of the art, the cost function can be of the rate-distortion type, or rate, or distortion, or perceptual. To measure the rate R, it is necessary to simulate the coding of FM cards i , then measure the associated coding rate (the size of the stream B1). According to one embodiment, the coding of the parameters W is not simulated k and / or V kbecause their influence is less important than that of the feature maps. According to one embodiment, the coding of the parameters W is also simulated k and / or V k and the associated flow rate (the size of the flow B2) is measured. To measure the distortion D, it is necessary to simulate the coding then the decoding of at least part of the image I, to obtain at least one pixel P' n resulting from a simulation of coding then decoding, then measuring the difference between this part of the image I (P n ) as input and a corresponding part of image I (P' n ) encoded then decoded.

[0123] Then during a second phase, called the coding phase, the FM cards i and the V parameters k and possibly W k are encoded to produce the FMc encoded values i and Vc k (and possibly Wc k) before transmission or storage. They constitute the compressed representation of the input signal I (P n ).

[0124] We will now describe the steps of a method according to one embodiment of the invention.

[0125] During a step E20, a signal I (P n ) to be coded, comprising a plurality of N samples P n , is provided as input to the process.

[0126] During a step E21, the M FM characteristic maps i are initialized. Subsequently, the parameters W k of the MLP synthesis module, the values ​​of the FM cards i , and the parameters V k of (at least one) convolutional neural network must be optimized during the construction phase.

[0127] According to one embodiment, the FM cards i are of the same resolution as the input signal I (P n) and therefore each have the same number of values ​​N as there are samples P n to be coded. According to one embodiment, the FM cards i are of resolution less than or equal to that of the input signal I (P n ) and therefore comprise, for at least one of them, a number of values ​​to be coded less than N. According to a variant, the first FMi card is at the resolution of the images and each following card is at a resolution half that of the previous one. According to one embodiment, several FMi cards have the same resolution, lower than that of the input signal I (P n ).

[0128] According to one embodiment, the FM cards i are initialized by predefined constant values. According to another embodiment, the feature maps are initialized by a set of random real values.

[0129] In one embodiment, the structure and parameters of the neural networks are initialized during this step. These parameters are subsequently updated, or refined, during the construction phase, during subsequent iterations of the method. According to one embodiment, the parameters of the synthesis module and / or the convolutional neural network are initialized by predefined values ​​known to give a satisfactory cost function value (for example, following training on a corpus of images). According to another embodiment, the parameters of the synthesis module and / or the convolutional neural network are initialized by a set of random values.

[0130] FM Feature Cards i and the parameters of the neural networks are subsequently updated, or refined, during a step E22, by the MAJ update module of the encoder during its learning.

[0131] During an E23 step, the FM cards i are encoded by the encoder's FMC module. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is an actual coding and the encoded values ​​constitute the B1 stream. The simulation can be identical to the actual coding but it can also be different (for example, simplified). For this coding, we use, for example, an entropy coding of the quantized values ​​of the feature map(s).

[0132] In one embodiment, the FM cards i are encoded in order (FM1, FM2,…, FM4), and the variables of each map in a predefined order, for example lexicographic. Each map undergoes entropy coding. The entropy coding produces a compressed stream B1 whose flow rate is subsequently measured during a step E28.

[0133] During a step E24, according to one embodiment, the M FM cards iare quantified, transformed, and data is extracted from them by the XTR module.

[0134] According to one embodiment, at least one FM card i is of lower resolution than the images of the sequence to be coded and the transformation operation involves oversampling so that the FMt card i transform has the same number of samples as the images in the sequence. Oversampling consists of adding values ​​to the FMt maps i to achieve the resolution of the images in the input sequence. It can be simple (by nearest neighbor replication) or include interpolation (linear, polynomial, filtering, etc.). It can be achieved by an upsampling neural network.

[0135] According to one embodiment, the XTR module performs an extraction of values ​​in the transformed characteristic maps FMt i , for a sample P nto be encoded, based on its coordinates in the input signal. The extracted values ​​constitute the vector Z n . Z n is a J-tuple, that is, it has J elements, or data z i . The vector Z n of index n refers to the characteristic vector of the pixel P' n .

[0136] In one embodiment, Z n is a J-tuple (z1, z2,…, z J ), consisting of the values ​​of the FM cards i located at the coordinates (x n , y n ) of the current pixel P n . In one embodiment, Z n is a J-tuple constructed from values ​​taken from FMi charts at coordinates that may be different depending on the chart. For example, if the FM charts iare at different resolutions because they have been downsampled, the coordinates are adapted (by scaling) to match the resolution of each map. In one embodiment, Z n is a J-tuple constructed from values ​​taken from FM maps i by applying the processing to one or more values ​​of the maps, for example filtering the values ​​close to the target value in a map. For example, for a current sample P n in an FMi map which is at the same resolution as the input signal, we can extract the values ​​located at the coordinates (x n , y n ), (x n -1, y n ), (x n , y n -1) and (x n -1, y n -1) and apply processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (z i ) of element i of vector Z n relating to this FM cardi . According to another example, in an FM card i which is at half the resolution of the input signal, we can consider the values ​​located at the coordinates (x n / 2, y n / 2), (x n / 2-1, y n / 2), (x n / 2, y n / 2-1) and (x n / 2-1, y n / 2-1) and apply processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (z i ) of element i of vector Z n relating to this FM card i . The samples to be coded are for example processed in sequential order, from n=1 to n=N.

[0137] In step E25, the feature maps are processed by the synthesis module to generate latent value maps. The synthesis module is, for example, a neural network, or a filter bank, or a linear inverse transformation (wavelet, DCT, etc.) defined by its parameters W k, capable of processing the transformed characteristic maps as input, to generate as output a set of NL latent maps L j with 1 <j <=NL.

[0138] According to one embodiment, the vector Z n is processed by an MLP synthesis neural network to generate a second ZL vector as output n containing NL latent values ​​for sample P n to be coded. According to one embodiment, NL is equal to 3 and the output vector is a triplet to be inserted into three maps L1, L2 and L 3. In another embodiment, NL is equal to 4 and the output vector is a quadruple to be inserted into four L1–L4 maps of latent values ​​at the pixel coordinates. In another embodiment, NL is between the number M of FM maps iand the number of components in the image, M≤NL≤3 for the case of NC=3 components. It is also possible to organize the latent values ​​differently, for example by forming a single signal comprising, for each sample, a vector of NL scalar values, without loss of generality. The latent value maps represent an intermediate signal before convolutional processing.

[0139] During a step E26, the (at least one) MPP convolutional neural network is applied at the output of the MLP module to the latent value maps (L j ) to obtain decoded samples P' n , for example vectors representative of the samples P nà coder, selon un mode de réalisation le triplet (R, G,B) de l’échantillon P’n(l’échantillon Pncodé puis décodé).

[0140] During a step E27, the parameters V k and W ksynthesis and convolution modules are quantized and coded. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is an actual coding and the coded values ​​constitute the B2 stream. The simulation can be identical to the actual coding but it can also be different (for example, simplified). Any known technique can be used for this purpose, for example the neural network coding standard proposed by the MPEG-7 part 17 standard, also called NNR (Neural Network Representation). Note that in this case, it is necessary to choose the amount of degradation that the coding brings to the parameters V k and W k .

[0141] During step E28, a performance measure is evaluated.

[0142] For this purpose, the coding simulation rates associated with the feature maps (simulation of the B1 flow by coding the FM maps i) and optionally to the parameters of the synthesis and convolution modules (simulation of the B2 flow by coding the V parameters k / W k ) are measured.

[0143] According to one embodiment, the cost function is of the rate-distortion type, denoted (D+L*R), where D is for example the squared error measured between the input signal and the decoded signal (or the error measured on a subset of samples of the signal). According to another example, D is calculated from a perceptual function such as the SSIM (for Structural SIMilarity), or MSSSIM (for Multi-scale Structural SIMilarity). According to one embodiment, R is the simulated rate of the stream B1; according to another embodiment, R is the total rate used to code this image or sequence of images, i.e. the sum of the simulated rates of B1 and B2. L is a parameter which adjusts the rate-distortion trade-off. Other cost functions are possible.

[0144] As long as the cost function has not reached its minimum, the performance measurement is not satisfactory, and the process is repeated from step E22. This minimization can be performed by a mechanism known as gradient descent with parameter updates during step E22. If the cost function has reached its minimum, training stops. If an encoded version corresponding to the last simulation of the parameters of the synthesis and convolution modules (V k / W k ) and feature maps (FM i ) is available, the streams B1 and B2 can be made up of them. According to another embodiment, the effective coding of the updated parameters of the synthesis and convolution modules (V k / W k ) and values ​​of the characteristic maps (FM i ) is performed at this step to produce the encoded parameters Vc k / Toilet k and FMc iwhich constitute streams B2 and B1. The streams B1 and B2 can be concatenated to produce a final stream. According to one embodiment, the stream B2 of the coded parameters of the synthesis and convolution modules is stored or transmitted before the stream B1, so that it can be decoded before the stream B1.

[0145] This is a flowchart representing an example of a decoding method that can be implemented by the decoding device of the.

[0146] During a step E30, the streams B1 and B2 are extracted from the encoded stream. They respectively contain the coded representations of the FMc maps i and parameters of the Wc synthesis and convolution modules k and Vc k .

[0147] During a step E31, the M FMd cards i are generated by decoding FMc values i. For this decoding, a decoding technique corresponding to that used by the encoder is used, for example entropy (de)coding. In one embodiment, the FMd cards i are decoded in order (FMd1, FMd2,… FMd4), and the values ​​of each card in a predefined order, for example lexicographically. The Wd parameters k and Vd k synthesis and convolution modules are generated by decoding Wc values k and Vc k of the B2 stream. Any known decoding technique corresponding to the coding technique used by the encoder can be used for this purpose. The (at least one) MPP convolution neural network is similar to the MPP synthesis network, i.e. it has the same structure and the same parameters, except for the coding, which can be done with or without losses. The same applies to the synthesis module, which can be implemented as a neural network.

[0148] According to one embodiment, the B2 stream is decoded before the B1 stream, in order to have the parameters of the synthesis and convolution modules before starting to decode the samples.

[0149] According to embodiments as described for the encoder: the FMd cards i are of the same resolution as the signal to be reconstructed I (Pd n ) and therefore each have the same number of values ​​N as there are samples Pd n to decode; FMd cards i are of resolution less than or equal to that of the signal and therefore comprise, for at least one of them, a number of values ​​to be decoded less than N. According to a variant, the first FMdi card is at the resolution of the images and each subsequent card is at a resolution half that of the previous one. According to one embodiment, several FMdi cards are of the same resolution, lower than that of the signal.

[0150] During a step E33, according to embodiments, the M FMd cards i are dequantized, transformed, and data is extracted from them by the XTR module.

[0151] According to one embodiment, at least one FMd card i is of lower resolution than the decoded images and the transformation operation involves oversampling so that the FMt' card i transformed has the same number of samples as the images in the sequence. Oversampling consists of adding values ​​to the FMt' maps i to achieve the resolution of the images in the input sequence. It can be simple (by nearest neighbor replication) or include interpolation (linear, polynomial, filtering, etc.). It can be achieved by an upsampling neural network.

[0152] According to one embodiment, the XTR' module performs an extraction of values ​​in the transformed characteristic maps FMt' i , for a Pd sample n to be decoded, depending on its coordinates in the signal. The extracted values ​​constitute the vector Zd n . Zd n is a J-tuple, that is, it has J elements, or data zd i . The vector Zd n of index n refers to the characteristic vector of the pixel Pd n .

[0153] This step is similar to step E24 which was described for the encoder in support of the, and the embodiments apply. In particular: In one embodiment, Zd n is a J-tuple (zd1, zd2,…, zd J ), consisting of the values ​​of the FMt' cards i / FMd i located at the coordinates (x n , y n ) of the current pixel Pd n . In one embodiment, Zd nis a J-uplet constructed from values ​​taken from the FMt' maps i / FMd i to coordinates that may be different depending on the maps. For example, if the FMt' maps i / FMd i are at different resolutions because they have been downsampled, the coordinates are adapted (by scaling) to match the resolution of each map. In one embodiment, Zd n is a J-uplet constructed from values ​​taken from the FMt' maps i / FMd i by applying the processing to one or more values ​​of the maps, for example filtering the values ​​close to the targeted value in a map. For example, for a current sample Pd n in an FMdi map which is at the same resolution as the image signal to be decoded, we can extract the values ​​located at the coordinates (x n , y n ), (x n -1, y n ), (x n , yn -1) and (x n -1, y n -1) and apply processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (zd i ) of element i of vector Z n relating to this FMt' card i / FMd i . According to another example, in an FMt' card i / FMd i which is at half the resolution of the signal, we can consider the values ​​located at the coordinates (x n / 2, y n / 2), (x n / 2-1, y n / 2), (x n / 2, y n / 2-1) and (x n / 2-1, y n / 2-1) and apply processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (zd i ) of element i of vector Zd n relating to this FM card i . The samples to be decoded are for example processed in sequential order, from n=1 to n=N

[0154] In a step E34, the feature maps are processed by the MLP' synthesis module to generate latent value maps. The synthesis module is for example a neural network, or a filter bank, or a linear inverse transformation (wavelet, DCT, etc.) defined by its parameters Wd k , capable of processing the transformed characteristic maps as input, to generate as output a set of NL latent maps L' j with 1 <j <=NL.

[0155] This step is similar to step E25 which was described for the encoder in support of the, and the embodiments apply. In particular, according to one embodiment, the vector Zd n is processed by an MLP' synthesis neural network to generate a second ZLd vector as output n containing NL latent values ​​for sample Pd nto be decoded. According to one embodiment, NL is equal to 3 and the output vector is a triplet to be inserted into three maps L'1, L'2 and L' 3. According to another embodiment, NL is equal to 4 and the output vector is a quadruple to be inserted into four maps L'1–L'4 of latent values ​​at the pixel coordinates. According to another embodiment, NL is between the number M of maps FMd i and the number of components in the image, M≤NL≤3 for the case where NC=3 components. It is also possible to organize the latent values ​​differently, for example by forming a single signal comprising, for each sample, a vector of NL scalar values, without loss of generality. The latent value maps represent an intermediate signal before convolutional processing.

[0156] During a step E35, the (at least one) convolutional neural network MPP' is applied at the output of the MLP' module on the latent value maps (L' j) to obtain decoded Pd samples n, for example vectors representative of Pd samples n to decode, according to one embodiment, the triplet (R, G, B) of the sample Pd n . This step is similar to step E26 which was described for the encoder in support of the.

[0157] When all the latent value maps have been processed, the decoded signal corresponding for example to image I (Pd n ) is available.

[0158] Illustratively represents a decoding method used in one embodiment of the invention.

[0159] In this embodiment, the FMd cards i are 5 in number. In a preferred mode, there are 7.

[0160] In this embodiment, the first map FMd1 has the same resolution as image I, and therefore has WxH variables, where W represents the width of the image in pixels, and H its height. The second map FMd2 has half the resolution (in each dimension) of map FMd1. Map FMd3 has half the resolution (in each dimension) of map FMd 2. The FMd4 and FMd5 maps have half the resolution (in each dimension) of the FMd3 map. This structure allows to reduce the number of variables in the feature maps, which facilitates decoding while minimizing the coding cost.

[0161] The XTR' module dequantizes the values ​​of the characteristic maps if necessary.

[0162] The feature maps are then processed by the XTR' module, which, according to this embodiment, oversamples them to make them the same resolution as the image to be decoded. (FMd2 is oversampled by a factor of 2 in each dimension, FMd3 is oversampled by a factor of 4 in each dimension, etc.)

[0163] Then, the XTR' module extracts, for each sample position, a vector Zd n in the form of a 5-tuple (zd1...zd5) made up of the values ​​of the FMd cards i transforms located at the coordinates (x n , y n ) of the current pixel Pd n . The vector Zd n is processed by the MLP' neural network to generate a ZPd vector as output n , according to the example a quadruplet representative of the sample Pd n to decode, which is reinjected into the latent value maps L' i at the coordinates (x n , y n ) in the components.

[0164] According to this example, the latent value maps L' i there are 4 of them.

[0165] The 4 latent value maps are processed by the MPP' convolution module, to produce as output, according to this example, the 3 components Rd, Gd and Bd of the decoded image.

Claims

Method of coding at least one image (I(P n )) comprising the following steps: - a step of constructing said at least one coded then decoded image, comprising the sub-steps of: - construction (E21, E22) of a group of (M) characteristic maps (FM i ) ;- processing (E24, E25) of said group of characteristic maps by a synthesis module (MLP), defined by synthesis parameters (W k ), to provide a group of (NL) latent value maps (L j ), the number of latent value maps being less than or equal to the number of feature maps;- processing (E26) of said group of latent value maps by at least one convolution neural network (MPP), defined by convolution parameters (V k ), to provide said at least one coded then decoded image (I(P' n)) ;- updating (E22, E28) at least one value of one of said feature maps and / or at least one synthesis parameter and / or at least one convolution parameter, as a function of a coding performance measurement;- a coding step (E23, E27) of said group of feature maps (FM i ) and said set of synthesis and convolution parameters (W k / V k ). Coding method according to claim 1, characterized in that said at least one image comprises a plurality of samples (P n ) to be coded, and in that the step of processing said group of feature maps (FM i ) to provide a group of latent value maps (L i ) includes the following sub-steps: - for at least one sample, called the current sample (P n ), of said at least one image to be coded, associated with a position (x n , y n) :- construction (E24) of a characteristic vector (Z n ) from said feature maps (FM i ), depending on said position (x n , y n ) of said current sample (P n ) ;- processing (E25) of said characteristic vector (Z n ) by said synthesis module (MLP), to provide a latent vector, representative of a latent value (ZL n ) of the current sample; - construction of the group of latent value maps (L i ) from said at least one latent vector (ZL n ) ; Method for decoding at least one image (I(Pd n )) from a binary stream (B1, B2), comprising the following steps:- decoding (E31) of a group of (M) characteristic maps (FMd i ) ;- obtaining (E30, E32) a set of synthesis parameters (Wc k) representative of a synthesis module (MLP');- obtaining (E30, E32) a set of convolution parameters (Vc k ) representative of at least one convolutional neural network;- processing (E33, E34) of said group of feature maps by said synthesis module (MLP') to provide a group of latent value maps (L' j ), the number (NL) of latent value maps being less than or equal to the number (M) of feature maps;- processing (E35) of said group of latent value maps by said at least one convolutional neural network (MPP) to provide said at least one decoded image (I(Pd n )). Method for decoding a signal according to claim 3, characterized in that said at least one image comprises a plurality of samples (Pd n) to be decoded, and in that the step of processing said group of feature maps to provide a group of latent value maps comprises the following sub-steps:- for at least one sample, called the current sample (Pd n ), of said at least one image to be decoded, associated with a position (x n , y n ) :- construction (E33) of a characteristic vector (Zd n ) from the feature maps, as a function of said position (x n , y n ) of said current sample, and:- processing (E34) of said characteristic vector (Zd n ) by said synthesis module (MLP') to provide a latent vector (ZLd n ), representative of a latent value of the current sample;- construction of the group of latent value maps (L' i ) from said at least one latent vector (ZLd n ). Decoding or coding method according to one of the preceding claims, characterized in that said synthesis module (MLP, MLP') comprises a multi-layer perceptron. Decoding or coding method according to one of the preceding claims, characterized in that said at least one convolution network (MPP, MPP') comprises at least one convolution layer (C, C1, C2, C3, C4, C5). Decoding or coding method according to one of the preceding claims, characterized in that the number (NL) of latent value maps is equal to the number of components (NC) of said at least one image. Decoding method according to one of claims 3 to 7, characterized in that at least part of the set of synthesis and / or convolution parameters (Wc k , Vc k ) is obtained from data of said bitstream. Decoding method according to one of claims 3 to 8, characterized in that at least part of the set of synthesis and / or convolution parameters (Wc k , Vc k ) is obtained from predetermined parameters. Decoding method according to one of claims 3 to 9, characterized in that at least part of the set of synthesis and / or convolution parameters (Wc k , Vc k ) is decoded based on reference parameters. Decoding method according to one of claims 3 to 10, characterized in that said at least one convolution network comprises at least two convolution networks. Decoding method according to the preceding claim, for which the binary stream comprises an indicator indicating the use of at least one of said networks. Decoding method according to one of claims 6 to 12, characterized in that said at least one image comprises at least two dimensions (x,y), and said convolution layer uses a kernel separable according to said at least two dimensions. Device for coding at least one image (I(P n )), characterized in that said device is configured to implement:- a step of constructing said at least one coded then decoded image, comprising the sub-steps of:- constructing (INIT, MAJ) a group of (M) characteristic maps (FM i ) ;- processing (XTR, MLP) of said group of characteristic maps by a synthesis module (MLP) defined by synthesis parameters (W k ) to provide a group of latent value maps (L j), the number of latent value maps (NL) being less than or equal to the number of feature maps (M);- processing (MPP) of said group of latent value maps by a convolution neural network (MPP) defined by convolution parameters (V k ) to provide said at least one coded then decoded image (I(P' n )) ;- updating (EVAL, MAJ) at least one value of one of said feature maps and / or at least one synthesis parameter and / or at least one convolution parameter, as a function of a coding performance measurement;- a coding step (FMC, NNC) of said group of feature maps (FM i ) and said set of synthesis and convolution parameters (W k / V k ). Device for decoding at least one image (I(Pd n)), from a binary stream, characterized in that said device is configured to implement:- decoding (FMD) of a group of (M) characteristic maps (FMd i ) ;- obtaining (NND) a set of synthesis parameters (Wc k ) representative of a synthesis module (MLP');- obtaining (NND) a set of convolution parameters (Vc k ) representative of at least one convolutional neural network;- processing (XTR', MLP') of said group of feature maps by said synthesis module (MLP') to provide a group of (NL) latent value maps (L' j ), the number of latent value maps being less than or equal to the number of feature maps;- processing (MPP ) of said group of latent value maps by said at least one convolutional neural network to provide said at least one decoded image (I(Pd n )). Computer program comprising instructions for carrying out the steps of an encoding or decoding method according to claim 1 or 3 when said program is executed by a computer.

Images