Method and device for encoding and decoding images.

Abstract

Method and device for encoding and decoding images. The invention relates to a method and device for encoding and decoding images comprising a plurality of samples to be encoded or decoded. The decoding method comprises the following steps: - decoding (E31) a group of (M) feature maps (FMdi); - obtaining (E32) a set of synthesis parameters (Wck) representative of a synthesis module (MLP'); - obtaining (E32) a set of convolution parameters (Vck) representative of a convolutional neural network; - processing (E33, E34) said group of feature maps by the synthesis neural network (MLP') to provide a group of (NL) latent value maps (L'j), the number of latent value maps being less than or equal to the number of feature maps; .- processing (E35) of said group of latent value maps by the convolutional neural network (MPP") to provide said at least one decoded image (I(Pdn)) Figure for the abstract: Fig. 2.

Description

Title of the invention: Method and device for coding and decoding images. Prior art

[0001] The invention relates to the general field of coding digital image sequences.

[0002] Digital images or videos are generally subject to source coding aimed at compressing them in order to limit the resources required for their transmission and / or storage. There are many coding standards, such as the ITU / MPEG standards (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) as well as their extensions (MVC, SVC, 3D-HEVC, etc.).

[0003] As video formats continue to evolve to compress more and more and adapt to the variety of expected formats and communication networks, the possibilities for prediction are becoming ever greater and the classic coding and decoding algorithms very complex.

[0004] In addition to these classic approaches proposed by compression standards (MPEG, ITU), approaches based on artificial intelligence, and in particular neural ones, tend to develop.

[0005] Some of these neural approaches can be seen as a simple extension of the notion of competition of the aforementioned compression techniques, such as the competition of prediction and transformation mode in video coding.

[0006] Other approaches use the concept of "auto-encoder". Auto-encoders are learning algorithms based on artificial neural networks, which allow a new representation of a data set to be constructed. The architecture of an auto-encoder consists of two parts: the encoder and the decoder. The encoder consists of a set of layers of neurons, which process the data in order to construct new representations called "encoded", also called "latent representations". In turn, the layers of neurons in the decoder receive these representations and filter them in order to try to reconstruct the initial data. The differences between the reconstructed data and the initial data make it possible to measure the error made by the auto-encoder. Training consists of modifying the parameters of the auto-encoder in order to reduce the reconstruction error measured on the different samples of the data set.The performance of such autoencoder-based systems comes at the cost of a considerable increase in memory footprint and complexity compared to conventional approaches such as those proposed by compression standards. They . can have millions of parameters and may require on the order of a million MACs (multiply-accumulate) to decode a single pixel. This makes such decoders much more complex than conventional decoders, which could hamper the adoption of learning-based compression. In addition, the neural networks used at the encoder and decoder are different. US 2022 / 0385907 describes such an autoencoder system.

[0007] More recently, a simple image encoding technique based on a neural network has been described in the article "Compression with Implicit Neural representations" by Emilien Dupont et al. (arXiv:2103.03123). The proposed encoding technique consists of fitting a neural network to an image, quantizing the network weights and transmitting them. At decoding time, the neural network is evaluated at each pixel position to reconstruct the image. However, such a technique remains inefficient in terms of compression.

[0008] Document US 2022 / 0385907 also describes a coding and decoding system based on the use of such implicit neural representations.

[0009] However, there is a need for a solution that allows a sequence of images to be coded / compressed in a simple and efficient manner. Statement of the invention

[0010] The invention relates to a method for coding at least one image comprising the following steps: - a step of constructing said at least one coded then decoded image, comprising the sub-steps of: - construction of a group of characteristic maps; - processing said group of characteristic maps by a synthesis module, defined by synthesis parameters, to provide a group of latent value maps, the number of latent value maps being less than or equal to the number of characteristic maps; - processing said group of latent value maps by at least one convolution neural network, defined by convolution parameters, to provide said at least one coded and then decoded image; - updating at least one value of one of said feature maps and / or at least one synthesis parameter and / or at least one convolution parameter, based on a coding performance measurement; - a step of coding said group of feature maps and said set of synthesis and convolution parameters.

[0011] The invention also relates to a method for decoding an image at least from a binary stream, comprising the following steps: - decoding a group of feature maps; - obtaining a set of synthesis parameters representative of a synthesis module; - obtaining a set of convolution parameters representative of at least one convolution neural network; - processing said group of feature maps by said synthesis module to provide a group of latent value maps, the number of latent value maps being less than or equal to the number of feature maps; - processing said group of latent value maps by said at least one convolutional neural network to provide said at least one decoded image.

[0012] For the purposes of the invention, encoding, or “coding”, means the operation which consists in representing a set of samples, or pixels, of the image or of the plurality of images, in a compact form carried for example by a digital binary train. Decoding means the operation which consists in processing a digital binary train to restore decoded samples of the image or of the plurality of images.

[0013] By “image” we mean a two-dimensional digital image comprising one or more components (color, depth, views, etc.)

[0014] By "at least one image" is meant a plurality of ordered two-dimensional images, for example temporally in the case of a video. According to another example, the images may be views of the same scene represented in multi-views. According to another example, the images may be a plurality of temporal and multi-view images (immersive video).

[0015] By "sample" or "pixel" is meant a value or a set of values ​​taken from an image. Sampling a signal produces a sequence of discrete values ​​called samples. In the case of an image signal, the sample is called a pixel, which can be, for example, a color pixel traditionally represented by a triplet of values, each value belonging to a component, for example (R,G,B) or (Y,U,V). The position of the sample can be identified by its abscissa (x) and ordinate (y) coordinates in the image.

[0016] By "feature map" is meant a part of an abstract representation of an image or a sequence of images, comprising a plurality of variable scalar data, which are also called characteristic values, for example real numbers, or integers, signed or unsigned. The abstract representation of an image or a sequence may therefore comprise several feature maps. The abstract representation of one or more images may also correspond to a single feature map.

[0017] By "latent value map" is meant a part of an abstract representation of an image or a sequence of images, comprising a plurality of variable scalar data, also called latent values, for example real numbers, or integers, signed or unsigned. The abstract representation of an image or a sequence can therefore comprise several latent value maps. The abstract representation of one or more images can also correspond to a single latent value map. These maps result from the processing of the characteristic maps by a synthesis module, for example a synthesis neural network.

[0018] By "synthesis module" we mean for example a neural network, or a filter bank, or a linear inverse transformation (wavelet, DCT, etc.) defined by its Wk parameters, capable of processing the transformed characteristic maps as input, to generate as output a set of NL latent value maps.

[0019] By "synthetic neural network" and "convolutional neural network" is meant a neural network such as a convolutional neural network, a multi-layer perceptron, etc. The neural network is defined for example by a plurality of layers of artificial neurons and by a set of activation, weighting and addition functions (for example, a layer can calculate y = f(Ax+b), where y and b are vectors of dimension N, x is a vector of dimension M, A is a matrix of dimension MxN, and f is the activation function).

[0020] By "parameter of the neural network" is meant one of the values ​​which characterizes the neural network, or the plurality of neural networks, for example a weight associated with one of the neurons (filter coefficient, values ​​of a matrix, weighting, bias, value affecting the operation of the non-linearity, etc.)

[0021] By "generation of said at least one image" is meant a step which aims to reconstruct the decoded samples of the image or of the plurality of images.

[0022] By "performance measurement" is meant a measurement between at least one value of a sample to be coded and a decoded value of said sample. The measurement can evaluate for example a distortion, or a perceptual error. It can be carried out on a sample or a plurality of samples (for example, the current samples, or the current images, etc.). The measurement can also include a measurement of the flow rate, in particular associated with the coding of the neural network and / or the coding of the characteristic maps of the group. The measurement can be a joint measurement between the flow rate and the distortion through their weighting. As is well known in the state of the art, the value of this measurement is generally minimized until a target value is reached.

[0023] By “construction step” is meant a step which aims to construct the coded then decoded values ​​of the pixels of the image or of the plurality of images, and the parameters representative of the image, in particular the values ​​of the characteristic maps and the parameters of the neural networks, before their actual coding. The sub-steps construction tests can be repeated as many times as necessary to obtain an acceptable performance measure.

[0024] Generally speaking, it is considered that the steps of a coding or decoding method should not be interpreted as being linked to a notion of temporal succession. In other words, the steps may be carried out in a different order than that indicated in the independent coding or decoding claim, or even in parallel.

[0025] The coding method according to the invention carries out a construction of the coding parameters, from an input image or sequence of images, by training neural networks on feature maps then latent value maps which can be at the resolution of the input images, or at a lower resolution. During the training, or construction, the parameters of the neural network and the values ​​of the feature maps and the latent value maps are updated according to a performance measurement, for example of the rate-distortion type. When the training is finished, that is to say that the performance measurement obtained is satisfactory, the actual coding of the parameters of the neural networks and the values ​​of the feature maps can be carried out and stored or transmitted to the decoder. The latent value maps are neither coded nor transmitted.

[0026] Advantageously, the training process makes it possible to refine the parameters of the neural networks and / or the values ​​of the feature maps until an adequate representation in terms of performance is obtained, for example a desired balance between the generated bit rate and the distortion undergone by the input image or sequence of images being encoded. The training of the values ​​of the feature maps and the parameters of the neural networks can be joint. Advantageously, the encoding method according to the invention makes it possible to efficiently compress the signal.

[0027] Advantageously, the compression of the characteristic maps is effective since this allows it to be transported over networks or on storage media with reduced capacities.

[0028] Furthermore, the use of two sets of networks, synthesis then convolution, is advantageous in itself, because it allows the two networks to be trained separately to ensure better convergence. According to certain embodiments, it is possible to freeze one of the two networks and learn only one, so that only the parameters of the learned network will be transmitted to the decoder.

[0029] Furthermore, since the number of latent value maps is less than or equal to the number of feature maps, the number of operations to be performed by the convolutional neural network is reduced. This results in reduced complexity, since less data is processed. Advantageously, the decoding method is simple since it is sufficient to decode the feature maps associated with an image or to a group of images, and to obtain the synthesis and / or convolution neural networks, to reconstruct a decoded version of the current image or group of current images.

[0030] Such a synthetic or convolutional neural network can advantageously be of very simple structure with few parameters and few operations per sample.

[0031] According to embodiments of the coding or decoding method:

[0032] - Said at least one image comprises a plurality of samples to be decoded, and the step of processing said group of feature maps to provide a group of latent value maps comprises the following substeps: - for at least one sample, called current sample, of said at least one image to be decoded, associated with a position: - construction of a characteristic vector from the characteristic maps, as a function of said position of said current sample, and: - processing of said characteristic vector by said synthesis module to provide a latent vector, representative of a latent value of the current sample; - construction of the group of latent value maps from said at least one latent vector.

[0033] Advantageously according to this mode, the synthesis module is of the positional type; the chaining of a positional type synthesis network and a convolution neural network makes it possible to carry out local processing at the level of the synthesis network, then to correct it by the use of the convolution neural network.

[0034] - The synthesis module includes a multi-layer perceptron, or MLP. Such a The neural network is advantageously very simple in structure with few parameters. In addition, decoding can be performed sample by sample, progressively and on a particular block of the image.

[0035] - The convolutional neural network comprises at least one layer of convolution. Thus, non-localized processing of the image can be performed, which improves the consistency of the generated image by limiting noise and enhancing the contours present in the image.

[0036] - The number of latent value maps is equal to the number of components of the image. Advantageously, the complexity is limited by processing exactly the number of components of the output pixels (for example, three in the case of an RGB or YUV image). Alternatively, each of the latent value maps (corresponding, for example, to the Y, U, and V components) can be processed independently. This further limits the processing complexity, since there are fewer multiplication / addition operations at the convolution neural network level, and therefore a limitation of the memory space required by the encoder and decoder, as well as a lower cost of the convolution parameters.

[0037] - At least part of the set of synthesis and / or convolution parameters is obtained (resp. coded) from data of said binary stream (resp. coded in the binary stream); advantageously according to this mode, the parameters of the neural networks are transmitted in the stream. The synthesis and / or convolution module can be transmitted in a quantized form and coded in a compact form using any quantizer and entropy coder accessible to those skilled in the art. A format similar to that of the MPEG-7 NNR standard can be used.

[0038] - At least part of the set of synthesis and / or convolution parameters is obtained (resp. coded) from predetermined parameters. Advantageously, according to this mode, the parameters of the neural networks are accessible to the decoder, for example they can be stored in a storage memory (resp. coded in a storage memory). In particular, part of the synthesis and / or convolution module can be stored in a quantized form and coded in a compact form using any quantizer and entropy coder accessible to those skilled in the art. A format similar to that of the MPEG-7 NNR standard can be used.

[0039] - At least part of the set of synthesis and / or convolution parameters is obtained (resp. coded) as a function of reference parameters. Advantageously, according to this mode, the parameters of the neural network are decoded by taking into account reference parameters which can be stored in a memory (storage memory, network, etc.) accessible from the decoder. Thus, the storage space or the transmission rate on the network can be effectively reduced: certain parameters can be coded / decoded in a complementary manner, others not be coded / decoded at all, because they are available in the reference parameters. According to embodiment variants, which can be combined with each other: • the reference parameters can constitute a set of variants of a synthesis and / or convolution network: type of convolution, attention modules, etc. Thus, only these parameters need to be coded, inserted into the stream and decoded by the decoder; • certain synthesis and / or convolution parameters are coded in a complementary manner to the parameters of the reference network. Thus, the decoder will only need to decode these residual parameters and then add them, multiply them (or combine them in any other known way) with those of the reference neural network; • part of the parameters of the reference network is reused for the synthesis and / or convolution network. For example, a complete layer of the network can be copied from the reference network, the parameters of this layer are therefore neither encoded, nor transmitted, nor decoded; • a reference network indicator is transmitted in the flow. It is thus

[0040]

[0041]

[0042]

[0043]

[0044] possible to indicate a network to use for decoding from among a plurality of possible networks known to the decoder. - Said at least one convolutional neural network comprises at least two convolutional networks. Advantageously, the use of several cascaded networks makes it possible in particular to enhance the filtering and improve the consistency of the generated image by limiting its noise and improving its contours. In addition, each network can be learned individually or frozen independently. According to embodiment variants, which can be combined with each other: • the bitstream includes an indicator indicating the use of at least one of these networks. Thus, depending on the quality and complexity required, one or more networks can be used for decoding. The indicator indicates, for example, that network 1 is mandatory, but that networks 2 and 3, which are cascaded behind network 1, are optional. The decoder can then choose whether or not to use them: if it uses them, it will obtain better quality at the expense of complexity, and vice versa. • At least two of the convolution networks have common parameters. It is thus advantageous to transmit the characteristic parameters of one network, which can be used for the others, which reduces the transmission cost. The same network is thus shared. • At least two of the networks have an identical structure and only their weights differ in part or completely. The same calculation elements are thus shared. - At least one image has at least two dimensions, and said convolution layer uses a kernel separable according to said at least two dimensions. Advantageously, according to this mode, the transmission cost and complexity are reduced by the use of operations (for example, convolution, oversampling, etc.) that are separable, which require fewer operations than non-separable operations. Correlatively, the invention also relates to a coding device and a decoding device. The characteristics and advantages of the coding or decoding method apply in the same way to the coding or decoding device according to the invention and vice versa. The invention also relates to a computer program on a recording medium, this program being capable of being implemented in a computer or a coding or decoding device in accordance with the invention. This program comprises instructions adapted to the implementation of the corresponding method. This program can use any programming language, and be in the form of source code, object code, or intermediate code between code source and object code, such as in partially compiled form, or in any other desirable form.

[0045] The invention also relates to an information carrier or a recording medium readable by a computer, and comprising computer program instructions mentioned above. The information or recording carriers can be any entity or device capable of storing the programs. For example, the carriers can comprise a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a floppy disk or a hard disk, a DNA sequence, or a flash memory. Furthermore, the information or recording carriers can be transmissible media such as an electrical or optical signal, which can be conveyed via an electrical or optical cable, by radio link, by wireless optical link or by other means.

[0046] A program according to the invention can in particular be downloaded from an Internet-type network.

[0047] Alternatively, each information or recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of a method according to the invention. Brief description of the drawings

[0048] Other characteristics and advantages of the present invention will emerge from the description given below, with reference to the appended drawings which illustrate exemplary embodiments thereof which are not in any limiting nature.

[0049] [Fig-1] [Fig.l] schematically represents a coding device used in the scope of the invention;

[0050] [Fig.2] [Fig.2] schematically represents a decoding device used in the context of the invention;

[0051] [Fig.3] [Fig.3] schematically represents artificial synthesis and convolution neural networks used in one embodiment of the invention for coding and decoding.

[0052] [Fig.4] [Fig.4] schematically represents artificial synthesis and convolution neural networks used in another embodiment of the invention for coding and decoding.

[0053] [Fig.5] [Fig.5] schematically represents convolution networks according to embodiments of the invention.

[0054] [Fig.6] [Fig.6] schematically represents convolution networks according to embodiments of the invention.

[0055] [Fig.7] [Fig.7] is a flowchart representing an example of a coding method which can be implemented by the coding device of [Fig.l].

[0056] [Fig.8] [Fig.8] is a flowchart representing an example of a decoding method which can be implemented by the decoding device of [Fig.2].

[0057] [Fig.9] [Fig.9] illustrates a decoding method used in an embodiment of the invention. Description of the embodiments

[0058] [Fig.l] schematically represents a digital image coding device ENC.

[0059] The coding device ENC of [Fig.l] receives as input a succession of samples to be coded, denoted Pn, for example a set of image data denoted I(Pn). The image signal I(Pn) can represent a two-dimensional image, or a plurality of two-dimensional images (video, color components, stereoscopic, multiscopic components, etc.). Pn designates a sample n of the input signal comprising N samples. In one embodiment, the signal is a color image signal represented by means of at least one two-dimensional representation, such as a matrix of pixels, of width W and height H (with N = WxH) each pixel comprising a red (R), green (G), blue (B) component, or, alternatively, a luminance (Y) component and at least one chrominance (U and / or V) component of width Wu / v and height Hu / v. The location of each pixel is defined by its abscissa and ordinate (x and y) coordinates in the image.In one embodiment, the image is grayscale, represented using a two-dimensional representation, such as a matrix of pixels, each pixel having a grayscale component, or luminance. Generally, the number of components is denoted NC.

[0060] This ENC coding device comprises an INIT module for generating characteristic maps, an XTR module for processing and extracting data, an MLP synthesis module, an MPP convolution module corresponding to at least one artificial convolution neural network, an FMC module for coding characteristic maps, an NNC module for coding a neural network, an EVAL module for evaluating a performance function, and an MAJ update module.

[0061] The synthesis module used for coding, MLP, and the synthesis module used for decoding, MLP', are defined by an identical structure, comprising for example a set of filter banks, or at least one artificial neural network and a set of weights and activation functions associated respectively with the artificial neurons of the network concerned.

[0062] The (at least one) artificial convolutional neural network used for coding, MPP, and the artificial convolutional neural network used for decoding, MPP', are defined by an identical structure, comprising for example a plurality of layers of artificial neurons, and by a set of weights and activation functions associated respectively with the artificial neurons of the network concerned.

[0063] During the process of training, or construction, of the coding, that is to say as long as the step of evaluating a performance is not satisfactory, the coding modules (FMC, NNC) carry out a coding simulation, followed by a decoding, intended for the evaluation module. Subsequently, they carry out the effective coding of the values ​​of the M FM characteristic maps;, with 1 ​

[0064] Each feature map contains scalar data, in accordance with the definition of a feature map given previously, and there are M feature maps. It is however possible to organize the feature values ​​differently, for example by forming a single signal comprising, for each sample, a vector of M scalar values, without loss of generality. Even if the data are organized differently, it is considered in the remainder of the document that there are M feature maps. The maps coded by the FMC module for a signal, for example a current image (or a plurality of images) are denoted FMc;. The FMC coding module quantizes if necessary the values ​​of the feature maps by using a quantizer to generate an ordered collection of quantized values. Then the coding module compresses the quantized data, for example by using an entropy coder.

[0065] The XTR module performs processing and extraction of data from the feature maps. The XTR module may perform quantization of the values ​​of the feature maps by using a quantizer Q to generate an ordered collection of quantized values. The XTR module may alternatively obtain the quantized data by the FMC module. It is recalled that quantization of a value refers to mapping that value to a member of a discrete set of possible code symbols. For example, the set of possible code symbols may consist of integer values, and the quantization system performs simple rounding of a real value to an integer value. According to another example, the quantization consists of multiplication by a given value and then rounding.Then, depending on the embodiments, the XTR module can perform a transformation of the values ​​of at least one of the characteristic maps such as for example oversampling, interpolation, filtering, etc. In one embodiment, at the end of the transformation, a transformed characteristic map has the same resolution as the input signal. Then the XTR module performs a data extraction from the characteristic maps thus quantified and optionally. transformed, denoted FMf, to present the data to the MLP module.

[0066] The MLP module is a synthesis module, for example a neural network, or a filter bank, or an inverse linear transformation (wavelet, DCT, etc.) defined by its parameters Wk, capable of processing the transformed characteristic maps as input, to generate as output a set of NL latent maps Lj with 1 <j <= NL.

[0067] According to one embodiment:

[0068] - The XTR module performs an extraction of values ​​in the characteristic maps FMf transforms, for a sample Pn to be encoded, according to its coordinates in the input signal. The extracted values ​​constitute the vector Zn. Zn is a J-tuple, that is to say it has J elements, or data z;. The vector Zn of index n refers to the characteristic vector of the pixel Pn.

[0069] - The MLP module comprises a neural network defined by K parameters Wk, with 1 <k <=K, apte à traiter le vecteur Zn, ou J-Uplet, en entrée, pour générer en sortie un second vecteur contenant des valeurs latentes représentatives de l’échantillon Pn à coder. Le réseau de neurones de synthèse comprend selon un mode de réalisation un MLP, ou Multi Layer Perceptron, constitué d’une couche d’entrée adaptée au format d’entrée (le J-uplet), optionnellement une ou plusieurs couche(s) cachée(s), et une couche de sortie adaptée au format de sortie du vecteur de sortie, de manière générale un vecteur comportant NL éléments, chaque élément étant destiné à une carte de valeurs latentes. Selon un mode de réalisation, NL est égal à 3 et le vecteur de sortie est un triplet à insérer dans les trois cartes Lb L2 et L3 (par exemple, Y, U et V) de valeurs latentes aux coordonnées du pixel. Selon un autre mode de réalisation, NL est égal à 4 et le vecteur de sortie est un quadruplet à insérer dans quatre cartes Li - L4 de valeurs latentes aux coordonnées du pixel.According to another embodiment, NL is between the number M of FM cards; and the number of components of the image, M< NL< 3 for the case of three components.

[0070] According to one embodiment, the NL latent value maps comprise any real values.

[0071] Latent value maps represent an intermediate signal before convolutional processing.

[0072] The (at least one) MPP neural network is of the convolutional type, that is to say that it comprises at least one convolutional layer. It may include convolutional elements, comprising a residual structure or not and / or attention modules, as will be described later.

[0073] The Wk parameters of the synthesis module coded by the NNC module are noted Wck.

[0074] The parameters Vk of the convolutional neural network(s) coded by the NNC module are denoted Vck.

[0075] The MPP network is applied at the output of MLP on the latent value maps (Lj) to obtain decoded samples P'n. When all the samples Pn of the signal have been coded and then decoded, we have a reconstructed signal I(P'n), that is to say at least one image I comprising N decoded samples.

[0076] The EVAL module performs an evaluation and minimization of a coding performance. The evaluation function is for example of the rate-distortion type. It aims to minimize the differences between the input representation of the current image I(Pn) and its output representation I(P'n), while optimizing the quantity of data to be encoded, namely the encoded feature maps, the parameters of the synthesis module Wck, and the parameters of the convolution neural network(s) Vck. The minimization can be performed by gradient descent, or any other method within the reach of a person skilled in the art.

[0077] The MAJ module updates the values ​​of the FM maps to be encoded in the current image, based on the results of the performance function.

[0078] Once the encoder training is complete, the parameters of the neural networks are encoded, either lossless, in which case the decoder's MLP7MPP' neural network is identical to MLP / MPP, or lossy, in which case the MLP7MPP' network may be slightly different from MLP / MPP. The values ​​of the feature maps, FMi, are also encoded as FMc; coded data, for example via an entropy encoder.

[0079] [Fig.2] schematically represents a DEC decoding device.

[0080] The decoding device DEC of [Fig.2] receives as input a first group of encoded data organized into M FMc characteristic maps, the Wck parameters of a synthesis module and the Vck parameters of at least one convolution neural network.

[0081] This DEC decoding device comprises a neural network decoding module NND, a feature map decoding module FMD, an inverse transformation and data extraction module XTR', a synthesis module MLP', a convolution module MPP' corresponding to at least one convolution neural network.

[0082] The cards decoded by the FMD module, numbering M, are noted FMd;.

[0083] The parameters decoded by the NND module are noted Wdk and Vdk. In the case where the synthesis module does not include a neural network, a different decoder can be used to decode the Wdk parameters. The Vdk and Wdk parameters are identical to the Vk and Wk parameters of the encoder if the encoding was done losslessly.

[0084] The XTR' module performs processing and extraction of data from the decoded feature maps FMd;. The XTR' module optionally performs an inverse quantization corresponding to the quantization performed at the encoder. The inverse quantization is not necessary if the quantizer Q of the encoder has performed a simple rounded of the real values ​​submitted to it. Inverse quantization is also not necessary if the neural network is able to take into account a quantization of its input data. Otherwise, the decoder performs the inverse operation of the quantizer Q. Then, depending on the embodiments, the XTR' module can perform a transformation of the values ​​of at least one of the feature maps such as oversampling, interpolation, filtering, etc. In one embodiment, at the end of the transformation, a transformed feature map has the same resolution as the input signal. Then the XTR' module performs a data extraction from the feature maps thus quantized and optionally transformed, denoted FMf;, to present the data to the MLP module.

[0085] As previously described in support of [Fig. 1], the MLP' module is a synthesis module, for example a neural network or an inverse linear transformation (wavelet, DCT, etc.), defined by its Wdk parameters, capable of processing the transformed characteristic maps as input, to generate as output a set of NL latent maps L'j with 1 <j < NL.

[0086] In a preferred embodiment:

[0087] - The XTR' module is identical to the XTR module in [Fig.l]. It performs an ex traction of values ​​of the M FMf characteristic maps; transformed, to construct the vector Zdn taking the form of a J-tuple for a sample Pn to be decoded, according to its coordinates in the signal to be decoded. In one embodiment, J=M.

[0088] - The MLP' module includes a neural network defined by K parameters Wdk, able to process the vector Zdn, or J-Uplet, as input, extracted from the feature maps, to generate as output a second vector ZLdn of latent values ​​representative of the sample Pn to be decoded. The synthetic neural network comprises, according to one embodiment, an MLP, or Multi Layer Perceptron, consisting of an input layer adapted to the input format (the J-uplet), optionally one or more hidden layer(s), and an output layer adapted to the output format of the output vector, generally a vector comprising NL elements, each element being intended for a map of latent values. According to one embodiment, NL is equal to 3 and the output vector is a triplet to be inserted into the three maps L\, L'2 and L'3 of latent values ​​at the pixel coordinates. According to another embodiment, R is equal to 4 and the output vector is a quadruple to be inserted into four maps L\ - L'4 of latent values ​​at the pixel coordinates.According to another embodiment, R is between the number M of FMd cards; and the number of components of the image or images to be decoded M< R< 3 for the case of three components.

[0089] Latent value maps represent an intermediate signal before convolutional processing.

[0090] The (at least one) MPP' neural network is of the convolutional type, that is to say that it comprises at least one convolutional layer. It may include convolutional elements, comprising a residual structure or not and / or attention modules, as will be described later. At the output of the MPP' module, the reconstructed image or plurality of images, I(Pdn), is available.

[0091] The decoding device DEC can be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the modules mentioned above can then be realized by the cooperation of the processor and computer program instructions stored in the aforementioned memory and designed to carry out the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0092] [Fig.3] schematically represents artificial synthesis and convolution neural networks used in one embodiment of the invention for coding and decoding.

[0093] According to this embodiment, the synthesis module comprises an MLP / MLP' neural network and the convolution module comprises an MPP / MPP' neural network. The MLP / MPP artificial neural network for coding and the MLP / MPP' artificial neural network for decoding are defined by an identical structure, comprising for example a plurality of layers of artificial neurons, and by a set of weights and activation functions associated respectively with the artificial neurons of the network concerned.

[0094] A vector representation of a current sample (a vector Zn or Zdn from the feature maps FMt, or FMt';) is applied as input (i.e. to an input layer) of the artificial synthesis neural network MLP or MLP». The artificial synthesis neural network produces as output a vector ZLn or ZLdn of latent values ​​representative of the decoded sample, which are reinserted into the NL latent value maps Lj / L'j. In [Fig.3], a layer of the synthesis neural network is represented which has 40 outputs denoted Ui...u40. Each output is followed by a RELU type activation function. The RELU function (for Rectified Linear Unit) is well known to those skilled in the art. The output layer has three components denoted vl...v3, similarly followed by a RELU function.At the output of the MLP / MLP' module, the resulting values ​​of each RELU function, composing the vector ZLn / ZLdn, are injected into the maps Lj at the position defined by that of the pixel n being processed. The concatenation of all these values ​​constitutes the decoded latent value maps Lj / L'j. In [Fig.3], there are therefore 3 latent value maps.

[0095] the decoded latent value maps Lj / L'j are then processed by the module of MPP / MPP' convolution, which produces as output a decoded signal, according to one embodiment the color components (R, G, B) constituting the image I.

[0096] According to the example of [Fig.3], the convolutional neural network consists of a residual convolution layer (CRI) as input, a set of RELU functions, and a residual convolution layer (CR2) as output. The CR modules are residual convolution elements, without limiting aspect. They can include kernels of any size. They can be of simple convolutional types or include attention elements. They can also include strides (shifting of the convolution kernel at each iteration), dilation (spacing of the input values ​​provided to the kernel), etc. A convolution element is potentially followed by a bias as is well known to those skilled in the art. Any activation function within the reach of those skilled in the art can be used (ReLU, sigmoid, Softmax, Softplus, Softsign, etc.)

[0097] The convolution elements may be composed of 2D convolutions, i.e., the convolution kernel is applied jointly to the rows and columns of the latent value map. The convolution elements may be composed of 1D convolutions, in a separable form. In this case, rather than performing the convolution on the two-dimensional latent value map, a first one-dimensional (1D) convolution operator of length Lh is performed on the rows, then a second convolution of length Lv on the columns of the map obtained after the first operator. The row-column processing may be reversed to perform processing by column and then by row. This implementation has the effect of reducing the storage of Lh*Lv for a given non-separable kernel to Lv+Lh, and limits the complexity to Lh+Lv multiplications per map value versus Lh*Lv in the 2D case.

[0098] Naturally, the convolution module can for its part comprise several networks, in cascade. Each network can comprise several convolution layers with or without bias, accompanied by activation functions of the relu type or other, etc.

[0099] [Fig.4] schematically represents artificial synthesis and convolution neural networks used in another embodiment of the invention for coding and decoding.

[0100] These networks are capable of processing images whose components are of different resolutions.

[0101] As for [Fig.3], according to this embodiment, the synthesis module comprises an MLP / MLP' neural network and the convolution module comprises an MPP / MPP' neural network. A vector representation of a current sample (a vector Zn or Zdn from the FMt or FMt'i ​​characteristic maps) is applied as input to the MLP or MLP' artificial synthesis neural network. The synthesis neural network produces as output a vector ZLn or ZLdn of values latent values ​​representative of the decoded sample, which are reinserted into the NL latent value maps Lj / L'j. In [Fig.4], a layer of the synthetic neural network is shown which has 40 neuron outputs denoted ul...u40. Each output is followed by a RELU type activation function. The output layer has six components denoted vl...v6, similarly followed by a RELU function. At the output of the MLP / MLP' module, the resulting values ​​of each RELU function are injected into the Lj maps at the position defined by that of the pixel being processed. The concatenation of all these values ​​constitutes the decoded latent value maps Lj / L'j. In [Fig.4], there are therefore 3 latent value maps.

[0102] Unlike [Fig.3], however, in this mode, the latent value maps Lj do not all three contain the same number of samples. The MLP / MLP' module generates more latent values ​​for the first map than for the following ones: 4 values ​​are generated for the first map denoted Ly against 1 for the maps Lu and Lv. This advantageously results in latent value maps corresponding to chrominance components which contain fewer samples than the latent value maps corresponding to luminances. Such a size adaptation makes it possible to reduce the number of operations carried out and the number of network parameters to be coded and transmitted (resp. decoded). This number can be adjusted to the dimension of the components.

[0103] The decoded latent value maps Lj / L'j are then processed by the MPP / MPP' convolution module, which produces as output a decoded signal, according to this embodiment the color components (IY, Iu, Iv) constituting the image I, the U and V components conventionally comprising four times fewer samples than the Y component.

[0104] The architecture of the convolutional neural network is identical to that of [Fig.3], and the same comments apply. The convolution elements are denoted CRI, CR2, CR3 and CR4.

[0105] [Fig.5] schematically represents convolution networks according to embodiments of the invention.

[0106] Cases (a) to (d) respectively illustrate convolution layers used in the context of the invention for processing three input latent value maps (NL=3), and a YUV three-component image (NC=3) at the output of the convolution module. In the figure, the oblique lines surmounted by a number indicate the number of input latent value maps and output image components.

[0107] Naturally, other embodiments, not shown, are conceivable, with a greater or lesser number of latent value maps and a variable number of output components (for example, NL = 4 latent value maps for an image with NC = 3 YUV components at output), without loss of generality.

[0108] Furthermore, in the embodiment presented, each latent value map contains only scalar data, in accordance with the definition of a latent value map given previously, and there are three latent value maps each corresponding to one of the three coordinates of the samples of said signal. It is however possible to organize the latent values ​​differently, for example by forming a single signal comprising, for each sample, a vector of three values, without loss of generality.

[0109] These convolution layers, which use convolution elements denoted Ci, and attention elements (SGM: sigmoid function), can be combined by cascade. The complexity increases with the cases presented, both from the point of view of memory occupation (number of filter coefficients) and that of the number of arithmetic operations they involve.

[0110] Case (a) presents a simple convolution that processes the NL latent value maps Li / L'i as input. The NL (according to the example, NL=3) maps undergo a simple convolution, denoted C. A filtering is performed using the kernel of the convolution, of size kxk (for example, k=3). The kernel (3x3) is applied to the different positions of the maps to obtain the NC (according to the example, NC=3) components of the image (for example, YUV) as output. The number of operations required is therefore 3*3*3*3 (= 81). Generally, the number of operations is NL*NC*k*k. NL*NC kernels of size k*k are used and must be stored and / or transmitted. In addition, NC biases must also be stored and / or transmitted.

[0111] Case (b) has the same complexity, but a residual structure: unlike case (a), the signal from the convolution is combined (added, subtracted, multiplied) with the original signal composed of the three intermingled latents. The memory footprint (number of operations) is identical to that of case (a), i.e. NL*NC*k*k operations, but this structure can facilitate the learning of the neural network.

[0112] Case (c) presents a residual structure which treats in a differentiated way the NL (according to the example, NL=3) latent value maps: one map (L'o) is used for the generation of the Y component, denoted IY, of the image. Its convolution is denoted Cl, its kernel is of size kl*kl. The other two maps (L'iet L'2) are used for the generation of the U and V components of the image, denoted Uct Iv. Their convolution is denoted C2, its kernel is of size k2*k2. The memory footprint is smaller than during the combined processing of the 3 maps, since the number of operations required is l*l*kl*kl for Cl and 2*2*k2*k2 for C2. It is necessary in this case to store 3 biases. As explained in support of [Fig.4], this structure is well adapted to process images of different resolutions.

[0113] Case (d) presents a structure comprising an SGM attention module, processing jointly the NL latent value maps. The memory footprint is larger since the number of operations required is NL*NC*k3*k3 for C3 and NL*l*k4*k4 for C4. In this case, it is necessary to store NC+1 bias. This structure, however, improves local image processing, particularly of the contours present in the images. a. For C2: 2*2*k2*k2

[0114] [Fig.6] schematically represents convolution networks according to embodiments of the invention.

[0115] The left part of the figure shows a low complexity differentiated processing which independently processes the three latent value maps. It is capable of processing different resolution maps, so each component of the image can have its own resolution, for example to manage a YUV 420 format for which the U and V components are of reduced size compared to the Y component. The convolution elements are denoted C5, C6 and C7.

[0116] The right part of the figure shows an alternative view of the differentiated processing of [Fig.5], case (c). The convolution neural network is able to process YUV formats separately; the Y component is processed separately while the processing of the U and V components is linked. The convolution elements are denoted C8, C9, CIO, Cil and C12.

[0117] The table below summarizes the number of operations required (MAC) and the number of values ​​stored (storage) for each of the cases presented in Figures 5 and 6. In this example, we consider that there are three latent value maps (NL=3), that there are three outputs (NC=3) and that all the convolution kernels are of size 3x3 (k=kl=k2=...=3).

[0118] [Tables 1] figure MAC storage 5-a 81 84 5-b 81 84 5-c 45 48 5-d 108 112 6 right 45 48 6 left 27 30

[0119] [Fig.7] is a flowchart representing an example of a coding method that can be implemented by the coding device of [Fig.l].

[0120] According to this embodiment, the signal is a two-dimensional image, each sample to be coded is therefore a pixel Pn with coordinates (xn, yn).

[0121] Encoding takes place in two main phases:

[0122] In a first phase, called the construction phase, learning is carried out in order to determine, for an input signal I(Pn), the values ​​of the FM maps; and of the parameters Wk of the synthesis module and Vk of the convolution neural network(s) to optimize a cost function. The learning is for example carried out by gradient descent, followed by an update of the values ​​of the FM characteristic maps;, and optionally of the synthesis and convolution neural networks. As is known in the state of the art, the cost function can be of the rate-distortion, or rate, or distortion, or perceptual type. To measure the rate R, it is necessary to simulate the coding of the FM maps;, then to measure the associated coding rate (the size of the flow Bl). According to one embodiment, the coding of the parameters Wk and / or Vk is not simulated because their influence is less important than that of the characteristic maps.According to one embodiment, the coding of the parameters Wk and / or Vk is also simulated and the associated flow rate (the size of the flow B2) is measured. To measure the distortion D, it is necessary to simulate the coding then the decoding of at least part of the image I, to obtain at least one pixel P'n resulting from a simulation of coding then decoding, then to measure the difference between this part of the image I (Pn) at input and a corresponding part of the image I (P'n) coded then decoded.

[0123] Then during a second phase, called the coding phase, the FM maps; and the parameters Vk and possibly Wk are encoded to produce the coded values ​​FMc; and Vck (and possibly Wck) before transmission or storage. They constitute the compressed representation of the input signal I (Pn).

[0124] We will now describe the steps of a method according to one embodiment of the invention.

[0125] During a step E20, a signal I (Pn) to be coded, comprising a plurality of N samples Pn, is provided as input to the method.

[0126] During a step E21, the M FM feature maps; are initialized. Subsequently, the parameters Wk of the MLP synthesis module, the values ​​of the FM maps;, and the parameters Vk of the (at least one) convolutional neural network must be optimized during the construction phase.

[0127] According to one embodiment, the FM cards; are of the same resolution as the input signal I (Pn) and therefore each comprise the same number of values ​​N as there are samples Pn to be coded. According to one embodiment, the FM cards; are of a resolution less than or equal to that of the input signal I (Pn) and therefore comprise, for at least one of them, a number of values ​​to be coded less than N. According to one variant, the first FM card is at the resolution of the images and each subsequent card is at half the resolution of the previous one. According to one embodiment, several FMi cards have the same resolution, lower than that of the input signal I (Pn).

[0128] According to one embodiment, the FM maps are initialized by predefined constant values. According to another embodiment, the feature maps are initialized by a set of random real values.

[0129] In one embodiment, the structure and parameters of the neural networks are initialized during this step. These parameters are subsequently updated, or refined, during the construction phase, during subsequent iterations of the method. According to one embodiment, the parameters of the synthesis module and / or the convolution neural network are initialized by predefined values ​​known to give a satisfactory cost function value (for example, following training on a corpus of images). According to another embodiment, the parameters of the synthesis module and / or the convolution neural network are initialized by a set of random values.

[0130] The FM characteristic maps; and the parameters of the neural networks are subsequently updated, or refined, during a step E22, by the update module MAJ of the encoder during its learning.

[0131] During a step E23, the FM maps are coded by the FMC module of the encoder. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is an actual coding and the coded values ​​constitute the stream B1. The simulation can be identical to the actual coding but it can also be different (for example, simplified). For this coding, for example, an entropic coding of the quantized values ​​of the characteristic map(s) is used.

[0132] In one embodiment, the FM cards are coded in the order (FMb FM2,..., FM4), and the variables of each card in a predefined order, for example lexico-graphic. Each card undergoes entropic coding. The entropic coding produces a compressed stream B1 whose flow rate is subsequently measured during a step E28.

[0133] During a step E24, according to one embodiment, the M FM cards are quantified, transformed, and data is extracted from them by the XTR module.

[0134] According to one embodiment, at least one FM map; has a lower resolution than the images of the sequence to be coded and the transformation operation comprises oversampling so that the transformed FMt map; has the same number of samples as the images of the sequence. Oversampling consists of adding values ​​to the FMt maps; to achieve the resolution of the images of the input sequence. It can be simple (by nearest neighbor replication) or include interpolation (linear, polynomial, by filtering, etc.). It can be carried out by an oversampling neural network.

[0135] According to one embodiment, the XTR module performs an extraction of values ​​in the transformed characteristic maps FMf, for a sample Pn to be coded, as a function of its coordinates in the input signal. The extracted values ​​constitute the vector Zn. Zn is a J-tuple, that is to say that it comprises J elements, or data z;. The vector Zn of index n refers to the characteristic vector of the pixel P'n.

[0136] In one embodiment, Zn is a J-tuple (zb z2,..., Zj), consisting of the values ​​of the FM maps; located at the coordinates (xn, yn) of the current pixel Pn. In one embodiment, Zn is a J-tuple constructed from values ​​taken from the FMi maps at coordinates that may be different depending on the maps. For example, if the FM maps; are at different resolutions because they have been subsampled, the coordinates are adapted (by scaling) to correspond to the resolution of each map. In one embodiment, Zn is a J-tuple constructed from values ​​taken from the FM maps; by applying the processing to one or more values ​​of the maps, for example a filtering of the values ​​neighboring the targeted value in a map.For example, for a current sample Pn in an FMi map which is at the same resolution as the input signal, we can extract the values ​​located at the coordinates (xn, yn), (xn-l, yn), (xn, yn-l) and (xn-l, yn-l) and apply a processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (¾) of the element i of the vector Zn relative to this FM map;. According to another example, in an FM map; which is at half the resolution of the input signal, we can consider the values ​​located at the coordinates (xn / 2, yn / 2), (xn / 2-l, yn / 2), (xn / 2, yn / 2-1) and (xn / 2-l, yn / 2-1) and apply a processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (z;) of the element i of the vector Zn relative to this FM map;. The samples to be coded are for example processed in sequential order, from n=l to n=N.

[0137] During a step E25, the characteristic maps are processed by the synthesis module to generate latent value maps. The synthesis module is for example a neural network, or a filter bank, or a linear inverse transformation (wavelet, DCT, etc.) defined by its parameters Wk, capable of processing the transformed characteristic maps as input, to generate as output a set of NL latent maps Lj with 1 <j <=NL.

[0138] According to one embodiment, the vector Zn is processed by an MLP synthesis neural network to generate as output a second vector ZLn containing NL latent values ​​for the sample Pn to be coded. According to one embodiment, NL is equal to 3 and the output vector is a triplet to be inserted into three maps Lb L2 and L3 According to another embodiment, NL is equal to 4 and the output vector is a quadruplet to be inserted into four maps Li - L4 of latent values ​​at the pixel coordinates. According to another embodiment, NL is between the number M of maps FM; and the number of image components, M <NL<3 pour le cas de NC=3 composantes. Il est également possible d’organiser les valeurs latentes différemment, par exemple en formant un unique signal comportant, pour chaque échantillon, un vecteur de NL valeurs scalaires, sans perte de généralité. Les cartes de valeurs latentes représentent un signal intermédiaire avant le traitement convolutif.

[0139] During a step E26, the (at least one) MPP convolutional neural network is applied at the output of the MLP module to the latent value maps (Lj) to obtain decoded samples P'n, for example vectors representative of the samples Pn to be coded, according to one embodiment the triplet (R, G, B) of the sample P'n (the sample Pn coded then decoded).

[0140] During a step E27, the parameters Vk and Wk of the synthesis and convolution modules are quantified and coded. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is an actual coding and the coded values ​​constitute the stream B2. The simulation can be identical to the actual coding but it can also be different (for example, simplified). Any known technique can be used for this purpose, for example the neural network coding standard proposed by the MPEG-7 part 17 standard, also called NNR (Neural Network Representation). It should be noted that in this case, the amount of degradation that the coding brings to the parameters Vk and Wk must be chosen.

[0141] During a step E28, a performance measurement is evaluated.

[0142] For this purpose, the coding simulation rates associated with the characteristic maps (simulation of the B1 flow by coding the FM maps;) and optionally with the parameters of the synthesis and convolution modules (simulation of the B2 flow by coding the Vk / Wk parameters) are measured.

[0143] According to one embodiment, the cost function is of the rate-distortion type, noted (D+L*R), where D is for example the quadratic error measured between the input signal and the decoded signal (or the error measured on a subset of samples of the signal). According to another example, D is calculated from a perceptual function such as SSIM (for Structural SIMilarity), or MSSSIM (for Multi-scale Structural SI-Milarity). According to one embodiment, R is the simulated rate of the stream B1; according to another embodiment, R is the total rate used to code this image or sequence of images, that is to say the sum of the simulated rates of B1 and B2. L is a parameter which regulates the rate-distortion trade-off. Other cost functions are possible.

[0144] As long as the cost function has not reached its minimum, the performance measurement is not satisfactory, and the method is repeated from step E22. This minimization can be carried out by a mechanism known as gradient descent with updating of the parameters during step E22. If the cost function has reached its minimum, the training stops. If an encoded version corresponding to the last simulation of the parameters of the synthesis and convolution modules (Vk / Wk) and the feature maps (FM;) is available, the streams B1 and B2 can be constituted from them. According to another embodiment, the actual encoding of the updated parameters of the synthesis and convolution modules (V^WJ and the values ​​of the feature maps (FM;) is carried out at this stage to produce the encoded parameters Vck / Wcket FMc; which constitute streams B2 and B1. The streams B1 and B2 can be concatenated to produce a final stream. According to one embodiment, the stream B2 of the encoded parameters of the synthesis and convolution modules is stored or transmitted before the stream Bl, so that it can be decoded before the stream Bl.

[0145] [Fig.8] is a flowchart representing an example of a decoding method that can be implemented by the decoding device of [Fig.2].

[0146] During a step E30, the streams B1 and B2 are extracted from the encoded stream. They respectively contain the coded representations of the FMc maps; and the parameters of the synthesis and convolution modules Wck and Vck.

[0147] During a step E31, the M FMd; cards are generated by decoding the EMC; values. For this decoding, a decoding technique corresponding to that used by the coder is used, for example entropic (de)coding. In one embodiment, the FMd; cards are decoded in the order (FMdi, FMd2,... FMd4), and the values ​​of each card in a predefined order, for example lexicographic. The parameters Wdk and Vdk of the synthesis and convolution modules are generated by decoding the Wck and Vck values ​​of the stream B2. Any known decoding technique corresponding to the coding technique used by the coder can be used for this purpose. The (at least one) MPP' convolution neural network is similar to the MPP synthesis network, i.e. it has the same structure and includes the same parameters, except for the coding, which can be carried out with or without losses.The same applies to the synthesis module, which can be implemented as a neural network.

[0148] According to one embodiment, the stream B2 is decoded before the stream B1, in order to be able to have the parameters of the synthesis and convolution modules before starting to decode the samples.

[0149] According to embodiments as described for the coder: the FMd cards; are of the same resolution as the signal to be reconstructed I (Pdn) and therefore each comprise the same number of values ​​N as there are Pdn samples to be decoded; the FMd cards; are of a resolution less than or equal to that of the signal and therefore comprise, for at least one of them, a number of values ​​to be decoded less than N. According to a variant, the first FMdi card is at the resolution of the images and each subsequent card is at a resolution half that of the previous one. According to one embodiment, several FMdi cards are of the same resolution, lower than that of the signal.

[0150] During a step E33, according to embodiments, the M FMd cards are dequantized, transformed, and data is extracted from them by the XTR module.

[0151] According to one embodiment, at least one FMd; map has a lower resolution than the decoded images and the transformation operation includes oversampling so that the transformed FMf; map includes the same number of samples as the images of the sequence. Oversampling consists of adding values ​​to the FMf; maps to achieve the resolution of the images of the input sequence. It can be simple (by nearest neighbor replication) or include interpolation (linear, polynomial, by filtering, etc.). It can be performed by an oversampling neural network.

[0152] According to one embodiment, the module XTR' performs an extraction of values ​​in the transformed characteristic maps FMf,, for a sample Pdn to be decoded, according to its coordinates in the signal. The extracted values ​​constitute the vector Zdn. Zdn is a J-tuple, that is to say that it comprises J elements, or data zd;. The vector Zdn of index n refers to the characteristic vector of the pixel Pdn.

[0153] This step is similar to step E24 which was described for the encoder in support of [Fig.7], and the embodiments apply. In particular: In one embodiment, Zdn is a J-tuple (zdi, zd2,..., zdj), consisting of the values ​​of the FMf; / FMdi maps located at the coordinates (xn, yn) of the current pixel Pdn. In one embodiment, Zdn is a J-tuple constructed from values ​​taken from the EMC / FMdi maps at coordinates which may be different depending on the maps. For example, if the EMC / FMd; maps are at different resolutions because they have been subsampled, the coordinates are adapted (by scaling) to correspond to the resolution of each map. In one embodiment, Zdn is a J-tuple constructed from values ​​taken from the EMC / FMd; maps. by applying the processing to one or more values ​​of the maps, for example filtering the values ​​close to the targeted value in a map.For example, for a current sample Pdn in an FMdi map which is at the same resolution as the image signal to be decoded, we can extract the values ​​located at the coordinates (xn, yn), (xn-l, yn), (xn, yn -1) and (xn-l, yn-l) and apply a processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (zd;) of the element i of the vector Zn relative to this FMf; / FMd; map. According to another example, in an FMf; / FMd; map which is at half the resolution of the signal, we can consider the values ​​located at the coordinates (xn / 2, yn / 2), (xn / 2-l, yn / 2), (xn / 2, yn / 2-1) and (xn / 2-l, yn / 2-l) and apply a processing to these values ​​(filtering, averaging, interpolation, etc.) to obtain the final value (zd;) of the element i of the vector Zdn relative to this FM card;. The samples to be decoded are for example processed in sequential order, from n=l to n=N.

[0154] During a step E34, the characteristic maps are processed by the synthesis module MLP' to generate latent value maps. The synthesis module is for example a neural network, or a filter bank, or a linear inverse transformation (wavelet, DCT, etc.) defined by its parameters Wdk, capable of processing the transformed characteristic maps as input, to generate as output a set of NL latent maps L'j with 1 <j <=NL.

[0155] This step is similar to step E25 which was described for the encoder in support of [Fig.7], and the embodiments apply. In particular, according to one embodiment, the vector Zdn is processed by a synthesis neural network MLP' to generate as output a second vector ZLdn containing NL latent values ​​for the sample Pdn to be decoded. According to one embodiment, NL is equal to 3 and the output vector is a triplet to be inserted into three maps L'b L'2 and L'3 According to another embodiment, NL is equal to 4 and the output vector is a quadruplet to be inserted into four maps L'i - L'4 of latent values ​​at the pixel coordinates. According to another embodiment, NL is between the number M of maps FMd; and the number of components of the image, M <NL<3 pour le cas où NC=3 composantes.It is also possible to organize the latent values ​​differently, for example by forming a single signal comprising, for each sample, a vector of NL scalar values, without loss of generality. Latent value maps represent an intermediate signal before convolutional processing.

[0156] During a step E35, the (at least one) convolutional neural network MPP' is applied at the output of the module MLP' on the latent value maps (L)) to obtain decoded samples Pdn, for example vectors representative of the samples Pdn to be decoded, according to one embodiment the triplet (R, G, B) of the sample Pdn. This step is similar to step E26 which was described for the coder with the support of [Fig.7],

[0157] When all the latent value maps have been processed, the decoded signal corresponding for example to the image I (Pdn) is available.

[0158] [Fig.9] illustrates a decoding method used in one embodiment of the invention.

[0159] In this embodiment, there are 5 FMd cards. In a preferred embodiment, there are 7.

[0160] In this embodiment, the first map FMdi has the same resolution as the image I, and therefore comprises WxH variables, where W represents the width of the image in pixels, and H its height. The second map FMd2 has half the resolution (in each dimension) of the map FMdb. The map FMd3 has half the resolution (in each dimension) of the map FMd2. The maps FMd4 and FMd5 have half the resolution (in each dimension) of the map FMd3. This structure makes it possible to reduce the number of variables of the feature maps which facilitates decoding while minimizing the coding cost.

[0161] The XTR' module dequantizes the values ​​of the characteristic maps if necessary.

[0162] The feature maps are then processed by the XTR' module, which, according to this embodiment, oversamples them to make them the same resolution as the image to be decoded. (FMd2 is oversampled by a factor of 2 in each dimension, FMd3 is oversampled by a factor of 4 in each dimension, etc.)

[0163] Then, the XTR' module extracts, for each sample position, a vector Zdn in the form of a 5-tuple (zdi...zd5) consisting of the values ​​of the transformed FMd; maps located at the coordinates (xn, yn) of the current pixel Pdn. The vector Zdn is processed by the MLP' neural network to generate as output a vector ZPdn, according to the example a quadruplet representative of the sample Pdn to be decoded, which is reinjected into the latent value maps L'; at the coordinates (xn, yn) in the components.

[0164] According to this example, there are 4 latent value maps L';.

[0165] The 4 latent value maps are processed by the MPP' convolution module, for produce as output, according to this example, the 3 components Rd, Gd and Bd of the decoded image.

Claims

Claims

1. Method for coding at least one image (I(Pn)) comprising the following steps: - a step of constructing said at least one image, comprising the sub-steps of: - construction (E21, E22) of a group of (M) characteristic maps (FMi); - processing (E24, E25) of said group of characteristic maps by a synthesis module (MLP), defined by synthesis parameters (Wk), to provide a group of (NL) latent value maps (Lj), the number of latent value maps being less than or equal to the number of characteristic maps; - processing (E26) said group of latent value maps by at least one convolution neural network (MPP), defined by convolution parameters (Vk), to provide said at least one image (I(P'n)); - updating (E22, E28) at least one value of one of said characteristic maps and / or at least one synthesis parameter and / or at least one convolution parameter, as a function of a coding performance measurement; - a coding step (E23, E27) of said group of characteristic maps (FM;) and of said set of synthesis and convolution parameters (Wk / Vk).

2. Coding method according to claim 1, characterized in that said at least one image comprises a plurality of samples (Pn) to be coded, and in that the step of processing said group of feature maps (FM;) to provide a group of latent value maps (L;) comprises the following sub-steps: - for at least one sample, called current sample (Pn), of said at least one image to be coded, associated with a position (xn, yn): - construction (E24) of a characteristic vector (Zn) from said characteristic maps (FM;), as a function of said position (xn, yn) of said current sample (Pn); - processing (E25) of said characteristic vector (Zn) by said synthesis module (MLP), to provide a latent vector, representative of a latent value (ZLn) of the current sample; - construction of the group of latent value maps (L;) from said at least one latent vector (ZLn);

3. Method for decoding at least one image (I(Pdn)) from a binary stream (Bl, B2), comprising the following steps: - decoding (E31) a group of (M) characteristic maps (FMd;); - obtaining (E30, E32) a set of synthesis parameters (Wck) representative of a synthesis module (MLP'); - obtaining (E30, E32) a set of convolution parameters (Vc k) representative of at least one convolution neural network; - processing (E33, E34) said group of characteristic maps by said synthesis module (MLP1) to provide a group of latent value maps (L j), the number (NL) of latent value maps being less than or equal to the number (M) of characteristic maps; - processing (E35) said group of latent value maps by said at least one convolutional neural network (MPP) to provide said at least one decoded image (I(Pdn)).

4. Method for decoding a signal according to claim 3, characterized in that said at least one image comprises a plurality of samples (Pdn) to be decoded, and in that the step of processing said group of characteristic maps to provide a group of latent value maps comprises the following sub-steps: - for at least one sample, called current sample (Pdn), of said at least one image to be decoded, associated with a position (xn, yn): - construction (E33) of a characteristic vector (Zdn) from the characteristic maps, as a function of said position (xn, yn) of said current sample, and: - processing (E34) of said characteristic vector (Zdn) by said synthesis module (MLP') to provide a latent vector (ZLdn), representative of a latent value of the current sample; - construction of the group of latent value maps (L';) from said at least one latent vector (ZLdn).

5. Decoding or coding method according to one of the preceding claims, characterized in that said synthesis module (MLP, MLP') comprises a multi-layer perceptron.

6. Decoding or coding method according to one of the preceding claims, characterized in that said at least one convolution neural network (MPP, MPP') comprises at least one convolution layer (C, Cl, C2, C3, C4, C5).

7. Decoding or coding method according to one of the preceding claims- preceding, characterized in that the number (NL) of latent value maps is equal to the number of components (NC) of said at least one image.

8. Decoding method according to one of claims 3 to 7, characterized in that at least part of the set of synthesis and / or convolution parameters (Wck, Vck) is obtained from data from said binary stream.

9. Decoding method according to one of claims 3 to 8, characterized in that at least part of the set of synthesis and / or convolution parameters (Wck, Vck) is obtained from predetermined parameters.

10. Decoding method according to one of claims 3 to 9, characterized in that at least part of the set of synthesis and / or convolution parameters (Wck, Vck) is decoded as a function of reference parameters.

11. Decoding method according to one of claims 3 to 10, characterized in that said at least one convolution neural network comprises at least two convolution networks.

12. Decoding method according to the preceding claim, for which the binary stream comprises an indicator indicating the use of at least one of said networks.

13. Decoding method according to claim 6, characterized in that said at least one image comprises at least two dimensions (x,y), and said at least one convolution layer uses a kernel separable according to said at least two dimensions.

14. Device for coding at least one image (I(Pn)), characterized in that said device is configured to implement: - a step of constructing said at least one image, comprising the sub-steps of: - constructing (INIT, MAJ) a group of (M) characteristic maps (FM;); - processing (XTR, MLP) said group of characteristic maps by a synthesis module (MLP) defined by synthesis parameters (W k) to provide a group of latent value maps (Lj), the number of latent value maps (NL) being less than or equal to the number of characteristic maps (M); - processing (MPP) said group of latent value maps by a convolutional neural network (MPP) defined by synthesis parameters (W k) to provide a group of latent value maps (Lj), the number of latent value maps (NL) being less than or equal to the number of characteristic maps (M); convolution (Vk) to provide said at least one image (I(P'n)); - updating (EVAL, MAJ) at least one value of one of said characteristic maps and / or at least one synthesis parameter and / or at least one convolution parameter, as a function of a coding performance measurement; - a coding step (FMC, NNC) of said group of feature maps (FM;) and of said set of synthesis and convolution parameters (Wk / Vk).

15. Device for decoding at least one image (I(Pdn)), from a binary stream, characterized in that said device is configured to implement: - decoding (FMD) of a group of (M) feature maps (FMd;) - obtaining (NND) a set of synthesis parameters (Wck) representative of a synthesis module (MLP'); - obtaining (NND) a set of convolution parameters (Vck) representative of at least one convolution neural network; - processing (XTR1, MLP') of said group of characteristic maps by said synthesis module (MLP1) to provide a group of (NL) latent value maps (L'j), the number of latent value maps being less than or equal to the number of characteristic maps; - processing (MPP) said group of latent value maps by said at least one convolutional neural network to provide said at least one decoded image (I(Pdn)).

16. A computer program comprising instructions for carrying out the steps of an encoding or decoding method according to claim 1 or 3 when said program is executed by a computer.

Images