Method and device for encoding and decoding images with an attention mechanism

Abstract

The invention relates to a method and device for decoding a current image (Iv) of an image sequence (Si). The method comprises the following steps: - obtaining (E31) a set of correction information (IC, FMC) for said at least one current image; - obtaining (E32) a set of prediction information (IP, FMP) for said at least one current image; - decoding (E34) a set of parameters (WMX) representative of at least one reconstruction neural network (MIX); - processing (E35) said correction information and said prediction information using said at least one reconstruction neural network so as to produce a representation of the decoded current image (FMREC, Ivd).

Description

Method and device for coding and decoding images with attention mechanism.

[0001] The invention relates to the general field of coding digital image sequences. It relates more particularly to the compression of digital videos.

[0002] Digital videos are generally source-encoded to compress them in order to reduce the resources required for transmission and / or storage. There are many encoding standards, such as the ITU / MPEG standards (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) and their extensions (MVC, SVC, 3D-HEVC, etc.).

[0003] Encoding an image is generally done by predicting pixels using previously encoded and then decoded pixels present in the image being encoded, in which case we speak of "Intra prediction", or previously coded images, in which case we speak of "Inter prediction".

[0004] In the case of inter-prediction, a classical approach is often used: after motion estimation and compensation at the encoder, a residual frame is calculated by subtracting the prediction frame from the original image. At the decoder, the prediction frame and the residual frame are added to obtain the reconstructed frame. Since this simple addition is not optimal, new approaches, particularly neural ones, have recently been developed.

[0005] In this area, a conditional neural approach is proposed for example in the paper "Conditional Residual Coding: A Remedy for Bottleneck Problems in Conditional Inter Frame Coding", by F. Brand et al (arXiv:2307.12864). The proposed conditional coding models the conditional distribution of an image as a function of its predictor. However (in this paper), the residual data and the predicted data are considered to be of equal importance. This limits the compression performance.

[0006] There is therefore a need for a solution to combine a residual signal with a predicted signal in an image sequence more efficiently.

[0007] The invention relates to a method for coding a current image at least of a sequence of images, comprising the following steps: - generation of a set of prediction information for said at least one current image; - generation of a set of correction information for said at least one current image; - generation of at least one reconstruction neural network, defined by a set of reconstruction parameters; - processing of said correction information and said prediction information by said at least one reconstruction neural network to produce a representation of the current image; - updating at least part of said prediction information and / or said correction information and / or said reconstruction parameters, as a function of a coding performance measurement; - coding of at least part of said correction and / or prediction information and said reconstruction parameters.

[0008] The invention also relates to a method for decoding a current image of a sequence of images, comprising the following steps: - obtaining a set of correction information for said current image; - obtaining a set of prediction information for said current image; - decoding a set of reconstruction parameters representative of at least one reconstruction neural network; - processing of said correction information and said prediction information by said at least one reconstruction neural network to produce a representation of the decoded current image.

[0009] For the purposes of the invention, encoding, or "coding", means the operation which consists of representing a set of samples, or pixels, in a compact form carried for example by a digital binary train. Decoding means the operation which consists of processing a digital binary train to restore decoded samples.

[0010] By "sequence of images" is meant a plurality of ordered two-dimensional images, for example temporally in the case of a video. According to one example, the sequence corresponds to a scene. According to one example, the sequence corresponds to a set of predefined images, for example a fixed number, or, within the meaning of the MPEG standards, a GOP (Group Of Pictures) comprising the images located between two images of the Intra-image type, also called "intra period". According to another example, the images can be views of the same scene represented in multi-views. According to another example, the images can be a plurality of temporal and multi-view images (immersive video).

[0011] By "current image" we mean an image in the sequence.

[0012] By "prediction information" we mean any information useful for generating a prediction of the current image. Typically, this information can be inter prediction information (motion, reference images, etc.), intra prediction information (prediction direction), mode selection information (intra or inter mode or combined intra / inter or combination), etc.

[0013] By "correction information" we mean any information useful for generating a correction of the prediction of the current image. Typically, this information can be residual information, for example a residual image.

[0014] "Prediction information" and "correction information" are the inputs to the reconstruction neural network. They can contain data in the image domain or in the latent domain. For example, a prediction (resp. correction) image can consist of pixels or feature maps representative of the prediction (resp. correction) image in the latent domain.

[0015] A "latent domain" is a representation composed of a set of two-dimensional feature maps, the size of which is proportional to the input image. It can also include a set of one-dimensional vectors, or a set of scalar values.

[0016] By "reconstruction neural network" is meant a neural network such as a convolutional neural network, a multi-layer perceptron, an LSTM (for "Long Short Term Memory"), etc. The neural network is defined for example by a plurality of layers of artificial neurons and by a set of activation, weighting and addition functions (for example, a layer can calculate y = f(Ax+b), where y and b are vectors of dimension N, x a vector of dimension M, A is a matrix of dimension MxN, and f is the activation function). Preferably the reconstruction neural network is convolutional, that is to say that it comprises at least one convolution layer. A plurality of such networks can be cascaded.

[0017] By "neural network parameter" we mean one of the values ​​that characterizes the neural network, for example a weight associated with one of the neurons (filter or convolution coefficient, weighting, bias, value affecting the operation of non-linearity, etc.)

[0018] By "processing (of input values) by a reconstruction neural network" is meant the application of a function expressed by a neural network to the input values ​​to produce output values ​​representative of the samples of the current image to be decoded (reconstructed) (resp. encoded).

[0019] By "representation of the current image" we mean its representation in the image domain (representation in the form of pixels) or latent (representation in the form of feature maps).

[0020] By "performance measurement" is meant a measurement between at least one value of a sample to be coded and a decoded value of said sample. The measurement can evaluate, for example, a distortion, or a perceptual error. It can be carried out on a sample or a plurality of samples (for example, the current samples, or the current image, etc.). The measurement can also include a measurement of the flow rate, in particular associated with the coding of the neural network and / or the coding of the feature maps. The measurement can be a joint measurement between the flow rate and the distortion through their weighting. As is well known in the state of the art, the value of this measurement is generally minimized until a target or minimum value, or a predefined time, is reached.

[0021] A "sample" is a value taken from an image in the sequence. Sampling a signal produces a series of discrete values ​​called samples. In the case of an image signal, the sample is called a pixel, which can be, for example, a color pixel traditionally represented by a triplet of values, for example (R,G,B) or (Y,U,V). The position of the sample is identified by its abscissa (x) and ordinate (y) coordinates in the image.

[0022] Generally speaking, it is considered that the steps of an coding or decoding method should not be interpreted as being linked to a notion of temporal succession. In other words, the steps may be carried out in a different order than that indicated in the independent coding or decoding claim, or even in parallel.

[0023] The coding method according to the invention carries out during its training a construction of the coding parameters, from a sequence of input images, by training a reconstruction neural network on data sets of prediction and correction information, to obtain a faithful reconstruction of the image. Advantageously, the coding method encodes the reconstruction neural network. This can be optimized, and adapted in complexity or quality, according to the targeted complexity and the targeted ratio between flow rate and distortion. Similarly, the encoded and transmitted prediction and correction information can thus be optimized and adapted.

[0024] During training, or construction, or learning, the parameters of the neural network and the information to be coded are updated according to a performance measure, for example of the rate-distortion type. When training is finished, that is to say the performance measure obtained is satisfactory, the actual coding of the parameters of the reconstruction neural network as well as that of the prediction and reconstruction information can be carried out and the result of the coding stored or transmitted to the decoder. Advantageously, the training process therefore makes it possible to refine the parameters of the reconstruction neural network, as well as its input parameters, until an adequate representation in terms of performance is obtained, for example a desired balance between the generated rate and the distortion undergone by the input image being coded.Advantageously, the coding method according to the invention makes it possible to efficiently compress the signal.

[0025] Advantageously, the decoding process is simple since it is sufficient to obtain the correction and prediction information and the reconstruction neural network to reconstruct a decoded version of the current image.

[0026] Advantageously, a transmission system can be designed that works image by image with low latency, each current image being decoded upon receipt of the neural network parameters and the associated prediction and correction information.

[0027] Advantageously, decoding allows working with prediction data and residual data having the same format and resolution, which facilitates hardware implementation.

[0028] According to embodiments of the coding and / or decoding method:

[0029] - Said at least one reconstruction neural network comprises at least one attention module. Advantageously, the neural network uses attention mechanisms which make it possible to generate attention masks. Coding is thus improved.

[0030] An "attention module" means a layer of a neural network comprising an attention element. An attention element is an element that allows an attention mechanism to be applied to generate one or more masks that are used to weight (by multiplication or use of more complex functions) the characteristics of the image in order, for example, to adapt the allocation of the flow to the different regions and / or characteristics of the image according to their importance. The attention masks are applied to the inputs of the neural network (correction and prediction information). They are encoded and decoded as parameters of the neural network. Advantageously, it is not necessary to use additional bits to encode the masks.

[0031] - Said at least one reconstruction neural network comprises at least one convolution layer; thus, non-localized processing of the image can be carried out, which makes it possible to improve the consistency of the generated image by limiting the noise and improving the contours present in the image.

[0032] - Said at least one reconstruction neural network applies the attention module to said prediction information. Advantageously, according to this mode, the attention module is used to correct the prediction channel. Thus, better consideration of certain areas of the image can be carried out. The prediction information can be weighted. According to one example, if the generated mask "erases" the prediction information, this results in coding of the image in intra-image mode (only the correction information is used).

[0033] - Said at least one reconstruction neural network applies the attention module to said correction information. Advantageously, according to this mode, the attention module is used to correct the correction channel. Thus, better consideration of certain areas of the image to be corrected can be carried out. The correction information can be weighted. According to one example, if the generated mask "erases" the correction information, this results in coding of the image in inter-image mode (only the prediction information is used).

[0034] - Said at least one reconstruction neural network applies the attention module to a combination of said correction and prediction information. Advantageously, according to this mode, the attention module is used to jointly correct the two input channels. Thus, a compromise can be found concerning the importance, or weighting, given to each of the two channels. The attention module can also correct artifacts related to the effects of the combination of the two inputs, or channels, of prediction and correction.

[0035] - At least part of the set of reconstruction parameters is obtained from data of said binary stream; advantageously according to this mode, the parameters of the neural network are transmitted in the stream. The reconstruction module can be transmitted in whole or in part in a quantized form and coded in a compact form using any quantizer and entropy coder accessible to those skilled in the art. A format similar to that of the MPEG-7 NNR standard can be used.

[0036] - At least part of the set of reconstruction parameters is decoded according to predetermined parameters; advantageously according to this mode, part of the reconstruction parameters of the neural network is accessible to the decoder, for example they can be stored in a storage memory. In particular, part of the reconstruction module can be stored in a quantized form and coded in a compact form using any quantizer and entropy coder accessible to those skilled in the art. A format similar to that of the MPEG-7 NNR standard can be used.

[0037] - At least part of the set of reconstruction parameters is decoded as a function of reference parameters; advantageously according to this mode, certain parameters of the neural network are decoded by taking into account reference parameters which can be stored in a memory (storage memory, network, etc.) accessible to the decoder. Thus the storage space or the transmission rate on the network can be effectively reduced: certain parameters can be coded / decoded in a complementary manner (for example by updating a kernel, a layer, a bias, etc.), others not be coded / decoded at all, because they are available in the reference parameters. The reference parameters can constitute a reference network. According to alternative embodiments, which can be combined with each other: the reference parameters can constitute a set of variants of a reconstruction network: type of convolution, attention modules, etc.Thus, only these parameters or their identification need to be coded, inserted into the stream and decoded by the decoder; certain parameters are coded in a complementary manner to the parameters of the reference network. Thus, the decoder will only need to decode these residual parameters and then add them, multiply them (or combine them in any other known way) with those of the reference neural network; part of the parameters of the reference network is reused for the reconstruction network. For example, a complete layer of the network can be copied from the reference network, the parameters of this layer are therefore neither coded, nor transmitted, nor decoded; a reference network indicator is transmitted in the stream. It is thus possible to indicate a network to be used for decoding among a plurality of possible networks known to the decoder.

[0038] - Said at least one reconstruction network is selected from a plurality of reconstruction networks, and the bitstream comprises an indicator indicating the selection of said at least one network from the plurality.

[0039] - Said at least one reconstruction network corresponds to a plurality of cascaded reconstruction networks. Advantageously, the use of several cascaded networks makes it possible in particular to enhance the filtering and improve the consistency of the generated image by limiting its noise and improving its contours. In addition, each network can be learned individually or fixed independently. According to alternative embodiments, which can be combined with each other: the binary stream comprises an indicator indicating the use of at least one of said networks. Thus, depending on the quality and complexity required, one or more networks can be used for decoding. The indicator indicates, for example, that network 1 is mandatory, but that networks 2 and 3, which are cascaded behind network 1, are optional.The decoder can then choose whether to use them or not: if it uses them, it will obtain better quality at the expense of complexity, and vice versa. At least two of the networks have common parameters. It is thus possible, advantageously, to transmit characteristic parameters of one network, which can be used for the others, which reduces the transmission cost. The network is thus shared through common parameters. At least two of the networks have an identical structure and only their weights differ in part or completely. The same calculation elements are thus shared.

[0040] Correlatively, the invention also relates to a coding device and a decoding device.

[0041] The characteristics and advantages of the coding or decoding method apply in the same way to the coding or decoding device according to the invention and vice versa.

[0042] The invention also relates to a computer program on a recording medium, this program being capable of being implemented in a computer or an encoding or decoding device in accordance with the invention. This program comprises instructions adapted to the implementation of the corresponding method. This program can use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0043] The invention also relates to a computer-readable information carrier or recording medium comprising computer program instructions mentioned above. The information or recording media may be any entity or device capable of storing the programs. For example, the media may comprise a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a floppy disk or a hard disk, a DNA sequence, or a flash memory. Furthermore, the information or recording media may be transmissible media such as an electrical or optical signal, which may be conveyed via an electrical or optical cable, by radio link, by wireless optical link or by other means.

[0044] The program according to the invention can in particular be downloaded from an Internet-type network.

[0045] Alternatively, each information or recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of a method according to the invention.

[0046] Other characteristics and advantages of the present invention will emerge from the description given below, with reference to the appended drawings which illustrate exemplary embodiments thereof which are not limiting in nature.

[0047] schematically represents a decoding device used in the context of the invention;

[0048] schematically represents a coding device used in the context of the invention;

[0049] Illustrates an example of a mixing module used in encoding and decoding according to two embodiments of the invention.

[0050] Illustrates a reconstruction neural network used for encoding and decoding according to one embodiment of the invention.

[0051] Illustrates a variant of a reconstruction neural network used for encoding and decoding according to one embodiment of the invention.

[0052] Illustrates another variant of a reconstruction neural network used for encoding and decoding according to an embodiment of the invention.

[0053] is a flowchart representing an example of a decoding method that can be implemented by the decoding device of the.

[0054] Schematically represents a decoding device used in the context of the invention.

[0055] The decoding device DECv decodes an image or sub-sequence v of the sequence, proceeding image by image. At the end of the decoding, the V images of the sequence are decoded.

[0056] A common subsequence to be decoded denoted Id V comprises at least one current image to be decoded, each image respectively comprising a plurality of samples. Subsequently, it is considered, without loss of generality, that the current subsequence comprises a single current image Iv. However, a current subsequence to be decoded may comprise, according to one embodiment, several images (which may for example be concatenated to form an image of larger size Iv).

[0057] The DECv decoder dedicated to the image receives as input the data necessary to decode it:

[0058] - IPc encoded data corresponding to prediction information. This prediction information can be for example:- motion information (motion field, reference image numbers, etc.);- intra-image prediction information (prediction mode, prediction directions, etc.);

[0059] - signal deformation information, such as geometric transformation parameters (translations, homotheties, etc.);

[0060] - etc.

[0061] – ICc encoded data corresponding to coded IC correction information, for example prediction residuals, or in general, data aimed at correcting the prediction obtained from the prediction information.

[0062] - the WMXc coded reconstruction parameters of at least one reconstruction neural network. Other neural networks can be used, such as networks:

[0063] - synthesis of prediction information;- synthesis of correction information;- predictive decoding of latent prediction or correction feature maps;

[0064] - oversampling and / or processing of latent feature maps for prediction or correction;

[0065] - post-processing of reconstructed data after mixing;

[0066] - etc.

[0067] The DEC decoding module v comprises, for a current image, a REF module for obtaining reference images, an FPD module for decoding prediction information, an FCD module for decoding correction information, a MIX module for reconstruction by combining the IP / FMP compensated images / maps and the IC / FMC correction images / maps, a NND module for decoding a neural network. According to one embodiment, DEC v produces as output a current decoded image, noted Idv, comprising a plurality of decoded samples.

[0068] The FCD decoding module decodes correction information ICc. According to one embodiment, it is a conventional decoder, for example entropic, of the JPEG, or MPEG, etc. type, which produces as output a correction image (of residues) noted IC. According to another embodiment, it decodes quantized data, for example using an entropic decoder, of one or more correction characteristic maps in the latent domain. According to one embodiment, the FCD module may comprise for this purpose a neural network for synthesizing correction information. According to one embodiment, the maps decoded by the FCD module, numbering NFC, are noted FMCet are called maps de correction de l’image courante, ou cartes de correction courantes.

[0069] The FPD decoding module decodes prediction information IPc. According to one embodiment, it is a conventional decoder, for example entropic, of the JPEG, or MPEG, etc. type, which produces as output a prediction image denoted IP. According to one embodiment, it is a conventional decoder which produces as output a motion field MV. The FPD module may comprise for this purpose a motion synthesis neural network. According to another embodiment, it decodes quantized data, for example using an entropic decoder, of one or more prediction feature maps. The maps decoded by the FPD module, numbering NFP, are denoted FMP and are called maps de prédiction de l’image courante, ou cartes de prédiction courantes.

[0070] The optional REF module constitutes a set of NFR reference images / maps, denoted IREF / FMREF, corresponding to one or more images / maps previously encoded and then decoded from the sequence. They can be predetermined or their reference can be decoded in the stream in the form of an indicator. In one embodiment, a single reference image is selected. In one embodiment, several reference images are selected. In one embodiment, a set of reference maps is selected in the latent domain. In one embodiment, for example in the case where an intra image is decoded, no reference map is selected and the REF module is absent.

[0071] The optional WARP module performs motion compensation of the reference images / maps from the MV motion field in order to obtain one or more IP / FMP prediction images or maps. The WARP module may be absent if the FPD decoder directly decodes the prediction images or maps. In one embodiment, the WARP module produces the FMP maps with data only from the FPD module, i.e. the optional REF module does not exist, or the WARP module does not take into account the reference images submitted to it.

[0072] The MIX module uses as input one or more IP / FMP prediction image(s) or maps and one or more IC / FMC correction image(s) or maps and performs a combination of the two pieces of information to obtain the reconstructed image. The MIX module will be detailed in the following description. It comprises at least one convolutional CNN neural network, i.e. it comprises at least one convolutional layer. A convolutional layer may include convolutional elements, comprising a residual structure or not and an attention module or not, as will be described later.

[0073] The coded information ICc, IPc and WMXc are extracted from the BS stream which can be received over a communication network, or stored in a memory accessible by the decoder.

[0074] The DEC decoding device may be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the above-mentioned modules may then be realized by the cooperation of the processor and computer program instructions stored in the above-mentioned memory and designed to perform the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0075] Schematically represents a coding device used in the context of the invention.

[0076] The ENCv coding device encodes a current image or subsequence Iv of the sequence SV, proceeding image by image. At the end of the coding, the V images of the sequence are coded.

[0077] The ENCv encoder receives as input a sequence of images including image Iv and produces as output coded parameters. These parameters include:

[0078] – IPc encoded data corresponding to prediction information.

[0079] – ICc encoded data corresponding to correction information, for example prediction residuals.

[0080] - the encoded parameters of at least one reconstruction neural network. As mentioned when describing the, other neural networks can be used.

[0081] The ENC coding module v includes, for a current image, an NNC module for coding neural network parameters, an FCC module for coding correction information, an FPC module for coding prediction information, an RD-OPT evaluation module, an INIT / MAJ initialization and update module.

[0082] Furthermore, in a conventional manner, the coding module includes a decoder similar to the DECv decoder which was previously described, in order to reconstruct the current image I'v coded then decoded.

[0083] During the process of training, or building, the coding, that is to say as long as the performance evaluation step is not satisfactory, the coding modules carry out a coding simulation, followed by a decoding, intended for the RD-OPT evaluation module. Subsequently, they carry out the actual coding of the data. As is known, the coding simulation can be identical to the actual coding, or carry out an approximation thereof.

[0084] The INIT / MAJ module is responsible for initializing and updating the values ​​of the images / feature maps and the parameters of the neural network(s). It updates the values ​​to be encoded from the current image, based on the results of the performance function. Once the prediction and correction information and the neural network(s) are stabilized, they can be encoded.

[0085] The RD-OPT module performs an evaluation and minimization of a coding performance. The evaluation function is for example of the rate-distortion type. The distortion can be evaluated between Iv and I'v, the rate can include the cumulative rate linked to the contributions of ICc, WMXc and IPc, or only a sub-part. The minimization can be carried out by gradient descent, or any other optimization method within the reach of a person skilled in the art.

[0086] The ENC coding device may be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the above-mentioned modules may then be realized by the cooperation of the processor and computer program instructions stored in the above-mentioned memory and designed to perform the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0087] Illustrates an example of a mixing module used for coding and decoding according to two embodiments of the invention (a) and (b).

[0088] The MIX module includes a CNN reconstruction neural network which will be described later using figures 4, 5 and 6.

[0089] On the left of figure (a), according to a first embodiment, the MIX module receives as input a correction IC image and a prediction IP image. It generates as output a reconstructed Ivd image.

[0090] On the right of figure (b), according to a second embodiment, the MIX module receives as input a set of correction characteristic maps, FMC, and a set of prediction characteristic maps FMP.

[0091] The FMC and FMD feature maps are not necessarily the same in number or in the same form. The MIX module is adapted to process these inputs in their respective formats.

[0092] According to one embodiment, the MIX module generates as output a reconstructed image Ivd.

[0093] According to one embodiment, the MIX module generates as output a set of FMREC feature maps representative (in the latent domain) of a reconstructed Ivd image. In this case, these maps can be processed by a so-called combination neural network, comprising for example an MLP (multi-layer perceptron) or convolutional type network, to produce the reconstructed Ivd image.

[0094] Illustrates a reconstruction neural network (CNN) used by the MIX mixing module for encoding and decoding according to one embodiment of the invention.

[0095] The MIX module can contain one or more reconstruction networks, in cascade. The networks can be identical in their structure and parameters, or different.

[0096] According to the example of the, the reconstruction network comprises a CC (Cross Channel) mixing layer of the two input sets, IC / FMC correction images / maps on the first input and IP / FMP prediction images / maps on the second input. The mixing layer can perform a simple concatenation of the inputs or a more complex combination including for example a convolution or a combination by neural network on the concatenated inputs. The reconstruction network then comprises a set of layers, or subnetworks, N1, N2..NN.

[0097] The Ni modules (N1, N2..NN) include convolution elements, and non-linearity elements (RELU, etc.) without limiting aspect.

[0098] A convolution element may be of residual type or not, and may comprise kernels of any size. It is potentially followed by a bias as is well known to those skilled in the art. Any activation function within the reach of those skilled in the art may be used (ReLU, sigmoid, Softmax, Softplus, Softsign, etc.). The convolution elements may be composed of 2D convolutions, i.e. the convolution kernel is applied jointly to the rows and columns of the image or latent value map. The convolution elements may be composed of 1D convolutions, in a separable form. In this case, in a known manner, rather than performing the convolution on the two-dimensional image or latent value map, a first one-dimensional (1D) convolution operator is performed on the rows, then a second convolution on the columns of the representation obtained after the first operator.Row-column processing can be reversed to perform column-by-column and then row-by-row processing.

[0099] According to a first embodiment, not shown, the module N2 comprises, like the other modules Ni, one or more convolution layers followed by non-linearities.

[0100] The upper part of the figure details another preferred embodiment of the N2 module. It receives as input an intermediate signal z representative of a composition, or mixing, by the CC module, of the two inputs, possibly followed by a first network (N1), for example convolution.

[0101] According to this embodiment, the N2 module includes a CL attention element. The attention mechanism allows to apply a differentiated processing on certain areas of the image to be reconstructed, thanks to a masking in the image or latent domain. Such operations are used in many learning-based compression systems. For example, the paper "Non-local Attention Optimized Deep Image Compression" by Liu et al. (available at https: / arxiv.org / pdf / 1904.09757.pdf) describes such attention mechanisms useful in the context of entropy coding. In particular, the introduction of an attention mask allows to give particular importance to certain areas; in the context of this publication, it is a question of adapting the bitrate allocation to the different values ​​of the feature maps (for example, by allocating more bitrate to the textured areas of the image).

[0102] The upper branch of N2 receives as input the signal z from the network N1 or from a combination of the two inputs. The network (N2) teaches (the encoder) to construct a mask M which is obtained by optional processing of the input signal by a neural network (NN3) comprising a set of convolutions followed by non-linearities, then processing of the values ​​by the CL module which can, in a known manner, be of the Clamp, sigmoid, Arctg type, etc.

[0103] The central branch of N2 corresponds to the main processing of the signal z by a neural network (NN4). This module is optional. The output of the NN4 module is multiplied by the mask M. According to one embodiment, only certain areas are kept after the multiplication by the values ​​of the mask M. According to one embodiment, the inputs are erased after the multiplication by the values ​​of the mask M

[0104] According to one embodiment, M is an attention mask which is applied to an image from the optional NN4 network or from the signal z.

[0105] According to one embodiment, M is an attention mask which applies to all the latents from the optional NN4 network or the z latents.

[0106] According to one embodiment, M is an attention mask that includes as many latents as the output of the optional network NN4 or the signal z.

[0107] According to one embodiment, M is an attention mask whose real values ​​are between 0 and 1.

[0108] According to another embodiment, M contains only the values ​​0 or 1.

[0109] A residual connection (optional) is present on the lower branch, for well-known reasons of stability of the N3, N4 neural networks.

[0110] This produces a richer combination of the two channels (prediction and correction) thanks to the masking mechanism. This combination is represented by the output y of module N2.

[0111] The NN network is an optional neural network including a convolutional network that adapts the output of the previous network to the format of the Ivd image or FMREC latent feature maps.

[0112] The CNN network itself may have a residual connection, shown in the lower part of the figure, for known stability reasons.

[0113] The processing of IP / FMP and IC / FMC signals by the CNN module produces an image or Ivd / FMREC feature maps as output.

[0114] Illustrates a variant of a reconstruction neural network used for encoding and decoding according to one embodiment of the invention.

[0115] The N2 module described above is applied only to the IP / FMP prediction signal. In other words, only the prediction signal is masked by the attention module.

[0116] According to the example of the, the input channel of the IP / FMP prediction images / maps feeds a module N2, which has already been described previously in support of the, with input z and output y, which can be images, or feature maps of latent values. The module N2 has at least one attention element CL, as described previously. In the case of the, the upper branch of N2 therefore receives as input the signal z representative of the IP / FMP input alone. The network (N2) learns (the encoder) to construct a mask M from this input. This achieves a richer processing of the IP / FMP channel thanks to the masking mechanism.

[0117] The N2 network produces an output y which is the first channel feeding a CC (Cross Channel) mixing layer having two input channels, the second channel being fed by the IC / FMC correction images / cards. The mixing layer can perform a simple concatenation of the inputs or a more complex combination.

[0118] The reconstruction network then comprises a set of layers, or sub-networks, N1..NN. The N1..NN modules include convolution elements, and non-linearity elements (RELU, etc.) without limiting aspect, already described previously.

[0119] Illustrates another variant of a reconstruction neural network used for encoding and decoding according to one embodiment of the invention.

[0120] The N2 module described above is applied only to the IC / FMC correction signal. In other words, only the correction signal is masked by the attention module.

[0121] According to the example of the, the input channel of the IC / FMC prediction images / maps feeds a N2 module which has already been described previously in support of the. This is therefore a similar implementation mode to that of the, the roles of the IP / FMP and IC / FMC signals being reversed.

[0122] This achieves richer IC / MFC channel processing thanks to the masking mechanism.

[0123] The N2 network produces an output y which is the first channel feeding a CC (Cross Channel) mixing layer having two input channels, the second channel being fed by the IP / FMP correction images / maps. The mixing layer can perform a simple concatenation of the inputs or a more complex combination.

[0124] The reconstruction network then comprises a set of layers, or sub-networks, N1..NN. The N1..NN modules include convolution elements, and non-linearity elements (RELU, etc.) without limiting aspect, already described previously.

[0125] This is a flowchart representing an example of a decoding method that can be implemented by the decoding device of the.

[0126] The decoding described concerns a sub-sequence of images comprising at least one current image of the sequence to be decoded.

[0127] In a step E30, the coded data stream is obtained. It can be received from a communication network, or read from a storage medium. In addition, certain information (for example, a reference reconstruction network) can be read from accessible memories of the decoder. The data obtained are the correction information ICc, the prediction data IPc, and the parameters WMXc of a reconstruction network.

[0128] During a step E31, the correction information ICc is decoded.

[0129] According to one embodiment, they are decoded by a conventional technique within the reach of those skilled in the art, for example a prediction followed by decoding by a contextual entropic coder then dequantization, or they come from a standard decoder (JPEG, MPEG, etc.)

[0130] According to one embodiment, an IC image is decoded.

[0131] According to one embodiment, one or more FMC (latent) correction feature maps are decoded.

[0132] During a step E32, the IPc prediction information is decoded.

[0133] According to one embodiment, they are decoded by a conventional technique within the reach of those skilled in the art, for example a prediction followed by decoding by a contextual entropic coder then dequantization, or they come from a standard decoder (JPEG, MPEG, etc.)

[0134] According to one embodiment, an IP image is decoded.

[0135] According to one embodiment, one or more FMP (latent) prediction feature maps are decoded.

[0136] According to one embodiment, a motion field MV is decoded. This motion field may for example be dense (one piece of motion information per pixel). It may be decoded by a neural network, from decoded feature maps representative of the motion of the current image or estimated on the basis of the several IREF / FMREF reference images / maps. Conventionally, this motion field is used to compensate (warp) one or more IREF / FMREF reference images / maps during a step E33 in order to obtain the motion-compensated IP / FMP prediction images / maps.

[0137] In a step E34, the WMX reconstruction parameters of a reconstruction neural network of the MIX module are generated by decoding the WMXc values. For this purpose, any known neural network decoding technique corresponding to the coding technique used by the encoder can be used, for example the neural network coding standard proposed by the MPEG-7 part 17, NNR standard.

[0138] According to one embodiment, the reconstruction parameters are received in the bitstream by the decoder. According to one embodiment, certain parameters of the reconstruction neural network are accessed in an accessible memory of the decoder. According to one embodiment, certain reference parameters of the reconstruction neural network are accessed in an accessible memory of the decoder.

[0139] According to one embodiment, an indicator received in the stream makes it possible to select one or more reconstruction networks from a plurality of networks accessible to the decoder.

[0140] In a step E35, a combination of the IC / FMC correction images / maps and the IP / FMP prediction feature images / maps is performed by the MIX module.

[0141] According to one embodiment, the MIX module uses a CNN reconstruction neural network which takes as input the IC / FMC and IP / FMP images / maps and produces, for example using a series of convolutional layers and attention modules, as described previously, the Idv image.

[0142] According to another embodiment, the MIX module uses several cascaded reconstruction networks. They can be identical or different.

[0143] According to one embodiment, the MIX module produces as output a reconstructed image Ivd.

[0144] According to another embodiment, the MIX module produces as output one or more FMREC (latent) feature maps. The feature maps are then provided as input to another neural network, called a combination network, for example an MLP type network, to produce the final image Ivd.

Claims

Method for coding a current image (Iv) of a sequence of images, comprising the following steps:- generation of a set of prediction information (IP, FMP) of said at least one current image;- generation of a set of correction information (IC, FMC) of said at least one current image;- generation of at least one reconstruction neural network (MIX), defined by a set of reconstruction parameters (WMX);- processing of said correction information and said prediction information by said at least one reconstruction neural network to produce a representation of the current image (I'v);- updating of at least part of said prediction information and / or of said correction information and / or of said reconstruction parameters, as a function of a coding performance measurement;- coding of at least part of said correction and / or prediction information and of said reconstruction parameters. Method for decoding a current image (Idv) of an image sequence (SV), comprising the following steps: - obtaining (E31) a set of correction information (IC, FMC) of said current image; - obtaining (E32) a set of prediction information (IP, FMP) of said current image; - decoding (E34) a set of reconstruction parameters (WMX) representative of at least one reconstruction neural network (MIX, MLP); - processing (E35) of said correction information and said prediction information by said at least one reconstruction neural network to produce a decoded representation of said current image (FMREC, Ivd). Decoding or coding method according to one of the preceding claims, characterized in that said at least one reconstruction neural network comprises at least one attention module (N2, CL, M). Decoding or coding method according to one of the preceding claims, characterized in that said at least one reconstruction neural network comprises at least one convolution layer (N1...NN). Decoding or coding method according to one of the preceding claims, characterized in that said at least one reconstruction neural network applies the attention module (CL) to said prediction information (IP, FMP). Decoding or coding method according to one of the preceding claims, characterized in that said at least one reconstruction neural network applies the attention module (CL) to said correction information (IC, FMC). Decoding or coding method according to one of the preceding claims, characterized in that said at least one reconstruction neural network applies the attention module (CL) to a combination of said correction and prediction information (IP, FMP). Decoding method according to one of claims 2 to 7, characterized in that at least part of the set of reconstruction parameters (WMX) is obtained from data from said binary stream. Decoding method according to one of claims 2 to 8, characterized in that at least part of the set of reconstruction parameters (WMX) is decoded according to predetermined parameters. Decoding method according to one of claims 2 to 9, characterized in that at least part of the set of reconstruction parameters (WMX) is decoded as a function of reference parameters. Decoding method according to one of claims 2 to 10, characterized in that said at least one reconstruction network is selected from a plurality of reconstruction networks, and the binary stream comprises an indicator indicating the selection of said at least one network from the plurality. Decoding method according to one of claims 2 to 11, characterized in that said at least one reconstruction network corresponds to a plurality of cascaded reconstruction networks. Device for coding a current image (Iv) of a sequence of images (S v), said device being configured to implement:- generation (INIT / MAJ) of a set of prediction information (IP, FMP) of said current image;- generation (INIT / MAJ) of a set of correction information (IC, FMC) of said current image;- generation (INIT / MAJ) of at least one reconstruction neural network (MIX), defined by a set of parameters (WMX);- processing (MIX, CNN) of said correction information and said prediction information by said at least one reconstruction neural network to produce a representation of the current image (I'v);- updating (INIT / MAJ) of at least part of said prediction information and / or of said correction information and / or of said reconstruction parameters, as a function of a coding performance measurement;- coding (FPC, FCP, NNC) of at least part of said correction and / or prediction information and of said reconstruction parameters. Device for decoding a current image (Iv) of a sequence of images (SV), said device being configured to implement:- obtaining (FCD) a set of correction information (IC, FMC) of said current image;- obtaining (FPD) a set of prediction information (IP, FMP) of said current image;- decoding (NND) a set of reconstruction parameters (WMX) representative of at least one reconstruction neural network (MIX);- processing (MIX, CNN) of said correction information and said prediction information by said at least one reconstruction neural network to produce a decoded representation of said current image (FMREC, Ivd). Computer program comprising instructions for carrying out the steps of an encoding or decoding method according to claim 1 or 2 when said program is executed by a computer.

Images