Method and device for encoding and decoding images with an attention mechanism.

Abstract

Method and device for encoding and decoding images with an attention mechanism. The invention relates to a method and device for decoding a current image (Iv) from a sequence of images (Si). The method comprises the following steps: - obtaining (E31) a set of correction information (IC, FMC) for said at least one current image; - obtaining (E32) a set of prediction information (IP, FMP) for said at least one current image; - decoding (E34) a set of parameters (WMX) representative of at least one reconstruction neural network (MIX); - processing (E35) said correction information and said prediction information by said at least one reconstruction neural network to produce a representation of the decoded current image (FMREC, Ivd). Figure for the abstract: Fig. 1.

Description

Title of the invention: Method and device for coding and decoding images with attention mechanism. Prior art

[0001] The invention relates to the general field of coding digital image sequences. It relates more particularly to the compression of digital videos.

[0002] Digital videos are generally subject to source coding aimed at compressing them in order to limit the resources required for their transmission and / or storage. There are many coding standards, such as the ITU / MPEG standards (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) as well as their extensions (MVC, SVC, 3D-HEVC, etc.).

[0003] The encoding of an image is generally carried out by a prediction of the pixels using previously coded then decoded pixels present in the image being encoded, in which case we speak of “Intra prediction”, or of previously coded images, in which case we speak of “Inter prediction”.

[0004] In the case of inter-prediction, a classic approach is often used: after estimation and compensation of the motion at the encoder, a residual frame is calculated by subtracting the prediction frame from the original image. At the decoder, the prediction frame and the residual frame are added to obtain the reconstructed frame. This simple addition is not optimal, new approaches, notably neural, have recently been developed.

[0005] In this field, a conditional neural approach is proposed for example in the paper "Conditional Residual Coding: A Remedy for Bottleneck Problems in Conditional Inter Frame Coding", by F. Brand et al (arXiv:2307.12864). The proposed conditional coding models the conditional distribution of an image as a function of its predictor. However ([Fig.3] of this paper), the residual data and the predicted data are considered to be of equal importance. This limits the compression performance.

[0006] There is therefore a need for a solution for combining a residual signal with a predicted signal in an image sequence more efficiently. Statement of the invention

[0007] The invention relates to a method for coding a current image at least from a sequence of images, comprising the following steps: - generation of a set of prediction information for said at least one current image; - generation of a set of correction information for said at least one current image; - generation of at least one reconstruction neural network, defined by a set of reconstruction parameters; - processing said correction information and said prediction information by said at least one reconstruction neural network to produce a representation of the current image; - updating at least part of said prediction information and / or said correction information and / or said reconstruction parameters, based on a coding performance measurement; - coding at least part of said correction and / or prediction information and said reconstruction parameters.

[0008] The invention also relates to a method for decoding a current image from a sequence of images, comprising the following steps: - obtaining a set of correction information for said current image; - obtaining a set of prediction information for said current image; - decoding a set of reconstruction parameters representative of at least one reconstruction neural network; - processing said correction information and said prediction information by said at least one reconstruction neural network to produce a representation of the decoded current image.

[0009] For the purposes of the invention, encoding, or “coding”, means the operation which consists of representing a set of samples, or pixels, in a compact form carried for example by a digital binary train. Decoding means the operation which consists of processing a digital binary train to restore decoded samples.

[0010] By "sequence of images" is meant a plurality of ordered two-dimensional images, for example temporally in the case of a video. According to one example, the sequence corresponds to a scene. According to one example, the sequence corresponds to a set of predefined images, for example a fixed number, or, in the sense of the MPEG standards, a GOP (Group Of Pictures) comprising the images located between two images of the Intra-image type, also called "intra period". According to another example, the images can be views of the same scene represented in multi-views. According to another example, the images can be a plurality of temporal and multi-view images (immersive video).

[0011] By “current image” we mean an image of the sequence.

[0012] By "prediction information" is meant any information useful for generating a prediction of the current image. Conventionally, this information can be inter prediction information (motion, reference images, etc.), intra prediction (prediction direction), mode selection information (intra or inter mode or combined intra / inter or combination), etc.

[0013] By "correction information" is meant any information useful for generating a correction of the prediction of the current image. Conventionally, this information can be residual information, for example a residual image.

[0014] The "prediction information" and "correction information" are the inputs to the reconstruction neural network. They may contain data in the image domain or in the latent domain. For example, a prediction (resp. correction) image may consist of pixels or feature maps representative of the prediction (resp. correction) image in the latent domain.

[0015] A "latent domain" is a representation composed of a set of feature maps in two-dimensional form, the size of which is proportional to the input image. It can also include a set of one-dimensional vectors, or a set of scalar values.

[0016] By "reconstruction neural network" is meant a neural network such as a convolutional neural network, a multi-layer perceptron, an LSTM (for "Long Short Term Memory" in English), etc. The neural network is defined for example by a plurality of layers of artificial neurons and by a set of activation, weighting and addition functions (for example, a layer can calculate y = f(Ax+b), where y and b are vectors of dimension N, x a vector of dimension M, A is a matrix of dimension MxN, and f is the activation function). Preferably the reconstruction neural network is convolutional, that is to say that it comprises at least one convolution layer. A plurality of such networks can be cascaded.

[0017] By "parameter of the neural network" is meant one of the values ​​which characterizes the neural network, for example a weight associated with one of the neurons (filter or convolution coefficient, weighting, bias, value affecting the operation of the non-linearity, etc.)

[0018] By “processing (of input values) by a reconstruction neural network” is meant the application of a function expressed by a neural network to the input values ​​to produce output values ​​representative of the samples of the current image to be decoded (reconstructed) (resp. encoded).

[0019] By "representation of the current image" we mean its representation in the image domain (representation in the form of pixels) or latent (representation in the form of feature maps).

[0020] By "performance measurement" is meant a measurement between at least one value of a sample to be coded and a decoded value of said sample. The measurement can evaluate for example a distortion, or a perceptual error. It can be carried out on a sample or a plurality of samples (for example, the samples currents, or the current image, etc.). The measurement may also include a measurement of the flow rate, in particular associated with the coding of the neural network and / or the coding of the feature maps. The measurement may be a joint measurement between the flow rate and the distortion through their weighting. As is well known in the state of the art, the value of this measurement is generally minimized until a target or minimum value, or a predefined time, is reached.

[0021] By "sample" is meant a value taken from an image of the sequence. Sampling a signal produces a series of discrete values ​​called samples. In the case of an image signal, the sample is called a pixel, which can be, for example, a color pixel traditionally represented by a triplet of values, for example (R,G,B) or (Y,U,V). The position of the sample is identified by its abscissa (x) and ordinate (y) coordinates in the image.

[0022] Generally speaking, it is considered that the steps of a coding or decoding method should not be interpreted as being linked to a notion of temporal succession. In other words, the steps may be carried out in a different order than that indicated in the independent coding or decoding claim, or even in parallel.

[0023] The coding method according to the invention carries out during its training a construction of the coding parameters, from a sequence of input images, by training a reconstruction neural network on data sets of prediction and correction information, to obtain a faithful reconstruction of the image. Advantageously, the coding method encodes the reconstruction neural network. This can be optimized, and adapted in complexity or quality, according to the targeted complexity and the targeted ratio between flow rate and distortion. Similarly, the encoded and transmitted prediction and correction information can thus be optimized and adapted.

[0024] During training, or construction, or learning, the parameters of the neural network and the information to be coded are updated according to a performance measurement, for example of the rate-distortion type. When the training is finished, that is to say that the performance measurement obtained is satisfactory, the actual coding of the parameters of the reconstruction neural network as well as that of the prediction and reconstruction information can be carried out and the result of the coding stored or transmitted to the decoder. Advantageously, the training process therefore makes it possible to refine the parameters of the reconstruction neural network, as well as its input parameters, until an adequate representation in terms of performance is obtained, for example a desired balance between the rate generated and the distortion undergone by the input image being coded.Advantageously, the coding method according to the invention makes it possible to efficiently compress. the signal.

[0025] Advantageously, the decoding method is simple since it is sufficient to obtain the correction and prediction information and the reconstruction neural network to reconstruct a decoded version of the current image.

[0026] Advantageously, it is possible to design a transmission system which works image by image with low latency, each current image being decoded upon receipt of the parameters of the neural network and the associated prediction and correction information.

[0027] Advantageously, the decoding makes it possible to work with prediction data and residual data having the same format and the same resolution, which facilitates hardware implementation.

[0028] According to embodiments of the coding and / or decoding method:

[0029] - Said at least one reconstruction neural network comprises at least one Attention module. Advantageously, the neural network uses attention mechanisms that allow attention masks to be generated. Coding is thus improved.

[0030] By "attention module" is meant a layer of a neural network comprising an attention element. An attention element is an element that makes it possible to apply an attention mechanism to generate one or more masks that are used to weight (by multiplication or use of more complex functions) the characteristics of the image in order, for example, to adapt the allocation of the flow rate to the different regions and / or characteristics of the image according to their importance. The attention masks are applied to the inputs of the neural network (correction and prediction information). They are coded and decoded as parameters of the neural network. Advantageously, it is not necessary to use additional bits to code the masks.

[0031] - Said at least one reconstruction neural network comprises at least one convolution layer; thus, non-localized processing of the image can be performed, which improves the consistency of the generated image by limiting noise and enhancing the contours present in the image.

[0032] - Said at least one reconstruction neural network applies the module attention on said prediction information. Advantageously, according to this mode, the attention module is used to correct the prediction channel. Thus, better consideration of certain areas of the image can be carried out. The prediction information can be weighted. According to one example, if the generated mask "erases" the prediction information, the result is coding of the image in intra-image mode (only the correction information is used).

[0033] - Said at least one reconstruction neural network applies the module attention on said correction information. Advantageously, according to this mode, the attention module is used to correct the correction channel. Thus, better consideration of certain areas of the image to be corrected can be carried out. Weighting of the correction information can be carried out. According to one example, if the generated mask "erases" the correction information, this results in coding of the image in inter-image mode (only the prediction information is used).

[0034] - Said at least one reconstruction neural network applies the module attention on a combination of said correction and prediction information. Advantageously, according to this mode, the attention module is used to jointly correct the two input channels. Thus, a compromise can be found concerning the importance, or weighting, given to each of the two channels. The attention module can also correct artifacts related to the effects of the combination of the two inputs, or channels, of prediction and correction.

[0035] - At least part of the set of reconstruction parameters is obtained at from data from said binary stream; advantageously in this mode, the parameters of the neural network are transmitted in the stream. The reconstruction module can be transmitted in whole or in part in a quantized form and coded in a compact form using any quantizer and entropy coder accessible to those skilled in the art. A format similar to that of the MPEG-7 NNR standard can be used.

[0036] - At least part of the set of reconstruction parameters is decoded into function of predetermined parameters; advantageously according to this mode, part of the reconstruction parameters of the neural network is accessible to the decoder, for example they can be stored in a storage memory. In particular, part of the reconstruction module can be stored in a quantized form and coded in a compact form using any quantizer and entropy coder accessible to those skilled in the art. A format similar to that of the MPEG-7 NNR standard can be used.

[0037] - At least part of the set of reconstruction parameters is decoded into function of reference parameters; advantageously according to this mode, certain parameters of the neural network are decoded by taking into account reference parameters which can be stored in a memory (storage memory, network, etc.) accessible from the decoder. Thus the storage space or the transmission rate on the network can be effectively reduced: certain parameters can be coded / decoded in a complementary manner (for example by updating a kernel, a layer, a bias, etc.), others not be coded / decoded at all, because they are available in the reference parameters. The reference parameters can constitute a reference network. According to embodiment variants, which can be combined with each other: • the reference parameters can constitute a set of variants of a reconstruction network: type of convolution, attention modules, etc. Thus, only these parameters or their identification need to be coded, inserted into the flow and decoded by the decoder; • certain parameters are coded in a complementary manner to the parameters of the reference network. Thus, the decoder will simply have to decode these residual parameters and then add them, multiply them (or combine them in any other known way) with those of the reference neural network; • part of the parameters of the reference network is reused for the reconstruction network. For example, a complete layer of the network can be copied from the reference network, the parameters of this layer are therefore neither encoded, nor transmitted, nor decoded; • a reference network indicator is transmitted in the stream. It is thus possible to indicate a network to be used during decoding from among a plurality of possible networks known to the decoder.

[0038] - Said at least one reconstruction network is selected from a plurality of reconstruction networks, and the bitstream includes an indicator indicating selection of the at least one network of the plurality.

[0039] - Said at least one reconstruction network corresponds to a plurality of networks of Cascade reconstruction. Advantageously, the use of several cascaded networks makes it possible to enhance filtering and improve the consistency of the generated image by limiting its noise and improving its contours. In addition, each network can be learned individually or frozen independently. According to implementation variants, which can be combined with each other: • the bitstream includes an indicator indicating the use of at least one of these networks. Thus, depending on the quality and complexity required, one or more networks can be used for decoding. The indicator indicates, for example, that network 1 is mandatory, but that networks 2 and 3, which are cascaded behind network 1, are optional. The decoder can then choose whether or not to use them: if it uses them, it will obtain better quality at the expense of complexity, and vice versa. • At least two of the networks have common parameters. It is thus possible, advantageously, to transmit parameters characteristic of one network, which can be used for the others, which reduces the transmission cost. The network is thus shared through common parameters. • At least two of the networks have an identical structure and only their weights differ in part or completely. The same calculation elements are thus mu tualized.

[0040] Correlatively, the invention also relates to a coding device and a decoding device.

[0041] The characteristics and advantages of the coding or decoding method apply in the same way to the coding or decoding device according to the invention and vice versa.

[0042] The invention also relates to a computer program on a recording medium, this program being capable of being implemented in a computer or a coding or decoding device in accordance with the invention. This program comprises instructions adapted to the implementation of the corresponding method. This program can use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0043] The invention also relates to an information carrier or a recording medium readable by a computer, and comprising computer program instructions mentioned above. The information or recording carriers can be any entity or device capable of storing the programs. For example, the carriers can comprise a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a floppy disk or a hard disk, a DNA sequence, or a flash memory. Furthermore, the information or recording carriers can be transmissible media such as an electrical or optical signal, which can be conveyed via an electrical or optical cable, by radio link, by wireless optical link or by other means.

[0044] The program according to the invention can in particular be downloaded from an Internet-type network.

[0045] Alternatively, each information or recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of a method according to the invention. Brief description of the drawings

[0046] Other characteristics and advantages of the present invention will emerge from the description given below, with reference to the appended drawings which illustrate exemplary embodiments thereof which are not in any limiting nature.

[0047] [Fig-1] [Fig.l] schematically represents a decoding device used in the scope of the invention;

[0048] [Fig.2] [Fig.2] schematically represents a coding device used in the context of the invention;

[0049] [Fig.3] [Fig.3] illustrates an example of a mixing module used in coding and decoding according to two embodiments of the invention

[0050] [Fig.4] [Fig.4] illustrates a reconstruction neural network used in coding and to decoding according to one embodiment of the invention.

[0051] [Fig.5] [Fig.5] illustrates a variant of reconstruction neural network used to coding and decoding according to one embodiment of the invention.

[0052] [Fig.6] [Fig.6] illustrates another variant of reconstruction neural network used for encoding and decoding according to one embodiment of the invention

[0053] [Fig.7] [Fig.7] is a flowchart representing an example of a decoding method which can be implemented by the decoding device of [Fig.l]. Description of the embodiments

[0054] [Fig.l] schematically represents a decoding device used in the context of the invention.

[0055] The decoding device DECv of [Fig.l] decodes an image or sub-sequence v of the sequence, proceeding image by image. At the end of the decoding, the V images of the sequence are decoded.

[0056] A current sub-sequence to be decoded denoted Idv comprises at least one current image to be decoded, each image respectively comprising a plurality of samples. Subsequently, it is considered, without loss of generality, that the current sub-sequence comprises a single current image Iv. However, a current sub-sequence to be decoded may comprise, according to one embodiment, several images (which may for example be concatenated to form an image of greater size Iv).

[0057] The DECv decoder dedicated to the image receives as input the data necessary to decode it:

[0058] - IPc encoded data corresponding to prediction information. These in prediction training can be for example: - motion information (motion field, reference frame numbers, etc.); - intra-image prediction information (prediction mode, prediction directions, etc.);

[0059] - signal distortion information, such as trans parameters geometric formations (translations, homotheties, etc.);

[0060] - etc.

[0061] - ICc encoded data corresponding to IC correction information encoded, for example prediction residuals, or generally, data aimed at correcting the prediction obtained from the prediction information.

[0062] - the WMXc coded reconstruction parameters of at least one neural network of reconstruction. Other neural networks can be used, such as networks:

[0063] - for synthesizing prediction information; - summary of correction information; - predictive decoding of latent feature maps for prediction or correction;

[0064] - of oversampling and / or processing of latent feature maps of prediction or correction;

[0065] - post-processing of reconstructed data after mixing;

[0066] - etc.

[0067] The DECV decoding module comprises, for a current image, a REF module for obtaining reference images, an FPD module for decoding prediction information, an FCD module for decoding correction information, a MIX module for reconstruction by combining the compensated images / maps IP / FMP and the correction images / maps IC / FMC, a NND module for decoding a neural network. According to one embodiment, DECV produces as output a current decoded image, denoted Idv, comprising a plurality of decoded samples.

[0068] The FCD decoding module decodes correction information ICc. According to one embodiment, it is a conventional decoder, for example entropic, of the JPEG, or MPEG, etc. type, which produces as output a correction image (of residues) denoted IC. According to another embodiment, it decodes quantized data, for example using an entropic decoder, of one or more correction characteristic maps in the latent domain. According to one embodiment, the FCD module may comprise for this purpose a neural network for synthesizing correction information. According to one embodiment, the maps decoded by the FCD module, to the number of NFCs, are denoted FMC and are called correction maps of the current image, or current correction maps.

[0069] The FPD decoding module decodes prediction information IPc. According to one embodiment, it is a conventional decoder, for example entropic, of the JPEG, or MPEG, etc. type, which produces as output a prediction image denoted IP. According to one embodiment, it is a conventional decoder which produces as output a motion field MV. The FPD module may comprise for this purpose a motion synthesis neural network. According to another embodiment, it decodes quantized data, for example using an entropic decoder, of one or more prediction characteristic maps. The maps decoded by the FPD module, numbering NFP, are denoted FMP and are called prediction maps of the current image, or current prediction maps.

[0070] The optional REF module constitutes a set of NFR reference images / maps, noted IREF / FMREF, corresponding to one or more images / maps previously encoded and then decoded from the sequence. They can be predetermined or their reference can be decoded in the stream in the form of an indicator. In one embodiment, a single reference image is selected. In one embodiment, several reference images are selected. In one embodiment, a set of reference maps is selected in the latent domain. In one embodiment, for example in the case where an intra image is decoded, no reference map is selected and the REF module is absent.

[0071] The optional WARP module performs motion compensation of the reference images / maps from the MV motion field in order to obtain one or more prediction images or maps, IP / FMP. The WARP module may be absent if the FPD decoder directly decodes the prediction images or maps. In one embodiment, the WARP module produces the FMP maps with data only from the FPD module, i.e. the optional REF module does not exist, or the WARP module does not take into account the reference images submitted to it.

[0072] The MIX module uses as input one or more IP / FMP prediction image(s) or maps and one or more IC / FMC correction image(s) or maps and performs a combination of the two pieces of information to obtain the reconstructed image. The MIX module will be detailed in the remainder of the description. It comprises at least one convolutional type CNN neural network, i.e. it comprises at least one convolution layer. A convolution layer may include convolutional elements, comprising a residual structure or not and an attention module or not, as will be described subsequently.

[0073] The coded information ICc, IPc and WMXc are extracted from the BS stream which can be received on a communication network, or stored in a memory accessible to the decoder.

[0074] The decoding device DEC can be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the modules mentioned above can then be realized by the cooperation of the processor and computer program instructions stored in the aforementioned memory and designed to carry out the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0075] [Fig.2] schematically represents a coding device used in the context of the invention.

[0076] The coding device ENCv of [Fig.2] codes a current image or sub-sequence Iv of the sequence SV, proceeding image by image. At the end of the coding, the V images of the sequence are coded.

[0077] The ENCv coder receives as input a sequence of images including the image Iv and produces as output coded parameters. These parameters include:

[0078] - IPc encoded data corresponding to prediction information.

[0079] - ICc encoded data corresponding to correction information, for example of prediction residuals.

[0080] - the coded parameters of at least one reconstruction neural network. As mentioned when describing [Fig.l], other neural networks can be used.

[0081] The ENCV coding module comprises, for a current image, an NNC module for coding neural network parameters, an FCC module for coding correction information, an FPC module for coding prediction information, an RD-OPT evaluation module, an INIT / MAJ initialization and update module.

[0082] Furthermore, in a conventional manner, the coding module comprises a decoder similar to the DECv decoder which was previously described, in order to reconstruct the current image l'v coded then decoded.

[0083] During the process of training, or building, the coding, that is to say as long as the step of evaluating a performance is not satisfactory, the coding modules carry out a coding simulation, followed by a decoding, intended for the RD-OPT evaluation module. Subsequently, they carry out the actual coding of the data. In a known manner, the coding simulation can be identical to the actual coding, or carry out an approximation thereof.

[0084] The INIT / MAJ module is responsible for initializing and updating the values ​​of the images / feature maps and the parameters of the neural network(s). It updates the values ​​to be encoded of the current image, based on the results of the performance function. Once the prediction and correction information and the neural network(s) are stabilized, they can be encoded.

[0085] The RD-OPT module performs an evaluation and minimization of a coding performance. The evaluation function is for example of the rate-distortion type. The distortion can be evaluated between Iv and l'v, the rate can include the cumulative rate linked to the contributions of ICc, WMXc and IPc, or only a sub-part. The minimization can be carried out by gradient descent, or any other optimization method within the reach of those skilled in the art.

[0086] The ENC coding device may be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the above-mentioned modules may then be implemented by the cooperation of the processor and computer program instructions stored in the above-mentioned memory and designed to carry out the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0087] [Fig.3] illustrates an example of a mixing module used for coding and decoding according to two embodiments of the invention (a) and (b).

[0088] The MIX module includes a CNN reconstruction neural network which will be described later with reference to figures 4, 5 and 6.

[0089] On the left of figure (a), according to a first embodiment, the MIX module receives as input a correction image IC and a prediction image IP. It generates as output a reconstructed image Ivd.

[0090] On the right of figure (b), according to a second embodiment, the MIX module receives as input a set of correction characteristic maps, FMC, and a set of prediction characteristic maps FMP.

[0091] The FMC and FMD feature maps are not necessarily in the same number or in the same form. The MIX module is adapted to process these inputs in their respective format.

[0092] According to one embodiment, the MIX module generates a reconstructed image Ivd as output.

[0093] According to one embodiment, the MIX module generates as output a set of FMREC characteristic maps representative (in the latent domain) of a reconstructed image Ivd. In this case, these maps can be processed by a so-called combination neural network, comprising for example an MLP (multi-layer perceptron) or convolutional type network, to produce the reconstructed image Ivd.

[0094] [Fig.4] illustrates a reconstruction neural network (CNN) used by the MIX mixing module for encoding and decoding according to one embodiment of the invention.

[0095] The MIX module may comprise one or more reconstruction networks, in cascade. The networks may be identical in their structure and parameters, or different.

[0096] According to the example of [Fig.4], the reconstruction network comprises a CC (Cross Channel) mixing layer of the two input sets, IC / FMC correction images / maps on the first input and IP / FMP prediction images / maps on the second input. The mixing layer can perform a simple concatenation of the inputs or a more complex combination including for example a convolution or a combination by neural network on the concatenated inputs. The reconstruction network then comprises a set of layers, or sub-networks, NI, N2..NN.

[0097] The modules Ni (NI, N2..NN) include convolution elements, and non-linearity elements (RELU, etc.) without limiting aspect.

[0098] A convolution element can be of residual type or not, have kernels of any size. It is potentially followed by a bias as is well known to those skilled in the art. Any activation function within the reach of those skilled in the art can be used (ReLU, sigmoid, Softmax, Softplus, Softsign, etc.). The convolution elements can be composed of 2D convolutions, that is to say that the convolution kernel is applied jointly to the rows and columns of the image or latent value map. The convolution elements can be composed of 1D convolutions, in a separable form. In this case, in a known manner, rather than performing the convolution on the two-dimensional image or latent value map, a first one-dimensional (1D) convolution operator is performed on the rows, then a second convolution on the columns of the representation obtained after the first operator. The row-column processing can be reversed to perform processing by column and then by row.

[0099] According to a first embodiment, not shown, the module N2 comprises, like the other modules Ni, one or more convolution layers followed by non-linearities.

[0100] The upper part of the figure details another preferred embodiment of the module N2. It receives as input an intermediate signal z representative of a composition, or mixing, by the module CC, of ​​the two inputs, possibly followed by a first network (NI), for example convolution.

[0101] According to this embodiment, the module N2 comprises an attention element CL. The attention mechanism makes it possible to apply a differentiated processing on certain areas of the image to be reconstructed, thanks to a masking in the image or latent domain. Such operations are used in many learning-based compression systems. For example, the document "Non-local Attention Optimized Deep Image Compression" by Liu et al. (available at https: / / arxiv.org / pdf / 1904.09757.pdf) describes such attention mechanisms useful in the context of entropy coding. In particular, the introduction of an attention mask makes it possible to place particular importance on certain areas; in the context of this publication, this involves adapting the bit rate allocation to the different values ​​of the feature maps (for example, by allocating more bit rate to the textured areas of the image).

[0102] The upper branch of N2 receives as input the signal z from the NI network or from a combination of the two inputs. The network (N2) teaches (the encoder) to construct a mask M which is obtained by optional processing of the input signal by a neural network (NN3) comprising a set of convolutions followed by non-linearities, then processing of the values ​​by the CL module which can, in a known manner, be of the Clamp, sigmoid, Arctg, etc. type.

[0103] The central branch of N2 corresponds to the main processing of the signal z by a neural network (NN4). This module is optional. The output of the NN4 module is multiplied by the mask M. According to one embodiment, only certain areas are retained after multiplication by the values ​​of the mask M. According to one embodiment, the entries are erased after multiplication by the values ​​of the mask M

[0104] According to one embodiment, M is an attention mask which is applied to an image from the optional network NN4 or from the signal z.

[0105] According to one embodiment, M is an attention mask which applies to all the latents from the optional network NN4 or the latents z.

[0106] According to one embodiment, M is an attention mask which comprises as many latents as the output of the optional network NN4 or the signal z.

[0107] According to one embodiment, M is an attention mask whose real values ​​are between 0 and 1.

[0108] According to another embodiment, M contains only the values ​​0 or 1.

[0109] A residual connection (optional) is present on the lower branch, for well-known reasons of stability of the neural networks N3, N4.

[0110] This produces a richer combination of the two channels (prediction and correction) thanks to the masking mechanism. This combination is represented by the output y of module N2.

[0111] The NN network is an optional neural network including a convolutional network that adapts the output of the previous network to the format of the Ivd image or FMREC latent feature maps.

[0112] The CNN network of [Fig.4] may itself include a residual connection, shown in the lower part of the figure, for known reasons of stability.

[0113] The processing of the IP / FMP and IC / FMC signals by the CNN module produces as output an image or Ivd / FMREC characteristic maps.

[0114] [Fig.5] illustrates a variant of reconstruction neural network used for coding and decoding according to one embodiment of the invention.

[0115] The N2 module described above is applied only to the IP / FMP prediction signal. In other words, only the prediction signal is masked by the attention module.

[0116] According to the example of [Fig.5], the input channel of the IP / FMP prediction images / maps feeds a module N2, which has already been described previously in support of [Fig.4], with input z and output y, which can be images, or feature maps of latent values. The module N2 comprises at least one attention element CL, as described previously. In the case of [Fig.5], the upper branch of N2 therefore receives as input the signal z representative of the IP / FMP input alone. The network (N2) learns (the encoder) to construct a mask M from this input. This achieves richer processing of the IP / FMP channel thanks to the masking mechanism.

[0117] The N2 network produces an output y which is the first channel feeding a CC (Cross Channel) mixing layer having two input channels, the second channel being fed by the IC / FMC correction images / cards. The mixing layer can perform a simple concatenation of the inputs or a more complex combination.

[0118] The reconstruction network then comprises a set of layers, or sub-networks, NL.NN. The NL.NN modules include convolution elements, and non-linearity elements (RELU, etc.) without limiting aspect, already described previously.

[0119] [Fig.6] illustrates another variant of reconstruction neural network used for coding and decoding according to an embodiment of the invention.

[0120] The N2 module described above is applied only to the IC / FMC correction signal. In other words, only the correction signal is masked by the attention module.

[0121] According to the example of [Fig.6], the input channel of the IC / FMC prediction images / maps feeds a module N2 which has already been described previously in support of [Fig.4]. This is therefore an implementation mode similar to that of [Fig.5], the roles of the IP / FMP and IC / FMC signals being reversed.

[0122] This achieves richer processing of the IC / MFC channel thanks to the masking mechanism.

[0123] The N2 network produces an output y which is the first channel feeding a CC (Cross Channel) mixing layer having two input channels, the second channel being fed by the IP / FMP correction images / maps. The mixing layer can perform a simple concatenation of the inputs or a more complex combination.

[0124] The reconstruction network then comprises a set of layers, or sub-networks, NL.NN. The NL.NN modules include convolution elements, and non-linearity elements (RELU, etc.) without limiting aspect, already described previously.

[0125] [Fig.7] is a flowchart representing an example of a decoding method that can be implemented by the decoding device of [Fig.l].

[0126] The decoding described relates to a sub-sequence of images comprising at least one current image of the sequence to be decoded.

[0127] During a step E30, the coded data stream is obtained. It can be received from a communication network, or read from a storage medium. In addition, certain information (for example, a reference reconstruction network) can be read from accessible memories of the decoder. The data obtained are the correction information ICc, the prediction data IPc, and the parameters WMXc of a reconstruction network.

[0128] During a step E31, the correction information ICc is decoded.

[0129] According to one embodiment, they are decoded by a conventional technique within the reach of those skilled in the art, for example a prediction followed by decoding by a contextual entropic coder then dequantization, or they come from a standard decoder (JPEG, MPEG, etc.)

[0130] According to one embodiment, an IC image is decoded.

[0131] According to one embodiment, one or more characteristic maps of (latent) FMC corrections are decoded.

[0132] During a step E32, the prediction information IPc is decoded.

[0133] According to one embodiment, they are decoded by a conventional technique within the reach of those skilled in the art, for example a prediction followed by decoding by a contextual entropic coder then dequantization, or they come from a standard decoder (JPEG, MPEG, etc.)

[0134] According to one embodiment, an IP image is decoded.

[0135] According to one embodiment, one or more characteristic maps of (latent) FMP predictions are decoded.

[0136] According to one embodiment, a motion field MV is decoded. This motion field movement can be for example dense (one piece of motion information per pixel). It can be decoded by a neural network, from decoded feature maps representative of the motion of the current image or estimated on the basis of the several IREF / FMREF reference images / maps. Conventionally, this motion field is used to compensate (warp) one or more IREF / FMREF reference images / maps during a step E33 in order to obtain the motion-compensated IP / FMP prediction images / maps.

[0137] During a step E34, the WMX reconstruction parameters of a reconstruction neural network of the MIX module are generated by decoding the WMXc values. For this purpose, any known neural network decoding technique corresponding to the coding technique used by the coder can be used, for example the neural network coding standard proposed by the MPEG-7 part 17, NNR standard.

[0138] According to one embodiment, the reconstruction parameters are received in the bitstream by the decoder. According to one embodiment, certain parameters of the reconstruction neural network are accessed in an accessible memory of the decoder. According to one embodiment, certain reference parameters of the reconstruction neural network are accessed in an accessible memory of the decoder.

[0139] According to one embodiment, an indicator received in the stream makes it possible to select one or more reconstruction networks from among a plurality of networks accessible to the decoder.

[0140] During a step E35, a combination of the correction images / maps IC / FMC and IP / FMP prediction feature images / maps is performed by the MIX module.

[0141] According to one embodiment, the MIX module uses a CNN reconstruction neural network which takes as input the IC / FMC and IP / FMP images / maps and produces, for example using a series of convolutional layers and attention modules, as described previously, the Idv image.

[0142] According to another embodiment, the MIX module uses several cascaded reconstruction networks. They may be identical or different.

[0143] According to one embodiment, the MIX module produces as output a reconstructed image Ivd.

[0144] According to another embodiment, the MIX module produces as output one or more FMREC (latent) characteristic maps. The characteristic maps are then provided as input to another neural network, called a combination network, for example an MLP type network, to produce the final image Ivd.

Claims

Claims

1. Method for coding a current image (Iv) of a sequence of images, comprising the following steps: - generation of a set of prediction information (IP, FMP) of said at least one current image; - generation of a set of correction information (IC, FMC) of said at least one current image; - generation of at least one reconstruction neural network (MIX), defined by a set of reconstruction parameters (WMX); - processing of said correction information and of said prediction information by said at least one reconstruction neural network to produce a representation of the current image (Iv); - updating at least part of said prediction information and / or of said correction information and / or of said reconstruction parameters, as a function of a coding performance measurement;- coding at least part of said correction and / or prediction information and said reconstruction parameters;

2. Method for decoding a current image (Idv) of an image sequence (SV), comprising the following steps: - obtaining (E31) a set of correction information (IC, FMC) of said current image; - obtaining (E32) a set of prediction information (IP, FMP) of said current image; - decoding (E34) a set of reconstruction parameters (WMX) representative of at least one reconstruction neural network (MIX, MLP); - processing (E35) said correction information and said prediction information by said at least one reconstruction neural network to produce a decoded representation of said current image (FMREC, Ivd).

3. Decoding or coding method according to one of the preceding claims, characterized in that said at least one reconstruction neural network comprises at least one attention module (N2, CL, M).

4. Decoding or coding method according to one of the preceding claims, characterized in that said at least one reconstruction neural network comprises at least one convolution layer (N1...NN).

5. Decoding or coding method according to one of the preceding claims, characterized in that said at least one reconstruction neural network applies the attention module (CL) to said prediction information (IP, FMP).

6. Decoding or coding method according to one of the preceding claims, characterized in that said at least one reconstruction neural network applies the attention module (CL) to said correction information (IC, FMC).

7. Decoding or coding method according to one of the preceding claims, characterized in that said at least one reconstruction neural network applies the attention module (CL) to a combination of said correction and prediction information (IP, FMP).

8. Decoding method according to one of claims 2 to 7, characterized in that at least part of the set of reconstruction parameters (WMX) is obtained from data from said binary stream.

9. Decoding method according to one of claims 2 to 8, characterized in that at least part of the set of reconstruction parameters (WMX) is decoded according to predetermined parameters.

10. Decoding method according to one of claims 2 to 9, characterized in that at least part of the set of reconstruction parameters (WMX) is decoded as a function of reference parameters.

11. Decoding method according to one of claims 2 to 10, characterized in that said at least one reconstruction network is selected from a plurality of reconstruction networks, and the bit stream comprises an indicator indicating the selection of said at least one network from the plurality.

12. Decoding method according to one of claims 2 to 11, characterized in that said at least one reconstruction network corresponds to a plurality of cascaded reconstruction networks.

13. Device for coding a current image (Iv) of a sequence of images (Sv), said device being configured to implement: - generation (INIT / MAJ) of a set of prediction information (IP, FMP) of said current image; - generation (INIT / MAJ) of a set of correction information (IC, FMC) of said current image; - generation (INIT / MAJ) of at least one reconstruction neural network (MIX), defined by a set of parameters (WMX); - processing (MIX, CNN) of said correction information and of said prediction information by said at least one reconstruction neural network to produce a representation of the current image (l'v); - updating (INIT / MAJ) of at least part of said prediction information and / or said correction information and / or said reconstruction parameters, as a function of a coding performance measurement; - coding (FPC, FCP, NNC) of at least part of said correction and / or prediction information and said reconstruction parameters.

14. Device for decoding a current image (Iv) of a sequence of images (SV), said device being configured to implement: - obtaining (FCD) a set of correction information (IC, FMC) of said current image; - obtaining (FPD) a set of prediction information (IP, FMP) of said current image; - decoding (NND) a set of reconstruction parameters (WMX) representative of at least one reconstruction neural network (MIX); - processing (MIX, CNN) of said correction information and said prediction information by said at least one reconstruction neural network to produce a decoded representation of said current image (FMREC, Ivd).

15. A computer program comprising instructions for carrying out the steps of an encoding or decoding method according to claim 1 or 2 when said program is executed by a computer.

Images