Method and device for coding and decoding images with correction mechanism.

Abstract

Method and device for coding and decoding images with correction mechanism The invention relates to a method for coding and decoding a signal of a plurality of samples. The decoding method comprises the following steps: – obtaining at least one characteristic map, called the original map, representative of the signal in the latent domain; – decoding the parameters of a first neural network, called the synthetic neural network, comprising at least two synthetic neural layers; – decoding the parameters of a second neural network, called the control network; – processing said at least one original characteristic map by said synthetic neural network to produce the decoded signal at output, said processing comprising at least one step of producing at least one characteristic map, called the intermediate map, at the output of a layer of one of said synthetic neural layers, called the generator layer.– application of said control neural network to said at least one intermediate characteristic map to obtain a set of control parameters; – application of at least one function parameterized by at least part of said set of control parameters to the input or output of a layer of said synthesis neural layers, called the corrected layer. Figure for the abstract: Fig. 1.

Description

Title of the invention: Method and device for coding and decoding images with correction mechanism. Prior art

[0001] The invention relates to the general field of digital image coding. It relates more particularly to the compression of digital videos.

[0002] Digital videos are generally subject to source coding aimed at compressing them in order to limit the resources required for their transmission and / or storage. There are many coding standards, such as the ITU / MPEG standards (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) as well as their extensions (MVC, SVC, 3D-HEVC, etc.). In these approaches, the encoding of an image is generally carried out by predicting the pixels using previously coded and then decoded pixels present in the image being encoded, in which case we speak of “Intra prediction”, or previously coded images, in which case we speak of “Inter prediction”.

[0003] In addition to these classic approaches, approaches based on artificial intelligence, and in particular neural ones, tend to develop.

[0004] Some approaches use the concept of "auto-encoder", learning algorithms based on artificial neural networks, which allow to construct a new representation of a data set. The architecture of an auto-encoder is made up of two parts: the encoder and the decoder. The encoder is made up of a set of layers of neurons, which process the data in order to construct new representations called "encoded", also called "latent representations". In turn, the layers of neurons of the decoder receive these representations and filter them in order to try to reconstruct the initial data. The differences between the reconstructed data and the initial data make it possible to measure the error made by the auto-encoder. Training consists of modifying the parameters of the auto-encoder in order to reduce the reconstruction error measured on the different samples of the data set.The performance of such autoencoder-based systems comes at the cost of a considerable increase in memory footprint and complexity compared to conventional approaches as proposed by compression standards. They can have millions of parameters and may require up to a million MACs (multiply-accumulate) to decode a single pixel. This makes such decoders significantly more complex than conventional decoders, which could hamper the adoption of learning-based compression.

[0005] To improve signal processing by neural networks, so-called attention mechanisms have been proposed. An attention mechanism allows to focus the processing (attention) on certain parts of the signal. For example, in the article "Deep Attentional Features for Prostate Segmentation in Ultrasound" by Y. Wang et al. (International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 523-530, 2018), it is proposed to modify the latent representations of the input image by an attention network. Such mechanisms are very little used in the field of signal compression, because they are not very effective in this context.

[0006] In patent application WO2023199172A, it is proposed to improve neural networks used in a video encoder. According to one example, these neural networks compose autoencoders. According to other examples, they complement or replace an element of the video encoder, for example a loop filter. The improvement takes the form of improvement parameters, for example parameters of the aforementioned filtering neural network, which are evaluated by an estimator (LST-SQ) during the training of the encoder, and can be transmitted to the decoder in the form of an update of parameters of the neural network. However, such a method remains very expensive.

[0007] There is therefore a need for a solution to efficiently correct a neural network in a neural decoder. Statement of the invention

[0008] The invention relates to a method for coding a signal of a plurality of samples, comprising the following steps: - generation of at least one original feature map; - generation of at least one synthetic neural network, defined by a set of synthetic network parameters comprising at least two synthetic neural layers, and a control neural network defined by a set of control network parameters; - processing of said at least one original characteristic map by said synthesis neural network to produce a representation of the decoded signal at output, said processing comprising at least one step of producing at least one characteristic map, called generator map, at the output of one of said synthesis neural layers, called generator layer; - applying said control neural network to said at least one generator card to obtain a set of control parameters; - application of at least one function parameterized by at least part of said set of parameters to the input or output of one of said synthesis neuronal layers, called corrected layer; - updating at least a portion of said original feature maps and / or said synthesis and / or control neural network parameters, based on a coding performance measure; - encoding at least a portion of said original feature maps and / or said synthetic and / or control neural network parameters, based on a coding performance measure.

[0009] The invention also relates to a method for decoding a signal from a plurality of samples, comprising the following steps: - obtaining at least one characteristic map, called the original map, representative of the signal in the latent domain; - decoding the parameters of a first neural network, called a synthetic neural network, comprising at least two synthetic neural layers; - decoding the parameters of a second neural network, called the control neural network; - processing of said at least one original characteristic map by said synthesis neural network to produce the decoded output signal, said processing comprising at least one step of producing at least one characteristic map, called generator map, at the output of one of said synthesis neural layers, called generator layer. - applying said control neural network to said at least one generator card to obtain a set of control parameters; - application of at least one function parameterized by at least part of said set of parameters to the input or output of one of said synthesis neuronal layers, called the corrected layer.

[0010] For the purposes of the invention, encoding, or "coding", means the operation which consists of representing a set of samples in a compact form carried for example by a digital binary train. Decoding means the operation which consists of processing a digital binary train to restore decoded samples.

[0011] By "sample" of the signal is meant a value taken from the signal. Sampling the signal produces a series of discrete values ​​called samples. In the case of an image signal, the sample is called a pixel which can be, for example, a color pixel traditionally represented by a triplet of values, for example (R,G,B) or (Y,U,V). Generally speaking, a sample is represented by a vector of one or more components, or data.

[0012] By "signal comprising a plurality of samples" is meant a signal with one (audio, sound), two (image) or more than two (stereoscopic, multiscopic image, image associated with a depth map, video, etc.) dimensions. Depending on this dimensionality, the sample has one, two or more coordinates in the signal. In the case of an image signal, the position of the sample is identified by its abscissa (x) and ordinate (y) coordinates.

[0013] By "feature map" is meant an abstract representation of the signal comprising a plurality of variable scalar data, which are also called characteristic values, for example real or integer numbers. In a known manner, these maps are also referred to as "latent representation" or "representation in the latent domain" of the signal. These feature maps may have a two-dimensional representation in the form of images of scalar values, or one-dimensional in the form of vectors, or correspond to a set of scalar values.

[0014] By "synthetic feature maps" is meant any feature map manipulated by the synthesis network, including the input and output feature maps of the synthesis network, as well as the intermediate feature maps, manipulated by the layers of the synthesis network. One of the neural layers of the synthesis network produces as output the generator map(s) as output, applied as input to the control neural network.

[0015] By "neural network" is meant a neural network such as a convolutional neural network, a multi-layer perceptron, etc. The neural network is defined for example by a plurality of layers of artificial neurons and by a set of activation, weighting and addition functions (for example, a layer can calculate y = f(Ax+b), where y and b are vectors of dimension N, x a vector of dimension M, A is a matrix of dimension MxN, and f is the activation function). A plurality of such networks can be cascaded.

[0016] By "parameter of the neural network" is meant one of the values ​​which characterizes the neural network, for example a weight associated with one of the neurons (filter or convolution coefficient, weighting, bias, value affecting the operation of the non-linearity, etc.)

[0017] By "processing (of input values) by a neural network" is meant the application of a function expressed by the neural network to the input values ​​to produce output values ​​representative of the samples of the current signal to be decoded (reconstructed) (resp. encoded).

[0018] By "performance measurement" is meant a measurement between at least one value of a sample to be coded and a decoded value of said sample. The measurement can evaluate for example a distortion, or a perceptual error. It can be carried out on a sample or a plurality of samples (for example, the current samples, or the current image, etc.) The measurement can also include a measurement of the flow rate, in particular associated with the coding of the neural network and / or the coding of the characteristic maps. The measurement can be a joint measurement between the flow rate and the distortion through their weighting. As is well known in the state of the art, the value of this measurement is generally minimized until a target or minimum value, or a predefined time, is reached.

[0019] By "control parameters" is meant any information useful for correcting one or more synthetic feature maps. Such a map is called a corrected map. The layer to which the correction is applied is called the corrected layer.

[0020] By "parameterized function" is meant a function defined by a set of parameters. Any known mathematical function or combination or composition of such functions can be considered. The function can be represented by an artificial neuron, also called a perceptron, classically corresponding to a set of parameters that are its weights, biases and possibly activation functions. Naturally, several functions can be applied to an input or output layer. A set of parameterized functions can therefore represent one or more layers of artificial neural networks. The function or set of functions are applied to the output or input of one or more layers of the synthesis network. For example, a complete neural network, defined by the parameters of its parameterizable functions, can be applied to a synthesis layer.In another example, a simple mathematical function can be applied to the values ​​of a synthesis layer.

[0021] Generally speaking, it is considered that the steps of a coding or decoding method should not be interpreted as being linked to a notion of temporal succession. In other words, the steps may be carried out in a different order than that indicated in the independent coding or decoding claim, or even in parallel.

[0022] The coding method according to the invention carries out during its training a construction of the coding parameters, from an input signal, for example a sequence of images, by training a synthesis neural network and a characteristic map control neural network, to obtain a faithful reconstruction of the signal. Advantageously, the coding method encodes the neural networks in the form of parameters. These can be optimized, and adapted in complexity or quality, according to the targeted complexity and a targeted ratio between bit rate and distortion. Similarly, the encoded and transmitted characteristic maps can thus be optimized and adapted.

[0023] During training, or construction, or learning, the parameters of the neural networks and the information to be coded are updated according to a performance measure, for example of the rate-distortion type. When the training is finished, that is to say that the performance measure obtained is satisfactory, the effective coding of the parameters of the reconstruction neural network as well as that of the feature maps can be carried out and the result of the coding stored or transmitted to the decoder. Advantageously, the training process therefore makes it possible to refine the parameters of the neural networks, as well as its input parameters, until an adequate representation in terms of performance is obtained, for example a desired balance between the generated bit rate and the distortion undergone by the input image being coded. Advantageously, the coding method according to the invention makes it possible to efficiently compress the signal.

[0024] Advantageously, the decoding method is simple since it is sufficient to obtain the neural networks and the characteristic maps to reconstruct a decoded version of the signal, for example a current image of a sequence of images being processed. Advantageously in this context, it is possible to design a transmission system which works image by image with low latency, each current image being decoded upon receipt of the parameters of the neural networks and the associated characteristic maps.

[0025] Advantageously, the control parameters are generated by a control network at the input of which are presented latent characteristic maps from the synthesis network. Thus, the layers of the network which are located before the generating layer make it possible to produce a more relevant signal for the control network, which improves the quality of the control parameters. In addition, the cost of coding the control parameters is low, since it is limited to that of the parameters of the control neural network, which make it possible to reconstitute the control neural network, which itself makes it possible to produce the control parameters. This makes it possible to reduce the quantity of data necessary for the transmission of the signal.

[0026] Advantageously, the control parameters parameterize a function that can be applied in whole or in part to one or more synthesis layers. They can thus be shared, that is to say that several parameters (and / or parameterized functions) can be used as input or output of several layers. This makes it possible both to reduce the quantity of data necessary for the transmission of the signal and to reduce the complexity, since it is sufficient to produce the set of parameters only once.

[0027] Advantageously, the control parameters can be applied to several outputs or inputs of a layer. They can thus be shared, that is to say that several parameters (and / or parameterized functions) can be used for several inputs or outputs of a layer, for example feature maps with several resolutions.

[0028] The control parameters make it possible to drive a function for modifying the low-complexity (latent) feature maps (for example, affine), which makes it possible to replace more complex processing that would have had to be carried out by the synthesis network.

[0029] According to embodiments of the coding and / or decoding method:

[0030] - Said generating layer is equal to said corrected layer. Advantageously in this mode, the proximity between the corrected layer and the generating layer allows for control parameters that are most locally adapted, thus contributing to the overall performance of the system.

[0031] - Said generating layer is located before said corrected layer; by "before" we mean means that the generator layer is located before the corrected layer in the order of the synthesis layers organized from the first (network input) to the last (network output); advantageously, this avoids the complexity of a "recursive" network, i.e. in which the generator layer, necessary for the input of the control network, would be located behind the layer corrected by the control network.

[0032] - Said generator layer is the first layer of the synthesis network. Advantageously, this allows us to start from a more abstract representation, far from the signal to be decoded and with lower resolution, the calculations are therefore reduced and the control network can be less complex.

[0033] - Said generator layer is the last layer of the synthesis network. Advantageously, the signal is then corrected as close as possible to the image domain, this allows contours to be corrected with a precise profile; the last layers of the synthesis network being the most detailed, this allows the control network to contribute as much as possible to the quality of the reconstructed signal.

[0034] - The method comprises a step of decoding said at least one map of original features. Advantageously, the feature maps can be encoded following optimization at the encoder, and transmitted to the decoder. Any encoding / decoding tool can be used: standard encoder (HEVC, VVC, JPEG, etc.), encoding of a frequency map from a wavelet transform, DCT, entropy encoder, etc.

[0035] - At least one synthesis neural layer is applied to at least one map of input characteristics and produces at least one characteristic map as output, said input and output maps being called synthesis characteristic maps; advantageously according to this mode, the synthesis network produces step by step a series of synthesis characteristic maps which can be seen as an increasingly faithful approximation of the signal in the latent domain.

[0036] - The same parameterized function is applied to at least two maps of synthesis characteristics; advantageously according to this mode, the same parameterized function (for example, a convolution, a multiplication, a masking, etc.) can be applied either to several places in the synthesis (at the input or output of distinct layers) or to several characteristic maps of the same synthesis layer, or both. We can thus share the functions to be applied for the control of several synthesis characteristic maps.

[0037] - The same control parameters are used to set at least two functions applied respectively to at least two synthesis characteristic maps; advantageously, the control parameters can thus be pooled to correct several synthesis characteristic maps using the same function or distinct functions.

[0038] - The set of control parameters comprises at least one kernel of convolution; advantageously, one or more convolution kernels are generated by the control module, so as to perform a convolution operation on the synthesis feature maps.

[0039] - The set of control parameters comprises at least one map of characteristics, called control characteristic map; advantageously, one or more latent value characteristic maps are generated by the control neural network. These characteristic maps can be used to parameterize a simple function (for example, multiplication and / or addition), or a more complex one (such as filtering), to be applied directly to the values ​​of the synthesis characteristic maps.

[0040] - The resolution of said at least one control characteristic map is identical to that of said at least one synthesis characteristic map; advantageously according to this mode, the parameterized function can make it possible to perform a simple operation between a value of the control characteristic map and one or more values ​​located in the same neighborhood of the synthesis characteristic map (addition, multiplication, masking, convolution, etc.). In this case, there is at least one control parameter available for each element of the synthesis characteristic map, this degree of precision making it possible to obtain maximum coding performance.

[0041] - The resolution of said at least one control characteristic map is different from that of said at least one synthesis characteristic map; advantageously according to this mode, the parameterized function can make it possible to carry out a simple operation between a value of the control characteristic map and several values ​​of the synthesis characteristic map (addition, multiplication, masking, convolution, etc.) which can be for example in the vicinity of the position of the value of the control map, after scaling. It can be a block, for example.

[0042] - The resolution of said at least one control characteristic map is lower than that of said at least one synthetic characteristics map; advantageously according to this mode, the complexity of the map generation step of control characteristics is lowered, and the same control parameter is shared between several elements of the synthesis characteristics map.

[0043] - Said at least one function parameterized by said at least one card of control characteristics is applied to at least two separate synthesis maps.

[0044] - Said at least one function parameterized by said at least one card of control characteristics is applied to at least two synthesis maps of the same layer; advantageously, according to this mode, a function parameterized by the same control characteristics map applies to several synthesis characteristics maps at the same location in the synthesis network. For example, a control map can be applied (via the same function or several distinct functions) to apply a treatment at the same location to several synthesis characteristics maps at different resolutions. It is thus possible to share the control parameters to correct several synthesis characteristics maps.

[0045] - Said at least one (synthetic or) control neural network comprises at least least one convolution layer; thus, image filtering processing can be performed, which improves the consistency of the generated image by limiting noise and enhancing the contours present in the image.

[0046] - Said at least one (synthetic or) control neural network comprises at least least one attention module; "attention module" means a layer of a neural network comprising an attention element. An attention element is an element that allows an attention mechanism to be applied to generate one or more masks as parameters. These masks can parameterize a function that uses them to weight (by multiplication or use of more complex functions) the synthesis feature maps in order, for example, to adapt the allocation of the flow to the different regions and / or characteristics of the image according to their importance. The attention masks are encoded and decoded as parameters of the control network. Advantageously, it is not necessary to use additional bits to encode the masks.

[0047] - At least part of the set of parameters of the neural networks synthesis and / or control is obtained from data of said binary stream; advantageously according to this mode, the parameters of the neural networks are transmitted in the stream. The parameters can be transmitted in whole or in part in a quantized form and coded in a compact form using any quantizer and entropy coder accessible to those skilled in the art. A format similar to that of the MPEG-7 NNR standard can be used.

[0048] - At least part of the set of parameters of the neural networks of synthesis and / or control is decoded according to predetermined parameters; advantageously according to this mode, part of the parameters of the neural network is accessible to the decoder, for example they can be stored in a storage memory. In particular, a part can be stored in a quantized form and encoded in a compact form using any quantizer and entropy coder accessible to those skilled in the art. A format similar to that of the MPEG-7 NNR standard can be used.

[0049] - At least part of the set of parameters of the neural networks of synthesis and / or control is decoded according to reference parameters; advantageously according to this mode, certain parameters of the neural network are decoded by taking into account reference parameters which can be stored in a memory (storage memory, network, etc.) accessible from the decoder. Thus the storage space or the transmission rate on the network can be effectively reduced: certain parameters can be coded / decoded in a complementary manner (for example by updating a kernel, a layer, a bias, etc.), others not be coded / decoded at all, because they are available in the reference parameters. The reference parameters can constitute a reference network. According to embodiment variants, which can be combined with each other: • the reference parameters can constitute a set of variants of a neural network: type of convolution, attention modules, etc. Thus, only these parameters or their identification need to be coded, inserted into the flow and decoded by the decoder; • certain parameters are coded in a complementary manner to the parameters of the reference network. Thus, the decoder will simply have to decode these residual parameters and then add them, multiply them (or combine them in any other known way) with those of the reference neural network; • part of the parameters of the reference network is reused for the target neural network. For example, a complete layer of the synthesis network can be copied from the reference network, the parameters of this layer are therefore neither encoded, nor transmitted, nor decoded; • a reference network indicator is transmitted in the stream. It is thus possible to indicate a network to be used during decoding from among a plurality of possible networks known to the decoder.

[0050] - Said at least one synthetic and / or control neural network is selected from among a plurality of neural networks, and the bitstream includes an indicator indicating selection of said at least one network from the plurality.

[0051] Correlatively, the invention also relates to a coding device and a decoding device.

[0052] The characteristics and advantages of the coding or decoding method apply in the same way to the coding or decoding device according to the invention and vice versa.

[0053] The invention also relates to a computer program on a recording medium, this program being capable of being implemented in a computer or a coding or decoding device in accordance with the invention. This program comprises instructions adapted to the implementation of the corresponding method. This program can use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0054] The invention also relates to an information carrier or a recording medium readable by a computer, and comprising computer program instructions mentioned above. The information or recording carriers can be any entity or device capable of storing the programs. For example, the carriers can comprise a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a floppy disk or a hard disk, a DNA sequence, or a flash memory. Furthermore, the information or recording carriers can be transmissible media such as an electrical or optical signal, which can be conveyed via an electrical or optical cable, by radio link, by wireless optical link or by other means.

[0055] The program according to the invention can in particular be downloaded from an Internet-type network.

[0056] Alternatively, each information or recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of a method according to the invention. Brief description of the drawings

[0057] Other characteristics and advantages of the present invention will emerge from the description given below, with reference to the appended drawings which illustrate exemplary embodiments thereof which are not in any limiting nature.

[0058] [Fig-1] [Fig.l] schematically represents a decoding device used in the scope of the invention;

[0059] [Fig.2] [Fig.2] schematically represents a coding device used in the context of the invention;

[0060] [Fig.3] [Fig.3] illustrates an architecture of control and synthesis networks used for coding and decoding according to an embodiment of the invention;

[0061] [Fig.4] [Fig.4] illustrates an architecture of control and synthesis networks used for coding and decoding according to another embodiment of the invention;

[0062] [Fig.5] [Fig.5] illustrates an example of a synthesis network used for coding and decoding according to an embodiment of the invention;

[0063] [Fig.6] [Fig.6] illustrates an example of correction applied by the control network on the synthesis characteristic maps;

[0064] [Fig.7] [Fig.7] illustrates another example of correction applied by the control network on the synthetic feature maps.

[0065] [Fig.8] [Fig.8] is a flowchart representing an example of a decoding method which can be implemented by the decoding device of [Fig.l]. Description of the embodiments

[0066] [Fig.l] schematically represents a decoding device used in the context of the invention.

[0067] The decoding device DEC of [Fig.l] decodes the coded information of a binary stream BS to generate a signal Sd, which can be for example an image of a sequence of digital images.

[0068] The signal comprises a plurality of samples.

[0069] The decoder DEC receives as input the data necessary to decode the signal Sd:

[0070] - encoded data FMc corresponding to one or more characteristic maps encoded, for example a plurality of multi-resolution feature maps.

[0071] - the coded parameters WcCon and WcSyn of at least one neural network of control, respectively synthesis. Other neural networks can be used, such as neural networks:

[0072] - predictive decoding of latent feature maps;

[0073] - oversampling and / or processing of latent feature maps;

[0074] - post-processing of the signal resulting from the synthesis;

[0075] - etc.

[0076] The coded information is extracted from the BS stream which can be received over a communication network, obtained from an accessible memory of the decoder, etc.

[0077] The DEC decoding module comprises a FMD module for decoding feature maps, a NND module for decoding neural networks.

[0078] The FMD decoding module decodes encoded feature maps FMc. According to one embodiment, it is a conventional decoder, for example entropic, of the JPEG, or MPEG, etc. type, which produces as output a set of at least one decoded characteristic map noted FMOR. According to one embodiment, the FMD module may comprise for this purpose a neural network for synthesizing characteristic maps. According to one embodiment, the maps decoded by the FMD module, numbering NFC, are noted FMOR and indexed by i.

[0079] The neural network decoding module NND decodes the encoded parameters of the control and synthesis neural networks, noted respectively after decoding WdCon and WdSyn.

[0080] The NNSYN module, comprising the synthesis neural network, uses the original FMOR maps as input and produces the decoded signal Sd as output (for example, in the form of Y,U,V components). The NNSYN module will be detailed in the remainder of the description. It comprises one or more synthesis layers Si, in cascade. It may comprise an MLP. It may also comprise convolution layers, including convolutional elements, comprising a residual structure or not and an attention module or not. It may also comprise oversampling, undersampling, etc. layers. At least one of the synthesis feature maps (latent input, output or intermediate maps of the synthesis process) is corrected, at the input or output of a layer, by a function parameterized by control parameters (PAR).

[0081] The NNCON module, comprising the control neural network, uses as input an output from a layer of the synthesis network, for example so-called generator feature maps, denoted FMGEN, and produces as output a set of parameters denoted PAR. NNCON can be in the form of a neural network and include weighting and non-linear elements. It can include an MLP. It can also include convolution layers, including convolutional elements, including a residual structure or not and an attention module or not.

[0082] The decoding device DEC can be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the modules mentioned above can then be realized by the cooperation of the processor and computer program instructions stored in the aforementioned memory and designed to carry out the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0083] [Fig.2] schematically represents a coding device used in the context of the invention.

[0084] The coding device ENC of [Fig.2] codes the signal S, for example one or more images of a video sequence.

[0085] The coder ENC receives the signal S as input and produces coded parameters as output. These parameters comprise encoded data FMc corresponding to one or more encoded feature maps, for example a plurality of multiresolution maps, and the coded parameters WcCon and WcSyn of at least one control neural network, respectively of synthesis, called network parameters of control and synthesis. As mentioned when describing [Fig.l], other neural networks can be used.

[0086] The ENC coding module comprises, for a current image, an NNC module for coding neural network parameters, an FMC module for coding characteristic maps, an RD-OPT evaluation module, an INIT / MAJ initialization and update module.

[0087] Furthermore, in a conventional manner, the coding module comprises a decoder similar to the DEC decoder which was previously described, in order to reconstruct a coded then decoded signal S'. This decoder is denoted DEC'.

[0088] During the process of training, or building, the coding, that is to say as long as the step of evaluating a performance is not satisfactory, the coding modules carry out a coding simulation, followed by a decoding, intended for the RD-OPT evaluation module. Subsequently, they carry out the actual coding of the data. In a known manner, the coding simulation can be identical to the actual coding, or carry out an approximation thereof.

[0089] The INIT / MAJ module is responsible for initializing and updating the values ​​of the images / characteristic maps and the parameters of the neural networks. It updates the values ​​to be encoded in the signal, based on the results of a performance function. Once the values ​​of the characteristic maps and the neural networks are stabilized, the maps and networks can be encoded. The INIT / MAJ module can perform an update on a subset of the images / characteristic maps and the parameters of the neural networks. The non-updated elements may, if necessary, retain values ​​known to the decoder.

[0090] The RD-OPT module performs an evaluation and minimization of a coding performance. The evaluation function is for example of the rate-distortion type. The distortion can be evaluated between the original signal S and the signal S' resulting from a coding simulation then decoding, the rate can comprise the accumulation of the rates linked to the contributions of WcCon, WcSyn and FMc, or only a sub-part. The minimization can be carried out by a gradient descent and updating of the parameters of the neural network(s), or any other optimization method within the reach of a person skilled in the art.

[0091] The ENC coding device can be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the above-mentioned modules can then be realized by the cooperation of the processor and computer program instructions stored in the above-mentioned memory and designed to carry out the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0092] [Fig.3] illustrates an architecture of control and synthesis neural networks used for coding and decoding according to one embodiment of the invention.

[0093] According to the example of [Fig.3], the synthesis network comprises a set of synthesis layers, or sub-networks, denoted SI, S2,... SN. It is generated from the decoded WdSyn parameters. According to one embodiment, part of the WdSyn parameters is obtained from known reference parameters of the decoder. The first synthesis layer (SI) takes as input the decoded so-called original feature maps (FMOR).

[0094] Each synthesis layer can perform a simple concatenation of the inputs or a more complex combination including for example a convolution or a combination by neural network on the concatenated inputs.

[0095] The modules Si (SI, S2,... SN) can include MLPs (MultiLayer Perceptrons), convolution elements, and non-linearity elements (RELU, etc.) without limiting aspect.

[0096] A convolution element may be of residual type or not, comprising kernels of any size. It is followed by a bias (possibly zero) as is well known to those skilled in the art. Any activation function within the reach of those skilled in the art may be used (ReLU, sigmoid, Softmax, Softplus, Softsign, etc.) When operating on 2D signals, such as latent value feature maps, the convolution elements may be composed of 2D convolutions, i.e. the convolution kernel is applied jointly to the rows and columns of the image or latent value map. The convolution elements may be composed of 1D convolutions, in a separable form.In this case, as is known, rather than performing the convolution on the image or the two-dimensional latent value map, a first one-dimensional (1D) convolution operator is performed on the rows, then a second convolution on the columns of the representation obtained after the first operator. The row-column processing can be reversed to perform processing by column and then by row.

[0097] One of the synthesis layers (in the example, the first SI layer) generates at output at least one layer of characteristics, noted FMGEN, called generator, which is applied at the input of the NNCON control network.

[0098] According to the example of [Fig.3], the NNCON control network comprises a set of layers, or sub-networks, denoted Cl, C2,... CN. It is generated from the decoded WdCon parameters. According to one embodiment, part of the WdCon parameters is obtained from reference parameters known to the decoder. It takes as input at least one generating feature layer produced by the synthesis network, denoted FMGEN. It produces as output a set of parameters, denoted PAR.

[0099] Each layer Ci can perform a simple concatenation of the inputs or a more complex combination including for example a convolution or a combination by neural network on the concatenated inputs. The modules Ci (Cl, C2.. .CN) can include MLP (MultiLayer Perceptron), convolution elements, and non-linearity elements (RELU, etc.) without limiting aspect.

[0100] PAR parameters are control parameters of the control functions. They may take the form of one-, two-, three-, or more-dimensional data, for example, one or more control feature maps in the latent domain. In one example, a control feature map may include gains to be applied by a function to the synthesis feature maps. In another example, it may include biases to be applied to the synthesis feature maps. In yet another example, it may include convolution kernels, nonlinearity parameters, etc.

[0101] The control functions Fi (Fb F2...FN) parameterized by the control parameters are used to correct the synthesis characteristic maps, at the input or output of the different layers of the synthesis network. They are parameterized by the PAR parameters. The functions Fi can perform any operation within the reach of a person skilled in the art, depending on the nature of the data handled: multiplications, thresholds, additions, non-linearities, linear combinations, etc. Any known mathematical function or combination or composition of such functions can be used. For example, the function may include: - an affine function, of type f(x)=a. FM(x)+b, where a and b are the parameters of the function and FM(x) expresses a value of the feature map being corrected at the x coordinate, for example in the case where the map is represented in the form of a one-dimensional vector; - a polynomial function, for example of order 2, of type f(x) = a. FM(x)2 +b.FM(x)+c, where a,b,c are the parameters of the function; - any known mathematical function or combination or composition of such functions, for example, f(x)=a.cos(b.FM(x)+c)+d.

[0102] The function can naturally be a function of several variables, for example: - a function of type f(x,y) = a.FM(x,y) + b.FM(x,y) + c.sqrt(xy) where sqrt() is the square root and x and y are the coordinates of a value of the FM feature map; - a filtering function allowing elementary filtering to be carried out on a FM characteristic map with integer coordinates indexed x,y, of type f(x, y) = f(x,y) = a.FM(xl,y) + b.FM(x,y) + c.FM(x+l,y).

[0103] The function is applied to the values ​​of at least one synthesis feature map of a synthesis layer that is located at the same level or after (in the output direction) the generating layer. According to the example shown, a function is applied to the output of the last layer (SN, function FN) and to the output of the first layer SI (dotted arrow, function Fl). For example, the parameters (a, b, c, etc.) are specific to each value of the map. According to another example, the same parameters are applied to several values ​​of the map.

[0104] According to embodiments:

[0105] All Fi functions are distinct and use their own parameters;

[0106] At least two functions Fi are identical;

[0107] At least two functions Fi, distinct or not, share a set of parameters of the control parameter set.

[0108] [Fig.4] illustrates an architecture of control and synthesis neural networks used for coding and decoding according to another embodiment of the invention.

[0109] Unlike the mode proposed in support of [Fig.3], the mode illustrated here takes as generator layer the last SN layer of the synthesis neural network. The SN layer generates at output at least one layer of characteristics, noted FMGEN, which is applied at the input of the NNCON control network. The PAR parameters generated by the NNCON network apply only to the output of this last layer, via a parameterized function FN which is used to correct the final reconstructed signal.

[0110] [Fig.5] illustrates an example of a synthetic neural network used for coding and decoding according to one embodiment of the invention.

[0111] In [Fig.5], a single layer Sk of the synthesis network is shown. It takes as input feature maps denoted FMk_IN (subset of the FMSYN synthesis feature maps) and produces as output feature maps FMk_OUT. For example, layer Sk is a convolution layer, an upsampling layer, etc.

[0112] The second layer denoted Fk, whose input corresponds to the output FMk_OUT of Sk, is a control layer corresponding to the application of the function Fk to correct the synthesis network. The function Fk is parameterized by PAR parameters, generated by the control network, corresponding here to a set of control characteristic maps (in the latent domain). The function Fk takes at least one of the control characteristic maps as a parameter, and performs an operation (function) on the set or a subset of synthesis characteristic maps FMk_OUT of the layer Sk, to generate as output a set of synthesis characteristic maps FMk+1_IN which can be applied as input to the layer Sk+1.

[0113] Thus, step by step, the synthesis network generates synthesis characteristic maps which represent the signal in the latent domain more and more finely.

[0114] The example in [Fig.5] provides control charts and synthesis charts at the same resolution.

[0115] Naturally, the synthesis maps can be of different resolutions. For example, an FMk_OUT map can include a larger number of values ​​than its counterpart in FMk_IN. In this case, Sk performs oversampling, including for example a usual interpolation function, bicubic, bilinear or any other filtering; according to another example, the oversampling can be carried out using neural networks comprising sampling, pixel multiplication (per pixel "shuffling") or convolution with bias, stride (shifting of the inverse convolution kernel at each iteration), dilation (spacing of the input values ​​supplied to the kernel), etc. Furthermore, the synthesis maps of the same layer can be of different resolutions (for example, they can constitute a pyramid of multiresolution maps).

[0116] Similarly, as will be illustrated in [Fig.8], the control maps may be of different resolution from the synthesis maps. Furthermore, the control maps of the same layer may be of different resolution from each other (for example, they may constitute a pyramid of multiresolution maps).

[0117] The numbers of summary and control cards may also be identical (a control card may be used to correct a summary card) or different (a control card may be used to correct several summary cards, or several control cards may be used to correct a summary card).

[0118] [Fig.6] illustrates an example of correction applied by the control network on the characteristic maps of the synthesis network.

[0119] According to the embodiment presented, the control characteristic maps (PAR) and synthesis (FMk) are 4 in number and are all at the same resolution. The control network generates as many PAR control characteristic maps (denoted PARi... PAR4) as there are FMk synthesis characteristic maps (denoted FMki... FMk4) at the output or input of one of the synthesis layers Sk. Each control characteristic map number i is at the same resolution as the corresponding synthesis map and serves as a parameter to a function Fki. A control characteristic map is used to correct a synthesis map to provide a corrected map FMkC (denoted FMkCi... FMkC4). Naturally, the respective maps may be of different number and resolution. The Fk functions thus parameterized may perform identical or different operations, of any type, for example, and in a non-limiting manner: - application of a mathematical function parameterized by the values ​​of the control chart, or combination or composition of such mathematical functions; Examples of functions have been proposed before; for example, a simple function is to multiply each value of a synthesis map by a value of the corresponding control map; another simple function is to add to each value of a synthesis map a value of the control map; according to another example, each function performs a masking of the synthesis feature map to which it applies, using the corresponding control map. For example, the control feature map includes values ​​corresponding to mask values ​​(e.g. an attention mask whose integer values ​​are between 0 and 255, or the real values ​​between 0 and 1, or a mask containing only the values ​​0 or 1, etc.) A value v of coordinates (x,y) of the synthesis feature map number i FMk; is multiplied by the value of the mask at the coordinates x and y of the corresponding control feature map to provide the corrected value. .

[0120] - convolution of the values ​​of a synthesis map by a convolution kernel from of the control map; in a very simple example, a filtering of the values ​​of the synthesis map is performed using one or more convolution kernels defined by the parameters or control maps. For example, a single kernel of size 3x3 is used. The 3x3 kernel is applied to the different positions of the synthesis map by convolution to obtain the corrected output map. Any complementary operation within the reach of the person skilled in the art can be added to the convolution: bias, stride (shift of the inverse convolution kernel at each iteration), dilation (spacing of the input values ​​provided to the kernel), etc. Similarly, any activation function within the reach of the person skilled in the art can be used (ReLU, sigmoid, Softmax, Softplus, Softsign, etc.) - etc.

[0121] More generally, the values ​​v of the characteristic map number i FMki produce a value v' of the map FMkC; by applying a function Fk; (x,y). For example, if the function is of affine type, the characteristic map FMk i is corrected by a function of type Fk;(x,y) =a;(x,y)* FMk;(x,y) + b;(x,y) where a;and b i are the parameters associated with the control characteristic map PAR; of index i, at the coordinates x and y. A similar reasoning applies for the functions mentioned previously.

[0122] According to the same formalism, the correction function can be expressed in the form: v' = Fki(x,y) =ai*FMki(xl,y) + b; * FMk; (x,y) + c; * FMk; (x+l,y) in order to carry out horizontal elementary filtering using parameters (ai5 b,, cO common to the whole of the characteristic map (in this case the control parameters are made up of this triplet), or variables in x and y (in this case the control parameters consist of three maps comprising the parameters ai5 b; and c; respectively, or a single map comprising a triplet of values ​​at each coordinate x,y).

[0123] [Fig.7] illustrates another example of correction applied by the control network on the characteristic maps of the synthesis network.

[0124] According to the embodiment presented, there are 4 control characteristic maps (PAR) and synthesis (FMk). As previously described in support of [Fig. 5], the control network generates as many PAR control characteristic maps (denoted PARi... PAR4) as there are FMk synthesis characteristic maps (denoted FMki... FMk4) at the output or input of one of the synthesis layers Sk. Each control characteristic map number i serves as a parameter to a function Fki. A control characteristic map is used to correct a synthesis map of a correction synthesis layer to provide a corrected map FMkC (denoted FMkCi... FMkC4). The Fk functions thus parameterized can perform identical or different operations, of any type, as previously mentioned in support of [Fig. 6].

[0125] On the other hand, contrary to what has been explained in support of [Fig.6], the control characteristic maps are at a lower resolution than that of the synthetic characteristic maps. For example, the resolution is lower by a factor of 4 in each dimension.

[0126] A value v of coordinates (x,y) of the characteristic map number i FMk; produces a value v' of the map FMkC; by applying a function Fk;. For example, if the function is of affine type, a value v of coordinates (x,y) of the characteristic map FMk; undergoes a function of type v' = Fk;(v) =a;(x / 4,y / 4)FMki (x,y)+ bj(x / 4,y / 4), where ^and b^ are the parameters associated with the control characteristic map PAR; of index i, at coordinates x / 4 and y / 4 (in integer division).

[0127] In other words, this amounts to dividing the feature map to be corrected into blocks of predetermined size (here, 16 values), and using a function parameterized by a single set of control parameters per block.

[0128] [Fig.8] is a flowchart representing an example of a decoding method that can be implemented by the decoding device of [Fig.l].

[0129] The coded BS data stream is obtained during step E30. It can be received from a communication network, or read from a storage medium. In addition, certain information (for example, relating to a reference control or synthesis network) can be read from accessible memories of the decoder. The data obtained are FMc encoded data corresponding to one or more encoded characteristic maps, for example a plurality of multiresolution maps, and WcCon coded parameters of at least one control neural network and WcSyn of at least one synthesis neural network.

[0130] During a step E31, the NFC cards of original characteristics FMOR are decoded by the FMD module. According to one embodiment, they are decoded by a conventional technique within the reach of those skilled in the art, for example a prediction followed by decoding by a contextual entropy coder then dequantization. According to another mode, they come from a standard decoder (JPEG, MPEG, etc.). According to yet another mode, they are decoded (synthesized) by a decoding neural network.

[0131] During a step E32, the control parameters WcCon of a control neural network and the synthesis parameters WcSyn of a synthesis neural network are decoded by the NNC module. For this purpose, any known neural network decoding technique corresponding to the coding technique used by the coder can be used, for example the neural network coding standard proposed by the MPEG-7 part 17, NNR, standard.

[0132] According to one embodiment, the parameters of a neural network are received in the bit stream by the decoder. According to one embodiment, certain parameters of the neural network are accessed in an accessible memory of the decoder. According to one embodiment, certain reference parameters of the neural network are accessed in an accessible memory of the decoder. According to one embodiment, an indicator received in the stream makes it possible to select one or more synthesis and / or correction networks from among a plurality of accessible networks of the decoder.

[0133] During a step E33, a layer of the decoded neural network NNSYN is applied to the synthesis feature maps (corrected or not by a parameterized control function) to generate, according to one embodiment, output feature maps. The first synthesis layer is applied directly to the original FMOR maps, that is to say that the first iteration of this step E33 consists of applying the function corresponding to the SI layer to the original FMOR feature maps. As long as the generative layer is not reached, this step is repeated. When the generative layer is reached, the output of this layer, corresponding to the generative feature maps, is applied, during a step E34, to the inputs of the control neural network NNCON.

[0134] During a step E34, the decoded control neural network, NNCON, generates the control PAR parameters. According to one embodiment, the PAR parameters take the form of a set of control characteristic maps.

[0135] During a step E35, at least part of the PAR parameters is applied to an input or output of a synthesis layer Sk of the synthesis network. According to one embodiment, the PAR parameters take the form of a set of maps of control characteristics (PAR) each used to parameterize a function Fk applied to the input or output of the layer, for example on one of the input or output FMSYN synthesis characteristic maps.

[0136] Step E35 of applying a parameterized function is followed by one or more iterations of step E33 when layers of the synthesis network remain to be applied. Otherwise, the method stops. When the last SN layer of the synthesis network has been applied, the method stops.

[0137] The decoded signal is thus generated from the characteristic maps representative of the signal. The number of iterations is defined by the structure of the network.

[0138] The decoded signal Sd corresponds to the output of the last synthesis layer, corrected or not by a parameterized function.

Claims

Claims

1. Method for coding a signal (S) of a plurality of samples, comprising the following steps: - generation of at least one original characteristic map (FMOR); - generation of at least one synthesis neural network (NNSYN), defined by a set of synthesis network parameters (WoSyn) comprising at least two synthesis neural layers (S1, SN), and a control neural network (NNCON) defined by a set of control network parameters (WoCon); - processing of said at least one original characteristic map by said synthesis neural network to produce a representation of the decoded signal (S1) at output, said processing comprising at least one step of producing at least one characteristic map, called generator map, at the output of one of said synthesis neural layers (S1, SN), called generator layer;- applying said control neural network to said at least one generator map to obtain a set of control parameters (PAR); - applying at least one function (Fl.. .FN) parameterized by at least part of said set of control parameters (PAR) to the input or output of one of said synthesis neural layers (S1,SN), called corrected layer; - updating at least part of said at least one original feature map and / or said synthesis and / or control neural network parameters, as a function of a coding performance measurement; - coding at least part of said at least one original feature map and / or said synthesis and / or control neural network parameters, as a function of a coding performance measurement.;

2. Method for decoding a signal (Sd) from a plurality of samples, comprising the following steps: - obtaining (E31) at least one characteristic map (FMOR), called original map, representative of the signal (Sd) in the latent domain;

3.

4.

5.

6.

7.

8. - decoding (E32) of the parameters (WcSyn) of a first neural network (NNSYN), called a synthetic neural network, comprising at least two neural synthesis layers (Sl,SN); - decoding (E32) of the parameters (WcCon) of a second neural network (NNCON), called the control neural network; - processing (E33) of said at least one original characteristic map by said synthesis neural network to produce the decoded signal (Sd) at output, said processing comprising at least one step of producing at least one characteristic map called generator map, at the output of one of said synthesis neural layers (Sl,SN), called generator layer; - application (E34) of said control neural network on said at least one generator card to obtain a set of control parameters (PAR); - application (E35) of at least one function (F1...FN) parameterized by at least part of said set of control parameters (PAR) to the input or output of one of said synthesis neural layers (S1,SN), called the corrected layer. Decoding method according to claim 2, characterized in that said generating layer is equal to said corrected layer. Decoding method according to claim 2, characterized in that said generating layer is located before said corrected layer in the synthesis neural network. Decoding method according to claim 2, characterized in that said generating layer is the first layer of the synthetic neural network. Decoding method according to claim 2, characterized in that said generating layer is the last layer of the synthesis neural network. Decoding method according to claim 2, comprising a step of decoding said at least one original characteristic map. Decoding method according to claim 2, for which said at least one synthesis neural layer (SI,..., SN) is applied to at least one input characteristic map (FMk_IN) and produces at least one output characteristic map (FMk_OUT), said input and output maps being called synthesis characteristic maps (FMSYN).

9. Decoding method according to claim 8, characterized in that the same parameterized function is applied to at least two synthesis characteristic maps (FMSYN).

10. Decoding method according to claim 8, characterized in that the same control parameters are used to parameterize at least two distinct functions applied respectively to at least two synthesis characteristic maps.

11. Decoding method according to one of claims 2 to 10, for which said set of control parameters (PAR) comprises at least one characteristics map, called control characteristics map.

12. Device for coding a signal (S) of a plurality of samples, said device being configured to implement the following steps: - generation (INIT / MAJ) of at least one original characteristic map (FMOR); - generation of at least one synthesis neural network (NNSYN), defined by a set of synthesis network parameters (WoSyn) comprising at least two synthesis neural layers (S1,SN), and of a control neural network (NNCON) defined by a set of control neural network parameters (WoCon); - processing of said at least one original characteristic map by said synthesis neural network to produce a representation of the decoded signal (S1) at output, said processing comprising at least one step of producing at least one characteristic map, called generator map, at the output of one of said synthesis neural layers (S1,SN), called generator layer.- applying said control neural network to said at least one generator map to obtain a set of control parameters (PAR); - applying at least one function (Fl.. .FN) parameterized by at least part of said set of control parameters (PAR) to the input or output of one of said neural layers (S1,SN) of said synthesis neural network, called corrected layer; - updating (MAJ) of at least part of at least one original characteristic map and / or of said network parameters. synthesis and / or control neurons, depending on a coding performance measure; - coding (FMC, NNC) of at least part of at least one original feature map and / or of said synthesis and / or control neural network parameters, depending on a coding performance measure.

13. Device for decoding a signal (Sd) from a plurality of samples, said device being configured to implement the following steps: - obtaining (FMD) at least one characteristic map (FMOR), called original map, representative of the signal (Sd) in the latent domain; - decoding (NND) the parameters (WcSyn) of a first neural network (NNSYN), called synthesis neural network, comprising at least two synthesis neural layers (S1,SN); - decoding (NND) the parameters (WcCon) of a second neural network (NNCON), called control neural network; - processing (NNSYN) of said at least one original characteristic map by said synthesis neural network to produce the decoded signal (Sd) at output, said processing comprising at least one step of producing at least one characteristic map, called generator map, at the output of one of said synthesis neural layers (S1,SN), called generator layer.- application (NNCON) of said control neural network on said at least one generator map to obtain a set of control parameters (PAR); - application of at least one function (Fl.. .FN) parameterized by at least part of said set of control parameters (PAR) to the input or output of one of said synthesis neural layers (S1,SN), called corrected layer.

14. A computer program comprising instructions for carrying out the steps of an encoding or decoding method according to claim 1 or 2 when said program is executed by a computer.

Images