Method and device for coding and decoding images with correction mechanism.

Abstract

Method and device for coding and decoding images with correction mechanism The invention relates to a method for coding and decoding a signal of a plurality of samples.The decoding method comprises the following steps: – obtaining at least one feature map, called the original map, representative of the signal in the latent domain; – decoding the parameters of a first neural network, called the synthesis network, comprising at least one synthesis neural layer; – decoding the parameters of a second neural network, called the control network; – applying said control neural network to said at least one original feature map to obtain a set of control parameters; – processing said at least one original feature map by said synthesis neural network to produce the decoded signal as output, said processing comprising at least one step of applying at least one function parameterized by at least part of said set of parameters to at least one synthesis neural layer. Figure for the abstract: Fig. 1.

Description

Title of the invention: Method and device for coding and decoding images with correction mechanism. Prior art

[0001] The invention relates to the general field of digital image coding. It relates more particularly to the compression of digital videos.

[0002] Digital videos are generally subject to source coding aimed at compressing them in order to limit the resources required for their transmission and / or storage. There are many coding standards, such as the ITU / MPEG standards (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) as well as their extensions (MVC, SVC, 3D-HEVC, etc.). In these approaches, the encoding of an image is generally carried out by predicting the pixels using previously coded and then decoded pixels present in the image being encoded, in which case we speak of “Intra prediction”, or previously coded images, in which case we speak of “Inter prediction”.

[0003] In addition to these classic approaches, approaches based on artificial intelligence, and in particular neural ones, tend to develop.

[0004] Some approaches use the concept of "auto-encoder", learning algorithms based on artificial neural networks, which allow to construct a new representation of a data set. The architecture of an auto-encoder is made up of two parts: the encoder and the decoder. The encoder is made up of a set of layers of neurons, which process the data in order to construct new representations called "encoded", also called "latent representations". In turn, the layers of neurons of the decoder receive these representations and filter them in order to try to reconstruct the initial data. The differences between the reconstructed data and the initial data make it possible to measure the error made by the auto-encoder. Training consists of modifying the parameters of the auto-encoder in order to reduce the reconstruction error measured on the different samples of the data set.The performance of such autoencoder-based systems comes at the cost of a considerable increase in memory footprint and complexity compared to conventional approaches as proposed by compression standards. They can have millions of parameters and may require up to a million MACs (multiply-accumulate) to decode a single pixel. This makes such decoders significantly more complex than conventional decoders, which could hamper the adoption of learning-based compression.

[0005] To improve signal processing by neural networks, so-called attention mechanisms have been proposed. An attention mechanism allows to focus the processing (attention) on certain parts of the signal. For example, in the article "Deep Attentional Features for Prostate Segmentation in Ultrasound" by Y. Wang et al. (International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 523-530, 2018), it is proposed to modify the latent representations of the input image by an attention network. Such mechanisms are very little used in the field of signal compression, because they are not very effective in this context.

[0006] In patent application WO2023199172A, it is proposed to improve neural networks used in a video encoder. According to one example, these neural networks compose autoencoders. According to other examples, they complement or replace an element of the video encoder, for example a loop filter. The improvement takes the form of improvement parameters, for example parameters of the aforementioned filtering neural network, which are evaluated by an estimator (LST-SQ) during the training of the encoder, and can be transmitted to the decoder in the form of an update of parameters of the neural network. However, such a method remains very expensive.

[0007] There is therefore a need for a solution to efficiently correct a neural network in a neural decoder. Statement of the invention

[0008] The invention relates to a method for coding a signal of a plurality of samples, comprising the following steps: - generation of a set of original feature maps; - generation of at least one synthetic neural network comprising at least one neural layer, defined by a set of synthetic network parameters and a control neural network defined by a set of control network parameters; - applying said control neural network to said at least one original feature map to obtain a set of control parameters; - processing of said at least one original feature map by said synthetic neural network to produce a representation of the decoded output signal, said processing comprising at least one step of applying a function parameterized by at least part of said set of parameters to the input or output of said at least one neural layer of said synthetic neural network; - updating at least a portion of said original feature maps and / or said synthesis and / or control neural network parameters, based on a coding performance measure; - encoding at least a portion of said original feature maps and / or said synthetic and / or control neural network parameters, based on a coding performance measure.

[0009] The invention also relates to a method for decoding a signal from a plurality of samples, comprising the following steps: - obtaining at least one characteristic map, called the original map, representative of the signal in the latent domain; - decoding the parameters of a first neural network, called a synthetic neural network, comprising at least one synthetic neural layer; - decoding the parameters of a second neural network, called the control neural network; - applying said control neural network to said at least one original feature map to obtain a set of control parameters; - processing of said at least one original feature map by said synthesis neural network to produce the decoded output signal, said processing comprising at least one step of applying at least one function parameterized by at least part of said set of control parameters to the input or output of said at least one synthesis neural layer.

[0010] For the purposes of the invention, encoding, or “coding”, means the operation which consists of representing a set of samples in a compact form carried for example by a digital binary train. Decoding means the operation which consists of processing a digital binary train to restore decoded samples.

[0011] By "sample" of the signal is meant a value taken from the signal. Sampling the signal produces a series of discrete values ​​called samples. In the case of an image signal, the sample is called a pixel which can be, for example, a color pixel traditionally represented by a triplet of values, for example (R,G,B) or (Y,U,V). Generally speaking, a sample is represented by a vector of one or more components, or data.

[0012] By "signal comprising a plurality of samples" is meant a signal with one (audio, sound), two (image) or more than two (stereoscopic, multiscopic image, image associated with a depth map, video, etc.) dimensions. Depending on this dimensionality, the sample has one, two or more coordinates in the signal. In the case of an image signal, the position of the sample is identified by its abscissa (x) and ordinate (y) coordinates.

[0013] By "feature map" is meant an abstract representation of the signal comprising a plurality of variable scalar data, which are also called characteristic values, for example real or integer numbers. In a known manner, these maps are also referred to as "latent representation" or "latent domain representation" of the signal. These feature maps can have a two-dimensional representation as images of scalar values, one-dimensional as vectors, or correspond to a set of scalar values.

[0014] By "synthesis feature maps" is meant any feature map manipulated by the synthesis network, including the input and output feature maps of the synthesis network, as well as the intermediate feature maps, manipulated by the layers of the synthesis network.

[0015] By "neural network" is meant a neural network such as a convolutional neural network, a multi-layer perceptron, etc. The neural network is defined for example by a plurality of layers of artificial neurons and by a set of activation, weighting and addition functions (for example, a layer can calculate y = f(Ax+b), where y and b are vectors of dimension N, x a vector of dimension M, A is a matrix of dimension MxN, and f is the activation function). A plurality of such networks can be cascaded.

[0016] By "parameter of the neural network" is meant one of the values ​​which characterizes the neural network, for example a weight associated with one of the neurons (filter or convolution coefficient, weighting, bias, value affecting the operation of the non-linearity, etc.)

[0017] By “processing (of input values) by a neural network” is meant the application of a function expressed by the neural network to the input values ​​to produce output values ​​representative of the samples of the current signal to be decoded (reconstructed) (resp. encoded).

[0018] By "performance measurement" is meant a measurement between at least one value of a sample to be coded and a decoded value of said sample. The measurement can evaluate for example a distortion, or a perceptual error. It can be carried out on a sample or a plurality of samples (for example, the current samples, or the current image, etc.). The measurement can also include a measurement of the flow rate, in particular associated with the coding of the neural network and / or the coding of the characteristic maps. The measurement can be a joint measurement between the flow rate and the distortion through their weighting. As is well known in the state of the art, the value of this measurement is generally minimized until a target or minimum value, or a predefined time, is reached.

[0019] By “control parameters” is meant any information useful for the correction of one or more synthetic characteristic maps.

[0020] By "parameterized function" is meant a function defined by a set of parameters. Any known mathematical function or combination or composition of such functions may be considered. The function may be represented by a Artificial neuron, also called perceptron, classically corresponding to a set of parameters that are its weights, biases and possibly activation functions. Naturally, several functions can be applied to an input or output layer. A set of parameterized functions can therefore represent one or more layers of artificial neural networks. The function or set of functions are applied to the output or input of one or more layers of the synthesis network. For example, a complete neural network, defined by the parameters of its parameterized functions, can be applied to a synthesis layer. According to another example, a simple mathematical function can be applied to the values ​​of a synthesis layer.

[0021] Generally speaking, it is considered that the steps of a coding or decoding method should not be interpreted as being linked to a notion of temporal succession. In other words, the steps may be carried out in a different order from that indicated in the independent coding or decoding claim, or even in parallel.

[0022] The coding method according to the invention carries out during its training a construction of the coding parameters, from an input signal, for example a sequence of images, by training a synthesis neural network and a characteristic map control neural network, to obtain a faithful reconstruction of the signal. Advantageously, the coding method encodes the neural networks in the form of parameters. These can be optimized, and adapted in complexity or quality, according to the targeted complexity and a targeted ratio between bit rate and distortion. Similarly, the encoded and transmitted characteristic maps can thus be optimized and adapted.

[0023] During training, or construction, or learning, the parameters of the neural networks and the information to be coded are updated according to a performance measurement, for example of the rate-distortion type. When the training is finished, that is to say that the performance measurement obtained is satisfactory, the actual coding of the parameters of the reconstruction neural network as well as that of the characteristic maps can be carried out and the result of the coding stored or transmitted to the decoder. Advantageously, the training process therefore makes it possible to refine the parameters of the neural networks, as well as its input parameters, until an adequate representation in terms of performance is obtained, for example a desired balance between the rate generated and the distortion undergone by the input image being coded. Advantageously, the coding method according to the invention makes it possible to efficiently compress the signal.

[0024] Advantageously, the decoding method is simple since it is sufficient to obtain the neural networks and the characteristic maps to reconstruct a version decoded from the signal, for example a current image of a sequence of images being processed. Advantageously in this context, one can design a transmission system that works image by image with low latency, each current image being decoded upon receipt of the parameters of the neural networks and the associated feature maps.

[0025] Advantageously, the control parameters are generated by a control network at the input of which the latent characteristic maps representative of the signal are presented. A single set of characteristic maps is thus used for the input of the two networks (control and synthesis). In addition, the cost of coding the control parameters is low, since it is limited to that of the parameters of the control neural network, which make it possible to reconstitute the control neural network, which itself makes it possible to produce the control parameters. This makes it possible to reduce the quantity of data necessary for the transmission of the signal.

[0026] Advantageously, the control parameters parameterize a function that can be applied in whole or in part to one or more synthesis characteristic maps. They can thus be shared, that is to say that several parameters (and / or parameterized functions) can be used as input or output of several layers. This makes it possible both to reduce the quantity of data necessary for the transmission of the signal and to reduce the complexity, since it is sufficient to produce the set of parameters only once.

[0027] Advantageously, the control parameters can be applied to several outputs or inputs of a layer. They can thus be shared, that is to say that several parameters (and / or parameterized functions) can be used for several inputs or outputs of a layer, for example feature maps with several resolutions.

[0028] The control parameters make it possible to drive a modification function, or correction, of the low-complexity (latent) characteristic maps (for example, affine), which makes it possible to replace more complex processing that should have been carried out by the synthesis network.

[0029] According to embodiments of the coding and / or decoding method:

[0030] - The method comprises a step of decoding said at least one map of original features. Advantageously, the feature maps can be encoded following optimization at the encoder, and transmitted to the decoder. Any encoding / decoding tool can be used: standard encoder (HEVC, VVC, JPEG, etc.), encoding of a frequency map from a wavelet transform, DCT, entropy encoder, etc.

[0031] - At least one synthesis neural layer is applied to at least one map of input features and produces at least one feature map as output, said input and output maps being called synthesis characteristic maps; advantageously according to this mode, the synthesis network produces step by step a series of synthesis characteristic maps which can be seen as an increasingly faithful approximation of the signal in the latent domain.

[0032] - The same parameterized function is applied to at least two maps of synthesis characteristics; advantageously, according to this mode, the same parameterized function (for example, a convolution, a multiplication, a masking, etc.) can be applied either to several places in the synthesis (at the input or output of distinct layers) or to several characteristic maps of the same synthesis layer, or to both. It is thus possible to share the functions to be applied for the control of several synthesis characteristic maps.

[0033] - The same control parameters are used to set at least two functions applied respectively to at least two synthesis characteristic maps; advantageously, the control parameters can thus be pooled to correct several synthesis characteristic maps using the same function or distinct functions.

[0034] - The set of control parameters comprises at least one kernel of convolution; advantageously, one or more convolution kernels are generated by the control module, so as to perform a convolution operation on the synthesis feature maps.

[0035] - The set of control parameters comprises at least one map of characteristics, called control characteristic map; advantageously, one or more latent value characteristic maps are generated by the control neural network. These characteristic maps can be used to parameterize a simple function (for example, multiplication and / or addition) or a more complex one (such as filtering) to be applied directly to the values ​​of the synthesis characteristic maps.

[0036] - The resolution of said at least one control characteristic map is identical to that of said at least one synthesis characteristic map; advantageously according to this mode, the parameterized function can make it possible to perform a simple operation between a value of the control characteristic map and one or more values ​​located in the same neighborhood of the synthesis characteristic map (addition, multiplication, masking, convolution, etc.). In this case, there is a control parameter for each element of the synthesis characteristic map, this degree of precision making it possible to obtain maximum coding performance.

[0037] - The resolution of said at least one control characteristic map is different from that of said at least one synthesis characteristic map; advantageously according to this mode, the parameterized function can make it possible to carry out a simple operation between a value of the control feature map and several values ​​of the synthesis feature map (addition, multiplication, masking, convolution, etc.) which can be for example in the vicinity of the position of the value of the control map, after scaling. It can be a block, for example.

[0038] - The resolution of said at least one control characteristic map is lower than that of said at least one synthesis characteristic map; advantageously according to this mode, the complexity of the step of generating the control characteristic map is reduced, and the same control parameter is shared between several elements of the characteristic map.

[0039] - Said at least one function parameterized by said at least one card of control characteristics is applied to at least two separate synthesis maps.

[0040] - Said at least one function parameterized by said at least one card of control characteristics is applied to at least two synthesis maps of the same layer; advantageously, according to this mode, a function parameterized by the same control characteristics map applies to several synthesis characteristics maps at the same location in the synthesis network. For example, a control map can be applied (via the same function or several distinct functions) to apply a treatment at the same location to several synthesis characteristics maps at different resolutions. It is thus possible to share the control parameters to correct several synthesis characteristics maps.

[0041] - Said at least one control neural network comprises at least one layer convolution; thus, a filtering treatment of the image can be carried out, which makes it possible to improve the consistency of the generated image by limiting the noise and improving the contours present in the image.

[0042] - Said at least one (synthetic or) control neural network comprises at least least one attention module; "attention module" means a layer of a neural network comprising an attention element. An attention element is an element that allows an attention mechanism to be applied to generate one or more masks as parameters. These masks can parameterize a function that uses them to weight (by multiplication or use of more complex functions) the synthesis feature maps in order, for example, to adapt the allocation of the flow rate to the different regions and / or characteristics of the image according to their importance. The attention masks are encoded and decoded as parameters of the control network. Advantageously, it is not necessary to use additional bits to encode the masks.

[0043] - At least part of the set of parameters of the neural networks synthesis and / or control is obtained from data of said binary stream; Advantageously, according to this mode, the parameters of the neural networks are transmitted in the stream. The parameters can be transmitted in whole or in part in a quantized form and coded in a compact form using any quantizer and entropy coder accessible to those skilled in the art. A format similar to that of the MPEG-7 NNR standard can be used.

[0044] - At least part of the set of parameters of the neural networks of synthesis and / or control is decoded according to predetermined parameters; advantageously according to this mode, part of the parameters of the neural network are accessible to the decoder, for example they can be stored in a storage memory. In particular, part can be stored in a quantized form and coded in a compact form using any quantizer and entropy coder accessible to those skilled in the art. A format similar to that of the MPEG-7 NNR standard can be used.

[0045] - At least part of the set of parameters of the neural networks of synthesis and / or control is decoded according to reference parameters; advantageously according to this mode, certain parameters of the neural network are decoded by taking into account reference parameters which can be stored in a memory (storage memory, network, etc.) accessible from the decoder. Thus the storage space or the transmission rate on the network can be effectively reduced: certain parameters can be coded / decoded in a complementary manner (for example by updating a kernel, a layer, a bias, etc.), others not be coded / decoded at all, because they are available in the reference parameters. The reference parameters can constitute a reference network. According to embodiment variants, which can be combined with each other: • the reference parameters can constitute a set of variants of a neural network: type of convolution, attention modules, etc. Thus, only these parameters or their identification need to be coded, inserted into the flow and decoded by the decoder; • certain parameters are coded in a complementary manner to the parameters of the reference network. Thus, the decoder will simply have to decode these residual parameters and then add them, multiply them (or combine them in any other known way) with those of the reference neural network; • part of the parameters of the reference network is reused for the target neural network. For example, a complete layer of the synthesis network can be copied from the reference network, the parameters of this layer are therefore neither encoded, nor transmitted, nor decoded; • a reference network indicator is transmitted in the stream. It is thus possible to indicate a network to be used during decoding from among a plurality of possible networks known to the decoder.

[0046] - Said at least one synthetic and / or control neural network is selected from among a plurality of neural networks, and the bitstream includes an indicator indicating selection of said at least one network from the plurality.

[0047] Correlatively, the invention also relates to a coding device and a decoding device.

[0048] The characteristics and advantages of the coding or decoding method apply in the same way to the coding or decoding device according to the invention and vice versa.

[0049] The invention also relates to a computer program on a recording medium, this program being capable of being implemented in a computer or an encoding or decoding device in accordance with the invention. This program comprises instructions adapted to the implementation of the corresponding method. This program can use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0050] The invention also relates to an information carrier or a recording medium readable by a computer, and comprising computer program instructions mentioned above. The information or recording carriers can be any entity or device capable of storing the programs. For example, the carriers can comprise a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a floppy disk or a hard disk, a DNA sequence, or a flash memory. Furthermore, the information or recording carriers can be transmissible media such as an electrical or optical signal, which can be conveyed via an electrical or optical cable, by radio link, by wireless optical link or by other means.

[0051] The program according to the invention can in particular be downloaded from an Internet-type network.

[0052] Alternatively, each information or recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of a method according to the invention. Brief description of the drawings

[0053] Other characteristics and advantages of the present invention will emerge from the description given below, with reference to the appended drawings which illustrate exemplary embodiments thereof which are not in any limiting nature.

[0054] [Fig-1] [Fig.l] schematically represents a decoding device used in the scope of the invention;

[0055] [Fig.2] [Fig.2] schematically represents a coding device used in the context of the invention;

[0056] [Fig.3] [Fig.3] illustrates an architecture of control and synthesis networks used for coding and decoding according to an embodiment of the invention;

[0057] [Fig.4] [Fig.4] illustrates an example of a synthesis network used for coding and decoding according to an embodiment of the invention;

[0058] [Fig.5] [Fig.5] illustrates an example of correction applied by the control network on the synthesis characteristic maps;

[0059] [Fig.6] [Fig.6] illustrates another example of correction applied by the control network on the synthetic feature maps;

[0060] [Fig.7] [Fig.7] is a flowchart representing an example of a decoding method which can be implemented by the decoding device of [Fig.l]. Description of the embodiments

[0061] [Fig.l] schematically represents a decoding device used in the context of the invention.

[0062] The decoding device DEC of [Fig.l] decodes the coded information of a binary stream BS to generate a signal Sd, which can be for example an image of a sequence of digital images.

[0063] The signal comprises a plurality of samples.

[0064] The DEC decoder receives as input the data necessary to decode the signal Sd:

[0065] - FMc encoded data corresponding to one or more feature maps encoded, for example a plurality of multi-resolution feature maps.

[0066] - the coded parameters WcCon and WcSyn of at least one neural network of control, respectively synthesis. Other neural networks can be used, such as neural networks:

[0067] - predictive decoding of latent feature maps;

[0068] - oversampling and / or processing of latent feature maps;

[0069] - post-processing of the signal resulting from the synthesis;

[0070] - etc.

[0071] The coded information is extracted from the BS stream which can be received over a communication network, obtained from an accessible memory of the decoder, etc.

[0072] The DEC decoding module comprises a FMD module for decoding feature maps, a NND module for decoding neural networks.

[0073] The FMD decoding module decodes coded feature maps FMc. According to one embodiment, it is a conventional decoder, for example entropic, of the JPEG, or MPEG, etc. type, which produces as output a set of at least one decoded feature map denoted FMOR. According to one embodiment, the FMD module may comprise for this purpose a neural network for synthesizing feature maps. According to one embodiment, the maps decoded by the FMD module, numbering NFC, are denoted FMOR and indexed by i.

[0074] The neural network decoding module NND decodes the encoded parameters of the control and synthesis neural networks, noted respectively after decoding WdCon and WdSyn.

[0075] The NNCON module, comprising the control neural network, uses the original FMOR maps as input and produces as output a set of parameters denoted PAR. NNCON can be in the form of a neural network and include weighting and non-linear elements. It can include an MLP. It can also include convolution layers, including convolutional elements, including a residual structure or not and an attention module or not.

[0076] The NNSYN module, comprising the synthesis neural network, also uses the original FMOR maps as input and produces the decoded signal Sd as output (for example, in the form of Y,U,V components). The NNSYN module will be detailed in the remainder of the description. It comprises one or more synthesis layers Si, in cascade. It may comprise an MLP. It may also comprise convolution layers, including convolutional elements, comprising a residual structure or not and an attention module or not. It may also comprise oversampling, undersampling, etc. layers. At least one of the synthesis feature maps (latent input, output or intermediate maps of the synthesis process) is corrected, at the input or output of a layer, by a function parameterized by the PAR control parameters.

[0077] The decoding device DEC can be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the modules mentioned above can then be realized by the cooperation of the processor and computer program instructions stored in the aforementioned memory and designed to carry out the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0078] [Fig.2] schematically represents a coding device used in the context of the invention.

[0079] The coding device ENC of [Fig.2] codes the signal S, for example one or more images of a video sequence.

[0080] The coder ENC receives the signal S as input and produces coded parameters as output. These parameters comprise encoded data FMc corresponding to one or more encoded feature maps, for example a plurality of multiresolution maps, and the coded parameters WcCon and WcSyn of at least one control neural network, respectively of synthesis, called control and synthesis network parameters. As mentioned during the description of [Fig.l], other neural networks can be used.

[0081] The coding module ENC comprises, for a current image, an NNC module for coding neural network parameters, an FMC module for coding characteristic maps, an RD-OPT evaluation module, an INIT / MAJ initialization and update module.

[0082] Furthermore, in a conventional manner, the coding module comprises a decoder similar to the DEC decoder which was previously described, in order to reconstruct a coded then decoded signal S'. This decoder is denoted DEC'.

[0083] During the process of training, or building, the coding, that is to say as long as the step of evaluating a performance is not satisfactory, the coding modules carry out a coding simulation, followed by a decoding, intended for the RD-OPT evaluation module. Subsequently, they carry out the actual coding of the data. In a known manner, the coding simulation can be identical to the actual coding, or carry out an approximation thereof.

[0084] The INIT / MAJ module is responsible for initializing and updating the values ​​of the images / characteristic maps and the parameters of the neural networks. It updates the values ​​to be encoded in the signal, based on the results of a performance function. Once the values ​​of the characteristic maps and the neural networks are stabilized, the maps and networks can be encoded. The INIT / MAJ module can perform an update on a subset of the images / characteristic maps and the parameters of the neural networks. The non-updated elements may, if necessary, retain values ​​known to the decoder.

[0085] The RD-OPT module performs an evaluation and minimization of a coding performance. The evaluation function is for example of the rate-distortion type. The distortion can be evaluated between the original signal S and the signal S' resulting from a coding simulation then decoding, the rate can include the cumulative rate linked to the contributions of WcCon, WcSyn and FMc, or only a sub-part. The minimization can be carried out by a gradient descent and updating of the parameters of the neural network(s), or any other optimization method within the reach of a person skilled in the art.

[0086] The ENC coding device can be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the above-mentioned modules can then be realized by the cooperation of the processor and computer program instructions stored in the above-mentioned memory and designed to carry out the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0087] [Fig.3] illustrates an architecture of control and synthesis neural networks used for coding and decoding according to one embodiment of the invention.

[0088] According to the example of [Fig.3], the synthesis network comprises a set of synthesis layers, or sub-networks, denoted SI, S2,.. SN. It is generated from the decoded WdSyn parameters. According to one embodiment, the first synthesis layer (SI) takes as input the decoded so-called original feature maps (FMOR). According to another embodiment, the first layer SI takes as input the results of a first parameterized function applied to the original FMOR feature maps. According to one embodiment, a part of the WdSyn parameters is obtained from known reference parameters of the decoder.

[0089] Each layer can perform a simple concatenation of the inputs or a more complex combination including for example a convolution or a combination by neural network on the concatenated inputs.

[0090] The modules Si (SI, S2.. SN) can include MLP (MultiLayer Perceptron), convolution elements, and non-linearity elements (RELU, etc.) without limiting aspect.

[0091] A convolution element may be of residual type or not, comprising kernels of any size. It is followed by a bias (possibly zero) as is well known to those skilled in the art. Any activation function within the reach of those skilled in the art may be used (ReLU, sigmoid, Softmax, Softplus, Softsign, etc.). When operating on 2D signals, such as latent value feature maps, the convolution elements may be composed of 2D convolutions, i.e. the convolution kernel is applied jointly to the rows and columns of the image or latent value map. The convolution elements may be composed of 1D convolutions, in a separable form.In this case, as is known, rather than performing the convolution on the image or the two-dimensional latent value map, a first one-dimensional (1D) convolution operator is performed on the rows, then a second convolution on the columns of the representation obtained after the first operator. The row-column processing can be reversed to perform processing by column and then by row.

[0092] According to the example of [Fig.3], the NNCON control network comprises a set of layers, or sub-networks, denoted Cl, C2,.. CN. It is generated from the decoded WdCon parameters. It takes as input the decoded so-called original feature maps (FMOR). It produces as output a set of parameters, denoted PAR. According to one embodiment, part of the WdCon parameters is obtained from known reference parameters of the decoder.

[0093] Each layer Ci can perform a simple concatenation of the inputs or a more complex combination including for example a convolution or a combination by neural network on the concatenated inputs. The modules Ci (Cl, C2.. .CN) can include MLP (MultiLayer Perceptron), convolution elements, and non-linearity elements (RELU, etc.) without limiting aspect.

[0094] PAR parameters are control parameters of the control functions. They may take the form of one-, two-, three-, or more-dimensional data, for example, one or more control feature maps in the latent domain. In one example, a control feature map may include gains to be applied to the synthesis feature maps. In another example, it may include biases to be applied to the synthesis feature maps. In yet another example, it may include convolution kernels, etc.

[0095] The control functions Fi (Fb F2...FN) parameterized by the control parameters are used to correct the synthesis characteristic maps, at the input or output of the different layers of the synthesis network. They are parameterized by the PAR parameters. The functions Fi can perform any operation within the reach of a person skilled in the art, depending on the nature of the data handled: multiplications, thresholds, additions, non-linearities, linear combinations, etc. Any known mathematical function or combination or composition of such functions can be used. For example, the function may include: - an affine function, of type f(x)=a.FM(x)+b, where a and b are the parameters of the function and FM(x) expresses a value of the characteristic map at the x coordinate, for example in the case where the map is represented in the form of a one-dimensional vector; - a polynomial function, for example of order 2, of type f(x) = a.FM(x)2 +b.FM(x)+c, where a,b,c are the parameters of the function; - any known mathematical function or combination or composition of such functions, for example, f(x)=a.cos(b.FM(x)+c)+d.

[0096] The function can naturally be a function of several variables, for example: - a function of type f(x,y) = a.FM(x,y) + b.FM(x,y) + c.sqrt(xy) where sqrt() is the square root and x and y are the coordinates of a value of the FM characteristic map; - a filtering function allowing elementary filtering to be carried out on a FM characteristic map with integer coordinates indexed x,y, of type f(x,y) = a.FM(xl,y) + b.FM(x,y) + c.FM(x+l,y).

[0097] The function is applied to the values ​​of a synthetic feature map. For example, the parameters a, b, and c are specific to each value of the map. In another example, the same parameters are applied to multiple values ​​of the map.

[0098] According to embodiments:

[0099] All Fi functions are distinct and use their own parameters;

[0100] At least two functions Fi are identical;

[0101] At least two functions Fi, distinct or not, share a set of parameters of the set of control parameters.

[0102] [Fig.4] illustrates an example of a synthetic neural network used for coding and decoding according to one embodiment of the invention.

[0103] In [Fig.4], a single layer Sk of the synthesis network is shown. It takes as input feature maps denoted FMk_IN (subset of the FMSYN synthesis feature maps) and produces as output feature maps FMk_OUT. For example, layer Sk is a convolution layer, an upsampling layer, etc.

[0104] The second layer denoted Fk, whose input corresponds to the output FMk_OUT of Sk, is a control layer corresponding to the application of the function Fk to correct the synthesis network. The function Fk is parameterized by PAR parameters, generated by the control network, corresponding here to a set of control characteristic maps (in the latent domain). The function Fk takes at least one of the control characteristic maps as a parameter, and performs an operation (function) on the synthesis characteristic maps FMk_OUT of the layer Sk, to generate as output a set of synthesis characteristic maps FMk+l_IN which can be applied as input to the layer Sk+1.

[0105] Thus, step by step, the synthesis network generates synthesis characteristic maps which represent the signal in the latent domain more and more finely.

[0106] The example in [Fig.4] provides control charts and synthesis charts at the same resolution.

[0107] Naturally, the synthesis maps can be of different resolutions. For example, an FMk_OUT map can include a larger number of values ​​than its counterpart in FMk_IN. In this case, Sk performs oversampling, including for example a usual interpolation function, bicubic, bilinear or any other filtering; according to another example, oversampling can be carried out using neural networks comprising sampling, pixel multiplication (per pixel "shuffling") or convolution with bias, stride (kernel shift inverse convolution at each iteration), dilation (spacing of input values ​​provided to the kernel), etc. Furthermore, the synthesis maps of the same layer can be of different resolutions (for example, they can constitute a pyramid of multiresolution maps).

[0108] Similarly, as will be illustrated in [Fig.6], the control maps may be of different resolution from the synthesis maps. Furthermore, the control maps of the same layer may be of different resolution from each other (for example, they may constitute a pyramid of multiresolution maps).

[0109] The numbers of summary and control cards may also be identical (a control card may be used to correct a summary card) or different (a control card may be used to correct several summary cards, or several control cards may be used to correct a summary card).

[0110] [Fig.5] illustrates an example of correction applied by the control network on the characteristic maps of the synthesis network.

[0111] According to the embodiment presented, the control characteristic maps (PAR) and synthesis (FMk) are 4 in number and are all at the same resolution. The control network generates as many PAR control characteristic maps (denoted PARi... PAR4) as there are FMk synthesis characteristic maps (denoted FMki... FMk4) at the output or input of one of the synthesis layers Sk. Each control characteristic map number i is at the same resolution as the corresponding synthesis map and serves as a parameter to a function Fki. A control characteristic map is used to correct a synthesis map to provide a corrected map FMkC (denoted FMkCi... FMkC4). Naturally, the respective maps may be of different number and resolution. The Fk functions thus parameterized may perform identical or different operations, of any type, such as for example, and in a non-limiting manner: - application of a mathematical function parameterized by the values ​​of the control chart, or combination or composition of such mathematical functions; examples of functions have been proposed before; for example, a simple function consists of multiplying each value of a synthesis chart by a value of the corresponding control chart; another simple function consists of adding to each value of a synthesis chart a value of the control chart; according to another example, each function performs a masking of the synthesis feature map to which it applies, using the corresponding control chart. For example, the control feature map comprises values ​​corresponding to mask values ​​(for example an attention mask whose integer values ​​are between 0 and 255, or the real values ​​between 0 and 1, or a mask containing only values ​​0 or 1, etc.) A value v of coordinates (x,y) of the synthesis feature map number i FMk; is multiplied by the mask value at the x and y coordinates of the corresponding control feature map to provide the corrected value.

[0112] - convolution of the values ​​of a synthesis map by a convolution kernel from of the control map; in a very simple example, a filtering of the values ​​of the synthesis map is performed using one or more convolution kernels defined by the parameters or control maps. For example, a single kernel of size 3x3 is used. The 3x3 kernel is applied to the different positions of the synthesis map to obtain the corrected output map. Any complementary operation within the reach of the person skilled in the art can be added to the convolution: bias, stride (shift of the inverse convolution kernel at each iteration), dilation (spacing of the input values ​​provided to the kernel), etc. Similarly, any activation function within the reach of the person skilled in the art can be used (ReLU, sigmoid, Softmax, Softplus, Softsign, etc.) - etc.

[0113] More generally, the values ​​v of the characteristic map number i FMki produce a value v' of the map FMkC; by applying a function Fk; (x,y). For example, if the function is of affine type, the characteristic map FMk i is corrected by a function of type Fk;(x,y) =a;(x,y)* FMk;(x,y) + b;(x,y) where a;and b i are the parameters associated with the control characteristic map PAR; of index i, at the coordinates x and y. A similar reasoning applies for the functions mentioned previously.

[0114] According to the same formalism, the correction function can be expressed in the form: v' = Fki(x,y) =ai*FMki(xl,y) + b; * FMk; (x,y) + c; * FMk; (x+l,y) in order to carry out horizontal elementary filtering using parameters (ai5 b;, cO common to the whole of the characteristic map (in this case the control parameters are made up of this triplet), or variables in x and y (in this case the control parameters are made up of three maps comprising respectively the parameters a;, b; and Ci, or a single map comprising a triplet of values ​​at each coordinate x,y).

[0115] [Fig.6] illustrates another example of correction applied by the control network on the characteristic maps of the synthesis network.

[0116] According to the embodiment presented, there are 4 control characteristic maps (PAR) and synthesis characteristic maps (FMk). As described previously in support of [Fig. 5], the control network generates as many PAR control characteristic maps (denoted PARi... PAR4) as there are FMk synthesis characteristic maps (denoted FMki... FMk4) at the output or input of one of the synthesis layers Sk. Each control characteristic map number i serves as a parameter to a function Fki. A control characteristics map is used to correct a synthesis map to provide a corrected FMkC map (denoted FMkCi... FMkC4). The Fk functions thus parameterized can perform identical or different operations, of any type, as mentioned previously in support of [Fig.5].

[0117] On the other hand, contrary to what has been explained in support of [Fig.5], the control characteristic maps are at a lower resolution than that of the synthesis characteristic maps. For example, the resolution is lower by a factor of 4 in each dimension.

[0118] A value v of coordinates (x,y) of the characteristic map number i FMk; produces a value v' of the map FMkC; by applying a function Fk;. For example, if the function is of affine type, a value v of coordinates (x,y) of the characteristic map FMk; undergoes a function of type v' = Fk;(v) =a;(x / 4,y / 4)FMki (x,y)+ bj(x / 4,y / 4), where ^andb^ are the parameters associated with the control characteristic map PAR; of index i, at coordinates x / 4 and y / 4 (in integer division).

[0119] In other words, this amounts to dividing the feature map to be corrected into blocks of predetermined size (here, 16 values), and using a function parameterized by a single set of control parameters per block.

[0120] [Fig.7] is a flowchart representing an example of a decoding method that can be implemented by the decoding device of [Fig.l].

[0121] The coded BS data stream is obtained during step E30. It can be received from a communication network, or read from a storage medium. In addition, certain information (for example, relating to a reference control or synthesis network) can be read from accessible memories of the decoder. The data obtained are encoded data FMc corresponding to one or more encoded characteristic maps, for example a plurality of multiresolution maps, and coded parameters WcCon of at least one control neural network and WcSyn of at least one synthesis neural network.

[0122] During a step E31, the NFC cards of original characteristics FMOR are decoded by the FMD module. According to one embodiment, they are decoded by a conventional technique within the reach of those skilled in the art, for example a prediction followed by decoding by a contextual entropy coder then dequantization. According to another mode, they come from a standard decoder (JPEG, MPEG, etc.). According to yet another mode, they are decoded (synthesized) by a decoding neural network.

[0123] During a step E32, the control parameters WcCon of a control neural network and the synthesis parameters WcSyn of a synthesis neural network are decoded by the NNC module. Any known technique can be used for this purpose. neural network decoding corresponding to the encoding standard used by the encoder, for example the neural network coding standard proposed by the MPEG-7 part 17, NNR standard.

[0124] According to one embodiment, the parameters of a neural network are received in the bit stream by the decoder. According to one embodiment, certain parameters of the neural network are accessed in an accessible memory of the decoder. According to one embodiment, certain reference parameters of the neural network are accessed in an accessible memory of the decoder. According to one embodiment, an indicator received in the stream makes it possible to select one or more synthesis and / or correction networks from among a plurality of accessible networks of the decoder.

[0125] During a step E33, the control neural network thus decoded, NNCON, is applied to the original characteristic maps FMOR to generate the control PAR parameters. According to one embodiment, the PAR parameters take the form of a set of control characteristic maps.

[0126] During a step E34, a layer of the decoded neural network NNSYN is applied to the synthesis feature maps (corrected or not by a control function) to generate, according to one embodiment, output feature maps. According to one embodiment, a first parameterized function is applied to the original FMOR maps. According to another embodiment, the first synthesis layer is applied directly to the original FMOR maps.

[0127] During a step E35, at least part of the PAR parameters is applied to an input or output of a synthesis layer Sk of the synthesis network. According to one embodiment, the PAR parameters take the form of a set of control characteristic maps (PAR) each used to parameterize a function Fk applied to the input or output of the layer, for example on one of the input or output FMSYN synthesis characteristic maps.

[0128] Steps E34 and E35 can be reversed.

[0129] Steps E34 and E35 are repeated as many times as necessary (as the synthesis network has layers) to generate the decoded signal from the characteristic maps representative of the signal. The number of iterations is defined by the structure of the network.

[0130] The decoded signal Sd corresponds to the output of the last synthesis layer, corrected or not by a parameterized function.

Claims

Claims

1. Method for coding a signal (S) of a plurality of samples, comprising the following steps: - generation of a set of original characteristic maps (FMOR); - generation of at least one synthesis neural network (NNSYN) comprising at least one synthesis neural layer (S1,SN), defined by a set of synthesis network parameters (WoSyn) and a control neural network (NNCON) defined by a set of control network parameters (WoCon); - application of said control neural network to said at least one original characteristic map (FMOR) to obtain a set of control parameters (PAR); - processing of said at least one original characteristic map by said synthesis neural network to produce a representation of the decoded signal (S1) as output, said processing comprising at least one step of applying a function (Fl.. .FN) parameterized by at least part of said set of control parameters (PAR) at the input or output of said at least one synthesis neural layer (S1,SN); - updating at least part of said original feature maps and / or said synthesis and / or control neural network parameters, as a function of a coding performance measure; - coding at least part of said original feature maps and / or said synthesis and / or control neural network parameters, as a function of a coding performance measure.

2. Method for decoding a signal (Sd) from a plurality of samples, comprising the following steps: - obtaining (E31) at least one characteristic map (FMOR), called original map, representative of the signal (Sd) in the latent domain; - decoding (E32) the parameters (WcSyn) of a first neural network (NNSYN), called synthetic neural network, comprising at least one synthesis neural layer (S1,SN); - decoding (E32) the parameters (WcCon) of a second neural network (NNCON), called the control neural network; - applying (E33) said control neural network to said at least one original characteristic map (FMOR) to obtain a set of control parameters (PAR); - processing (E34, E35) said at least one original characteristic map by said synthesis neural network to produce the decoded signal (Sd) as output, said processing comprising at least one step of applying at least one function (Fl.. .FN) parameterized by at least part of said set of control parameters (PAR) to the input or output of said at least one synthesis neural layer (S1, SN).

3. A decoding method according to claim 2, comprising a step of decoding said at least one original feature map.

4. Decoding method according to claim 2, for which said at least one synthesis neural layer (SI,..., SN) is applied to at least one input feature map (FMk_IN) and produces at least one output feature map (FMk_out), said input and output maps being called synthesis feature maps (FMSYN).

5. Decoding method according to claim 4, characterized in that the same parameterized function is applied to at least two synthesis characteristic maps (FMSYN).

6. Decoding method according to claim 4, characterized in that the same control parameters are used to parameterize at least two distinct functions applied respectively to at least two synthesis characteristic maps.

7. Decoding method according to one of claims 2 to 6, for which said set of control parameters (PAR) comprises at least one convolution kernel.

8. Decoding method according to one of claims 2 to 7, for which said set of control parameters (PAR) comprises at least one characteristics map, called control characteristics map.

9. Decoding or coding method according to one of the preceding claims, characterized in that said at least one control neural network comprises at least one attention module.

10. Decoding or coding method according to one of the preceding claims, characterized in that said at least one control neural network comprises at least one convolution layer.

11. Device for coding a signal (S) of a plurality of samples, said device being configured to implement the following steps: - generation (INIT / MAJ) of a set of original characteristic map (FMOR); - generation of at least one synthesis neural network (NNSYN), defined by a set of synthesis network parameters (WoSyn) comprising at least one synthesis neural layer (S1,SN) and a control neural network (NNCON) defined by a set of control network parameters (WoCon); - application of said control neural network to said at least one original characteristic map (FMOR) to obtain a set of control parameters (PAR); - processing of said at least one original characteristic map by said synthesis neural network to produce a representation of the decoded signal (S1) at output, said processing comprising at least one step of applying a function (Fl.. .FN) parameterized by at least part of said set of control parameters (PAR) at the input or output of said at least one synthesis neural layer (S1, SN); - updating (MAJ) of at least part of said original feature maps and / or of said synthesis and / or control neural network parameters, as a function of a coding performance measurement; - coding (FMC, NNC) of at least part of said original feature maps and / or of said synthesis and / or control neural network parameters, as a function of a coding performance measurement.

12. Device for decoding a signal (Sd) from a plurality of samples, said device being configured to implement the following steps: - obtaining (FMD) at least one characteristic map (FMOR), called original map, representative of the signal (Sd) in the latent domain;

13. - decoding (NND) of the parameters (WcSyn) of a first neural network (NNSYN), called a synthetic neural network, comprising at least one synthetic neural layer (S1,SN); - decoding (NND) of the parameters (WcCon) of a second neural network (NNCON), called the control network; - application (NNCON) of said control neural network on said at least one original characteristic map (FM) to obtain a set of control parameters (PAR); - processing (NNSYN) of said at least one original characteristic map by said synthesis neural network to produce the decoded signal (Sd) as output, said processing comprising at least one step of applying at least one function (Fl.. .FN) parameterized by at least part of said set of control parameters (PAR) to the input or output of said at least one synthesis neural layer (S1,SN). Computer program comprising instructions for carrying out the steps of an encoding or decoding method according to claim 1 or 2 when said program is executed by a computer.

Images