Method and device for encoding and decoding images.
By constructing lower resolution feature maps and using oversampling neural networks to update parameters, the method addresses inefficiencies in existing image compression technologies, achieving efficient and low-complexity encoding and decoding.
Patent Information
- Application Number
- FR2023005152
- Authority / Receiving Office
- FR · FR
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-24
- Publication Date
- 2024-11-29
- Estimated Expiration
- Not applicable · inactive patent
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
Title of the invention: Method and device for coding and decoding images. Prior art
[0001] The invention relates to the general field of coding digital image sequences. It relates more particularly to the compression of digital videos.
[0002] Digital videos are generally subject to source coding aimed at compressing them in order to limit the resources required for their transmission and / or storage. There are many coding standards, such as the ITU / MPEG standards (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) as well as their extensions (MVC, SVC, 3D-HEVC, etc.).
[0003] As video formats continue to evolve to compress more and more and adapt to the variety of expected formats and communication networks, the possibilities for prediction are becoming ever greater and the classic coding and decoding algorithms very complex.
[0004] In addition to these classic approaches proposed by compression standards (MPEG, ITU), approaches based on artificial intelligence, and in particular neural ones, tend to develop.
[0005] Some of these neural approaches can be seen as a simple extension of the notion of competition of the aforementioned compression techniques, such as the competition of prediction and transformation mode in video coding.
[0006] Other approaches use the concept of "auto-encoder". Auto-encoders are learning algorithms based on artificial neural networks, which allow the construction of a new representation of a data set. The architecture of an auto-encoder consists of two parts: the encoder and the decoder. The encoder is made up of a set of layers of neurons, which process the data in order to construct new representations called "encoded", also called "latent representations". In turn, the layers of neurons of the decoder receive these representations and filter them in order to try to reconstruct the initial data. The differences between the reconstructed data and the initial data make it possible to measure the error made by the auto-encoder. Training consists of modifying the parameters of the auto-encoder in order to reduce the reconstruction error measured on the different samples of the data set.The performance of such autoencoder-based systems comes at the cost of a considerable increase in memory footprint and complexity compared to conventional approaches such as those proposed by compression standards. They . can have millions of parameters and may require on the order of a million MACs (multiply-accumulate) to decode a single pixel. This makes such decoders much more complex than conventional decoders, which could hamper the adoption of learning-based compression. In addition, the neural networks used at the encoder and decoder are different. US 2022 / 0385907 describes such an autoencoder system.
[0007] More recently, a simple image encoding technique based on a neural network has been described in the article "Compression with Implicit Neural representations" by Emilien Dupont et al. (arXiv:2103.03123). The proposed encoding technique consists of fitting a neural network to an image, quantizing the network weights and transmitting them. At decoding time, the neural network is evaluated at each pixel position to reconstruct the image. However, such a technique remains inefficient in terms of compression.
[0008] Document US 2022 / 0385907 also describes a coding and decoding system based on the use of such implicit neural representations.
[0009] However, there is a need for a solution that allows a sequence of images to be coded / compressed in a simple and efficient manner. Statement of the invention
[0010] The invention relates to a method for coding an image at least comprising a plurality of samples to be coded, comprising the following steps: - a construction stage, comprising the sub-stages of: - construction of a group of feature maps comprising at least one feature map of resolution lower than the resolution of the input image, called lower resolution map; - upsampling said at least one lower resolution feature map using an upsampling artificial neural network at least defined by a set of upsampling parameters to obtain an upsampled map at the resolution of said at least one image; - generation of said at least one image from said at least one oversampled map; - updating at least one value of one of said feature maps of said group and / or at least one oversampling parameter, based on a coding performance measurement, - a step of coding said group of feature maps and said set of oversampling parameters.
[0011] The invention also relates to a method for decoding at least one image, from a binary stream comprising a plurality of samples to be decoded, comprising the steps following: - decoding a group of characteristic maps representative of the current image, said group comprising at least one characteristic map of resolution lower than the resolution of the input image, called lower resolution map; - obtaining a set of oversampling parameters representative of at least one neural network, called an oversampling neural network; - upsampling said at least one lower resolution feature map using said at least one upsampling network to obtain an upsampled map at the resolution of said at least one image. - generation of said at least one image from said at least one oversampled map.
[0012] For the purposes of the invention, encoding, or “coding”, means the operation which consists of representing a set of samples, or pixels, in a compact form carried for example by a digital binary train. Decoding means the operation which consists of processing a digital binary train to restore decoded samples, for example of a digital image or of a plurality of digital images.
[0013] By “image” we mean a two-dimensional digital image comprising one or more components (color, depth, views, etc.)
[0014] By "at least one image" is meant a plurality of ordered two-dimensional images, for example temporally in the case of a video. According to another example, the images may be views of the same scene represented in multi-views. According to another example, the images may be a plurality of temporal and multi-view images (immersive video).
[0015] By "sample" is meant a value taken from an image. Sampling a signal produces a series of discrete values called samples. In the case of an image signal, the sample is called a pixel, which can be, for example, a color pixel traditionally represented by a triplet of values, for example (R,G,B) or (Y,U,V). The position of the sample can be identified by its abscissa (x) and ordinate (y) coordinates in the image.
[0016] By "feature map" is meant an abstract representation of an image or a sequence of images, comprising a plurality of variable scalar data, which are also called characteristic values, for example real numbers, or integers, signed or unsigned. The abstract representation of an image or a sequence may therefore comprise several feature maps. The abstract representation of several images may also correspond to a single feature map.
[0017] By "oversampling of feature maps" is meant an operation which makes it possible to transform the values of a first map into values of a second map. The first map, intended for coding (resp. decoding), is arbitrary. The second map, called the oversampled map, is at a higher resolution. For example, it can be at the same resolution as the input images, i.e. it contains as many values as an input image (resp. to be decoded) contains samples (N).
[0018] By "neural network", or "oversampling neural network", we mean a neural network such as a convolutional neural network, a multi-layer perceptron, etc. The neural network is defined for example by a plurality of layers of artificial neurons and by a set of activation, weighting and addition functions (for example, a layer can calculate y = f(Ax+b), where y and b are vectors of dimension N, x is a vector of dimension M, A is a matrix of dimension MxN, and f is the activation function).
[0019] By "parameter of the neural network" is meant one of the values which characterizes the neural network, for example a weight associated with one of the neurons (filter coefficient, value of a matrix, weighting, bias, value affecting the operation of the non-linearity, etc.)
[0020] By “generation of said at least one image” is meant a step which aims to reconstruct the decoded samples of the image.
[0021] The samples are generated from the oversampled feature maps. The generation of a sample is thus advantageously simplified, in particular in the case where the oversampled feature maps are the same size as the image, because extraction is facilitated. In addition, the oversampling neural network is able to automatically adapt to the nature of the maps and the image.
[0022] By "performance measurement" is meant a measurement between at least one value of a sample to be coded and a decoded value of said sample. The measurement can evaluate for example a distortion, or a perceptual error. It can be carried out on a sample or a plurality of samples (for example, the current samples, or the current images, etc.). The measurement can also include a measurement of the flow rate, in particular associated with the coding of the neural network and / or the coding of the characteristic maps of the group. The measurement can be a joint measurement between the flow rate and the distortion through their weighting. As is well known in the state of the art, the value of this measurement is generally minimized until a target value is reached.
[0023] By "construction step" is meant a step which aims to construct the representative parameters of the image, in particular the values of the characteristic maps and the parameters of the neural network, before their actual coding. The construction sub-steps can be repeated as many times as necessary to obtain an acceptable performance measurement.
[0024] Generally speaking, it is considered that the steps of a coding or decoding method should not be interpreted as being linked to a notion of temporal succession. In other words, the steps may be carried out in a different order than that indicated in the independent coding or decoding claim, or even in parallel.
[0025] The coding method according to the invention carries out a construction of the coding parameters, from an input image or sequence of images, by training a neural network on feature maps which may be at the resolution of the input images, or at a lower resolution. During the training, or construction, the parameters of the neural network and the values of the feature maps are updated according to a performance measurement, for example of the rate-distortion type. When the training is finished, that is to say when the performance measurement obtained is satisfactory, the actual coding of the parameters of the oversampling neural network and the values of the feature maps can be carried out and the result stored or transmitted to the decoder.
[0026] Advantageously, the training process makes it possible to refine the parameters of the upsampling neural network and / or the values of the feature maps until an adequate representation in terms of performance is obtained, for example a desired balance between the generated bit rate and the distortion undergone by the input image or sequence of images being encoded. The training of the values of the feature maps and the parameters of the upsampling neural network may be joint. Advantageously, the coding method according to the invention makes it possible to efficiently compress the signal.
[0027] Advantageously, the compression of the characteristic maps is efficient since at least one of the characteristic maps to be coded (resp. decoded) comprises fewer values than if it were at the resolution of the signal. For example, in the case of a digital image, one of the characteristic maps to be coded (resp. decoded) may be at 1 / 2 resolution, that is to say that it comprises half as many values on the abscissa and ordinate as the input image comprises samples, or in total four times fewer values than a characteristic map at the resolution of the image. On the other hand, the oversampled characteristic map is at a higher resolution, for example of a resolution identical to that of the signal. Oversampling makes it possible to obtain the same number of values in this transformed map as the input signal (resp. to be decoded) comprises samples.
[0028] Advantageously, the decoding method is simple since it suffices to decode the characteristic maps associated with an image or a group of images, and to obtain the oversampling neural network, to reconstruct a decoded version of the current image or the group of current images.
[0029] Such an oversampling neural network can advantageously be of very simple structure with few parameters and few operations per sample.
[0030] In addition, decoding can be performed sample by sample, in a progressive manner.
[0031] According to embodiments of the coding or decoding method:
[0032] - At least a portion of the set of oversampling parameters is decoded from data of said binary stream (resp. coded in the binary stream); advantageously according to this mode, the parameters of the neural network are transmitted in the stream. The module can be transmitted in a quantized form and coded in a compact form using any quantizer and entropy coder accessible to those skilled in the art. A format analogous to that of the MPEG-7 NNR standard can be used.
[0033] - At least part of the set of oversampling parameters is obtained from predetermined values; advantageously according to this mode, the parameters of the neural network are obtained, for example from a network, or from a storage memory.
[0034] - At least a portion of the set of oversampling parameters is decoded (resp. coded) according to reference parameters. Advantageously, according to this mode, the parameters of the neural network are decoded by taking into account reference parameters which can be stored in a memory accessible to the decoder, or obtained from a network. Thus, the storage space or the transmission rate on the network can be effectively reduced: certain parameters can be coded in a complementary manner, others not be coded at all, because they are available in the reference parameters. According to alternative embodiments, which can be combined with each other: • the reference parameters can constitute a set of variants of an upsampling network: type of convolution, attention modules, etc. Thus, only these parameters need to be known to the decoder. They can be, for example, coded, inserted into the stream and decoded by the decoder which can reconstruct, from the reference parameters, the at least one upsampling network. • the oversampling parameters are coded in a complementary manner to the parameters of a reference network. Thus, the decoder will only need to decode these residual parameters and then correct them by addition, multiplication (or combination in any other known manner) with those of the reference neural network. • part of the parameters of a reference network is reused for the upsampling network. For example, a complete layer of the upsampling network can be copied from the reference network, the parameters of this layer are therefore neither coded, nor transmitted, nor decoded. • a reference network indicator is transmitted and decoded in the stream. It is thus possible to indicate a network to be used for decoding from among a plurality of possible networks known to the decoder.
[0035] - Said set of oversampling parameters is representative of a plurality of upsampling neural networks, one upsampling neural network being used to upsample at least one lower resolution feature map of the group of feature maps. Advantageously according to this embodiment, a plurality of upsampling neural networks can be decoded, each of them being intended for decoding one or more maps. Thus the upsampling of the different maps can be carried out separately. According to an alternative embodiment, at least two upsampling neural networks are used to upsample the same lower resolution feature map of the group of feature maps.
[0036] - An upsampling neural network is used to upsample at least two lower resolution feature maps from the feature map group.
[0037] - Said upsampling neural network is used to upsample at least one already oversampled map from the group of feature maps. Advantageously, in this mode, it is possible to use the same oversampling network in cascade. For example, oversampling by a factor of four in each dimension can be achieved by cascading two oversampling networks by a factor of two. This limits the cost of data transmission and storage and makes it possible to limit the modules to be implemented by the encoder and decoder, and therefore the algorithmic complexity, the space occupied in memory, etc.
[0038] - Said at least one lower resolution feature map is oversampled divided in at least two dimensions, and separate upsampling is used for each dimension. Advantageously, in this mode, transmission cost and complexity are reduced by the use of separable upsampling transformations (e.g., convolution).
[0039] - The generation of said at least one image uses a neural network of synthesis capable of generating a sample of the image to be decoded from data extracted from the oversampled feature maps at positions defined by that of said sample in the image. Advantageously, according to this mode, the coding method produces the coded values by training a neural network on vectors of values extracted from the feature maps, associated with a position of a sample to be coded. When the training is finished, that is to say that the performance measurement obtained is satisfactory, the effective coding of the parameters of the synthesis neural network can be performed and stored or transmitted to the decoder. The decoding process is simple since it is sufficient to decode the feature maps, the upsampling neural network and the synthesis neural network to reconstruct a decoded version of the image signal. According to variants of this embodiment: • such a synthesis network includes an MLP. Such a neural network is advantageously of very simple structure with few parameters. In addition, decoding can be carried out sample by sample, in a progressive manner. • The synthesis network includes convolution-type filtering elements. • The construction of the data vector includes an extraction sub-step of a value of said at least one oversampled feature map at a position identical to that of the current sample in the signal to be coded (resp. decoded). Advantageously, this method is simple to implement. For example, if J oversampled feature maps of the same resolution as the signal are available, a simple extraction of the values of the maps at the coordinates of the current sample (at the same abscissa and the same ordinate in the feature map) makes it possible to construct a data vector of J elements to be applied as input to the synthesis neural network to obtain a sample or a plurality of samples as output.
[0040] - At least one of said feature maps is of identical resolution to that of the signal to be coded (resp. decoded). Advantageously, according to this mode, at least one of the characteristic maps, at the same resolution as the input signal to be coded (resp. decoded), allows significant fidelity and respect for the details of the initial resolution of the signal. In this case, oversampling preserves the number of values of the characteristic map; it can be reduced to identity.
[0041] - At least one oversampled map is obtained from a plurality of maps of lower resolution features.
[0042] - Multiple decoded cards are used to produce multiple oversampled cards furrowed.
[0043] Correlatively, the invention also relates to a coding device and a decoding device.
[0044] The characteristics and advantages of the coding or decoding method apply in the same way to the coding or decoding device according to the invention and vice versa.
[0045] The invention also relates to a computer program on a recording medium, this program being capable of being implemented in a computer or a coding or decoding device in accordance with the invention. This program comprises instructions adapted to the implementation of the method cor respondent. This program may use any programming language, and may be in the form of source code, object code, or code intermediate between source code and object code, such as in a partially compiled form, or in any other desirable form.
[0046] The invention also relates to an information carrier or a recording medium readable by a computer, and comprising computer program instructions mentioned above. The information or recording carriers can be any entity or device capable of storing the programs. For example, the carriers can comprise a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a floppy disk or a hard disk, a DNA sequence, or a flash memory. Furthermore, the information or recording carriers can be transmissible media such as an electrical or optical signal, which can be conveyed via an electrical or optical cable, by radio link, by wireless optical link or by other means.
[0047] The program according to the invention can in particular be downloaded from an Internet-type network.
[0048] Alternatively, each information or recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of a method according to the invention. Brief description of the drawings
[0049] Other characteristics and advantages of the present invention will emerge from the description given below, with reference to the appended drawings which illustrate exemplary embodiments thereof which are not in any limiting nature.
[0050] [Fig-1] [Fig.l] schematically represents a coding device used in the scope of the invention;
[0051] [Fig.2] [Fig.2] schematically represents a decoding device used in the context of the invention;
[0052] [Fig.3] [Fig.3] schematically represents an artificial neural network for oversampling used in the context of the invention.
[0053] [Fig.4] [Fig.4] illustrates an example of an oversampling module used in the context of the invention.
[0054] [Fig.5] [Fig.5] presents other examples of implementation of the oversampling module used in the context of the invention.
[0055] [Fig.6] [Fig.6] is a flowchart representing an example of a coding method which can be implemented by the coding device of [Fig.l].
[0056] [Fig.7] [Fig.7] illustrates an encoding method used in a embodiment of the invention.
[0057] [Fig-8] [Fig.8] is a flowchart representing an example of a decoding method which can be implemented by the decoding device of [Fig.2].
[0058] [Fig.9] [Fig.9] illustrates a decoding method used in an embodiment of the invention. Description of the embodiments
[0059] [Fig.l] schematically represents a digital image coding device ENC.
[0060] The coding device ENC of [Fig.l] receives as input a succession of samples to be coded, denoted Pn, for example a temporal succession of sound samples, or a set of image data denoted I(Pn). In this second case, the image signal I(Pn) can represent a two-dimensional image, or a plurality of two-dimensional images (video, color components, stereoscopic components, multiscopic components, etc.). Pn designates a sample n of the input signal comprising N samples. In one embodiment, the signal is a color image signal represented by means of at least one two-dimensional representation, such as a matrix of pixels, of width W and height H (with N = WxH) each pixel comprising a red (R), green (G), blue (B) component, or, alternatively, a luminance (Y) component and at least one chrominance component. The location of each pixel is defined by its abscissa and ordinate (x and y) coordinates in the image.In one embodiment, the image is grayscale represented using a two-dimensional representation, such as a matrix of pixels, each pixel having a grayscale component, or luminance.
[0061] This ENC coding device comprises an INIT module for generating initial characteristic maps, a Q quantization module, a UP module for oversampling the characteristic maps and a PROD signal synthesis module, according to the example represented a digital image I(P'n), an FMC module for coding the characteristic maps, an NNC neural network coding module, a PRC signal coding module, an EVAL module for evaluating a performance function, an MAJ update module.
[0062] An oversampling artificial neural network used for coding, contained in the UP module, and an oversampling artificial neural network used for decoding, contained in the UP' module, are defined by an identical structure, comprising for example a plurality of layers of artificial neurons, and by a set of weights and activation functions associated respectively with the artificial neurons of the network concerned.
[0063] During the process of training, or building, the coding, that is to say as long as that the performance evaluation step is not satisfactory, the coding modules (FMC, NNC, optionally PRC) carry out a coding simulation, followed by decoding, for the evaluation module. Subsequently, they carry out the effective coding of the values of the FM maps, of the parameters of the WUk oversampling neural network, and optionally of the coding parameters of the PROD synthesis module. In a known manner, the coding simulation can be identical to the actual coding, or carry out an approximation thereof.
[0064] Each feature map contains scalar data, in accordance with the definition of a feature map given previously, and there are M feature maps. It is however possible to organize the feature values differently, for example by forming a single signal comprising, for each sample, a vector of M scalar values, without loss of generality. Even if the data are organized differently, it is considered in the remainder of the document that there are M feature maps. The maps coded by the FMC module for a current image (or a plurality of images) are denoted FMc;. The FMC coding module quantizes if necessary the values of the feature maps by using a quantizer to generate an ordered collection of quantized values. Then the coding module compresses the quantized data, for example by using an en-tropic coder.
[0065] The Q module optionally performs a quantization. The Q module produces the FMq maps;. During the coding step, a quantization is essential for the proper functioning of the system if the characteristic maps contain, for example, floating point or real values. It is necessary to quantize them before coding them and / or providing them as input to the oversampling neural network. During decoding, however, inverse quantization is not necessary, depending on the embodiments.
[0066] The UP module performs an oversampling of the FM(q) characteristic maps. This module comprises WUk oversampling parameters representative of an oversampling neural network. This oversampling is performed on at least one map, obtained after the Q module, with a resolution lower than the resolution of the image. Some maps may not require oversampling, in particular if their resolution is equal to that of the image to be coded. Such maps are not processed by the UP module (or the UP module is reduced to the identity function). At the output of the UP module, a set of FMU characteristic maps is obtained; comprising the maps actually oversampled and the non-oversampled maps for which FMU; = FMq;.
[0067] According to embodiments, the UP module may be a convolutional neural network, comprising at least one convolution layer. Each layer convolution module can include nonlinearities, such as rectification, arctangent, sigmoid, etc. It can also include a post-processing module of the convolutional or convolutional with residual type or an attention module. The Up module can also include several neural networks.
[0068] The PROD module is a synthesis module, for example a neural network or an inverse linear transformation (wavelet, DCT, etc.) defined by its synthesis parameters WSk, capable of processing the FMU characteristic maps; as input, to generate as output the synthesized image (coded then decoded) I(P'n). The image samples decoded by the PROD module are denoted P'n
[0069] According to one embodiment, the synthesis module is a filter bank, the WUk parameters are the parameters of the filters, and the output synthesized image results from the application of the filter bank to the FMU characteristic maps;.
[0070] According to one embodiment, the synthesis module is a synthesis neural network and for each sample to be decoded, the synthesis parameters WSk are the parameters of the synthesis neural network.
[0071] According to another embodiment, the neural network is of the convolutional type or of the multilayer perceptron (MLP) type. A combination of these two types of topology is also possible. In order to limit the number of arithmetic operations, an MLP is placed upstream of a convolution module. The synthesis module may comprise a post-processing module of the convolutional type or convolutional with residual applied to a (R,G,B) or (Y,U,V) signal.
[0072] A neural network of the PROD module can behave like the implicit neural networks mentioned above: it can advantageously process vectors from co-located pixels from oversampled feature maps. In this case, the positional decoding of the samples can be carried out independently, which provides an advantage in terms of parallelization of the decoding. A vector (P'n) representative of the sample Pn to be coded is generated at output according to its position in the image. According to one embodiment, the output vector is the triplet (R,G,B) of the decoded pixel P'n, or the triplet (Y,U,V) of the decoded pixel P'n when the image is in YUV 444 format, or a vector comprising four Y values, a U value and a V value, capable of representing four decoded pixels P'n when the image is in YUV 420 format.
[0073] The oversampling parameters WUk of the oversampling neural network encoded by the NNC module are denoted WUck.
[0074] The WSk synthesis parameters of the synthesis module coded by the PRC module are denoted WSck.
[0075] When all the samples Pndu signal have been coded and then decoded, we have a reconstructed signal I(P'n), according to one example an image I comprising N samples decoded in the form of N vectors.
[0076] The EVAL module performs an evaluation and minimization of a coding performance. The evaluation function is for example of the rate-distortion type. It aims to minimize the differences between the input representation of the current image I(Pn) and its output representation I(P'n), while optimizing the quantity of data to be encoded. The minimization can be carried out by gradient descent, or any other optimization method within the reach of those skilled in the art.
[0077] The MAJ module updates the values of the FM maps to be encoded in the current image, based on the results of the performance function.
[0078] Once the encoder training is complete, the parameters of the upsampling network are encoded, either lossless, in which case the decoder's UP' neural network is identical to UP, or lossy, in which case the UP' network may be slightly different from UP. The values of the feature maps, FM. are also encoded as FMc; coded data, for example via an en-tropic encoder. The parameters of the WSck synthesis module are optionally encoded.
[0079] [Fig.2] schematically represents a DEC decoding device.
[0080] The decoding device DEC of [Fig.2] receives as input a first group of encoded data organized into M FMc feature maps; and WUck oversampling parameters of an oversampling neural network. Optionally, the decoding device DEC receives the WSck synthesis parameters of a synthesis module, as illustrated by the dotted arrow.
[0081] This decoding device DEC comprises a neural network decoding module NND, a feature map decoding module FMD, an optional parameter decoding module PRD of the synthesis module, an optional inverse quantization module DQ, a feature map oversampling module UP' and a signal synthesis module PROD', according to the example represented a digital image I(Pdn).
[0082] The maps decoded by the FMD module, numbering M, are denoted FMd;. It is however possible to organize the characteristic maps differently, for example by forming a single signal comprising, for each sample, a vector of M scalar values to be decoded, without loss of generality. Even if the data are organized differently, it is considered in the remainder of the document that there are M characteristic maps to be decoded.
[0083] The oversampling parameters decoded by the NND module are denoted WUdk. According to embodiments, certain WUdk parameters may be predetermined, or obtained by decoding the WUck parameters using reference parameters known to the decoder.
[0084] The synthesis parameters decoded by the PRD module are noted WSdk.
[0085] The DQ module optionally performs an inverse quantization corresponding to the quantization performed at the encoder. The DQ module produces the maps FMdq;. Inverse quantization is not necessary if the encoder's Q quantizer has performed a simple rounding of the real values submitted to it. Otherwise, the decoder performs the inverse operation of the Q quantizer. If quantization is not necessary, FMdq; =FMd;
[0086] The UP' module performs an oversampling of the characteristic maps. This module is parameterized by the WUdk parameters. This oversampling is performed on at least one map, obtained after the DQ module, of resolution lower than the resolution of the image. Some maps may not require oversampling, in particular if their resolution is equal to that of the image to be decoded. Such maps are not processed by the UP' module (i.e. for example it is reduced to identity). At the output of the UP' module, a set of characteristic maps FMUd; comprising the maps actually oversampled and the non-oversampled maps for which FMUd; = FMdq; is obtained. The embodiments described previously in support of the encoder apply. The WUdk parameters are identical to the WUk parameters of the encoder if the coding has been carried out without losses.
[0087] The PROD' module is a synthesis module, for example a neural network or an inverse linear transformation (wavelet, DCT, etc.) defined by its WSd k parameters, capable of processing the characteristic maps FMUd; at input, to generate at output the synthesized (decoded) image I(Pdn). The embodiments described previously in support of the encoder apply. The WSdk parameters are identical to the WS k parameters of the encoder if the encoding has been carried out without losses. The WSdk parameters are optional, that is to say that the PROD' module can use parameters known to the decoder.
[0088] As previously described in support of [Fig. 1], a positional decoding of the Pn samples of the signal can be carried out independently, which provides an advantage in terms of parallelization of the decoding. A vector (Pdn) representative of the Pn sample to be decoded is generated at output according to its position in the image. According to one embodiment, the output vector is the triplet (R,G,B) of the decoded Pdn pixel, or the triplet (Y,U,V) of the decoded Pdn pixel when the image is in YUV 444 format, or a vector comprising four Y values, a U value and a V value, capable of representing four decoded Pdn pixels when the image is in YUV 420 format. When all the Pn samples of the signal have been decoded, a reconstructed signal I(Pd n) is available, according to one example an image I comprising N samples decoded in the form of N vectors (Pdn).
[0089] The decoding device DEC can be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the modules mentioned above can then be realized by the cooperation of the processor and computer program instructions stored in the above memory. mentioned and designed to perform the functionalities of the relevant module, in particular as described below, when these instructions are executed by the processor.
[0090] [Fig.3] illustrates an oversampling module (UP, UP') used in coding and decoding in the context of embodiments of the invention.
[0091] The purpose of the upsampling module is to produce a feature map containing more pixels than the input feature map. The input feature map is denoted FM;, but may correspond to a map FM(d)q; where, as explained in support of Figures 1 and 2, d indicates that the map of index i of the group of feature maps has been previously encoded and decoded, and q that it has been (de)quantized (if necessary) by the Q or DQ module. The output feature map is denoted FMU;. A map FM; of resolution W*H comprising W*H values, where W indicates the width of the map and H its height, is upsampled into a map FMUi of resolution s*W*s'*H. The scale parameters s and s' are positive and greater than or equal to one.If the scale parameters are larger than one, the FM map intended for coding contains fewer coded values than a map with the same resolution as the image, which results in a lowering of the required bitrate.
[0092] In one embodiment s=s' to maintain a proportionality ratio.
[0093] In a general framework, the oversampling module corresponds to at least one artificial neural network. It includes weighting coefficients, biases, and nonlinearities specific to the operation of a neural network. This upsampling module can take the form of a simple convolution after inserting zero values. More generally, a neural network can take the form of a succession of layers, where a layer can calculate y = f(Ax+b), where y and b are vectors of dimension N, x is a vector of dimension M, A is a matrix of dimension MxN, and f is the activation function.
[0094] In one embodiment s and are written as integer powers of 2 (s=2k, s'=2k'), which allows for a simpler implementation by inserting zero values into the FM map; to be processed, as will be explained in support of [Fig.4].
[0095] [Fig.4] illustrates an example of an oversampling module used in the framework of the invention.
[0096] The example in [Fig.4] represents a transposed convolution (CT) upsampling operating on an input feature map FM; of size 4x3 (W= 4 and H = 3 pixels) undergoing an interpolation by a factor of 2 in each direction (s=s'=2), so the size of the output map FMUi is 8x6.
[0097] The transposed convolution takes the form of a 3-dimensional convolution kernel, i.e. a 3*3-dimensional coefficient matrix.
[0098] In a first step, as illustrated in the middle of the figure, the FM map; is interpolated by inserting pixels of zero value between each pair of pixels of the input map. We therefore obtain 4 times more pixels on a grid noted GR of size 8x6 pixels. In a second step, we extend this resulting grid by a number of pixels (1 or 2 depending on the example) depending on the size kxk of the convolution filter (k=3 in this example). In a general and known manner, this extension, or padding, is of size (kl) / 2 if k is odd. The padding appears in the figure in a hatched area. We obtain an extended grid FM'; of size 10x8.
[0099] Then a filtering is carried out using the convolution kernel, of size kxk (here k=3) operating on the extended grid FM';. The 3x3 kernel is applied to the different positions of the grid to obtain the output map of the desired dimension. In [Fig.4], on the right, two positions of the convolution kernel are represented in transparent gray.
[0100] Naturally, any alternative (transposed) convolution operation can be applied. Similarly, any complementary operation within the reach of a person skilled in the art can be added to the transposed convolution: bias, stride (shifting the inverse convolution kernel at each iteration), dilation (spacing the input values provided to the kernel), etc.
[0101] Similarly, any activation function within the reach of those skilled in the art can be used (ReLU, sigmoid, Softmax, Softplus, Softsign, etc.)
[0102] According to the embodiment as presented in [Fig.4], the convolution modules are composed of 2D convolution, that is to say that the convolution kernel is applied jointly on the rows and columns of the feature map.
[0103] According to another embodiment, they can be in a separable form. In this case, rather than performing the convolution on the two-dimensional feature map, a first one-dimensional (1D) convolution operator of length Lh is performed on the rows, then a second convolution of length Lv on the columns of the feature map obtained after the first operator, or vice versa. This implementation has the effect of reducing the storage of Lh*Lv for a given kernel to Lv+Lh and limits the complexity to Lh+Lw multiplications per feature map value versus Lh*Lv in the 2D case. The two 1D operators can be identical in length and have the same number of coefficients. The coefficients can be identical in the horizontal and vertical dimensions. As with non-separable convolution, in the case of transposed convolution the 1D operators are applied after interpolation of the signal.
[0104] [Fig.5] presents examples of implementation of an oversampling module used in the context of the invention.
[0105] A simple module, corresponding to the description provided above in support of [Fig.4], is presented in (a). It comprises a transposed convolution CT. In a manner preferred, it is followed by bias and nonlinearity.
[0106] A module comprising a convolution (C) followed by a transposed convolution (CT) is presented in (b). For example, module C performs a filtering preprocessing on the feature map at its initial resolution.
[0107] A module using a residual connection is presented in (c). According to this mode, the result of the transposed convolution (the oversampled map, for example by a factor of 2 in each dimension) is subjected to a new convolution whose result is added to or subtracted from the oversampled map. The result is a residual map which advantageously corrects the oversampled signal.
[0108] A module including an attention element is presented in (d). Instead of the simple convolution presented on the low branch of the previous case (c), it comprises a convolution function (C2) followed by a sigmoid operator SGM representing the sigmoid-type activation function. An attention module generally allows training a neural network to automatically associate each element of a set with a relevance relative to the other elements based on the context, resulting in the example from C2. This allows more localized processing than a pure residual system which would not have this relevance element.
[0109] Naturally, the preceding elements, given as examples, can be combined.
[0110] In particular, in one embodiment, the oversampling module can take the form of a succession of sampling sub-modules. For example, oversampling by a factor s can be achieved by a succession of modules with smaller factors. An oversampling factor s=6 can be obtained by cascading two modules with respective factors s 1=2 and s2=3. A factor 8 can be obtained by three modules s 1=2, s2=2 and s2=2. Thus, a single oversampling sub-module can be replicated to obtain oversamplings adapted to the different resolutions of the FMd characteristic maps. This embodiment has the advantage of limiting the number of modules and pooling the oversampling modules. Thus, the number of parameters of this network, and consequently their storage and / or transmission volume, is reduced, as is the associated implementation complexity.
[0111] [Fig.6] is a flowchart representing an example of a coding method that can be implemented by the coding device of [Fig.l].
[0112] According to this embodiment, the signal is a two-dimensional image, each sample to be coded is therefore a pixel Pn with coordinates (xn, yn).
[0113] Encoding takes place in two main phases:
[0114] In a first phase, called the construction phase, learning is carried out in order to determine, for an input signal I(Pn), the values of the FM maps; and of the pa parameters WUk of the upsampling neural network and optionally WSk of the production module to optimize a global cost function. The learning is for example carried out by a gradient descent, followed by an update of the parameters of the upsampling neural network UP, optionally of the parameters of the synthesis module PROD, of the values of the feature maps FM;. As is known in the state of the art, the cost function can be of the rate-distortion type, or rate, or distortion, or include a perceptual metric. To measure the rate R, it is necessary to simulate the coding of the maps FM;, then to measure the associated coding rate (the size of the stream Bl). According to one embodiment, the coding of the parameters WUk and / or WSk> is not simulated because their influence is less important than that of the feature maps.According to one embodiment, the coding of the parameters WUk and / or WSk is also simulated and the associated flow rate (the size of the flow B2) is measured. To measure the distortion D, it is necessary to simulate the coding then the decoding of at least part of the image I, to obtain at least one pixel P'n resulting from a simulation of coding then decoding, then to measure the difference between this part of the image or of the set of images I (Pn) at input and a corresponding part of the image or of the set of images I (P'n) coded then decoded.
[0115] Then during a second phase, called the coding phase, the FM maps; and the parameters WUk and possibly WSk are encoded to produce the coded values FMCi and WUck (and possibly WSck) before transmission or storage. They constitute the compressed representation of the input signal I (Pn).
[0116] We will now describe the steps of a method according to one embodiment of the invention.
[0117] During a step E20, a set of images I (Pn) to be coded, comprising a plurality of N samples Pn, is provided as input to the method.
[0118] During a step E21, the M FM cards; are initialized. Subsequently, the WUk parameters of the UP oversampling neural network and the values of the FM cards;, and possibly the WSk parameters of the PROD synthesis module must be optimized during the construction phase.
[0119] According to one embodiment, the FM cards; have a resolution less than or equal to that of the input signal I (Pn) and therefore comprise, for at least one of them, a number N' of values to be coded less than N. According to a variant, the first FM card; is at the resolution of the images and each following card is at a resolution half that of the previous one.
[0120] According to one embodiment, several FM cards; have the same resolution, lower than that of the input signal I (Pn). According to one embodiment, a single FM card; comprises, for each sample, a vector of M scalar values to be decoded, without loss of generality. Even if the data are organized differently, we consider that there are M feature maps to decode.
[0121] According to one embodiment, the FM cards are initialized by the INIT module to predefined constant values.
[0122] According to another embodiment, the FM characteristic maps are initialized by the INIT module to a set of random real values.
[0123] The FM characteristic maps are subsequently updated, or refined, during a step E22, by the update module MAJ of the encoder during its learning.
[0124] During a step E23, the FM maps are quantized.
[0125] During a step E24 the FM maps are coded by the FMC module of the encoder. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is an actual coding and the coded values constitute the stream B1. The simulation can be identical to the actual coding, but it can also be different (for example, simplified). For this coding, for example, an entropic coding of the quantized values is used.
[0126] In one embodiment, the FM cards are coded in the order (FMb FM2,..., FM4), and the variables of each card in a predefined order, for example lexico-graphic. Each card undergoes entropic coding. The entropic coding produces a compressed stream B1 whose flow rate is subsequently measured during a step E28 or E29.
[0127] During a step E25, the M FM cards are oversampled by the FMU module to generate FMU cards at the resolution of the images of the input sequence. At least one FM card has a lower resolution than the images to be coded and the oversampling operation is performed so that the oversampled FMU card has the same number of samples as the images to be coded. Oversampling consists of adding values to the FMU cards and processing them by a neural network at least to achieve the resolution of the input images. The oversampled cards are not coded; they are only used to supply the PROD module.
[0128] The structure and WUk parameters of the upsampling neural network are initialized for example during the first iteration of this step. These parameters are subsequently updated, or refined, during the construction phase, during subsequent iterations of the method.
[0129] According to one embodiment, the parameters of the oversampling network are initialized by predefined values known to give a satisfactory result (for example, following training on a corpus of images).
[0130] According to another embodiment, the parameters of the oversampling neural network are initialized by a set of random values.
[0131] According to one embodiment, M FMU cards are generated.
[0132] According to one embodiment, each FM card; is transformed into an FMU card;.
[0133] During a step E26, the PROD module is called for a synthesis of the image I(P'n ). According to one embodiment, the PROD module is a neural network of the MLP or convolutional type, or a combination of the two, which transforms an input vector into an output vector representative of the sample being processed, for example a pixel. For example, the extraction is carried out in the oversampled maps according to the coordinates (xn, yn) of the sample Pn of the input signal. According to another example, the extraction is carried out by block in the maps to produce a set of output pixels. In this case, the input vector corresponds to the block. The structure and the WSk parameters of the synthesis neural network are initialized for example during the first iteration of this step. These parameters are subsequently updated, or refined, during the construction phase, during subsequent iterations of the method.
[0134] The samples to be coded are for example processed in sequential order, from n=1 to n=N.
[0135] During a step E27, the WSk parameters of the synthesis neural network of the PROD module, if they exist, and the WUk parameters of the upsampling neural network, are quantized and encoded. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is an actual coding and the encoded values constitute the B2 stream. The simulation can be identical to the actual coding, but it can also be different (for example, simplified). Any known technique can be used for this purpose, for example the neural network coding standard proposed by the MPEG-7 part 17 standard, also called NNR (Neural Network Representation). Note that in this case, it is necessary to choose the amount of degradation that the coding brings to the WUk weights, optionally WSk.
[0136] According to one embodiment, the stream B2 is transmitted. According to another embodiment, the stream B2 is stored in memory, for example in ROM or RAM.
[0137] This storage space can contain a bank of parameters allowing to constitute variants of oversampling network. For example, several convolutions (simple interpolators such as cubic, linear interpolations or extended to two dimensions...) several residual convolutions or with attention adapted to specific contents (directional images, graphic images, satellite...) can be stored, an indicator can then be inserted in the stream to inform the decoder of the network to use.
[0138] According to one embodiment, certain parameters WUck of the UP oversampling network may be expressed in relation to a reference network stored in memory. This has the advantage of reducing the space required for storage or network transmission. For example, a layer of the upsampling network can be taken from the reference network, the weights of a layer can be differentially encoded, that is, a corrective value is applied by addition or multiplication to each weight (convolution, bias) of the reference network. These corrective values can be transmitted or stored efficiently using an en-tropic code. Thus, the upsampling network is adapted, with little additional information, to the nature of the signal contained in the feature map and to the processed image(s).
[0139] The two embodiments mentioned above can be combined: a reference indicator can then be inserted into the stream to inform the decoder of the reference network to be used, and the received parameters are used to reconstruct the upsampling network from the reference network to be used.
[0140] During a step E28, a performance measurement is evaluated.
[0141] For this purpose, the coding simulation rates associated with the characteristic maps (simulation of the B1 flow by coding the FM maps;) and optionally with the parameters of the neural network(s) (simulation of the B2 flow by coding the WUk and optionally WSk parameters) are measured.
[0142] According to one embodiment, the cost function is of the rate-distortion type, noted (D+L*R), where D for example the quadratic error measured between the input signal and the decoded signal (or the error measured on a subset of samples of the signal). According to another example, D is calculated from a perceptual function such as the SSIM (for Structural SIMilarity), or MSSSIM (for Multi-scale Structural SI-Milarity). According to one embodiment, R is the simulated rate of the stream B1; according to another embodiment, R is the total rate used to code this image, that is to say the sum of the simulated rates of B1 and B2. L is a parameter which regulates the rate-distortion trade-off. Other cost functions are possible.
[0143] As long as the cost function has not reached its minimum, the performance measurement is not satisfactory, and the method is repeated from step E22. This minimization can be carried out by a mechanism known as gradient descent with updating of the parameters during step E22 for the values of the characteristic maps and E25, E26 for the parameters of the network(s).
[0144] During a step E29, if the cost function has reached its minimum, the training stops. If a coded version corresponding to the last simulation of the parameters of the oversampling neural network (WUk), optionally of synthesis (WSk) and of the characteristic maps (FM;) is available, the streams B1 and B2 can be constituted from it. According to another embodiment, the effective coding of the updated parameters of the neural network(s) and of the values of the characteristic maps (FM;) is carried out at this step to produce the encoded parameters WUck (optionally WSck) and FMc; which constitute streams B1 and B2. Streams B1 and B2 may be concatenated to produce a final stream. According to one embodiment, stream B2 of the coded parameters of the neural network(s) is (are) stored or transmitted before stream Bl, so that it can be decoded before stream Bl.
[0145] [Fig.7] illustrates an encoding method used in one embodiment of the invention.
[0146] In this embodiment, the generated FM maps are four in number. In a preferred embodiment, they are seven in number.
[0147] The first map FMi has the same resolution as the image I (Pn), and therefore has WxH variable values, or scalars, where W represents the width of the image in pixels, and H its height. The second map FM2 has half the resolution (in each dimension) of the map FMb. Each additional map has half the resolution of the previous map. This structure makes it possible to reduce the number of variables in the feature maps, which facilitates coding and learning while minimizing the coding cost. Other types of structure are possible, for example, a reduction rate different from half can be used between the maps (a quarter, or a third, etc.).
[0148] According to this embodiment, the FM cards are quantized by the Q module. Each card has its own quantization sub-module Q. In another embodiment, the quantization module is common to all the cards.
[0149] According to this embodiment, the FMq maps are oversampled by the UP module, with the exception of FMqi which is already at the resolution of the output image. The UPi module is reduced to identity (or non-existent).
[0150] In one embodiment, each card has its own UP oversampling sub-module; which allows it to move from the resolution of the FM card; to the resolution of the image. Thus the FM2 card is oversampled by the UP2 module by a factor of 2 in each dimension, according to a method illustrated with reference to Figures 3 to 5. The FM3 card is oversampled by a factor of 4, which can be achieved by two successive oversamplings by a factor of 2, in each dimension, and the FM4 card by a factor of 8 in each dimension.
[0151] The resulting FMU maps have the same resolution as the image I(Pn), and therefore each have WxH values, where W represents the width of the image in pixels and H its height (N=WxH).
[0152] In another embodiment, the oversampling module is common to all cards; for example, a sub-module of a factor of 2 can be used in cascade and / or shared by several cards.
[0153] In the illustrated embodiment, the synthesis PROD module is a synthesis neural network of the MLP or convolutional type, or a combination of the two, and a vector Zn (zi...z4) is formed from the values extracted from the FMU maps; located at the coordinates (xn, yn) of the current pixel Pn. The vector Zn consisting of the extracted (quantized) values of the FMU maps; is processed by the synthesis neural network of the PROD module to generate a second vector as output, according to the example the output vector is the triplet (R,G,B) of the coded pixel P'n (insofar as the feature maps used for extraction correspond to those coded in the stream) then decoded (i.e. here generated, or produced, by the PROD neural network). The triplet is inserted into the decoded image I (P'n) at the positions (xn, yn) of the color components (R', G', B') and can be used by the quality assessment module, by comparison with the input image I (Pn).
[0154] [Fig.8] is a flowchart representing an example of a decoding method that can be implemented by the decoding device of [Fig.2].
[0155] The decoding described relates to a sub-sequence of images comprising at least one image to be decoded.
[0156] During a step E30, the streams B1 and B2 are extracted from the encoded stream. They contain respectively, for a current subsequence of the sequence, the coded representations of the current maps FMc; and oversampling parameters WUck of the oversampling neural network UP, optionally synthesis parameters WSck of the synthesis module.
[0157] According to one embodiment, the B2 stream is received. According to another embodiment, the B2 stream is stored in memory, for example in ROM or RAM. In accordance with what has been described for the encoder, this storage space may contain a bank, or plurality, of parameters making it possible to constitute oversampling network variants. An indicator can then be read in the stream to inform the decoder of the network to use or to construct from a combination of parameters.
[0158] According to one embodiment, the WUck parameters of the oversampling network may be decoded in relation to a reference network stored in memory. The examples described at the encoder apply. For example, the received parameters are correction values relative to the reference network, which may be decoded using an entropy code and combined with the parameters of the reference network.
[0159] The two embodiments mentioned above can be combined: a reference indicator can then be read in the stream to inform the decoder of the reference network to use, and the received parameters are used to reconstruct the upsampling network from the reference network.
[0160] During a step E31, the M FMd maps are generated by decoding the FMCi values. For this decoding, an entropic decoder is used, for example.
[0161] According to embodiments as described for the encoder:
[0162] - The FMd maps; are at a resolution lower than or equal to that of the signal I (Pdn) at to reconstitute.
[0163] - Several FMd cards; are of the same resolution, lower than the signal resolution.
[0164] - A unique FMd map comprises, for each sample, a vector of M values scalars to decode. Even if the data are organized differently, we consider that there are M feature maps to decode
[0165] During a step E32, according to one embodiment, the FMd; maps are dequantized.
[0166] During a step E33, the upsampling neural network of the UP module is decoded. The WUdk parameters of the UP' upsampling neural network and possibly the WSdk parameters of the synthesis neural network of the PROD module are generated by decoding the WUck and WSck values of the B2 stream. For this purpose, any known decoding technique corresponding to the coding technique used by the coder can be used, for example a format similar to that of the MPEG-7 NNR standard. The synthesis neural network of the PROD' module is similar to the synthesis network of the PROD module, i.e. it has the same structure and includes the same parameters, except for the coding, which can be performed with or without losses. Similarly, the UP' upsampling neural network is similar to the UP upsampling network, i.e. it has the same structure and includes the same parameters, except for the coding, which can be performed with or without losses.
[0167] According to one embodiment, the stream B2 is decoded before the stream B1, in order to be able to have the oversampling neural network UP and possibly the synthesis neural network PROD before starting to decode the samples.
[0168] During a step E34, the M decoded FMd; maps are oversampled by the UP' module to generate FMUd; maps at the resolution of the images of the input sequence. At least one FMd; map has a lower resolution than the images to be coded and the oversampling operation is performed so that the oversampled FMUdi map has the same number of samples as the images to be decoded. The oversampling consists of adding values to the FMd; maps and processing them by a neural network to achieve the resolution of the output images. This step is similar to step E25 which was described for the coder with the support of [Fig.6], and the embodiments apply. In particular:
[0169] According to one embodiment, M FMUd maps are generated.
[0170] According to one embodiment, each FMd map; is upsampled into a map FMUdi.
[0171] According to one embodiment, at least one FMd map; is of lower resolution than that of the images to be coded and the oversampling operation is performed so that the Oversampled FMUdi map comprises the same number of samples as the output image.
[0172] During a step E35, the PROD module is called to generate a synthesis of the samples decoded from the upsampled feature maps. According to one embodiment, the PROD module is an MLP or convolutional neural network, or a combination of both, which transforms an input vector into an output vector representative of the sample being processed, for example, a pixel. For example, the extraction is performed in the upsampled maps according to the coordinates (xn, yn) of the sample Pn of the input signal.
[0173] This step is similar to step E26 which was described for the encoder in support of [Fig.6], and the embodiments apply.
[0174] The samples to be decoded are for example processed in sequential order, from n=1 to n=N.
[0175] When all the samples of the signal have been processed, the decoded signal corresponding for example to the image I (Pdn) is available.
[0176] [Fig.9] illustrates a decoding method used in a embodiment of the invention.
[0177] In this embodiment, the decoded FMd; cards are four in number. In a preferred embodiment, there are seven of them.
[0178] In this embodiment, the first map FMdi has the same resolution as the image I, and therefore comprises WxH variable values, where W represents the width of the image in pixels, and H its height. The second map FMd2 has half the resolution (in each dimension) of the map FMdi. Each additional map has half the resolution of the previous map. This structure makes it possible to reduce the number of variables in the feature maps, which facilitates decoding while minimizing the coding cost. Other types of structure are possible, for example, a reduction rate different from half can be used between the maps (a quarter, or a third, etc.).
[0179] According to this embodiment, the FMd cards are dequantized by the DQ module. Each card has its own DQ quantization sub-module. In another embodiment, the inverse quantization module is common to all cards. In another embodiment, it does not exist.
[0180] According to this embodiment, the FMq maps are oversampled by the UP' module, with the exception of FMqi which is already at the resolution of the output image. The UPi module is reduced to the identity (or non-existent).
[0181] In one embodiment, each map has its own upsampling sub-module UP; which allows it to switch from the resolution of the map FMdi to the resolution of the image. The map FMd2 is upsampled by the module UP'2 by a factor of 2 in each dimension, according to a method illustrated in support of Figures 3 to 5. The map FMd3 is upsampled by a factor of 4, which can be achieved by two successive upsamplings by a factor of 2, in each dimension, and the map FMd4 by a factor of 8 in each dimension.
[0182] The resulting FMUd; maps are of the same resolution as the image I (Pdn), and therefore each have WxH values, where W represents the width of the image in pixels and H its height (N=WxH).
[0183] In another embodiment, as described to the encoder, the oversampling module is common to all cards; for example, a sub-module of a factor of 2 can be used in cascade and / or shared by several cards.
[0184] In the illustrated embodiment, the PROD' module is a synthesis neural network of the MLP or convolutional type, or a combination of both, and a vector Zdn (zdi...zd4) is formed from the values extracted from the FMU maps; located at the coordinates (xn, yn) of the current pixel Pn. The vector Zdn is processed by the synthesis neural network of the PROD' module to generate as output the triplet (R, G, B) representative of the sample Pdn to be decoded. The triplet (R, G, B) is inserted into the decoded image I (Pdn) at the coordinates (xn, yn) in the color components (Rd, Gd, Bd).
Claims
Claims
1. Method for coding at least one current image (I(Pn)) comprising a plurality of samples (Pn) to be coded, comprising the following steps: - a construction step, comprising the sub-steps of: - construction (E21, E22) of a group of characteristic maps (FM; ) comprising at least one characteristic map of resolution lower than the resolution of the current image, called lower resolution map; - oversampling (E25) of said at least one characteristic map of lower resolution using an oversampling artificial neural network (UP) at least defined by a set of oversampling parameters (WUck) to obtain an oversampled map at the resolution of said at least one current image; - generation (E26) of said at least one current image from said at least one oversampled map;- updating (E28, E22) at least one value of one of said feature maps of said group and / or at least one oversampling parameter, as a function of a coding performance measurement, - a coding step (E24, E27) of said group of feature maps (FMci) and of said set of oversampling parameters (WUck).;
2. Coding method according to claim 1, characterized in that said generation (E26) of said at least one current image uses a synthesis module (PROD) defined by a set of synthesis parameters, capable of generating a sample of the current image from data extracted from said at least one oversampled characteristic map at positions defined by that of said sample in the current image.
3. Coding method according to claim 2, characterized in that the method comprises a step of coding said set of synthesis parameters.
4. Method for decoding at least one current image, from a binary stream comprising a plurality of samples (Pdn) to be decoded, comprising the following steps: - decoding (E31) a group of characteristic maps (FMd;) representative of the current image, said group comprising at least one lower resolution feature map than the resolution of the current image, called lower resolution map; - obtaining (E33) a set of oversampling parameters (WUck) representative of at least one neural network (UP1), called oversampling neural network; - oversampling (E34) of said at least one lower resolution feature map using said at least one oversampling network to obtain an oversampled map at the resolution of said at least one current image; - generating (E35) of said at least one current image from said at least one oversampled map.
5. Decoding method according to claim 4, characterized in that at least part of the set of oversampling parameters (WUck) is decoded from data of said binary stream.
6. Decoding method according to claim 4 or 5, characterized in that at least part of the set of oversampling parameters (WUck) is obtained from predetermined values.
7. Decoding method according to one of claims 4 to 6, characterized in that at least part of the set of oversampling parameters (WUck) is decoded as a function of reference parameters.
8. Decoding method according to one of claims 4 to 7, characterized in that the set of oversampling parameters (WUck) is representative of a plurality of oversampling neural networks (UP'i, UP'2, UP'3, UP'4), an oversampling neural network being used to oversample at least one lower resolution feature map of the group of feature maps.
9. A decoding method according to one of claims 4 to 8, characterized in that an upsampling neural network is used to upsample at least two lower resolution feature maps of the feature map group.
10. Decoding method according to one of claims 4 to 9, characterized in that an oversampling neural network is used to oversample at least one already oversampled map of the group of feature maps.
11. A decoding method according to one of claims 4 to 10, characterized in that said at least one lower resolution feature map is oversampled in at least two dimensions, and in that a separate oversampling is used for each dimension.
12. Decoding method according to one of claims 4 to 11, characterized in that said generation of said at least one current image uses a synthetic neural network (PROD1) capable of generating a sample of the current image to be decoded from data extracted from the oversampled feature maps at positions defined by that of said sample in the current image.
13. Device for coding at least one current image (I(Pn)) comprising a plurality of samples (Pn) to be coded, characterized in that said device is configured to implement: - a construction step, comprising the sub-steps of: - construction (INIT, MAJ) of a group of characteristic maps (FM;) comprising at least one characteristic map of resolution lower than the resolution of the current image, called lower resolution map; - oversampling (UP) of said at least one characteristic map of lower resolution using an artificial neural network of oversampling at least (UP) defined by a set of oversampling parameters (WUck) to obtain an oversampled map at the resolution of said at least one current image; - generation (PROD) of said at least one current image from said at least one oversampled map;- updating (EVAL, MAJ) at least one value of one of said feature maps of said group and / or at least one oversampling parameter, as a function of a coding performance measurement; - a coding step (FMC, NNC) of said group of feature maps (FMi) and of said set of oversampling parameters (WUck).;
14. Device for decoding at least one current image, from a binary stream, comprising a plurality of samples (Pdn) to be decoded, characterized in that said device is configured to implement: - decoding (FMD) of a group of characteristic maps (FMd;) representative of the current image, said group comprising at least one characteristic map of resolution lower than the resolution of the current image, called lower resolution map; - obtaining (NND) of a set of oversampling parameters (WUck) representative of at least one neural network (UP1), called oversampling neural network; - oversampling (UP1) of said at least one characteristic map lower resolution characteristics using said upsampling network to obtain an upsampled map to the resolution of said at least one current image; - generation (PROD1) of said at least one current image from said at least one oversampled map.
15. A computer program comprising instructions for carrying out the steps of an encoding or decoding method according to claim 1 or 4 when said program is executed by a computer.
Citation Information
Patent Citations
Implicit image and video compression using machine learning systems
US20220385907A1
Combining convolution and deconvolution for object detection
US20190303715A1
Systems and methods for reducing noise in reconstructed feature data in coding of multi-dimensional data
WO2023037977A1
Cited By
Method and device for decoding data representative of a sound or visual content, method and device for encoding such data, and associated data stream
EP4770093A1
Method and device for decoding data representative of sound or visual content, method and device for encoding such data, and associated data stream
FR3170669A1
