Method and device for encoding and decoding images - Patents.com

Abstract

The present invention relates to a method and device for encoding and decoding a signal comprising a plurality of samples, the decoding method comprising the steps of: decoding a first group of feature maps, transforming said first group of feature maps to obtain a second group of feature maps having the resolution of the signal to be decoded, decoding a set of parameters representing a neural network, for at least one sample of the signal to be decoded, referred to as the current sample, associated with a position in the signal to be decoded: constructing a feature vector from the feature maps of said second group according to the position of the current sample, and processing said feature vector using an artificial neural network defined by the decoded parameters to provide a vector representing the decoded value of the current sample.

Description

[Technical Field]

[0001] The present invention relates to the general field of coding of one-dimensional or multi-dimensional signals, and more particularly to the compression of digital images and videos. [Background technology]

[0002] Digital images are generally source-encoded to be compressed in order to limit the resources required for their transmission and / or storage. There are many encoding standards, such as the JPEG family of standards for still images and the standards of the ITU / MPEG organization for moving images or video (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) and their extensions (MVC, SVC, 3D-HEVC, etc.).

[0003] An image is generally encoded by dividing the image into a number of rectangular blocks and encoding these blocks of pixels in a given processing sequence. In existing video compression techniques, processing of a block typically involves making a prediction of the pixels of the block using previously coded and decoded pixels that are present in the image (in which case it is called "intra-prediction") or in a previously coded image (in which case it is called "inter-prediction") in the process of being encoded. This use of any spatial and / or temporal redundancy avoids transmitting or storing values ​​of the pixels of each pixel block by representing at least some of the blocks using residuals that represent the difference between the predicted values ​​of the pixels of the block and the actual values ​​of the pixels of the predicted block.

[0004] As video formats continue to evolve to provide better compression and to accommodate a wide variety of anticipated formats and communication networks, the number of possible predictions is increasing and conventional encoding and decoding algorithms are becoming very complex.

[0005] In addition to these traditional methods provided by compression standards (JPEG, MPEG, ITU), there is a trend towards developing methods based on artificial intelligence, in particular neural methods.

[0006] Some of these neural approaches can be seen as simple extensions of the concepts of prediction mode competition and competition of compression techniques mentioned above, such as transforms in video coding.

[0007] Another approach uses the "autoencoder" concept. An autoencoder is an artificial neural network-based learning algorithm that allows for the construction of new representations of a dataset. The autoencoder architecture consists of two parts: an encoder and a decoder. The encoder consists of a set of layers of neurons that process data to construct a new representation, referred to as "encoded" (also called a "latent representation"). The decoder's neuron layers then receive these representations and filter them to attempt to reconstruct the original data. The difference between the reconstructed data and the initial data can measure any error introduced by the autoencoder. The essence of training is to modify the autoencoder's parameters to reduce the reconstruction error measured on various samples of the dataset. The performance achieved with such autoencoder-based systems comes at the cost of a significantly increased memory footprint and complexity compared to traditional approaches, such as those provided by compression standards. They can have millions of parameters and require up to one million multiply-accumulate, or MAC, operations to decode a single pixel. This makes such decoders significantly more complex than traditional decoders, potentially hindering the adoption of learning-based compression.

[0008] Recently, a simple neural network-based coding technique was described in (Non-Patent Document 1). The essence of the proposed coding technique is to adjust a neural network to the image and quantize and transmit the network weights. When decoding, the neural network is evaluated at each pixel location to reconstruct the image. Nevertheless, such a technique remains inefficient in terms of compression. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] Emilien Dupont et al., “Compression with Implicit Neural representations” (arXiv:2103.03123) Summary of the Invention [Problem to be solved by the invention]

[0010] Therefore, there is a need for a solution that allows simple and efficient encoding / compression of an image or sequence of images. [Means for solving the problem]

[0011] The present invention relates to an encoding method according to claim 1 and a decoding method according to claim 7.

[0012] In the context of the present invention, by encoding is meant, for example, an operation performed by a digital bitstream, the essence of which is to represent a set of samples in a compact form, and by decoding is meant an operation the essence of which is to process a digital bitstream to recover the decoded samples.

[0013] By a "sample" of a signal, what is meant is a value sampled from the signal. Sampling a signal produces a series of discrete values ​​called samples. In the case of an image signal, the samples are called pixels and may be, for example, color pixels, conventionally represented by a triplet of values, e.g., (R,G,B) or (Y,U,V). Generally, a sample is represented by one or more components or vectors of data.

[0014] By "a signal containing a plurality of samples" is meant a signal that contains one dimension (audio, sound), two dimensions (image), or three or more dimensions (stereoscopic images, multi-view images, images associated with depth maps, video, etc.). Depending on the number of dimensions, a sample may have one, two or several coordinates in the signal. In the case of an image signal, the location of a sample is identified by an x-coordinate and a y-coordinate.

[0015] By "feature map" is meant an abstract representation of a signal that contains a number of potentially discrete variable data, called values, real numbers, or integers, for example. As is known, these maps are also called "latent representations" of the input signal.

[0016] By "transformation of a feature map" is meant the application of a mathematical operation that allows the values ​​of a first map to be transformed into values ​​of a second map. The first map, called a map of the first group and intended for encoding, can be of any type. The second map, called a transformed map or a map of the second group, has the same resolution as the input signal, i.e., contains the same number of values ​​as the number of samples (N) that the input signal contains. The transformation can include, for example, interpolation, oversampling, filtering, quantization, Fourier transformation, etc.

[0017] By "data feature vector constructed from feature maps as a function of position" is meant a vector consisting of one or more, preferably discrete, elements or data, the data being constructed from a second group of feature maps at a location determined by the location of the sample being processed in the signal. This feature vector is one input to the neural network. For example, in the case of a one-dimensional audio signal, such a vector can be constructed from multiple values ​​sampled in each feature map at the same coordinates as the sample being encoded. In the case of an image, such a vector can be constructed from multiple values ​​in each feature sample at the same x and y coordinates as the sample being encoded (or decoded).

[0018] By "neural network" is meant a neural network such as a convolutional neural network, a multi-layer perceptron, an LSTM (an acronym for long short-term memory), etc. A neural network is defined, for example, by multiple layers of artificial neurons containing a set of activation, weighting, and summation functions (e.g., one layer may compute y=f(Ax×b), where y and b are vectors of dimension N, x is a vector of dimension M, A is a matrix of dimension M×N, and f is the activation function). Hereinafter, references will be made to "neural network" or "artificial neural network."

[0019] By "parameter of a neural network" is meant one of the values ​​that characterize a neural network, such as a weight (filter coefficient, weighting, bias, a value that influences the operation of a nonlinearity) associated with one of the neurons.

[0020] By "processing by a neural network" is meant applying a function represented by a neural network to an input feature vector to generate an output vector representing the samples to be encoded (or decoded), which may include one or more pieces of data representing the samples.

[0021] By "performance measure" is meant a measure between at least one value of a sample to be coded and the decoded value of said sample. The measure can, for example, evaluate distortion or perceptual error. The measure can be performed on one sample or on multiple samples (e.g., the entire image). The measure can also include a measure of bit rate, in particular the bit rate associated with the coding of the neural network and / or the coding of the first group of feature maps. The measure can be a joint measure of bit rate and distortion achieved by weighting. As is known in the art, the value of this measure is generally minimized until a target value is reached.

[0022] By "construction step" is meant a step aimed at constructing parameters that represent the image prior to the actual encoding. The construction sub-step can be repeated as many times as necessary to obtain an acceptable performance measure.

[0023] In general, the steps of the encoding method or decoding process should not be interpreted as relating to the notion of temporal succession, in other words the steps may be performed in a different order or even in parallel than indicated in the encoding or decoding independent claims.

[0024] The coding method according to the invention constructs coding parameters from an input signal, e.g., an image, by training a neural network with feature vectors associated with the positions of the samples to be coded. These feature vectors are constructed from feature maps having the resolution of the input signal. During training or construction, the parameters of the neural network and the values ​​of the feature maps are updated as a function of a performance measure, e.g., of the rate-distortion type. Once the training is finished, i.e., if the obtained performance measure is satisfactory, the actual coding of the parameters of the neural network and / or the values ​​of the feature maps is performed and stored or transmitted to a decoder.

[0025] Advantageously, the training process allows for the refinement of the neural network parameters and / or the values ​​of the feature maps until a suitable representation is obtained in terms of performance (e.g., until a desired balance is achieved between the rate generated and the distortion suffered by the input signal). The training of the feature map values ​​and the training of the neural network parameters may be performed jointly. Advantageously, the encoding method according to the invention allows for efficient compression of signals.

[0026] Advantageously, the decoding method is simple, as it is sufficient to decode the first group of feature maps and the neural network to reconstruct a decoded version of the signal (e.g., an image).

[0027] Such neural networks can advantageously be of very simple structure with a small number of parameters.

[0028] Additionally, the decoding can be performed incrementally, sample by sample.

[0029] According to an embodiment of the encoding or decoding method, The method comprises a step of constructing a third group of feature maps, from which a feature vector is further constructed. Advantageously, these additional maps of the third group, constructed identically in the coder and decoder, are neither stored nor transmitted in the coder, nor decoded in the decoder. Thus, it becomes possible to employ additional data to improve compression without reducing the bit rate. The additional data may comprise, for example, coordinates, data available in the maps of the first or second group, data relating to other images already processed by the coder or decoder, etc.

[0030] At least one of the feature maps of the first group has a lower resolution than the signal to be coded (or decoded), and the transformation operation includes oversampling. Advantageously, according to this embodiment, the compression of the feature maps is more efficient, since at least one of the feature maps of the first group to be coded (or decoded) contains fewer values ​​than it would if it had the resolution of the signal. For example, in the case of a digital image, one of the feature maps in the first group may have half the resolution, i.e., contain half the number of x and y values ​​as the number of samples contained in the input signal, i.e., contain a total of 1 / 4 the number of values ​​of a feature map with the resolution of the signal. In contrast, the feature map of the second group, corresponding to the transformation of this map of the first group, has the same resolution as the signal. Therefore, the transformation in this case includes at least one oversampling operation, thereby obtaining in the transformed map the same number of values ​​as the number of samples contained in the input signal (or the signal to be decoded).

[0031] At least one of the feature maps of the first or third group has the same resolution as the signal to be encoded (or decoded). Advantageously, according to this embodiment, at least one of the feature maps has the same resolution as the input signal (or the signal to be decoded), allowing high fidelity and adherence to the details of the initial resolution of the signal. In this case, the transformation does not change the number of values ​​in the transformed feature map. It may be simplified to an identity transformation (no processing is performed on the values ​​of the first group) or may include filtering operations, quantization operations, Fourier transforms, etc. In encoding, if the feature maps contain, for example, float or real values, quantization is essential for the correct operation of the system. They need to be quantized before being encoded and / or input to the neural network. In contrast, in decoding, depending on the embodiment, dequantization may not be necessary.

[0032] Constructing said feature vector comprises the sub-step of extracting a value from said at least one feature map of the second or third group at the same position as the current sample in the signal.

[0033] Therefore, advantageously, it is possible to extract values ​​from the feature maps of the second or third group at the same position as the sample in the signal (the input signal in the case of encoding, or the decoded signal in the case of decoding) to construct the elements of the feature vector. This method is easy to implement. For example, if there are J feature maps of the same resolution as the signal, a feature vector of J elements can be directly constructed by simply extracting the value of the map at the coordinates of the current sample (at the same x and y coordinates of the feature maps).

[0034] - Encoding (or decoding) said first group of feature maps comprises the sub-step of entropy encoding (or decoding). Advantageously, entropy encoding makes it possible to exploit any signal redundancy that may remain in the feature maps, and thus the signal is compressed more efficiently.

[0035] Correspondingly, the invention also relates to an encoding device and a decoding device.

[0036] The features and advantages of the encoding or decoding method apply equally to the encoding or decoding device according to the invention, and vice versa.

[0037] The present invention also relates to a computer program on a recording medium, which can be implemented on a computer or controller according to the present invention, the program comprising instructions configured to implement the corresponding method, which program may use any programming language and may be in the form of source code, object code, or any intermediate code between source code and object code, such as code in partially compiled form or in any other desired form.

[0038] The present invention also relates to a computer-readable information medium or recording medium containing instructions for the computer program. The information or recording medium can be any entity or device capable of storing a program. For example, the medium can include a storage means such as a ROM, e.g., a CD-ROM or a microelectronic circuit ROM, or a magnetic storage means, e.g., a floppy disk or a hard disk, a DNA sequence, or a flash memory. Furthermore, the information or recording medium can be a transmissible medium, such as an electrical or optical signal, which can be routed via an electrical or optical cable, by a radio wave link, by a wireless optical link, or by other means.

[0039] The program according to the invention can be downloaded in particular via the Internet.

[0040] Alternatively, each information medium or recording medium may be an integrated circuit incorporating the program, the circuit being designed to perform or to be used for performing the method according to the invention.

[0041] Other characteristics and advantages of the invention will become apparent from the description given below with reference to the accompanying drawings, which show examples of embodiments that are in no way limiting, and in which: [Brief explanation of the drawings]

[0042] [Figure 1] 1 shows a schematic representation of an encoding device for use in the context of the present invention; [Figure 2] 1 shows a schematic representation of a decoding device for use in the context of the present invention; [Figure 3] 1 shows an example of a synthetic artificial neural network used in the context of the present invention during encoding and decoding. [Figure 4] 2 is a flowchart illustrating an example of an encoding method that may be implemented by the encoding device of FIG. 1. [Figure 5] 1 illustrates an encoding method used in one embodiment of the present invention. [Figure 6]1 illustrates an oversampling method used in one embodiment of the present invention. [Figure 7] 3 is a flowchart illustrating an example of a decoding method that may be implemented by the decoding device of FIG. 2. [Figure 8] 3 illustrates a decoding method used in one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0043] FIG. 1 shows diagrammatically the encoding device ENC.

[0044] This encoding device ENC comprises a module GEN for generating feature maps, a transformation module SE, a data extraction module XTR, a module MLP corresponding to an artificial neural network, a module NNC for encoding the neural network, a module FMC for encoding the feature maps, a module EVAL for evaluating a performance function, an update module MAJ and an optional module FME for generating additional feature maps.

[0045] The encoding device ENC may be implemented by an electronic device including a processor and a memory (not shown), each of said modules then being realized through the interaction of the processor with computer program instructions stored in said memory and designed, when executed by the processor, to carry out the functions of the module in question, in particular as described below.

[0046] The encoding device ENC of Figure 1 takes as input P n A sequence of samples to be coded, e.g., a time sequence of sound samples or I(P n In the second case, the image signal I(P n ) can represent a two-dimensional image or multiple two-dimensional images (video, color components, stereo components, multi-view components, etc.). ndenotes one sample n of an input signal containing N samples. In one embodiment, the signal is a color image signal represented by at least one two-dimensional representation, such as a matrix of pixels of width W and height H (N=W×H), where each pixel has one red component (R), one green component (G), and one blue component (B), or, as a variant, one lightness component (Y) and at least one saturation component. The location of each pixel is defined by its x- and y-coordinates (x and y) in the image. In one embodiment, the image is grayscale and is represented by a two-dimensional representation, such as a matrix of pixels, where each pixel has a grayscale or lightness component. In this case, the vector representing the pixel is simplified to a single component or datum.

[0047] As will be explained in more detail below with reference to FIGS. 3 to 8, the module GEN for generating feature maps comprises FM i The optional module FME is configured to generate, in particular initialize, a plurality of M feature maps, denoted as FME . The FME is not encoded or transmitted. l One or more additional maps (L), denoted as

[0048] The module SE transforms the first group of feature maps to generate a second group of feature maps having the same resolution as the input signal. The module SE may quantize the values ​​of the first group of maps using a quantizer Q to generate an ordered collection of quantized values. Recall that quantizing a value refers to mapping this value to a member of a discrete set of possible code symbols. For example, the set of possible code symbols may consist of integer values, and the quantization system simply rounds the actual value to an integer value. According to another example, quantization consists of multiplying by a given value and then rounding. The module SE then performs a transformation, such as oversampling, interpolation, or filtering, of at least one value of the feature maps. At the end of the transformation, the transformed feature maps of the second group have the same resolution as the input signal.

[0049] The module XTR receives the sample P to be encoded. n , as a function of its coordinate in the input signal, the feature map FMS i and optionally FME l The extracted values ​​are stored in the vector Z n Form Z n is a J-tuple, i.e., a set of J elements or data z i Contains the vector Z of index n n is pixel P' n Refers to the feature vector of

[0050] A modular MLP has K parameters W k and as input, a vector Z n or J tuples and, as output, P samples to be encoded n According to one embodiment, the neural network is an MLP or multi-layer perceptron consisting of an input layer adapted to the input format (J-tuple), optionally one or more hidden layers, and an output layer adapted to the output format of an output vector, generally a vector containing A elements. According to one embodiment, A is equal to 3 and the output vector is the encoded and then decoded pixel P' n is an (R,G,B) triplet.

[0051] The module NNC calculates the neural network, in particular its parameters W k During the coding training or construction process, i.e. as long as the step of evaluating the performance remains unsatisfactory, the module NNC performs a simulation of the coding, followed by the decoding, the results of which are sent to the evaluation module. The evaluation module updates the values ​​of the parameters as a function of the results of the performance measurements performed by the module EVAL. Subsequently, the parameters W of the neural network are k The actual encoding of Wc is performed. kThey are stored or transmitted in the form of stream B2. As is known, the coding simulation may be identical to the actual coding or may approximate it.

[0052] The module FMC calculates the maps FMi, i.e. the quantized values ​​of each map of the first group (map FME l During the coding training or construction process, i.e. as long as the step of evaluating the performance remains unsatisfactory, the module FMC performs a simulation of the coding, followed by decoding, the results of which are sent to the evaluation module. i The encoded map is then used to perform the actual encoding of the values ​​of FMc i They are stored or transmitted in the form of stream B1. As is known, the encoding simulation may be identical to or approximate the actual encoding. The encoding module quantizes the latent representations of the values ​​of the first group of maps, if necessary, by using a quantizer to generate an ordered collection of quantized values. The encoding module then compresses the quantized data, for example using entropy coding.

[0053] The module EVAL performs evaluation and minimization of the coding performance. The evaluation function is, for example, of the rate-distortion type. The minimization may be performed via gradient descent or any other method within the capabilities of a person skilled in the art.

[0054] The module MAJ encodes the FM as a function of the results of the performance function. i Update the map values.

[0055] FIG. 2 shows diagrammatically the decoding device DEC.

[0056] The coding device DEC of FIG. 2 receives as input M feature maps FMc from the stream B1. iThe first group of coded data organized in the FM layer (also called the FM layer) and the coded parameters Wc of the neural network from stream B2. k and receive.

[0057] This decoding device DEC comprises a module NND for decoding the neural network, a module FMD for decoding feature maps, a data extraction module XTR′, an inverse transformation module SE′, a module MLP′ corresponding to the neural network and an optional module FME′ for generating additional feature maps.

[0058] The maps or layers (M) decoded by the module FMD are i The parameters decoded by the module NND are Wd k It is shown as follows.

[0059] The decoder module FME' l It is also possible to generate one or more additional maps, denoted as FME, where the number of these is L and the additional maps generated by the coder are FME. l is the same as

[0060] The module SE' receives the decoded feature map FMd i transform the first group of feature maps to generate a second group of feature maps with the same resolution as the input signal, and these maps are called FMS' i The module SE′ optionally performs an inverse quantization corresponding to the quantization performed in the coder. If the quantizer Q of the coder simply rounds the actual values ​​submitted, inverse quantization is not necessary. Also, if the neural network is able to take into account the quantization of the input data, inverse quantization is not necessary. Otherwise, the decoder performs an inverse quantization of the quantizer Q. The module SE′ also performs a transformation of the values ​​of the feature maps, e.g., oversampling, interpolation, filtering, etc. At the end of the transformation, the transformed feature maps of the second group are at the same resolution as the signal to be decoded.

[0061] The module XTR' is identical to the module XTR in Figure 1. The module XTR' generates M transformed feature maps FMS' i and optionally L additional maps FME' l Extract the value from the decoded sample P n , a vector Z in the form of a J-tuple as a function of the coordinates in the signal to be decoded. n In one embodiment, J=M. In one embodiment, J=M+L.

[0062] The module MLP' has K parameters Wd k and takes as input a vector Zd n or receives J tuples and outputs the decoded samples P n , typically a vector with A elements. According to one embodiment, A=3 and the output vector is n In one variant, the output vector includes one lightness component (Y) and at least one saturation component. The module MLP' has the same structure as the module MLP, and its parameters are the parameters W k are identical if the encoding is lossless, or different if the encoding is lossy.

[0063] All samples of the signal P n is decoded, the reconstructed signal I(Pd n ) is obtained, and according to one example, N vectors Pd n An image I is obtained that contains N decoded samples, which are of the form:

[0064] The decoding device DEC may be implemented by an electronic device including a processor and a memory (not shown), each of said modules then being realized through the interaction of the processor with computer program instructions stored in said memory and designed, when executed by the processor, to carry out the functions of the module in question, in particular as described below.

[0065] FIG. 3 illustrates an example of a synthetic artificial neural network used for encoding and decoding in accordance with an embodiment of the present invention.

[0066] The composite artificial neural network MLP used for encoding and the composite artificial neural network MLP' used for decoding are defined by the same structure, e.g., including multiple layers of artificial neurons, and a set of weights and activation functions associated with each of the artificial neurons of the network in question.

[0067] Vector representation of the current sample (feature map FMS i and FME l or FMS' i and FME' l Vector Z obtained from n or Zd n ) is applied to the input (i.e., input layer) of the composite artificial neural network MLP or MLP'. The artificial neural network outputs as output a vector P' representing the decoded samples. n or Pd n According to one embodiment, the constituent color components (R, G, B) of a color pixel of an image are generated.

[0068] The decoded or reconstructed image is obtained by concatenating all the reconstructed pixels into a (2D, 3D) image.

[0069] In the coder, a synthetic artificial neural network (MLP) optimizes the current image I(P n ) and its output representation I(P' n) is trained on images to minimize the difference between the images. The module EVAL performs performance measurements in this sense.

[0070] Once trained, the network's parameters are coded either reversibly (the neural network MLP' is identical to the MLP) or irreversibly (the network MLP' can be slightly different from the MLP).

[0071] FIG. 4 is a flowchart illustrating an example of an encoding method that may be implemented by the encoding device of FIG.

[0072] According to this embodiment, the signal is a two-dimensional image, and therefore each sample to be coded has coordinates (x n ,y n ) pixel P n is.

[0073] The encoding occurs in two main phases.

[0074] In the first phase, called the construction phase, the input signal I(P n ) and optimize the overall cost function. i and parameter W k Learning is performed to determine the values ​​of the parameters of the neural network MLP and the feature map FM. i This is followed by updating the value of . As is known in the art, the cost function may be of the rate-distortion type or of the rate, distortion or perceptual type. To measure the bit rate R, the map FM i The encoding of the parameter W must be simulated and then the associated encoding bitrate (size of stream B1) must be measured. k The encodings of the parameters W are not simulated because their influence is smaller than that of the feature maps. kis also simulated and the associated bit rate (size of stream B2) is measured. To measure the distortion D, it is necessary to simulate the encoding and decoding of at least a portion of image I, so that at least one pixel P' resulting from the encoding and then decoding simulation is n Then, the input image I(P n ) and the encoded and then decoded image I(P' n ) and the corresponding part of

[0075] Then, during the second phase, called the encoding phase, the map FM i and parameter W k is encoded before being transmitted or stored, and the encoded value FMc i and Wc k They generate the input signal I(P n ) to form a compressed representation of

[0076] The steps of the method according to one embodiment of the present invention will now be described.

[0077] In step E20, a number N of samples P n The signal I(P n ) is sent as input to the method.

[0078] In step E21, a first group of M maps FM i is initialized by the module GEN. Next, the neural network parameters W k and Map FM i The value of must be optimized during the construction phase.

[0079] According to one embodiment, the map FM i is the input signal I(P n ), and therefore each has the same resolution as the encoded sample P n Contains as many values ​​N as there are

[0080] According to one embodiment, the map FM i is the input signal I(P n ) or less, and therefore at least one of them contains N' coded values, the number of which is less than N.

[0081] According to one embodiment, the first map FM i has the resolution of the signal, with each subsequent map having half the resolution of the previous map.

[0082] According to one embodiment, a plurality of maps FM i has the same resolution, which is less than the resolution of the signal.

[0083] According to one embodiment, the map FM i is initialized to a predetermined constant value.

[0084] According to another embodiment, the feature map is initialized with a set of random real numbers.

[0085] According to one embodiment, one or more maps FME forming a third group of L additional feature maps l are generated and added to the first group. They are used to construct the feature vector but are not stored or transmitted.

[0086] This feature map FM of the first group i is subsequently updated or refined in step E22 by updating the coder's module MAJ during the training of the construction phase.

[0087] In step E23, the map FM of the first group is iare coded by the coder module FMC. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is the actual coding, the coded values ​​forming the stream B1. The simulation can be identical to the actual coding, but it may also be different (for example simplified). For this coding, it is possible to use any known technique aimed at compressing the values ​​of a map.

[0088] In one embodiment, the map FM i are coded in the order (FM1, FM2, ..., FM4), the variables of each map being coded in a predetermined order, for example lexicographical order. Each map is then subjected to entropy coding, which generates a compressed stream B1, the bit rate of which is subsequently measured during step E28.

[0089] During step E24, the first group of M maps FM i is converted by the module SE to map the second group with the resolution of the input signal FMS i Generate.

[0090] According to one embodiment, M map FMSs i is generated.

[0091] According to one embodiment, each map FM i Map FMS i is converted to

[0092] According to one embodiment, at least one map FMi has a lower resolution than the signal to be coded and the transformation operation comprises oversampling, so that the transformed map FMS i contains the same number of samples as the input signal. Oversampling is performed by oversampling the map FMS i to achieve the resolution of the input signal. i It may be simple (by replicating nearest neighbors) or may involve interpolation (linear, polynomial, filtered, etc.).

[0093] In step E25, the vector Z n But for each sample P n Map FMS of the second group as a function of its coordinates i and optionally a third group of FMEs l Constructed by module XTR from values ​​extracted from

[0094] In one embodiment, Z n is the input map FMi (and optionally FME l ) contains as many values ​​as there are in the array. In this case, J = M(+L).

[0095] In one embodiment, Z n is the current pixel P as explained with reference to Figure 5. n coordinates (x n ,y n ) Map FMS i (and optionally FME l ) values ​​form a J-tuple (z1,z2,...,z J ) Map FMS i or FME l coordinates (x n ,y n ) values ​​in this map FMS i or FME l Vector Z related to n The final value of element i of (z i ) is given.

[0096] The samples to be coded are processed sequentially, for example from n=1 to n=N.

[0097] In step E26, the vector Z n is processed by the neural network MLP and the output is the encoded sample P n , according to one embodiment, a vector representing the sample P′ n (encoded and then decoded sample P n ) to generate an (R,G,B) triplet.

[0098] Neural network structure and parameters W k are initialized, for example, during the first iteration of this step. These parameters are then updated or refined during the construction phase in subsequent iterations of the process.

[0099] According to one embodiment, the parameters of the neural network are initialized with predetermined values ​​that are known to give satisfactory results (eg, after training on a corpus of images).

[0100] According to another embodiment, the parameter W of the neural network k is initialized with a set of random values.

[0101] In step E27, the parameters W of the neural network MLP are calculated. k is quantized and coded. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is the actual coding, and the coded values ​​form the stream B2. The simulation may be identical to the actual coding, but it may also be different (for example simplified). For this, any known technique can be used, for example the neural network coding standard proposed in part 17 of MPEG-7, also called neural network representation or NNR. In this case, the coding is performed using a parameter W k Note that it is necessary to choose the amount of reduction that occurs in the

[0102] In step E28, the performance measures are evaluated.

[0103] For this purpose, the coding simulation rate associated with the feature map (map F i ) and optionally a coding simulation rate associated with the parameters of the neural network (parameters W kThe measured values ​​are a simulation of stream B2 by encoding and decoding the stream B1.

[0104] According to one embodiment, the cost function is of the rate-distortion type, denoted (D+L*R), where D is for example the squared error measured between the input signal and the decoded signal (or the error measured on a subset of the signal's samples). According to another example, D is calculated using a perceptual function such as SSIM (Structural Similarity) or MSSSIM (Multi-Scale Structural Similarity). According to one embodiment, R is the simulated bitrate of stream B1, and according to another embodiment, R is the overall bitrate used to code this image, i.e. the sum of the simulated bitrates of B1 and B2. L is a parameter that adjusts the rate-distortion compromise. Other cost functions are possible.

[0105] The performance measure remains unsatisfactory until the cost function reaches a minimum and the method is repeated from step E22. This minimization can be performed via known mechanisms such as gradient descent, and the parameters are updated in step E22 for the feature map values ​​and in step E26 for the network parameters.

[0106] In step E29, training stops when the cost function reaches a minimum. k ) and feature map (FM i ) is available, streams B1 and B2 can be formed from it. According to another embodiment, the updated parameters of the neural network (W k ) and feature map (FM i The actual encoding of the values ​​of Wc ) is performed in this step to generate the encoded parameters Wc forming the streams B1 and B2. k and FMc i Generate.

[0107] Streams B1 and B2 can be concatenated to generate a final stream. According to one embodiment, stream B2 of coded parameters of the neural network is stored or transmitted before stream B1 so that it can be decoded before stream B1.

[0108] Figure 5 shows the image I(P n 1 illustrates an encoding method used in one embodiment of the present invention to encode a .times. ...

[0109] In this embodiment, the map FM i The number of features is 4. In a preferred embodiment, the number is 7. Map FM1 has the resolution of the image to be encoded. The second map FM2 has half the resolution (in each dimension) of map FM1. Each additional map has half the resolution of the previous map. This structure allows for a lower number of variables in the feature maps, thus facilitating encoding and learning while minimizing encoding costs.

[0110] Map FM2 is oversampled by a factor of 2 in each dimension using the method described with reference to Figure 6. Map FM3 is oversampled by a factor of 4 in each dimension, and map FM4 is oversampled by a factor of 8 in each dimension.

[0111] Generated Map FMS i is the image I(P n ), and therefore each contains W×H values, where W is the width of the image in pixels and H is its height (N=W×H).

[0112] Other types of structures are possible, for example, reduction levels other than 1 / 2 (such as 1 / 4 or 1 / 3) may be used between maps.

[0113] In one variation, shown by the dotted lines, there are five feature maps, and an additional map, FME0, is introduced, which is not stored, encoded, or transmitted. This additional map typically contains data that may assist the MLP network in the task of reconstructing the signal. Thus, the added map may be one or more from the following non-limiting list: - a map containing, for each point, the x coordinate of this point, - a map containing, for each point, the y coordinate of this point, - a map containing the location coding for each point (for example, as explained at the following internet address: https: / / skosmos.loterre.fr / P66 / fr / page / -K0D65X2X-X), - a map representing an image that is distinct from the image being processed and that is able to provide information about the image to be coded (for example a previously processed image, if the current image is part of a sequence of images to be coded, such as a video, a set of medical images, a multi-view representation, etc.), - a map representing the feature maps of an image (e.g. a previously processed image, if the current image is part of a sequence of images to be coded, such as a video, a set of medical images, a multi-view representation, etc.) that is distinct from the image being processed and that is able to provide information about the image to be coded, - Already decoded samples of the same map, e.g. a map containing the values ​​of the previous samples in decoding order.

[0114] In this embodiment, the vector Z n is the current pixel P n coordinates (x n ,y n ) Map FMS i The extracted values ​​are quantized and the vector Z n is processed by the neural network MLP and outputs as output a second vector, in this example the samples P to be coded. n The output vector, in this case, is the encoded and then decoded pixel P'n The triplet is the (R,G,B) triplet of the color components (R',G',B') at the position (x n ,y n ) the decoded image I(P' n )

[0115] In one embodiment, the vector Z n is quantized after extraction.

[0116] In another embodiment, a four map FMS i is quantized by the module SE, and the vectors generated in this case contain already quantized values.

[0117] In the deformation shown by the dotted line, the vector Z n is a 5-tuple (z0···z4), where the value z0 is extracted from the additional map FME0.

[0118] FIG. 6 illustrates the oversampling method used in one embodiment of the present invention during encoding and decoding.

[0119] According to the example shown in Figure 6, the map FM2 (or FMd2) of the first group has a resolution of 1 / 2. The interpolation-based oversampling method generates an image FMS2 with the same resolution as the input signal.

[0120] According to a simple example, to generate the value of the map FMS2 at position (x, y), the values ​​at positions (x / 2, y / 2), (x / 2-1, y / 2), (x / 2, y / 2-1), (x / 2-1, y / 2-1) in the map FM2 (or FMd2) are taken into account and an operation is applied to them to obtain the value FMS2 (or FMS'2) at position (x, y). This may for example be a matter of averaging, interpolation, etc.

[0121] Of course, any oversampling algorithm within the capabilities of one skilled in the art may be used.

[0122] FIG. 7 is a flowchart illustrating an example of a decoding method that may be implemented by the decoding device of FIG.

[0123] At step E30, streams B1 and B2 are extracted from the coded stream, each of which contains a coded representation FMc of the map of the first group. i and the coded representation of the parameters Wc k Includes.

[0124] In step E31, M maps FMd i But the value FMc i To perform this decoding, any known technique can be used, preferably entropy decoding, similar to that used in coders. In one embodiment, the map FMd i are decoded in the order of (FMd1, FMd2, . . . , FMd4), and the variables of each map are decoded in a predetermined order, for example, lexicographic order.

[0125] According to an embodiment equivalent to that described for the coder, the map FMd i is the signal to be reconstructed I(Pd n ), and therefore each decoded sample Pd n It contains N values, the same number as the number of

[0126] According to an embodiment equivalent to that described for the coder, the map FMd i is the signal to be reconstructed I(Pd n ) has a resolution of

[0127] According to an embodiment equivalent to that described for the coder, the first map FMd i has the resolution of the signal, with each subsequent map having half the resolution of the previous map.

[0128] According to an embodiment equivalent to that described for the coder, a plurality of maps FMd ihas the same resolution, which is less than the resolution of the signal.

[0129] In step E32, according to one embodiment, one or more maps FME' forming a third group of L additional feature maps are l are generated and added to the second group. They are not decoded, but are generated by the decoder in the same manner as they are generated in the coder. They typically contain data that can aid the MLP' network in the task of reconstructing the signal. The non-limiting list of possible additional feature maps described with reference to FIG. 4 in the context of the coder also applies here.

[0130] In step E33, a first group of M maps FMd i is transformed by the module SE into a map FMS' of the second group with the resolution of the input signal. i Generate.

[0131] According to an embodiment equivalent to that described for the coder, M maps FMS' i is generated.

[0132] According to an embodiment equivalent to that described for the coder, each map FMd i Map FMS i is converted to

[0133] According to an embodiment equivalent to that described for the coder, at least one map FMd i is of lower resolution than the signal to be coded, and the transformation operation involves oversampling such that the transformed map FMS'i contains the same number of samples as the input signal. Oversampling involves oversampling the values ​​of the map FMS' i to achieve the resolution of the input signal. Oversampling may be simple (by replicating nearest neighbors) or may involve interpolation (linear, polynomial, filtered, etc.).

[0134] The transformation may optionally include dequantization of the extracted values ​​if necessary, although dequantization is not required.

[0135] In step E34, the vector Zd n is the sample P to be decoded. n For the coordinates (x n ,y n ) as a function of the second group map FMS' i and optionally a third group of FMEs' l This step is identical to step E25 described in the context of the coder with reference to FIG. 4 and to which the described embodiment applies.

[0136] In particular, in one embodiment, Zd n is the current pixel P as shown with reference to FIG. n coordinates (x n ,y n ) Map FMS' i (and optionally FME' l ) values ​​form a J-tuple (z1, z2, , z J ) The coordinates (x n ,y n ) values ​​in this map FMS' i or FME' l Vector Zd for n The final value of element i of (zd i ) is given.

[0137] If necessary, the extraction is performed by extracting the extracted values ​​or forming vector Zd n This may include inverse quantization of

[0138] The samples to be decoded are processed sequentially, for example from n=1 to n=N.

[0139] At step E35, the parameters Wd of the neural network MLP' are calculated. k However, the value Wc of stream B2 kFor this purpose, any known decoding technique corresponding to the encoding technique used by the coder can be used. The neural network MLP' is similar to the network MLP, i.e., it has the same structure and the same parameters regardless of the encoding, which may be lossy or lossless.

[0140] According to one embodiment, stream B2 is decoded before stream B1 in order to acquire the neural network before starting to decode the samples.

[0141] In step E36, the vector Zd n is processed by the neural network MLP' and the output is the decoded sample Pd n , according to one embodiment, the position (x n ,y n ) the decoded image I(Pd n ) This step is identical to step E26 described for the coder with reference to FIG.

[0142] If all samples of a signal are processed, for example, the image I(Pd n ) is obtained.

[0143] FIG. 8 illustrates the decoding method used in one embodiment of the present invention.

[0144] In this embodiment, the map FMd i The number is 4. In a preferred embodiment, the number is 7.

[0145] In this embodiment, the first map FMd1 has the same resolution as image I and therefore contains W x H variables, where W is the width of the image in pixels and H is its height. The second map FMd2 has half the resolution (in each dimension) of map FMd1. Each additional map has half the resolution of the previous map. This structure allows the number of variables in the feature maps to be reduced, facilitating decoding while minimizing encoding costs.

[0146] Map FMd2 is oversampled by a factor of 2 in each dimension using the method described with reference to Figure 6. Map FMd3 is oversampled by a factor of 4 in each dimension, and map FMd4 is oversampled by a factor of 8 in each dimension.

[0147] Map FMS' i has the same resolution as the image to be decoded and therefore contains W×H variables, where W is the width of the image in pixels and H is its height.

[0148] In this embodiment, the vector Zd n is the current pixel Pd n coordinates (x n ,y n ) Map FMS' i The vector Zd is a 4-tuple (zd1···zd4) formed from the values ​​of n is optionally dequantized and then processed by the neural network MLP′ to produce as output a second vector, according to the example, of decoded samples Pd n The (R,G,B) triplet is the coordinate (x n ,y n ), the decoded image I(Pd n )

[0149] In one variant, shown in dotted lines, the number of maps is 5 and an additional map FME'0 is introduced. In this embodiment, the vector Zdn is a 5-tuple.

Claims

1. The number of samples to be coded (P ｎ ) containing the signal (I(P ｎ )) a method for encoding a signal comprising the steps of: A construction step comprising: Feature Map (FM ｉ a substep (E21, E22) of constructing a first group of The feature map (FM ｉ ) to generate a feature map (FMS ｉ a substep (E24) of obtaining a second group of Position (x ｎ , y ｎ ) of the signal to be coded, which is associated with the current sample (P ｎ ) for at least one sample, called The current sample (P ｎ ) at the position (x ｎ , y ｎ ) as a function of the feature map (FMS ｉ ) to the feature vector (Z ｎ ) a sub-step (E25), The decoded value of the current sample (P' ｎ ) to provide a vector representing the parameters (W ｋ ) is used to generate the feature vector (Z ｎ a substep (E26) of processing a substep (E22, E26) of updating at least one value of one of the feature maps of the first group and / or at least one parameter of the network as a function of the coding performance measure; a construction step, The feature map (FM ｉ ) and the first group of parameters (W ｋ ) (E23, E27, E29) 1. A method for encoding a signal, comprising:

2. Feature Map (FME) ｌ 2. The method for encoding a signal according to claim 1, further comprising a step (E21, E22) of constructing a third group of feature maps of said first group and said feature vector being further constructed from said feature maps of said third group.

3. The feature map (FM) of the first group ｉ 3. A method for encoding a signal according to claim 1 or 2, characterized in that at least one of the first and second inputs is of lower resolution than the signal to be encoded and the transformation operation comprises oversampling.

4. The feature map (FM) of the first or third group ｉ , FME ｌ 4. A method for encoding a signal according to claim 1, wherein at least one of the first and second inputs has the same resolution as the signal to be encoded.

5. The feature vector (Z ｎ ) is constructed by constructing the current sample (P ｎ ) and the same position (x ｎ , y ｎ ) the at least one feature map (FMS ｉ , FME ｉ 5. A method for encoding a signal according to claim 1, further comprising the substep of extracting a value from the signal.

6. Method for coding a signal according to any one of claims 1 to 5, characterized in that the coding of the first group of feature maps comprises a sub-step of entropy coding.

7. The number of samples to be decoded (Pd ｎ 1. A method for decoding a signal comprising: Feature map (FMd ｉ a step (E31) of decoding a first group of The feature map (FMd ｉ ) to generate a feature map (FMS') having the resolution of the signal to be decoded. ｉ a step (E33) of obtaining a second group of The parameter (Wc ｋ ) and Position (x ｎ , y ｎ ) of the current sample (Pd ｎ ) for at least one sample, called The position (x ｎ , y ｎ ) as a function of the feature map (FMS') of the second group ｉ ) to the feature vector (Zd ｎ ) (E34), The current sample (Pd ｎ ) to provide a vector representing the decoded values ​​of the decoded parameters (Wd ｋ ) is used to calculate the feature vector (Zd ｎ ) processing step (E36); 10. A method for decoding a signal, comprising:

8. Feature map (FME' ｌ 8. The decoding method according to claim 7, characterized in that it comprises a step (E32) of constructing a third group of feature maps of the first group, and the feature vector is further constructed from the feature maps of the third group.

9. The feature map (FMd) of the first group ｉ 9. A decoding method according to claim 7 or 8, characterized in that at least one of the signals is of lower resolution than the signal to be decoded and the transformation operation comprises oversampling.

10. The feature map (FMd) of the first or third group ｉ , FME' ｌ 10. A method for decoding a signal according to any one of claims 7 to 9, characterized in that at least one of the first and second inputs has the same resolution as the signal to be decoded.

11. The feature vector (Zd ｎ ) is constructed by constructing the current sample (Pd ｎ ) and the same position (x ｎ , y ｎ ) the at least one feature map (FMS') of the second or third group in ｉ , FME' ｉ 11. A method for decoding a signal according to any one of claims 7 to 10, characterized in that it comprises the substep of extracting a value from

12. Method for decoding a signal according to any one of claims 7 to 11, characterized in that the decoding (E31) of the first group of feature maps comprises a sub-step of entropy decoding.

13. The number of samples to be coded (P ｎ ) containing the signal (I(P ｎ )) a device for encoding a signal, the device comprising: Feature Map (FM ｉ ) to construct the first group (GEN, MAJ) The feature map (FM ｉ ) to generate a feature map (FMS ｉ ) (SE), Position (x ｎ , y ｎ ) of the signal to be coded, which is associated with the current sample (P ｎ ) for at least one sample, called The current sample (P ｎ ) at the position (x ｎ , y ｎ ) as a function of the feature map (FMS ｉ ) to the feature vector (Z ｎ ) (XTR), The decoded value of the current sample (P' ｎ ) vector (S ｎ ) to provide the parameter (W ｋ ) is used to generate the feature vector (Z ｎ ) (MLP), updating at least one value of one of the feature maps of the first group and / or at least one parameter of the network (MAJ, NNC) as a function of the coding performance measure; The feature map (FM ｉ ) and the first group of parameters (W ｋ ) to encode the set (FMC, NNC) 1. A device for encoding a signal, configured to:

14. The number of samples to be decoded (Pd ｎ 1. A device for decoding a signal comprising: Feature map (FMd ｉ ) decoding a first group (FMD); The feature map (FMd ｉ ) to generate a feature map (FMS') having the resolution of the signal to be decoded. ｉ ) (SE′), and The parameter (Wc ｋ ) (NND), and Position (x ｎ , y ｎ ) of the current sample (Pd ｎ ) for at least one sample, called The position (x ｎ , y ｎ ) as a function of the feature map (FMS') of the second group ｉ ) to the feature vector (Zd ｎ ) (XTR'); The current sample (Pd ｎ ) to provide a vector representing the decoded values ​​of the decoded parameters (Wd ｋ ) is used to calculate the feature vector (Zd ｎ ) (MLP'); 10. A device for decoding a signal, configured to:

15. A computer program comprising instructions for carrying out the steps of the encoding or decoding method according to claim 1 or 7 when said program is executed by a computer.

Images