Method and device for encoding and decoding images - Patents.com

Abstract

The embodiments relate to a method and a device for encoding and decoding a signal comprising a plurality of samples (Pdn). The decoding method comprises the following steps: i ) parameter (Wd k ) in the signal to be decoded, n ,y n for at least one sample of the signal to be decoded, called the current sample (Pdn), associated with the first group (FMd) as a function of the position (xn,yn) of the current sample; i ) feature map to the feature vector (Zd n ) (E33, E34, XTR', TT'), and - the decoded parameters (Wd k ) is used to generate the above feature vector (Zd n ) to process the current sample (Pd n ) and providing a vector representing the decoded value of (E36, MLP').

Description

[Technical Field]

[0001] The present invention relates to the general field of coding of one-dimensional or multi-dimensional signals, and more particularly to the compression of digital images and videos. [Background technology]

[0002] Digital images are generally source-encoded to be compressed in order to limit the resources required for their transmission and / or storage. There are many encoding standards, such as the JPEG family of standards for still images and the standards of the ITU / MPEG organization for moving images or video (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) and their extensions (MVC, SVC, 3D-HEVC, etc.).

[0003] An image is generally encoded by dividing the image into a number of rectangular blocks and encoding these blocks of pixels in a given processing sequence. In existing video compression techniques, the processing of a block typically involves prediction of the pixels of the block performed using previously encoded and then decoded pixels present in the image being encoded, in which case it is referred to as "intra-prediction," or prediction of the pixels of the block performed using a previously encoded image, in which case it is referred to as "inter-prediction." This use of any spatial and / or temporal redundancy avoids transmitting or storing values ​​of the pixels of each pixel block by representing at least some of the blocks using residuals that represent the difference between predicted values ​​of the pixels of the block and the actual values ​​of the pixels of the predicted block.

[0004] Video formats are constantly evolving in search of higher compression ratios and becoming more predictable to accommodate a wide variety of expected formats and communication networks, making conventional encoding and decoding algorithms very complex.

[0005] In addition to these traditional methods proposed by compression standards (JPEG, MPEG, ITU), there is a trend towards the development of methods based on artificial intelligence, in particular neural intelligence.

[0006] Some of these neural approaches can be seen as simple extensions of the concepts of compression technique competition mentioned above, such as prediction mode competition and video coding transformation.

[0007] Another approach uses the "autoencoder" concept. An autoencoder is an artificial neural network-based learning algorithm that allows for the construction of new representations of a dataset. The autoencoder architecture consists of two parts: an encoder and a decoder. The encoder consists of a set of layers of neurons that process data to construct new representations, called "encoded" representations, also known as "latent representations." The decoder's neural layers then receive these representations and filter them to attempt to reconstruct the original data. The difference between the reconstructed data and the initial data allows for measurement of any errors introduced by the autoencoder. Training involves modifying the autoencoder's parameters to reduce the measurement error, measured on various samples of the dataset. The performance capabilities of such autoencoder-based systems come at the expense of a significant increase in memory footprint and complexity compared to traditional approaches, such as those proposed by compression standards. They can have millions of parameters and require up to a million MAC (multiply-accumulate) operations to decode a single pixel. This makes such decoders significantly more complex than traditional decoders and hinders the adoption of learning-based compression.

[0008] Recently, a simple neural network-based coding technique was described in (Non-Patent Document 1). The proposed coding technique involves adjusting a neural network to an image and quantizing and transmitting the network weights. When decoding, the neural network is evaluated at each pixel location to reconstruct the image. Nevertheless, such techniques remain inefficient in terms of compression. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] “Compression with Implicit Neural representations” by Emilien Dupont et al.,(arXiv:2103.03123) Summary of the Invention [Problem to be solved by the invention]

[0010] Therefore, there is a need for a solution for simple and efficient encoding / compression of an image or a sequence of images. [Means for solving the problem]

[0011] The present invention relates to an encoding method according to claim 1 and a decoding method according to claim 8.

[0012] Within the meaning of the present invention, the terms "decoding" or "encoding" are understood to mean operations that involve representing a set of samples in a compact form, for example carried by a digital bitstream. Decoding is understood to mean operations that involve processing a digital bitstream in order to render decoded samples.

[0013] By a "sample" of a signal, what is meant is a value sampled from the signal. Sampling a signal produces a series of discrete values ​​called samples. In the case of an image signal, the samples are called pixels and may be, for example, color pixels, conventionally represented by a triplet of values, e.g., (R,G,B) or (Y,U,V). Generally, a sample is represented by one or more components or vectors of data.

[0014] By "a signal containing a plurality of samples" is meant a signal that contains one dimension (audio, sound), two dimensions (image), or three or more dimensions (stereoscopic images, multi-view images, images associated with depth maps, video, etc.). Depending on the number of dimensions, a sample may have one, two or several coordinates in the signal. In the case of an image signal, the location of a sample is identified by an x-coordinate and a y-coordinate.

[0015] The term "feature map" is understood to mean an abstract representation of a signal that contains a plurality of potentially discrete variable data, called for example values, real numbers or integers. As is known, these maps are also called "latent representations" of the input signal.

[0016] By "data feature vector constructed from feature maps as a function of position" is meant a vector consisting of one or more, preferably discrete, elements or data, constructed from feature maps at positions determined by the position of the sample being processed in the signal. This feature vector is one input to the neural network. For example, in the case of a one-dimensional audio signal, such a vector can be constructed from multiple values ​​sampled in each feature map at the same coordinates as the sample being encoded. In the case of an image, such a vector can be constructed from multiple values ​​in each feature sample at the same x and y coordinates as the sample being encoded. It can also be constructed from multiple values ​​sampled in each feature map at positions that depend not only on the sample's position but also on the size of the feature map. Once sampled from the feature maps, these values ​​can be processed, for example by quantization, filtering, interpolation, etc., before being input to the neural network to form the feature vector.

[0017] By "neural network" is meant a neural network such as a convolutional neural network, a multi-layer perceptron, an LSTM (an acronym for long short-term memory), etc. A neural network is defined, for example, by multiple layers of artificial neurons containing a set of activation, weighting, and summation functions (e.g., one layer may compute y=f(Ax×b), where y and b are vectors of dimension N, x is a vector of dimension M, A is a matrix of dimension M×N, and f is the activation function). Hereinafter, references will be made to "neural network" or "artificial neural network."

[0018] By "parameter of a neural network" is meant one of the values ​​that characterize a neural network, such as a weight (filter coefficient, weighting, bias, a value that influences the operation of a nonlinearity) associated with one of the neurons.

[0019] By "processing by a neural network" is meant applying a function represented by a neural network to an input feature vector to generate an output vector representing the samples to be encoded (or decoded), which may include one or more pieces of data representing the samples.

[0020] By "performance measure" is meant a measure between at least one value of a sample to be coded and the decoded value of said sample. The measure can, for example, evaluate distortion or perceptual error. The measure can be performed on one sample or on multiple samples (e.g., the entire image). The measure can also include a measure of bit rate, in particular the bit rate associated with the coding of the neural network and / or the coding of the feature maps. The measure can also be a joint measure of bit rate and distortion achieved by weighting. As is known in the art, the value of this measure is generally minimized until a target value is reached.

[0021] By "construction step" is meant a step aimed at constructing parameters that represent the image prior to the actual encoding. The construction sub-step can be repeated as many times as necessary to obtain an acceptable performance measure.

[0022] In general, the steps of the encoding or decoding methods should not be interpreted as relating to the notion of temporal succession, in other words the steps may be performed in a different order than indicated in the encoding or decoding independent claims or even simultaneously.

[0023] The coding method according to the invention constructs coding parameters from an input signal, e.g., an image, by training a neural network with feature vectors associated with the positions of the samples to be coded. These feature vectors are constructed from feature maps. During training or construction, the parameters of the neural network and the values ​​of the feature maps are updated as a function of a performance measure, e.g., of the bit-rate distortion type. Once the training is finished, i.e., if the obtained performance measure is satisfactory, the actual coding of the parameters of the neural network and / or the values ​​of the feature maps is performed and stored or transmitted to a decoder.

[0024] Advantageously, the training process makes it possible to refine the neural network parameters and / or the values ​​of the feature maps until a suitable representation is obtained in terms of performance (e.g., until a desired balance is achieved between the rate at which it is generated and the distortion suffered by the input signal). The training of the feature map values ​​and the neural network parameters can be carried out jointly. Advantageously, the coding method according to the invention makes it possible to compress signals efficiently.

[0025] Advantageously, the decoding method is simple, as one simply needs to decode the feature maps and neural networks to reconstruct a decoded version of the signal, e.g., an image.

[0026] Such neural networks may advantageously have a very simple structure with a small number of parameters.

[0027] Additionally, the decoding may be performed incrementally, sample by sample.

[0028] According to an embodiment of the encoding and / or decoding method, the method comprises a step of constructing a second group of feature maps, from which a feature vector is also constructed. Advantageously, these additional maps of the second group, constructed identically in the coder and decoder, are neither stored in the coder nor transmitted or decoded in the decoder. They therefore allow additional data to be used to improve compression without reducing the bit rate. These may include, for example, coordinates, data available in the maps of the first group, data relating to other images already processed by the coder or decoder, etc.

[0029] At least one of the feature maps has a lower resolution than the signal to be encoded (respectively decoded). Advantageously, according to this embodiment, at least one of the feature maps contains fewer values ​​than it would if it had the resolution of the signal, so that compression of the feature maps is more efficient. For example, in the case of a digital image, one of the feature maps may have half the resolution, i.e., contain half the number of x and y values ​​as the input signal contains, i.e., contain a total of 1 / 4 the number of values ​​of a feature map with the resolution of the signal.

[0030] At least one of said feature maps has the same resolution as the signal to be encoded (respectively decoded). Advantageously, according to this embodiment, at least one of the feature maps has the same resolution as the input signal (or the signal to be decoded), allowing high fidelity and adherence to the details of the initial resolution of the signal.

[0031] - constructing said feature vector comprises the sub-step of extracting a value from said at least one feature map at the same position as a current sample in the signal.

[0032] Therefore, advantageously, to construct the elements of the feature vector, it is possible to extract values ​​from the feature maps of the first or second group at the same positions as the samples in the signal (input signal in the case of encoding, signal to be decoded in the case of decoding). This method is easy to implement. For example, if J input feature maps with the same resolution as the signal are available, a feature vector of J elements can be directly constructed by simply extracting the values ​​of the maps at the coordinates of the current sample (same abscissa and same coordinate in the feature maps).

[0033] Constructing said feature vector comprises the following sub-steps: extracting a plurality of values ​​from the feature map as a function of the position of the current sample; - processing said extracted values ​​to obtain a feature vector.

[0034] Advantageously, it is possible to process the values ​​extracted from the feature maps in order to construct a feature vector. Such processing can correspond, for example, to quantization, scaling, or filtering of the extracted data. In the case of encoding, if the feature maps contain, for example, float or real values, quantization is essential for proper operation of the system. They need to be quantized before being encoded and / or input to the neural network. In contrast, in decoding, dequantization is not required depending on the embodiment.

[0035] - Encoding (respectively decoding) said first group of feature maps comprises the sub-step of entropy coding (respectively decoding). Advantageously, entropy coding makes it possible to exploit any signal redundancy that may remain in the feature maps, and thus the signal is compressed more efficiently.

[0036] Correspondingly, the invention also relates to an encoding device and a decoding device.

[0037] The features and advantages of the encoding or decoding method apply equally to the encoding or decoding device according to the invention, and vice versa.

[0038] The invention also relates to a computer program on a recording medium, which program is capable of implementing a computer or control device according to the invention, the program comprising instructions designed to implement the corresponding method, which program can use any programming language and can be in the form of source code, object code, an intermediate code between source code and object code, such as a partially compiled form, or any other desired form.

[0039] The present invention also relates to a computer-readable information medium or storage medium containing instructions for the computer program described above. The information or storage medium may be any entity or device capable of storing a program. For example, the medium may include a storage means such as a ROM, e.g., a CD-ROM or a microelectronic circuit ROM, or even a magnetic storage means, e.g., a floppy disk or hard disk, a DNA sequence, or a flash memory. Furthermore, the information or storage medium may be a transmissible medium, such as an electric or optical signal, which may be routed via an electric or optical cable, by a radio wave link, by a wireless optical link, or by other means.

[0040] The program according to the invention can in particular be downloaded via the Internet.

[0041] Alternatively, each information medium or storage medium may be an integrated circuit incorporating a program, the circuit being designed to perform or to be used for performing the method according to the invention.

[0042] Further characteristics and advantages of the invention will become apparent from the following description, with reference to the accompanying drawings, which show exemplary, non-limiting embodiments. [Brief explanation of the drawings]

[0043] [Figure 1] 1 shows a schematic representation of a coding device for use within the scope of the present invention; [Figure 2] 1 shows a schematic representation of a decoding device for use within the scope of the present invention; [Figure 3] 1 shows an example of a synthetic artificial neural network used within the scope of the present invention during encoding and decoding. [Figure 4] 2 is a flowchart illustrating an example of an encoding method that can be implemented by the encoding device of FIG. 1. [Figure 5] 1 shows a diagram of an encoding method used in one embodiment of the present invention; [Figure 6] 4 shows a diagram of an encoding method used in another embodiment of the present invention; [Figure 7] 3 is a flowchart illustrating an example of a decoding method that may be implemented by the decoding device of FIG. 2. [Figure 8] 1 shows a diagram of a decoding method used in one embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0044] FIG. 1 shows diagrammatically the encoding device ENC.

[0045] This encoding device ENC comprises a module GEN for generating feature maps, a processing and quantization module TT, a data extraction module XTR, a module MLP corresponding to the artificial neural network, a module NNC for encoding the neural network, a module FMC for encoding the feature maps, a module EVAL for evaluating a performance function, an update module MAJ and an optional module FME for generating additional feature maps.

[0046] The encoding device ENC may be implemented by an electronic device including a processor and a memory (not shown), each of said modules then being realized through the interaction of the processor with computer program instructions stored in said memory and designed, when executed by the processor, to carry out the functions of the module in question, in particular as described below.

[0047] The encoding device ENC of Figure 1 takes as input P n A sequence of samples to be coded, e.g., a time sequence of sound samples or I(P n In the second case, the image signal I(P n ) can represent a two-dimensional image or multiple two-dimensional images (video, color components, stereo components, multi-view components, etc.). n denotes one sample n of an input signal containing N samples. In one embodiment, the signal is a color image signal represented by at least one two-dimensional representation, such as a matrix of pixels of width W and height H (N=W×H), where each pixel has one red component (R), one green component (G), and one blue component (B), or, as a variant, one lightness component (Y) and at least one saturation component. The location of each pixel is defined by its x- and y-coordinates (x and y) in the image. In one embodiment, the image is grayscale and is represented by a two-dimensional representation, such as a matrix of pixels, where each pixel has a grayscale or lightness component. In this case, the vector representing the pixel is simplified to a single component or datum.

[0048] As will be explained in more detail below with reference to FIGS. 3 to 8, the module GEN for generating feature maps comprises FM i The optional module FME is configured to generate, in particular initialize, a plurality of M feature maps, denoted as FME . The FME is not encoded or transmitted. l One or more additional maps (L), denoted as

[0049] The module XTR receives the sample P to be encoded. n , as a function of its coordinate in the input signal, the feature map FM i and optionally FME l Extract the value at

[0050] The module TT processes the extracted values ​​to obtain the vector Z n Generate Z n is a J-tuple, i.e., a set of J elements or data z i Contains the vector Z of index n n is the pixel P' n Refers to the feature vector of

[0051] Module TT, M Map FM i From this set of data or from these data, a vector Z n quantizes the data extracted from Q. To prepare it for further encoding (e.g., entropy coding), the encoding system quantizes the latent representation of the input data using a quantizer Q to generate an ordered collection of quantized values. Note that quantizing a value refers to matching this value with some discrete set of possible code symbols. For example, the set of possible code symbols may consist of integer values, and the quantization system simply rounds the actual value to an integer value. According to another example, quantization involves multiplying by a given value and then rounding.

[0052] A modular MLP has K parameters W k and as input, a vector Z n or J tuples and, as output, P samples to be encoded nAccording to one embodiment, the neural network is an MLP or multi-layer perceptron consisting of an input layer adapted to the input format (J-tuple), optionally one or more hidden layers, and an output layer adapted to the output format of an output vector, generally a vector containing A elements. According to one embodiment, A is equal to 3 and the output vector is the encoded and then decoded pixel P' n is an (R,G,B) triplet.

[0053] The module NNC calculates the neural network, in particular its parameters W k During the coding training or construction process, i.e. as long as the step of evaluating the performance remains unsatisfactory, the module NNC simulates the coding, followed by the decoding, the results of which are sent to the evaluation module. The evaluation module updates the values ​​of the parameters as a function of the results of the performance measurements carried out by the module EVAL. Subsequently, the parameters W of the neural network k The actual encoding of Wc is performed. k As is known, the coding simulation may be identical to or may approximate the actual coding.

[0054] The module FMC calculates the maps FMi, i.e. the quantized values ​​of each map of the first group (map FME l During the coding training or construction process, i.e. as long as the step of evaluating the performance remains unsatisfactory, the module FMC simulates the coding, followed by the decoding, the results of which are sent to the evaluation module. i The encoded map is then used to perform the actual encoding of the values ​​of FMc i As is known, the encoding simulation may be identical to or approximate the actual encoding. The encoding module compresses the quantized data using, for example, entropy coding.

[0055] The module EVAL performs the evaluation and minimization of the coding performance. The evaluation function is, for example, of the bit rate distortion type. The minimization can be performed via gradient descent or any other method within the capabilities of a person skilled in the art.

[0056] The module MAJ updates the coded FMi map values ​​as a function of the results of the performance function. Optionally, an additional map FME l The value of can be updated.

[0057] FIG. 2 shows diagrammatically the decoding device DEC.

[0058] The coding device DEC in Fig. 2 takes as input M feature maps FMc i The first group of coded data organized in layers (also called layers FM) and the coded parameters Wc of the neural network. k and receive.

[0059] This decoding device DEC comprises a module NND for decoding the neural network, a module FMD for decoding the feature maps, a data extraction module XTR′, an inverse processing module TT′, a module MLP′ corresponding to the neural network and an optional module FME′ for generating additional feature maps.

[0060] The M maps decoded by the module FMD are i The parameters decoded by the module NND are denoted as Wd k It is shown as follows.

[0061] The decoder module FME' is L, FME' l and the additional map FME generated by the encoder. l It is also possible to generate one or more additional maps that are identical to

[0062] The module XTR' is identical to the module XTR in Fig. 1. The module XTR' receives M decoded feature maps FMd i and optionally L additional maps FME' l Extract the value from the decoded sample P n , a vector Z in the form of a J-tuple as a function of the coordinates in the signal to be decoded. n In one embodiment, J=M. In one embodiment, J=M+L.

[0063] Module TT′ optionally performs an inverse quantization corresponding to the quantization performed in the encoder. If the quantizer Q of the encoder simply rounds the actual values ​​received, then inverse quantization is not necessary. If the neural network is able to take into account the quantization of its input data, then inverse quantization is not necessary. In other cases, the decoder performs the inverse operation of quantizer Q. Furthermore, like module TT of the encoder, module TT′ optionally processes the values ​​extracted from the feature maps to generate a decoded vector Zd n Generate.

[0064] The module MLP' has K parameters Wd k The sample P to be decoded is defined by n To generate as output a second vector, typically a vector with A elements, representing n or J-tuples. According to one embodiment, A=3 and the output vector is the decoded pixel Pd n The (R, G, B) triplet of the MLP module has the same structure as the MLP module, and its parameters are the parameters W k are identical if the encoding is lossless, or different if the encoding is lossy.

[0065] All samples of the signal P n is decoded, the reconstructed signal I(Pd n) is obtained, and according to one example, N vectors Pd n An image I is obtained that contains N decoded samples, which are of the form:

[0066] The decoding device DEC may be implemented by an electronic device including a processor and a memory (not shown), each of said modules then being realized through the interaction of the processor with computer program instructions stored in said memory and designed, when executed by the processor, to carry out the functions of the module in question, in particular as described below.

[0067] FIG. 3 illustrates an example of a synthetic artificial neural network used for encoding and decoding in accordance with an embodiment of the present invention.

[0068] The composite artificial neural network MLP used for encoding and the composite artificial neural network MLP' used for decoding are defined by the same structure, e.g., including multiple layers of artificial neurons, and a set of weights and activation functions associated with each of the artificial neurons of the network in question.

[0069] A vector representation of the current sample (feature map FM i and FME l or FMd i and FME' l Vector Z obtained from n or Zd n ) is applied to the input (i.e., input layer) of the composite artificial neural network MLP or MLP'. The artificial neural network outputs as output a vector P' representing the decoded samples. n or Pd n According to one embodiment, the constituent color components (R, G, B) of a color pixel of an image are generated.

[0070] Concatenating all these reconstructed pixels into a (2D, 3D) image forms the decoded or reconstructed image.

[0071] In the coder, a synthetic artificial neural network (MLP) optimizes the current image I(P n ) and its output representation I(P' n ) is trained on images to minimize the difference between the images. The module EVAL performs performance measurements in this sense.

[0072] Once trained, the network's parameters are coded either reversibly (the neural network MLP' is identical to the MLP) or irreversibly (the network MLP' can be slightly different from the MLP).

[0073] FIG. 4 is a flowchart illustrating an example of an encoding method that may be implemented by the encoding device of FIG.

[0074] According to this embodiment, the signal is a two-dimensional image, and therefore each sample to be coded has coordinates (x n ,y n ) pixel P n is.

[0075] The encoding occurs in two main phases.

[0076] In the first phase, called the construction phase, the input signal I(P n ) to optimize the overall cost function. i and parameter W k Training is performed to determine the values ​​of the parameters of the neural network MLP and the feature map FM. Training is performed, for example, via gradient descent, and then i This is followed by updating the parameters of the values ​​of . As known in the art, the cost function can be of bitrate distortion type, or of bitrate, distortion or perceptual type. To measure the bitrate R, the map FM i The encoding of the parameter W must be simulated and then the associated encoding bitrate (size of stream B1) must be measured. kThe encodings of W are not simulated because their influence is smaller than that of the feature maps. k is also simulated and the associated bit rate (size of stream B2) is measured. In order to measure the distortion D, it is necessary to simulate the encoding and then decoding of at least one part of image I, so that at least one pixel P' resulting from the encoding and then decoding simulation is measured. n Then, the input image I(P n ) and the encoded and then decoded image I(P' n ) and the corresponding part of

[0077] Then, during the second phase, called the encoding phase, the map FM i and parameter W k is encoded before being transmitted or stored, and the encoded value FMc i and Wc k They generate the input signal I(P n ) to form a compressed representation of

[0078] The steps of the method according to one embodiment of the present invention will now be described.

[0079] In step E20, a number N of samples P n The signal I(P n ) is sent as input to the method.

[0080] In step E21, a first group of M maps FM i is initialized by the module GEN. Next, the neural network parameters W k and Map FM i The value of must be optimized during the construction phase.

[0081] According to one embodiment, the map FM i is the input signal I(P n), and therefore each has the same resolution as the encoded sample P n Contains as many values ​​N as there are

[0082] According to one embodiment, the map FM i is the input signal I(P n ) or less, and therefore at least one of them contains N' coded values, the number of which is less than N.

[0083] According to one embodiment, the first map FM i has the resolution of the signal, with each subsequent map having half the resolution of the previous map.

[0084] According to one embodiment, a plurality of maps FM i has the same resolution, which is less than the resolution of the signal.

[0085] According to one embodiment, the map FM i is initialized to a predetermined constant value.

[0086] According to another embodiment, the feature map is initialized with a set of random real numbers.

[0087] According to one embodiment, one or more maps FME forming a second group of L additional feature maps l are generated and added to the first group. They are used to construct the feature vector but are not stored or transmitted.

[0088] This feature map FM of the first group i is subsequently updated or refined in step E22 by updating the coder's module MAJ during the training of the construction phase.

[0089] In step E23, the maps FMi of the first group are coded by the coder module FMC. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is the actual coding, the coded values ​​forming the stream B1. The simulation may be identical to the actual coding, but may also be different (for example simplified). This coding may use any known technique aimed at compressing the values ​​of the maps.

[0090] In one embodiment, the map FM i are coded in the order (FM1, FM2, ..., FM4), with the variables of each map coded in a predetermined order, for example lexicographical order. Each map is then subjected to entropy coding, which generates a compressed stream B1, the bit rate of which is subsequently measured during step E28.

[0091] During step E24, map FM i and optionally FME l The value from the input signal sample P is extracted by the module XTR. n coordinates (x n ,y n ) is executed as a function of

[0092] According to one embodiment, the coordinate values ​​(x n ,y n ) is extracted.

[0093] According to one embodiment, the coordinate values ​​are scaled to extract the lower resolution layers.

[0094] According to one embodiment, values ​​close to said value are also extracted for further processing in step E25.

[0095] During step E25, vector Z n is the coordinate of the input signal (x n ,y n ) for each sample P nAbout Map FM i and optionally constructed by module TT from values ​​extracted from Fme. i The processing may include other operations such as filtering, scaling, application of any function, preferably a monotonic function, etc.

[0096] In one embodiment, Z n is the input map or map FM i (and optionally FME l ) contains as many values ​​as there are in J, where J=M(+L).

[0097] In one embodiment, Z n is the current pixel P as explained with reference to Figure 5. n coordinates (x n ,y n ) Map FM i (and optionally FME l ) values ​​form a J-tuple (z1,z2,...,z J ) Map FM i coordinates (x n ,y n ) is the value in this map FM i or FME l Vector Z related to n The final value of element i of (z i ) In this case, the module TT does not apply any processing other than quantization.

[0098] In one embodiment, Z n is a map FM in coordinates that may vary depending on the map, as will be explained with reference to FIG. i (and optionally FME l ) is a J-tuple constructed from values ​​sampled from the map FM i (and / or FME l ) have different resolutions due to being downsampled, the coordinates are adapted (by scaling) to fit the resolution of each map.

[0099] In one embodiment, Z n is a method for generating a map FM by applying a process to one or more values ​​of the map, for example, by filtering neighboring values ​​of a targeted value in the map. i (and FME l ) is a J-tuple constructed from values ​​sampled from the input signal. For example, a map FM i In this map FM i or FME l Vector Z related to n The final value of element i of (z i To get the coordinates (x n ,y n ), (x n -1,y n ), (x n ,y n -1) and (x n -1,y n −1) and process these values ​​(filter, average, interpolate, etc.). According to another example, it is possible to map FM i So, this map FM i or FME l Vector Z related to n The final value of element i of (z i To get the coordinates (x n / 2,y n / 2), (x n / 2-1,y n / 2), (x n / 2,y n / 2-1) and (x n / 2-1,y n / 2-1) and can process these values ​​(filter, average, interpolate, etc.).

[0100] The samples to be coded are processed sequentially, for example from n=1 to n=N.

[0101] In step E26, the vector Z nis processed by the neural network MLP and the output is the encoded sample P n , according to one embodiment, a vector representing the sample P′ n (encoded and then decoded sample P n ) to generate an (R,G,B) triplet.

[0102] Neural network structure and parameters W k are initialized, for example, during the first iteration of this step. These parameters are then updated or refined during subsequent iterations of the method, during the construction phase.

[0103] According to one embodiment, the parameters of the neural network are initialized with predetermined values ​​that are known to give satisfactory results (eg, after training on a corpus of images).

[0104] According to another embodiment, the parameter W of the neural network k is initialized with a set of random values.

[0105] During step E27, the parameters W of the neural network MLP are k is quantized and coded. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is the actual coding, and the coded values ​​form the stream B2. The simulation may be identical to the actual coding, but it may also be different (for example simplified). For this, any known technique can be used, for example the neural network coding standard proposed in part 17 of MPEG-7, also called neural network representation or NNR. In this case, the coding is performed using a parameter W k Note that the amount of reduction provided to needs to be selected.

[0106] In step E28, the performance measures are evaluated.

[0107] For this purpose, the coding simulation rate associated with the feature map (map F i ) and optionally a coding simulation rate associated with the parameters of the neural network (parameters W k The measured values ​​are a simulation of stream B2 by encoding and decoding the stream B1.

[0108] According to one embodiment, the cost function is of the rate-distortion type denoted (D+L*R), where D is for example the squared error measured between the input signal and the decoded signal (or the error measured on a subset of the signal's samples). According to another example, D is calculated from a perceptual function such as SSIM (Structural Similarity) or MSSSIM (Multi-Scale Structural Similarity). According to one embodiment, R is the simulated bitrate of stream B1, and according to another embodiment, R is the overall bitrate used to code this image, i.e. the sum of the simulated bitrates of B1 and B2. L is a parameter that adjusts the rate-distortion compromise. Other cost functions are possible.

[0109] The performance measure remains unsatisfactory until the cost function reaches a minimum and the method is repeated from step E22. This minimization can be performed via known mechanisms such as gradient descent, and the parameters are updated in step E22 for the feature map values ​​and in step E26 for the network parameters.

[0110] In step E29, training stops when the cost function reaches a minimum. k ) and feature map (FM i ) is available, streams B1 and B2 can be formed from it. According to another embodiment, the updated parameters of the neural network (W k ) and feature map (FM iThe actual encoding of the values ​​of Wc ) is performed in this step to generate the encoded parameters Wc forming the streams B1 and B2. k and FMc i Generate.

[0111] Streams B1 and B2 can be concatenated to generate the final stream BS. According to one embodiment, stream B2 of coded parameters of the neural network is stored or transmitted before stream B1 so that it can be decoded before stream B1.

[0112] Figure 5 shows the image I(P n 1 illustrates an encoding method used in one embodiment of the present invention to encode a .times. ...

[0113] In this embodiment, the map FM i The number of is 4. In a preferred embodiment, the number is 7. Map FM i is the image I(P n ), and therefore each contains W×H values, where W represents the width of the image in pixels and H represents its height (N=W×H).

[0114] In one variation, shown by the dotted lines, there are five feature maps, and an additional map, FME0, is introduced, which is not stored, coded, or transmitted. This additional map typically contains data that may assist the network MLP in the task of reconstructing the signal. Thus, the added map may be one or more maps from the following non-limiting list: - a map containing, for each point, the x coordinate of this point, - a map containing, for each point, the y coordinate of this point, - a map containing the location coding for each point (for example, as explained at the following internet address: https: / / skosmos.loterre.fr / P66 / fr / page / -K0D65X2X-X), - a map representing an image that is distinct from the image being processed and that is able to provide information about the image to be coded (for example a previously processed image, if the current image is part of a sequence of images to be coded, such as a video, a set of medical images, a multi-view representation, etc.), - a map representing the feature maps of an image (e.g. a previously processed image, if the current image is part of a sequence of images to be coded, such as a video, a set of medical images, a multi-view representation, etc.) that is distinct from the image being processed and that is able to provide information about the image to be coded, - Already decoded samples of the same map, e.g. a map containing the values ​​of the previous samples in decoding order.

[0115] In this embodiment, the vector Z n is the current pixel P n coordinates (x n ,y n ) Map FM i The extracted values ​​are quantized by the module TT, and the vector Z formed from the quantized values ​​is n is processed by the neural network MLP and outputs as output a second vector, in this example the samples P to be coded. n The output vector, in this case, is the encoded and then decoded pixel P' n The triplet is the (R,G,B) triplet of the color components (R',G',B') at the position (x n ,y n ) the decoded image I(P' n )

[0116] According to one embodiment, four maps FM i is quantized.

[0117] According to the deformation shown as a dashed line, the vector Z n is a 5-tuple (z0···z4), where the value z0 is extracted from the additional map FME0.

[0118] FIG. 6 shows a diagram of an encoding method used in another embodiment of the present invention.

[0119] In this embodiment, the map FM i The number of variables in the feature maps is four. As in the previous embodiment, the first map FM1 has the same resolution as image I and therefore contains W × H values, where W represents the image's width in pixels and H represents its height. The second map FM2 has half the resolution (in each dimension) of map FM1. Each additional map has half the resolution of the previous map. This structure allows for a lower number of variables in the feature maps, thus facilitating encoding and training while minimizing encoding costs.

[0120] In this embodiment, the 4-tuple Z n is the current pixel P n (x n ,y n ) coordinates (x n ,y n ), (x n / 2,y n / 2), (x n / 4,y n / 4), (x n / 8,y n / 8) Map FM i These extracted values ​​are processed and quantized to form a 4-tuple Z n According to one embodiment, for the map FM2, the coordinates (x n / 2,y n / 2) is extracted and the vector Z associated with this map FM2 n According to another embodiment, for the map FM2, the coordinates (x n / 2,y n / 2), (x n / 2-1,y n / 2), (x n / 2,y n / 2-1), (x n / 2-1,y n / 2-1) are extracted and filtered to form the vector Z associated with this map FM2. n The oversampling is thus performed. The filtering or oversampling may for example use bicubic filtering or Lanczos filtering, which are known per se. A more general process may also be used, using a neural network, for example of the MLP type, to filter the coordinates (x n / 2,y n / 2), (x n / 2-1,y n / 2), (x n / 2,y n / 2-1), (x n / 2-1,y n The final value (z2) may be generated by weighting the values ​​in (z2 / 2-1).

[0121] As shown above, the vector Z n is processed by the neural network MLP to produce as output a second vector, here the samples P n The output vector in this case is the encoded and then decoded pixel P' n The (R,G,B) triplet is the (R,G,B) triplet of the image I(p' n ) color components (R', G', B') position (x n ,y n ) is re-injected into the decoded image.

[0122] Other types of structures are possible, for example, a drop-off ratio between maps different from 1 / 2 (such as 1 / 4 or 1 / 3) may be used, in which case an oversampling step is adapted to generate the final value of the vector Zn.

[0123] As explained with reference to Figure 5, other maps FME l may be introduced, which corresponds to the vector Z n is used to generate the , but is not stored or transmitted.

[0124] FIG. 7 is a flowchart illustrating an example of a decoding method that may be performed by the decoding device of FIG.

[0125] During a step E30, the streams B1 and B2 are extracted from the coded stream BS. The streams B1 and B2 are coded representations of the maps FMc of the first group. i and the coded representation of the parameters Wc k Each includes:

[0126] During step E31, M maps FMd i But the value FMc i This decoding can use any known technique, preferably entropy decoding, similar to that used in the encoder. In one embodiment, the map FMd i are decoded in the order of (FMd1, FMd2, . . . , FMd4), and the variables of each map are decoded in a predetermined order, for example, lexicographic order.

[0127] According to one embodiment of the encoder as described, the map FMd i is the signal to be reconstructed I(Pd n ), i.e. it contains N values.

[0128] According to an embodiment equivalent to that described for the coder, the map FMd i is the signal to be reconstructed I(Pd n ) has a resolution of

[0129] According to an embodiment equivalent to that described for the coder, a plurality of maps FMd i has the same resolution, which is less than the resolution of the signal.

[0130] During step E32, according to one embodiment, one or more maps FME' forming a second group of L additional feature maps are selected. lis generated and added to the first group. Map FME' l are not decoded, but are generated by the decoder in the same way as they are generated in the encoder. They may typically assist the network MLP' in the task of reconstructing the signal. The non-limiting list of possible additional feature maps described with reference to FIG. 5 for the encoder also applies in this case.

[0131] During step E33, the value is i and FME' where possible l This extraction is carried out by the module XTR', which extracts the samples P n The coordinates of the input signal (x n ,y n ) This step is identical to step E24 described for the encoder with reference to FIG. 4 and applies to the described embodiment.

[0132] The samples to be decoded are processed sequentially, for example from n=1 to n=N.

[0133] During step E34, the vector Zd n But the coordinate (x n ,y n ) decoded sample P n About Map FMd i and FME' where possible l The processing is carried out by the module TT' from the values ​​extracted from the vector Zd n , which may include inverse quantization of the quantized image. The processing may also include other operations such as filtering, scaling, etc. This step is identical to step E25 described for the encoder with reference to Figure 4, and the described embodiment applies. However, inverse quantization is not mandatory. Step E34 and module TT' are therefore optional in this particular case.

[0134] At step E35, the parameters Wd of the neural network MLP' are calculated.k However, the value Wc of stream B2 k For this purpose, any known decoding technique corresponding to the encoding technique used by the coder can be used. The neural network MLP' is similar to the network MLP, i.e., it has the same structure and the same parameters regardless of the encoding, which may be lossy or lossless.

[0135] According to one embodiment, stream B2 is decoded before stream B1 in order to acquire the neural network before starting to decode the samples.

[0136] During step E36, vector Zd n is processed by the neural network MLP' and the output is the decoded sample Pd n , according to one embodiment, the position (x n ,y n ) the decoded image I(Pd n ) This step is identical to step E26 described for the coder with reference to FIG.

[0137] If all samples of a signal are processed, for example, the image I(Pd n ) is obtained.

[0138] Figure 8 shows the image I(Pd n 1 illustrates a decoding method used in one embodiment of the present invention to decode

[0139] In this embodiment, the map FMd i The number is 4. In a preferred embodiment, the number is 7.

[0140] In this embodiment, the map FMd i has the same resolution as the image to be decoded and therefore contains W×H variables, where W represents the width of the image in pixels and H represents its height.

[0141] In this embodiment, the vector Zd n is the current pixel Pd n coordinates (x n ,y n ) Map FMd i The vector Zd is a 4-tuple (z1···z4) formed from the values ​​of n is optionally dequantized and then processed by the neural network MLP′ to produce as output a second vector, according to the example, of decoded samples Pd n The (R,G,B) triplet is the coordinate (x n ,y n ), the decoded image I(Pd n )

[0142] According to one variant, shown as a dashed line, the number of maps is 5 and an additional map FME'0 is introduced. In this embodiment, the vector Zd n is a 5-tuple.

Claims

1. The number of samples to be coded (P ｎ ) containing the signal (I(P ｎ )) a method for encoding a signal comprising the steps of: A construction step comprising: Feature Map (FM ｉ a substep (E21, E22) of constructing a first group of Position (x ｎ , y ｎ ) of the signal to be coded, which is associated with the current sample (P ｎ ) for at least one sample, called The current sample (P ｎ ) at the position (x ｎ , y ｎ ) of the first group as a function of the feature map (FM ｉ ) to the feature vector (Z ｎ ) (E24, E25), The decoded value of the current sample (P' ｎ ) to provide a vector representing the parameters (W ｋ ) is used to generate the feature vector (Z ｎ a substep (E25) of processing a substep (E22, E26) of updating at least one value of one of the feature maps of the first group and / or at least one parameter of the network as a function of the coding performance measure; a construction step, The feature map (FM ｉ ) and the first group of parameters (W ｋ ) (E23, E27, E29) 1. A method for encoding a signal, comprising:

2. Feature Map (FME) ｌ 2. The method for encoding a signal according to claim 1, characterized in that it comprises a step (E21) of constructing a second group of feature maps of the first signal and the second signal, and that the feature vector is further constructed from the feature maps of the second group.

3. The feature map (FM ｉ , FME ｌ 3. A method for encoding a signal according to claim 1 or 2, characterized in that at least one of the first and second inputs is of lower resolution than the signal to be encoded.

4. The feature map (FM ｉ , FME ｌ 4. A method for encoding a signal according to claim 1, wherein at least one of the first and second inputs has the same resolution as the signal to be encoded.

5. The feature vector (Z ｎ ) is constructed by constructing the current sample (P ｎ ) and the same position (x ｎ , y ｎ ) in the at least one feature map (FM ｉ , FME ｌ 5. A method for encoding a signal according to claim 1, further comprising the substep of extracting a value from the signal.

6. The feature vector (Z ｎ ) is The current sample (P ｎ ) at the position (x ｎ , y ｎ ) as a function of the feature map (FM ｉ , FME ｌ ) and a substep to extract multiple values ​​from it. processing the extracted values ​​to obtain the feature vector; A method for encoding a signal according to any one of claims 1 to 5, characterized in that it comprises:

7. Method for coding a signal according to any one of claims 1 to 6, characterized in that the coding of the first group of feature maps comprises a sub-step of entropy coding.

8. The number of samples to be decoded (Pd ｎ 1. A method for decoding a signal comprising: Feature map (FMd ｉ a step (E31) of decoding a first group of The parameter (Wd ｋ ) and Position (x ｎ , y ｎ ) of the current sample (Pd ｎ ) for at least one sample, called The position (x ｎ , y ｎ ) as a function of the feature map (FMd ｉ ) to the feature vector (Zd ｎ ) (E33, E34), The current sample (Pd ｎ ) to provide a vector representing the decoded values ​​of the decoded parameters (Wd ｋ ) is used to calculate the feature vector (Zd ｎ ) processing step (E36); 10. A method for decoding a signal, comprising:

9. Feature map (FME' ｌ 9. The decoding method according to claim 8, characterized in that it comprises a step (E32) of constructing a second group of feature maps of the first group, and the feature vector is further constructed from the feature maps of the second group.

10. The feature map (FMd ｉ , FME' ｌ 10. A decoding method according to claim 8 or 9, characterized in that at least one of the signals has a lower resolution than the signal to be decoded.

11. The feature map (FMd ｉ , FME' ｌ 11. A method for decoding a signal according to any one of claims 8 to 10, characterized in that at least one of the first and second inputs has the same resolution as the signal to be decoded.

12. The feature vector (Zd ｎ ) is constructed by constructing the current sample (Pd ｎ ) and the same position (x ｎ , y ｎ ) in the at least one feature map (FMd ｉ , FME' ｌ 12. A method for decoding a signal according to any one of claims 8 to 11, characterized in that it comprises the substep of extracting a value from

13. The feature vector (Zd ｎ ) is The current sample (Pd ｎ ) at the position (x ｎ , y ｎ extracting a plurality of values ​​from said feature map as a function of processing the extracted values ​​to obtain the feature vector. A method for decoding a signal according to any one of claims 8 to 12, characterized in that it comprises:

14. Method for decoding a signal according to any one of claims 8 to 13, characterized in that the decoding (E31) of the first group of feature maps comprises a sub-step of entropy coding.

15. The number of samples to be coded (P ｎ ) containing the signal (I(P ｎ )) a device for encoding a signal, the device comprising: Feature Map (FM ｉ ) to construct a first group (GEN, MAJ), Position (x ｎ , y ｎ ) of the signal to be coded, which is associated with the current sample (P ｎ ) for at least one sample, called The current sample (P ｎ ) at the position (x ｎ , y ｎ ) of the first group as a function of the feature map (FM ｉ ) to the feature vector (Z ｎ ) (XTR, TT) and The decoded value of the current sample (P' ｎ ) to provide a vector representing the parameters (W ｋ ) is used to generate the feature vector (Z ｎ ) (MLP), updating at least one value of one of the feature maps of the first group and / or at least one parameter of the network (MAJ, NNC) as a function of the coding performance measure; The feature map (FM ｉ ) and the first group of parameters (W ｋ ) to encode the set (FMC, NNC) 1. A device for encoding a signal, configured to:

16. The number of samples to be decoded (Pd ｎ 1. A device for decoding a signal comprising: Feature map (FMd ｉ ) decoding a first group (FMD); The parameter (Wd ｋ ) (NND), and Position (x ｎ , y ｎ ) of the current sample (Pd ｎ ) for at least one sample, called The position (x ｎ , y ｎ ) as a function of the first group (FMd ｉ ) from the feature map of the feature vector (Zd ｎ ) (XTR', TT'); The current sample (Pd ｎ ) to provide a vector representing the decoded values ​​of the decoded parameters (Wd ｋ ) is used to calculate the feature vector (Zd ｎ ) (MLP'); A device configured to:

17. A computer program comprising instructions for carrying out the steps of the encoding or decoding method according to claim 1 or 8 when said program is executed by a computer.

Images