Method and apparatus for encoding and decoding image

Abstract

The present invention relates to a method and apparatus for encoding and decoding a signal comprising a plurality of sample points. The decoding method comprises the following steps:-decoding (E31, FMD) the first set of feature maps (Fmdi); -decoding (E35, NND) a set of parameters (Wdk) representing the neural network (MLP '); for at least one sample point of the signal to be decoded, referred to as the current sample point (Pdn), associated with the position (xn, yn) in the signal to be decoded:-constructing (E33, E34, XTR ', TT') feature vectors (Zdn) from the feature maps (FMdi) in the first group from the position (xn, yn) of the current sample point; and-processing (E36, MLP ') said feature vector (Zdn) using an artificial neural network (MLP') defined by the decoded parameter (Wdk) in order to provide a vector representative of decoded values of the current sample point (Pdn).

Description

Technical Field

[0001] The present invention relates to the general field of encoding one-dimensional or multi-dimensional signals. More particularly, the present invention relates to compression of digital images and videos. Background Art

[0002] Digital images are typically source-coded to achieve compression, thereby limiting the resources required for their transmission and / or storage. Numerous coding standards exist. For still images, there is the JPEG family of standards, while for moving images or video, there are standards developed by the ITU / MPEG organizations (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) and their extensions (MVC, SVC, 3D-HEVC, etc.).

[0003] An image is typically encoded by dividing it into a number of rectangular blocks and encoding these blocks of pixels in a given processing sequence. In existing video compression techniques, the processing of the blocks typically involves block pixel prediction, which is performed using previously encoded and then decoded pixels present in the image being encoded (in this case, called "intra-frame prediction") or using previously encoded images (in this case, called "inter-frame prediction"). This exploitation of any spatial and / or temporal redundancy avoids the need to transmit or store the pixel values ​​of each pixel block by representing at least some of the blocks using a residual (which represents the difference between the predicted values ​​of the pixels in the block and the actual values ​​of the pixels in the predicted block).

[0004] As video formats continue to evolve to achieve higher compression rates and accommodate a diverse range of expected formats and communication networks, the prediction possibilities continue to grow, and conventional encoding and decoding algorithms become very complex.

[0005] In addition to these conventional methods proposed by compression standards (JPEG, MPEG, ITU), methods based on artificial intelligence (especially neural network intelligence) are also emerging.

[0006] Some of these neural network methods can be viewed as simple extensions of the competition concept in previous compression techniques (such as prediction mode competition, video coding transformation, etc.).

[0007] Other approaches use the concept of "autoencoders." Autoencoders are learning algorithms based on artificial neural networks that allow for the construction of new representations of a dataset. The autoencoder architecture consists of two parts: an encoder and a decoder. The encoder consists of multiple layers of neurons that process the data to construct new representations, called "encoded" representations, also known as "latent representations." The decoder's neural network layers, in turn, receive these representations and filter them to attempt to reconstruct the original data. The difference between the reconstructed and original data allows for the measurement of any error introduced by the autoencoder. Training involves modifying the autoencoder's parameters to reduce the reconstruction error measured at each sample point in the dataset. While these autoencoder-based systems offer superior performance, they come at the cost of significantly increased memory usage and complexity compared to conventional approaches, such as those proposed by compression standards. Such systems can have millions of parameters, and decoding a single pixel can require up to a million MAC (multiply-add) operations. This makes these decoders significantly more complex than conventional decoders, potentially hindering the application of learning-based compression.

[0008] A simple neural network-based encoding technique has recently been described in the article "Compression with Implicit Neural Representations" (arXiv:2103.03123) by Emilien Dupont et al. This encoding technique involves adapting a neural network to an image, quantizing the network's weights, and transmitting the quantized weights. During decoding, the neural network is evaluated at each pixel location to reconstruct the image. However, this technique still suffers from compression inefficiencies.

[0009] Therefore, there is a need for a solution for simply and efficiently encoding / compressing an image or a sequence of images. Summary of the Invention

[0010] The present invention relates to an encoding method as claimed in claim 1 and a decoding method as claimed in claim 8 .

[0011] Within the meaning of the present invention, the term "encoding" is understood to mean an operation involving representing a set of samples in a compact form, such as for transmission via a digital bit stream. Decoding is understood to mean an operation involving processing a digital bit stream to restore decoded samples.

[0012] A "sample" of a signal is a value sampled from the signal. Sampling a signal produces a sequence of discrete values, called samples. For image signals, a sample is called a pixel and can be, for example, a color pixel typically represented by a triplet of values, such as (R, G, B) or (Y, U, V). Typically, a sample is represented by a vector with one or more components or data.

[0013] A "signal comprising multiple samples" is a signal containing one dimension (audio, sound), two dimensions (images), or more than two dimensions (stereoscopic images, multi-view images, images associated with depth maps, video, etc.). Depending on the dimensionality, the samples have one, two, or several coordinates in the signal. For image signals, the location of a sample is identified by its x and y coordinates.

[0014] The term "feature map" should be understood to mean an abstract representation of a signal, comprising a plurality of underlying discrete variable data, also known as values, real numbers or integers. As is well known, these maps are also known as the "latent representation" of the input signal.

[0015] A "feature vector of data constructed from a feature map based on position" refers to a vector consisting of one or more (preferably discrete) elements or data, where the data is constructed from a feature map at a position determined by the position of the sample being processed in the signal. This feature vector is an input to the neural network. For example, in the case of a one-dimensional audio signal, such a vector can be constructed from multiple values ​​sampled from each of the feature maps at the same coordinates as the sample to be encoded. For images, such a vector can be constructed from multiple values ​​sampled from each of the feature maps at the same x and y coordinates as the sample to be encoded. It can also be constructed from multiple values ​​sampled from each of the feature maps at a position that depends not only on the position of the sample but also on the size of the feature map. Once these values ​​are sampled from the feature maps, they can be processed (e.g., through quantization, filtering, interpolation, etc.) to form a feature vector, which is then input to the neural network.

[0016] "Neural network" refers to neural networks such as convolutional neural networks, multilayer perceptrons, and LSTM (long short-term memory). A neural network, for example, is defined by multiple layers of artificial neurons, including a set of activation functions, weighting functions, and summation functions (for example, a layer may compute y = f(Ax + b), where y and b are N-dimensional vectors, x is an M-dimensional vector, A is an M x N-dimensional matrix, and f is the activation function). Below, we will refer to "neural networks" or "synthetic neural networks."

[0017] A “parameter of a neural network” refers to one of the values ​​that characterizes the neural network, for example, a weight associated with one of the neurons (filter coefficient, weight, bias, value affecting nonlinear operation), etc.

[0018] "Processing by a neural network" means applying a function expressed by the neural network to an input feature vector to produce an output vector representing the sample to be encoded (or decoded). The output vector may include one or more data representing the sample.

[0019] A "performance measure" is a measurement between at least one value of a sample to be encoded and the decoded value of that sample. This measurement can assess, for example, distortion or perceptual error. This measurement can be performed for a single sample or for multiple samples (e.g., an entire image). This measurement can also include a measurement of bitrate (particularly the bitrate associated with encoding a neural network and / or encoding a feature map). This measurement can be a joint measure of bitrate and distortion achieved by weighting the bitrate and distortion. As is well known in the art, the value of this measure is typically minimized until a target value is reached.

[0020] The "construction step" is the step where the parameters representing the image are constructed before the actual encoding. The construction sub-step can be repeated as many times as necessary to obtain an acceptable performance measure.

[0021] Generally, the steps of the encoding or decoding method should not be interpreted as being associated with the concept of a chronological order. In other words, these steps can be performed in a different order than indicated in the independent encoding or decoding claims, or even performed simultaneously.

[0022] The encoding method according to the present invention constructs encoding parameters based on an input signal (e.g., an image) by training a neural network on feature vectors associated with the locations of the samples to be encoded. These feature vectors are constructed from feature maps. During training or construction, the neural network parameters and feature map values ​​are updated based on performance measures (e.g., bitrate distortion type). When training is complete (i.e., the performance measures obtained are satisfactory), the actual encoding of the neural network parameters and / or feature map values ​​can be performed and stored or transmitted to a decoder.

[0023] Advantageously, the training process allows the parameters of the neural network and / or the values ​​of the feature maps to be refined until a representation that satisfies performance requirements is achieved (e.g., until a desired balance is achieved between the generated bit rate and the distortion experienced by the input signal). The training of the values ​​of the feature maps and the parameters of the neural network can be performed jointly. Advantageously, the encoding method according to the invention allows for efficient compression of signals.

[0024] Advantageously, the decoding method is simple because only the feature maps and the neural network need to be decoded to reconstruct the decoded version of the signal (e.g., image).

[0025] Advantageously, such a neural network can have a very simple structure, with very few parameters.

[0026] Alternatively, decoding may be performed step-by-step, sample-by-sample.

[0027] According to an embodiment of the encoding and / or decoding method:

[0028] The method includes the steps of constructing a second set of feature maps, and also constructing feature vectors based on these feature maps. Advantageously, these second set of additional maps, constructed identically in the encoder and decoder, are neither stored or transmitted in the encoder nor decoded in the decoder. Thus, these additional maps allow for improved compression using additional data without reducing the bitrate. This additional data may include, for example, coordinates, data available in the maps of the first set, data about other images already processed by the encoder or decoder, etc.

[0029] - at least one of the feature maps has a lower resolution than the resolution of the signal to be encoded (respectively to be decoded). Advantageously, according to this embodiment, compression of the feature maps is more efficient because at least one of the feature maps contains fewer values ​​than would be the case if it had the resolution of the signal. For example, in the case of a digital image, one of the feature maps may have a 1 / 2 resolution, i.e., it contains half the number of x-values ​​and y-values ​​as the number of samples contained in the input signal (i.e., a total of 1 / 4 the number of values ​​of the feature map having the resolution of the signal).

[0030] - at least one of the characteristic maps has the same resolution as the signal to be encoded (respectively to be decoded). Advantageously, according to this embodiment, at least one of the characteristic maps has the same resolution as the input signal (or signal to be decoded), thereby achieving high fidelity and preserving details of the original resolution of the signal.

[0031] - Constructing said feature vector comprises the sub-step of extracting the value of said at least one feature map in the same position as the position of the current sample in the signal.

[0032] Advantageously, the values ​​of the feature maps in the first or second group at the same locations as the samples in the signal (input signal for encoding, signal to be decoded for decoding) can be extracted to construct the elements of the feature vector. This approach is easy to implement. For example, if J input feature maps with the same resolution as the signal are available, simply extracting the values ​​of these maps at the coordinates of the current sample (at the same horizontal and vertical coordinates in the feature maps) allows the direct construction of a feature vector of J elements.

[0033] - The construction of the feature vector includes the following sub-steps:

[0034] - extracting multiple values ​​of a feature map according to the position of the current sample point;

[0035] - processing said extracted values ​​in order to obtain a feature vector.

[0036] Advantageously, the values ​​extracted from the feature map can be processed to construct a feature vector. This processing can correspond to, for example, quantization, scaling, or filtering of the extracted data. For encoding, for example, if the feature map includes floating-point values ​​or real values, quantization is crucial for the correct operation of the system. These feature maps need to be quantized before encoding and / or input to the neural network. In contrast, during decoding, depending on the embodiment, a dequantization operation is not required.

[0037] - Encoding (respectively decoding) said first set of feature maps comprises a sub-step of entropy encoding (respectively decoding). Advantageously, entropy encoding makes it possible to exploit any signal redundancy that may remain in the feature maps, thereby compressing the signal more efficiently.

[0038] Relatedly, the present invention also relates to an encoding device and a decoding device.

[0039] The features and advantages of the encoding or decoding method are also applicable to the encoding or decoding device according to the present invention, and vice versa.

[0040] The present invention also relates to a computer program on a recording medium, which can be implemented in a computer or controller according to the present invention. The program includes instructions designed to implement the corresponding method. The program can use any programming language and can be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled format, or in any other desired format.

[0041] The present invention also relates to a computer-readable information medium or storage medium comprising instructions for the aforementioned computer program. The information or storage medium may be any entity or device capable of storing a program. For example, the medium may include a storage device such as a ROM (e.g., a CD-ROM or a microelectronic circuit ROM), or even a magnetic storage device (e.g., a floppy disk, a hard disk, a DNA sequence, or a flash memory). Furthermore, the information or storage medium may be a transmissible medium such as an electrical or optical signal, which may be routed via an electrical or optical cable, a radio link, a wireless optical link, or other means.

[0042] The program according to the invention can in particular be downloaded via the Internet.

[0043] Alternatively, each information medium or storage medium may be an integrated circuit incorporating the program, the circuit being designed to execute or being used to execute the method according to the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Other characteristics and advantages of the invention will become apparent from the following description given with reference to the accompanying drawings, which show exemplary embodiments of the invention that are in no way limiting.

[0045]

Figure 1

[0046]

Figure 2

[0047]

Figure 3

[0048]

Figure 4

[0049]

Figure 5

[0050]

Figure 6

[0051]

Figure 7

[0052]

Figure 8

[0053] Figure 1 An encoding device ENC is schematically shown.

[0054] The encoding device ENC comprises a module GEN for generating feature maps, a processing and quantization module TT, a data extraction module XTR, a module MLP corresponding to an artificial neural network, a module NNC for encoding the neural network, a module FMC for encoding the feature maps, a module EVAL for evaluating the performance function, an updating module MAJ and an optional module FME for generating additional feature maps.

[0055] The encoding device ENC may be implemented by an electronic device comprising a processor and a memory (not shown); each of the aforementioned modules may then be generated via the interaction of the processor with computer program instructions stored in the aforementioned memory and designed, in particular as described below, to perform the functions of the module in question when these instructions are executed by the processor.

[0056] Figure 1 The encoding device ENC in the encoding device receives a series of samples to be encoded (denoted as P n , for example, a time-continuous sound sample) or a set of image data (denoted as I(P n ))as input. In the second case, the image signal I(P n ) can represent a two-dimensional image, or multiple two-dimensional images (video, color components, stereoscopic components, multi-view components, etc.). n Represents a sample n of an input signal consisting of N samples. In one embodiment, the signal is a color image signal represented by at least one two-dimensional representation (such as a pixel matrix) of width W and height H (where N = W × H), with each pixel having a red component (R), a green component (G), and a blue component (B), or, as a variant, a luma component (Y) and at least one chroma component. The position of each pixel is defined by its x and y coordinates (x and y) in the image. In one embodiment, the image is grayscale and represented by a two-dimensional representation (such as a pixel matrix) with each pixel having either a grayscale component or a luma component. In this case, the vector representing the pixel is reduced to a single component or data.

[0057] As will be referred to below Figures 3 to 8 Described in more detail:

[0058] The module GEN for generating feature maps is configured to generate (especially initialize) multiple (M) feature maps, denoted as FM i The optional module FME can generate one or more additional maps (number L) that are neither encoded nor transmitted and are represented as FME l .

[0059] Module XTR is for the sample point P to be coded n , extract the feature map FM according to its coordinates in the input signal i and optionally FME l The value in .

[0060] Module TT processes the extracted values ​​in order to generate a vector Z n . Z n is a J-tuple, i.e. it contains J elements or data zi . The vector Z with index n n Refers to pixel P' n The eigenvector of .

[0061] Module TT quantifies the M graphs FM from the set i The data extracted from or the vector Z constructed from these data n . In preparation for further encoding (e.g., entropy encoding), the encoding system quantizes the latent representation of the input data using a quantizer Q to generate an ordered set of quantized values. It should be noted that quantization of a value refers to matching that value to a member of a discrete set of possible code symbols. For example, the set of possible code symbols may consist of integer values, and the quantization system simply rounds the actual value to an integer value. According to another example, quantization involves multiplying a given value and then rounding.

[0062] The module MLP consists of K parameters W k The neural network defined here can receive the vector Z n (or J-tuple) as input to generate the sample point P to be encoded n According to one embodiment, the neural network is an MLP or multilayer perceptron consisting of an input layer adapted to the input format (J-tuples), optionally one or more hidden layers, and an output layer adapted to the output format of the output vector (typically a vector containing A elements). According to one embodiment, A is equal to 3, and the output vector is the encoded and then decoded pixel P' n The (R, G, B) triplet.

[0063] The module NNC affects the neural network (especially its parameters W k ) is encoded. During the encoding training or construction process, that is, as long as the step of evaluating the performance is still not satisfactory, the module NNC simulates encoding and then decoding, the results of which are sent to the evaluation module. The evaluation module updates the values ​​of the parameters according to the results of the performance measurements performed by the module EVAL. Subsequently, the parameters W of the neural network are adjusted. k The actual encoding of the coded parameters is represented by Wc k In a known manner, the coding simulation can be identical to or similar to the actual coding.

[0064] The module FMC encodes the graph FMi, i.e., the quantized value of each graph in the first group (excluding the graph FME). l During the encoding training or construction process, i.e. as long as the step of evaluating the performance is still unsatisfactory, the module FMC simulates encoding and then decoding, the result of which is sent to the evaluation module. i The actual encoding of the value of . The encoded graph is represented as FMc iIn a known manner, the encoding simulation can be identical to or similar to the actual encoding. The encoding module compresses the quantized data using, for example, entropy coding.

[0065] The module EVAL performs evaluation and minimization of coding performance. For example, the evaluation function is of the bit rate distortion type. Minimization can be performed via gradient descent or any other method within the capabilities of those skilled in the art.

[0066] The module MAJ updates the values ​​of the FMi map to be encoded based on the results of the performance function. Optionally, it can update additional maps FME l The value of .

[0067] Figure 2 A decoding device DEC is schematically shown.

[0068] Figure 2 The encoding device DEC receives as input a first set of encoded data, which is organized into M feature maps FMc i (also called layer FM) and the encoded parameters Wc of the neural network k .

[0069] The decoding device DEC comprises a module NND for decoding a neural network, a module FMD for decoding a feature map, a data extraction module XTR', an inverse processing module TT', a module MLP' corresponding to the neural network, and an optional module FME' for generating additional feature maps.

[0070] The graph (number M) decoded by the module FMD is denoted as FMd i The parameters decoded by the module NND are denoted as Wd k .

[0071] The decoder module FME' can also generate one or more additional graphs, which are represented by FME' l , and the number is L, with the additional graph FME generated by the encoder l The same number.

[0072] Module XTR' with Figure 1 The module is the same as the module XTR. This module is for the sample point P to be decoded n , extract M decoded feature maps FMd according to their coordinates in the signal to be decoded i and optional L additional maps FME' l to construct a vector Z in the form of a J-tuple n In one embodiment, J = M. In one embodiment, J = M + L.

[0073] The module TT' optionally performs an inverse quantization corresponding to the quantization performed at the encoder. If the encoder's quantizer Q only rounds the actual values ​​it receives, then there is no need to perform inverse quantization. If the neural network is able to take into account the quantization of its input data, then there is no need to perform inverse quantization. Otherwise, the decoder performs the inverse operation of the quantizer Q. Furthermore, similar to the module TT of the encoder, the module TT' optionally processes the values ​​extracted from the feature map to generate the decoded vector Zd n .

[0074] The module MLP' consists of K parameters Wd k The neural network defined here can receive the vector Zd n (or J-tuple) as input to generate a representation of the sample point to be decoded P n The second vector (usually a vector containing A elements) of is output. According to one embodiment, A=3, and the output vector is the decoded pixel Pd n The (R, G, B) triplet of . Module MLP' has the same structure as module MLP, and if the parameter W k If the encoding is lossless, its parameters are the same, or if the encoding is lossy, its parameters are different.

[0075] When all the sample points P of the signal n When all have been decoded, the reconstructed signal I(Pd n ), for example, the image I includes N vectors Pd n N decoded samples in the form of .

[0076] The decoding device DEC may be implemented by an electronic device comprising a processor and a memory (not shown); each of the aforementioned modules may then be generated via the interaction of the processor with computer program instructions stored in the aforementioned memory and designed, in particular as described below, to perform the functions of the module in question when these instructions are executed by the processor.

[0077] Figure 3 An example of a synthetic artificial neural network for encoding and decoding in the context of an embodiment of the present invention is presented.

[0078] The synthetic artificial neural network MLP for encoding and the synthetic artificial neural network MLP' for decoding are defined by the same structure (eg, comprising multiple layers of artificial neurons) and a set of weights and activation functions respectively associated with the artificial neurons of the networks in question.

[0079] The vector representation of the current sample point (from the feature map FM i and FME l or FMd i and FME'l The vector Z obtained n or Zd n ) is applied to the input (i.e., input layer) of the synthetic artificial neural network MLP or MLP'. The artificial neural network produces a vector P' representing the decoded sample point n or Pd n As output, according to one embodiment, are the constituent color components (R, G, B) of the color pixels of the image.

[0080] All of these reconstructed pixels are stitched together into a (2D, 3D) image, forming the decoded or reconstructed image.

[0081] In the encoder, a synthetic artificial neural network MLP is trained on the image to minimize the input representation I(P n ) and its output representation I(P' n ), while also minimizing the amount of data to be encoded. In this sense, the module EVAL performs a performance measurement.

[0082] Once trained, the network's parameters are either encoded losslessly (in which case the neural network MLP' is identical to the MLP) or lossily (in which case the network MLP' may be slightly different from the MLP).

[0083] Figure 4 It shows that Figure 1 Flowchart of an example of an encoding method implemented by an encoding device.

[0084] According to this embodiment, the signal is a two-dimensional image, so each sample point to be encoded has a coordinate (x n ,y n ) pixel P n .

[0085] Coding is done in two main stages:

[0086] In the first phase, called the construction phase, learning is performed so that for the input signal I(P n ) Determine the FM i The value and parameter W k , to optimize the total cost function. For example, via gradient descent, the parameters of the neural network MLP and the feature map FM are then updated i As is known in the art, the cost function can be of the bitrate-distortion type, or of the bitrate, or distortion, or perceptual type. To measure the bitrate R, it is necessary to simulate the FM i The encoding of , then the associated encoding bit rate (the size of stream B1) needs to be measured. According to one embodiment, the parameter W is not simulated. kThe encoding of , because their influence is smaller than that of the feature map. According to one embodiment, the parameter W is also simulated. k To measure the distortion D, it is necessary to simulate the encoding and then decoding of at least a portion of the image I in order to obtain at least one pixel P' resulting from the simulation of the encoding and then decoding. n , then, we need to measure the input image I(P n ) with the coded and then decoded image I(P' n ) between the corresponding parts.

[0087] Next, during the second phase, called the encoding phase, the graph FM i and parameter W k Encoded to produce the encoded value FMc i and WC k , and then transmitted or stored. They form the input signal I(P n ) is a compressed representation of .

[0088] The steps of a method according to one embodiment of the present invention will now be described.

[0089] In step E20, a plurality of (N) sample points P are included. n The signal to be coded I(P n ) is passed as input to this method.

[0090] In step E21, a first set of M graphs FM is initialized by the module GEN i Subsequently, the parameters W of the neural network MLP must be optimized during the construction phase k Hetu FM i The value of .

[0091] According to one embodiment, FIG. FM i With the input signal I(P n ) and therefore each picture contains the same resolution as the sample points P to be coded n The number of values ​​is the same as the number N.

[0092] According to one embodiment, FIG. FM i The resolution is less than or equal to the input signal I(P n ), and therefore, at least one of the maps comprises a number N' of values ​​to be encoded that is less than N.

[0093] According to one embodiment, the first map FM i has the resolution of the signal, and each subsequent plot has the Figure 1 Half resolution.

[0094] According to one embodiment, a plurality of graphs FM i has the same resolution, which is smaller than the resolution of the signal.

[0095] According to one embodiment, the map FM is initialized using predefined constant values. i .

[0096] According to another embodiment, the feature map is initialized using a set of random real values.

[0097] According to one embodiment, FME generates one or more graphs l And added to the first set, these one or more maps form the second set of L additional feature maps. They are used to construct the feature vector but are not stored or transmitted.

[0098] Subsequently, in a step E22, the updating module MAJ of the encoder updates or refines these feature maps FM of the first set during its learning phase of construction i .

[0099] In step E23, the module FMC of the encoder processes the graphs FM of the first group. i The encoding is performed. During the construction phase, this operation is a simulation of the encoding. During the encoding phase, this operation is the actual encoding, and the encoded values ​​form the stream B1. The simulation can be identical to the actual encoding, but can also be different (e.g., simplified). For this encoding, any known technique for compressing graph values ​​can be used.

[0100] In one embodiment, FIGFM i The encoding is performed in the order (FM1, FM2, ..., FM4), with the variables of each graph encoded in a predefined order (e.g., lexicographic order). Each graph undergoes entropy encoding. Entropy encoding produces a compressed stream B1, the bit rate of which is subsequently measured during step E28.

[0101] During a step E24, the map FM is extracted by the module XTR i and optionally FME l The extraction is based on the sample point P of the input signal. n The coordinates (x n ,y n ) to execute.

[0102] According to one embodiment, the coordinate values ​​(x n ,y n ).

[0103] According to one embodiment, the coordinate values ​​are scaled to extract lower resolution layers.

[0104] According to one embodiment, a value close to the above value is also extracted for subsequent processing in step E25 .

[0105] During a step E25 , the module TT determines the coordinates (x n ,y n ) for each sample point P n , according to Figure FM i and optionally the values ​​extracted by Fme to construct the vector Z n If necessary, the processing includes the i The extracted values ​​are quantized. This processing may include other operations such as filtering, scaling, applying any function (preferably a monotonic function), etc.

[0106] In one embodiment, Z n Included with the input map or map FMi (and optionally FME l ) is the same number of values ​​as the number of values ​​in the median. In this case, J = M(+L).

[0107] In one embodiment, Z n is located at the current pixel P n The coordinates (x n ,y n ) at Figure FM i (and optionally FME l ) values ​​form a J-tuple (z1, z2, ..., z J ), such as referring to Figure 5 As shown. Located in Figure FM i The coordinates (x n ,y n ) is given in Figure FM i or FME l Related vector Z n The final value of element i (z i ). In this case, module TT does not apply any processing other than quantization.

[0108] In one embodiment, Z n is generated by drawing from FMi (and optionally FME l ) is constructed from the values ​​sampled in the graph, where the sampling coordinates of these values ​​vary depending on the graph. For example, if the graph FM i (and / or FME l ) Since they have been downsampled and have different resolutions, the coordinates are adapted (by scaling) to match the resolution of each figure, as will be seen in the reference Figure 6 shown.

[0109] In one embodiment, Z nis a J-tuple constructed from values ​​that are retrieved from the graph FM by applying a process to one or more values ​​of the graph (e.g., filtering the neighbors of the target value in the graph). i (and FME l ) sampling. For example, in the image FMi with the same resolution as the input signal, the image at the coordinate (x n ,y n )、(x n -1,y n )、(x n ,y n -1) and (x n -1,y n -1) and process these values ​​(filtering, averaging, interpolation, etc.) to obtain the value of FM i or FME l Related vector Z n The final value of element i (z i According to another example, in the image FM having half the resolution of the input signal i In the example, we can consider the coordinates (x n / 2,y n / 2)、(x n / 2-1,y n / 2)、(x n / 2,y n / 2-1) and (x n / 2-1,y n / 2-1) and process these values ​​(filtering, averaging, interpolation, etc.) to obtain the value of FM in this figure. i or FME l Related vector Z n The final value of element i (z i ).

[0110] The samples to be coded are processed in order from n=1 to n=N, for example.

[0111] In step E26, vector Z n Processed by the neural network MLP to generate a representation of the sample point to be encoded P n As the output, according to one embodiment, (sample point P n The sample point P' is encoded and then decoded n The (R, G, B) triplet.

[0112] The structure and parameters W of the neural network k For example, they are initialized in the first iteration of this step. These parameters are then updated or refined during subsequent iterations of the method during the construction phase.

[0113] According to one embodiment, the parameters of the neural network are initialized with predefined values ​​that are known to produce satisfactory results (e.g., after training on a library of images).

[0114] According to another embodiment, the parameters W of the neural network are initialized with a set of random values. k .

[0115] In step E27, the parameters W of the neural network MLP are adjusted. k quantized and encoded. During the construction phase, this operation is a simulation of encoding. During the encoding phase, this operation is the actual encoding, and the encoded values ​​form the stream B2. The simulation can be identical to the actual encoding, but can also be different (e.g., simplified). For this purpose, any known technique can be used, such as the Neural Network Coding standard proposed in Part 17 of the MPEG-7 standard, also known as Neural Network Representation or NNR. It should be noted that in this case, the encoding pair parameter W needs to be selected k The amount of degradation caused.

[0116] In step E28 , the performance measure is evaluated.

[0117] To this end, the measurements and the characteristic map (by comparing the FM i encoding and decoding to simulate the flow B1) and optionally with the parameters of the neural network (by the parameters W k Encoding and decoding are performed to simulate the encoding simulation bit rate associated with stream B2).

[0118] According to one embodiment, the cost function is of rate-distortion type, expressed as (D + L * R), where D is, for example, the squared error measured between the input signal and the decoded signal (or the error measured on a subset of the signal's samples). According to another example, D is calculated from a perceptual function such as SSIM (Structural Similarity) or MSSSIM (Multi-Scale Structural Similarity). According to one embodiment, R is the simulated bit rate of stream B1; according to another embodiment, R is the total bit rate used to encode the image, i.e., the sum of the simulated bit rates of B1 and B2. L is a parameter that adjusts the rate-distortion tradeoff. Other cost functions may also be used.

[0119] As long as the cost function has not reached its minimum, the performance measure is not satisfactory and the method is repeated starting from step E22. This minimization can be performed via a known mechanism such as gradient descent, wherein the parameters are updated by updating the values ​​of the feature maps in step E22 and the parameters of the network in step E26.

[0120] In step E29, if the cost function has reached its minimum, the training stops. k) and feature maps (FM i ) is available, the streams B1 and B2 can be formed from them. According to another embodiment, the neural network (W) is processed in this step. k )’s updated parameters and feature maps (FM i ) to produce the coded parameters Wc that form the streams B1 and B2 k and FMc i .

[0121] Streams B1 and B2 can be concatenated to produce the final stream BS. According to one embodiment, the stream B2 of encoded parameters of the neural network is stored or transmitted before the stream B1 so as to be able to be decoded before the stream B1.

[0122] Figure 5 The method for processing an image I(P) in one embodiment of the present invention is shown. n ) (not shown) is an encoding method for encoding.

[0123] In this embodiment, FIGFM i The number of is 4. In a preferred embodiment, the number is 7. FIGFM i With the same image I(P n ) and therefore each image consists of W × H values, where W represents the width of the image in pixels and H represents its height (N = W × H).

[0124] In the variant drawn with a dashed line, the number of feature maps is 5: an additional map FME0 has been introduced, which is neither stored nor transmitted. This additional map typically contains data that can assist the network MLP in performing the task of reconstructing the signal. Thus, the additional map can be one or more maps from the following non-limiting list:

[0125] - A plot that contains at each point the x-coordinate of that point.

[0126] - A plot that contains at each point the y-coordinate of that point.

[0127] - a diagram containing a position code at each point (e.g. as described at the following Internet address: https: / / skosmos.loterre.fr / P66 / fr / page / -K0D65X2X-X).

[0128] - A map representing an image that is different from the one being processed and that can provide information about the image to be encoded, for example, previously processed images if the current image forms part of a sequence of images to be encoded (such as a video, a set of medical images, a multi-view representation, etc.).

[0129] - A map representing a feature map of an image different from the one being processed and which can provide information about the image to be encoded, for example, a previously processed map if the current image forms part of a sequence of images to be encoded (such as a video, a set of medical images, a multi-view representation, etc.).

[0130] - a picture containing the values ​​of an already decoded sample of the same picture (i.e. the previous sample in decoding order).

[0131] In this embodiment, vector Z n is obtained from the pixel at the current pixel P n The coordinates (x n ,y n ) at Figure FM i The extracted values ​​form a 4-tuple (z1...z4). The extracted values ​​are quantized by module TT, and the vector Z formed by the quantized values n The neural network MLP is processed to generate a second vector as output, which in this example is the vector representing the sample point P to be encoded. n In this case, the output vector is the encoded and then decoded pixel P' n The (R, G, B) triplet is inserted into the decoded image I(P' n ) in the color components (R', G', B') position (x n ,y n ) place.

[0132] According to one embodiment, the four images FM i Quantify.

[0133] According to the variant shown as the dotted line, the vector Z n It is a 5-tuple (z0...z4), where the value z0 is extracted from the attached map fme0.

[0134] Figure 6 A schematic diagram showing an encoding method used in another embodiment of the present invention is shown.

[0135] In this embodiment, FIGFM i The number of features is 4. In the previous embodiment, the first map FM1 has the same resolution as image I and therefore contains W × H values, where W represents the width of the image (in pixels) and H its height. The second map FM2 has half the resolution (in each dimension) of map FM1. The resolution of each additional map is half that of the previous one. This structure allows the number of variables in the feature map to be reduced, thereby facilitating encoding and learning while minimizing encoding costs.

[0136] In this embodiment, the 4-tuple Z nThe current pixel P n (x n ,y n ) coordinates (x n ,y n )、(x n / 2,y n / 2)、(x n / 4,y n / 4)、(x n / 8,y n / 8) at Figure FM i These extracted values ​​are processed and quantized to generate the 4-tuple Z n According to one embodiment, for Figure FM2, extract the n / 2,y n / 2) to obtain the vector Z associated with the graph FM2 n According to another embodiment, for the graph FM2, the value of the coordinate (x n / 2,y n / 2)、(x n / 2-1,y n / 2)、(x n / 2,y n / 2-1)、(x n / 2-1,y n / 2-1) and filtering to obtain the vector Z associated with the graph FM2 n The final value (z2) of . In this way, oversampling is performed. Filtering or oversampling can be performed using, for example, bicubic filtering or Lanczos filtering, which are known per se. A more general process can also be used: a neural network can be used to map the coordinates (x n / 2,y n / 2)、(x n / 2-1,y n / 2)、(x n / 2,y n / 2-1)、(x n / 2-1,y n / 2-1) to produce a final value (z2), such as an MLP type network.

[0137] As mentioned above, the vector Z n Processed by the neural network MLP to generate a second vector as output, here is the sample point P to be encoded n In this case, the output vector is the encoded and then decoded pixel P' n The (R, G, B) triplet is re-injected into the decoded image I(p'n )'s color components (R', G', B') n ,y n ) place.

[0138] Other types of structures are possible, for example, a drop rate other than half between graphs (such as one quarter, one third, etc.) An oversampling step is then used to produce the final value of the vector Zn.

[0139] As reference Figure 5 As mentioned above, other graphs can be imported into FME l , these graphs will be used to generate the vector Z n , but they are not stored or transmitted.

[0140] Figure 7 It shows that Figure 2 A flowchart of an example of a decoding method performed by a decoding device.

[0141] During a step E30, the streams B1 and B2 are extracted from the coded stream BS. These streams respectively contain a first set of images FMc i and parameter Wc k The encoded representation of .

[0142] During step E31, by i Decode to generate M graphs FMd i For this decoding, any known technique similar to that used in the encoder may be used, and preferably entropy decoding is used. In one embodiment, FIGFMd i Decoding is performed in the order of (FMd1, FMd2, ..., FMd4), and the variables of each graph are decoded in a predefined order (e.g., lexicographic order).

[0143] According to one embodiment described for an encoder, FIGFMd i With the signal to be reconstructed I(Pd n ) same resolution, i.e., the graphs contain N values.

[0144] According to an embodiment equivalent to that described for the encoder, FIG. FMd i The resolution is less than or equal to the reconstructed signal I(Pd n ) resolution.

[0145] According to an embodiment equivalent to that described for the encoder, a plurality of maps FMd i has the same resolution, which is lower than the resolution of the signal.

[0146] During a step E32, according to one embodiment, one or more graphs FME' are generatedl And added to the first set, this one or more maps form a second set of L additional feature maps. These maps are not decoded, but are generated by the decoder in the same way as they were generated in the encoder. These maps typically include data that can assist the network MLP's task of reconstructing the signal. Reference Figure 5 The non-limiting list of possible additional feature maps described for the encoder applies in this case as well.

[0147] During step E33, the module XTR' is taken from the map FMd i and possibly FME' l The extraction is based on the sample point P of the signal to be reconstructed. n The coordinates of the input signal (x n ,y n ) is performed. This step is the same as reference Figure 4 Step E24 described for the encoder is identical and the described embodiments are applicable.

[0148] The samples to be decoded are processed in order from n=1 to n=N, for example.

[0149] During a step E34, the module TT' determines the coordinates (x n ,y n ) of the sample point P to be decoded n , according to Figure FMd i and possibly FME' l Extract the values ​​to construct the vector Zd n If necessary, the processing may include the extraction of the value or the vector Zd formed n The inverse quantization is performed. The processing may include other operations such as filtering, scaling, etc. This step is similar to the reference Figure 4 The same step E25 as described for the encoder is used and the described embodiment is applicable. However, inverse quantization is not mandatory. Therefore, in this particular case, step E34 and module TT' are optional.

[0150] In step E35, the value Wc of the convection B2 is k Decode to generate the parameters Wd of the neural network MLP' k To this end, any known decoding technique corresponding to the encoding technique used by the encoder can be used. The neural network MLP' is similar to the network MLP, ie it has the same structure and the same parameters, except for the encoding, which can be lossy or lossless.

[0151] According to one embodiment, stream B2 is decoded before stream B1 in order to obtain the neural network before starting to decode samples.

[0152] During a step E36 , the vector Zd is processed by the neural network MLP′ n To generate the sample point Pd to be decoded n The second vector is outputted, according to one embodiment, at the position (x) of the color component (Rd, Gd, Bd) n ,y n ) is injected into the decoded image I(Pd n ). This step is the same as reference Figure 4 The step E26 is identical to the one described in the context of the encoder.

[0153] When all samples of the signal have been processed, we obtain an image corresponding to, for example, I(Pd n ) corresponds to the decoded signal.

[0154] Figure 8 The method for processing an image I(Pd) used in one embodiment of the present invention is shown. n ) is used to decode the decoding method.

[0155] In this embodiment, FIGFMd i The number of is 4. In a preferred embodiment, the number is 7.

[0156] In this embodiment, FIGFMd i Has the same resolution as the image to be decoded and therefore contains W × H variables, where W represents the width of the image in pixels and H is its height.

[0157] In this embodiment, the vector Zd n is obtained from the pixel at the current pixel Pd n The coordinates (x n ,y n ) at the FMd i The value of the vector Zd forms a 4-tuple (z1...z4). n is optionally dequantized and then processed by the neural network MLP′ to generate a second vector as output, which, according to an example, represents the sample to be decoded Pd n The (R, G, B) triplet is re-injected into the decoded image I(Pd n ) in the color components (Rd, Gd, Bd) (x n ,y n ) place.

[0158] According to the variant shown in dashed lines, the number of graphs is 5: an additional graph FME'0 has been introduced. In this embodiment, the vector Zd n It is a 5-tuple.

Claims

1. A method for performing a multi-sample n ) of the signal to be coded (I(P n )) is a method for encoding, the method comprising the following steps: - Build step, which includes the following sub-steps: - Construct (E21, E22) the first set of feature maps (FM i ); - For the position in the signal to be encoded (x n ,y n ) is associated with at least one sample point of the signal to be encoded, called the current sample point (P n ): - According to the current sample point (P n ) of the position (x n ,y n ) according to the feature map (FM) in the first group i ) Construct (E24, E25) feature vector (Z n ); - Use a set of parameters (W k ) defines an artificial neural network (MLP) to process the feature vector (Z) of (E25) n ), in order to provide a decoded value (P') representing the current sample point. n ) vector; - updating (E22, E26) at least one value of one of said feature maps of said first group and / or at least one parameter of said network as a function of a coding performance measure; - For the first set of feature maps (FM i ) and the set of parameters (W k ) to perform the encoding steps (E23, E27, E29).

2. The method for encoding a signal according to claim 1, wherein The method includes constructing (E21) a second set of feature maps (FME l ) step, and is characterized in that the feature vector is further constructed according to the feature map in the second group.

3. Method for encoding a signal according to any one of the preceding claims, characterized in that The feature map (FM i , FME l ) has a resolution lower than the resolution of the signal to be encoded.

4. Method for encoding a signal according to any one of the preceding claims, characterized in that The feature map (FM i , FME l ) has a resolution equal to the resolution of the signal to be encoded.

5. Method for encoding a signal according to any one of the preceding claims, characterized in that The eigenvector (Z n ) consists of extracting the current sample point (P n ) is the same position as (x n ,y n ) in the at least one feature map (FM i , FME l ) value of the substep.

6. Method for encoding a signal according to any one of the preceding claims, characterized in that The eigenvector (Z n ) consists of the following sub-steps: - According to the current sample point (P n ) of the position (x n ,y n ) extracts the feature map (FM i , FME l ) multiple values; - processing said extracted values ​​in order to obtain a feature vector.

7. Method for encoding a signal according to any one of the preceding claims, characterized in that Encoding the first set of feature maps includes an entropy encoding sub-step.

8. A method for performing a multi-sample n ) is a method for decoding a signal to be decoded, the method comprising the following steps: - For the first set of feature maps (FMd i ) to decode (E31); - A set of parameters (Wd k ) to decode (E35); - For the position in the signal to be decoded (x n ,y n ) is associated with at least one sample point of the signal to be decoded, called the current sample point (Pd n ): - According to the position of the current sample point (x n ,y n ) According to the feature map (FMd) in the first group i ) Construct (E33, E34) feature vector (Zd n ); as well as: - Using the decoded parameters (Wd k ) defines an artificial neural network (MLP') to process the feature vector (Zd n ), in order to provide a representation of the current sample point (Pd n ) is a vector of decoded values.

9. The decoding method according to claim 8, wherein: The method involves constructing a second set of feature maps (FME' l ) step (E32), and is characterized in that the feature vector is further constructed according to the feature map in the second group.

10. The decoding method according to claim 8 or 9, characterized in that: The characteristic map (FMd i , FME' l ) has a resolution lower than the resolution of the signal to be decoded.

11. The method for decoding a signal according to any one of claims 8 to 10, characterized in that: The characteristic map (FMd i , FME' l ) has a resolution equal to the resolution of the signal to be decoded.

12. The method for decoding a signal according to any one of claims 8 to 11, characterized in that The characteristic vector (Zd n ) consists of extracting the current sample point (Pd n ) is the same position as (x n ,y n ) in the at least one feature map (FMd i , FME' i ) value of the substep.

13. The method for decoding a signal according to any one of claims 8 to 12, characterized in that The characteristic vector (Zd n ) consists of the following sub-steps: - According to the current sample point (Pd n ) of the position (x n ,y n ) extract multiple values ​​of these feature maps; - processing said extracted values ​​in order to obtain a feature vector.

14. The method for decoding a signal according to any one of claims 8 to 13, characterized in that Decoding (E31) said first set of feature maps comprises an entropy encoding sub-step.

15. A method for performing a multi-sample n ) of the signal to be coded (I(P n )) A device for encoding, characterized in that The device is configured to: - Construct (GEN, MAJ) the first set of feature maps (FM i ); - For the position in the signal to be encoded (x n ,y n ) is associated with at least one sample point of the signal to be encoded, called the current sample point (P n ): - According to the current sample point (P n ) of the position (x n ,y n ) according to the feature map (FM) in the first group i ) Construct (XTR, TT) feature vector (Z n ),as well as; - Use a set of parameters (W k ) defines an artificial neural network (MLP) to process the feature vector (Z n ), in order to provide a decoded value (P') representing the current sample point. n ) vector; - updating (MAJ, NNC) at least one value of one of said feature maps of said first group and / or at least one parameter of said network as a function of a coding performance measure; - For the first set of feature maps (FM i ) and the set of parameters (W k ) for encoding (FMC, NNC).

16. A method for detecting a plurality of sample points (Pd n ) of a device for decoding a signal to be decoded, characterized in that The device is configured to: - For the first set of feature maps (FMd i ) for decoding (FMD); - A set of parameters (Wd k ) for decoding (NND); - For the position in the signal to be decoded (x n ,y n ) is associated with at least one sample point of the signal to be decoded, called the current sample point (Pd n ): - According to the position of the current sample point (x n ,y n ) According to the feature map (FMd) in the first group i ) Construct (XTR', TT') feature vector (Zd n ); as well as: - Using the decoded parameters (Wd k ) defines an artificial neural network (MLP') to process the feature vector (Zd n ), in order to provide a representation of the current sample point (Pd n ) is a vector of decoded values.

17. A computer program comprising instructions for executing the steps of the encoding method according to claim 1 or the decoding method according to claim 8 when said program is executed by a computer.

Images