Method and device for encoding and decoding images - Patents.com

Abstract

The present invention relates to a method and device for encoding and decoding a signal comprising a plurality of samples, the decoding method comprising the steps of: decoding a first group of feature maps representing the signal, decoding a set of parameters representing a neural network, referred to as a synthetic neural network, for at least one sample of the signal to be decoded, referred to as the current sample, associated with a position in the signal to be decoded, constructing a feature vector from the feature maps of said first group as a function of the position of said feature vector using the synthetic neural network defined by the decoded parameters, in order to provide a vector representing a decoded value of the current sample, decoding and encoding said first group of feature maps, comprising, for at least one value of one of the feature maps, referred to as the current value, entropy coding said value as a function of at least its neighboring values.

Description

[Technical Field]

[0001] The present invention relates to the general field of coding of one-dimensional or multi-dimensional signals, and more particularly to the compression of digital images. [Background technology]

[0002] Digital images are generally source-encoded to be compressed in order to limit the resources required for their transmission and / or storage. There are many encoding standards, such as the JPEG family of standards for still images and the standards of the ITU / MPEG organization for moving images or video (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) and their extensions (MVC, SVC, 3D-HEVC, etc.).

[0003] An image is generally encoded by dividing the image into a number of rectangular blocks and encoding these blocks of pixels in a given processing sequence. In existing video compression techniques, the processing of a block typically involves prediction of the pixels of the block performed using previously encoded and then decoded pixels present in the image being encoded, in which case it is referred to as "intra-prediction," or prediction of the pixels of the block performed using a previously encoded image, in which case it is referred to as "inter-prediction." This use of any spatial and / or temporal redundancy avoids transmitting or storing values ​​of the pixels of each pixel block by representing at least some of the blocks using residuals that represent the difference between predicted values ​​of the pixels of the block and the actual values ​​of the pixels of the predicted block.

[0004] Video formats are constantly evolving in search of higher compression ratios and becoming more predictable to accommodate a wide variety of expected formats and communication networks, making conventional encoding and decoding algorithms very complex.

[0005] In addition to these traditional methods proposed by compression standards (JPEG, MPEG, ITU), there is a trend towards the development of methods based on artificial intelligence, in particular neural intelligence.

[0006] Some of these neural approaches can be seen as simple extensions of the concepts of compression technique competition mentioned above, such as prediction mode competition and video coding transformation.

[0007] Another approach uses the "autoencoder" concept. An autoencoder is an artificial neural network-based learning algorithm that allows for the construction of new representations of a dataset. The autoencoder architecture consists of two parts: an encoder and a decoder. The encoder consists of a set of layers of neurons that process data to construct new representations, called "encoded" representations, also known as "latent representations." The decoder's neural layers then receive these representations and filter them to attempt to reconstruct the original data. The difference between the reconstructed data and the initial data allows for measurement of any errors introduced by the autoencoder. Training involves modifying the autoencoder's parameters to reduce the measurement error, measured on various samples of the dataset. The performance capabilities of such autoencoder-based systems come at the expense of a significant increase in memory footprint and complexity compared to traditional approaches, such as those proposed by compression standards. They can have millions of parameters and require up to a million MAC (multiply-accumulate) operations to decode a single pixel. This makes such decoders significantly more complex than traditional decoders and hinders the adoption of learning-based compression.

[0008] Recently, a simple neural network-based image coding technique was described in (Non-Patent Document 1). The proposed coding technique involves adjusting a neural network to the image and quantizing and transmitting the network weights. When decoding, the neural network is evaluated at each pixel location to reconstruct the image. Nevertheless, such techniques remain inefficient in terms of compression. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] “Compression with Implicit Neural representations” by Emilien Dupont et al.,(arXiv:2103.03123) Summary of the Invention [Problem to be solved by the invention]

[0010] Therefore, there is a need for a solution for simple and efficient encoding / compression of an image or a sequence of images. [Means for solving the problem]

[0011] The object of the invention is an encoding method according to claim 1 and a decoding method according to claim 8.

[0012] Within the meaning of the present invention, the terms "decoding" or "encoding" are understood to mean operations that involve representing a set of samples in a compact form, for example carried by a digital bitstream. Decoding is understood to mean operations that involve processing a digital bitstream in order to render decoded samples.

[0013] The term "sample" of a signal is understood to mean a value sampled from the signal. Sampling a signal produces a sequence of discrete values ​​called samples. In the case of an image signal, the samples are called pixels, which may be, for example, color pixels conventionally represented by a triplet of values, e.g., (R, G, B) or (Y, U, V). The location of a sample is identified by its coordinates on the horizontal (x) and vertical (y) axes in the image.

[0014] The expression "signal comprising a plurality of samples" is understood to mean a signal that is one-dimensional (audio, sound), two-dimensional (image) or three or more dimensional (stereo, multi-view images, images associated with depth maps, video, etc.). Depending on this dimensionality, a sample may have one, two or several coordinates in the signal. In the case of an image signal, the position of a sample is identified by its x and y coordinates.

[0015] The term "feature map" is understood to mean an abstract representation of a signal that contains a plurality of potentially discrete variable data, also called values, real numbers or integers, for example. As is known, these maps are also called "latent representations".

[0016] The expression "transformation of feature maps" is understood to mean the application of a mathematical operation that allows the values ​​of a first map to be transformed into values ​​of a second map. The first map, called a map of the first group, is intended for encoding and can be any type of map. The second map, called a transformed map or a map of the second group, has the same resolution as the input signal, i.e., contains the same number (N) of values ​​as the input signal (or the signal to be decoded) contains samples. The transformation can include, for example, interpolation, upsampling, filtering, quantization, Fourier transformation, etc.

[0017] The expression "data feature vector constructed from feature maps as a function of position" is understood to mean a vector composed of one or more, preferably discrete, elements or data, the data being constructed from feature maps at positions determined by the position of the sample being processed in the signal. This feature vector is one input to the synthetic neural network. For example, in the case of a one-dimensional audio signal, such a vector may be constructed from multiple values ​​sampled in each of the feature maps at the same coordinates as the sample being encoded. In the case of an image, such a vector may be constructed from multiple values ​​sampled in each of the feature maps at the same x and y coordinates as the sample being encoded (or decoded). Once sampled from the feature maps, these values ​​may be processed, for example by quantization, filtering, interpolation, etc., to form a feature vector before entering the synthetic neural network.

[0018] The term "synthetic neural network" is understood to mean neural networks such as convolutional neural networks, multi-layer perceptrons, LSTM (long short-term memory), etc. A neural network is defined, for example, by multiple layers of artificial neurons and a set of activation, weighting and summation functions (for example, one layer can calculate y=f(Ax+b), where y and b are vectors of dimension N, x is a vector of dimension M, A is a matrix of dimension MxN, and f is the activation function).

[0019] The term "parameter of a neural network" is understood to mean one of the values ​​that characterize a neural network, such as a weight (filter coefficient, weighting, bias, value that influences the operation of nonlinearities) associated with one of the neurons.

[0020] The expression "processing using a synthetic neural network" is understood to mean applying the function represented by the synthetic neural network to the input feature vector to generate an output vector representing the samples to be encoded (or decoded), which may include one or more data representing the samples.

[0021] The term "performance measure" should be understood to mean a measure between at least one value of a sample to be coded and the decoded value of said sample. The measure can, for example, evaluate distortion or perceptual error. The performance measure can be performed on one sample or on multiple samples (e.g., the current sample or the current image, etc.). The measure can also include a bitrate measure, in particular, associated with the coding of the synthetic neural network and / or the coding of the first group of feature maps. The measure can be a joint measure of bitrate and distortion, achieved by weighting them. As is well known in the art, the value of this measure is generally minimized until a target value is reached.

[0022] The term "construction step" is understood to mean a step aimed at constructing parameters that represent the image before it is actually coded. The construction sub-step can be repeated as many times as necessary to obtain an acceptable performance measure.

[0023] In general, the steps of the encoding or decoding methods should not be interpreted as relating to the notion of temporal succession, in other words the steps may be performed in a different order than indicated in the encoding or decoding independent claims or even simultaneously.

[0024] The coding method according to the invention constructs coding parameters from an input signal, e.g., an image, by training a neural network, called a synthetic neural network, with feature vectors associated with the positions of the samples to be coded. These feature vectors are constructed from feature maps that may have the resolution of the input signal or a lower resolution. During training or construction, the values ​​of the parameters of the neural network or the feature maps are updated as a function of a performance measure, e.g., of the bitrate distortion type. Once the training is over, i.e., if the obtained performance measures are satisfactory, the actual coding of the parameters of the synthetic neural network and the values ​​of the feature maps can be performed and stored or transmitted to a decoder.

[0025] Advantageously, the training process allows the parameters of the synthetic neural network and / or the values ​​of the feature maps to be refined until a suitable representation in terms of performance is obtained, for example until a desired balance is achieved between the generated bit rate and the distortion suffered by the input signal. The training of the feature map values ​​and the training of the parameters of the synthetic neural network may be joint training. Advantageously, the coding method according to the invention allows for efficient compression of signals.

[0026] Advantageously, the decoding method is simple, as it only requires decoding the feature maps and synthetic neural networks to reconstruct a decoded version of the signal, e.g., an image.

[0027] Advantageously, the coding of the feature maps is rendered particularly efficient by taking into account the coding neighborhood, thereby making use of redundancies present in the maps.

[0028] Such synthetic neural networks may advantageously have a very simple structure with a small number of parameters.

[0029] Additionally, the decoding may be performed incrementally, sample by sample.

[0030] According to an embodiment of the encoding or decoding method, the encoding method comprises the following sub-steps of encoding said current value of one of said feature maps: - constructing a neighborhood vector from the feature maps of the first group; - processing said neighborhood vectors using an artificial neural network, called a predictive neural network, defined by a set of parameters, in order to provide a prediction of said current value; - updating at least one parameter of said prediction network as a function of the coding performance measure, - encoding said set of parameters of the prediction network.

[0031] The encoding method comprises the following sub-steps of decoding the current value of one of the feature maps: - a substep of decoding a set of parameters representing a neural network called a predictive neural network; - constructing a neighborhood vector from the feature maps of the first group; and - processing said vector using a predictive neural network to provide a prediction of said current value.

[0032] Advantageously, according to this encoding or decoding mode, the feature map is efficiently compressed by a predictive neural network capable of predicting the map's values ​​as a function of its neighbors. The term "neighborhood vector" is understood to mean a vector composed of one or more elements or data constructed from the feature map at a position close to the current sample's position, which is also the position of the current value in the feature map. The neighboring positions may refer to values ​​in the map being processed (e.g., neighboring values ​​in the upper left corner of the map being processed) or in another feature map (e.g., neighboring values ​​in the same position in a previous map). This neighborhood vector is one input to the predictive neural network. The term "prediction" is understood to mean at least one piece of data for estimating the current value of the feature map, e.g., a probability, a statistical value, etc. A predictive neural network trained on a map of an image can efficiently represent the map of an image. Furthermore, it is inexpensive to encode.

[0033] The method includes transforming the first group of feature maps to obtain a second group of feature maps having a resolution of the input signal, the method being characterized in that the feature vector is constructed from the feature maps of the second group.

[0034] Advantageously, according to this embodiment, the feature maps are divided into two groups, one reserved for the extraction of feature vectors and the other reserved for encoding. It is therefore possible to separate two methods with different objectives: the first group of maps to be encoded (or decoded) must be compressed as efficiently as possible, while the second group of maps must be able to facilitate the process of extracting and building feature vectors.

[0035] According to one variant, at least one of the feature maps of the first group has a lower resolution than the signal to be coded (or decoded), and the transformation operation involves upsampling. Advantageously, according to this embodiment, at least one of the feature maps of the first group to be coded (or decoded) contains fewer values ​​than it would have with the resolution of the signal, so that the compression of the feature maps is more efficient. For example, in the case of a digital image, one of the feature maps in the first group may have a resolution half the number of samples contained in the input signal, i.e. it contains half the number of x and y values, i.e. it contains a quarter of the number of values ​​in total of a feature map with the resolution of the signal. In contrast, the feature map of the second group, corresponding to the transformation of this map of the first group, has the same resolution as the signal. Therefore, the transformation in this case involves at least one upsampling operation in order to obtain in this transformed map the same number of values ​​as the number of samples contained in the input signal (or the signal to be decoded).

[0036] At least one of said feature maps of the first group has the same resolution as the signal to be encoded (or decoded).

[0037] Advantageously, according to this embodiment, at least one of the feature maps has the same resolution as the input signal to be encoded (or decoded), allowing high fidelity and matching to the details of the initial resolution of the signal. The transformation according to one embodiment in this case preserves the number of values ​​of the transformed feature map and may be simplified to identity (no processing is performed on the values ​​of the first group of maps) or may include filtering, quantization, Fourier transform operations, etc.

[0038] During the encoding step, quantization is essential for proper operation of the system, for example, when the feature maps contain floating or real values. Values ​​need to be quantized before being encoded and / or provided as input to synthetic and / or predictive neural networks. In contrast, for decoding, dequantization is not required depending on the embodiment.

[0039] - constructing said feature vector comprises the sub-step of extracting a value from said at least one feature map at the same position as a current sample in the encoded (or decoded) signal.

[0040] Advantageously, to construct the elements of the feature vector, it is possible to extract values ​​from the feature maps of the first or second group at the same positions as the samples in the signal (input signal in case of encoding, signal to be decoded in case of decoding). This method is easy to implement. For example, if J input feature maps with the same resolution as the signal are available, a feature vector of J elements can be directly constructed by simply extracting the map values ​​at the coordinates of the current sample (same abscissa and same coordinate in the feature maps).

[0041] said construction of the feature vector comprises the following sub-steps: - extracting a plurality of values ​​of the feature map of the first group as a function of the position of the current sample; - processing said extracted values ​​to obtain a feature vector.

[0042] Advantageously, according to this embodiment, the feature vector is extracted from a feature map having a lower resolution than the signal to be encoded (or decoded), in particular before undergoing processing, which may be optional. Such processing may correspond, for example, to quantization, scaling or filtering of the extracted data. In the case of encoding, quantization is essential for the proper operation of the system if the feature map contains, for example, floating or real values. Values ​​need to be quantized before being encoded and / or provided as input to a synthetic neural network and / or a predictive neural network. In contrast, in the case of decoding, dequantization is not necessary depending on the embodiment.

[0043] the method comprises a step of constructing a third group of feature maps, and a feature vector is also constructed from said feature maps.

[0044] Advantageously, these additional maps of the third group, which are constructed identically in the encoder and decoder, are not stored or transmitted to the decoder, nor are they decoded therein, thus making it possible to use additional data to improve compression without reducing the bit rate. For example, the additional data may include coordinates, causal data available in the maps of the first or second group, data relating to other images already processed by the encoder or decoder, etc.

[0045] Correlatively, further objects of the invention are encoding and decoding devices.

[0046] The features and advantages of the encoding method or the decoding method apply equally to the encoding and decoding device according to the invention, and vice versa.

[0047] A further object of the invention is a computer program on a storage medium, which program is capable of implementing a computer or control device according to the invention, the program comprising instructions designed to implement the corresponding method, which program can use any programming language and can be in the form of source code, object code, an intermediate code between source code and object code, such as a partially compiled form, or any other desired form.

[0048] The present invention also relates to a computer-readable information medium or storage medium containing instructions for the computer program described above. The information or storage medium may be any entity or device capable of storing a program. For example, the medium may include a storage means such as a ROM, e.g., a CD-ROM or a microelectronic circuit ROM, or even a magnetic storage means, e.g., a floppy disk or hard disk, a DNA sequence, or a flash memory. Furthermore, the information or storage medium may be a transmissible medium, such as an electric or optical signal, which may be routed via an electric or optical cable, by a radio wave link, by a wireless optical link, or by other means.

[0049] The program according to the present invention can be downloaded via the Internet.

[0050] Alternatively, each information medium or storage medium may be an integrated circuit incorporating a program, the circuit being designed to perform or to be used for performing the method according to the invention.

[0051] Further characteristics and advantages of the invention will become apparent from the following description, with reference to the accompanying drawings, which show non-limiting embodiments. [Brief explanation of the drawings]

[0052] [Figure 1] 1 shows a schematic representation of a coding device for use within the scope of the present invention; [Figure 2] 1 shows a schematic representation of a decoding device for use within the scope of the present invention; [Figure 3] 1 shows an example of a synthetic artificial neural network for use within the scope of the present invention. [Figure 4] 1 shows an example of a predictive artificial neural network for use within the scope of the present invention. [Figure 5] 2 is a flowchart illustrating an example of an encoding method that may be implemented by the encoding device of FIG. 1. [Figure 6] 1 shows a diagram of an encoding method used in one embodiment of the present invention; [Figure 7] 3 is a flowchart illustrating an example of a decoding method that may be implemented by the decoding device of FIG. 2. [Figure 8] 1 shows a diagram of a decoding method used in one embodiment of the present invention; [Figure 9] 6 is a flowchart illustrating a method for encoding a feature map, which may be implemented by the encoding device of FIG. 1 and the encoding method of FIG. 5. [Figure 10] 8 is a flowchart illustrating a method for decoding a feature map, which may be implemented by the decoding device of FIG. 2 and the decoding method of FIG. 7. [Figure 11]1 shows a diagram of a method for decoding or decoding feature maps. [Figure 12] 10 shows a diagram of another method of encoding or decoding feature maps. DETAILED DESCRIPTION OF THE INVENTION

[0053] FIG. 1 shows diagrammatically the encoding device ENC.

[0054] The encoding device ENC comprises a feature map generation module GEN, an additional feature map generation module FME, a transformation module SE, a data extraction module XTR, a processing and quantization module TT, a module MLP corresponding to a synthetic artificial neural network, a neural network encoding module NNC capable of encoding synthetic neural networks and optionally predictive neural networks, a feature map encoding module FMC, a coding performance evaluation module EVAL, an update module MAJ.

[0055] The encoding device ENC may be implemented by an electronic device including a processor and a memory (not shown), which is then able to generate each of the above-mentioned modules through the interaction of the processor with computer program instructions stored in said memory and designed, when executed by the processor, to carry out the functions of the module in question, in particular as described below.

[0056] The encoding device ENC of Figure 1 takes as input P n A sequence of samples to be coded, denoted as I(P n In the second case, the image signal I(P n ) can represent a two-dimensional image or multiple two-dimensional images (video, color components, stereo components, multi-view components, etc.). ndenotes one sample n of an input signal containing N samples. In one embodiment, the signal is a color image signal represented by at least one two-dimensional representation, such as a matrix of pixels, each pixel having one red component (R), one green component (G), and one blue component (B), or, as a variant, one lightness component (Y) and at least one chroma component (U, V). The location of each pixel is defined by its x- and y-coordinates (x and y) in the image. In one embodiment, the image is grayscale and represented by a two-dimensional representation, such as a matrix of pixels, each pixel having a grayscale or lightness component. In this case, the vector representing the pixel is simplified to one component.

[0057] As will be explained in more detail below with reference to FIGS. 3 to 7, the feature map generation module GEN generates the FM i The optional module FME is configured to generate a plurality of M feature maps, denoted as FME. l One or more additional maps (L), denoted as

[0058] In one embodiment, the module SE generates a feature map FMS having the same resolution as the input signal. i To generate the second group of features, we use the feature map FM i Transform the first group of

[0059] Optional module FME, M maps FM i This set of data or the vector Z formed from these data nThe data extracted from the input sequence can be quantized. Note that quantizing a value refers to matching it with a member of a discrete set of possible code symbols. For example, the set of possible code symbols may consist of integer values, and the quantization system simply rounds the actual value to an integer value. According to another example, quantization involves multiplication by a given value followed by rounding. The SE module then transforms at least one value of the feature map, for example by upsampling, interpolation, filtering, etc. At the end of the transformation, the transformed feature maps of the second group have the same resolution as the images of the input sequence. Advantageously, according to this embodiment, the encoded feature maps may have a lower resolution than the resolution of the images to be encoded, while the maps of the second group used to construct the feature vector have the same resolution as the sequence of images, thereby facilitating the extraction of values.

[0060] In one embodiment, the module SE is not present, in which case the values ​​used to construct the feature vector are extracted from the first group of feature maps.

[0061] The module XTR encodes the current sample P n About Feature Map FM i (and / or FMS according to one of the above-mentioned embodiments) i and / or FME l ) as a function of its coordinate in the input signal. For example, n ,y n ) sample P n If you intend to encode the coordinates (x n ,y n ) to extract the value in the map at the position given by

[0062] In one embodiment, the extracted values ​​are represented by a vector Z n Form Z n is a J-tuple, i.e., J elements or data z i Contains the vector Z of index n n is the pixel P'n refers to the feature vector of

[0063] In one embodiment, the optional module TT calculates the vector Z n The module TT processes the extracted values ​​to generate a set of feature maps. The module TT can quantize the data extracted from the set of feature maps. The processing can also include other operations such as filtering, scaling, etc. In particular, if the module SE is not used and the feature maps of the first group have a lower resolution than the images of the sequence, the module TT can take into account the coordinates of the values ​​in the maps with lower resolution.

[0064] Note that at least one of the modules SE or TT must quantize the feature maps.

[0065] The module MLP receives the samples P to be coded. n To generate as output a second vector representing k and takes as input a vector Z n or a synthetic neural network capable of processing a J-tuple. According to one embodiment, the synthetic neural network is an MLP, or multi-layer perceptron, consisting of an input layer adapted to the input format (J-tuple), optionally one or more hidden layers, and an output layer adapted to the output format of an output vector, generally a vector containing A elements. According to one embodiment, A is equal to 3 and the output vector is a pixel P' that is coded and then decoded. n is an (R,G,B) triplet.

[0066] The module NNC is responsible for the synthesis of neural networks, in particular their parameters W k Optionally, the module NNC encodes the predictive neural network ARM, in particular its parameters O bDuring the coding training or construction process, i.e., as long as the step of evaluating the performance remains unsatisfactory, the module NNC simulates the coding, followed by the decoding, the results of which are sent to the evaluation module. Subsequently, the parameters W of the synthetic neural network are k , and optionally performs the encoding of the predictive neural network ARM. The encoded parameters are k and Oc b As is known, the coding simulation can be identical to or approximate the actual coding.

[0067] Module FMC, Map FM i , i.e., encode the values ​​of the feature maps of the first group (any additional maps FME l and optionally excluding the second group of maps resulting from the upsampling by the module SE). During the coding training or construction process, i.e. as long as the step of evaluating the performance remains unsatisfactory, the module FMC simulates the coding, followed by the decoding, the results of which are sent to the evaluation module. Subsequently, the maps FM i The encoded map is then used to i As is known, the coding simulation may be identical to or approximate actual coding. The coding module quantizes the latent representation of the first group of map values, if necessary, by generating an ordered set of quantized values ​​using a quantizer. The coding module then compresses the quantized data using coding that takes into account the neighborhood of the feature map values ​​being coded. As described below, the module FMC may include a predictive neural network (ARM).

[0068] The module EVAL performs the evaluation and minimization of the coding performance. The evaluation function is, for example, of the bit rate distortion type. The minimization can be performed via gradient descent or any other method within the capabilities of a person skilled in the art.

[0069] The module MAJ encodes the map FM as a function of the results of the performance function. i Update the value of

[0070] FIG. 2 shows diagrammatically the decoding device DEC.

[0071] The decoding device DEC in Fig. 2 receives as input M feature maps FMc i The first group of coded data organized in a layer (also called layer FM) and the coded parameters Wc of the synthetic neural network MLP'. k , optionally the encoded parameters Oc of the synthetic neural network ARM' b and receive.

[0072] This decoding device DEC comprises a neural network decoding module NND capable of decoding a synthetic neural network MLP' and optionally a predictive neural network ARM', a feature map decoding module FMD, a data extraction module XTR', an inverse transformation module SE', a processing and inverse quantization module TT', a module corresponding to the synthetic neural network MLP' and an additional feature map generation module FME'. According to one embodiment, the decoding device comprises a plurality of decoded samples Pd n Including I(Pd n ) and outputs the decoded image.

[0073] The M maps decoded by the module FMD are decoded by FMd i The parameters of the synthetic neural network (MLP') coded by the module NND are Wd k The parameters of the predictive neural network (ARM') decoded by the module NND are denoted as Od b It is shown as follows.

[0074] The decoder module FME' l and the additional map FME generated by the encoder. lIt is also possible to generate one or more additional maps, numbered L, that are identical to

[0075] In one embodiment, the module SE' is i to generate a second group of feature maps with the same resolution as the signal being decoded, denoted as i The decoder transforms the first group of feature maps. Module SE′ optionally performs an inverse quantization corresponding to the quantization performed in the encoder. Inverse quantization is not necessary if the quantizer Q of the encoder simply rounds the actual values ​​received. Inverse quantization is also not necessary if the neural network is able to take into account the quantization of its input data. In other cases, the decoder performs the inverse operation of quantizer Q. Module SE′ then transforms the values ​​of the feature maps in a manner similar to that performed by the encoder, including, for example, upsampling, interpolation, filtering, etc. Once the transformation is complete, the transformed feature maps of the second group have the same resolution as the images of the sequence to be decoded.

[0076] In one embodiment, module SE' is not present, in which case the values ​​used to construct the feature vector are extracted from the first group of feature maps.

[0077] The module XTR' is identical to the module XTR of FIG. 1. The module XTR' receives the sample Pd n For M feature maps FMd i (and / or FMS' according to one of the above-mentioned embodiments) i and / or FME' l ) as a function of its coordinate in the signal to be decoded. In one embodiment, J=M. In one embodiment, J=M+L.

[0078] In one embodiment, the extracted values ​​are represented by a vector Zd n Form Zd n is a J-tuple, i.e., J elements or data zd i Includes:

[0079] In one embodiment, the optional module TT' is n The extracted values ​​are processed to generate . The module TT' can perform inverse quantization of the data extracted from the set of feature maps. The processing may also include other operations such as filtering, scaling, etc., similar to those performed by an encoder.

[0080] The module MLP' has K parameters Wd k The sample P to be decoded is defined by n To generate as output a second vector, typically a vector with A elements, representing n or J-tuples, called synthetic neural networks. According to one embodiment, K=3 and the output vector is a vector of decoded pixels Pd n The (R, G, B) triplet of the MLP module has the same structure as the MLP module, and its parameters are the parameters W k are identical if the encoding is lossless, or different if the encoding is lossy.

[0081] All samples of the signal P n is decoded, the reconstructed signal I(Pd n ) is obtained, and according to one example, N vectors Pd n An image I is obtained that contains N decoded samples, which are of the form:

[0082] The decoding device DEC may be implemented by an electronic device including a processor and a memory (not shown), which is then able to generate each of the above-mentioned modules through interaction of the processor with computer program instructions stored in said memory and designed, when executed by the processor, to carry out the functions of the module in question, in particular as described below.

[0083] FIG. 3 illustrates an example of a synthetic artificial neural network used for encoding and decoding in accordance with an embodiment of the present invention.

[0084] The composite artificial neural network MLP used for encoding and the composite artificial neural network MLP' used for decoding are defined by the same structure, e.g., including multiple layers of artificial neurons, and a set of weights and activation functions associated with each of the artificial neurons of the network in question.

[0085] A vector representation of the current sample (in the encoder we have the feature map FM i / FMS i or FMd in the decoder i / FMS' i and FME' l Vector Z obtained from n or Zd n ) is applied to the input (i.e., input layer) of the compound artificial neural network MLP or MLP'. The compound artificial neural network outputs a vector, according to one embodiment, whose color components (R, G, B) form the color pixels of the image.

[0086] Concatenating all these reconstructed pixels into a (2D, 3D) image forms the decoded or reconstructed image.

[0087] In the encoder, a synthetic artificial neural network (MLP) optimizes the current image I(P n ) and its output representation I(P' n ) is trained on images to minimize the difference between the images. The module EVAL performs performance measurements in this sense.

[0088] Once the encoder has been trained, the network parameters are coded either losslessly (the neural network MLP' is identical to the MLP) or lossily (the network MLP' can be slightly different from the MLP).

[0089] FIG. 4 illustrates an example of a predictive artificial neural network used for encoding (ARM) and decoding (ARM') feature maps in accordance with an embodiment of the present invention.

[0090] The predictive artificial neural network ARM used for encoding and the predictive artificial neural network ARM' used for decoding are defined by the same structure, e.g., including multiple layers of artificial neurons, and a set of weights and activation functions associated with each of the artificial neurons of the network in question.

[0091] A vector representation of the current sample (in the encoder we have the feature map FM i / FMS i and FME l or FMd in the decoder i / FMS' i and FME' l Vector C obtained from n or Cd n ) is applied to the input (i.e., input layer) of a predictive artificial neural network ARM (for encoding) or ARM' (for decoding).

[0092] A predictive neural network behaves as a function that outputs a prediction of the current value of the feature map being processed, which may be in the form of predicted values ​​or probability data.

[0093] According to one embodiment, in the encoder, the network determines the maps FMi to be encoded. i The current value of V n A function f that provides the expected mean and / or variance (μ, σ) for the current value of ψThese statistics are used to perform entropy coding of this value. For example, if the function produces a mean, this mean is subtracted from the current value and only the difference is entropy coded, with this mean being considered a prediction of the current value. Alternatively, if the function produces a mean and a variance, the mean is subtracted from the current value and the difference is coded using entropy coding adapted to the variance, for example by quantizing the variance into a predetermined set of variances and associating a type of entropy coding to each quantized variance value. At the encoder, the network calculates the map FMd to be decoded. i The current value of Vd n A function f that provides the expected mean and / or variance for the current value of ψ These statistics are used to perform entropy decoding of the value. For example, if the function produces a mean, the current value is decoded by the entropy decoder and this mean is added to the current value. Alternatively, if the function produces a mean and a variance, the current value is decoded by the decoder using entropy decoding adapted to the variance, for example by quantizing the variance into a predetermined set of variances and associating a type of entropy decoding with each quantized variance value.

[0094] According to another embodiment, the neural network can generate an expected probability (pr) for each possible value of the current sample, in which case the entropy coding or decoding is adapted to this probability (as is known for Huffman or arithmetic entropy coding).

[0095] In the encoder, a predictive artificial neural network ARM is trained on the images to minimize the amount of data to be coded. The module EVAL performs a performance measurement in this sense. The overall performance measurement is the amount of data that is coded and then decoded I(P') while minimizing the decoding rate. n ) and the input image I(P nNote that the objective of this method is to minimize the distortion between the feature maps and the image. According to one embodiment, the feature maps are losslessly encoded via entropy coding. In this case, encoding the feature maps affects the bit rate but not the distortion of the encoded image. According to another embodiment, if the feature maps are lossy encoded, encoding the feature maps affects not only the bit rate but also the distortion.

[0096] Once training is complete, the network has B parameters Oc b is encoded either losslessly (the neural network ARM' is identical to ARM) or lossily (the network ARM' can be slightly different from ARM).

[0097] FIG. 5 is a flowchart illustrating an example of an encoding method that may be implemented by the encoding device of FIG.

[0098] According to this embodiment, the signal is a two-dimensional image, and therefore each sample to be coded has coordinates (x n ,y n ) pixel P n is.

[0099] The encoding occurs in two main phases.

[0100] In the first phase, called the construction phase, the input signal I(P n ) to optimize the overall cost function. i and parameter W k and optionally O b Training is performed to determine the values ​​of the parameters of the synthetic neural network MLP and the feature map FM. i This is followed by updating the values ​​of , and optionally the parameters of the predictive neural network ARM. As known in the art, the cost function can be of the bitrate distortion type, or of the bitrate, distortion or perceptual type. To measure the bitrate R, the map FMi The encoding of the parameter W must be simulated and then the associated encoding bitrate (size of stream B1) must be measured. k and / or O b The encodings of W are not simulated because their influence is smaller than that of the feature maps. k and / or O b is also simulated and the associated bit rate (size of stream B2) is measured. To measure the distortion D, it is necessary to simulate the encoding and then decoding of at least one part of image I, so that at least one pixel P' resulting from the encoding and then decoding simulation is measured. n Then, the input image I(P n ) and the encoded and then decoded image I(P' n ) and the corresponding part of

[0101] Then, during the second phase, called the encoding phase, the map FM i and parameter W k and optionally O b is encoded before being transmitted or stored, and the encoded value FMc i and Wc k (and optionally Oc b ) are generated by the input signal I(P n ) to form a compressed representation of

[0102] The steps of the method according to one embodiment of the present invention will now be described.

[0103] During step E20, a number N of samples P n The signal to be coded I(P n ) is given as input to the method.

[0104] During step E21, a first group of M maps FM iare initialized. Then, the parameters W of the synthetic neural network MLP are k and Map FM i and optionally the values ​​of the parameters O of the predictive neural network b must be optimized during the construction phase.

[0105] According to one embodiment, the map FM i is the input signal I(P n ), and therefore each has the same resolution as the encoded sample P n Contains the same number of values ​​N as

[0106] According to one embodiment, the map FM i The resolution of the input signal I(P n ) and therefore contains N' values ​​less than N, at least one of which is coded. i has the resolution of the image, and each subsequent map has half the resolution of the previous map.

[0107] According to one embodiment, the plurality of maps FMi are n ) has the same resolution less than the resolution of

[0108] According to one embodiment, the map FM i is the transformed feature map FMS i In this embodiment, the feature vector is preferably extracted from the transformed map of the second group, rather than directly from the map of the first group. Thus, in this embodiment, the feature vector is extracted indirectly from the map of the first group. The map of the second group is not encoded and is only used to construct the feature vector.

[0109] According to one embodiment, the map FM i is initialized to a predetermined constant value.

[0110] According to another embodiment, the feature map is initialized with a set of random real numbers.

[0111] According to one embodiment, one or more maps FME forming an additional group of L additional feature maps l are generated and added to the first group. They are used to construct the feature vector but are not stored or transmitted.

[0112] First group feature map FM i is subsequently updated or improved during a step E22 by updating the module MAJ of the encoder during the learning phase.

[0113] During step E23, the first group of maps FMi l are coded by the module FMC of the encoder. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is the actual coding, the coded values ​​forming the stream B1. The simulation can be identical to the actual coding, but it can also be different (for example simplified). For this coding, a technique is used to predict the values ​​of the feature maps via their neighbours, as will be described for example with reference to FIG. 9. In one embodiment, the structure and parameters O of the predictive neural network are calculated. b are initialized, for example, during the first iteration of this step. These parameters are then updated or refined during the construction phase during subsequent iterations of the method.

[0114] In one embodiment, the map FM i are coded in the order (FM1, FM2, ..., FM4), with the variables of each map coded in a predetermined order, for example lexicographical order. Each map is then subjected to entropy coding, which generates a compressed stream B1, the bit rate of which is subsequently measured during step E29.

[0115] During step E24, according to one embodiment, a first group of M maps FM iThe second group is transformed by the module SE and maps FMS with the resolution of the images of the input sequence. i Generate.

[0116] According to one embodiment, M map FMSs i is generated.

[0117] According to one embodiment, each map FM i Map FMS i is converted to

[0118] According to one embodiment, at least one map FM i has a resolution lower than the resolution of the image of the signal to be coded, and the conversion operation includes upsampling, so that the converted map FMS i contains the same number of samples as the number of images in the sequence. Upsampling is done by mapping FMS to achieve the image resolution of the input sequence. i It can be simple (by replicating the nearest neighbor) or it can involve interpolation (linear, polynomial, filtering, etc.).

[0119] During step E25, the converted map FM i or optionally FMS i and optionally additional maps FME i is extracted by the module XTR. This extraction is performed by the samples P n coordinates (x n ,y n ) It can also be implemented as a function of the resolution of the map of interest.

[0120] According to one embodiment, the feature vector Z n results directly from this extraction.

[0121] The samples to be coded are processed sequentially, for example from n=1 to n=N.

[0122] According to one embodiment, during step E26, the feature vector Z n is the coordinate of the input signal (x n ,y n ) for each sample P n About Map FM i or FMS i and optionally FME l The processing is carried out by the module TT from the values ​​extracted from the map FM i The value or component vector Z extracted from n The processing may include other operations such as filtering, scaling, application of any function, preferably a monotonic function, etc.

[0123] In one embodiment, Z n is the input map FM i or FMS i (and optionally FME l ) contains as many values ​​as there are in J, where J=M(+L).

[0124] In one embodiment, Z n is the current pixel P as will be explained with reference to Figure 6. n coordinates (x n ,y n ) Map FM i or FMS i (and optionally FME l ) values ​​form a J-tuple (z1,z2,...,z J )

[0125] In one embodiment, Z n is the map FM in coordinates which may vary depending on the map i (and optionally FME l ) is a J-tuple constructed from values ​​sampled from the map FM i (and / or FME l ) have different resolutions due to being downsampled, the coordinates are adapted (by scaling) to fit the resolution of each map.

[0126] In one embodiment, Z n is a method for generating a map FM by applying a process to one or more values ​​of the map, for example, by filtering neighboring values ​​of a targeted value in the map. i (and optionally FME l ) is a J-tuple constructed from values ​​sampled from the input signal. For example, a map FM i Current sample P n About this map FM i or FME l Vector Z related to n The final value of element i of (z i ) to get the coordinates (x n ,y n ), (x n -1,y n ), (x n ,y n -1) and (x n -1,y n −1) and process these values ​​(filter, average, interpolate, etc.). According to another example, it is possible to map FM i So, this map FM i or FME l Vector Z related to n The final value of element i of (z i ) to get the coordinates (x n / 2,y n / 2), (x n / 2-1,y n / 2), (x n / 2,y n / 2-1) and (x n / 2-1,y n / 2-1) and can process these values ​​(filter, average, interpolate, etc.).

[0127] During step E27, vector Z n is processed by the synthetic neural network MLP to generate the sample P n , i.e., according to one embodiment, the sample P′n (encoded and then decoded sample P n ) produces as output an (R,G,B) triplet of

[0128] The structure and parameters of the synthetic neural network W k are initialized, for example, during the first iteration of this step. These parameters are then updated or refined during subsequent iterations of the method, during the construction phase.

[0129] According to one embodiment, the parameters of the synthetic neural network and / or the predictive neural network are initialized with predetermined values ​​that are known to produce satisfactory results (e.g., after training on a corpus of images).

[0130] According to another embodiment, the parameters of the synthetic neural network and / or the predictive neural network are initialized with a set of random values.

[0131] During step E28, the parameters W of the synthetic neural network MLP are k and the parameters O of the predictive neural network ARM, if any. b are quantized and coded. During the construction phase, this operation is a coding simulation. During the coding phase, this operation is the actual coding, with the coded values ​​forming the stream B2. The simulation can be identical to the actual coding, but it can also be different (for example simplified). For this, any known technique can be used, for example the neural network coding standard proposed in part 17 of MPEG-7, also called neural network representation or NNR. In this case, the coding uses weights W k and optionally O b Note that you must be selective about the amount of scaling you provide.

[0132] During step E29, the performance measures are evaluated.

[0133] For this purpose, the first group of feature maps and the associated coding simulation bit rate (map F i ) and optionally a coding simulation bit rate associated with the parameters of the neural network (parameters W k and optionally O b The measured values ​​are used to simulate the stream B2 by encoding the

[0134] According to one embodiment, the cost function is of the bitrate-distortion type, denoted (D+L*R), where D is for example the squared error measured between the input signal and the decoded signal (or the error measured on a subset of the signal's samples). According to another example, D is calculated from a perceptual function such as SSIM (Structural Similarity) or MSSSIM (Multi-Scale Structural Similarity). According to one embodiment, R is the simulated bitrate of stream B1, and according to another embodiment, R is the overall bitrate used to code this image, i.e. the sum of the simulated bitrates of B1 and B2. L is a parameter that adjusts the bitrate-distortion compromise. Other cost functions are possible.

[0135] The performance measure remains unsatisfactory until the cost function reaches a minimum and the method is repeated from step E22. This minimization can be performed via known mechanisms such as gradient descent, with parameters being updated during step E22 for the values ​​of the feature maps and for the parameters of one or more networks during steps E23, E27.

[0136] During step EF, training stops when the cost function reaches a minimum. The parameters of the synthetic neural network (W k ) and feature map (FM i If an encoded version corresponding to the last simulation of W is available, streams B1 and B2 can be formed from it. According to another embodiment, the updated parameters of the synthetic neural network (W k) and feature map (FM i ) and optionally those of the predictive neural network (O b ) is performed during this step to obtain the coded parameters Wc forming the streams B1 and B2. k (Optionally O b ) and FMc i Streams B1 and B2 can be concatenated to generate a final stream. According to one embodiment, stream B2 of coded parameters of one or more neural networks is stored or transmitted before stream B so that it can be decoded before stream B1. Figure 6 shows a diagram of a decoding method used in one embodiment of the present invention.

[0137] In this embodiment, four generated maps FM i In the preferred embodiment, there are seven maps.

[0138] The first map FM1 is the image I(P n ), and therefore contains X × H variables, where W represents the width of the image in pixels and H represents its height. The second map FM2 has half the resolution of map FM1 (in each dimension). Each additional map has half the resolution of the previous second map. This structure allows for a reduction in the number of variables in the feature maps, thus facilitating encoding and learning while minimizing encoding costs.

[0139] Map FM2 is upsampled by module SE by a factor of 2 in each dimension in the manner illustrated with reference to Figure 6. Map FM3 is upsampled by a factor of 4 in each dimension and map FM4 is upsampled by a factor of 8 in each dimension.

[0140] Generated Map FMS i is the image I(P n), and therefore each contains W×H values, where W represents the width of the image in pixels and H represents its height (N=W×H).

[0141] According to this embodiment, the layer FMS i is quantized by the module SE.

[0142] Other types of structures may be used between maps, for example reduction ratios other than half (such as 1 / 4 or 1 / 3).

[0143] In one variant, shown in dashed lines, there are five feature maps, and an additional map, FME0, is introduced, which is neither coded nor transmitted. This additional map typically contains data that may assist the network MLP in the task of reconstructing the signal. The added map may therefore be one or more maps from the following non-limiting list: - a map containing, for each point, the x coordinate of this point, - a map containing, for each point, the y coordinate of this point, - a map containing the location coding of each point (for example, as explained on the following website: https: / / skosmos.loterre.fr / P66 / fr / page / -K0D65X2X-X), - an image separate from the image being processed and capable of providing information about the image to be coded, for example a map representing a previously processed image if the current image is part of a sequence of images to be coded, such as a video, a set of medical images, a multi-view representation, etc. an image separate from the image being processed and capable of providing information about the image to be coded, for example a map representing the feature maps of a previously processed image if the current image is part of a sequence of images to be coded, such as a video, a set of medical images, a multi-view representation, etc. - Already decoded samples of the same map, e.g. a map containing the values ​​of the previous samples in decoding order.

[0144] In this embodiment, the vector Z nis the current pixel P n coordinates (x n ,y n ) Map FMS i The map FMS is a 4-tuple (z1...z4) formed from values ​​extracted from i A vector Z consisting of the sampled (quantized) values ​​of n is processed by the synthetic neural network MLP to output a second vector, which, according to this example, is the encoded and then decoded pixel P' n The triplet is the (R,G,B) triplet of the color components (R',G',B') at the position (x n ,y n ), the decoded image I(P' n )

[0145] In another embodiment not shown, the vector Z n is the layer FM at the recalculated position as a function of the map size. i , and the extracted values ​​are then optionally processed and quantized after extraction.

[0146] According to a variant shown as a dashed line, the vector Z n is a 5-tuple (z0...z4), where the value z0 is extracted from the additional map FME0.

[0147] FIG. 7 is a flowchart illustrating an example of a decoding method that may be performed by the decoding device of FIG.

[0148] During step E30, streams B1 and B2 are extracted from the coded stream. The streams B1 and B2 are coded representations of the maps FMc of the first group. i and the coded representation of the parameters Wc k , optionally an encoded representation of the parameters Oc b Each includes:

[0149] During step E31, M maps FMd i is the value FMci , which uses techniques to predict feature map values ​​via their neighbors, e.g., as described with reference to Figure 10. In one embodiment, the map FMd i are decoded in the order (FMd1, FMd2, ..., FMd4), and the values ​​of each map are decoded in a predetermined order, for example, lexicographic order.

[0150] According to the embodiment as described for the encoder: - Map FMd i is the signal to be reconstructed I(Pd n ), i.e. it contains N values. - Map FMd i is the signal to be reconstructed I(Pd n ) or less resolution. - Multiple maps FMd i has the same resolution, which is lower than the resolution of the signal.

[0151] During step E32, according to one embodiment, one or more maps FME' forming an additional group of L additional feature maps are l is generated to fill the first group. Map FME' l are not decoded, but are generated by the decoder in the same way as they are generated in the encoder. They may typically assist the network MLP' in the task of reconstructing the signal. The non-limiting list of possible additional feature maps described with reference to FIG. 6 for the encoder also applies in this case.

[0152] During step E33, according to one embodiment, the first group FMd i The second group of maps FMS' has the resolution of the input image. i This step is similar to step E24 described for the encoder with reference to Figure 5, and its implementation also applies here.

[0153] In particular, according to one embodiment, M maps FMS' i is generated.

[0154] According to one embodiment, each map FMd i Map FMS i is converted to

[0155] According to one embodiment, at least one map FMd i has a resolution lower than the resolution of the plurality of images of the image to be coded, and the transformation operation is performed to obtain a transformed map FMS' i contains the same number of samples as the input image. Upsampling involves mapping FMS' to achieve the resolution of the input image. i This can be simple (nearest neighbor replication) or can involve interpolation (linear, polynomial, filtering, etc.).

[0156] The transformation may optionally include dequantization of the extracted values ​​if desired, although dequantization is not required.

[0157] During step E34, the transformed map FMd i or optionally FMS' i and optionally an additional map FME' i The value of is extracted by the module XTR'. This extraction is carried out by the sample P of the input signal. n coordinates (x n ,y n ) as a function of the resolution of the map of interest. This step is similar to step E25 described for the encoder with reference to FIG. 5, and those embodiments are also applicable here. In particular, according to one embodiment, the feature vector Z n results directly from this extraction.

[0158] In one embodiment, Zd n is the current pixel P as shown with reference to FIG. n coordinates (x n ,yn ) Map FMd i or FMS' i The J-tuple (z1, z2,..., z J )

[0159] The samples to be decoded are processed sequentially, for example from n=1 to n=N.

[0160] According to one embodiment, during step E35, the vector Zd n is the decoded coordinate (x n ,y n ) for each sample Pd n For coordinates (x n ,y n ) as a function of the first group map FMd i Or the second group's map FMS' i and optionally additional groups of maps FME' i This step is similar to step E26 described for the encoder with reference to Figure 5, and the described embodiment also applies here. The extraction can be carried out by means of the extracted values ​​or the formed vector Zd n This may include inverse quantization of

[0161] During step E36, the parameters Wd of the synthetic neural network MLP' k and optionally the parameters Od of the predictive neural network b is the value Wc of stream B. k and Oc b, which is generated by decoding the feature maps. For this purpose, any known decoding technique corresponding to the encoding technique used by the encoder can be used. The synthetic neural network MLP' is similar to the synthetic neural network MLP, i.e., it has the same structure and contains the same parameters as the nearest neighbor encoding, which may be lossy or lossless. Similarly, if a predictive neural network ARM' is used to decode the feature maps, the predictive neural network ARM' is similar to the predictive network ARM, i.e., it has the same structure and contains the same parameters as the nearest neighbor encoding, which may be lossy or lossless.

[0162] According to one embodiment, stream B2 is decoded before stream B1 so as to have a synthesis neural network MLP' and optionally a prediction neural network ARM' before starting the decoding of the samples.

[0163] During step E37, vector Zd n is processed by the synthetic neural network MLP' to obtain the decoded sample Pd n , according to one embodiment the position (x n ,y n ) and the decoded image I(Pd n ) as output. This step is similar to step E27 described for the encoder with reference to FIG.

[0164] If all samples of a signal are processed, for example, the image I(Pd n ) is available.

[0165] FIG. 8 shows a diagram of the decoding method used in one embodiment of the present invention.

[0166] In this embodiment, four decoded maps FMd i In the preferred embodiment, there are seven maps.

[0167] In this embodiment, the first map FMd i has the same resolution as image I and therefore contains W × H variables, where W is the width of the image in pixels and H is its height. l (in each dimension). Each additional map has half the resolution of the previous map. This structure allows us to reduce the number of variables in the feature maps, thus facilitating decoding while minimizing decoding cost.

[0168] Map FMd2 is upsampled by a factor of 2 in each dimension by any upsampling method within the capabilities of one skilled in the art. Map FMd3 is upsampled by a factor of 4 in each dimension, and map FMd4 is upsampled by a factor of 8 in each dimension.

[0169] Map FMS' i have the same resolution as the image to be decoded and therefore each contain W×H variables, where W is the width of the image in pixels and H is its height.

[0170] In this embodiment, the vector Zd n is the current pixel Pd n coordinates (x n ,y n ) Map FMS' i The vector Zd is a 4-tuple (z1...z4) formed from the values ​​of n is optionally dequantized and then processed by a synthetic neural network MLP′ to obtain the decoded samples Pd n The (R,G,B) triplet is generated as output, which represents the coordinates (x n ,y n ) and the decoded image I(Pd n )

[0171] According to one variant, shown as a dashed line, there are five maps and an additional map FME'0 is introduced. In this embodiment, the vector Zd n is a 5-tuple.

[0172] FIG. 9 is a flowchart illustrating a method for encoding a feature map, which may be implemented by the encoding device of FIG. 1 and the encoding method of FIG.

[0173] These steps form sub-steps of step E23 described above with reference to figure 5. These steps use the neighbourhood values ​​to generate a feature map FM of the first group being processed. i The current value of V n is intended to encode

[0174] During substep E231, the value V n A neighborhood vector (C n ) are established. As will be explained below with reference to FIGS. 11 and 12, these neighborhood values ​​are calculated using a number of M maps FM i This neighborhood vector is formed by C values ​​or data corresponding to neighboring values ​​(e.g., C=10). These values ​​must be known by the encoder and decoder, and therefore the value V n must be in the causal neighborhood of

[0175] According to a first embodiment, these values ​​are used to determine the context of an entropy encoder for decoding the current value during step E234. This encoder may be a CABAC (Context-Adaptive Binary Arithmetic Coding) encoder. This type of encoder is well known to those skilled in the art. It is used in particular in the H.265 / HEVC video compression standard. It is an arithmetic encoder with lossless compression. It decomposes all non-binary symbols into binary symbols. Then, for each bit, the encoder selects the most suitable probability model and optimizes the probability estimate using the context. This context may be defined by information about neighboring elements. Arithmetic coding is then applied to compress the generated data. As known to those skilled in the art, there are several ways to generate context information using neighborhood vectors. For example, the number of neighboring values ​​different from zero can be counted and a context can be associated with each number. Alternatively, a comparison can be made between several neighboring values, and a given context can be associated with an ordering configuration between the neighboring values, for example by sorting the neighboring values ​​in ascending order and associating a context with each possible ordering.

[0176] In a second embodiment, during step E232, a neighborhood is used to predict the current value from an autoregressive model. Note that an autoregressive model predicts a sample from a series by its past values. In this embodiment, the past values ​​are formed by a context, and the difference between the predicted variable and the actual value is quantized and then entropy coded during step E234.

[0177] In a third embodiment, as illustrated with reference to Figure 12, during step E233, a predictive neural network ARM is used to predict the statistical characteristics of the variables to be coded. The neighbourhood vectors are provided as input to the network ARM in order to output a prediction of the current value. According to one embodiment, the network ARM is based on a function f, as explained with reference to Figure 4. ψIt acts as a feature map generator and provides a set of statistical parameters (mean, variance, median, etc.) that are used to entropy code the current value. The role of this module ARM is to reduce the bit rate required for encoding the feature maps. i All values ​​V to be encoded n According to another embodiment, a predictive neural network ARM is used to generate expected probabilities (pr) of the possible values ​​of the current sample. Entropy coding is applied to this probability (as known for Huffman or arithmetic entropy coding).

[0178] In the fourth embodiment, each feature map is divided into blocks having a predetermined size, and the encoding of each block includes a transform (e.g., DCT, discrete cosine transform, Haar transform, etc.), and the transformed values ​​are encoded by entropy coding.

[0179] In the fifth embodiment, each feature map is divided into blocks having a predetermined size, and each block is represented by a lattice vector quantization type product code.

[0180] Once the method is complete, the processing map FM i The current encoded value of Vc n is encoded.

[0181] FIG. 10 is a flowchart illustrating a method for decoding feature maps, which may be implemented by the decoding device of FIG. 2 and the decoding method of FIG.

[0182] These steps form sub-steps of step E31 described above with reference to figure 7. The purpose of these steps is to use the neighbourhood values ​​to generate the feature map FMd of the first group being processed. i The current value of Vd n The purpose is to decrypt the

[0183] During substep E311, the value Vd n A neighborhood vector (Cn ) is established. This step is similar to step E231 described above and the same implementation applies here. This neighbourhood vector is then calculated by dividing a number of M maps FMd i The value Vd is composed of C values ​​or data (for example, C=10) corresponding to neighboring values ​​in the same map and / or different maps. n These values ​​that are within the causal neighborhood of are known to the decoder.

[0184] According to a first embodiment, these values ​​are used to determine a context for the entropy encoder in order to decode the current value during step E314. This decoding is similar to that used in an encoder, for example a CABAC encoder. The use of neighborhoods to generate context information is similar to that selected in the encoder. For example, the number of neighborhood values ​​different from zero can be counted and a context can be associated with each number. Alternatively, a comparison can be made between several neighborhood values ​​and a given context can be associated with an ordering configuration between the neighborhood values, for example by sorting the neighborhood values ​​in ascending order and associating a context with each possible ordering.

[0185] In a second embodiment, during step E312, the neighborhood is used to predict the current value from an autoregressive model. Note that an autoregressive model predicts a sample from a series by its past values. In this embodiment, the past values ​​are formed by a context, and the difference between the predicted variable and the actual value is quantized and then entropy coded during step E314.

[0186] As shown with reference to Figure 12, in a third embodiment, during step E313, a predictive neural network ARM' is used to predict the statistical characteristics of the variable to be decoded. The neighbourhood vectors are provided as input to the network ARM' in order to output a prediction of the current value. According to one embodiment, the network ARM is configured to predict a function f defined by a set of statistical parameters (mean, variance, median, etc.) used to entropy code the current value, as explained with reference to Figure 4. ψ According to another embodiment, a predictive neural network ARM' is used to generate expected probabilities (pr) of the possible values ​​of the current sample. The entropy decoding is adapted to these probabilities (as is known for Huffman or arithmetic entropy coding). If the coding is performed losslessly, the network ARM' is identical to the network ARM.

[0187] In a fourth embodiment, each feature map is divided into blocks having a predetermined size, and decoding each block involves entropy decoding the values, which then undergo an inverse transform (e.g., inverse DCT, discrete cosine transform, inverse Haar transform, etc.).

[0188] In the fifth embodiment, each feature map is divided into blocks having a predetermined size, and each block is decoded by a lattice vector quantization type product code to generate a decoded block.

[0189] Once the method is complete, the map FMd in progress i The current decoded value Vd n is decrypted.

[0190] FIG. 11 shows a diagram of a method for encoding or decoding feature maps according to one embodiment.

[0191] In this figure, the current (i.e., encoding or simulating encoding) (or FMd l , i.e., the one being decoded) feature map FM l coordinates (x n ,y n) the current value V n (or Vd n The encoding (or decoding) of layer FM uses the context information of its own map and the previous map FM2 (or FMd2). l (or FMd l ) coordinates (x n ,y n -1), (x n ,y n -2), (x n -1,y n -1), (x n -1,y n ), (x n -1,y n +1), (x n -2,y n ) and the coordinates (x n -1,y n -1), (x n -1,y n ), (x n -1,y n +1), (x n ,y n -1), (x n ,y n ), (x n ,y n +1), (x n +1,y n -1), (x n +1,y n ), (x n +1,y n +1) are used to determine the neighborhood for encoding (or decoding) the current value. These values ​​are all available to the encoder and decoder, and are used in one of the embodiments described above with reference to FIG. 9 (or FIG. 10). n (or Vd n ) neighborhood vector C n (or Cd n ) is formed.

[0192] FIG. 12 shows a diagram of another method for encoding or decoding a feature map according to one embodiment.

[0193] In this figure, the current feature map FM i(or FMd i ) coordinates (x n ,y n ) the current value V n (or Vd n ) uses the context information of its own map. The values ​​shown in grey are used to determine the neighborhood for encoding (or decoding) the current value. These values ​​are all available to the encoder and decoder and are the values ​​V n (or Vd n ) neighborhood vector C n (or Cd n ) is formed.

[0194] In the illustrated embodiment, the neighborhood vectors are extracted by a module CTX (or CTX') of the encoding module FMC (or FMD) and then applied to the input of a predictive neural network ARM (or ARM') which is used to predict the statistical features (μ, σ) or probabilities (pr) of the values ​​to be encoded (or decoded) by the entropy encoder CE (or DE).

Claims

1. The number of samples to be coded (P ｎ ) containing the signal (I(P ｎ )) a method for encoding a signal, the method comprising the steps of: a construction step, said signal (I(P ｎ )) ｉ a substep (E21, E22) of constructing a first group of - the position (x ｎ , y ｎ ) of the signal to be coded, which is associated with the current sample (P ｎ ) for at least one sample, called - the current sample (P ｎ ) at the position (x ｎ , y ｎ ) of the first group as a function of the feature map (FM ｉ ) to the feature vector (Z ｎ ) a sub-step (E25), a vector (P') representing the decoded value of the current sample ｎ ) to provide the parameter (W ｋ ) is defined by a set of ｎ a substep (E27) of processing a substep (E22, E27) of updating at least one value of one of said feature maps of said first group and / or at least one parameter of said network as a function of the coding performance measure; A construction step including - the feature map (FM ｉ ) of one of said feature maps, ｎ a step (E23, E27, EF) comprising, for at least one value called ∑i=1, ∑j=2, ∑j=3, ∑j=4, ∑j=5, ∑j=6, ∑j=7, ∑j=8, ∑j=9, ∑j=10, ∑j=11, ∑j=12, ∑j=13, ∑j=14, ∑j=15, ∑j=16, ∑j=17, ∑j=18, ∑j=19, ∑j=20, ∑j=21, ∑j=22, ∑j=23, ∑j=24, ∑j=25, ∑j=26, ∑j=27 the parameters of the synthetic neural network (W ｋ ) encoding the set of A method comprising:

2. The current value (V ｎ ) the feature map (FM) of the first group ｉ ) to the neighboring vector (C ｎ ) (E231), - the current value (V ｎ ) to provide a prediction of the parameter (O ｂ ) is defined by a set of vectors (C ｎ a substep (E233) of processing a substep (E22, E23, E28) of updating at least one parameter of said prediction network as a function of said coding performance measure, the parameters of the prediction network (O ｂ ) encoding the set of 2. The encoding method of claim 1, comprising:

3. 3. The method of encoding a sequence of images according to claim 1 or 2, further comprising the step of generating a feature map (FMS) having the resolution of the images of the input sequence. ｉ ) to obtain a second group of feature maps (FM ｉ ) a step (E24) of transforming a first group of The feature vector (Z ｎ ) is the feature map (FM ｉ ) the transformed feature map (FMS ｉ )

4. The feature map (FM) of the first group ｉ 4. The method of claim 3, wherein at least one of the first and second inputs has a lower resolution than the resolution of the signal to be coded, and the conversion operation involves upsampling.

5. The feature vector (Z ｎ ) the construction of - the current sample (P ｎ ) at the position (x ｎ , y ｎ ) of the first group as a function of the feature map (FM ｉ ) and a sub-step to extract multiple values ​​of - processing said extracted values ​​to obtain said feature vectors 2. The encoding method of claim 1, comprising:

6. Feature Map (FME) ｌ 6. The method according to claim 1, further comprising a step (E21, E22) of constructing additional groups of feature maps of said additional groups, and wherein said feature vector is also constructed from said feature maps of said additional groups.

7. The number of samples to be decoded (Pd ｎ 1. A method for decoding a signal comprising: - a feature map (FMd) representing said signal ｉ ) of one of said feature maps, ｎ a step (E31) comprising, for at least one value called ∑ ... - parameters (Wd ｋ ) and - the position (x ｎ , y ｎ ) of the current sample (Pd ｎ ) for at least one sample, called the position (x ｎ , y ｎ ) as a function of the feature map (FMd ｉ ) to the feature vector (Zd ｎ ) (E34), - the current sample (Pd ｎ ) to provide a vector representing the decoded values ​​of the decoded parameters (Wd ｋ ) to generate the vector (Zd ｎ ) processing step (E37); A method comprising:

8. The current value (V ｎ ) - Parameters (Od ｋ a substep (E36) of decoding the set of the feature map (FM) of the first group ｉ ) to the neighboring vector (C ｎ ) (E311), - the current value (V ｎ ) using the predictive neural network (ARM′) to provide a prediction of the neighborhood vector (C ｎ ) processing substep (E313) 8. The decoding method of claim 7, comprising:

9. A feature map (FMS') having the resolution of the input signal ｉ ) to obtain a second group of decoded feature maps (FMd ｉ ) a step (E33) of transforming a first group of The feature vector (Zd ｎ ) is the decoded feature map (FMd ｉ ) the transformed feature map (FMS′) of the second group obtained from ｉ 9. A method according to claim 7 or 8, characterized in that the decoding is constructed from:

10. The feature map (FMd) of the first group ｉ 10. The decoding method of claim 9, wherein at least one of the first and second inputs has a lower resolution than the resolution of the signal to be decoded, and the conversion operation comprises upsampling.

11. The feature vector (Zd ｎ ) is constructed by the current sample (Pd ｎ ) and the same position (x ｎ , y ｎ ) in the at least one feature map (FMd ｉ , FME' ｉ 11. A decoding method according to any one of claims 7 to 10, characterized in that it comprises a substep (E34) of extracting the value of .

12. The feature vector (Zd ｎ ) the construction of - the current sample (P ｎ ) at the position (x ｎ , y ｎ ) as a function of the feature map (FMd ｉ ) and a sub-step to extract multiple values ​​of a substep (E35, TT') of processing said extracted values ​​in order to obtain said feature vectors; 8. The decoding method of claim 7, comprising:

13. Feature map (FME' ｌ 13. The decoding method according to claim 7, further comprising a step (E32) of constructing additional groups of feature maps, wherein the feature vector is also constructed from the feature maps of the additional groups.

14. The number of samples to be coded (P ｎ ) containing the signal (I(P ｎ )) in a device for encoding said signal (I(P ｎ )) ｉ ) to construct a first group (GEN, MAJ), - the position (x ｎ , y ｎ ) of the signal to be coded, which is associated with the current sample (P ｎ ) for at least one sample, called - the current sample (P ｎ ) at the position (x ｎ , y ｎ ) as a function of the feature map (FMS ｉ ) to the feature vector (Z ｎ ) (XTR), the decoded value of the current sample (P' ｎ ) vector (S ｎ ) to provide the parameter (W ｋ ) is defined by a set of ｎ ) (MLP), updating at least one value of one of the feature maps of the first group and / or at least one parameter of the synthesis network (MAJ, NNC) as a function of the coding performance measure; - the feature map (FM ｉ ) of one of said feature maps, ｎ ), entropy coding of said value as a function of values ​​in at least its neighborhood (FMC), the parameters of the synthetic neural network (W ｋ ) encoding the set (NNC) 20. A device configured to perform the steps of:

15. The number of samples to be decoded (Pd ｎ 1. A device for decoding a signal comprising: - a feature map (FMd) representing said signal ｉ ) of one of the feature maps, ｎ ), comprising entropy decoding of said value as a function of values ​​in at least its neighborhood; - parameters (Wd ｋ ) (NND), - the position (x ｎ , y ｎ ) of the current sample (Pd ｎ ) for at least one sample, called the position (x ｎ , y ｎ ) as a function of the feature map (FMd ｉ ) to the feature vector (Zd ｎ ) (XTR'), - the current sample (Pd ｎ ) to provide a vector representing the decoded values ​​of the decoded parameters (Wd ｋ ) to generate the feature vector (Zd ｎ ) (MLP') 20. A device configured to perform the steps of:

16. A computer program comprising instructions for carrying out the steps of the encoding or decoding method according to claim 1 or 7, when said program is executed by a computer.

Images