Method and apparatus for encoding and decoding image sequences

Abstract

The present invention relates to a method and apparatus for encoding and decoding a sequence of images. The decoding method comprises the following steps for a current image: obtaining at least one feature map from a first set of decoded feature maps representing at least one image different from the current image; -decoding a second set of feature maps representing the current image, the step comprising: for at least one value, entropy decoding according to a neighborhood pair comprising at least one decoded value extracted from the at least one reference map:-decoding a synthetic neural network; for at least one sample point of the current image: constructing a feature vector from a second set of feature maps according to the position of said current sample point; and-processing the vector using the synthetic neural network in order to provide a decoded value of the sample point.

Description

Background Technology

[0001] This invention relates to the general field of encoding and decoding digital image sequences. More specifically, this invention relates to compressing digital video.

[0002] Digital video is typically source-coded to achieve compression, thereby limiting the resources required for its transmission and / or storage. Many coding standards exist, such as those developed by the ITU / MPEG organization (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) and their extensions (MVC, SVC, 3D-HEVC, etc.).

[0003] As video formats continue to evolve to achieve higher compression rates and match diverse expected formats and communication networks, the possibilities for prediction are constantly increasing, and conventional encoding and decoding algorithms are becoming increasingly complex.

[0004] In addition to the conventional methods proposed by compression standards (MPEG, ITU), artificial intelligence-based methods (especially neural methods) are also emerging.

[0005] Some of these neural methods can be considered as simple extensions of the competing concepts among the aforementioned compression techniques (such as the competition between prediction and video coding transform methods).

[0006] Other methods use the concept of an "autoencoder." An autoencoder is a learning algorithm based on artificial neural networks that allows the construction of new representations of a dataset. The architecture of an autoencoder consists of two parts: an encoder and a decoder. The encoder is formed by a set of neuron layers that process the data to construct new representations, called "encoded" representations or "latent representations." The decoder's neuron layers then receive these representations and filter them to attempt to reconstruct the original data. The difference between the reconstructed data and the initial data allows measurement of any errors produced by the autoencoder. Training involves changing the parameters of the autoencoder to reduce the reconstruction error measured at individual samples in the dataset. While these autoencoder-based systems offer superior performance, they come at the cost of a significant increase in memory footprint and complexity compared to conventional methods, such as those proposed by compression standards. These systems may have millions of parameters, and decoding a single pixel may require approximately a million MAC (multiply-accumulate) operations. This makes such decoders far more complex than conventional decoders, potentially hindering the application of learning-based compression.

[0007] Recently, Emilien Dupont et al. described a simple neural network-based image coding technique in their paper "Compression with Implicit Neural Representations" (arXiv: 2103.03123). The proposed coding technique involves adjusting the neural network for the image, quantizing the network weights, and transmitting the quantized weights. During decoding, the neural network is evaluated at each pixel location to reconstruct the image. However, this technique remains inefficient in terms of compression and involves encoding images independently within the video.

[0008] Therefore, a solution is needed for encoding / compressing image sequences simply and efficiently. Summary of the Invention

[0009] The subject of this invention is a method for encoding an image sequence comprising a plurality of sample points to be encoded, the method comprising the following steps:

[0010] - A construction step, which includes the following sub-steps for at least one current image in the sequence:

[0011] - Construct a first set of feature maps representing at least one image, referred to as a reference image, that is different from the current image;

[0012] - Construct a second set of feature maps representing the current image;

[0013] - For at least one sample point of the current image associated with a location in the current image, referred to as the current sample point:

[0014] - Construct a feature vector from the feature map in the second group based on the position of the current sample point;

[0015] - The feature vector is processed using an artificial neural network, called a synthetic neural network, defined by a set of parameters, in order to provide a vector representing the decoded value of the current sample point;

[0016] - Update at least one value of one feature map in the second group and / or at least one parameter of the network based on the coding performance metric;

[0017] - The step of encoding the second set of feature maps includes: for at least one value of a feature map called the current value in the feature maps called the current map, entropy encoding the value based on the neighborhood of the at least one value extracted from at least one map called the reference map in the first set of feature maps;

[0018] - The step of encoding the set of parameters of the synthetic neural network.

[0019] The subject of this invention is a method for decoding an image sequence comprising a plurality of samples to be decoded from a bitstream, the method comprising the following steps for at least one current image in the sequence:

[0020] - Obtain at least one feature map, referred to as the reference map, from a first set of decoded feature maps representing at least one image different from the current image;

[0021] - Decoding a second set of feature maps representing the current image, the step comprising: for at least one value called the current value in a feature map called the current map, entropy decoding the value based on a neighborhood including at least one decoded value extracted from the at least one reference map;

[0022] - Decode a set of parameters representing a synthetic neural network;

[0023] - For at least one sample point of the current image associated with a location in the current image, referred to as the current sample point:

[0024] - Construct a feature vector from the feature map of the second group based on the position of the current sample point; and:

[0025] - The vector is processed using the synthetic neural network to provide a vector representing the decoded value of the current sample.

[0026] Within the meaning of this invention, the term "encoding" should be understood to refer to operations involving representing a set of samples (or pixels) in a compact form, for example, transmitted via a digital bitstream. The term "decoding" should be understood to refer to operations involving processing a digital bitstream to reconstruct the decoded samples.

[0027] The term "image sequence" should be understood to mean, for example, multiple two-dimensional images ordered chronologically in the case of video. As another example, images can be views of the same scene displayed from multiple perspectives. As yet another example, images can be multi-view images from multiple times (immersive video).

[0028] The term "sample" should be understood as referring to a value taken from a sequence of images. Sampling a signal produces a discrete sequence of values ​​called samples. In the case of image signals, samples are known as pixels, which can be, for example, grayscale pixels represented by a single component, or color pixels conventionally represented by triplet values ​​(e.g., (R, G, B) or (Y, U, V)). The location of a sample is identified by its x and y coordinates in the image.

[0029] The term "feature map" should be understood as referring to a portion of an abstract representation of an image that contains multiple variable scalar data, also known as feature values, such as signed or unsigned real numbers or integers. Therefore, an abstract representation of an image can contain multiple feature maps. An abstract representation of an image can also correspond to a single feature map that includes a set of values ​​associated with a location.

[0030] The term "feature vector constructed from a feature map based on location" should be understood as meaning a vector formed by one or more (preferably discrete) elements or data constructed from the locations of the sample points being processed in the image, as determined by the positions of the feature maps. This feature vector is the feature vector applied to the input of a synthetic neural network. In the case of an image, such a vector can be constructed, for example, from multiple values ​​taken from the same x and y coordinates in each of these feature maps as the sample point to be encoded (or decoded).

[0031] The term "synthetic neural network" should be understood to refer to neural networks, such as convolutional neural networks, multilayer perceptrons, and LSTM (Long Short-Term Memory). A neural network is defined, for example, by multiple layers of artificial neurons and a set of activation functions, weighting functions, and addition functions (e.g., a layer can compute y = f(Ax + b), where y and b are N-dimensional vectors, x is an M-dimensional vector, A is an M × N-dimensional matrix, and f is the activation function).

[0032] The term "parameters of a neural network" should be understood to mean one of the values ​​that characterizes the neural network, such as the weights (filter coefficients, weights, biases, values ​​that affect nonlinear operations, etc.) associated with one of the neurons.

[0033] The term "processing using a synthetic neural network" should be understood as applying a function expressed by a neural network to an input feature vector to produce an output vector representing samples to be encoded (or decoded). This output vector may include one or more data representing the samples.

[0034] The term "performance metric" should be understood as a measure between at least one value of a sample to be encoded and the decoded value of said sample. This metric can evaluate, for example, distortion or perceptual error. This can be performed on one or more samples (e.g., the current sample or the current image, etc.). The metric can also include a measure of bitrate (particularly the bitrate associated with encoding a neural network and / or encoding a second set of feature maps). This metric can be a joint measure of bitrate and distortion, achieved by weighting bitrate and distortion. As is well known in the art, the value of this metric is typically minimized until a target value is reached.

[0035] The term "build step" should be understood as referring to the steps involved in building up the parameters representing the image before actually encoding them. The build sub-step can be repeated multiple times as needed to obtain acceptable performance metrics.

[0036] Generally, it is assumed that the steps of an encoding or decoding method should not be interpreted as being associated with a concept of chronological order. In other words, these steps may be performed in a different order than indicated in the independent encoding or decoding claims, or even simultaneously.

[0037] The encoding method according to the invention constructs encoding parameters from an input image sequence by training a neural network on feature vectors associated with the locations of the samples to be encoded in the input image. These feature vectors are constructed from feature maps, which may have the same resolution as the input image or may have a lower resolution. During training or construction, the parameters of the neural network and the values ​​of the feature maps are updated according to a performance metric, such as rate-distortion. When training is complete (i.e., the obtained performance metric is satisfactory), the actual encoding of the neural network parameters and the values ​​of the feature maps can be performed and stored or transmitted to the decoder.

[0038] Advantageously, the training process allows for refining the values ​​of the parameters and / or feature maps of the synthetic neural network until a sufficiently representative representation in terms of performance is obtained, for example, until the desired balance between the generated bit rate and the distortion experienced by the encoded input image is achieved. Training of the feature map values ​​and training of the parameters of the synthetic neural network can be joint training. Advantageously, the encoding method according to the invention allows for efficient signal compression.

[0039] Advantageously, the decoding method is simple because it only requires decoding the feature map of the synthetic neural network and the current image based on other decoded feature maps of other images in the sequence (e.g., previous images) to recover the decoded version of the current image.

[0040] Advantageously, by taking into account the coding neighborhood in the feature maps of other images in the sequence, redundancy present in the maps associated with different images is allowed, making the coding of the feature maps particularly efficient.

[0041] Advantageously, this synthetic neural network can have a very simple structure and few parameters.

[0042] In addition, decoding can be performed step by step, sample by sample.

[0043] According to an embodiment of the encoding or decoding method:

[0044] - The encoding method includes the following sub-steps for encoding the current value:

[0045] - Construct a neighborhood vector comprising at least one neighborhood value extracted from the at least one reference graph; and

[0046] - The neighborhood vector is processed using an artificial neural network called a predictive neural network, which is defined by a set of parameters, in order to provide a prediction of the current value;

[0047] - Update at least one parameter of the prediction network based on the coding performance metric;

[0048] - Encode the set of parameters of the prediction network.

[0049] - The decoding method includes the following sub-steps: decoding the current value of one of the feature maps:

[0050] - Decode a set of parameters representing a neural network known as a prediction neural network;

[0051] - Construct a neighborhood vector comprising the at least one decoded value extracted from the at least one reference graph; and

[0052] - The vector is processed using a predictive neural network to provide a prediction of the current value.

[0053] Advantageously, according to this encoding or decoding embodiment, the feature maps are efficiently compressed by a predictive neural network capable of predicting the values ​​of the maps based on their temporal neighborhood. The term "neighborhood vector" should be understood as meaning a vector composed of one or more elements or data constructed from feature maps (or reference maps) of other images in the sequence, preferably extracted from locations close to the current sample location (which is also the location of the current value in the current feature map). The neighboring location can indicate the value in any map associated with another image in the sequence within that set of feature maps (e.g., the value being processed is the top-left neighbor in the feature map of the previous image). This neighborhood vector is the neighborhood vector applied to the input of the predictive neural network. The term "prediction" should be understood as meaning at least one piece of data used to estimate the current value of the feature map, such as a probability, statistic, etc. Predictive neural networks trained on the maps of images in the sequence can efficiently represent these maps. Furthermore, the encoding cost is low.

[0054] According to the variant:

[0055] - At least one of the feature maps in the second set has a lower resolution than the image to be decoded, and the method involves upsampling. Advantageously, according to this embodiment, the compression of the feature maps is more efficient because at least one of the feature maps in the second set to be encoded (or decoded) contains fewer values ​​compared to the case with the resolution of the signal. For example, in the case of a digital image, one of the feature maps may have a 1 / 2 resolution, that is, it contains half the number of x and y values ​​as the number of samples contained in the input image (i.e., the total number is 1 / 4 of the number of values ​​in the feature map with the resolution of the image). On the other hand, the feature map used for synthesis may have the same resolution as the image. Therefore, in this case, the method includes at least one upsampling operation to obtain the same number of values ​​in the map as the number of samples contained in the input image (or the image to be decoded).

[0056] - At least one of the feature maps in the second group has the same resolution as the signal to be encoded (or decoded). Advantageously, according to this embodiment, at least one of the feature maps has the same resolution as the image to be encoded (or decoded), thereby achieving high fidelity and preserving the details of the original resolution of the image signal.

[0057] Constructing a feature vector involves a sub-step of extracting the values ​​of the feature map at the same locations as the current sample point in the image to be encoded or decoded. Advantageously, values ​​can be extracted from the same locations in the feature map as the sample point in the image to form the elements of the feature vector. This approach is easy to implement. For example, if J feature maps with the same resolution as the image are available at the input, simply extracting the values ​​from these maps at the coordinates of the current sample point (at the same x-coordinate and the same coordinates in the feature map) allows for the direct construction of a feature vector with J elements.

[0058] - Constructing the feature vector includes the following sub-steps:

[0059] - Extract multiple values ​​of the feature map based on the position of the current sample point;

[0060] - Process the extracted values ​​to obtain a feature vector.

[0061] Advantageously, according to this embodiment, feature vectors are extracted from feature maps prior to processing. These feature maps can be arbitrary, particularly those with a resolution lower than the image to be encoded (or decoded). This processing can correspond to, for example, quantization, scaling, or filtering of the extracted data. For encoding, quantization is crucial for the proper functioning of the system, for example, if the feature maps contain floating-point or real-point values. They need to be quantized before being encoded and / or provided as input to the synthesis and / or prediction neural network. On the other hand, for decoding, depending on the embodiment, dequantization is not necessary.

[0062] - The first and second groups contain the same number (M) of feature maps; advantageously, according to this embodiment, different images in the sequence are associated with the same number of feature maps, which may also have the same structure. For example, images in the sequence are associated with five feature maps having decreasing resolution levels. This facilitates the generation and extraction of values ​​from the feature maps to produce neighborhood vectors.

[0063] - The at least one reference map has the same resolution as the current map. Advantageously, according to this embodiment, since the reference map has the same resolution as the current map, it facilitates the construction of neighborhood vectors; extracting values ​​at positions that are at the same or close to the current map value is very simple.

[0064] - The at least one reference image is indicated in the bitstream. Advantageously, according to this embodiment, a simple indicator encoded in the stream allows the decoder to directly access one or more reference images to be used for decoding the current image, for example, by an indicator of the reference image number.

[0065] - The at least one reference image is predetermined. Advantageously, according to this embodiment, one or more reference images to be used for decoding the current image of the current image are predetermined. For example, the reference image is the previous image in the sequence. According to another example, the first image in the sequence is the reference image. According to another example, the first image in the sequence and the image after the current image are two reference images.

[0066] - The at least one reference map is indicated in the bitstream. Advantageously, according to this embodiment, simple indicators encoded in the stream allow the decoder to directly access one or more reference maps to be used for decoding the current map, for example, by means of indicators of the reference map numbers in a feature map group associated with the reference image.

[0067] - The at least one reference map is predetermined. Advantageously, according to this embodiment, one or more feature maps to be used for decoding the current map are predetermined. For example, a reference feature map is a map with the same index in its respective feature map group as the current map. According to another example, it is expected that all feature maps of the reference image are used as reference maps for the current map.

[0068] The at least one decoded value extracted from the at least one reference map is located at the same position in the at least one reference map as the current value in the current feature map. Advantageously, according to this embodiment, once the reference map is known, a neighborhood vector is generated using the values ​​in the current map that are located at the same position as the value being encoded (or decoded). Since these values ​​correspond to the same position, they are highly likely to be related, and therefore the extracted values ​​provide relevant information about the current value.

[0069] - The neighborhood vector further includes at least one second decoded value (Vd) extracted from another reference graph. n Advantageously, according to this embodiment, at least one second value can be extracted from another reference image. This allows values ​​at different times (e.g., two previous times, or future and past times depending on the order in which the images are encoded / decoded), and thus better accounts for changes in the signal used to encode / decode the current value.

[0070] Relatedly, the subject of this invention is also an encoding device and a decoding device.

[0071] The features and advantages of the encoding or decoding methods also apply to the encoding or decoding devices according to the present invention, and vice versa.

[0072] The subject of this invention is also a computer program on a storage medium that can be executed in a computer or encoding or decoding device according to the invention. The program contains instructions suitable for carrying out corresponding methods. The program can use any programming language and can take the form of source code, object code, or intermediate code between source code and object code, such as partially compiled code or any other desired form of code.

[0073] The subject of this invention is also a computer-readable information medium or storage medium containing instructions for a aforementioned computer program. This information or storage medium can be any entity or device capable of storing a program. For example, the medium can include storage devices such as ROMs (e.g., CD-ROMs or microelectronic circuit ROMs), or even magnetic storage devices (e.g., floppy disks, hard disks, DNA sequences, or flash memory). Furthermore, the information or storage medium can be a transmissible medium such as electrical or optical signals, which can be routed via cables or optical fibers, through radio links, through wireless optical links, or through other means.

[0074] The program according to the invention can be downloaded specifically via the Internet.

[0075] Alternatively, each piece of information or storage medium may be an integrated circuit incorporating the program, the circuit being adapted to implement or be used to implement the method according to the invention. Attached Figure Description

[0076] Other features and advantages of the invention will become apparent from the following description with reference to the accompanying drawings, which illustrate exemplary embodiments of the invention in a manner that is by no means limiting.

[0077] [ Figure 1 ] Figure 1 An encoding device used within the scope of this invention is illustrated schematically;

[0078] [ Figure 2 ] Figure 2 A decoding device used within the scope of this invention is illustrated schematically;

[0079] [ Figure 3 ] Figure 3 Examples of synthetic artificial neural networks used within the scope of this invention are shown;

[0080] [ Figure 4 ] Figure 4 Examples of predictive artificial neural networks used within the scope of this invention are shown;

[0081] [ Figure 5 ] Figure 5 It shows that it can be used Figure 1 A flowchart illustrating an example of an encoding method performed by an encoding device;

[0082] [ Figure 6 ] Figure 6 It shows that it can be used Figure 2 A flowchart illustrating an example of a decoding method performed by a decoding device;

[0083] [ Figure 7 ] Figure 7 An illustration of a decoding method used in one embodiment of the present invention is shown;

[0084] [ Figure 8 ] Figure 8 It shows that it can be used Figure 1 Encoding devices and using Figure 5 The flowchart shows the method for encoding the feature map, executed by the encoding method.

[0085] [ Figure 9 ] Figure 9 It shows that it can be used Figure 2 Decoding devices and using Figure 6 The flowchart shows the method executed by the decoding method for decoding the feature map;

[0086] [ Figure 10 ] Figure 10 A diagram illustrating a method for encoding or decoding a feature map is shown.

[0087] [ Figure 11 ] Figure 11 An illustration shows another method for encoding or decoding feature maps. Detailed Implementation

[0088] Figure 1 The digital image sequence encoding device ENC is illustrated schematically.

[0089] The encoding device ENC includes as many encoding sub-modules as the image to be encoded. v For the image v to be encoded in the sequence (denoted as Iv or I(P)), Vn (referred to as the current image), the sub-modules include the feature map generation module GEN, the transformation module SE, the data extraction module XTR, the module MLP corresponding to the synthetic artificial neural network, the neural network encoding module NNC which can encode the synthetic neural network and the optional predictive neural network, the feature map encoding and decoding module FMC, the encoding performance evaluation module EVAL, and the update module MAJ.

[0090] The encoding device ENC can be implemented by an electronic device including a processor and a memory (not shown); thus, each of the aforementioned modules can be generated by the interaction of the processor with computer program instructions stored in the aforementioned memory and designed, in particular as described below, to perform the function of the module in question when these instructions are executed by the processor.

[0091] Figure 1 The input of the encoding device ENC in the middle receives at least two representations as I(P) Vn The sequence of images to be encoded, each image containing multiple samples P. Vn The subscript v indicates a sequence S of V images. V The image number in the image. The subscript n indicates the pixel number in an image with N pixels. Image I(P) Vn A sequence can represent a single two-dimensional image, and a sequence can represent a sequence of multiple two-dimensional images (video sequence, stereoscopic component, multi-view component, medical image series, etc.). In one embodiment, the images in the sequence are represented by at least one two-dimensional representation (e.g., a pixel matrix), each pixel having a vector containing red (R) components, green (G) components, and blue (B) components, or alternatively, a vector containing a luminance component and at least one chrominance component. The position of each pixel is determined by its x and y coordinates in the image (x, y). n and y nThe sequence is defined as follows: In one embodiment, the sequence is a grayscale image sequence represented by a two-dimensional representation (such as a pixel matrix), where each pixel has a grayscale component or a luminance component. In this case, the vector representing the pixel is reduced to a single component.

[0092] The following text will refer to Figures 3 to 11 A more detailed description:

[0093] The feature map generation module GEN is configured to generate multiple (M) feature maps (denoted as FMv) for the current image v. i An optional module (not shown) can generate one or more additional graphs of number L, which are neither encoded nor transmitted.

[0094] In one embodiment, module SE processes a set of feature maps FMv i A transformation is performed to generate a set of feature maps FMSv with the same resolution as the input signal. i .

[0095] The optional module SE can quantize the M graphs FMv in this set. i The data is extracted from the image. As a reminder, quantization of a value refers to matching that value with members of a discrete set of possible code symbols. For example, this set of possible code symbols can be formed from integer values, and the quantization system simply rounds the actual value to an integer. According to another example, quantization involves multiplying by a given value and then rounding. The module SE can then transform the value of at least one feature map, for example, through upsampling, interpolation, filtering, etc. According to one embodiment, at the end of the transformation, the transformed feature map has the same resolution as the image in the input sequence. Advantageously, according to this embodiment, the encoded feature map can have a lower resolution than the image to be encoded, while the transformed map used to construct the feature vector has the same resolution as the image sequence, thereby facilitating value extraction.

[0096] In one embodiment, module SE is absent; in this case, the values ​​used to construct the feature vector are extracted directly from the second set of feature maps.

[0097] The XTR module targets the current sample Pv to be encoded. n Based on its coordinates in the input signal, from the feature map FMv i (and / or FMSv) i Extracting values ​​(according to one of the previously described embodiments). For example, if the intention is to extract values ​​from the coordinates (x, y) of the input image. n , y n Sample point Pv at ) n Encoding is performed so that module XTR is derived from the coordinates (x) in the graph. n , y nExtract the value at the determined location.

[0098] In one embodiment, the extracted values ​​form a vector Z. n Z n It is a J-tuple, meaning it contains J elements or data z. i A vector Z with index n n Refers to pixels (Pv) n eigenvectors.

[0099] The XTR module can also quantize the data extracted from this set of feature maps. In particular, if the SE module is not used and the feature maps have a lower resolution than the images in the sequence, the XTR module can take into account the coordinates of the values ​​in the lower-resolution maps.

[0100] It should be noted that at least one of the modules SE or XTR must quantize the values ​​extracted from the feature map of the current image.

[0101] The MLP module consists of K parameters Wv k Defined synthetic neural network, which can process the vector Z at the input. n Alternatively, a J-tuple can be used to output a sample Pv representing the current image to be encoded. n The vector. According to one embodiment, the synthetic neural network is an MLP or multilayer perceptron, consisting of an input layer adapted to an input format (J-tuples), optionally one or more hidden layers, and an output layer adapted to an output vector format, which is typically a vector containing NC elements corresponding to NC components of the image to be decoded. According to one embodiment, NC equals 3, and the output vector is the encoded and then decoded pixel P'v of the current image. n The (R, G, B) triplet.

[0102] The NNC module synthesizes the neural network for the current image (especially its parameters Wv). k Encoding is performed. Optionally, the NNC module... For predictive neural networks ARM (especially their parameter Ov) b Encoding is performed. During the encoding training or construction process, i.e., whenever the performance evaluation step is still unsatisfactory, the NNC module simulates encoding, then decodes it, and the result is sent to the evaluation module. Subsequently, this module processes the parameters Wv of the synthetic neural network. k Actual encoding is performed, and optionally, the predictive neural network ARM is encoded. The encoded parameters of the current image are represented as Wcv. k (and Ocv) b In known ways, coding simulation can be identical to or approximate the actual coding.

[0103] Module FMC to FMv i (That is, the values ​​of the feature maps of the current image, referred to as the second set of maps (excluding the maps optionally generated by module SE upsampling and additional maps)) are encoded. During the encoding training or construction process, i.e., whenever the performance evaluation step is still unsatisfactory, module FMC simulates encoding and then decoding, the results of which are sent to the evaluation module. Subsequently, an evaluation of the graph FMv is performed. i The actual encoding of the value. The encoded image representation of the current image is FMcv. i In a known manner, the encoding simulation can be identical to or approximate the actual encoding. If necessary, the encoding module quantizes the latent representation of the values ​​of the second set of graphs using a quantizer to generate an ordered set of quantized values. The encoding module then compresses the quantized data using encoding that considers the temporal neighborhood of the values ​​to be encoded in the feature maps. To construct the neighborhood, the FMC module uses encoded and decoded feature maps corresponding to previously encoded and then decoded images v' in the sequence that are different from the current image v, such as feature maps of the previous image in the sequence, or feature maps of two past reference images in the sequence, or feature maps of a past reference image and a future reference image (provided the decoding order of the sequence allows it (in a known manner, the decoding order can differ from the temporal order of the sequence)), etc. These feature maps are called the feature maps of the first set, or reference maps, denoted as FMdv'. i As will be explained later, the FMC module can include a predictive neural network (ARM) for the current image.

[0104] The module EVAL evaluates and minimizes coding performance. For example, the evaluation function is of the rate-distortion type. Minimization can be performed using gradient descent or any other method within the capabilities of those skilled in the art.

[0105] The MAJ module updates the FMv to be encoded for the current image based on the result of the performance function. i The value of .

[0106] Figure 2 The decoding device DEC is shown schematically.

[0107] Figure 2 The encoding device DEC includes as many decoding submodules as there are images to be decoded in the sequence. v For the current image v to be decoded (denoted as I(Pdv)... n The input terminal of the submodule receives:

[0108] - FMcv with M feature maps i (Also known as the second set of feature maps, or the current map) organized encoded data;

[0109] - The encoded parameters Wvc of the synthetic neural network corresponding to the current image k ;

[0110] - Optionally, the encoded parameters Ocv of the prediction neural network corresponding to the current image b ;

[0111] - Using feature map FMdv' i The data organized corresponds to previously decoded images v' in the sequence that are different from the current image v (e.g., feature maps of the previous image in the sequence, or feature maps of two reference images in the sequence). These feature maps are referred to as the first set of feature maps, or reference maps.

[0112] Submodule DEC v For N sample points Pdv of image v in the sequence n Decoding is performed. This submodule decodes the V images to be decoded in the sequence.

[0113] For the current image, the decoding device DEC v The device includes a neural network decoding module (NND) capable of decoding a synthetic neural network (MLP) and an optional predictive neural network (ARM), a feature map decoding module (FMD), a data extraction module (XTR), an inverse transform module (SE), and a module corresponding to the synthetic neural network (MLP). According to one embodiment, the decoding device outputs the decoded current image (denoted as I(Pdv)). n The decoded current image contains multiple decoded sample points Pdv. n .

[0114] The graph representation of M decoded by module FMD is FMdv i (Feature map of the current image v). The parameters of the synthetic neural network (MLP') decoded by module NND are represented as Wdv. k The parameters of the predictive neural network (ARM') optionally decoded by the module NND are represented as Odv. b In order to analyze the current graph FMdv i For decoding, the FMD module uses one or more feature maps, called reference maps, from an image v' that is different from the current image in the sequence. These feature maps have previously been decoded by another decoding submodule and represented as FMdv'. i .

[0115] Module NND corresponds to module NNC. For example, this module can use the neural network coding standard proposed in Part 17 of the MPEG-7 standard, also known as NNR (Neural Network Representation). According to one embodiment, module NND uses parameters previously decoded from other neural networks in the sequence to decode the neural network associated with image v.

[0116] The decoder can also generate one or more arbitrary additional graphs, not shown.

[0117] In one embodiment, module SE' processes the first set of decoded feature maps FMdv. i Perform a transformation to generate a set of transformed feature maps, denoted as FMS'v i Module SE' optionally performs inverse quantization corresponding to the quantization performed on the encoder. If the encoder's quantizer Q only rounds the actual value transmitted to it, inverse quantization is not required. Similarly, if the neural network can account for the quantization of its input data, inverse quantization is also unnecessary. Otherwise, the decoder performs the inverse operation of quantizer Q. Module SE' then optionally transforms the values ​​of the feature map, involving operations such as upsampling, interpolation, filtering, etc., similar to those performed by the encoder. At the end of the transformation, the transformed feature map FMS'v... i Preferably, it has the same resolution as the image to be decoded in the sequence.

[0118] In one embodiment, module SE' is not present; in this case, the values ​​used to construct the feature vector are extracted from the second set of feature maps.

[0119] In one embodiment, module SE' uses an upsampling neural network.

[0120] Module XTR and Figure 1 The module XTR is the same. This module is for the sample Pdv to be decoded. n Based on its coordinates in the image to be decoded, from the M feature maps FMdv of the current image. i (and / or according to one of the previously described embodiments, from FMS'v) i Extract values ​​from ).

[0121] In one embodiment, the extracted values ​​form a vector Zd. n Zd n It is a J-tuple, meaning it contains J elements or data. i In one embodiment, J = M.

[0122] In one embodiment, module XTR' also processes the extracted values ​​to generate vector Zd. nThe XTR module can perform inverse quantization on the data extracted from this set of feature maps, especially in the absence of the SE module. Processing can involve other operations similar to those performed by the encoder, such as filtering, scaling, etc.

[0123] Module MLP' consists of K parameters Wdv k The defined neural network (called a synthetic neural network) is capable of processing the input vector Zd. n Or a J-tuple, so that the output represents the sample Pv to be decoded. n The output vector is typically a vector containing NC element-valued vectors corresponding to NC components of the image to be decoded. In one embodiment, K = 3, and the output vector is the decoded pixel Pdv. n The (R, G, B) triplet. Module MLP' has the same structure as module MLP, and if its parameter Wv k If the encoding is lossless, then the parameters of the two are the same; if the encoding is lossy, then the parameters of the two are different.

[0124] When all samples of the signal are Pv n Once all signals have been decoded, the reconstructed signal I(Pdv) is obtained. n For example, an image Iv contains N vectors Pdv n N decoded samples in the form of .

[0125] The decoding device DEC can be implemented by an electronic device including a processor and memory (not shown); therefore, each of the aforementioned modules can be generated by the interaction of the processor with computer program instructions stored in the aforementioned memory and designed, in particular as described below, to perform the function of the module in question when these instructions are executed by the processor.

[0126] Figure 3 Examples of synthetic artificial neural networks used for encoding and decoding in the context of embodiments of the present invention are shown.

[0127] Synthetic artificial neural networks (MLPs) used for encoding and synthetic artificial neural networks (MLPs) used for decoding are defined, for example, by the same structure comprising multiple layers of artificial neurons, and a set of weights and activation functions associated with the artificial neurons of the network in question.

[0128] The vector representation of the current sample (from the feature map FMv on the encoder) i / FMSv i vector Z n Or from FMdv on the decoder i / FMSdv ivector Zd n This is applied to the input layer (i.e., the input layer) of a synthetic artificial neural network (MLP) or MLP'. According to one embodiment, the synthetic artificial neural network outputs a vector, which represents the color components (R, G, B) of the colored pixels that form the image.

[0129] All these reconstructed pixels are stitched together to form a (2D, 3D) image, resulting in a decoded or reconstructed image.

[0130] On the encoder, a synthetic artificial neural network (MLP) is trained on the image to minimize the representation I(Pv) of the current image at the input. n ) and its representation at the output terminal I(P'v n The difference between ) is minimized while also minimizing the amount of data to be encoded. In this sense, the EVAL module performs a performance metric.

[0131] Once the encoder training is complete, the network parameters are either losslessly encoded (in which case, the neural network MLP' is the same as MLP) or lossily encoded (in which case, the network MLP' may be slightly different from MLP).

[0132] Figure 4 An example of a predictive artificial neural network for encoding (ARM) and decoding (ARM') feature maps is shown in the context of embodiments of the present invention.

[0133] The predictive artificial neural network (ARM) used for encoding and the predictive artificial neural network (ARM') used for decoding are defined, for example, by including the same structure of multiple layers of artificial neurons, and a set of weights and activation functions respectively associated with the artificial neurons of the network in question.

[0134] The vector representation of the current neighborhood (from the feature map FMdv of another image in the sequence on the encoder or decoder) i or FMdv' i vector Cv n or Cdv n It is applied to the input of predictive artificial neural networks ARM (for encoding) or ARM' (for decoding) (i.e., the input layer).

[0135] Predictive artificial neural networks behave as functions whose output is a prediction of the current value of the feature map being processed; this prediction can be in the form of predicted values ​​or probability data.

[0136] According to one embodiment, on the encoder, the network implements function f This function is for encoding the FMv graph. i The current value Vv nThe current value provides the expected mean and / or variance (µ, σ). These statistics are used to perform entropy encoding on the value. For example, if the function produces a mean, the mean is subtracted from the current value, and only the difference is entropy encoded, with the mean being considered a prediction of the current value. Alternatively, if the function produces both a mean and a variance, the mean is subtracted from the current value, and the difference is encoded using an entropy code suitable for the variance, for example, by quantizing the variance into a predetermined set of variances and associating one type of entropy code with each quantized variance value. On the decoder, the network implements the function f. This function is for decoding the image FMdv. i The current value Vdv n The current value provides the expected mean and / or variance. These statistics are used to perform entropy decoding on the value. For example, if the function produces a mean, the current value is decoded by an entropy decoder, and the mean is added to the current value. Alternatively, if the function produces a mean and a variance, the current value is decoded by the decoder using entropy decoding appropriate for the variance, for example, by quantizing the variance into a predetermined set of variances and associating a type of entropy decoding with each quantized variance value.

[0137] According to another embodiment, the neural network can generate the expected probability (pr) for each possible value of the current sample. In this case, entropy encoding or decoding will be adapted to that probability (such as the well-known Huffman or arithmetic entropy encoding).

[0138] On the encoder, the predictive artificial neural network (ARM) is trained on the current image and a reference image to minimize the amount of data to be encoded. In this sense, the EVAL module performs performance metrics. As a reminder, the overall performance metric involves minimizing the encoded and then decoded image I(P'v) while minimizing the encoding bitrate. n ) and input image I(Pv n Distortion between () and (). According to one embodiment, the feature map is losslessly encoded using entropy coding. In this case, encoding the feature map affects the bit rate but does not affect the distortion of the encoded image. According to another embodiment, if the feature map is lossily encoded, encoding the feature map affects not only the bit rate but also the distortion.

[0139] Once training is complete, the network's B parameters Ocv b It can be either lossless encoded (in which case, the neural network ARM' is the same as ARM) or lossy encoded (in which case, the network ARM' may be slightly different from ARM).

[0140] Figure 5 It shows that it can be used Figure 1A flowchart illustrating an example of an encoding method performed by an encoding device.

[0141] According to this embodiment, the input signal is a two-dimensional image in an image sequence; therefore, each sample point to be encoded has coordinates (x, y) in the current image v. n , y n ) of pixels Pv n .

[0142] The encoding process is carried out in two main stages:

[0143] In the first stage, known as the construction stage, learning is performed to adapt the input image I(Pv) to the given image. n Determine the FMv diagram. i The value and parameter Wv k and optional Ov b This is done to optimize the total cost function. For example, by using gradient descent, the parameters and feature maps FMv of the synthetic neural network MLP are subsequently updated. i The value of the cost function and the optional prediction of the neural network ARM are used to perform learning. As known in the art, the cost function can be rate-distortion type, or bitrate type, or distortion type, or perceptual type. To measure the bitrate R, it is necessary to simulate the graph FMv. i The encoding is then performed, and the associated code rate (the size of stream B1) needs to be measured. According to one embodiment, the parameter Wv is not simulated. k and / or Ov b The encoding is performed because their influence is less than that of the feature map. According to one embodiment, simulation is also performed on the parameter Wv. k and / or Ov b The encoding is performed, and the associated bitrate (the size of stream B2) is measured. To measure the distortion D, it is necessary to simulate the encoding and then decoding of at least a portion of the image Iv to obtain at least one pixel P'v generated by the simulation of the encoding and then decoding. n Then, measure the image I(Pv) at the input end. n This part is related to the encoded and then decoded image I(P'v) n The difference between the corresponding parts of ).

[0144] In the second stage, known as the encoding stage, then the graph FMv is processed. i and parameter Wv k And possibly Ov b Encode to produce the encoded value FMcv i and Wcv k (and possibly Ocv) b They are then transmitted or stored. These form images (PVs). n The compressed representation of ).

[0145] The steps of a method according to an embodiment of the present invention will now be described.

[0146] In step E20, the image I(Pv) to be encoded with index v in the image sequence is provided at the input of the method. n The image to be encoded includes multiple (N) sample points Pv n And a certain number (NC) of components.

[0147] In step E21, the M graphs FMv of the second group are initialized. i Or the feature map of the current image v. Subsequently, the graph FMv must be optimized during the construction phase. i The values ​​of , and possibly the parameters of the neural network.

[0148] According to one embodiment, Figure FMv i Having the same characteristics as the input image I(Pv) n The same resolution, and therefore each image contains the same number of samples Pv to be encoded. n They have the same number of values ​​for N. According to one embodiment, Figure FMv i The resolution is less than or equal to the input signal I(Pv) n The resolution of the first image FMv1 is such that for at least one of the images, the number of values ​​N' to be encoded is less than N. According to one variant, the first image FMv1 has the resolution of the image, and each subsequent image has the resolution of the previous image. Figure 1 Half the resolution. According to one embodiment, multiple images FMv i It has the same resolution, which is smaller than the input signal I(Pv) n ) resolution.

[0149] According to one embodiment, for graph FMv i Perform a transformation to provide a set of transformed feature maps FMSv i In this embodiment, the feature vectors are preferably extracted from the transformed graphs rather than directly from the graphs in the second group. Therefore, in this embodiment, the feature vectors are extracted indirectly from the graphs in the second group. The transformed graphs are not encoded; they are only used to construct the feature vectors.

[0150] According to one embodiment, the current image's FMv is initialized using a predefined constant value. i According to another embodiment, a set of random actual values ​​is used to initialize the feature maps. According to one embodiment, one or more maps are generated and added to a second set, forming another set of additional feature maps. These additional feature maps are used to construct feature vectors but are not stored or transmitted.

[0151] Subsequently, in step E22, the encoder's update module MAJ updates or refines the feature maps FMv of the second group during its learning period. i .

[0152] In step E23, the encoder module FMC processes the second group of images FMv. i Encoding is performed. During the construction phase, this operation is an encoding simulation. During the encoding phase, this operation is the actual encoding, and the encoded values ​​form stream B1. The simulation can be the same as the actual encoding, but it can also be different (e.g., simplified). For this encoding, a technique is used to predict the feature map value through the temporal neighborhood of the feature map value, that is, prediction is made from the feature map of the first set (formed by the feature maps of images in the sequence other than the current image (called the reference map of the reference image)). Figure 8 In a more detailed description, in one embodiment, the structure and parameters of the synthetic (optionally predictive) neural network are initialized, for example, in the first iteration of this step. These parameters are then updated or refined in subsequent iterations of the method during the construction phase.

[0153] In one embodiment, Figure FMv i Encoding is performed in the order (FMv1, FMv2, ..., FMv4), and the variables of each graph are encoded in a predefined order (e.g., lexicographical order). Each graph undergoes entropy encoding. Entropy encoding produces a compressed stream B1, the bitrate of which can then be measured during step E29.

[0154] In step E24, according to one embodiment, module SE processes the M graphs FM of the first group. i Perform a transformation to generate a transformed image FMSv with the resolution of the images in the input sequence. i .

[0155] According to one embodiment, M graphs FMSv are generated. i According to one embodiment, each graph FMv i Transform into graph FMSv i According to one embodiment, at least one figure FMv i It has a lower resolution than the image to be encoded in the sequence, and the transform operation includes upsampling, so that the transformed image FMSv i It contains the same number of samples as the current image. Upsampling involves adding values ​​to the FMSv graph. i This upsampling is performed to achieve the same resolution as the images in the input sequence. This upsampling can be simple (by copying nearest neighbors) or involve interpolation (linear, polynomial, filtering, etc.). Upsampling can also be performed using an upsampling neural network.

[0156] In step E25, module XTR is transformed from graph FMv i Or possibly FMSv i Values ​​are extracted from the input signal, and optionally from an additional graph. This extraction is based on the sample points Pv of the input signal. n coordinates (x) n , y n This is performed using [the function / method]. The extraction can also be performed based on the resolution of the graph in question.

[0157] According to one embodiment, the feature vector Z n It is generated directly from this extraction.

[0158] The samples to be encoded are processed, for example, in order from n = 1 to n = N.

[0159] According to one embodiment, in step E25, module XTR is configured for the coordinates (x, y) of the input image. n , y n Each sample point Pv n According to the graph FMv i or FMSv i Extracted values ​​to construct feature vector Z n If necessary, this processing can involve processing the graph FMv. i Extracted value or vector Z n Perform quantization. This process can involve other operations such as filtering, scaling, applying any function (preferably a monotonic function), etc.

[0160] In one embodiment, Z n Including the graph FMv at the input end i or FMSv i The same number of values. In this case, J = M.

[0161] In one embodiment, Z n It is located at the current pixel Pv n coordinates (x) n , y n The image at ) FMv i or FMSv i The values ​​form J-tuples (z1, z2, ..., z...) J ), such as reference Figure 7 What is shown.

[0162] In one embodiment, Z n It is taken from the image FMv i The values ​​are used to construct J-tuples, and the coordinates at which these values ​​are obtained vary depending on the graph. For example, if the graph FMv iSince they have been downsampled to have different resolutions, the coordinates (by scaling) are adapted to match the resolution of each graph.

[0163] In one embodiment, Z n It is taken from the image FMv i The J-tuples are constructed from the values ​​of the graph, obtained by applying processing to one or more values ​​of the graph (e.g., filtering values ​​of neighboring target values ​​in the graph). For example, for a graph FMv with the same resolution as the input signal... i The current sample point Pv n It can extract the coordinates (x) n , y n ), (x n -1, y n ), (x n , y n -1) and (x n -1, y n The values ​​of -1) are then processed (filtering, averaging, interpolation, etc.) to obtain the values ​​corresponding to the FMv graph. i The relevant vector Z n The final value of element i (z) i According to another example, in a graph FMv with half the resolution of the input signal... i In the middle, it can be considered to be located at coordinate (x n / 2, y n / 2), (x n / 2-1, y n / 2), (x n / 2, y n / 2-1) and (x n / 2-1, y n The values ​​of / 2-1) can be processed (filtering, averaging, interpolation, etc.) to obtain the values ​​corresponding to the FMv graph. i The relevant vector Z n The final value of element i (z) i ).

[0164] In step E26, vector Z n Processed by a synthetic neural network MLP to output a representation of the sample Pv to be encoded. n The vector, according to one embodiment, is the vector of sample points P'v. n (Sample Pv after encoding and then decoding) n The (R, G, B) triplet.

[0165] For example, in the first iteration of this step, the structure and parameters Wv of the synthetic neural network are initialized. k(And optionally, the structure and parameters of the synthetic neural network, and the structure and parameters of the upsampled neural network, if such a neural network exists). These parameters are then updated or refined in subsequent iterations of the method during the construction phase.

[0166] According to one embodiment, the parameters of the synthetic neural network and / or the predictive neural network are initialized with predefined values ​​known to produce satisfactory results (e.g., after training on an image database). According to another embodiment, the parameters of the synthetic neural network and / or the predictive neural network are initialized with a set of random values.

[0167] In step E27, the parameters Wv of the synthetic neural network MLP are... k And the parameters Ov of the predictive neural network ARM b (If the neural network exists) Quantization and encoding are performed. During the build phase, this operation is an encoding simulation. During the encoding phase, this operation is the actual encoding, and the encoded values ​​form stream B2. The simulation can be the same as the actual encoding, but it can also be different (e.g., simplified). For this purpose, any known technique can be used, such as the neural network coding standard proposed in Part 17 of the MPEG-7 standard, also known as NNR (Neural Network Representation). It should be noted that in this case, the encoding pair weights Wv need to be selected. k And optional Ov b The amount of downgrade caused.

[0168] In step E28, performance metrics are evaluated.

[0169] Therefore, the measurement is performed on the feature map of the first group (by analyzing the FMv graph). i Encoding to simulate flow B1) and optionally with the parameters of (multiple) neural networks (via parameters Wv) k And optional Ov b Encode to simulate the bitrate associated with stream B2).

[0170] According to one embodiment, the cost function is of rate-distortion type, denoted as (D+L). R), where D is, for example, the squared error measured between the input signal and the decoded signal (or the error measured on a subset of samples of the signal). According to another example, D is calculated from a perceptual function such as SSIM (Structural Similarity) or MSSSIM (Multi-Scale Structural Similarity). According to one embodiment, R is the analog bitrate of stream B1; according to another embodiment, R is the total bitrate used to encode the image, i.e., the sum of the analog bitrates of B1 and B2. L is a parameter for adjusting the rate-distortion tradeoff. Other cost functions may also be used.

[0171] As long as the cost function has not reached its minimum, the performance metric is unsatisfactory, and the method repeats from step E22 onwards. Alternatively, the method can be interrupted at a predefined time or at the end of a predefined number of iterations to control its complexity or duration. This minimization can be performed using known mechanisms such as gradient descent, where parameters are updated by updating the values ​​of the feature maps during step E22 and updating the parameters of the network(s) during steps E26 and E27.

[0172] In step E29, training stops if the cost function has reached its minimum. This is similar to training with a synthetic neural network (Wv). k ) and feature map (FMv) i The final simulation of the parameters of the synthetic neural network, corresponding to the encoded version, is available, thereby forming streams B1 and B2. According to another embodiment, in this step, the updated parameters (Wv) of the synthetic neural network are performed. k ) and feature map (FMv) i The value of ) and the optional prediction neural network (Ov) b The actual encoding of Wcv is used to produce the encoded parameters that form streams B1 and B2. k (Optional Ov) b ) and FMcv i Streams B1 and B2 can be concatenated to produce the final stream. According to one embodiment, stream B2 of encoded parameters of (multiple) neural networks is stored or transmitted before stream B1 so that it can be decoded before stream B1.

[0173] Figure 6 It shows that it can be used Figure 2 A flowchart illustrating an example of a decoding method performed by a decoding device.

[0174] In step E30, streams B1 and B2 are extracted from the encoded stream. For the current image v in the sequence, these two streams each contain the second set of images FMcv. i The encoded representation and parameters Wcv of the images in the sequence k (Optional parameter Ocv) b ).

[0175] In step E31, the decoded feature maps FMdv' of other images in the sequence that are different from the current image are used. i The value of FMcv for the current image i Decode to generate M images FMdv of the current image v in the sequence. iFor example, if the current image is image 3 (v = 3), it can be decoded using feature maps of one or more previous images (v' = 1 and / or v' = 2). According to another example, it can be decoded using feature maps of one or more previous images (v' = 1 and / or v' = 2) and one or more future images (v' = 4, ...) in the sequence (provided that one or more future images have been decoded). Naturally, if the image is the first in the sequence, only the current image FMd is decoded. 1i Available. In this specific case, decoding does not depend on previous image graphs, but only on values ​​present in the graph associated with the current image.

[0176] For decoding, a technique is used to predict the feature map value through its temporal neighborhood, such as by referencing, for example,... Figure 9 Described. In one embodiment, Figure FMdv i Decoding is performed in the order of (FMdv1, FMdv2, ..., FMdv4), and the values ​​of each graph are decoded in a predefined order (e.g., lexicographical order).

[0177] According to the embodiment described for the encoder:

[0178] - Image FMdv i Having the signal I(Pdv) to be recovered n They have the same resolution, meaning these graphs contain N values.

[0179] - Image FMdv i The resolution is less than or equal to the signal I(Pdv) to be recovered. n ) resolution.

[0180] - Multiple images FMdv i It has the same resolution, but this resolution is lower than the resolution of the signal.

[0181] - The decoded reference image v' used to decode the current feature map is indicated in the bitstream B1, for example, in the form of an index indicating the image number in the sequence.

[0182] - The decoded reference image v' used to decode the current feature map is located at a predefined position. According to some examples, these images can be the previous image in the sequence, the next image (as long as it has been decoded), the first image, etc.

[0183] The feature map used to decode the current feature map is indicated in bitstream B1, for example, in the form of an index indicating the position of the feature map in a feature map group associated with the reference image (between 1 and M).

[0184] - The feature maps used to decode the current feature map are located in predefined positions. According to some examples, these feature maps may be feature maps that are in the same position as the current feature map in a set of feature maps associated with a reference image, or the previous feature map, or the previous and next feature maps, etc.

[0185] - The second group of feature maps associated with the current image are also used to decode the current feature map. For example, a spatial neighborhood of the current feature map's value can be extracted from the current feature map (causal neighborhood) or from the decoded feature maps in the same (second) group.

[0186] According to one embodiment, one or more maps are generated and added to a second set, forming another set of additional feature maps. These maps are not decoded, but are generated by the decoder in the same manner as they are generated in the encoder. These maps typically include data that can assist the network MLP in reconstructing the signal.

[0187] In step E32, according to one embodiment, module SE' processes the M graphs FMdv of the second group. i A transformation is performed to generate a transformed image FMS'v with the resolution of the input image. i This step is consistent with the reference. Figure 5 The steps described for the encoder are similar to E24, and these embodiments are applicable. Specifically:

[0188] According to one embodiment, M graphs FMS'v are generated. i .

[0189] According to one embodiment, each graph FMdv i Transform into graph FMS'v i .

[0190] According to one embodiment, at least one graph FMdv i It has a lower resolution than the image to be encoded, and the transform operation includes upsampling, so that the transformed image FMS'v i Includes the same number of samples as the input image. Upsampling involves adding values ​​to the FMS'v image. i This upsampling is done to achieve the desired resolution for the input image. The upsampling can be simple (by copying nearest neighbors) or involve interpolation (linear, polynomial, filtering, etc.). Upsampling can also be performed using an upsampling neural network.

[0191] If necessary, the transformation may optionally involve dequantization of the extracted values. However, dequantization is not mandatory.

[0192] In step E33, module XTR' is transformed from graph FMdv iOr possibly FMS'v i The value is extracted from the sample points Pv of the current image. n coordinates (x) n , y n This extraction is performed using [the specified method / method]. The extraction can also be performed based on the resolution of the graph in question. This step is consistent with the reference [reference / section / method]. Figure 5 The steps described for the encoder are similar to E25, and these embodiments are applicable. Specifically:

[0193] According to one embodiment, the feature vector Zd n It is generated directly from this extraction.

[0194] In one embodiment, Zd n It is located at the current pixel Pdv n coordinates (x) n , y n The image at ) FMdv i or FMS'v i The values ​​form J-tuples (z1, z2, ..., z...) J ), such as reference Figure 7 What is shown.

[0195] The samples to be decoded are processed, for example, in order from n = 1 to n = N.

[0196] According to one embodiment, in step E33, for coordinates (x... n , y n The Pdv of each sample to be decoded n The module XTR' follows this coordinate (x n , y n According to the graph FMdv from the second group i and FMdv' i From or through the transformed graph FMS'v i The vector Zd is constructed from values ​​extracted from the graph of the additional group, and optionally from the graph of the additional group. n This step is consistent with the reference. Figure 5 The steps described for encoder E25 are similar, and the described embodiments are applicable. If desired, the extraction may involve the extracted values ​​or the resulting vector Zd. n The execution of inverse quantization.

[0197] In step E34, the parameters Wdv of the synthesized neural network MLP' are... k And possibly predict the parameters Odv of the neural network b (And optionally the parameters of the upsampling network of module SE) are obtained through the value Wcv of stream B2. k and Ocvb This is generated through decoding. For this purpose, any known decoding technique corresponding to the encoding technique used by the encoder can be used. A synthetic neural network MLP' is similar to a synthetic network MLP, i.e., it has the same structure and includes the same parameters, except for the encoding, which can be lossy or lossless. Similarly, a predictive neural network ARM' (if used for decoding feature maps) is similar to a predictive network ARM, i.e., it has the same structure and includes the same parameters, except for the encoding, which can be lossy or lossless.

[0198] According to one embodiment, stream B2 is decoded before stream B1 so that the synthetic neural network MLP' and possibly the predictive neural network ARM' can be obtained before the decoding of the samples begins.

[0199] In step E35, the vector Zd is processed by the synthetic neural network MLP'. n The output represents the sample Pdv to be decoded. n The second vector, according to one embodiment, is the position (x) of the color components (Rd, Gd, Bd) of the image v. n , y n Injected into the decoded image I(Pdv) at point ) n The triplet in ) . This step is consistent with the reference Figure 5 The steps described for the encoder are similar to those in step E26.

[0200] When all samples have been processed, an image I(Pdv) is obtained, for example. n The corresponding currently decoded image.

[0201] Figure 7 An illustration shows a decoding method used in one embodiment of the present invention.

[0202] In this embodiment, there are four decoded images FMdv associated with the current image v in the sequence. i In a preferred embodiment, there are 7.

[0203] In this embodiment, the first image FMdv1 has the same resolution as image Iv and therefore contains W × H variables, where W is the width of the image in pixels and H is its height. The second image FMdv2 has half the resolution of image FMdv1 (in each dimension). The resolution of each additional image is half that of the previous image. This structure allows for a reduction in the number of variables in the feature maps, thereby aiding decoding while minimizing the encoding cost.

[0204] In this embodiment, a set of reference maps associated with an already decoded image v' in the sequence (e.g., the previous image in the sequence, i.e., v' = v-1) is available in the decoder. This set of maps associated with image v' has the same structure as the set of maps currently being decoded associated with the current image v: the first map FMdv'1 has the same resolution as image v', the second map FMdv'2 has half the resolution of map FMdv'1 (in each dimension), and each additional map has the same resolution as the previous image v'. Figure 1 Half the resolution.

[0205] When decoding the current graph (e.g., FMdv1), at least one reference graph (e.g., FMdv'1 and FMdv'2) is used. (The reference graph is then used.) Figure 10 and Figure 11 As shown, a reference graph is used to construct the neighborhood of the value to be decoded in the current graph.

[0206] When looking at the graph FMdv i During decoding, any upsampling method within the capabilities of those skilled in the art is used to upsample FMdv2 by 2x in each dimension, FMdv3 by 4x in each dimension, and FMdv4 by 8x in each dimension.

[0207] The resulting image is FMS'v i It has the same resolution as the image to be decoded, and therefore contains W × H values, where W is the width of the image (in pixels) and H is its height.

[0208] In this embodiment, vector Zd n It is located at the current pixel Pdv n coordinates (x) n , y n The figure at point FMS'v i The values ​​form a 4-tuple (z1 ... z4). Vector Zd n Optionally, it is dequantized and then processed by a synthetic neural network MLP to output a representation of the sample Pdv to be decoded. n The (R, G, B) triplet. Insert this (R, G, B) triplet into the decoded image I(Pdv). n The coordinates (x, y) of the color components (Rd, Gd, Bd) in the graph. n , y n ) place.

[0209] According to a variant shown using dashed lines, there are 5 graphs: an additional graph FME'0 is introduced. In this embodiment, vector Zd n It is a 5-tuple. Without limitation, for example, such an additional graph could be:

[0210] - A graph containing the x-coordinate of each point.

[0211] - A graph containing the y-coordinate of each point.

[0212] - A graph containing location-coded locations at each point (e.g., as described at the following web address: https: / / skosmos.loterre.fr / P66 / fr / page / -K0D65X2X-X).

[0213] - A diagram showing an image different from the image being processed, which can provide information about the image to be encoded, such as if the current image is part of a series of images to be encoded, like a video, a set of medical images, a multi-view representation, etc., then this information is from previously processed images.

[0214] - A graph showing the feature map of an image that is different from the image being processed. This graph can provide information about the image to be encoded, for example, if the current image is part of a series of images to be encoded, such as a video, a set of medical images, a multi-view representation, etc., then this information is from a previously processed graph.

[0215] - A graph that contains the values ​​of decoded samples from the same graph (e.g., the previous sample in the decoding order).

[0216] Figure 8 It shows that it can be used Figure 1 Encoding devices and using Figure 5 The flowchart illustrates the method for encoding the feature map.

[0217] These steps form the basis of previous references. Figure 5 The described sub-step of step E23. Its purpose is to use the reference image FMv' associated with the reference image v'. i The extracted neighborhood values ​​are used to compare the current feature map FMv of the second group of the current image being processed. i The current value Vv n Encode it.

[0218] In sub-step E231, a neighborhood vector (Cv) is established. n The neighborhood vector includes the value Vv n Nearest values. (See later for reference.) Figure 10 and Figure 11 As shown, these neighboring values ​​can be located in equivalent graphs (i.e., graphs with the same ordinal number within the group) and / or in multiple (M) graphs FMv' i In one or more different graphs. The neighborhood vector is formed by values ​​or data corresponding to a number C of neighborhood values ​​(e.g., C = 10).

[0219] According to one variant, it is also possible to obtain the decoded image FMv associated with the current image. i Extracting values ​​from Vv. In this case, the encoder and decoder must know these values; therefore, they must be located at the value Vv. n In the causal neighborhood.

[0220] According to the first embodiment, in step E234, these values ​​are used to determine the context of the entropy encoder used to encode the current value. This encoder can be a CABAC (Context Adaptive Binary Arithmetic Coding) encoder. This type of encoder is well known to those skilled in the art. It is particularly used in the H.265 / HEVC video compression standard. It is a lossless compression arithmetic encoder. It decomposes all non-binary symbols into binary symbols. Then, for each bit, the encoder selects the most suitable probability model and uses the context to optimize the probability estimate. This context can be defined by information about neighboring elements. Arithmetic coding is then applied to compress the resulting data. Those skilled in the art know that there are various ways to use neighborhood vectors to generate contextual information. For example, the number of non-zero neighboring values ​​can be calculated, and the context can be associated with each number. Alternatively, multiple neighboring values ​​can be compared, and a given context can be associated with an order configuration between neighboring values, for example, by classifying neighboring values ​​in ascending order and associating the context with each possible order.

[0221] In the second embodiment, during step E232, the current value is predicted using the neighborhood according to an autoregressive model. As a reminder, the autoregressive model predicts a range of samples based on past values. In this embodiment, past values ​​are formed by context, and during step E234, the difference between the predicted variable and the actual value is quantized and then entropy-encoded.

[0222] In the third embodiment, as referenced Figure 11 As shown, during step E233, a predictive neural network (ARM) is used to predict the statistical characteristics of the variable to be encoded. Neighborhood vectors are input into the ARM network to output a prediction of the current value. According to one embodiment, the ARM network behaves as a function f For reference Figure 4 The document describes a set of statistical parameters (mean, variance, median, etc.) for entropy encoding of the current value. The ARM module is designed for graph FMv. i All values ​​Vv to be encoded nThe best approach is to predict the current value to reduce the bit rate required to encode the feature map. According to another embodiment, the expected probability (pr) of the possible values ​​that the neural network ARM can use to generate the current sample is predicted. Entropy coding will be applied to this probability (such as the well-known Huffman or arithmetic entropy coding).

[0223] In the fourth embodiment, each feature map is divided into blocks of a predetermined size, and the encoding of each block involves a transformation (e.g., DCT (Discrete Cosine Transform), Haar Transform, etc.), and the transformed values ​​are encoded by entropy coding.

[0224] At the end of this method, the current graph FMv being processed is... i The current encoded value Vv n Encoded as variable Vcv n In the form of.

[0225] Figure 9 It shows that it can be used Figure 2 Decoding devices and using Figure 6 The flowchart shows the method executed by the decoding method for decoding the feature map.

[0226] These steps form the basis of previous references. Figure 6 The described sub-step of step E31. Its purpose is to use the reference image FMdv' associated with the decoded reference image. i The extracted neighborhood values ​​are used to compare the current feature map FMdv of the second group of the current image being processed. i The current value Vdv n Decode it.

[0227] In sub-step E311, the neighborhood vector (Cdv) is established. n The neighborhood vector includes the value Vdv n Neighboring values. This step is similar to step E231 described previously, and the same embodiments apply. In particular, these neighboring values ​​may be located in equivalent graphs (i.e., graphs with the same index in this set of feature graphs) and / or in multiple (M) graphs FMdv'. i In one or more different graphs. The neighborhood vector is formed by the graphs located in the same graph and / or in multiple (M) graphs FMdv. i The number of neighborhood values ​​corresponding to different graphs in the graph is C (e.g., C = 10) of values ​​or data.

[0228] According to one variant, it is also possible to obtain the current image from the FMdv graph. i Extracting values ​​from Vdv. In this case, the encoder and decoder must know these values; therefore, they must be located in the value Vdv. n In the causal neighborhood.

[0229] According to the first embodiment, in step E314, these values ​​are used to determine the context of the entropy decoder used to decode the current value. This decoding is similar to that used on an encoder (e.g., CABAC). Using neighborhoods to generate context information is similar to the selection on an encoder. For example, the number of non-zero neighboring values ​​can be calculated, and the context can be associated with each number. Alternatively, multiple neighboring values ​​can be compared, and a given context can be associated with an order configuration between the neighboring values, for example, by classifying the neighboring values ​​in ascending order and associating the context with each possible order.

[0230] In the second embodiment, during step E312, the current value is predicted using the neighborhood according to an autoregressive model. In this embodiment, past values ​​are formed by context, and during step E314, the difference between the predicted variable and the actual value is quantized and then entropy encoded.

[0231] In the third embodiment, as referenced Figure 11 As shown, in step E313, a predictive neural network ARM' is used to predict the statistical characteristics of the variable to be decoded. Neighborhood vectors are applied to the input of the network ARM' to output a prediction of the current value. According to one embodiment, the network ARM behaves as a function f For reference Figure 4 The method is defined by a set of statistical parameters (mean, variance, median, etc.) used for entropy decoding of the current value. According to another embodiment, a predictive neural network ARM' is used to generate the expected probability (pr) of the possible values ​​of the current sample. Entropy decoding will be applied to this probability (such as well-known Huffman or arithmetic entropy coding). If the encoding is performed losslessly, then network ARM' is identical to network ARM.

[0232] In the fourth embodiment, each feature map is divided into blocks of a predetermined size, and decoding each block involves entropy decoding of the values ​​and then performing an inverse transformation on them (e.g., inverse DCT (discrete cosine transform), inverse Haar transform, etc.).

[0233] At the end of this method, the current graph FMdv being processed is... i The current decoded value Vdv n Decoded.

[0234] Figure 10 An illustration shows a method for encoding or decoding a feature map according to one embodiment.

[0235] In this illustration, the coordinate (x) in the current feature map FMv1 (i.e., the map being encoded or undergoing analog encoding) (or FMdv1, i.e., the map being decoded) is... n , yn The current value Vv at () n (or Vdv) n The encoding (or decoding) uses context information from two graphs of the already decoded reference image v': graph FMv'1, which is located in the same position as the current graph in this set of reference graphs, and the previous graph FMv'2 (or FMdv'2). The coordinates (x, y, y) in graph FMv'1 (or FMdv'1) are used. n -1, y n -1), (x n -1,y n ), (x n -1, y n +1), (x n , y n -1), (x n , y n ), (x n , y n +1), (x n +1, y n -1), (x n +1, y n ), (x n +1, y n The value at +1), and the coordinate (x) in layer FMv'2 (or FMdv2). n / 2-1, y n / 2-1), (x n / 2-1, y n / 2), (x n / 2,y n / 2-1), (x n / 2, y n The variable at / 2) determines the neighborhood used to encode (or decode) the current value. These values, available in both the encoder and decoder, form the value Vv to be encoded. n (or Vdv) n The neighborhood vector Cv n (or Cdv) n This neighborhood vector can be used in the previous reference. Figure 8 (or Figure 9 In one of the embodiments described.

[0236] Figure 11 An illustration shows another method for encoding or decoding a feature map according to one embodiment.

[0237] In this illustration, the coordinate (x) in the current feature map FMv1 (or FMdv1) is... n , y n The current value Vv at () n(or Vdv) n The encoding (or decoding) of the current value uses contextual information from a reference image located at the same position in the reference image v'. Values ​​shown in gray are used to determine the neighborhood for encoding (or decoding) the current value. These values, available in both the encoder and decoder, form the value Vv. n (or Vdv) n The neighborhood vector Cv n (or Cdv) n This neighborhood vector can be used in the previous reference. Figure 8 (or Figure 9 In one of the embodiments described.

[0238] In the illustrated embodiment, the neighborhood vector is extracted by module CTX (or CTX') of the encoding module FMC (or FMD) and then applied to the input of the predictive neural network ARM (or ARM') to predict the statistical features (µ, σ) or probability (pr) of the value to be encoded (or decoded), represented in a fork shape. This value is then encoded (or decoded) by the entropy encoder CE (or DE).

Claims

1. A method for processing image sequences (S v The method for encoding includes the following steps: - Construction steps, which include targeting at least one current image (v, I(Pv)) in the sequence. n The following sub-steps of )) are: - Construct (E21, E22) to represent the current image (v', I(Pv')). n The first set of feature maps (FMv') of at least one different image referred to as the reference image i ); - Construct (E21, E22) to represent the current image (v, I(Pv)). n The second set of feature maps (FMv) i ); - For the position in the current image ((x n , y n The associated sample is called the current sample (Pv). n At least one sample point of the current image: - Based on the current sample point (Pv) n The position (x) n , y n From the feature map (FMv) of the second group i Constructing (E25) eigenvectors (Z) n ); - Use a set of parameters (Wv) k The artificial neural network (MLP) defined as a synthetic neural network is used to process the feature vector (Z) described in (E26). n ), so as to provide a vector (P'v) representing the decoded value of the at least one current sample. n ); - Update at least one value of one feature map in the second group and / or at least one parameter of the network based on the coding performance metric (E22, E27); - For the second set of feature maps (FMv) i The step of encoding (E23) includes: for a feature map referred to as the current map, a value referred to as the current value (Vv) is given. n At least one value of ), based on at least one value (Vv') extracted from at least one map, referred to as a reference map, including the first set of feature maps. n The value is entropy encoded using the neighborhood of the given value. - For the set of parameters (Wv) of the synthetic neural network k The steps for encoding (E27).

2. The encoding method as described in claim 1, characterized in that, For the current value (Vv) n The encoding process includes the following sub-steps: - Construct (E231) neighborhood vectors (Cv' n The neighborhood vector includes data from the at least one reference graph (FMv'). i The at least one neighborhood value extracted from ) - Use a set of parameters (Ov) b The artificial neural network processing (E233) defined as a predictive neural network (ARM) uses the neighborhood vector (Cv') to process the neural network. n ), so as to provide the current value (Vv) n (Prediction); - Update at least one parameter of the prediction network (E22, E23, E28) based on the coding performance metric; - For the set of parameters (Ov) of the prediction network b The steps for encoding.

3. A method for extracting image sequences (S) from a bitstream v A method for decoding ) for at least one current image (v, I(Pdv) in the sequence. n This includes the following steps: - From at least one image (v', I(Pdv') that represents a reference image that is different from the current image. n The first set of decoded feature maps obtained (E31) includes at least one feature map (FMdv') referred to as the reference map. i ); - For the second set of feature maps (FMdv) representing the current image i Decoding (E31) involves: for a feature map referred to as the current map, a value referred to as the current value (Vdv) is performed. n At least one value of ), based on at least one decoded value (Vdv') extracted from the at least one reference graph. n Entropy decoding (DE) is performed on the value in the neighborhood of ). - For a set of parameters (Wdv) representing a synthetic neural network (MLP') k Decode (E34); - For the position in the current image ((x n , y n The associated sample is called the current sample (Pdv). n At least one sample point of the current image: - Based on the position of the current sample point ((x) n , y n From the feature map of the second group (FMdv) i Constructing (E33) feature vectors (Zd) n );as well as: - Use the synthetic neural network (MLP') to process (E35) the vector (Zd) n ), so as to provide a representation of the at least one current sample (Pdv) n The vector of decoded values.

4. The decoding method as described in claim 3, characterized in that, For the current value (Vdv) n The decoding process includes the following sub-steps: - For a set of parameters (Odv) representing a neural network called a predictive neural network (ARM') k Decode (E34); - Construct (E311) neighborhood vectors (Cdv) n The neighborhood vector includes data from the at least one reference graph (FMdv'). i The at least one decoded value (Vv') extracted from ) n ); - Use the predictive neural network (ARM') to process (E313) the neighborhood vector (Cdv) n ), so as to provide the current value (Vdv) n (Prediction).

5. The encoding or decoding method as described in any one of the preceding claims, characterized in that, The first group and the second group (FMdv' i FMdv i Each contains the same number (M) of feature maps.

6. The encoding or decoding method as described in any one of the preceding claims, characterized in that, The at least one reference diagram (FMdv') i ) has the same characteristics as the current graph (FMdv) i (Same resolution) 7. The encoding or decoding method as described in any one of the preceding claims, characterized in that, The at least one reference image (v') is indicated in the bitstream.

8. The encoding or decoding method as described in any one of the preceding claims, characterized in that, The at least one reference image (v') is predetermined.

9. The encoding or decoding method as described in any one of the preceding claims, characterized in that, The at least one reference diagram is indicated in the bitstream.

10. The encoding or decoding method as described in any one of the preceding claims, characterized in that, The at least one reference drawing is predetermined.

11. The encoding method as described in claim 2 or the decoding method as described in any one of claims 4 to 10, characterized in that, The at least one decoded value (Vdv') extracted from the at least one reference graph n In at least one reference diagram (FMdv') i The current feature map (FMdv) i The current value (Vdv) in ) n ) are in the same position ((x) n , y n )).

12. The encoding method as described in claim 2 or the decoding method as described in any one of claims 4 to 11, characterized in that, The neighborhood vector further includes at least one second decoded value (Vd) extracted from another reference graph. n ').

13. A method for processing image sequences (S v The device for encoding is configured to perform the following steps: - Construction steps, which include targeting at least one current image (v, I(Pv)) in the sequence. n The following sub-steps of )) are: - Construct (GEN, MAJ) to represent the current image (v', I(Pv') n The first set of feature maps (FMv') of at least one different image referred to as the reference image i ); - Construct (GEN, MAJ) to represent the current image (v, I(Pv)). n The second set of feature maps (FMv) i ); - For the position in the current image ((x n , y n The associated sample is called the current sample (Pv). n At least one sample point of the current image: - Based on the current sample point (Pv) n The position (x) n , y n From the feature map (FMv) of the second group i Constructing (XTR) feature vectors (Z) n ); - Use a set of parameters (Wv) k Artificial neural networks, defined as synthetic neural networks (MLPs), are used to process the feature vectors (Z). n ), so as to provide a vector (P'v) representing the decoded value of the at least one current sample. n ); - Update (MAJ) at least one value of one feature map in the second group and / or at least one parameter of the network based on the coding performance metric; - For the second set of feature maps (FMv) i The encoding step (FMC, ARM) includes: for a feature map referred to as the current map, a value (Vv) is given. n At least one value of ), based on at least one value (Vv') extracted from at least one map, referred to as a reference map, including the first set of feature maps. n The value is entropy encoded using the neighborhood of the given value. - For the set of parameters (Wv) of the synthetic neural network k The steps for encoding (NNC).

14. A method for extracting image sequences (S) from a bitstream. v A device for decoding, the device being configured to decode at least one current image (v, I(Pdv)) in the sequence. n Perform the following steps: - From at least one image (v', I(Pdv') that represents a reference image that is different from the current image. n The first set of decoded feature maps obtained (FMD) is referred to as the reference map, which contains at least one feature map (FMdv'). i ); - For the second set of feature maps (FMdv) representing the current image i Decoding (FMD) involves the following steps: For a feature map referred to as the current map, the current value (Vdv) is referred to as the current value. n At least one value of ), based on at least one decoded value (Vdv') extracted from the at least one reference graph. n Entropy decoding (DE) is performed on the value in the neighborhood of ). - For a set of parameters (Wdv) representing a synthetic neural network (MLP') k Decode (NND); - For the position in the current image ((x n , y n The associated sample is called the current sample (Pdv). n At least one sample point of the current image: - Based on the position of the current sample point ((x) n , y n From the feature map of the second group (FMdv) i Constructing (XTR') feature vector (Zd) n );as well as: - Use the synthetic neural network (MLP') to process the vector (Zd) of the MLP'. n ), so as to provide a representation of the at least one current sample (Pdv) n The vector of decoded values.

15. A computer program comprising instructions which, when executed by a computer, are used to perform the steps of the encoding method of claim 1 or the decoding method of claim 3.

Images