Method and apparatus for encoding and decoding image

Abstract

The present invention relates to a method and an apparatus for encoding and decoding an image comprising a plurality of samples to be encoded or decoded. The decoding method comprises the following steps:-decoding (E31) a set (M) of feature maps (FMdi); obtaining (E32) a set of synthesis parameters (Wck) representing the synthesis module (MLP '); obtaining (E32) a set of convolution parameters (Vck) representing the convolutional neural network; -processing (E33, E34) said set of feature maps by means of a synthetic neural network (MLP ') to provide a set (NL) of potential value maps (L'j), the number of potential value maps being less than or equal to the number of feature maps; -processing (E35) said set of potential value maps by said convolutional neural network (MPP '') to provide said at least one decoded image (I (Pdn)).

Description

Technical Field

[0001] This invention relates to the general field of encoding digital image sequences. Background Technology

[0002] Digital images or videos are typically source-encoded for the purpose of compression to limit the resources required for their transmission and / or storage. Many coding standards exist, such as those developed by the ITU / MPEG organization (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) and their extensions (MVC, SVC, 3D-HEVC, etc.).

[0003] As video formats continue to evolve to achieve higher compression rates and match diverse expected formats and communication networks, the possibilities for prediction are constantly increasing, and conventional encoding and decoding algorithms are becoming increasingly complex.

[0004] In addition to the conventional methods proposed by compression standards (MPEG, ITU), artificial intelligence-based methods (especially neural methods) are also emerging.

[0005] Some of these neural methods can be considered as simple extensions of the competing concepts among the aforementioned compression techniques (such as the competition between prediction and video coding transform methods).

[0006] Other methods use the concept of an "autoencoder." An autoencoder is a learning algorithm based on artificial neural networks that allows the construction of new representations of a dataset. The architecture of an autoencoder consists of two parts: an encoder and a decoder. The encoder is formed by a set of neural layers that process the data to construct new representations, called "encoded" representations or "latent representations." The decoder's neural layers then receive these representations and filter them to attempt to reconstruct the original data. The difference between the reconstructed data and the initial data allows measurement of any errors produced by the autoencoder. Training involves changing the parameters of the autoencoder to reduce the reconstruction error measured at individual samples in the dataset. While these autoencoder-based systems offer superior performance, they come at the cost of a significant increase in memory footprint and complexity compared to conventional methods, such as those proposed by compression standards. These systems may have millions of parameters, and decoding a single pixel may require approximately a million MAC (multiply-accumulate) operations. This makes such decoders far more complex than conventional decoders, potentially hindering the application of learning-based compression. Furthermore, the neural networks used for the encoder and decoder are different. This automatic encoder system is described in document US 2022 / 0385907.

[0007] Recently, Emilien Dupont et al. described a simple neural network-based image coding technique in their paper "Compression with Implicit Neural Representations" (arXiv: 2103.03123). The proposed coding technique involves adjusting the neural network for the image, quantizing the network weights, and transmitting the quantized weights. During decoding, the neural network is evaluated at each pixel location to reconstruct the image. However, this technique remains inefficient in terms of compression.

[0008] The document US 2022 / 0385907 also describes an encoding and decoding system based on the use of such implicit neuron representations.

[0009] However, a simple and efficient solution for encoding / compressing image sequences is needed. Summary of the Invention

[0010] The subject of this invention is a method for encoding at least one image, the method comprising the following steps:

[0011] - The step of constructing the at least one encoded and then decoded image includes the following sub-steps:

[0012] - Construct a set of feature maps;

[0013] - The set of feature maps is processed by a synthesis module defined by synthesis parameters to provide a set of latent value maps, the number of which is less than or equal to the number of feature maps;

[0014] - The set of latent value maps are processed by at least one convolutional neural network defined by convolutional parameters to provide at least one encoded and then decoded image;

[0015] - Update at least one value of one of the feature maps and / or at least one synthesis parameter and / or at least one convolution parameter based on coding performance measurements;

[0016] - The step of encoding the set of feature maps and the set of synthesis parameters and convolution parameters.

[0017] The subject of this invention is also a method for decoding at least one image from a bitstream, the method comprising the following steps:

[0018] - Decode a set of feature maps;

[0019] - Obtain a set of synthesis parameters representing the synthesis module;

[0020] - Obtain a set of convolution parameters representing at least one convolutional neural network;

[0021] - The synthesis module processes the set of feature maps to provide a set of latent value maps, the number of which is less than or equal to the number of feature maps;

[0022] - The set of latent value maps are processed by the at least one convolutional neural network to provide the at least one decoded image.

[0023] Within the meaning of this invention, the term "encoding" should be understood to refer to operations involving representing a set of samples (or pixels) of one or more images in a compact form, for example, transmitted via a digital bitstream. The term "decoding" should be understood to refer to operations involving processing a digital bitstream to render decoded samples of one or more images.

[0024] The term "image" should be understood to mean a two-dimensional digital image that contains one or more components (color, depth, viewpoint, etc.).

[0025] The term "at least one image" should be understood to mean, for example, multiple ordered two-dimensional images arranged chronologically in the case of video. According to another example, the images can be views of the same scene displayed from multiple perspectives. According to yet another example, the images can be multi-view images from multiple times (immersive video).

[0026] The terms "sample" or "pixel" should be understood as referring to a value or set of values ​​obtained from an image. Sampling a signal produces a discrete sequence of values ​​called samples. In the case of an image signal, samples are called pixels, which can be, for example, color pixels traditionally represented by triplet values ​​(e.g., (R, G, B) or (Y, U, V)), each value belonging to a component. The location of a sample can be identified by its coordinates on the x-axis and y-axis of the image.

[0027] The term "feature map" should be understood as referring to a portion of an abstract representation of an image or sequence of images that contains multiple variable scalar data, also known as feature values, such as signed or unsigned real numbers or integers. Therefore, an abstract representation of an image or sequence can contain multiple feature maps. An abstract representation of one or more images can also correspond to a single feature map.

[0028] The term "latent value map" should be understood as referring to a portion of an abstract representation of an image or sequence of images that contains multiple variable scalar data, also known as latent values, such as signed or unsigned real numbers or integers. Therefore, an abstract representation of an image or sequence can contain multiple latent value maps. An abstract representation of one or more images can also correspond to a single latent value map. These maps are generated by processing feature maps through a synthesis module (e.g., a synthetic neural network).

[0029] The term "composition module" should be understood to mean, for example, a module whose parameter W is used to construct a composite module. k Defined as a neural network, filter bank, or linear inverse transform (wavelet, DCT, etc.) that can process the transformed feature map at the input to output a set of NL potential value maps.

[0030] The terms "synthetic neural network" and "convolutional neural network" should be understood as referring to neural networks, such as convolutional neural networks and multilayer perceptrons. A neural network is defined, for example, by multiple layers of artificial neurons and a set of activation functions, weighting functions, and addition functions (e.g., a layer can compute y = f(Ax+b), where y and b are N-dimensional vectors, x is an M-dimensional vector, A is an M × N-dimensional matrix, and f is an activation function).

[0031] The term "parameters of a neural network" should be understood to mean one of the values ​​that characterize a neural network or multiple neural networks, such as the weights (filter coefficients, matrix values, weights, biases, values ​​that affect nonlinear operations, etc.) associated with one of the neurons.

[0032] The term “generating the at least one image” should be understood to mean the steps aimed at reconstructing decoded samples of one or more images.

[0033] The term "performance measurement" should be understood as a measurement between at least one value of a sample to be encoded and the decoded value of said sample. This measurement can evaluate, for example, distortion or perceptual error. The measurement can be performed for one or more samples (e.g., the current sample or the current image, etc.). The measurement can also include a measurement of bitrate (particularly the bitrate associated with encoding a neural network and / or encoding feature maps in a group). This measurement can be a joint measurement of bitrate and distortion achieved by weighting the bitrate and distortion. As is well known in the art, the value of this measurement is typically minimized until a target value is reached.

[0034] The term "construction step" refers to the steps aimed at building the encoded and then decoded values ​​of pixels, as well as the parameters representing the image (particularly the values ​​of the feature maps) and the parameters of the neural network, before effectively encoding the pixels of one or more images. The construction sub-step can be repeated multiple times as needed to obtain acceptable performance measurements.

[0035] Generally, it is assumed that the steps of an encoding or decoding method should not be interpreted as being associated with a concept of chronological order. In other words, these steps may be performed in a different order than indicated in the independent encoding or decoding claims, or even simultaneously.

[0036] The encoding method according to the invention constructs encoding parameters from an input image or image sequence by training a neural network on feature maps and then on latent value maps, which may have the same resolution as the input image or a lower resolution. During training or construction, the parameters of the neural network and the values ​​of the feature maps and latent value maps are updated according to performance measurements, such as bitrate / distortion type. When training is complete (i.e., the obtained performance measurements are satisfactory), the actual encoding of the neural network parameters and the values ​​of the feature maps can be performed and stored or transmitted to the decoder. The latent value maps are neither encoded nor transmitted.

[0037] Advantageously, the training process allows for refining the values ​​of the neural network parameters and / or feature maps until a sufficiently performance-oriented representation is obtained, e.g., until the desired balance is achieved between the generated bit rate and the distortion experienced by the encoded input image or image sequence. Training of the feature map values ​​and the neural network parameters can be performed jointly. Advantageously, the encoding method according to the invention allows for efficient signal compression.

[0038] Advantageously, feature map compression is efficient because it allows feature maps to be transmitted over a network or on storage media with reduced capacity.

[0039] Furthermore, using two sets of networks (synthesis and convolution) is advantageous in itself because it allows each network to be trained separately to ensure better convergence. According to some embodiments, one of the two networks can be frozen and only one network can be learned, thus only the parameters of the learned network will be transmitted to the decoder.

[0040] Furthermore, since the number of latent value maps is less than or equal to the number of feature maps, the number of operations that the convolutional network must perform is limited. This reduces complexity because less data is processed. Advantageously, the decoding method is simple, as it is sufficient to decode the feature maps associated with an image or a set of images and obtain synthetic and / or convolutional neural networks to recover a decoded version of the current image or a set of current images.

[0041] Such synthetic or convolutional neural networks can advantageously have a very simple structure, with each sample having few parameters and few operations.

[0042] According to an embodiment of the encoding or decoding method:

[0043] - The at least one image includes multiple samples to be decoded, and the step of processing the set of feature maps to provide a set of latent value maps includes the following sub-steps:

[0044] - For at least one sample point of the at least one image to be decoded, referred to as the current sample point, which is associated with a location:

[0045] - Construct feature vectors from these feature maps based on the positions of the current sample points, and:

[0046] - The feature vector is processed by the synthesis module to provide a latent vector representing the potential value of the current sample point;

[0047] - Construct the set of potential value maps from the at least one potential vector.

[0048] Advantageously, according to this embodiment, the synthesis module is positional; the combination of the positional synthesis network and the convolutional network allows for local processing to be performed in the synthesis network, and then corrected using the convolutional network.

[0049] - This synthesis module includes a multilayer perceptron or MLP. Advantageously, such a neural network can have a very simple structure with few parameters. Furthermore, decoding can be performed on a sample-by-sample, progressive basis on specific blocks of the image.

[0050] - A convolutional network contains at least one convolutional layer. Therefore, nonlocalization processing can be performed on an image, which improves the consistency of the generated image by limiting noise and improving the contours present in the image.

[0051] The number of latent value maps equals the number of components in the image. Advantageously, complexity is limited by precisely processing the number of components of the output pixels (e.g., three in the case of RGB or YUV images). Depending on the variant, each of the latent value maps (e.g., corresponding to the Y, U, and V components) can be processed independently. This further limits the processing complexity because the convolutional network has fewer multiplication / addition operations, and therefore the space required in the encoder and decoder's memory is limited, and the cost of the convolutional parameters is low.

[0052] - Obtain (or encode) at least a portion of a set of synthesis parameters and / or convolution parameters from the data of the bitstream (or encoded in the bitstream); advantageously, according to this embodiment, the parameters of the neural network are transmitted in the stream. The synthesis and / or convolution modules can be transmitted in quantized form and encoded in compact form using any quantizer and entropy encoder available to those skilled in the art. A format similar to the MPEG-7 NNR standard can be used.

[0053] - Obtain (or encode) at least a portion of a set of synthesis parameters and / or convolution parameters from predetermined parameters. Advantageously, according to this embodiment, the parameters of the neural network can be accessed from a decoder; for example, these parameters can be stored in (or encoded in) a storage memory. In particular, a portion of the synthesis and / or convolution module can be stored in quantized form and encoded in compact form using any quantizer and entropy encoder available to those skilled in the art. A format similar to that of the MPEG-7 NNR standard can be used.

[0054] - Obtain (or encode) at least a portion of a set of synthetic parameters and / or convolutional parameters based on reference parameters. Advantageously, according to this embodiment, the parameters of the neural network are decoded by taking into account reference parameters that can be stored in memory (storage memory, network, etc.) accessible from the decoder. Therefore, the transmission bitrate in storage space or on the network can be efficiently reduced: some parameters can be encoded / decoded in a complementary manner, while other parameters may not be encoded / decoded at all because they are available from the reference parameters. According to variant embodiments that can be combined with each other:

[0055] • Reference parameters can form a set of variations of the synthesized and / or convolutional networks: convolution type, attention module, etc. Therefore, only these parameters need to be encoded, inserted into the stream, and decoded by the decoder;

[0056] • Some synthetic and / or convolutional parameters are encoded in a manner complementary to the parameters of the reference network. Therefore, the decoder only needs to decode these residual parameters and then add or multiply them with the residual parameters of the reference neural network (or combine them in any other known way);

[0057] • A portion of the parameters of the reference network is reused in the synthesized network and / or convolutional network. For example, an entire layer of the network can be copied from the reference network, and therefore the parameters of that layer are neither encoded, transmitted, nor decoded;

[0058] • Transmit a reference network indicator in the stream. Therefore, one of the many possible networks known to the decoder can be indicated as the network to be used for decoding.

[0059] - The at least one convolutional network comprises at least two convolutional networks. Advantageously, using multiple cascaded networks makes it possible to enhance filtering and improve the consistency of the image by limiting noise and improving its contours. Furthermore, each network can be learned or frozen independently. According to variant embodiments that can be combined with each other:

[0060] The bitstream contains an indicator indicating the use of at least one of the networks. Therefore, depending on the desired quality and complexity, one or more networks can be used for decoding. For example, the indicator might suggest that network 1 is mandatory, but networks 2 and 3, concatenated after network 1, are optional. The decoder can then choose whether to use these networks: if it does, it will achieve better quality at the expense of complexity, and vice versa.

[0061] • At least two convolutional networks in a convolutional network share common parameters. Therefore, it is advantageous to transfer the feature parameters of one network, which can then be used in other networks, thereby reducing transfer costs. Thus, the same network is shared.

[0062] • At least two of these networks have the same structure, and only their weights are partially or completely different. Therefore, they share the same computational elements.

[0063] - At least one image has at least two dimensions, and the convolutional layer uses a separable kernel in said at least two dimensions. Advantageously, in this embodiment, transmission costs and complexity are reduced by using separable operations (e.g., convolution, upsampling, etc.), which require fewer operations than non-separable operations.

[0064] Relatedly, the subject of this invention is also an encoding device and a decoding device.

[0065] The features and advantages of the encoding or decoding methods also apply to the encoding or decoding devices according to the present invention, and vice versa.

[0066] The subject of this invention is also a computer program on a storage medium that can be executed in a computer or control device according to the invention. The program contains instructions suitable for carrying out the corresponding methods. The program can use any programming language and can take the form of source code, object code, or intermediate code between source code and object code, such as partially compiled code or any other desired form of code.

[0067] The subject of this invention is also a computer-readable information medium or storage medium containing instructions for a aforementioned computer program. This information or storage medium can be any entity or device capable of storing a program. For example, the medium can include storage devices such as ROMs (e.g., CD-ROMs or microelectronic circuit ROMs), or even magnetic storage devices (e.g., floppy disks, hard disks, DNA sequences, or flash memory). Furthermore, the information or storage medium can be a transmissible medium such as electrical or optical signals, which can be routed via cables or optical fibers, through radio links, through wireless optical links, or through other means.

[0068] The program according to the present invention can be downloaded specifically from a network such as the Internet.

[0069] Alternatively, each piece of information or storage medium may be an integrated circuit incorporating the program, the circuit being adapted to implement or be used to implement the method according to the invention. Attached Figure Description

[0070] Other features and advantages of the invention will become apparent from the following description with reference to the accompanying drawings, which illustrate exemplary embodiments of the invention in a manner that is by no means limiting.

[0071] [ Figure 1 ] Figure 1 An encoding device used within the scope of this invention is illustrated schematically;

[0072] [ Figure 2 ] Figure 2 A decoding device used within the scope of this invention is illustrated schematically;

[0073] [ Figure 3 ] Figure 3 The schematic illustration shows the synthetic and convolutional artificial neural networks used in one embodiment of the invention during encoding and decoding;

[0074] [ Figure 4 ] Figure 4 The schematic illustration shows a synthetic and convolutional artificial neural network used in another embodiment of the invention during encoding and decoding;

[0075] [ Figure 5 ] Figure 5 A convolutional network according to an embodiment of the present invention is illustrated schematically;

[0076] [ Figure 6 ] Figure 6 A convolutional network according to an embodiment of the present invention is illustrated schematically;

[0077] [ Figure 7 ] Figure 7 It shows that it can be made by Figure 1 A flowchart illustrating an example of an encoding method implemented by an encoding device;

[0078] [ Figure 8 ] Figure 8 It shows that it can be made by Figure 2 A flowchart illustrating an example of a decoding method implemented by a decoding device;

[0079] [ Figure 9 ] Figure 9 A schematic diagram of the decoding method used in one embodiment of the present invention is shown. Detailed Implementation

[0080] Figure 1 The digital image encoding device ENC is illustrated schematically.

[0081] Figure 1 The input of the encoding device ENC receives a series of samples to be encoded (denoted as P). n For example, a set of image data (represented as I(P)). n Image signal I(P) n ) can represent a two-dimensional image, or multiple two-dimensional images (video, color components, stereo components, multi-view components, etc.). P n A sample n represents an input signal containing N samples. In one embodiment, the signal is a color image signal represented by at least one two-dimensional representation (such as a pixel matrix) with a width of W and a height of H (where N = W × H), where each pixel has a red component (R), a green component (G), and a blue component (B), or, as a variant, a luminance component (Y) and a width of W. u / v And the height is H u / v At least one chromaticity component (U and / or V). The position of each pixel is defined by its x-axis and y-axis coordinates (x and y) in the image. In one embodiment, the image is grayscale and represented by a two-dimensional representation (such as a pixel matrix), with each pixel having either a grayscale component or a luminance component. Typically, the number of components is represented as NC.

[0082] The encoding device ENC includes a feature map generation module INIT, a data processing and extraction module XTR, a synthesis module MLP, a convolution module MPP corresponding to at least one convolutional artificial neural network, a feature map encoding module FMC, a neural network encoding module NNC, a performance function evaluation module EVAL, and an update module MAJ.

[0083] The synthesis module for encoding the MLP and the synthesis module for decoding the MLP are defined by the same structure (e.g., including a set of filter banks or at least one artificial neural network) and a set of weights and activation functions respectively associated with the artificial neurons of the network in question.

[0084] The convolutional artificial neural network (at least one) used to encode the MPP and the convolutional artificial neural network used to decode the MPP are defined by the same structure (e.g., including multiple layers of artificial neurons) and a set of weights and activation functions associated with the artificial neurons of the network in question, respectively.

[0085] During the encoding training or construction process, i.e., when the performance evaluation step is still unsatisfactory, the encoding modules (FMC and NNC) simulate encoding and then decode, and the results are sent to the evaluation module. Subsequently, they perform FM on M feature maps. i The values ​​of (where 1 < i <= M) and the parameters of the neural networks MLP and MPP (denoted as W) k and V k Perform the actual encoding. In a known manner, the encoding simulation can be the same as or approximately the actual encoding.

[0086] As defined previously, each feature map contains scalar data, and there are M feature maps. However, feature values ​​can be organized in different ways, such as by forming a single signal containing a vector with M scalar values ​​for each sample point, without loss of generality. Even with different data organization methods, the remainder of this document still assumes the existence of M feature maps. A map encoded by the module FMC for a signal (e.g., the current image (or multiple images)) is represented as an FMC. i If necessary, the encoding module FMC quantizes the feature map values ​​using a quantizer to generate an ordered set of quantized values. Next, the encoding module uses, for example, an entropy encoder to compress the quantized data.

[0087] The XTR module processes and extracts data from the feature map. The XTR module can quantize the values ​​of the feature map using a quantizer Q to generate an ordered set of quantized values. Alternatively, the XTR module can obtain data quantized by the FMC module. It should be noted that quantization of a value refers to matching that value with a member of a discrete set of possible code symbols. For example, this set of possible code symbols can be formed from integer values, and the quantization system simply rounds the actual value to an integer. According to another example, quantization involves multiplying by a given value and then rounding. Next, according to an embodiment, the XTR module can transform the values ​​of at least one feature map in the feature map, for example, by performing upsampling, interpolation, filtering, etc. In one embodiment, at the end of the transformation, the transformed feature map has the same resolution as the input signal. The XTR module then extracts data from the feature map that has been so quantized and optionally transformed (denoted as FMt). i Extract data from ) to present the data to the MLP module.

[0088] The MLP module is composed of its parameter W k The defined function processes the transformed feature map at the input to output a set of NL potential maps L j Synthesis modules (where 1 < j <= NL), such as neural networks, filter banks, or linear inverse transforms (wavelets, DCT, etc.).

[0089] According to one embodiment:

[0090] - The XTR module is for the sample P to be encoded. n Based on its coordinates in the input signal, from the transformed feature map FMt i Extract values ​​from the vector Z. The extracted values ​​form a vector Z. n Z n It is a J-tuple, meaning it contains J elements or data z. i The vector Z with index n n It refers to pixel P n eigenvectors.

[0091] - The MLP module consists of K parameters W k Define a neural network, where 1 < k <= K, that is able to process vector Z at its input. n Alternatively, J-tuples can be processed to output a sample P representing the data to be encoded. n The second vector of latent values. According to one embodiment, the synthetic neural network includes an MLP or multilayer perceptron, which consists of an input layer suitable for an input format (J-tuples), optionally one or more hidden layers, and an output layer suitable for an output vector (typically a vector containing NL elements, each element intended for use in a latent value map). According to one embodiment, NL equals 3, and the output vector is a triple (e.g., Y, U, and V) to be inserted at pixel coordinates into three latent value maps L1, L2, and L3. According to another embodiment, NL equals 4, and the output vector is a quadruple to be inserted at pixel coordinates into four latent value maps L1 through L4. According to yet another embodiment, NL is between the graph FM... i The relationship between the number of components M and the number of components in the image is such that, for three components, M ≤ NL ≤ 3.

[0092] According to one embodiment, the NL potential value map contains any real values.

[0093] The latent value map represents the intermediate signal before convolution processing.

[0094] (At least one) neural network MPP is of the convolutional type, that is, it contains at least one convolutional layer. It may include convolutional elements, which include residual or non-residual structures and / or attention modules, as described below.

[0095] The parameter W of the synthesis module encoded by module NNC k Represented as Wc k .

[0096] The parameters V of the (multiple) convolutional neural networks encoded by the NNC module k Represented as Vc k .

[0097] Network MPP is applied to the latent value graph (L) at the output of the MLP. j To obtain the decoded sample P' n When all samples P of the signal n When it has been encoded and then decoded, the reconstructed signal I(P') can be obtained. n That is, at least one image I containing N decoded samples.

[0098] The EVAL module evaluates and minimizes coding performance. For example, the evaluation function is of the bitrate / distortion type. Its goal is to minimize the current image I(P) n The representation of ) at the input and its representation at the output I(P') n Minimize the differences between them, while optimizing the amount of data to be encoded, i.e., the encoded feature maps and the parameters Wc of the synthesis module. k And the parameters Vc of (multiple) convolutional neural networks k Minimization can be performed using gradient descent or any other method within the capabilities of someone skilled in the art.

[0099] The MAJ module updates the FM (Factor Module) to be encoded for the current image based on the result of the performance function. i The value of .

[0100] Once the encoder training is complete, the neural network parameters are either losslessly encoded (in this case, the decoder's neural network MLP' / MPP' is the same as MLP / MPP) or lossily encoded (in this case, the network MLP' / MPP' may be slightly different from MLP / MPP). Feature Map FM i The value is also encoded data FMC i Encode in the form of, for example, via an entropy encoder.

[0101] Figure 2 The decoding device DEC is shown schematically.

[0102] Figure 2 The decoding device DEC receives the first set of encoded data at the input, which is organized into M feature maps FMC. i The parameter Wc of the synthesis module k and at least one convolutional neural network parameter Vc k .

[0103] The decoding device DEC includes a neural network decoding module NND, a feature map decoding module FMD, an inverse transform and data extraction module XTR', a synthesis module MLP', and a convolution module MPP' corresponding to at least one convolutional neural network.

[0104] The graph (of number M) decoded by module FMD is represented as FMd. i .

[0105] The parameters decoded by the NND module are represented as Wd. k and Vd k If the synthesis module does not contain a neural network, different decoders can be used to process the parameter Wd. k Perform decoding. If lossless encoding has already been performed, then the parameter Vd... k and Wd k With encoder parameter V k and W k same.

[0106] The XTR module processes and extracts the decoded feature map FMd. i The module XTR' optionally performs inverse quantization corresponding to the quantization performed on the encoder. Inverse quantization is not required if the encoder's quantizer Q only rounds the actual value transmitted to it. Inverse quantization is also not required if the neural network can take into account the quantization of its input data. Otherwise, the decoder performs the inverse operation of the quantizer Q. Next, according to an embodiment, the module XTR' can transform the values ​​of at least one feature map in the feature map, for example, by performing upsampling, interpolation, filtering, etc. In one embodiment, at the end of the transformation, the transformed feature map has the same resolution as the input signal. The module XTR' then extracts the values ​​from the feature map that has been so quantized and optionally transformed (denoted as FMt'). i Extract data from ) to present the data to the MLP module.

[0107] As referenced above Figure 1 As described, module MLP' is composed of its parameter Wd k The defined function processes the transformed feature map at the input to output a set of NL potential maps L' j Synthesis modules (where 1 < j < NL), such as neural networks or linear inverse transforms (wavelet, DCT, ...).

[0108] In a preferred embodiment:

[0109] - Module XTR and Figure 1 The module XTR is the same. This module is for the sample P to be decoded. n Based on its coordinates in the signal to be decoded, from M transformed feature maps FMt' i Extract values ​​to construct a vector Zd in the form of J-tuples. n In one embodiment, J = M.

[0110] - Module MLP' includes K parameters Wd kDefine a neural network that can process a vector Zd extracted from a feature map at its input. n Alternatively, J-tuples can be processed to output a representation of the sample P to be decoded. n The second vector of potential values ​​Zld n According to one embodiment, the synthetic neural network includes an MLP or a multilayer perceptron, which consists of an input layer suitable for an input format (J-tuples), optionally one or more hidden layers, and an output layer suitable for an output vector (typically a vector containing NL elements, each element intended for use in a latent value map). According to one embodiment, NL equals 3, and the output vector is a triple to be inserted at pixel coordinates into three latent value maps L'1, L'2, and L'3. According to another embodiment, R equals 4, and the output vector is a quadruple to be inserted at pixel coordinates into four latent value maps L'1 to L'4. According to yet another embodiment, R is between the values ​​in the graph FMd. i The number M is related to the number of components of the (multiple) images to be decoded. For three components, M ≤ R ≤ 3.

[0111] The latent value map represents the intermediate signal before convolution processing.

[0112] At least one neural network MPP' is of the convolutional type, i.e., it contains at least one convolutional layer. It may include convolutional elements comprising residual or non-residual structures and / or attention modules, as described below. At the output of module MPP', an image or multiple reconstructed images I(Pd) may be obtained. n ).

[0113] The decoding device DEC can be implemented by an electronic device including a processor and memory (not shown); therefore, each of the aforementioned modules can be generated by the interaction of the processor with computer program instructions stored in the aforementioned memory and designed, in particular as described below, to perform the function of the module in question when these instructions are executed by the processor.

[0114] Figure 3 The schematic illustration shows the synthetic and convolutional artificial neural networks used in one embodiment of the invention during encoding and decoding.

[0115] According to this embodiment, the synthesis module includes a neural network MLP / MLP', and the convolution module includes a neural network MPP / MPP'. The synthetic artificial neural network MLP / MPP used for encoding and the synthetic artificial neural network MLP' / MPP' used for decoding are defined by the same structure (e.g., including multiple layers of artificial neurons) and a set of weights and activation functions respectively associated with the artificial neurons of the network in question.

[0116] The vector representation of the current sample point (from the feature map FMt) i or FMt' i The obtained vector Z n or Zd n This is applied to the input (i.e., the input layer) of a synthetic artificial neural network (MLP) or MLP'. The output of the synthetic artificial neural network is a vector ZL representing the latent values ​​of the decoded samples. n or ZLd n These potential values ​​were re-inserted into the NL potential value map L. j / L' j middle. Figure 3 The diagram shows a layer of a synthetic neural network, which contains layers denoted as u1...u 40 The system has 40 outputs. Each output is followed by a ReLU activation function. The ReLU (Rectified Linear Unit) function is well-known to those skilled in the art. The output layer contains three components, denoted as v1...v3, followed by ReLU functions. At the output of module MLP / MLP', the value obtained by each ReLU function (containing the vector ZL) is... n / ZLd n ) is introduced into Figure L j The location is defined by the position of the pixel n being processed. The concatenation of all these values ​​forms the decoded latent value map L. j / L' j .exist Figure 3 Therefore, the number of potential value maps is 3.

[0117] Then, the decoded latent value map L j / L' j Processed by a convolutional module MPP / MPP', which outputs a decoded signal, which, according to one embodiment, is the color components (R, G, B) that form image I.

[0118] according to Figure 3 For example, a convolutional network consists of a residual convolutional layer (CR1) at the input, a set of ReLU functions, and a residual convolutional layer (CR2) at the output. Modules CR are residual convolutional elements without any restrictions. These modules can have kernels of any size. These modules can be simple convolutional types or include attention elements. These modules can also include stride (the offset of the convolutional kernel per iteration), dilation (the spacing of the input values ​​supplied to the kernel), etc. As is well known to those skilled in the art, a bias potentially follows the convolutional elements. Any activation function within the capabilities of those skilled in the art (ReLU, sigmoid, Softmax, Softplus, Softsign, etc.) can be used.

[0119] Convolutional elements can consist of 2D convolutions, where convolutional kernels are jointly applied to rows and columns of the latent value map. Convolutional elements can also consist of separable 1D convolutions. In this case, instead of performing convolutions on the 2D latent value map, a first 1D convolution operator of length Lh is applied to the rows, and then a second convolution of length Lv is performed on the columns of the map obtained after the first operator. Row / column processing can be reversed to perform column processing, followed by row processing. This embodiment reduces the inseparability of a given kernel pair Lh at Lv+Lh. The effect of Lv storage requirements, and limits the complexity to Lh+Lv multiplications per graph value, instead of Lh in the 2D case. Lv.

[0120] Naturally, a convolutional module itself can contain multiple cascaded networks. Each network can contain multiple biased or unbiased convolutional layers, accompanied by ReLU or another type of activation function, etc.

[0121] Figure 4 The schematic illustration shows the synthetic and convolutional artificial neural networks used in another embodiment of the invention during encoding and decoding.

[0122] These networks are able to process images whose components have different resolutions.

[0123] As in Figure 3 In this embodiment, the synthesis module includes a neural network MLP / MLP', and the convolution module includes a neural network MPP / MPP'. The vector representation of the current sample point (from the feature map FMt) is... i or FMt' i The obtained vector Z n or Zd n This is applied as input to a synthetic artificial neural network (MLP) or MLP'. The output of the synthetic neural network is a vector ZL representing the latent values ​​of the decoded samples. n or ZLd n These potential values ​​were re-inserted into the NL potential value map L. j / L' j middle. Figure 4 The layer of the synthetic neural network is shown, containing 40 neuron outputs denoted as u1...u40. Each output is followed by a ReLU activation function. The output layer contains six components denoted as v1...v6, each followed by a ReLU function. At the output of module MLP / MLP', the resulting value of each ReLU function is introduced into Figure L. j The location is defined by the position of the pixel being processed. The concatenation of all these values ​​forms the decoded latent value map L. j / L' j .exist Figure 4 Therefore, the number of potential value maps is 3.

[0124] However, with Figure 3 Unlike other embodiments, in this embodiment, the potential value map L j Not all three have the same number of samples. Module MLP / MLP' generates more latent values ​​for the first graph than for subsequent graphs: 4 values ​​for the first graph represented as Ly, and 4 values ​​for graph L. u Figure L v One value was generated. This advantageously produces latent value maps corresponding to the chromaticity components, which contain fewer samples compared to the latent value maps corresponding to the luminance components. This size adaptation allows for a reduction in the number of operations performed and the number of network parameters to be encoded and transmitted (or decoded). This number can be adjusted according to the dimension of the components.

[0125] Then, the decoded latent value map L j / L' j Processed by a convolutional module MPP / MPP', the output of which is a decoded signal, which, according to this embodiment, forms the color component (I) of image I. Y I u I v The U and V components typically contain one-quarter of the samples of the Y component.

[0126] The architecture of convolutional networks and Figure 3 The architecture is the same, and the same descriptions apply. Convolutional elements are represented as CR1, CR2, CR3, and CR4.

[0127] Figure 5 A convolutional network according to an embodiment of the present invention is illustrated schematically.

[0128] Cases (a) through (d) illustrate, in the context of this invention, the convolutional layers used to process three latent value maps (NL = 3) at the input and a three-component image YUV (NC = 3) at the output of the convolutional module. In these figures, the slashes with numbers at the top indicate the number of latent value maps at the input and the number of image components at the output.

[0129] Naturally, other embodiments (not shown) can be envisioned, which have a more or less high number of latent value maps and a variable number of output components (e.g., for an image with NC = 3 YUV components at the output, NL = 4 latent value maps), without loss of generality.

[0130] Furthermore, in the illustrated embodiment, according to the previously given definition of latent value maps, each latent value map contains only scalar data, and there are three latent value maps, each corresponding to one of the three coordinates of a sample point of the signal. However, the latent values ​​can be organized in different ways, for example by forming a single signal that contains a vector with three values ​​for each sample point, without loss of generality.

[0131] These convolutional layers (which use convolutional elements denoted as Ci and attention elements (SGM: sigmoid function)) can be combined in a cascaded manner. The complexity increases as shown, both in terms of memory usage (the number of filter coefficients) and the number of arithmetic operations involved.

[0132] Case (a) illustrates a simple convolution processing the NL latent value map Li / L'i at the input. The NL map (NL = 3 in the example) undergoes a simple convolution, denoted as C. Filtering is performed using a kernel of size k × k (e.g., k = 3). The kernel (3 × 3) is applied to different locations in the map to obtain NC (NC = 3 in the example) components of the image (e.g., YUV) at the output. Therefore, the required number of operations is 3. 3 3 3 (= 81). Typically, the number of operations is NL. NC k k. Use size k k of NL The NC kernel must be stored and / or transmitted. Additionally, the NC bias must also be stored and / or transmitted.

[0133] Case (b) exhibits the same complexity but with a residual structure: unlike case (a), the signal generated by the convolution is combined (added, subtracted, multiplied) with the original signal consisting of three mixed latent values. The memory footprint (number of operations) is the same as that of case (a), i.e., NL. NC k It requires k operations, but this structure can facilitate the learning of neural networks.

[0134] Case (c) shows the residual structure after processing the latent value maps of NL (based on the example NL = 3) in a differentiated manner: Figure (L' o The Y component used to generate the image is denoted as I. Y Its convolution is represented as C1, and its kernel size is k1. k1. The other two graphs (L'1 and L'2) are used to generate the U and V components of the image, denoted as I. U and I V The convolution of these two graphs is represented as C2, and its kernel size is k2. k2. The case where the memory footprint is less than the combined processing of three graphs, because C1 requires only one operation. 1 k1 k1, and C2 requires 2 operations. 2 k2 k2. In this case, it is necessary to store 3 biases. See reference. Figure 4 As explained, this structure is well-suited for processing images with different resolutions.

[0135] Case (d) shows the structure of the attention module SGM that performs joint processing on NL potential value maps. It has a larger memory footprint because C3 requires NL operations. NC k3 k3, and the number of operations required for C4 is NL. 1 k4 k4. In this case, it is necessary to store NC+1 biases. However, this structure improves the local processing of images (especially contours present in the image).

[0136] a. Regarding C2:2 2 k2 k2

[0137] Figure 6 A convolutional network according to an embodiment of the present invention is illustrated schematically.

[0138] The left side of the figure illustrates low-complexity differential processing that independently processes the three latent value maps. It is capable of processing maps with different resolutions, so each component of the image can have its own resolution, for example, to support the YUV 420 format, where the U and V components are smaller than the Y component. The convolutional elements are denoted as C5, C6, and C7.

[0139] The right side of the figure shows Figure 5 An alternative view of the differentiated processing in case (c). The convolutional network can process the YUV format separately; the Y component is processed separately, while the U and V components are processed together. The convolutional elements are represented as C8, C9, C10, C11, and C12.

[0140] The following table summarizes Figure 5 and Figure 6 The example shows the number of operations (MAC) and the number of values ​​stored (storage) required for each case. The example assumes that there are three potential value maps (NL = 3), three outputs (NC = 3), and all convolutional kernels are 3 × 3 in size (k = k1 = k2 = ... = 3).

[0141] [Table 1]

[0142]

[0143] Figure 7 It shows that it can be made by Figure 1 A flowchart illustrating an example of an encoding method implemented by an encoding device.

[0144] According to this embodiment, the signal is a two-dimensional image, and therefore each sample point to be encoded has coordinates (x, y). n , y n ) pixels P n .

[0145] The encoding process is carried out in two main stages:

[0146] In the first stage, known as the construction stage, learning is performed to adapt to the input signal I(P) n ) Determine the FM diagram i The value and the parameter W of the synthesis module k And the parameters V of (multiple) convolutional neural networks k This is done to optimize the cost function. For example, by using gradient descent and then updating the feature map FM. i The value of the cost function, along with the values ​​of the synthesized neural network and convolutional neural network, is used to perform learning. As known in the art, the cost function can be of the bitrate / distortion type, or of the bitrate, distortion, or perceptual type. To measure the bitrate R, a simulation of the graph FM is required. i The encoding is then performed, and then the associated code rate (the size of stream B1) needs to be measured. According to one embodiment, the parameter W is not simulated. k and / or V k The encoding is performed because their influence is less than that of the feature map. According to one embodiment, simulation is also performed on the parameter W. k and / or V k The encoding of the image I is performed, and the associated bitrate (the size of stream B2) is measured. To measure the distortion D, it is necessary to simulate the encoding and then decoding of at least a portion of the image I to obtain at least one pixel P' generated by the simulation of the encoding and then decoding. n Then, measure the image I(P) at the input. nThis part is related to the encoded and then decoded image I(P'). n The difference between the corresponding parts of ).

[0147] Next, in the second stage, known as the encoding stage, the graph FM is... i and parameter V k And possible W k Encode to produce the encoded value FMC i and Vitamin C k (and possible Wc) k ), and then they are transmitted or stored. They form the input signal I(P) n The compressed representation of ).

[0148] The steps of a method according to an embodiment of the present invention will now be described.

[0149] In step E20, P containing multiple (N) sample points will be... n The signal to be encoded is I(P) n The input passed to this method.

[0150] In step E21, the M feature maps FM i Initialization is performed. Subsequently, the parameters W of the synthesis module MLP must be optimized during the build phase. k FM i The value of and (at least one) parameter V of the convolutional neural network k .

[0151] According to one embodiment, Figure FM i Having the same characteristics as the input signal I(P) n The same resolution, and therefore each map contains the same number of samples P to be encoded. n The number of values ​​N is the same as the number of values ​​N. According to one embodiment, Figure FM i The resolution is less than or equal to the input signal I(P) n The resolution of ) and therefore for at least one of these graphs, it contains less than N values ​​to be encoded. According to one variant, the first graph FM i It has the resolution of the image, and each subsequent image has the resolution of the previous one. Figure 1 Half the resolution. According to one embodiment, multiple graphs FM i It has the same resolution, which is smaller than the input signal I(P) n (resolution).

[0152] According to one embodiment, the graph FM is initialized using a predefined constant value. i According to another embodiment, a set of random real values ​​is used to initialize the feature map.

[0153] In one embodiment, the structure and parameters of the neural network are initialized in this step. These parameters are then updated or refined in subsequent iterations of the method during the build phase. According to one embodiment, the parameters of the synthesis module and / or the convolutional neural network are initialized with predefined values ​​known to produce satisfactory cost function values ​​(e.g., after training on an image database). According to another embodiment, the parameters of the synthesis module and / or the convolutional neural network are initialized with a set of random values.

[0154] Subsequently, in step E22, the encoder's update module MAJ updates or refines the feature map FM during its learning period. i And the parameters of the neural network.

[0155] In step E23, the encoder module FMC processes the image FM. i Encoding is performed. During the construction phase, this operation is an encoding simulation. During the encoding phase, this operation is the actual encoding, and the encoded values ​​form stream B1. The simulation can be the same as the actual encoding, but it can also be different (e.g., simplified). For this type of encoding, for example, entropy encoding of the quantized values ​​of (multiple) feature maps can be used.

[0156] In one embodiment, Figure FM i Encoding is performed in the order (FM1, FM2, ..., FM4), and the variables of each graph are encoded in a predefined order (e.g., lexicographical order). Each graph undergoes entropy coding. Entropy coding produces a compressed stream B1, and the bitrate of this compressed stream is subsequently measured during step E28.

[0157] In step E24, according to one embodiment, for M graphs FM i The data is quantized, transformed, and then extracted by the XTR module.

[0158] According to one embodiment, at least one graph FM i The resolution of the image is smaller than the resolution of the sequence to be encoded, and the transform operation includes upsampling, such that the transformed image FMt i It contains the same number of samples as the images in the sequence. Upsampling involves adding values ​​to the image FMt. i This is done to achieve the resolution of the input sequence of images. This operation can be simple (through nearest neighbor copying) or include interpolation (linear, polynomial, filtering, etc.). It can be performed using an upsampling neural network.

[0159] According to one embodiment, the module XTR targets the sample P to be encoded. n Based on its coordinates in the input signal, from the transformed feature map FMt i Extract values ​​from the vector Z. The extracted values ​​form a vector Z. nZ n It is a J-tuple, meaning it contains J elements or data z. i The vector Z with index n n It refers to pixel P' n eigenvectors.

[0160] In one embodiment, Z n It is from the current pixel P n coordinates (x) n , y n ) at the FM position i The values ​​form J-tuples (z1, z2, ..., z...) J In one embodiment, Z n From the diagram FM i The J-tuples are constructed from the values ​​obtained in the graph, and the coordinates at which these values ​​were obtained vary depending on the graph. For example, if the graph FM... i Because they have been downsampled to have different resolutions, the coordinates (by scaling) are adapted to match the resolution of each graph. In one embodiment, Z... n J-tuples are constructed from values ​​obtained by applying processing to one or more values ​​in the graph (e.g., filtering neighboring values ​​of the target value in the graph). i Acquired. For example, for a graph FM with the same resolution as the input signal. i The current sample point P in n It can extract the coordinates (x) n , y n ), (x n -1, y n ), (x n , y n -1) and (x n -1, y n The values ​​of -1) are then processed (filtering, averaging, interpolation, etc.) to obtain the values ​​corresponding to the FM graph. i The relevant vector Z n The final value of element i (z) i According to another example, in a graph FM with half the resolution of the input signal... i In the middle, it can be considered to be located at coordinate (x n / 2,y n / 2), (x n / 2-1, y n / 2), (x n / 2, y n / 2-1) and (x n / 2-1, y nThe values ​​of / 2-1) can be processed (filtering, averaging, interpolation, etc.) to obtain the values ​​corresponding to the FM graph. i The relevant vector Z n The final value of element i (z) i The samples need to be encoded, for example, processed in order from n = 1 to n = N.

[0161] In step E25, the feature map is processed by the synthesis module to generate a latent value map. The synthesis module, for example, is composed of its parameter W. k The defined function processes the transformed feature map at the input to output a set of NL potential maps L j (where 1 < j <= NL) Neural networks, filter banks, or linear inverse transforms (wavelets, DCT, etc.).

[0162] According to one embodiment, vector Z n The second vector ZL is output by processing the data through a synthetic neural network MLP. n The second vector contains the sample point P to be encoded. n The output vector contains NL potential values. According to one embodiment, NL equals 3, and the output vector is a triple to be inserted into three graphs L1, L2, and L3. According to another embodiment, NL equals 4, and the output vector is a quadruple to be inserted at the pixel coordinates into four potential value graphs L1 through L4. According to yet another embodiment, NL is between graph FM... i The number of latent values ​​M is related to the number of components in the image; for NC = 3 components, M ≤ NL ≤ 3. Latent values ​​can also be organized in different ways, for example, by forming a single signal containing a vector with NL scalar values ​​for each sample, without loss of generality. The latent value map represents the intermediate signal before convolution processing.

[0163] In step E26, at least one convolutional neural network MPP is applied at the output of module MLP to the latent value map (L). j To obtain the decoded sample P' n For example, it means to encode sample point P. n The vector, according to one embodiment, is the sample point P' n (Sample P that has been encoded and then decoded) n The triple (R, G, B) of ).

[0164] In step E27, the parameters V of the synthesis module and the convolution module are... k and W kQuantization and encoding are performed. During the build phase, this operation is an encoding simulation. During the encoding phase, this operation is the actual encoding, and the encoded values ​​form stream B2. The simulation can be the same as the actual encoding, but it can also be different (e.g., simplified). For this, any known technique can be used, such as the neural network coding standard proposed in Part 17 of the MPEG-7 standard, also known as NNR (Neural Network Representation). It should be noted that in this case, the encoding pair parameter V needs to be selected. k and W k The amount of downgrade caused.

[0165] In step E28, the performance measurements are evaluated.

[0166] Therefore, measurement and feature map (through graph FM) i Encoding is performed to simulate stream B1) and optionally with the parameters of the synthesis and convolution modules (via parameter V). k / W k Encode to simulate the bitrate associated with stream B2).

[0167] According to one embodiment, the cost function is of the bitrate / distortion type, denoted as (D+L). R), where D is, for example, the squared error (or the error measured on a subset of samples of the signal) between the input signal and the decoded signal. According to another example, D is calculated from a perceptual function such as SSIM (Structural Similarity) or MSSSIM (Multi-Scale Structural Similarity). According to one embodiment, R is the analog bitrate of stream B1; according to another embodiment, R is the total bitrate used to encode the image or image sequence, i.e., the sum of the analog bitrates of B1 and B2. L is a parameter that adjusts the bitrate / distortion tradeoff. Other cost functions may also be used.

[0168] As long as the cost function has not reached its minimum, the performance measurement is unsatisfactory, and the method repeats from step E22 onwards. This minimization can be performed using a mechanism called gradient descent, where the parameters are updated during step E22. Training stops if the cost function has reached its minimum. If the parameters (V) of the synthesis module and the convolutional module are... k / W k ) and feature map (FM) i The final simulation corresponding to the encoded version is available, thus forming streams B1 and B2. According to another embodiment, in this step, updated parameters (V) of the synthesis and convolution modules are performed. k / W k ) and feature map (FM) i The actual encoding of the value of ) to produce the encoded parameter Vc that forms streams B2 and B1. k / Wck and FMC i Streams B1 and B2 can be cascaded to produce the final stream. According to one embodiment, the encoded parameters of the synthesis and convolution modules in stream B2 are stored or transmitted before stream B1 so that they can be decoded before stream B1.

[0169] Figure 8 It shows that it can be made by Figure 2 A flowchart illustrating an example of a decoding method implemented by a decoding device.

[0170] In step E30, streams B1 and B2 are extracted from the encoded stream. These streams respectively contain graph FMc i And the parameters Wc of the synthesis module and the convolution module. k and Vitamin C k The encoded representation.

[0171] In step E31, by adjusting the value FMc i Decode to generate M graphs FMd i For this type of decoding, a decoding technique corresponding to the technique used by the encoder is employed, such as entropy coding (decoding). In one embodiment, Figure FMd i Decoding is performed in the order (FMd1, FMd2, ..., FMd4), and the values ​​of each graph are decoded in a predefined order (e.g., lexicographical order). The parameters Wd of the synthesis and convolution modules... k and Vd k It is through the value Wc of convection B2 k and Vitamin C k The generation is achieved through decoding. For this purpose, any known decoding technique corresponding to the encoding technique used by the encoder can be used. A convolutional neural network (MPP) is similar to a synthetic network (MPP), i.e., it has the same structure and contains the same parameters, except for the encoding, which can be performed lossily or losslessly. The same applies to the synthesis module, which can be implemented as a neural network.

[0172] According to one embodiment, stream B2 is decoded before stream B1 in order to obtain the parameters of the synthesis module and the convolution module before the decoding of the samples begins.

[0173] According to the embodiment described for the encoder: Figure FMd i Having the signal I(Pd) to be recovered n The same resolution, and therefore each image contains the same number of samples Pd to be decoded. n The number of values ​​N is the same as the number of graphs FMd. iThe resolution is less than or equal to the signal resolution, and therefore for at least one of these figures, it contains a number of values ​​to be decoded that is less than N. According to one variant, the first figure FMd i It has the resolution of the image, and each subsequent image has the resolution of the previous one. Figure 1 Half the resolution. According to one embodiment, multiple graphs FMd i They have the same resolution, but this resolution is smaller than the signal's resolution.

[0174] In step E33, according to the embodiment, the M graphs FMd are... i The data is dequantized, transformed, and then extracted by module XTR'.

[0175] According to one embodiment, at least one figure FMd i The resolution of the image to be decoded is smaller than the resolution of the image to be decoded, and the transform operation includes upsampling, such that the transformed image FMt' i It contains the same number of samples as the images in the sequence. Upsampling involves adding values ​​to the image FMt'. i This is done to achieve the resolution of the input sequence of images. This operation can be simple (through nearest neighbor copying) or include interpolation (linear, polynomial, filtering, etc.). It can be performed using an upsampling neural network.

[0176] According to one embodiment, module XTR' targets the sample Pd to be decoded. n Based on its coordinates in the signal, from the transformed feature map FMt' i Extract values ​​from the vector Zd. n Zd n It is a J-tuple, meaning it contains J elements or data. i The vector Zd with index n n It refers to pixel Pd n eigenvectors.

[0177] This step and reference Figure 7 The steps described for the encoder are similar to E24, and these embodiments are applicable. Specifically: in one embodiment, Zd n It is from the current pixel Pd n coordinates (x) n , y n The figure at ) FMt' i / FMd i The values ​​form J-tuples (zd1, zd2, ..., zd) J In one embodiment, Z n It is from the figure FMt' i / FMd iThe J-tuples are constructed from the values ​​obtained in the graph, and the coordinates at which these values ​​were obtained vary depending on the graph. For example, if the graph FMt' i / FMd i Because they have been downsampled to have different resolutions, the coordinates (by scaling) are adapted to match the resolution of each graph. In one embodiment, Zd n J-tuples are constructed from values ​​obtained by applying processing to one or more values ​​in the graph (e.g., filtering the neighbors of the target value in the graph). i / FMd i Acquired. For example, for the current sample Pd in ​​a graph FMdi with the same resolution as the image signal to be decoded. n It can extract the coordinates (x) n , y n ), (x n -1, y n ), (x n , y n -1) and (x n -1, y n The values ​​of -1) are then processed (filtering, averaging, interpolation, etc.) to obtain a result similar to the graph FMt'. i / FMd i The relevant vector Z n The final value of element i (zd) i In another example, in a graph FMt' with half the resolution of the signal... i / FMd i In the middle, it can be considered to be located at coordinate (x n / 2, y n / 2), (x n / 2-1, y n / 2), (x n / 2,y n / 2-1) and (x n / 2-1, y n The values ​​of / 2-1) can be processed (filtering, averaging, interpolation, etc.) to obtain the values ​​corresponding to the FM graph. i The relevant vector Zd n The final value of element i (zd) i The samples to be decoded are processed, for example, in order from n = 1 to n = N.

[0178] In step E34, the feature map is processed by the synthesis module MLP' to generate a latent value map. The synthesis module, for example, is composed of its parameter Wd. kThe defined function processes the transformed feature map at the input to output a set of NL potential maps L' j (where 1 < j <= NL) Neural networks, filter banks, or linear inverse transforms (wavelets, DCT, etc.).

[0179] This step and reference Figure 7 The steps described for the encoder are similar to E25, and these embodiments are applicable. Specifically, according to one embodiment, vector Zd... n The second vector ZLd is output by processing the data through a synthetic neural network MLP'. n The second vector contains the sample Pd to be decoded. n The output vector contains NL potential values. According to one embodiment, NL equals 3, and the output vector is a triple to be inserted into three graphs L'1, L'2, and L'3. According to another embodiment, NL equals 4, and the output vector is a quadruple to be inserted at the pixel coordinates into four potential value graphs L'1 to L'4. According to yet another embodiment, NL is between graph FMd. i The number of latent values ​​M is related to the number of components in the image; for NC = 3 components, M ≤ NL ≤ 3. Latent values ​​can also be organized in different ways, for example, by forming a single signal containing a vector with NL scalar values ​​for each sample, without loss of generality. The latent value map represents the intermediate signal before convolution processing.

[0180] In step E35, at least one convolutional neural network MPP' is applied at the output of module MLP' to the latent value map (L'). j To obtain the decoded sample Pd n For example, it means to decode sample point Pd. n The vector, according to one embodiment, is the sample point Pd. n The triplet (R, G, B). This step is consistent with the reference. Figure 7 The steps described for the encoder are similar to those in step E26.

[0181] When all potential value maps have been processed, we obtain, for example, an image I(Pd) n The corresponding decoded signal.

[0182] Figure 9 A schematic diagram of the decoding method used in one embodiment of the present invention is shown.

[0183] In this embodiment, Figure FMd i The quantity is 5. In a preferred embodiment, the quantity is 7.

[0184] In this embodiment, the first image FMd1 has the same resolution as image I and therefore contains W × H variables, where W is the width of the image (in pixels) and H is its height. The second image FMd2 has half the resolution of image FMd1 (in each dimension). Image FMd3 has half the resolution of image FMd2 (in each dimension). Images FMd4 and FMd5 have half the resolution of image FMd3 (in each dimension). This structure allows for a reduction in the number of variables in the feature maps, thereby facilitating decoding while minimizing encoding costs.

[0185] The module XTR' dequantizes the values ​​of the feature map when necessary.

[0186] The feature maps are then processed by module XTR', which, according to this embodiment, upsamples these feature maps to give them the same resolution as the image to be decoded. (For example, FMd2 is upsampled by 2x in each dimension, FMd3 by 4x in each dimension, and so on.)

[0187] Then, module XTR' extracts the data from the current pixel Pd for each sample location. n coordinates (x) n , y n Transformed diagram FMd at point ) i The values ​​form a vector Zd in the form of 5-tuples (zd1...zd5). n Vector Zd n Processed by a neural network MLP' to output vector ZPd n According to the example, this vector represents the sample Pd to be decoded. n The quaternion, whose coordinates (x, y) are in the components. n , y n The value at point ) is reintroduced into the latent value map L' i middle.

[0188] Based on this example, the potential value map L' i The quantity is 4.

[0189] According to this example, the four latent value maps are processed by the convolution module MPP' to output three components Rd, Gd and Bd of the decoded image.

Claims

1. A method for processing at least one image (I(P) n The encoding method includes the following steps: - The step of constructing the at least one encoded and then decoded image includes the following sub-steps: - Construct a set (M) feature maps (FM) of type (E21, E22) i ); - By using the synthesis parameters (W) k The defined synthesis module (MLP) processes the set of feature maps (E24, E25) to provide a set (NL) of latent value maps (L j The number of potential value maps is less than or equal to the number of feature maps; - By the convolution parameters (V) k At least one convolutional neural network (MPP) as defined in E26 processes the set of latent value maps to provide at least one encoded and then decoded image (I(P')). n )); - Update at least one value of one of the feature maps (E22, E28) and / or at least one synthesis parameter and / or at least one convolution parameter based on coding performance measurements; - For the set of feature maps (FM) i ) and the set of synthesis parameters and convolution parameters (W) k / V k The steps for encoding (E23, E27) are as follows.

2. The encoding method as described in claim 1, characterized in that, The at least one image includes multiple samples (P) to be encoded. n ), and is characterized in that, for the set of feature maps (FM) i Processing is performed to provide a set of potential value maps (L i The steps include the following sub-steps: - For at least one image to be encoded, the current sample (P) is called the current sample. n At least one sample point of ), the sample point being associated with the location (x) n y n Related to: - Based on the current sample point (P) n The position (x) n , y n From the feature map (FM) i Constructing (E24) eigenvectors (Z) n ); - The feature vector (Z) is processed by the synthesis module (MLP). n The sample is processed (E25) to provide a potential value (ZL) representing the current sample point. n The latent vector of ); - From the at least one latent vector (ZL) n Construct the potential value map (L) for this set of values. i ).

3. A method for processing at least one image (I(Pd) from a bitstream (B1, B2). n The decoding method includes the following steps: - For a set (M) feature maps (FMd) i Decode (E31); - Obtain (E30, E32) representing a set of synthesis parameters (Wc) for the synthesis module (MLP'). k ); - Obtain (E30, E32) to represent a set of convolutional parameters (Vc) of at least one convolutional neural network. k ); - The set of feature maps is processed (E33, E34) by the synthesis module (MLP') to provide a set of latent value maps (L'). j The number of potential value maps (NL) is less than or equal to the number of feature maps (M); - The set of latent value maps are processed (E35) by the at least one convolutional neural network (MPP) to provide the at least one decoded image (I(Pd)). n )).

4. The method for decoding a signal as described in claim 3, characterized in that, The at least one image includes multiple samples (Pd) to be decoded. n The step of processing the set of feature maps to provide a set of latent value maps includes the following sub-steps: - For at least one image to be decoded, the current sample (P) is called the current sample. n At least one sample point of ), the sample point being associated with the location (x) n , y n Related: - Based on the position (x) of the current sample point n , y n Construct (E33) feature vectors (Zd) from these feature maps n ),as well as: - The feature vector (Zd) is processed by the synthesis module (MLP'). n The sample is processed (E34) to provide a latent vector (ZLd) representing the potential value of the current sample. n ); - From the at least one latent vector (ZLd) n Construct the potential value map (L') for this set of values. i ).

5. The decoding or encoding method as described in any one of the preceding claims, characterized in that, The synthesis module (MLP, MLP') includes a multilayer perceptron.

6. The decoding or encoding method as described in any one of the preceding claims, characterized in that, The at least one convolutional network (MPP, MPP') contains at least one convolutional layer (C, C1, C2, C3, C4, C5).

7. The decoding or encoding method as described in any one of the preceding claims, characterized in that, The number of potential value maps (NL) is equal to the number of components (NC) of the at least one image.

8. The decoding method according to any one of claims 3 to 7, characterized in that, This combination consists of parameters and / or convolution parameters (Wc) k , Vc k At least a portion of it is obtained from the data in the bitstream.

9. The decoding method as described in any one of claims 3 to 8, characterized in that, This combination consists of parameters and / or convolution parameters (Wc) k , Vc k At least a portion of it is obtained from predetermined parameters.

10. The decoding method according to any one of claims 3 to 9, characterized in that, Based on the reference parameters, this combination is used to form parameters and / or convolution parameters (Wc). k , Vc k Decode at least a portion of ).

11. The decoding method according to any one of claims 3 to 10, characterized in that, The at least one convolutional network comprises at least two convolutional networks.

12. The decoding method as described in the preceding claim, wherein, The bitstream contains indicators indicating the use of at least one of the networks.

13. The decoding method according to any one of claims 6 to 12, characterized in that, The at least one image has at least two dimensions (x, y), and the convolutional layer uses a kernel that is separable in the at least two dimensions.

14. A method for processing at least one image (I(P) n The device for encoding is characterized in that, The device is configured to perform: - The step of constructing the at least one encoded and then decoded image includes the following sub-steps: - Construct a set (M) feature maps (FM) (INIT, MAJ) i ); - By using the synthesis parameters (W) k The defined synthesis module (MLP) processes the set of feature maps (XTR, MLP) to provide a set of latent value maps (L). j The number of potential value maps (NL) is less than or equal to the number of feature maps (M); - By the convolution parameters (V) k A convolutional neural network (MPP) as defined in the diagram processes the set of latent value maps to provide at least one encoded and then decoded image (I(P')). n )); - Update at least one value of one of the feature maps and / or at least one synthesis parameter and / or at least one convolution parameter according to the coding performance measurement (EVAL, MAJ); - For the set of feature maps (FM) i ) and the set of synthesis parameters and convolution parameters (W) k / V k The steps for encoding (FMC, NNC).

15. A method for processing at least one image (I(Pd) from a bitstream) n The device for decoding is characterized in that, The device is configured to perform the following operations: - For a set (M) feature maps (FMd) i Decode (FMD); - Obtain (NND) represents a set of synthesis parameters (Wc) for the synthesis module (MLP'). k ); - Obtain (NND) represents a set of convolutional parameters (Vc) of at least one convolutional neural network. k ); - The set of feature maps is processed (XTR', MLP) by the synthesis module (MLP') to provide a set (NL) of latent value maps (L'). j The number of potential value maps is less than or equal to the number of feature maps; - Process the set of latent value maps using the at least one convolutional neural network (MPP) to provide the at least one decoded image (I(Pd)). n )).

16. A computer program comprising instructions which, when executed by a computer, are used to perform the steps of the encoding method of claim 1 or the decoding method of claim 3.

Images