Method and apparatus for encoding and decoding an image using a correction mechanism

Abstract

The invention relates to a method for encoding and decoding a signal containing a plurality of samples. The decoding method comprises the following steps: obtaining at least one feature map, called original map, representing the signal in a latent domain; decoding the parameters of a first neural network, called synthetic neural network, comprising at least two synthetic neural layers; decoding the parameters of a second neural network, called control network; processing the at least one original feature map by means of the synthetic neural network in order to produce at the output the decoded signal, said processing comprising at least one step of producing at the output of one of the synthetic neural layers, called generator layer, at least one feature map, called intermediate map; applying the control neural network to the at least one intermediate feature map in order to obtain a set of control parameters; applying to the input or output of a certain layer of the synthetic neural layers, called correction layer, at least one function parameterized by at least a part of the set of control parameters.

Description

Background Technology

[0001] This invention relates to the general field of encoding digital images. More specifically, this invention relates to the compression of digital video.

[0002] Digital video is typically source-encoded to compress it, thus limiting the resources required for its transmission and / or storage. Many coding standards exist, such as those developed by the ITU / MPEG organization (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) and their extensions (MVC, SVC, 3D-HEVC, etc.). In these methods, images are typically encoded based on either pixel prediction using previously encoded and then decoded pixels present in the image being encoded (in this case, called "intra-frame prediction") or pixel prediction using previously encoded images (in this case, called "inter-frame prediction").

[0003] In addition to these conventional methods, artificial intelligence-based methods (especially neural methods) are also emerging.

[0004] Some methods utilize the concept of autoencoders (i.e., learning algorithms based on artificial neural networks), which enable the construction of new representations of a dataset. An autoencoder architecture consists of two parts: an encoder and a decoder. The encoder comprises a set of neural layers that process the data to construct new representations, called "encoded" representations or "latent representations." The decoder's neural layers then receive these representations and filter them to attempt to reconstruct the original data. The difference between the reconstructed data and the initial data allows for the measurement of any errors produced by the autoencoder. Training involves modifying the autoencoder's parameters to reduce the reconstruction error measured at individual samples in the dataset. While these autoencoder-based systems offer superior performance, they come at the cost of significantly increased memory footprint and complexity compared to conventional methods, such as those proposed in compression standards. These systems may have millions of parameters, and decoding a single pixel can require up to a million multiply-accumulate (MAC) operations. This makes such decoders far more complex than conventional decoders, potentially hindering the application of learning-based compression.

[0005] To improve neural network-based signal processing, so-called attention mechanisms have been proposed. Attention mechanisms allow processing (attention) to be focused on certain parts of the signal. For example, the paper "DeepAttentional Features for Prostate Segmentation in Ultrasound" by Y. Wang et al. (International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 523-530, 2018) proposes using attention networks to modify the latent representation of the input image. This mechanism is rarely used in the field of signal compression because its effectiveness is not particularly pronounced in this context.

[0006] Patent application WO 2023199172A proposes improvements to neural networks used in video encoders. According to one example, these neural networks form an autoencoder. According to other examples, they supplement or replace elements of a video encoder, such as loop filters. The improvement takes the form of improved parameters, such as the parameters of the aforementioned filter neural network, which are evaluated by an estimator (LST-SQ) during encoder training and can be transferred to the decoder as neural network parameter updates. However, this approach remains very expensive.

[0007] Therefore, a solution is needed that enables efficient correction of neural networks in neural decoders. Summary of the Invention

[0008] This invention relates to a method for encoding a signal containing multiple samples, the method comprising the following steps: - Generate at least one original feature map; - Generate the following: at least one synthetic neural network defined by a set of synthetic network parameters, the at least one synthetic neural network comprising at least two synthetic neural layers; and a control neural network defined by a set of control network parameters; - The at least one original feature map is processed by the synthetic neural network to produce a representation of the decoded signal at the output, the processing including at least one step of producing at least one feature map called a generator map at the output of a layer called a generator layer in the synthetic neural layer; - Apply the control neural network to the at least one generator graph to obtain a set of control parameters; - Apply at least one function parameterized by at least a portion of the set of parameters to the input or output of a layer in the synthetic neural layer called the correction layer; - Update the original feature map and / or at least a portion of the parameters of the synthesized and / or control neural network based on coding performance metrics; - Encode the original feature map and / or at least a portion of the parameters of the synthesized and / or control neural network based on encoding performance metrics.

[0009] The present invention also relates to a method for decoding a signal containing multiple samples, the method comprising the following steps: - Obtain at least one feature map, referred to as the original map, which represents the signal in the latent domain; - Decode the parameters of a first neural network, referred to as a synthetic neural network, which includes at least two synthetic neural layers; - Decode the parameters of the second neural network, which is called the control neural network; - The at least one original feature map is processed by the synthetic neural network to generate the decoded signal at the output, the processing including at least one step of generating at least one feature map called a generator map at the output of a layer called a generator layer in the synthetic neural layer; - Apply the control neural network to the at least one generator graph to obtain a set of control parameters; - Apply at least one function parameterized by at least a portion of the set of parameters to the input or output of a layer in the synthetic neural layer, referred to as the correction layer.

[0010] Within the meaning of this invention, the term "encoding" should be understood to mean the operation of representing a set of samples in a compact form, for example, transmitted via a digital bitstream. The term "decoding" should be understood to mean the operation of processing a digital bitstream to reconstruct the decoded samples.

[0011] The term "sample" in signal should be understood as referring to the value sampled from the signal. Sampling a signal produces a discrete sequence of values ​​called samples. In the case of image signals, samples are called pixels and can be, for example, color pixels traditionally represented by triplet values ​​(e.g., (R, G, B) or (Y, U, V)). Generally, samples are represented by vectors containing one or more components or data.

[0012] The term "signal comprising multiple samples" should be understood to refer to a signal with one dimension (audio, sound), two dimensions (image), or more than two dimensions (stereo images, multi-view images, images associated with depth maps, video, etc.). Depending on the dimension, a sample has one, two, or more coordinates in the signal. For image signals, the position of a sample is identified by its x and y coordinates.

[0013] The term "feature map" should be understood as an abstract representation of a signal, containing multiple variable scalar data, also known as eigenvalues, such as real numbers or integers. These maps are also commonly referred to as the "latent representation" or "representation in the latent domain" of the signal. These feature maps can have a two-dimensional representation in the form of a scalar value image, a one-dimensional representation in the form of a vector, or correspond to a set of scalar values.

[0014] The term "synthetic feature map" should be understood to mean any feature map manipulated by a synthetic network, including the input and output feature maps of the synthetic network, as well as intermediate feature maps manipulated by the layers of the synthetic network. One of the neural layers of the synthetic network generates one or more generator maps at the output, which / these generator maps are used to control the input of the neural network.

[0015] The term "neural network" should be understood to refer to neural networks such as convolutional neural networks and multilayer perceptrons. A neural network is defined, for example, by multiple layers of artificial neurons and a set of activation functions, weighting functions, and addition functions (e.g., a layer can compute y = f(Ax + b), where y and b are N-dimensional vectors, x is an M-dimensional vector, A is an M × N-dimensional matrix, and f is the activation function). Multiple such networks can be cascaded.

[0016] The term "parameters of a neural network" should be understood to mean one of the values ​​that characterize the neural network, such as the weights (convolution or filtering coefficients, weights, biases, values ​​that affect nonlinear operations, etc.) associated with one of the neurons.

[0017] The term "processing (input values) via a neural network" should be understood as applying a function represented by a neural network to the input values ​​in order to produce output values ​​that represent samples of the current signal to be decoded (reconstructed) (or encoded).

[0018] The term "performance metric" should be understood as a measure between at least one value of a sample to be encoded and the decoded value of said sample. For example, this metric can evaluate distortion or perceptual error. This metric can be applied to one or more samples (e.g., the current sample or the current image). The metric can also include a bitrate metric, particularly a bitrate metric associated with the encoding of a neural network and / or the encoding of feature maps. This metric can be a joint measure of bitrate and distortion achieved by weighting bitrate and distortion. As is known in the art, the value of this metric is typically minimized until a target value or minimum value, or a predefined time, is reached.

[0019] The term "control parameters" should be understood to refer to any information that can be used to correct one or more synthetic feature maps. Such a map is called a correction map. The layer to which correction is applied is called a correction layer.

[0020] The term "parameterized function" should be understood as referring to a function defined by a set of parameters. Any known mathematical function, or a combination or composition of such functions, can be considered. The function can be represented by an artificial neuron (also called a perceptron), which typically corresponds to a set of parameters: its weights, biases, and possible activation functions. Of course, multiple functions can be applied to an input or output layer. Therefore, a set of parameterized functions can represent one or more layers of an artificial neural network. This function or set of functions can be applied to the output or input of one or more layers of a synthetic network. For example, a complete neural network defined by its parameters, which are parameterized functions, can be applied to a synthetic layer. According to another example, a simple mathematical function can be applied to the values ​​of a synthetic layer.

[0021] Generally speaking, the steps of an encoding or decoding method should not be interpreted as being associated with any concept of temporal sequence. In other words, these steps can be performed in a different order than indicated in the independent encoding or decoding claims, and can even be performed in parallel.

[0022] The encoding method according to the invention constructs encoding parameters based on the input signal (e.g., an image sequence) during its training by training a feature map synthesis neural network and a feature map control neural network to obtain a faithful reconstruction of the signal. Advantageously, the encoding method encodes the neural network in the form of parameters. These parameters can be optimized and adapted in terms of complexity or quality according to the target complexity and target bit rate-distortion ratio. Similarly, the encoded and transmitted feature maps can thus be optimized and adapted.

[0023] During the training, construction, or learning process, the parameters of the neural network and the information to be encoded are updated based on performance metrics, such as rate-distortion. When training ends, that is, when the obtained performance metrics are satisfactory, actual encoding can be performed on the parameters of the reconstructed neural network and the parameters of the feature maps, and the encoded results can be stored or transmitted to the decoder. Advantageously, the training process thus allows for the refinement of the neural network parameters and their input parameters until a sufficiently representative representation in terms of performance is obtained, for example, until the desired balance between the generated bit rate and the distortion experienced by the input image being encoded is achieved. Advantageously, the encoding method according to the invention allows for efficient signal compression.

[0024] Advantageously, the decoding method is simple because obtaining the neural network and feature maps is sufficient to reconstruct a decoded version of the signal (e.g., the current image in a sequence of images being processed). Advantageously, in this context, a transmission system that processes images per-image with low latency can be designed to decode each current image once the parameters of the neural network and the associated feature maps are received.

[0025] Advantageously, the control parameters are generated by a control network, to which the latent feature maps obtained from the synthesis network are submitted as input. Therefore, layers preceding the generator layers in the network allow for the generation of signals more relevant to the control network, thereby improving the quality of the control parameters. Furthermore, encoding the control parameters is inexpensive, as the cost is limited to the parameters of the control neural network, which allows for the reconstruction of the control neural network itself, enabling the generation of the control parameters. This allows for a reduction in the amount of data required to transmit the signal.

[0026] Advantageously, control parameters can be parameterized to functions that can be applied in whole or in part to one or more synthesis layers. Therefore, these control parameters can be shared; that is, multiple parameters (and / or parameterized functions) can be used as inputs or outputs for multiple layers. This allows for both a reduction in the amount of data required to transmit signals and a decrease in complexity, since generating the set of parameters only once is sufficient.

[0027] Advantageously, control parameters can be applied to multiple outputs or inputs of a layer. Therefore, these control parameters can be shared, meaning that multiple parameters (and / or parameterization functions) can be used for multiple inputs or outputs of a layer, for example, for feature maps with multiple resolutions.

[0028] Control parameters allow for the management of low-complexity (e.g., affine) functions used to modify (latent) feature maps, thereby enabling the replacement of more complex processing operations that would otherwise have to be carried out by synthetic networks.

[0029] According to some embodiments of the encoding and / or decoding methods: The generator layer is equal to the correction layer. Advantageously, according to this embodiment, the proximity between the correction layer and the generator layer allows for the acquisition of locally optimal control parameters, thereby contributing to the overall performance of the system.

[0030] - The generator layer is located before the correction layer; "before" should be understood as meaning that the generator layer is located before the correction layer in the order of the synthetic layers organized from the first layer (the network's input) to the last layer (the network's output); advantageously, this avoids the complexity of "recursive" networks, which are networks in which the generator layer necessary for the input of the control network is located after the layer corrected by the control network.

[0031] - The generator layer is the first layer of the synthesis network.

[0032] Advantageously, this allows us to start with a more abstract representation that is far removed from the signal to be decoded and has a lower resolution, thus reducing computation and allowing the control network to have lower complexity.

[0033] - The generator layer is the last layer of the synthesis network.

[0034] Advantageously, the signal is then corrected as close as possible to the image domain, allowing the contour to be corrected with precise shape; since the last few layers of the synthesis network are the finest, this enables the control network to contribute as much as possible to the quality of the reconstructed signal.

[0035] The method includes the step of decoding the at least one original feature map. Advantageously, the feature map can be optimized at the encoder before encoding and then transmitted to the decoder. Any encoding / decoding tool can be used: standard encoders (HEVC, VVC, JPEG, etc.), encoding frequency maps obtained from wavelet transform, DCT, entropy encoders, etc., etc.

[0036] - At least one synthetic neural layer is applied to at least one feature map at the input and produces at least one feature map at the output, the input map and the output map being referred to as synthetic feature maps; advantageously, according to this embodiment, the synthetic network progressively produces a series of synthetic feature maps, which can be regarded as increasingly faithful approximations of the signal in the latent domain.

[0037] - The same parameterization function can be applied to at least two synthesized feature maps; advantageously, according to this embodiment, the same parameterization function (e.g., convolution, multiplication, masking, etc.) can be applied to multiple locations in the synthesis (at the input or output of different layers) or to multiple feature maps of the same synthesis layer, or both. Therefore, the function to be applied to control multiple synthesized feature maps can be shared.

[0038] - Use the same control parameters to parameterize at least two functions applied to at least two synthetic feature maps respectively; advantageously, the control parameters can therefore be shared so that multiple synthetic feature maps can be corrected using the same function or different functions.

[0039] - The set of control parameters includes at least one convolution kernel; advantageously, one or more convolution kernels are generated by the control module to perform convolution operations on the synthesized feature map.

[0040] - The set of control parameters includes at least one feature map referred to as a control feature map; advantageously, one or more latent value feature maps are generated by a control neural network. These feature maps can be used to parameterize simple functions (e.g., multiplication and / or addition) or more complex functions (e.g., filtering) that are to be directly applied to the values ​​of the synthesized feature maps.

[0041] - The resolution of the at least one control feature map is the same as the resolution of the at least one synthetic feature map; advantageously, according to this embodiment, the parameterization function allows simple operations (addition, multiplication, masking, convolution, etc.) to be performed between the values ​​of the control feature map and one or more values ​​located in the same neighborhood of the synthetic feature map. In this case, there is at least one available control parameter for each element of the synthetic feature map, and this fineness allows for maximum coding performance.

[0042] - The resolution of the at least one control feature map differs from the resolution of the at least one synthetic feature map; advantageously, according to this embodiment, the parameterization function allows simple operations (addition, multiplication, masking, convolution, etc.) to be performed between the values ​​of the control feature map and multiple values ​​of the synthetic feature map, which may, for example, be located in a neighborhood of the position where the values ​​of the control map are scaled. For example, the neighborhood may be a block.

[0043] - The resolution of the at least one control feature map is lower than the resolution of the at least one synthetic feature map; advantageously, according to this embodiment, the complexity of the step of generating the control feature map is reduced, and the same control parameter is shared among multiple elements of the synthetic feature map.

[0044] - The at least one function parameterized by the at least one control feature map is applied to at least two different synthetic graphs.

[0045] - The at least one function parameterized by the at least one control feature map is applied to at least two synthetic feature maps in the same layer; advantageously, according to this embodiment, a function parameterized by the same control feature map is applied to multiple synthetic feature maps at the same location in the synthetic network. For example, a control map (via the same function or multiple different functions) can be applied to apply processing to multiple synthetic feature maps with different resolutions at the same location. Thus, control parameters can be shared to correct multiple synthetic feature maps.

[0046] - The at least one (synthetic or) control neural network includes at least one convolutional layer; therefore, filtering-based processing can be performed on the image, thereby improving the consistency of the generated image by limiting noise and improving the contours present in the image.

[0047] - The at least one (synthetic or) control neural network includes at least one attention module; the term "attention module" should be understood to mean a layer of the neural network that includes attention elements. Attention elements are elements that allow the application of attention mechanisms to generate one or more masks as parameters. These masks can be parameterized to a function that uses these masks to weight the synthesized feature maps (by multiplication or using more complex functions) to adapt the bitrate distribution over these regions and / or features, for example, according to the importance of individual regions and / or features of the image. The attention masks are encoded and decoded into parameters controlling the network. Advantageously, no additional bits are required to encode the masks.

[0048] - Based on the data from the bitstream, at least a portion of the set of parameters for synthesizing and / or controlling the neural network is obtained; advantageously, according to this embodiment, the parameters of the neural network are transmitted in a stream. All or a portion of these parameters can be transmitted in quantized form and encoded in compact form using any quantizer and entropy encoder available to those skilled in the art. A format similar to the MPEG-7 NNR standard can be used.

[0049] - Decoding at least a portion of the set of parameters for synthesizing and / or controlling the neural network based on predetermined parameters; advantageously, according to this embodiment, a portion of the parameters of the neural network can be accessed by the decoder; for example, these parameters can be stored in a storage memory. In particular, a portion can be stored in quantized form and encoded in compact form using any quantizer and entropy encoder available to those skilled in the art. A format similar to that of the MPEG-7 NNR standard can be used.

[0050] - Decoding at least a portion of the set of parameters of the synthesized and / or controlled neural network based on reference parameters; advantageously, according to this embodiment, certain parameters of the neural network are decoded by taking into account the reference parameters, which can be stored in a memory (storage memory, network, etc.) accessible by the decoder. Therefore, storage space or transmission rate on the network can be effectively reduced: some parameters can be encoded / decoded in a complementary manner (e.g., by updating kernels, layers, biases, etc.), while other parameters do not require encoding / decoding at all, as they are already provided in the reference parameters. The reference parameters can constitute a reference network. According to some variant embodiments that can be combined with each other: Reference parameters can form a set of variants of a neural network: convolution type, attention module, etc. Therefore, only these parameters or their identifiers need to be encoded, inserted into the stream, and decoded by the decoder; Some parameters are encoded in a way that is complementary to the parameters of the reference network. Therefore, the decoder only needs to decode these residual parameters and then add or multiply them with the residual parameters of the reference neural network (or combine them in any other known way); A portion of the parameters of the reference network is reused in the target neural network. For example, an entire layer of the synthetic network can be copied from the reference network, and therefore the parameters of that layer are neither encoded, transmitted, nor decoded. A reference network indicator is transmitted in the stream. Therefore, a network to be used for decoding can be indicated from among several possible networks known to the decoder.

[0051] - The at least one synthetic and / or control neural network is selected from a plurality of neural networks, and the bit stream contains an indicator indicating the selection of the at least one of the plurality of networks.

[0052] Relatedly, the present invention also relates to an encoding device and a decoding device.

[0053] The features and advantages of the encoding or decoding methods also apply to the encoding or decoding devices according to the present invention, and vice versa.

[0054] The present invention also relates to a computer program on a recording medium, which can be implemented in a computer or an encoding or decoding apparatus according to the invention. The program includes instructions suitable for implementing the corresponding methods. The program can use any programming language and can be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form or in any other desirable form.

[0055] The present invention also relates to a computer-readable information medium or recording medium comprising instructions of the aforementioned computer program. The information or recording medium can be any entity or device capable of storing a program. For example, the medium may include a storage device, such as a ROM (e.g., a CD-ROM or a microelectronic circuit ROM), or a magnetic recording device (e.g., a floppy disk, a hard disk, a DNA sequence, or a flash memory). Furthermore, the information or recording medium may be a transmissible medium, such as an electrical signal or an optical signal, which can be routed via cables or optical fibers, via radio links, via wireless optical links, or via other means.

[0056] The program according to the invention can be downloaded specifically via the Internet.

[0057] Alternatively, each information or recording medium may be an integrated circuit incorporating the program, the circuit being designed to execute or be used to execute the method according to the invention. Attached Figure Description

[0058] Other features and advantages of the invention will become apparent from the following description with reference to the accompanying drawings, which illustrate exemplary embodiments of the invention in a manner that is by no means limiting.

[0059] [ Figure 1 ] Figure 1 A decoding device used in the context of this invention is illustrated schematically; [ Figure 2 ] Figure 2 An encoding device used in the context of this invention is illustrated schematically; [ Figure 3 ] Figure 3 An architecture of a control and synthesis network used in encoding and decoding according to an embodiment of the present invention is shown; [ Figure 4 ] Figure 4 An architecture of a control and synthesis network used in encoding and decoding according to another embodiment of the present invention is shown; [ Figure 5 ] Figure 5 An example of a synthetic network used in encoding and decoding according to an embodiment of the present invention is shown; [ Figure 6 ] Figure 6 An example of correction applied to the synthesized feature map by a control network is shown; [ Figure 7 ] Figure 7 This demonstrates another example of correction applied to the synthesized feature map by a control network; [ Figure 8 ] Figure 8 It shows that it can be generated by Figure 1A flowchart of an example of a decoding method implemented by a decoding device. Detailed Implementation

[0060] Figure 1 A decoding device used in the context of this invention is illustrated schematically.

[0061] Figure 1 The decoding device DEC in the bitstream decodes the encoded information items from the bitstream BS to generate a signal Sd, which may be, for example, an image in a digital image sequence.

[0062] The signal includes multiple samples.

[0063] The decoder DEC receives the data needed to decode the signal Sd at its input: - Encoded data FMC corresponding to one or more encoded feature maps (e.g., multiple multi-resolution feature maps).

[0064] - At least one control neural network and coded parameters WcCon and WcSyn for the synthetic neural network. Other neural networks can be used, such as: - A neural network used for predictive decoding of latent feature maps; - A neural network used for upsampling and / or processing of latent feature maps; - A neural network used for post-processing the synthesized signal; - etc.

[0065] Encoded information is extracted from a stream BS, which can be received via a communication network, obtained from memory accessible to the decoder, and so on.

[0066] The decoding module DEC includes the feature map decoding module FMD and the neural network decoding module NND.

[0067] The decoding module FMD decodes the encoded feature map FMC. According to one embodiment, the decoding module is a conventional decoder, such as an entropy decoder of JPEG or MPEG type, that generates at least one set of decoded feature maps (denoted as FMOR) at the output. According to one embodiment, for this purpose, the module FMD may include a feature map synthesis neural network. According to one embodiment, the number of maps decoded by the module FMD, NFC in number, is represented as FMOR and indexed by i.

[0068] The neural network decoding module NND decodes the encoded parameters of the control neural network and the synthetic neural network. The decoded parameters are represented as WdCon and WdSyn, respectively.

[0069] The module NNSYN, which includes a synthetic neural network, uses the original image FMOR at the input and produces a decoded signal Sd (e.g., in the form of Y, U, V components) at the output. The module NNSYN will be described in detail in the remainder of this specification. The module includes one or more levels of cascaded layers Si. The module may include an MLP. The module may also include convolutional layers comprising convolutional elements, which may or may not include residual structures and attention modules. The module may also include upsampling layers, downsampling layers, etc. At least one of the synthetic feature maps (the latent input map, output map, or intermediate map of the synthesis process) is corrected at the input or output of the layer by a function parameterized by control parameters (PAR).

[0070] The module NNCON, which controls a neural network, uses the outputs of layers from a synthetic network (e.g., so-called generator feature maps, denoted as FMGEN) at the input and produces a set of parameters (denoted as PAR) at the output. NNCON can be in the form of a neural network and includes weighted and non-linear elements. The module may include an MLP. The module may also include convolutional layers comprising convolutional elements, which may or may not include residual structures, and may or may not include an attention module.

[0071] The decoding device DEC can be implemented by an electronic device containing a processor and memory (not shown); then, each of the aforementioned modules can be generated via the interaction of the processor with computer program instructions stored in the aforementioned memory and designed to perform the function of the module in question when these instructions are executed by the processor, in particular as described below.

[0072] Figure 2 An encoding device used in the context of this invention is illustrated schematically.

[0073] Figure 2 The encoding device ENC in the video encodes the signal S, which may be, for example, one or more images in a video sequence.

[0074] The encoder ENC receives a signal S at its input and generates encoded parameters at its output. These parameters include encoded data FMC corresponding to one or more encoded feature maps (e.g., multiple multi-resolution maps), and encoded parameters WcCon and WcSyn (referred to as control network parameters and synthetic network parameters) for at least one control neural network and one synthetic neural network, respectively. Figure 1 As mentioned in the description, other neural networks can be used.

[0075] For the current image, the encoding module ENC includes: the module NNC for encoding the parameters of one or more neural networks, the module FMC for encoding the feature map, the evaluation module RD-OPT, and the initialization and update module INIT / MAJ.

[0076] Furthermore, following conventional methods, the encoding module includes a decoder similar to the decoder DEC described above, in order to reconstruct the encoded and then decoded signal S'. This decoder is denoted as DEC'.

[0077] During the encoding training or construction process—that is, whenever the performance evaluation step is still unsatisfactory—the encoding module simulates encoding, followed by decoding, and the result is sent to the evaluation module RD-OPT. Subsequently, the encoding module performs the actual encoding of the data. It is well known that encoding simulation can be identical or approximate to actual encoding.

[0078] The INIT / MAJ module is responsible for initializing and updating the values ​​of the image / feature map and the parameters of the neural network. This module updates the values ​​of the signal to be encoded based on the results of a performance function. Once the values ​​of the feature map and the neural network have stabilized, the graph and network can be encoded. The INIT / MAJ module can perform updates on subsets of the image / feature map and subsets of the neural network parameters. Where applicable, elements that are not updated can retain values ​​known to the decoder.

[0079] The RD-OPT module evaluates and minimizes coding performance. For example, the evaluation function is of the rate-distortion type. The distortion between the original signal S and the analog-generated signal S' produced by encoding and then decoding can be evaluated, and the bitrate may include the sum of bitrates related to the contributions of WcCon, WcSyn, and FMC, or only a sub-part. Minimization can be performed using gradient descent and updating the parameters of one or more neural networks, or using any other optimization method within the capabilities of those skilled in the art.

[0080] The encoding device ENC can be implemented by an electronic device containing a processor and memory (not shown); then, each of the aforementioned modules can be generated via the interaction of the processor with computer program instructions stored in the aforementioned memory and designed to perform the function of the module in question when these instructions are executed by the processor, in particular as described below.

[0081] Figure 3 An architecture of a control and synthetic neural network used in encoding and decoding according to an embodiment of the present invention is shown.

[0082] according to Figure 3For example, the synthesized network comprises a set of synthesized layers or sub-networks, denoted as S1, S2, ..., SN. This synthesized network is generated based on the decoded parameters WdSyn. According to one embodiment, a portion of the parameters WdSyn is obtained based on reference parameters known to the decoder. The first synthesized layer (S1) acquires a decoded feature map, known as the original feature map (FMOR), at the input.

[0083] Each synthesis layer can perform simple concatenation of the input, or more complex combinations, including, for example, convolution or combination of the concatenated input by a neural network.

[0084] Modules Si (S1, S2, ..., SN) can contain, without limitation, multilayer perceptrons (MLP), convolutional elements, and nonlinear elements (ReLU, etc.).

[0085] Convolutional elements may or may not be residual and can include kernels of any size. As is known to those skilled in the art, a bias (which may be zero) follows the convolutional element. Any activation function within the capabilities of those skilled in the art (ReLU, sigmoid, Softmax, Softplus, Softsign, etc.) can be used. When convolutional elements operate on 2D signals (such as latent value feature maps), the convolutional elements can consist of 2D convolutions, meaning that convolutional kernels are jointly applied to rows and columns of an image or latent value map. Convolutional elements can also consist of 1D convolutions in a separable form. In this case, instead of performing convolution on the image or 2D latent value map in a known manner, a first 1D convolution operator is applied to the rows, and then a second convolution is applied to the columns of the representation obtained after the first operator. Row / column processing can be reversed to perform column processing first, followed by row processing.

[0086] One of the synthesis layers (the first layer S1 in this example) generates at least one feature layer (denoted as FMGEN and referred to as the generator layer) at the output, which is applied to the input of the control network NNCON.

[0087] according to Figure 3 For example, the control network NNCON comprises a set of layers or subnetworks, denoted as C1, C2, ..., CN. This control network is generated based on the decoded parameters WdCon. According to one embodiment, a portion of the parameters WdCon is obtained based on reference parameters known to the decoder. The control network employs at least one generator feature layer, denoted as FMGEN, generated by the synthesis network at its input. The control network produces a set of parameters, denoted as PAR, at its output.

[0088] Each layer Ci can perform simple concatenation of the input or more complex combinations, including, for example, convolution or combination of the concatenated input by a neural network. Modules Ci (C1, C2, ..., CN) can contain, without limitation, multilayer perceptrons (MLPs), convolutional elements, and nonlinear elements (ReLU, etc.).

[0089] The parameter PAR is the control parameter of the control function. These parameters can take the form of one-dimensional, two-dimensional, three-dimensional, or more-dimensional data, such as one or more control feature maps in the latent domain. According to one example, the control feature map may include the gain to be applied by the function to the synthesized feature map. According to another example, the control feature map may include the bias to be applied to the synthesized feature map. According to yet another example, the control feature map may include convolution kernels, nonlinear parameters, etc.

[0090] The control function Fi(F1, F2...F) is parameterized from the control parameters. N These functions are used to correct the synthesized feature maps at the inputs or outputs of various layers of the synthetic network. These functions are parameterized by the parameter PAR. Depending on the nature of the data being manipulated, the function Fi can perform any operation within the capabilities of someone skilled in the art: multiplication, thresholding, addition, nonlinearity, linear combination, etc. Any known mathematical function or combination or composition of such functions can be used. For example, the function may include: -f(x) = a. FM(x)+b, where a and b are the parameters of the function, and FM(x) represents the value of the corrected feature map at coordinate x, for example, if the map is represented as a one-dimensional vector; -f(x) = a. FM(x) 2 Polynomial functions of type +b.FM(x)+c, such as second-order polynomials, where a, b, and c are the parameters of the function; - Any known mathematical function or combination or composition of such functions, such as f(x) = a.cos(b.FM(x)+c)+d.

[0091] This function can, of course, be a multivariable function, for example: -f(x,y) = a.FM(x,y) + b.FM(x,y) + c.sqrt(xy) is a function of the type a.FM(x,y) + b.FM(x,y) + c.sqrt(xy), where sqrt() is the square root, and x and y are the coordinates of the value of the feature map FM; The filtering function of type -f(x,y) = f(x,y) = a.FM(x-1,y) + b.FM(x,y) + c.FM(x+1,y) is used to perform basic filtering on the feature map FM at integer coordinates indexed by x,y.

[0092] This function is applied to the values ​​of at least one synthesized feature map in a synthesized layer that is at the same level as the generator layer or that is located after the generator layer (in the output direction). According to the example shown, the function is applied to the output of the last layer (SN, function FN) and the output of the first layer S1 (dashed arrow, function F1). For example, the parameters (a, b, c, etc.) are specific to each value of the graph. According to another example, the same parameters are applied to multiple values ​​of the graph.

[0093] According to some embodiments: All functions Fi are distinct and use their own parameters; At least two functions Fi are identical; At least two functions Fi (whether different or not) share a set of parameters from this set of control parameters.

[0094] Figure 4 An architecture of a control and synthetic neural network used in encoding and decoding according to another embodiment of the present invention is shown.

[0095] Reference Figure 3 In contrast to the proposed embodiments, the embodiments shown herein employ the last layer (SN) of the synthetic neural network as the generator layer. Layer SN generates at least one feature layer (denoted as FMGEN) at its output, which is applied to the input of the control network NNCON. The parameters PAR generated by the network NNCON are applied only to the output of this last layer via a parameterization function FN used to correct the final reconstructed signal.

[0096] Figure 5 An example of a synthetic neural network for encoding and decoding according to an embodiment of the present invention is shown.

[0097] Figure 5 The diagram shows a single layer Sk of the synthetic network. This layer takes a feature map, represented as FMk_IN, at the input (a subset of the synthetic feature map FMSYN) and produces a feature map FMk_OUT at the output. For example, layer Sk can be a convolutional layer, an upsampling layer, etc.

[0098] The second layer (denoted as Fk, whose input corresponds to the output FMk_OUT of Sk) is the control layer, which corresponds to the application of the function Fk to correct the synthesis network. The function Fk is parameterized by the parameters PAR (which here correspond to a set of control feature maps (in the latent domain)) generated by the control network. The function Fk takes at least one control feature map as a parameter and performs an operation (function) on a set or subset of the synthesis feature maps FMk_OUT of layer Sk to generate a set of synthesis feature maps FMk+1_IN at the output that can be applied to the input of layer Sk+1.

[0099] Therefore, the synthetic network progressively generates synthetic feature maps, which represent the signal in the latent domain with progressively increasing granularity.

[0100] Figure 5 The examples presented here show control charts and composite charts with the same resolution.

[0101] Of course, the synthesized graphs can have different resolutions. For example, the number of values ​​included in the graph FMk_OUT can be greater than the number of corresponding values ​​in FMk_IN. In this case, Sk performs upsampling, including, for example, regular, bicubic, or bilinear interpolation functions or any other filtering function; according to another example, neural networks can be used to perform upsampling, which include sampling, pixel doubling (by pixel shuffling), or convolutions with bias, stride (the offset of the deconvolution kernel on each iteration), dilation (the spacing of the input values ​​supplied to the kernel), etc. Furthermore, synthesized graphs in the same layer can have different resolutions from each other (e.g., they can form a multi-resolution graph pyramid).

[0102] Similarly, as will Figure 8 As shown, control charts can have different resolutions than composite charts. Furthermore, control charts at the same level can have different resolutions from each other (e.g., they can form a multi-resolution chart pyramid).

[0103] The number of composite plots and control plots can be the same (one control plot can be used to correct one composite plot) or different (one control plot can be used to correct multiple composite plots, or multiple control plots can be used to correct one composite plot).

[0104] Figure 6 An example of the correction applied to the feature map of the synthetic network by the control network is shown.

[0105] According to the presented embodiment, the number of control feature maps and synthetic feature maps (PAR and FMk) is 4, and they all have the same resolution. The control network generates as many control feature maps (PAR1...PAR4) as the synthetic feature maps FMk (denoted as FMk1...FMk4) at the output or input of one of the synthetic layers Sk. Each control feature map numbered i has the same resolution as the corresponding synthetic map and is used as a parameter of the function Fki. The control feature maps are used to correct the synthetic map in order to provide a correction map FMkC (denoted as FMkC1...FMkC4). Of course, the number of each type of map can be different, and these maps can have different resolutions. The function Fk, thus parameterized, can perform any type of the same or different operations, such as, but not limited to: - Apply mathematical functions parameterized by the values ​​of the control chart, or combinations or composites of such mathematical functions; Examples of functions have been presented above; for instance, a simple function multiplies each value of the composite feature map by the corresponding control map value; another simple function adds the control map value to each value of the composite feature map; according to another example, each function masks the composite feature map to which the function is applied using the corresponding control map. For example, the control feature map includes values ​​corresponding to the mask values ​​(e.g., attention masks with integer values ​​between 0 and 255, or real values ​​between 0 and 1; or masks containing only values ​​0 or 1, etc.). The composite feature map FMk numbered i... i The value v of the coordinates (x, y) is multiplied by the mask value at the corresponding coordinates x and y of the control feature map in order to provide a correction value.

[0106] - The values ​​of the synthesized map are convolved with a convolution kernel obtained from the control chart; according to a very simple example, the values ​​of the synthesized map are filtered using one or more convolution kernels defined by parameters or the control chart. For example, a single kernel of size 3 × 3 is used. The 3 × 3 kernel is applied to various locations of the synthesized map by convolution to obtain a corrected output map. Any supplementary operations within the capabilities of those skilled in the art can be added to the convolution: bias, stride (the offset of the deconvolution kernel on each iteration), dilation (the spacing of the input values ​​supplied to the kernel), etc. Similarly, any activation function within the capabilities of those skilled in the art (ReLU, sigmoid, Softmax, Softplus, Softsign, etc.) can be used—and so on.

[0107] More generally, the feature map FMk numbered i i The value v is obtained by applying the function Fk i (x,y) and thus generate the graph FMkC i The value v'. For example, if the function is affine, then the feature map FMk i via Fk i (x,y) = a i (x,y) FMk i (x,y) + b i A function of type (x,y) is used for correction, where a i and b i The control feature map PAR at coordinates x and y with index i i The associated parameters. Similar reasoning applies to the functions described above.

[0108] Based on the same formal system, the correction function can be expressed in the following form: v' = Fk i (x,y) = a i FMk i (x-1,y) + b i FMk i (x,y) + c i FMk i (x+1,y), the purpose is to use the parameters common to the entire feature map (a i b i c i (In this case, the control parameters consist of the triplet) or variables on x and y (in this case, the control parameters consist of parameters a and y respectively). i b i and c i The basic horizontal filtering is performed by using three graphs (or a single graph consisting of triplet values ​​at each coordinate x, y).

[0109] Figure 7 This demonstrates another example of the correction applied to the feature map of the synthetic network by the control network.

[0110] According to the presented embodiment, the number of control feature maps and synthetic feature maps (PAR and FMk) is 4 each. (See above reference.) Figure 5 As described, the control network generates as many control feature maps PAR (denoted as PAR1...PAR4) as the synthesis feature maps FMk (denoted as FMk1...FMk4) at the output or input of one of the synthesis layers Sk. Each control feature map numbered i is used as a parameter of the function Fki. The control feature maps are used to correct the synthesis map of the correction synthesis layer in order to provide a correction map FMkC (denoted as FMkC1...FMkC4). The function Fk, thus parameterized, can perform any type of the same or different operation, as referenced above. Figure 6 As mentioned.

[0111] On the other hand, with reference to Figure 6 Conversely, the resolution of the control feature map is lower than that of the synthesized feature map. For example, the resolution is reduced to one-quarter of the original in each dimension.

[0112] Feature map FMk numbered i i The value v of the coordinates (x, y) is obtained by applying the function Fk. i And generate graph FMkC i The value v'. For example, if the function is affine, then the feature map FMk i The coordinates (x, y) and the value v will be applied by v' = Fk i (v) =a i (x / 4, y / 4) FMki (x,y) + b i A function of type (x / 4, y / 4), where a i and b i The control feature map PAR at coordinates x / 4 and y / 4 (integer division) with index i. i Related parameters.

[0113] In other words, this is equivalent to dividing the feature map to be corrected into blocks of a predetermined size (16 values ​​in this case), and equivalent to using a function parameterized by a single set of control parameters for each block.

[0114] Figure 8 It shows that it can be generated by Figure 1 A flowchart of an example of a decoding method implemented by a decoding device.

[0115] In step E30, an encoded data stream BS is obtained. This encoded data stream can be received from a communication network or read from a storage medium. Furthermore, certain information (e.g., information related to a reference control or synthesis network) can be read from a memory accessible by the decoder. The obtained data consists of encoded data FMC corresponding to one or more encoded feature maps (e.g., multiple multi-resolution maps), encoded parameters WcCon of at least one control neural network, and WcSyn of at least one synthesis neural network.

[0116] In step E31, the FMD module decodes the NFC raw feature maps FMOR. According to one embodiment, these raw feature maps are decoded using conventional techniques within the capabilities of those skilled in the art, such as prediction followed by decoding using a context entropy encoder, and then dequantization. According to another embodiment, these raw feature maps are obtained from a standard decoder (JPEG, MPEG, etc.). According to yet another embodiment, these raw feature maps are decoded (synthesized) by a decoding neural network.

[0117] In step E32, the NNC module decodes the control parameters WcCon of the control neural network and the synthesis parameters WcSyn of the synthesis neural network. For this purpose, any known neural network decoding technique corresponding to the encoding technique used by the encoder can be used, such as the neural network coding standard NNR proposed in Part 17 of the MPEG-7 standard.

[0118] According to one embodiment, the parameters of the neural network are received by the decoder in the bitstream. According to one embodiment, some parameters of the neural network are obtained from memory accessible by the decoder. According to one embodiment, some reference parameters of the neural network are obtained from memory accessible by the decoder. According to one embodiment, an indicator received in the stream enables the selection of one or more synthesis and / or correction networks from a plurality of networks accessible by the decoder.

[0119] In step E33, the layers of the decoding neural network NNSYN are applied to the synthesized feature map (corrected by a parameterized control function, or uncorrected) to generate an output feature map according to one embodiment. The first synthesized layer is applied directly to the original map FMOR; that is, the first iteration of step E33 involves applying the function corresponding to layer S1 to the original feature map FMOR. This step is repeated as long as the generator layer is not reached. When the generator layer is reached, in step E34, the output of that layer (corresponding to the generated feature map) is applied to the input of the control neural network NNCON.

[0120] In step E34, the decoding control neural network NNCON generates control parameters PAR. According to one embodiment, the parameters PAR take the form of a set of control feature maps.

[0121] In step E35, at least a portion of the parameter PAR is applied to the input or output of the synthesis layer Sk of the synthesis network. According to one embodiment, the parameter PAR takes the form of a set of control feature maps (PARs), each of which is used to parameterize a function Fk applied to the input or output of the layer, for example, to one of the input or output synthesis feature maps FMSYN.

[0122] After step E35, which applies the parameterized function, step E33 is iterated once or multiple times while the synthesized network still has layers to be applied. Otherwise, the method stops. The method stops when the last layer SN of the synthesized network has been applied.

[0123] Therefore, the decoded signal is generated based on the feature maps representing the signal. The number of iterations is limited by the network structure.

[0124] The decoded signal Sd corresponds to the output of the last synthesized layer, whether corrected or uncorrected by the parameterized function.

Claims

1. A method for encoding a signal (S) containing multiple samples, the method comprising the following steps: - Generate at least one original feature map (FMOR); - Generate the following: at least one synthetic neural network (NNSYN) defined by a set of synthetic network parameters (WoSyn), the at least one synthetic neural network comprising at least two synthetic neural layers (S1, SN); and a control neural network (NNCON) defined by a set of control network parameters (WoCon); - The at least one original feature map is processed by the synthetic neural network to generate a representation of the decoded signal (S') at the output, the processing including at least one step of generating at least one feature map called a generator map at the output of a layer called a generator layer in the synthetic neural layer (S1, SN); - Apply the control neural network to the at least one generator graph to obtain a set of control parameters (PAR); - Apply at least one function (F1...FN) parameterized by at least a portion of the set of control parameters (PAR) to the input or output of a layer called the correction layer in the synthetic neural layer (S1, SN); - Update at least one original feature map and / or at least a portion of the parameters of the synthesized and / or control neural network based on coding performance metrics; - Encode at least one original feature map and / or at least a portion of the parameters of the synthesized and / or control neural network based on encoding performance metrics.

2. A method for decoding a signal (Sd) containing multiple samples, the method comprising the following steps: - Obtain at least one feature map (FMOR) called the original map (E31), which represents the signal (Sd) in the latent domain. - Decode (E32) the parameters (WcSyn) of a first neural network (NNSYN) called a synthetic neural network, which includes at least two synthetic neural layers (S1, SN). - Decode the parameters (WcCon) of the second neural network (NNCON), which is called the control neural network (E32); - The at least one original feature map is processed (E33) by the synthetic neural network to generate the decoded signal (Sd) at the output, the processing including at least one step of generating at least one feature map called the generator map at the output of a layer called the generator layer in the synthetic neural layer (S1, SN); - Apply the control neural network (E34) to the at least one generator graph to obtain a set of control parameters (PAR). - Apply (E35) at least one function (F1…FN) parameterized by at least a portion of the set of control parameters (PAR) to the input or output of a layer called the correction layer in the synthetic neural layer (S1, SN).

3. The decoding method as described in claim 2, characterized in that, The generator layer is equal to the correction layer.

4. The decoding method as described in claim 2, characterized in that, The generator layer is located before the correction layer in the synthetic neural network.

5. The decoding method as described in claim 2, characterized in that, The generator layer is the first layer of the synthetic neural network.

6. The decoding method as described in claim 2, characterized in that, The generator layer is the last layer of this synthetic neural network.

7. The decoding method as described in claim 2, comprising the step of decoding the at least one original feature map.

8. The decoding method as described in claim 2, wherein, The at least one synthetic neural layer (S1, ..., SN) is applied to at least one feature map (FMk_IN) at the input and generates at least one feature map (FMk_OUT) at the output. The input map and the output map are referred to as synthetic feature maps (FMSYN).

9. The decoding method as described in claim 8, characterized in that, Apply the same parameterization function to at least two synthetic feature maps (FMSYN).

10. The decoding method as described in claim 8, characterized in that, Use the same control parameters to parameterize at least two different functions applied to at least two synthetic feature maps respectively.

11. The decoding method according to any one of claims 2 to 10, wherein, The set of control parameters (PAR) includes at least one feature map referred to as the control feature map.

12. An apparatus for encoding a signal (S) containing multiple samples, the apparatus being configured to perform the following steps: - Generate at least one original feature map (FMOR) (INIT / MAJ); - Generate the following: at least one synthetic neural network (NNSYN) defined by a set of synthetic network parameters (WoSyn), the at least one synthetic neural network comprising at least two synthetic neural layers (S1, SN); and a control neural network (NNCON) defined by a set of control neural network parameters (WoCon); - The at least one original feature map is processed by the synthetic neural network to generate a representation of the decoded signal (S') at the output, the processing including at least one step of generating at least one feature map called a generator map at the output of a layer called a generator layer in the synthetic neural layer (S1, SN); - Apply the control neural network to the at least one generator graph to obtain a set of control parameters (PAR); - Apply at least one function (F1...FN) parameterized by at least a portion of the set of control parameters (PAR) to the input or output of a layer called a correction layer in the neural layer (S1, SN) of the synthetic neural network; - Update at least one original feature map and / or at least a portion of the parameters of the synthesized and / or control neural network based on encoding performance metrics (MAJ); - Encode at least one original feature map and / or at least a portion of the parameters of the synthesized and / or control neural network based on encoding performance metrics (FMC, NNC).

13. An apparatus for decoding a signal (Sd) containing multiple samples, said apparatus being configured to perform the following steps: - Obtain at least one feature map (FMOR) called the original map (FMD), which represents the signal (Sd) in the latent domain. - Decode (NND) the parameters (WcSyn) of a first neural network (NNSYN) called a synthetic neural network, which includes at least two synthetic neural layers (S1, SN). - Decode (NND) the parameters (WcCon) of the second neural network (NNCON), which is called the control neural network. - The at least one original feature map is processed (NNSYN) by the synthetic neural network to generate the decoded signal (Sd) at the output, the processing including at least one step of generating at least one feature map called the generator map at the output of a layer called the generator layer in the synthetic neural layer (S1, SN); - Apply the control neural network (NNCON) to the at least one generator graph to obtain a set of control parameters (PAR). - Apply at least one function (F1...FN) parameterized by at least a portion of the set of control parameters (PAR) to the input or output of a layer called the correction layer in the synthetic neural layer (S1, SN).

14. A computer program comprising instructions which, when executed by a computer, are used to perform the steps of the encoding method of claim 1 or the decoding method of claim 2.

Images