Method and apparatus for encoding and decoding an image using a correction mechanism

Abstract

The invention relates to a method for encoding and decoding a signal containing a plurality of samples. The decoding method comprises the following steps: - obtaining at least one feature map, called original map, representing the signal in a latent domain; - decoding parameters of a first neural network, called synthesis network, comprising at least one synthesis neural layer; - decoding parameters of a second neural network, called control network; - applying the control neural network to the at least one original feature map in order to obtain a set of control parameters; - processing the at least one original feature map by the synthesis neural network in order to produce a decoded signal at the output, the processing comprising at least one step of applying at least one function parameterized by at least one part of the set of parameters to at least one synthesis neural layer.

Description

Background Technology

[0001] This invention relates to the general field of encoding digital images. More specifically, this invention relates to the compression of digital video.

[0002] Digital video is typically source-encoded to compress it, thus limiting the resources required for its transmission and / or storage. Many coding standards exist, such as those developed by the ITU / MPEG organization (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) and their extensions (MVC, SVC, 3D-HEVC, etc.). In these methods, images are typically encoded based on either pixel prediction using previously encoded and then decoded pixels present in the image being encoded (in this case, called "intra-frame prediction") or pixel prediction using previously encoded images (in this case, called "inter-frame prediction").

[0003] In addition to these conventional methods, artificial intelligence-based methods (especially neural methods) are also emerging.

[0004] Some methods utilize the concept of autoencoders (i.e., learning algorithms based on artificial neural networks), which enable the construction of new representations of a dataset. An autoencoder architecture consists of two parts: an encoder and a decoder. The encoder comprises a set of neural layers that process the data to construct new representations, called "encoded" representations or "latent representations." The decoder's neural layers then receive these representations and filter them to attempt to reconstruct the original data. The difference between the reconstructed data and the initial data allows for the measurement of any errors produced by the autoencoder. Training involves modifying the autoencoder's parameters to reduce the reconstruction error measured at individual samples in the dataset. While these autoencoder-based systems offer superior performance, they come at the cost of significantly increased memory footprint and complexity compared to conventional methods, such as those proposed in compression standards. These systems may have millions of parameters, and decoding a single pixel can require up to a million multiply-accumulate (MAC) operations. This makes such decoders far more complex than conventional decoders, potentially hindering the application of learning-based compression.

[0005] To improve neural network-based signal processing, so-called attention mechanisms have been proposed. Attention mechanisms allow processing (attention) to be focused on certain parts of the signal. For example, the paper "DeepAttentional Features for Prostate Segmentation in Ultrasound" by Y. Wang et al. (International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 523-530, 2018) proposes using attention networks to modify the latent representation of the input image. This mechanism is rarely used in the field of signal compression because its effectiveness is not particularly pronounced in this context.

[0006] Patent application WO 2023199172A proposes improvements to neural networks used in video encoders. According to one example, these neural networks form an autoencoder. According to other examples, they supplement or replace elements of a video encoder, such as loop filters. The improvement takes the form of improved parameters, such as the parameters of the aforementioned filtering neural network, which are evaluated by an estimator (LST-SQ) during encoder training and can be transferred to the decoder as neural network parameter updates. However, this approach remains very expensive.

[0007] Therefore, a solution is needed that enables efficient correction of neural networks in neural decoders. Summary of the Invention

[0008] This invention relates to a method for encoding a signal containing multiple samples, the method comprising the following steps: – Generate a set of original feature maps; – Generate the following: at least one synthetic neural network defined by a set of synthetic network parameters, the at least one synthetic neural network comprising at least one neural layer; and a control neural network defined by a set of control network parameters; – Apply the control neural network to the at least one original feature map to obtain a set of control parameters; – Processing the at least one original feature map through the synthetic neural network to produce a representation of a decoded signal at the output, the processing includes at least one step of applying a function parameterized by at least a portion of the set of parameters to the input or output of the at least one neural layer of the synthetic neural network; – Update the original feature map and / or at least a portion of the parameters of the synthesized and / or control neural network based on coding performance metrics; – Encode the original feature map and / or at least a portion of the parameters of the synthesized and / or control neural network based on encoding performance metrics.

[0009] The present invention also relates to a method for decoding a signal containing multiple samples, the method comprising the following steps: – Obtain at least one feature map, referred to as the original map, which represents the signal in the latent domain; – Decode the parameters of a first neural network, referred to as a synthetic neural network, which includes at least one synthetic neural layer; – Decode the parameters of the second neural network, which is called the control neural network; – Apply the control neural network to the at least one original feature map to obtain a set of control parameters; – Processing the at least one original feature map through the synthetic neural network to generate a decoded signal at the output, the processing includes: applying at least one function parameterized by at least a portion of the set of control parameters to at least one step of the input or output of the at least one synthetic neural layer.

[0010] Within the meaning of this invention, the term "encoding" should be understood to mean the operation of representing a set of samples in a compact form, for example, transmitted via a digital bitstream. The term "decoding" should be understood to mean the operation of processing a digital bitstream to reconstruct the decoded samples.

[0011] The term "sample" in signal should be understood as referring to the value sampled from the signal. Sampling a signal produces a discrete sequence of values ​​called samples. In the case of image signals, samples are called pixels and can be, for example, color pixels traditionally represented by triplet values ​​(e.g., (R, G, B) or (Y, U, V)). Generally, samples are represented by vectors containing one or more components or data.

[0012] The term "signal comprising multiple samples" should be understood to refer to a signal with one dimension (audio, sound), two dimensions (image), or more than two dimensions (stereo images, multi-view images, images associated with depth maps, video, etc.). Depending on the dimension, a sample has one, two, or more coordinates in the signal. For image signals, the position of a sample is identified by its x and y coordinates.

[0013] The term "feature map" should be understood as an abstract representation of a signal, containing multiple variable scalar data, also known as eigenvalues, such as real numbers or integers. These maps are also commonly referred to as the "latent representation" or "representation in the latent domain" of the signal. These feature maps can have a two-dimensional representation in the form of a scalar value image, a one-dimensional representation in vector form, or correspond to a set of scalar values.

[0014] The term “synthetic feature map” should be understood to mean any feature map manipulated by the synthetic network, including the input and output feature maps of the synthetic network, as well as the intermediate feature maps manipulated by the layers of the synthetic network.

[0015] The term "neural network" should be understood to refer to neural networks such as convolutional neural networks and multilayer perceptrons. A neural network is defined, for example, by multiple layers of artificial neurons and a set of activation functions, weighting functions, and addition functions (e.g., a layer can compute y = f(Ax + b), where y and b are N-dimensional vectors, x is an M-dimensional vector, A is an M × N-dimensional matrix, and f is the activation function). Multiple such networks can be cascaded.

[0016] The term "parameters of a neural network" should be understood to mean one of the values ​​that characterize the neural network, such as the weights (convolution or filtering coefficients, weights, biases, values ​​that affect nonlinear operations, etc.) associated with one of the neurons.

[0017] The term "processing (input values) via a neural network" should be understood as applying a function represented by a neural network to the input values ​​in order to produce output values ​​that represent samples of the current signal to be decoded (reconstructed) (or encoded).

[0018] The term "performance metric" should be understood as a measure between at least one value of a sample to be encoded and the decoded value of said sample. For example, this metric can evaluate distortion or perceptual error. This metric can be applied to one or more samples (e.g., the current sample or the current image). The metric can also include a bitrate metric, particularly a bitrate metric associated with the encoding of a neural network and / or the encoding of feature maps. This metric can be a joint measure of bitrate and distortion achieved by weighting bitrate and distortion. As is known in the art, the value of this metric is typically minimized until a target value or minimum value, or a predefined time, is reached.

[0019] The term "control parameters" should be understood to mean any information that can be used to correct one or more synthetic feature maps.

[0020] The term "parameterized function" should be understood as referring to a function defined by a set of parameters. Any known mathematical function, or a combination or composition of such functions, can be considered. The function can be represented by an artificial neuron (also called a perceptron), which typically corresponds to a set of parameters: its weights, biases, and possible activation functions. Of course, multiple functions can be applied to an input or output layer. Therefore, a set of parameterized functions can represent one or more layers of an artificial neural network. This function or set of functions can be applied to the output or input of one or more layers of a synthetic network. For example, a complete neural network defined by its parameters, which are parameterized functions, can be applied to a synthetic layer. According to another example, a simple mathematical function can be applied to the values ​​of a synthetic layer.

[0021] Generally speaking, the steps of an encoding or decoding method should not be interpreted as being associated with any concept of temporal sequence. In other words, these steps can be performed in a different order than indicated in the independent encoding or decoding claims, and can even be performed in parallel.

[0022] The encoding method according to the invention constructs encoding parameters based on the input signal (e.g., an image sequence) during its training by training a feature map synthesis neural network and a feature map control neural network to obtain a faithful reconstruction of the signal. Advantageously, the encoding method encodes the neural network in the form of parameters. These parameters can be optimized and adapted in terms of complexity or quality according to the target complexity and target bit rate-distortion ratio. Similarly, the encoded and transmitted feature maps can thus be optimized and adapted.

[0023] During the training, construction, or learning process, the parameters of the neural network and the information to be encoded are updated based on performance metrics, such as rate-distortion. When training ends, that is, when the obtained performance metrics are satisfactory, actual encoding can be performed on the parameters of the reconstructed neural network and the parameters of the feature maps, and the encoded results can be stored or transmitted to the decoder. Advantageously, the training process thus allows for the refinement of the neural network parameters and their input parameters until a sufficiently representative representation in terms of performance is obtained, for example, until the desired balance between the generated bit rate and the distortion experienced by the input image being encoded is achieved. Advantageously, the encoding method according to the invention allows for efficient signal compression.

[0024] Advantageously, the decoding method is simple because obtaining the neural network and feature maps is sufficient to reconstruct a decoded version of the signal (e.g., the current image in a sequence of images being processed). Advantageously, in this context, a transmission system that processes images per-image with low latency can be designed to decode each current image once the parameters of the neural network and the associated feature maps are received.

[0025] Advantageously, the control parameters are generated by a control network, and latent feature maps representing the signal are submitted as input to this control network. Therefore, a single set of feature maps is used as input to both (control and synthesis) networks. Furthermore, encoding the control parameters is inexpensive, as the cost is limited to the parameters of the control neural network, which allows for the reconstruction of the control neural network itself, enabling the generation of the control parameters. This allows for a reduction in the amount of data required to transmit the signal.

[0026] Advantageously, the control parameters are parameterized to functions that can be applied wholly or partially to one or more synthetic feature maps. Therefore, these control parameters can be shared; that is, multiple parameters (and / or parameterized functions) can be used as inputs or outputs for multiple layers. This allows for both a reduction in the amount of data required to transmit signals and a decrease in complexity, since generating the set of parameters only once is sufficient.

[0027] Advantageously, control parameters can be applied to multiple outputs or inputs of a layer. Therefore, these control parameters can be shared, meaning that multiple parameters (and / or parameterization functions) can be used for multiple inputs or outputs of a layer, for example, for feature maps with multiple resolutions.

[0028] Control parameters allow for the management of low-complexity (e.g., affine) functions used to modify or correct (latent) feature maps, thereby enabling the replacement of more complex processing operations that would otherwise have to be carried out by synthetic networks.

[0029] According to some embodiments of the encoding and / or decoding methods: The method includes the step of decoding the at least one original feature map. Advantageously, the feature map can be optimized at the encoder before encoding and then transmitted to the decoder. Any encoding / decoding tool can be used: standard encoders (HEVC, VVC, JPEG, etc.), encoding frequency maps obtained from wavelet transform, DCT, entropy encoders, etc., etc.

[0030] – At least one synthetic neural layer is applied to at least one feature map at the input and produces at least one feature map at the output, the input map and the output map being referred to as synthetic feature maps; advantageously, according to this embodiment, the synthetic network progressively produces a series of synthetic feature maps, which can be regarded as increasingly faithful approximations of the signal in the latent domain.

[0031] – The same parameterization function can be applied to at least two synthesized feature maps; advantageously, according to this embodiment, the same parameterization function (e.g., convolution, multiplication, masking, etc.) can be applied to multiple locations in the synthesis (at the input or output of different layers) or to multiple feature maps of the same synthesis layer, or both. Therefore, the function to be applied to control multiple synthesized feature maps can be shared.

[0032] – Use the same control parameters to parameterize at least two functions applied to at least two synthetic feature maps respectively; advantageously, the control parameters can therefore be shared so that multiple synthetic feature maps can be corrected using the same function or different functions.

[0033] – The set of control parameters includes at least one convolution kernel; advantageously, one or more convolution kernels are generated by the control module to perform convolution operations on the synthesized feature map.

[0034] – This set of control parameters includes at least one feature map referred to as a control feature map; advantageously, one or more latent value feature maps are generated by a control neural network. These feature maps can be used to parameterize simple functions (e.g., multiplication and / or addition) or more complex functions (e.g., filtering) that are to be directly applied to the values ​​of the synthesized feature maps.

[0035] – The resolution of the at least one control feature map is the same as the resolution of the at least one synthetic feature map; advantageously, according to this embodiment, the parameterization function allows simple operations (addition, multiplication, masking, convolution, etc.) to be performed between the values ​​of the control feature map and one or more values ​​located in the same neighborhood of the synthetic feature map. In this case, there is a control parameter for each element of the synthetic feature map, and this fineness allows for maximum coding performance.

[0036] – The resolution of the at least one control feature map differs from the resolution of the at least one synthetic feature map; advantageously, according to this embodiment, the parameterization function allows simple operations (addition, multiplication, masking, convolution, etc.) to be performed between the values ​​of the control feature map and multiple values ​​of the synthetic feature map, which may, for example, be located in a neighborhood of the position where the values ​​of the control map are scaled. For example, the neighborhood may be a block.

[0037] – The resolution of the at least one control feature map is lower than that of the at least one synthesized feature map; advantageously, according to this embodiment, the complexity of the steps for generating the control feature map is reduced, and the same control parameter is shared among multiple elements of the feature map.

[0038] – Apply the at least one function parameterized by the at least one control feature map to at least two different synthetic graphs.

[0039] – The at least one function parameterized by the at least one control feature map is applied to at least two synthetic feature maps in the same layer; advantageously, according to this embodiment, a function parameterized by the same control feature map is applied to multiple synthetic feature maps at the same location in the synthetic network. For example, a control map (via the same function or multiple different functions) can be applied to process multiple synthetic feature maps with different resolutions at the same location. Thus, control parameters can be shared to correct multiple synthetic feature maps.

[0040] – The at least one control neural network includes at least one convolutional layer; therefore, filtering-based processing can be performed on the image, thereby improving the consistency of the generated image by limiting noise and improving the contours present in the image.

[0041] – The at least one (synthetic or) control neural network includes at least one attention module; the term "attention module" should be understood to mean a layer of the neural network that includes attention elements. Attention elements are elements that allow the application of attention mechanisms to generate one or more masks as parameters. These masks can be parameterized to a function that uses these masks to weight the synthetic feature maps (by multiplication or using more complex functions) to adapt the bitrate distribution across those regions and / or features, for example, based on the importance of individual regions and / or features of the image. The attention masks are encoded and decoded into parameters controlling the network. Advantageously, no additional bits are required to encode the masks.

[0042] – Based on the data in the bitstream, at least a portion of the set of parameters for synthesizing and / or controlling the neural network is obtained; advantageously, according to this embodiment, the parameters of the neural network are transmitted in a stream. All or a portion of these parameters can be transmitted in quantized form and encoded in compact form using any quantizer and entropy encoder available to those skilled in the art. A format similar to the MPEG-7 NNR standard can be used.

[0043] – At least a portion of the set of parameters for synthesizing and / or controlling the neural network is decoded based on predetermined parameters; advantageously, according to this embodiment, a portion of the parameters of the neural network can be accessed by the decoder; for example, these parameters can be stored in a storage memory. In particular, a portion can be stored in quantized form and encoded in compact form using any quantizer and entropy encoder available to those skilled in the art. A format similar to that of the MPEG-7 NNR standard can be used.

[0044] – Decoding at least a portion of the set of parameters of the synthesized and / or controlled neural network based on reference parameters; advantageously, according to this embodiment, certain parameters of the neural network are decoded by taking into account the reference parameters, which can be stored in a memory (storage memory, network, etc.) accessible by the decoder. Therefore, storage space or transmission rate on the network can be effectively reduced: some parameters can be encoded / decoded in a complementary manner (e.g., by updating kernels, layers, biases, etc.), while other parameters do not require encoding / decoding at all, as they are already provided in the reference parameters. The reference parameters can constitute a reference network. According to some variant embodiments that can be combined with each other: Reference parameters can form a set of variants of a neural network: convolution type, attention module, etc. Therefore, only these parameters or their identifiers need to be encoded, inserted into the stream, and decoded by the decoder; Some parameters are encoded in a way that is complementary to the parameters of the reference network. Therefore, the decoder only needs to decode these residual parameters and then add or multiply them with the residual parameters of the reference neural network (or combine them in any other known way); A portion of the parameters of the reference network is reused in the target neural network. For example, an entire layer of the synthetic network can be copied from the reference network, and therefore the parameters of that layer are neither encoded, transmitted, nor decoded. A reference network indicator is transmitted in the stream. Therefore, a network to be used for decoding can be indicated from among several possible networks known to the decoder.

[0045] – The at least one synthetic and / or control neural network is selected from a plurality of neural networks, and the bitstream contains an indicator indicating the selection of the at least one of the plurality of networks.

[0046] Relatedly, the present invention also relates to an encoding device and a decoding device.

[0047] The features and advantages of the encoding or decoding methods also apply to the encoding or decoding devices according to the present invention, and vice versa.

[0048] The present invention also relates to a computer program on a recording medium, which can be implemented in a computer or an encoding or decoding apparatus according to the invention. The program includes instructions suitable for implementing the corresponding methods. The program can use any programming language and can be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form or in any other desirable form.

[0049] The present invention also relates to a computer-readable information medium or recording medium comprising instructions of the aforementioned computer program. The information or recording medium can be any entity or device capable of storing a program. For example, the medium may include a storage device, such as a ROM (e.g., a CD-ROM or a microelectronic circuit ROM), or a magnetic recording device (e.g., a floppy disk, a hard disk, a DNA sequence, or a flash memory). Furthermore, the information or recording medium may be a transmissible medium, such as an electrical signal or an optical signal, which can be routed via cables or optical fibers, via radio links, via wireless optical links, or via other means.

[0050] The program according to the invention can be downloaded specifically via the Internet.

[0051] Alternatively, each information or recording medium may be an integrated circuit incorporating the program, the circuit being designed to execute or be used to execute the method according to the invention. Attached Figure Description

[0052] Other features and advantages of the invention will become apparent from the following description with reference to the accompanying drawings, which illustrate exemplary embodiments of the invention in a manner that is by no means limiting.

[0053] [ Figure 1 ] Figure 1 A decoding device used in the context of this invention is illustrated schematically; [ Figure 2 ] Figure 2 An encoding device used in the context of this invention is illustrated schematically; [ Figure 3 ] Figure 3 An architecture of a control and synthesis network used in encoding and decoding according to an embodiment of the present invention is shown; [ Figure 4 ] Figure 4 An example of a synthetic network used in encoding and decoding according to an embodiment of the present invention is shown; [ Figure 5 ] Figure 5 An example of correction applied to the synthesized feature map by a control network is shown; [ Figure 6 ] Figure 6 This demonstrates another example of correction applied to the synthesized feature map by a control network; [ Figure 7 ] Figure 7 It shows that it can be made by Figure 1 A flowchart of an example of a decoding method implemented by a decoding device. Detailed Implementation

[0054] Figure 1 A decoding device used in the context of this invention is illustrated schematically.

[0055] Figure 1 The decoding device DEC in the bitstream decodes the encoded information items from the bitstream BS to generate a signal Sd, which may be, for example, an image in a digital image sequence.

[0056] The signal includes multiple samples.

[0057] The decoder DEC receives the data needed to decode the signal Sd at its input: – Encoded data FMC corresponding to one or more encoded feature maps (e.g., multiple multi-resolution feature maps).

[0058] – At least one control neural network and the synthesized neural network, each with encoded parameters WcCon and WcSyn. Other neural networks can be used, such as: - A neural network used for predictive decoding of latent feature maps; – A neural network used for upsampling and / or processing of latent feature maps; – A neural network used for post-processing of the synthesized signal; - etc.

[0059] Encoded information is extracted from a stream BS, which can be received via a communication network, obtained from memory accessible to the decoder, and so on.

[0060] The decoding module DEC includes the feature map decoding module FMD and the neural network decoding module NND.

[0061] The decoding module FMD decodes the encoded feature map FMC. According to one embodiment, the decoding module is a conventional decoder, such as an entropy decoder of JPEG or MPEG type, that generates at least one set of decoded feature maps (denoted as FMOR) at the output. According to one embodiment, for this purpose, the module FMD may include a feature map synthesis neural network. According to one embodiment, the number of maps decoded by the module FMD, NFC in number, is represented as FMOR and indexed by i.

[0062] The neural network decoding module NND decodes the encoded parameters of the control neural network and the synthetic neural network. The decoded parameters are represented as WdCon and WdSyn, respectively.

[0063] The module NNCON, which controls a neural network, uses the original graph FMOR at the input and produces a set of parameters represented as PAR at the output. NNCON can be in the form of a neural network and includes weighted elements and nonlinear elements. The module can include an MLP. The module can also include convolutional layers comprising convolutional elements, which may or may not include residual structures, and may or may not include an attention module.

[0064] The module NNSYN, which includes a synthetic neural network, also uses the original image FMOR at the input and produces a decoded signal Sd (e.g., in the form of Y, U, V components) at the output. The module NNSYN will be described in detail in the remainder of this specification. This module includes one or more levels of cascaded layers Si. This module may include an MLP. This module may also include convolutional layers comprising convolutional elements, which may or may not include residual structures and attention modules. This module may also include upsampling layers, downsampling layers, etc. At least one of the synthetic feature maps (the latent input map, output map, or intermediate map of the synthesis process) is corrected at the input or output of the layer by a function parameterized by the control parameter PAR.

[0065] The decoding device DEC can be implemented by an electronic device containing a processor and memory (not shown); then, each of the aforementioned modules can be generated via the interaction of the processor with computer program instructions stored in the aforementioned memory and designed to perform the function of the module in question when these instructions are executed by the processor, in particular as described below.

[0066] Figure 2 An encoding device used in the context of this invention is illustrated schematically.

[0067] Figure 2 The encoding device ENC in the video encodes the signal S, which may be, for example, one or more images in a video sequence.

[0068] The encoder ENC receives a signal S at its input and generates encoded parameters at its output. These parameters include encoded data FMC corresponding to one or more encoded feature maps (e.g., multiple multi-resolution maps), and encoded parameters WcCon and WcSyn (referred to as control network parameters and synthetic network parameters) for at least one control neural network and one synthetic neural network, respectively. Figure 1 As mentioned in the description, other neural networks can be used.

[0069] For the current image, the encoding module ENC includes: the module NNC for encoding the parameters of one or more neural networks, the module FMC for encoding the feature map, the evaluation module RD-OPT, and the initialization and update module INIT / MAJ.

[0070] Furthermore, following conventional methods, the encoding module includes a decoder similar to the decoder DEC described above, in order to reconstruct the encoded and then decoded signal S'. This decoder is denoted as DEC'.

[0071] During the encoding training or construction process—that is, whenever the performance evaluation step is still unsatisfactory—the encoding module simulates encoding, followed by decoding, and the result is sent to the evaluation module RD-OPT. Subsequently, the encoding module performs the actual encoding of the data. It is well known that encoding simulation can be identical or approximate to actual encoding.

[0072] The INIT / MAJ module is responsible for initializing and updating the values ​​of the image / feature map and the parameters of the neural network. This module updates the values ​​of the signal to be encoded based on the results of a performance function. Once the values ​​of the feature map and the neural network have stabilized, the graph and network can be encoded. The INIT / MAJ module can perform updates on subsets of the image / feature map and subsets of the neural network parameters. Where applicable, elements that are not updated can retain values ​​known to the decoder.

[0073] The RD-OPT module evaluates and minimizes coding performance. For example, the evaluation function is of the rate-distortion type. The distortion between the original signal S and the analog-generated signal S' produced by encoding and then decoding can be evaluated, and the bitrate may include the sum of bitrates related to the contributions of WcCon, WcSyn, and FMC, or only a sub-part. Minimization can be performed using gradient descent and updating the parameters of one or more neural networks, or using any other optimization method within the capabilities of those skilled in the art.

[0074] The encoding device ENC can be implemented by an electronic device containing a processor and memory (not shown); then, each of the aforementioned modules can be generated via the interaction of the processor with computer program instructions stored in the aforementioned memory and designed to perform the function of the module in question when these instructions are executed by the processor, in particular as described below.

[0075] Figure 3 An architecture of a control and synthetic neural network used in encoding and decoding according to an embodiment of the present invention is shown.

[0076] according to Figure 3For example, the synthesized network includes a set of synthesized layers or sub-networks, denoted as S1, S2, ..., SN. This synthesized network is generated based on the decoded parameters WdSyn. According to one embodiment, the first synthesized layer (S1) obtains a decoded feature map, referred to as the original feature map (FMOR), at the input. According to another embodiment, the first layer S1 obtains the result of a first parameterized function applied to the FMOR of the original feature map at the input. According to one embodiment, a portion of the parameters WdSyn is obtained based on reference parameters known to the decoder.

[0077] Each layer can perform simple concatenation of the input, or more complex combinations, including, for example, convolution or combination of the concatenated input by a neural network.

[0078] Modules Si (S1, S2, ..., SN) can contain, without limitation, multilayer perceptrons (MLP), convolutional elements, and nonlinear elements (ReLU, etc.).

[0079] Convolutional elements may or may not be residual and can include kernels of any size. As is known to those skilled in the art, a bias (which may be zero) follows the convolutional element. Any activation function within the capabilities of those skilled in the art (ReLU, sigmoid, Softmax, Softplus, Softsign, etc.) can be used. When convolutional elements operate on 2D signals (such as latent value feature maps), the convolutional elements can consist of 2D convolutions, meaning that convolutional kernels are jointly applied to rows and columns of an image or latent value map. Convolutional elements can also consist of 1D convolutions in a separable form. In this case, instead of performing convolution on the image or 2D latent value map in a known manner, a first 1D convolution operator is applied to the rows, and then a second convolution is applied to the columns of the representation obtained after the first operator. Row / column processing can be reversed to perform column processing first, followed by row processing.

[0080] according to Figure 3 For example, the control network NNCON comprises a set of layers or subnetworks, denoted as C1, C2, ..., CN. This control network is generated based on the decoded parameters WdCon. The control network acquires a decoded feature map at the input, known as the raw feature map (FMOR). The control network produces a set of parameters at the output, denoted as PAR. According to one embodiment, a portion of the parameters WdCon is obtained based on reference parameters known to the decoder.

[0081] Each layer Ci can perform simple concatenation of the input or more complex combinations, including, for example, convolution or combination of the concatenated input by a neural network. Modules Ci (C1, C2, ..., CN) can contain, without limitation, multilayer perceptrons (MLPs), convolutional elements, and nonlinear elements (ReLU, etc.).

[0082] The parameter PAR is the control parameter of the control function. These parameters can take the form of one-dimensional, two-dimensional, three-dimensional, or more-dimensional data, such as one or more control feature maps in the latent domain. According to one example, the control feature map may include a gain to be applied to the synthesized feature map. According to another example, the control feature map may include a bias to be applied to the synthesized feature map. According to yet another example, the control feature map may include convolutional kernels, etc.

[0083] The control function Fi(F1, F2...F) is parameterized from the control parameters. N These functions are used to correct the synthesized feature maps at the inputs or outputs of various layers of the synthetic network. These functions are parameterized by the parameter PAR. Depending on the nature of the data being manipulated, the function Fi can perform any operation within the capabilities of someone skilled in the art: multiplication, thresholding, addition, nonlinearity, linear combination, etc. Any known mathematical function or combination or composition of such functions can be used. For example, the function may include: – An affine function of the type f(x) = a.FM(x) + b, where a and b are the parameters of the function, and FM(x) represents the value of the feature map at coordinate x, for example, when the map is represented as a one-dimensional vector; – f(x) = a.FM(x) 2 Polynomial functions of type +b.FM(x)+c, such as second-order polynomials, where a, b, and c are the parameters of the function; – Any known mathematical function or combination or composition of such functions, such as f(x)=a.cos(b.FM(x)+c)+d.

[0084] This function can, of course, be a multivariable function, for example: – A function of type f(x,y) = a.FM(x,y) + b.FM(x,y) + c.sqrt(xy), where sqrt() is the square root, and x and y are the coordinates of the value of the feature map FM; A filter function of type f(x,y) = a.FM(x-1,y) + b.FM(x,y) + c.FM(x+1,y) is used to perform basic filtering on the feature map FM at integer coordinates indexed by x and y.

[0085] This function is applied to the values ​​of the synthesized feature map. For example, parameters a, b, and c are specific to each value of the map. According to another example, the same parameters are applied to multiple values ​​of the map.

[0086] According to some embodiments: All functions Fi are distinct and use their own parameters; At least two functions Fi are identical; At least two functions Fi (whether different or not) share a set of parameters from this set of control parameters.

[0087] Figure 4 An example of a synthetic neural network for encoding and decoding according to an embodiment of the present invention is shown.

[0088] Figure 4 The diagram shows a single layer Sk of the synthetic network. This layer takes a feature map, represented as FMk_IN, at the input (a subset of the synthetic feature map FMSYN) and produces a feature map FMk_OUT at the output. For example, layer Sk can be a convolutional layer, an upsampling layer, etc.

[0089] The second layer (denoted as Fk, whose input corresponds to the output FMk_OUT of Sk) is the control layer, which corresponds to the application of the function Fk to correct the synthesis network. The function Fk is parameterized by the parameters PAR (which here correspond to a set of control feature maps (in the latent domain)) generated by the control network. The function Fk takes at least one control feature map from the control feature maps as parameters and performs an operation (function) on the synthesis feature map FMk_OUT of layer Sk to generate a set of synthesis feature maps FMk+1_IN at the output that can be applied to the input of layer Sk+1.

[0090] Therefore, the synthetic network progressively generates synthetic feature maps, which represent the signal in the latent domain with progressively increasing granularity.

[0091] Figure 4 The examples presented here show control charts and composite charts with the same resolution.

[0092] Of course, the synthesized graphs can have different resolutions. For example, the number of values ​​included in the graph FMk_OUT can be greater than the number of corresponding values ​​in FMk_IN. In this case, Sk performs upsampling, including, for example, regular, bicubic, or bilinear interpolation functions or any other filtering function; according to another example, neural networks can be used to perform upsampling, which include sampling, pixel doubling (by pixel shuffling), or convolutions with bias, stride (the offset of the deconvolution kernel on each iteration), dilation (the spacing of the input values ​​supplied to the kernel), etc. Furthermore, synthesized graphs in the same layer can have different resolutions from each other (e.g., they can form a multi-resolution graph pyramid).

[0093] Similarly, as will Figure 6 As shown, control charts can have different resolutions than composite charts. Furthermore, control charts at the same level can have different resolutions from each other (e.g., they can form a multi-resolution chart pyramid).

[0094] The number of composite plots and control plots can be the same (one control plot can be used to correct one composite plot) or different (one control plot can be used to correct multiple composite plots, or multiple control plots can be used to correct one composite plot).

[0095] Figure 5 An example of the correction applied to the feature map of the synthetic network by the control network is shown.

[0096] According to the presented embodiment, the number of control feature maps and synthetic feature maps (PAR and FMk) is 4, and they all have the same resolution. The control network generates as many control feature maps PAR (denoted as PAR1, ..., PAR4) as the synthetic feature maps FMk (denoted as FMk1, ..., FMk4) at the output or input of one of the synthetic layers Sk. Each control feature map numbered i has the same resolution as the corresponding synthetic map and is used as a parameter of the function Fki. The control feature maps are used to correct the synthetic map in order to provide a correction map FMkC (denoted as FMkC1...FMkC4). Of course, the number of each type of map can be different, and these maps can have different resolutions. The parameterized function Fk can perform any type of the same or different operations, such as, but not limited to: – Apply a mathematical function parameterized by the values ​​of the control chart, or a combination or composition of such mathematical functions; examples of functions have already been given above; for example, a simple function is to multiply each value of the composite map by the corresponding value of the control chart; another simple function is to add the value of the control chart to each value of the composite map; according to another example, each function uses the corresponding control chart to mask the composite feature map to which the function is applied. For example, the control feature map includes values ​​corresponding to the mask values ​​(e.g., attention masks with integer values ​​between 0 and 255, or real values ​​between 0 and 1; or masks containing only values ​​0 or 1, etc.). The composite feature map FMk numbered i... i The value v of the coordinates (x, y) is multiplied by the mask value at the corresponding coordinates x and y of the control feature map in order to provide a correction value.

[0097] – The values ​​of the synthesized map are convolved with a convolution kernel obtained from the control map; following a very simple example, the values ​​of the synthesized map are filtered using one or more convolution kernels defined by parameters or the control map. For example, a single kernel of size 3 × 3 is used. The 3 × 3 kernel is applied to various locations in the synthesized map to obtain a corrected output map. Any supplementary operations within the capabilities of those skilled in the art can be added to the convolution: bias, stride (the offset of the deconvolution kernel on each iteration), dilation (the spacing of the input values ​​supplied to the kernel), etc. Similarly, any activation function within the capabilities of those skilled in the art (ReLU, sigmoid, Softmax, Softplus, Softsign, etc.) can be used.

[0098] --etc.

[0099] More generally, the feature map FMk numbered i i The value v is obtained by applying the function Fk i (x,y) and thus generate the graph FMkC i The value v'. For example, if the function is affine, then the feature map FMk i via Fk i (x,y) =a i (x,y) FMk i (x,y) + b i A function of type (x,y) is used for correction, where a i and b i The control feature map PAR at coordinates x and y with index i i The associated parameters. Similar reasoning applies to the functions described above.

[0100] Based on the same formal system, the correction function can be expressed in the following form: v' = Fki (x,y) =a i FMk i (x-1,y) + b i FMk i (x,y) + c i FMk i (x+1,y), the purpose is to use the parameters common to the entire feature map (a i b i c i (In this case, the control parameters consist of the triplet) or variables on x and y (in this case, the control parameters consist of parameters a and y respectively). i b i and c i The basic horizontal filtering is performed by using three graphs (or a single graph consisting of triplet values ​​at each coordinate x, y).

[0101] Figure 6 This demonstrates another example of the correction applied to the feature map of the synthetic network by the control network.

[0102] According to the presented embodiment, the number of control feature maps and synthetic feature maps (PAR and FMk) is 4 each. (See above reference.) Figure 5 As described, the control network generates as many control feature maps PAR (denoted as PAR1, ..., PAR4) as the synthetic feature maps FMk (denoted as FMk1, ..., FMk4) at the output or input of one of the synthetic layers Sk. Each control feature map numbered i is used as a parameter of the function Fki. The control feature maps are used to correct the synthetic map in order to provide a correction map FMkC (denoted as FMkC1...FMkC4). The function Fk, thus parameterized, can perform any type of the same or different operations, as referenced above. Figure 5 As mentioned.

[0103] On the other hand, with reference to Figure 5 Conversely, the resolution of the control feature map is lower than that of the synthesized feature map. For example, the resolution is reduced to one-quarter of the original in each dimension.

[0104] Feature map FMk numbered i i The value v of the coordinates (x, y) is obtained by applying the function Fk. i And generate graph FMkC i The value v'. For example, if the function is affine, then the feature map FMk i The coordinates (x, y) and the value v will be applied by v' = Fk i(v) =a i (x / 4, y / 4) FMk i (x,y)+ b i A function of type (x / 4, y / 4), where a i and b i The control feature map PAR at coordinates x / 4 and y / 4 (integer division) with index i. i Related parameters.

[0105] In other words, this is equivalent to dividing the feature map to be corrected into blocks of a predetermined size (16 values ​​in this case), and equivalent to using a function parameterized by a single set of control parameters for each block.

[0106] Figure 7 It shows that it can be generated by Figure 1 A flowchart of an example of a decoding method implemented by a decoding device.

[0107] In step E30, an encoded data stream BS is obtained. This encoded data stream can be received from a communication network or read from a storage medium. Furthermore, certain information (e.g., information related to a reference control or synthesis network) can be read from a memory accessible by the decoder. The obtained data consists of encoded data FMC corresponding to one or more encoded feature maps (e.g., multiple multi-resolution maps), encoded parameters WcCon of at least one control neural network, and WcSyn of at least one synthesis neural network.

[0108] In step E31, the FMD module decodes the NFC raw feature maps FMOR. According to one embodiment, these raw feature maps are decoded using conventional techniques within the capabilities of those skilled in the art, such as prediction followed by decoding using a context entropy encoder, and then dequantization. According to another embodiment, these raw feature maps are obtained from a standard decoder (JPEG, MPEG, etc.). According to yet another embodiment, these raw feature maps are decoded (synthesized) by a decoding neural network.

[0109] In step E32, the NNC module decodes the control parameters WcCon of the control neural network and the synthesis parameters WcSyn of the synthesis neural network. For this purpose, any known neural network decoding technique corresponding to the encoding technique used by the encoder can be used, such as the neural network coding standard NNR proposed in Part 17 of the MPEG-7 standard.

[0110] According to one embodiment, the parameters of the neural network are received by the decoder in the bitstream. According to one embodiment, some parameters of the neural network are obtained from memory accessible by the decoder. According to one embodiment, some reference parameters of the neural network are obtained from memory accessible by the decoder. According to one embodiment, an indicator received in the stream enables the selection of one or more synthesis and / or correction networks from a plurality of networks accessible by the decoder.

[0111] In step E33, the decoded control neural network NNCON is applied to the original feature map FMOR to generate control parameters PAR. According to one embodiment, the parameters PAR take the form of a set of control feature maps.

[0112] In step E34, the layers of the decoding neural network NNSYN are applied to the synthesized feature map (corrected by a control function, or uncorrected) to generate an output feature map according to one embodiment. According to one embodiment, a first parameterization function is applied to the original image FMOR. According to another embodiment, the first synthesis layer is applied directly to the original image FMOR.

[0113] In step E35, at least a portion of the parameter PAR is applied to the input or output of the synthesis layer Sk of the synthesis network. According to one embodiment, the parameter PAR takes the form of a set of control feature maps (PARs), each of which is used to parameterize a function Fk applied to the input or output of the layer, for example, to one of the input or output synthesis feature maps FMSYN.

[0114] Steps E34 and E35 can be interchanged.

[0115] Steps E34 and E35 are repeated as needed (as many times as the number of layers in the synthesis network) to generate the decoded signal based on the feature maps representing the signal. The number of iterations is limited by the network structure.

[0116] The decoded signal Sd corresponds to the output of the last synthesized layer, whether corrected or uncorrected by the parameterized function.

Claims

1. A method for encoding a signal (S) containing multiple samples, the method comprising the following steps: – Generate a set of original feature maps (FMOR); – Generate the following: at least one synthetic neural network (NNSYN) defined by a set of synthetic network parameters (WoSyn), the at least one synthetic neural network including at least one synthetic neural layer (S1, SN); and a control neural network (NNCON) defined by a set of control network parameters (WoCon); – Apply the control neural network to the at least one original feature map (FMOR) to obtain a set of control parameters (PAR); – The at least one original feature map is processed by the synthetic neural network to produce a representation of the decoded signal (S') at the output, the processing including at least one step of applying a function (F1…FN) parameterized by at least a portion of the set of control parameters (PAR) to the input or output of the at least one synthetic neural layer (S1, SN); – Update the original feature map and / or at least a portion of the parameters of the synthesized and / or control neural network based on coding performance metrics; – Encode the original feature map and / or at least a portion of the parameters of the synthesized and / or control neural network based on encoding performance metrics.

2. A method for decoding a signal (Sd) containing multiple samples, the method comprising the following steps: – Obtain at least one feature map (FMOR) called the original map (E31), which represents the signal (Sd) in the latent domain. – Decode (E32) the parameters (WcSyn) of a first neural network (NNSYN) called a synthetic neural network, which includes at least one synthetic neural layer (S1, SN). – Decode the parameters (WcCon) of the second neural network (NNCON), which is called the control neural network (E32); – Apply the control neural network (E33) to the at least one original feature map (FMOR) to obtain a set of control parameters (PAR); – Processing the at least one original feature map (E34, E35) by the synthetic neural network to generate a decoded signal (Sd) at the output, the processing includes at least one step of applying at least one function (F1…FN) parameterized by at least a portion of the set of control parameters (PAR) to the input or output of the at least one synthetic neural layer (S1, SN).

3. The decoding method as described in claim 2, comprising the step of decoding the at least one original feature map.

4. The decoding method as described in claim 2, wherein, The at least one synthetic neural layer (S1, ..., SN) is applied to at least one feature map (FMk_IN) at the input and generates at least one feature map (FMk_out) at the output. The input map and the output map are referred to as synthetic feature maps (FMSYN).

5. The decoding method as described in claim 4, characterized in that, Apply the same parameterization function to at least two synthetic feature maps (FMSYN).

6. The decoding method as described in claim 4, characterized in that, Use the same control parameters to parameterize at least two different functions applied to at least two synthetic feature maps respectively.

7. The decoding method according to any one of claims 2 to 6, wherein, The set of control parameters (PAR) includes at least one convolutional kernel.

8. The decoding method according to any one of claims 2 to 7, wherein, The set of control parameters (PAR) includes at least one feature map referred to as the control feature map.

9. The decoding or encoding method as described in any one of the preceding claims, characterized in that, The at least one control neural network includes at least one attention module.

10. The decoding or encoding method as described in any one of the preceding claims, characterized in that, The at least one control neural network includes at least one convolutional layer.

11. An apparatus for encoding a signal (S) containing multiple samples, the apparatus being configured to perform the following steps: – Generate a set of original feature maps (FMOR) (INIT / MAJ); – Generate the following: at least one synthetic neural network (NNSYN) defined by a set of synthetic network parameters (WoSyn), the at least one synthetic neural network including at least one synthetic neural layer (S1, SN); and a control neural network (NNCON) defined by a set of control network parameters (WoCon); – Apply the control neural network to the at least one original feature map (FMOR) to obtain a set of control parameters (PAR); – The at least one original feature map is processed by the synthetic neural network to produce a representation of the decoded signal (S') at the output, the processing comprising: At least one step of applying a function (F1…FN) parameterized by at least a portion of the set of control parameters (PAR) to the input or output of the at least one synthetic neural layer (S1, SN); – Update at least a portion of the original feature map and / or the synthetic and / or control neural network parameters based on encoding performance metrics (MAJ); – Encode at least a portion of the original feature map and / or the parameters of the synthesized and / or control neural network based on encoding performance metrics (FMC, NNC).

12. An apparatus for decoding a signal (Sd) containing multiple samples, the apparatus being configured to perform the following steps: – Obtain at least one feature map (FMOR) called the original map (FMD), which represents the signal (Sd) in the latent domain. – Decode (NND) the parameters (WcSyn) of a first neural network (NNSYN) called a synthetic neural network, which includes at least one synthetic neural layer (S1, SN). Decode (NND) the parameters (WcCon) of the second neural network (NNCON), which is called the control network. – Apply the control neural network (NNCON) to the at least one original feature map (FM) to obtain a set of control parameters (PAR); – Processing the at least one original feature map using the synthetic neural network (NNSYN) to generate a decoded signal (Sd) at the output, the processing includes: At least one step of applying at least one function (F1…FN) parameterized by at least a portion of the set of control parameters (PAR) to the input or output of the at least one synthetic neural layer (S1, SN).

13. A computer program comprising instructions which, when executed by a computer, are used to perform the steps of the encoding method of claim 1 or the decoding method of claim 2.

Images