Fine tuning of limited parameter sets in depth coding systems for images

Abstract

A deep neural network-based encoding system for an image determines update parameters of a deep neural network model for decoding an image. These parameters are determined by the encoder and provided to the decoder to update the model of the decoder prior to decoding the image. This provides structural sparsity by trimming only some parameters of the neural decoder. The updating takes place either for a predetermined set of parameters such that the structural sparsity is the same for all images, or for a set of parameters selected based on the image to be encoded such that the structural sparsity is specific to the image. A new training program and end-to-end trainable quantization are also presented, allowing trained parameters to be included in the bitstream and parameters in the decoder to be updated.

Description

Technical Field

[0001] At least one of the present embodiments generally relates to neural network based image compression, and more specifically to fine-tuning parameters of a deep decoder. Background Art

[0002] Image and video compression is a fundamental task in image processing, which has become crucial in the era of pandemics and increasing video streaming. Thanks to decades of tremendous community effort, traditional methods have reached state-of-the-art rate-distortion performance and dominate current industrial codec solutions. End-to-end trainable deep models have recently emerged as an alternative with promising results. They now beat the best traditional compression methods (VVC, Versatile Video Coding) even in terms of peak signal-to-noise ratio for single image compression. Summary of the invention

[0003] A novel deep neural network based coding system for an image to be encoded proposes determining update parameters of a deep neural network model used to decode the encoded image. These parameters are determined by the encoder and provided to the decoder to update the decoder's model before decoding the image. This provides structural sparsity by fine-tuning only some parameters of the neural decoder.

[0004] According to a first aspect of at least one embodiment, a method for encoding an image comprises: determining an embedding representing an input image using a deep neural network based on a first model including a parameter set; determining a parameter update to fine-tune a second model based on the first model, wherein the fine-tuning is based on a decoded version of the input image and the embedding decoded using the deep neural network based on the second model; and generating encoded data comprising at least encoding of a quantized embedding and encoding of a quantization parameter update, wherein the parameters are limited to a selected parameter set.

[0005] According to a second aspect of at least one embodiment, a method for decoding an image comprises: obtaining a decoded embedding and a decoded parameter update from encoded data; updating parameters of a model of a deep neural network by the obtained parameter update; and determining a decoded image based on the obtained decoded embedding using the deep neural network with the updated parameters.

[0006] According to a third aspect of at least one embodiment, an apparatus includes an encoder for encoding an image, the encoder configured to: determine an embedding representing an input image using a deep neural network based on a first model including a parameter set; determine a parameter update to fine-tune a second model based on the first model, wherein the fine-tuning is based on a decoded version of the input image and the embedding decoded using the deep neural network based on the second model; and generate encoded data comprising at least encoding of a quantized embedding and encoding of a quantization parameter update, wherein the parameters are limited to a selected parameter set.

[0007] According to a fourth aspect of at least one embodiment, a device includes a decoder for decoding an image, wherein the decoder is configured to: obtain decoded embeddings and parameter updates from encoded data; update parameters of a model of a deep neural network through the obtained parameter updates; and determine a decoded image based on the obtained decoded embedding using the deep neural network with updated parameters.

[0008] According to a fifth aspect of at least one embodiment, a computer program is proposed, comprising program code instructions executable by a processor, wherein the computer program implements the steps of the method according to at least the first or second aspect when executed on the processor.

[0009] According to a sixth aspect of at least one embodiment, a non-transitory computer-readable medium is proposed, comprising program code instructions executable by a processor, which when executed on the processor implements the steps of the method according to at least the first or second aspect.

[0010] In a variant of the first and third aspects, the selected parameter set is independent of the input image. In further variants of the first and third aspects, the selected parameter set is selected based on the input image, and wherein the encoded data also includes information representing the selection. In a variant of the first and third aspects, quantization of the parameter update is performed based on a trained quantization with a quantization parameter, and wherein the encoded data also includes information representing the quantization parameter. In a variant of the first and third aspects, the fine-tuning is based on a loss function to minimize a distortion measure between the input image and an image reconstructed using a deep neural network based on a second model with updated parameters.

[0011] In variations of the first, second, third and fourth aspects, the parameters are selected from a set comprising biases, weights, parameters of a nonlinear function of the model, a subset of layers of the model, a specific layer of the model, biases of a specific layer of the model, and a subset of neurons of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 An example of an end-to-end neural network-based compression system for encoding images using a deep neural network is shown.

[0013] Figure 2 An example of an image encoder that uses the same structural sparsity for any image in accordance with at least one embodiment is shown.

[0014] Figure 3 An example of an image decoder that uses the same structural sparsity for any image in accordance with at least one embodiment is shown.

[0015] Figure 4 An example of a flow chart for an image encoder using the same structural sparsity for any image in accordance with at least one embodiment is shown.

[0016] Figure 5 An example of a flow chart for an image decoder using the same structural sparsity for any image in accordance with at least one embodiment is shown.

[0017] Figure 6 An example of an image encoder using image specific structured sparsity in accordance with at least one embodiment is shown.

[0018] Figure 7 An example of an image decoder using image specific structured sparsity in accordance with at least one embodiment is shown.

[0019] Figure 8 An example of a flow chart for an image encoder using image specific structured sparsity in accordance with at least one embodiment is shown.

[0020] Fig. 9 An example of a flow chart for an image decoder using image specific structured sparsity in accordance with at least one embodiment is shown.

[0021] Fig.10 A block diagram of an example of a system implementing various aspects and embodiments is shown.

[0022] Fig.11 An example of a format for describing weight update quantization in accordance with at least one embodiment is shown.

[0023] Fig.12 The impact of the layer values ​​of the last few layers to be updated is shown.

[0024] Fig.13 The average performance for different values ​​of k is shown.

[0025] Fig.14 The performance achieved when using the optimal k value for each baseline model M is shown.

[0026] Fig.15 The PSNR vs. bits per pixel comparison of our scheme on two different baselines is shown, using six trained models on each baseline.

[0027] Fig.16 Shown is the effect of the new training procedure (new loss vs. old loss) and trainable weight quantization (learnable Q vs. non-learnable Q) on the 14th image of the test set, using one quality. DETAILED DESCRIPTION

[0028] Figure 1 An example of an end-to-end neural network based compression system for encoding an image using a deep neural network is shown. In such a system 100, an input image x to be compressed is first processed by a deep neural network encoder (hereinafter referred to as a deep encoder or encoder) in an encoding device 110. The output y of the encoder is called the embedding g of the image. The embedding is converted into a bit stream 120 by an arithmetic encoder AE after passing through a quantizer Q. The resulting bit stream 120 is provided to a decoding device 130 and decoded by an arithmetic decoder AD to reconstruct the quantized embedding The reconstructed quantized embedding is then processed by a deep neural network decoder (hereinafter referred to as deep decoder or decoder) To get the decompressed image

[0029] Deep encoders and decoders consist of multiple neural layers (e.g., convolutional layers). Each neural layer can be described as a function that first multiplies the input by a tensor, adds a vector called a bias, and then applies a nonlinear function to the resulting value. The properties of the tensor and the type of nonlinear function are called the architecture of the network. The values ​​of the tensors and biases are denoted by the term "weights." The parameters of the weights and nonlinear functions (if applicable) are called the parameters of the network. The architecture and parameters define the "model." Typically, encoders and decoders are fixed and based on a predetermined model that should be known at encoding and decoding time. The layers of the decoder are denoted as l1,...l i , ..., l n , the parameters of the decoder are denoted as θ. For example, the encoder and decoder models are trained simultaneously so that they are compatible. Sometimes, they are collectively referred to as "autoencoders", i.e. models that encode inputs and reconstruct them afterwards. The architecture of the decoder is usually roughly the opposite of that of the encoder, although some layers or their order may be slightly different.

[0030] Many end-to-end architectures have been proposed. Figure 1 The architectures shown in are more complex, but they all retain a deep encoder and decoder. In terms of rate-distortion trade-off, current state-of-the-art models are comparable to traditional video codecs such as Versatile Video Coding (VVC).

[0031] The model M must be trained on a massive image database D to learn the weights of the encoder and decoder.

[0032] Typically, weights are optimized to minimize the training loss, for example expressed as:

[0033]

[0034] where p M denotes the probability of a quantized embedding according to M (hence this term is a theoretical lower bound for the bitstream size of the encoded quantized embedding), d(.,.) denotes a measure of the distortion between the original image and the reconstructed image (e.g., mean squared error, multi-scale structural similarity index metric (MS-SSIM), information weighted structural similarity index metric (IWSSIM), video multi-method assessment fusion (VMAF), visual information fidelity (VIF), peak signal-to-noise ratio human visual system modification (PSNR-HVS-M), normalized Laplacian pyramid distance (NLPD), or feature similarity index metric (FSIM)), and λ denotes a parameter that controls the trade-off between the rate (r) and distortion (d) terms.

[0035] Typically, the architecture is trained multiple times with different values ​​of λ to produce a set of models with different rate / distortion (r / d) trade-offs {M i}. Typically, different architectures will produce models with different r / d points. To compare these architectures, the r / d points for each architecture are interpolated, resulting in a function d(r) for each architecture that provides a distortion estimate for any value of rate.

[0036] like Figure 1 As shown, the deep decoder can decode any type of image. In other words, it performs well on average for all images, but it may not be optimal for any single image. The rate-distortion tradeoff for a single video can be improved by retraining the decoder specifically for that video and transmitting weight updates δ to the decoder in addition to the quantized embeddings for the intra frames of the video. δ is added to θ before decoding the quantized embeddings. This technique is called fine-tuning. The weight update δ is determined by a fine-tuning algorithm that minimizes a loss function, which can be, for example:

[0037]

[0038] where p Δ (·) represents the probability density of weight update, represents the image reconstructed by the decoder with weights updated by δ, and β represents the trade-off between the two losses.

[0039] However, this scheme cannot achieve rate-distortion improvements for a single image due to the increased code size caused by the inclusion of weight updates. In an exemplary embodiment, an additional term can be added to the loss to impose a global sparsity constraint on δ, so that many weight updates have the same value (0), making the encoding more efficient.

[0040] Current schemes of fine-tuning the decoder using a global sparsity constraint bring rate-distortion performance gains for video coding. However, this scheme is not applicable to single images because even with a global sparsity constraint, the inclusion of weight updates still results in increased code size. Furthermore, fine-tuning the decoder requires optimizing the value of β. This can result in fine-tuning the decoder multiple times, which is an expensive process.

[0041] The embodiments described below have been designed with the above in mind and are based on imposing the structural sparsity of the deep neural network used in the image compression system, in other words, only fine-tuning some parameters of the neural decoder, thereby reducing the number of updates to be encoded. This results in better coding efficiency even for a single image, since the amount of data representing the encoded image is reduced. This principle also applies to images (i.e. frames) of a video sequence.

[0042] In an embodiment, a deep neural network-based encoding system for an image determines selected update parameters of a deep neural network model for an image to be encoded. These parameters are provided to a decoder to update the decoder's model before decoding the image. This provides structural sparsity by fine-tuning only a selected subset of parameters of the neural decoder. In this case, fine-tuning refers to a training algorithm that is adapted to train a machine learning model that has been trained on a typically much larger dataset using a small set of data points. In this particular case, the decoder (previously trained on a large dataset) is fine-tuned for a single image (a small dataset). For example, fine-tuning is performed by minimizing a loss function. In at least one embodiment, the update of the model is performed on a selected set of parameters that are independent of the image to be encoded (e.g., the biases of the last five convolutional layers of the model). In such an embodiment, the structural sparsity is the same for all images. In at least one embodiment, the set of parameters used to update the model is selected based on the image to be encoded. In such an embodiment, the structural sparsity is image-specific.

[0043] At least one embodiment proposes fine-tuning an end-to-end decoder using a training procedure that avoids optimizing hyperparameters and ensures better r / d performance by explicitly maximizing bitrate savings.

[0044] At least one embodiment proposes applying trainable quantization to weight updates in end-to-end decoder fine-tuning and including these training parameters in the bitstream to improve performance.

[0045] Figure 2 An example of an image encoder that uses the same structural sparsity for any image according to at least one embodiment is shown. Such an encoder 200 is, for example, Fig.10 In this embodiment, by fine-tuning only a limited set of selected parameters of the decoder to enforce structural sparsity. ft is the same for all images; in other words, the same subset of parameters is fine-tuned for all images. For example, the finite set may include biases and / or weights and / or parameters of nonlinear functions and / or any other parameters of the decoder and / or any subset of these elements. For example, such a subset may be defined as a subset of layers, such as the last k layers, or the biases of the last k layers, or a subset of neurons. In at least one embodiment, the selected parameter set θ ft The following description and figures use the example of weight updating, but the same principles apply to other parameters of the model.

[0046] The input image x is first encoded using a depth encoder 210 to obtain an embedding y. The embedding is then quantized, for example, by a quantizer 211, and encoded, for example, by an arithmetic encoder 212 or another encoder, resulting in an encoded quantized embedding 231.

[0047] Based on the input image x and the quantized embedding The weight updates are optimized by a fine-tuning algorithm 220. The fine-tuning algorithm is used for the selected parameters θ ft Different updates of δ ft Iterations are performed to jointly minimize the distortion measure between the original image and the reconstructed image (with updated parameters) and the code length of these updates. To this end, the fine-tuning loss function can be, for example:

[0048]

[0049] image is an image to be decoded using the updated decoder using the selected parameters θ ft The updated fine-tuning parameter δ ft .

[0050] The loss can also contain other terms, such as those that impose constraints on the weights (such as sparsity constraints).

[0051] These weight updates may then be quantized, for example, by quantizer 221. We use Finally, the weight updates are quantized using, for example, the arithmetic encoder 222 or another encoder. to encode.

[0052] The encoded data is then aggregated, for example, into a bitstream, and includes at least the quantized embedded 231 and weight update 232, for example they are encoded by an arithmetic encoder or another encoder.

[0053] The quantization and encoding of the weight updates depends on the parameters, which may be the same for all images, or some / all parameters may be fine-tuned for each image. In the latter case, the encoded data also includes the values ​​of these parameters 233, denoted by C in the figure. Fig.11 Examples of formats for carrying C are presented and the rationale discussed.

[0054] Those skilled in the art will appreciate that these elements 231, 232, 233 may be arranged in any order, and may even be interleaved in the bitstream.

[0055] In a variation of this embodiment, the quantized embedding Can be compared with δ ft In this case, the bitstream remains the same, but the loss may be:

[0056]

[0057] Figure 3 An example of an image decoder using the same structural sparsity for any image according to at least one embodiment is shown. The decoder 300 is, for example, Fig.10 The device 1000 is implemented in the embodiment of the present invention and is adapted to decode the Figure 2 The data is encoded by the encoder 200, which is arranged as a bit stream 230, for example, and includes the encoded quantized embedding 231, the weight update 232 and the optional coding information C 233. If present, the coding information C 233 is extracted from the bit stream. The quantized embedding is decoded, for example, by the arithmetic decoder 311 as The quantized weight update is decoded, for example, by arithmetic 312 as δ ft (optionally based on the encoding information C). Then, the depth decoder 320 is updated based on the quantization weight update. Finally, the updated depth decoder 330 (i.e., the selected parameter subset (e.g., weights) has been updated according to δ ft The updated deep decoder) is embedded from the quantized Decode the image

[0058] The figure shows a system in which the reversible operations associated with quantization of the weight updates are also reversed in AD block 312. The same system can be described using additional blocks (placed between 312 and 320) called, for example, "inverse quantization" or "inverse quantization" for performing these operations. An example of such a reversible operation is scaling the weight updates prior to quantization to change the quantization resolution.

[0059] Figure 4 An example flowchart for an image encoder using the same structural sparsity for any image according to at least one embodiment is shown. The flow is composed of Figure 2 The encoder 200 operates, for example, in Fig.10 In step 410, the device obtains an input image. In step 420, the device uses a depth encoder to determine a corresponding embedding. In step 430, the embedding is quantized and encoded. In step 440, the device determines a parameter update for a selected parameter subset of the depth decoder, such as described above with respect to Figure 3 In step 450, the parameter update is quantized and encoded. In step 460, the encoded data (including at least the quantized encoded embedding and the quantized encoded parameter update) is aggregated into a bitstream adapted, for example, for provision to another device or storage on a storage medium.

[0060] As described above, the parameters used for updating may include biases and / or weights and / or parameters of nonlinear functions and / or any other parameters of the decoder and / or any subset of these elements, and may be defined as a subset of layers, such as the last k layers.

[0061] Optionally, encoding information is determined and encoded so as to be embedded in the encoded data along with other data.

[0062] Figure 5 An example flowchart for an image decoder using the same structural sparsity for any image according to at least one embodiment is shown. Figure 3 The decoder 300 operates, and for example in Fig.10 The method is implemented in device 1000. In step 510, the device obtains encoded data aggregated together (for example, aggregated in a bit stream received from another device or read from a storage medium) and decodes the encoded data. The encoded data includes at least quantized encoded embeddings and quantized encoded parameter updates. As a result of the decoding, the decoded data includes at least quantized embeddings and parameter updates. In step 520, the device updates the values ​​of the selected parameter subset based on the parameter update, thereby updating the depth decoder. In step 530, the device determines an image based on the embedded and updated depth decoder. Due to the update, the difference between the original input image and the decoded image is reduced compared to decoding using a decoder that is not updated.

[0063] Figure 6 An example of an image encoder using image-specific structured sparsity according to at least one embodiment is shown. For example, such an encoder 600 is implemented in Fig.10 In the device 1000. Although as described above, for a fixed parameter subset θ ft Fine-tuning can improve the rate-distortion tradeoff for a single image, but this particular structural sparsity constraint may not be optimal for every image. In this embodiment, an image-specific structural sparsity constraint is used. In other words, the subset of parameters to be fine-tuned may be different for each image, and the subset of parameters is selected based on the input image to be encoded.

[0064] However, allowing the fine-tuning algorithm to select any subset of parameters may be counterproductive. In fact, in this case, the bitstream must also contain information for identifying this subset. This information can be included, for example, by including an index to the optimized weights. This will significantly increase the bitstream size and lead to a worse rate-distortion trade-off.

[0065] Therefore, in this embodiment, the optimization θ for each image is ft The degrees of freedom of the fine-tuning algorithm are limited to a subset of parameters. Let Denote the set of non-overlapping subsets of θ, let δ1,...,δ m represents the relevant parameter update. For each image x, the fine-tuning algorithm can fine-tune any parameter combination θ1,...,θ m Therefore, the fine-tuning algorithm attempts to solve the following combinatorial optimization problem to select the subset of weights to be fine-tuned:

[0066]

[0067] where Ω represents θ1, ..., θ m Then, we calculate ω as described in the previous section. * Update of weights

[0068] The input image x is first encoded using a depth encoder 610 to obtain an embedding y. The embedding is then quantized (e.g., by a quantizer 611) and encoded (e.g., by an arithmetic encoder 612 or another encoder) to obtain an encoded quantized embedding 641.

[0069] Selection box 620 selects the weight subset ω to be optimized according to the above combinatorial optimization problem * . Weight subset ω * Different techniques can be used to represent this. For example, a subset can be represented by ω * Index in Ω or ω *θ1,...,θ contained in m Then, based on the input image and quantized embedding The fine-tuning algorithm 630 optimizes the algorithm corresponding to the selected subset ω * Parameters of , thereby obtaining weight updates Fine-tune the use of the same Figure 2 The same mechanism as described for encoder 200 is described, except that the parameter set has previously been selected via selection box 620. Note that these two steps can occur simultaneously, i.e., both optimizations are performed simultaneously.

[0070] These weights are updated It is also quantized by, for example, quantizer 631. The result is The selection of the weights is then encoded, for example, by an arithmetic encoder 622, and the quantization weight updates are encoded, for example, by an arithmetic encoder 632. These elements may be encoded by an arithmetic encoder or another type of encoder.

[0071] The encoded data is then aggregated, for example, into a bitstream 640, and includes at least quantized embedded 641, weight subset ω * 642 and weight update 643.

[0072] Quantization and encoding Parameters optimized for each image may optionally be involved. In this case, the coded data also includes coding information 644 (denoted by C) representing the values ​​of these parameters.

[0073] As in reference Figure 2 , 3 , 4 and 5 above, these elements can be arranged in any order and can even be interleaved in the bitstream, and the quantization is embedded Can be used with Fine-tune together.

[0074] For example, each subset θ j It can be defined as the decoder layer l j In this case, Ω is the combination of all integers 1, ..., n. * The identifier of may be the index of the layer whose bias has been fine-tuned.

[0075] Figure 7 An example of an image decoder using image-specific structured sparsity according to at least one embodiment is shown. Such a decoder 700 may be used, for example, in Fig.10 The device 1000 is implemented in the embodiment of the present invention and is adapted to decode the Figure 7The encoder 600 encodes data, which is arranged as a bit stream 640, for example, and includes at least quantized embedded 641, weight subset ω * , weight update 643 and optional coding information C 644.

[0076] The quantized embedding is decoded by the arithmetic decoder 711 as Weight subset ω * For example, decoded by arithmetic decoder 712, and the quantization weight update is decoded by arithmetic decoder 713 as (optionally based on coding information C 644 present in the coded data). This information allows the weight subset ω to be used as the basis for the * and quantized weight update To perform the decoder update 720. Then, the image By the updated depth decoder 730 (ie some parameters have been Updated deep decoder) from quantized embedding Decoded.

[0077] Figure 8 An example flowchart for an image encoder using image-specific structured sparsity according to at least one embodiment is shown. Figure 6 The encoder 600 operates, for example, in Fig.10 is implemented in the device 1000.

[0078] In step 810, the device obtains an input image. In step 820, the device determines a corresponding embedding using a depth encoder. In step 830, the embedding is quantized and encoded. In step 835, the device determines a selected parameter subset based on the input image. In step 840, the device determines a parameter update for the selected parameter subset of the depth decoder, such as described above with respect to Figure 6 In step 850, the parameter updates are quantized and encoded. In step 860, the encoded data (including at least the quantized encoded embedding, the encoding information representing the selected parameter subset, and the quantized encoded parameter updates) are aggregated into a bitstream adapted, for example, for provision to another device or storage on a storage medium.

[0079] As described above, the parameters used for updating may include biases and / or weights and / or parameters of nonlinear functions and / or any other parameters of the decoder and / or any subset of these elements, and may be defined as a subset of layers, such as the last k layers.

[0080] Optionally, encoding information is determined and encoded so as to be embedded in the encoded data along with other data.

[0081] In addition to the encoding and decoding methods and devices described above, at least one embodiment relates to a new training procedure for fine-tuning a decoder. A key part of the training procedure is the use of a new fine-tuning loss that does not involve optimizing a hyperparameter β. Instead of directly optimizing the rate-distortion tradeoff, a loss is proposed that drives the fine-tuning algorithm to be better than the baseline model M. o This loss can be used for any decoder fine-tuning algorithm that optimizes the set of weight updates δ, including the above embodiments.

[0082] More specifically, the training procedure is performed on the distortion d achieved by the fine-tuned model. ft The following will make the two rates (the rate r of the fine-tuned model ft and the rate r of the original architecture o ) In other words, the following loss is proposed:

[0083]

[0084] Unfortunately, as mentioned above, the rate of the original architecture is not available for every distortion. However, it is possible to invert the function d o (r) to obtain the rate estimation function r of the original architecture o (d).

[0085] So the loss becomes:

[0086]

[0087] The denominator is the estimated rate of the original architecture at the distorted value of the image reconstructed by the fine-tuned encoder. The numerator is the actual rate of the fine-tuned decoder. The first term is the rate of the model M used as the baseline for fine-tuning It corresponds to the encoding of the quantized embedding. Therefore, Second term - log p Δ (δ ft ) corresponds to the encoding of the weight update, and len(C) corresponds to the encoding size of the features of the weight update quantizer that need to be transmitted.

[0088] This loss is advantageous because it does not contain any hyperparameters such as β that must be optimized. Therefore, it speeds up the fine-tuning process. The disadvantage is that it requires the function r o (d), thus requiring at least two trained models from the original architecture. This is usually not a problem since multiple models are trained for different operating points.

[0089] For example, estimating the rate We can use the baseline model M o and from M o The same model set {M i} but with a different r / d tradeoff model M p (For example, M p The rate is closest to M o The approximation is done by linear interpolation between the model with the next higher quality. In this case:

[0090]

[0091] in Represented by model M p Encode / decode images.

[0092] Any interpolation method can be used, such as polynomial interpolation of arbitrary order or approximation via a machine learning model.

[0093] Fig. 9 An example flowchart for an image decoder using image-specific structured sparsity according to at least one embodiment is shown. Figure 7 The decoder 700 in the embodiment of the present invention operates, for example, Fig.10 The device 1000 is implemented in the device 1000.

[0094] In step 910, the device obtains encoded data aggregated together (e.g., aggregated in a bitstream received from another device or read from a storage medium) and decodes the encoded data. As a result of the decoding, the decoded data includes at least a quantization embedding, information indicating a selected parameter subset, and a quantization parameter update. In step 920, the device selects a parameter set for a depth decoder based on the information indicating the selected parameter subset and updates the value of the selected parameter based on the parameter update, thereby updating the depth decoder, thereby generating an updated depth decoder. In step 930, the device determines an image based on the received embedding and the updated depth decoder.

[0095] Fig.10 A block diagram of an example system for implementing various aspects and embodiments is shown. The system 1000 may be embodied as a device including various components described below, and may be configured to perform one or more aspects described in the present application, such as Figure 2 Encoder 200, Figure 3 Decoder 300, Figure 6 Encoder 600 or Figure 7Decoder 700 of system 1000. Examples of such devices include, but are not limited to, a variety of electronic devices, such as personal computers, notebook computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, interconnected home appliances, encoders, transcoders and servers. The elements of system 1000 can be embodied in a single integrated circuit, multiple ICs and / or discrete components, either alone or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed over multiple ICs and / or discrete components. In various embodiments, system 1000 is coupled to other similar systems or other electronic devices, for example, by a communication bus or by dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more aspects described in this document.

[0096] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein, such as those used to implement the various aspects described in this document. The processor 1010 may include embedded memory, input-output interfaces, and various other circuits known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include a non-volatile memory and / or a volatile memory, including but not limited to an EEPROM, a ROM, a PROM, a RAM, a DRAM, an SRAM, a flash memory, a disk drive, and / or an optical drive. As a non-limiting example, the storage device 1040 may include an internal storage device, an attached storage device, and / or a network accessible storage device.

[0097] The system 1000 includes an encoder / decoder module 1030, which is configured to process data to provide encoded video or decoded video, for example, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a module that can be included in a device to perform encoding and / or decoding functions. As is well known, a device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000, or may be incorporated into the processor 1010 as a combination of hardware and software known to those skilled in the art.

[0098] Program code loaded onto the processor 1010 or the encoder / decoder 1030 to perform various aspects described in this document may be stored in the storage device 1040 and subsequently loaded onto the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of a variety of items during the execution of the processes described in this document. These stored items may include, but are not limited to, input video, decoded video or portions thereof, bitstreams, matrices, variables, and intermediate or final results from equations, formulas, operations, and arithmetic logic processing.

[0099] In some embodiments, memory inside the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory outside the processing device (e.g., the processing device can be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory can be a memory 1020 and / or a storage device 1040, such as a dynamic volatile memory and / or a non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used to store the operating system of the TV. In at least one embodiment, a fast external dynamic volatile memory (e.g., RAM) is used as working memory for video encoding and decoding operations (e.g., MPEG-2, HEVC, or VVC (Versatile Video Coding)).

[0100] Input to elements of system 1000 may be provided through a variety of input devices as shown at block 1130. Such input devices include, but are not limited to: (i) an RF section that receives an RF signal transmitted over the air, such as by a broadcaster, (ii) a composite input, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0101] In various embodiments, the input device of block 1130 has associated respective input processing elements known in the art. For example, the RF section may be associated with elements required for: (i) selecting a desired frequency (also referred to as selecting a signal or bandwidth limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) again bandwidth limiting to a narrower frequency band to select (for example) a signal frequency band (which may be referred to as a channel in some embodiments), (iv) demodulating the down-converted and bandwidth-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired packet stream. The RF section of various embodiments includes one or more elements for performing these functions, such as a frequency selector, a signal selector, a bandwidth limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs multiple of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or to baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal transmitted by wired (for example, cable) medium, and pass filtering, down-conversion and filtering to the desired frequency band again to perform frequency selection. Various embodiments rearrange the order of above-mentioned (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions.Adding element can include inserting element between existing element, for example inserting amplifier and analog-to-digital converter.In various embodiments, the RF part includes antenna.

[0102] In addition, the USB and / or HDMI terminals may include respective interface processors for connecting the system 1000 to other electronic devices via the USB and / or HDMI connections. It should be appreciated that aspects of input processing (e.g., Reed-Solomon error correction) may be implemented within a separate input processing IC or within the processor 1010 as desired. Similarly, aspects of USB or HDMI interface processing may be implemented within a separate interface IC or within the processor 1010 as desired. The demodulated, error corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 1010 and the encoder / decoder 1030, which operate in conjunction with memory and storage elements to process the data streams as desired for presentation on an output device.

[0103] The various elements of system 1000 may be disposed within an integrated housing. Within the integrated housing, the various elements may be interconnected and transmit data between each other using suitable connection means (eg, internal buses known in the art, including I2C buses, wiring, and printed circuit boards).

[0104] The system 1000 includes a communication interface 1050 that allows communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to send and receive data via the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.

[0105] In various embodiments, data is streamed to the system 1000 using a Wi-Fi network such as IEEE 802.11. The Wi-Fi signals of these embodiments are received by a communication channel 1060 and a communication interface 1050 adapted for Wi-Fi communication. The communication channel 1060 of these embodiments is typically connected to an access point or router that provides access to external networks including the Internet to allow streaming applications and other over-the-top communications. Other embodiments provide streaming data to the system 1000 using a set-top box that transmits data via an HDMI connection of an input box 1130. Other embodiments use an RF connection of an input box 1130 to provide streaming data to the system 1000.

[0106] The system 1000 can provide output signals to a variety of output devices, including a display 1100, speakers 1110, and other peripherals 1120. In examples of various embodiments, the other peripherals 1120 include one or more stand-alone DVRs, optical disc players, stereo systems, lighting systems, and other devices that provide functions based on the output of the system 1000. In various embodiments, control signals are communicated between the system 1000 and the display 1100, speakers 1110, or other peripherals 1120 using signaling such as AV.Link, CEC, or other communication protocols that implement device-to-device control with or without user intervention. The output devices can be communicatively coupled to the system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, the output devices can be connected to the system 1000 using a communication channel 1060 through a communication interface 1050. The display 1100 and speakers 1110 can be integrated with other components of the system 1000 in a single unit in an electronic device (e.g., a television). In various embodiments, the display interface 1070 includes a display driver, such as a timing controller (T Con) chip.

[0107] Display 1100 and speaker 1110 can also be separated from one or more other components, for example, if the RF part of input 1130 is part of a separate set-top box. In various embodiments where display 1100 and speaker 1110 are external components, output signals can be provided through dedicated output connections (for example, including HDMI ports, USB ports, or COMP outputs). The embodiments described herein can be implemented as methods or processes, devices, software programs, data streams, or signals, for example. Even if only discussed in the context of a single embodiment (for example, discussed only as a method), the implementation of the discussed features can also be implemented as other forms (for example, devices or programs). Devices can be implemented as appropriate hardware, software, and firmware, for example. Methods can be implemented in processors, for example, and processors generally refer to processing devices, for example, including computers, microprocessors, integrated circuits, or programmable logic devices. Processors also include communication devices, such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate information communication between end users.

[0108] Fig.11 An example of a format for describing the quantization of weight updates according to at least one embodiment is shown. Many existing quantization and encoding techniques can be used to quantize and encode weight updates δ of size u. ft The following scenario illustrates what C might be.

[0109] We propose to use uniform scalar quantization for the scaled bias updates during the test phase. Quantization is done by ft ,q)=round(δ ft .q) rounds the scaled input to the nearest integer value, where "." represents the product of a vector and a scalar. Since the value of q is learned for each image, it can be used to adjust the quantization resolution. Inverse quantization undoes the scaling: However, since the rounding operator has non-informative gradients, it cannot be used during the training phase. For training, the rounding operator is softened using the standard technique of additive uniform noise. Therefore, during the training phase, we apply quantization and inverse quantization as follows:

[0110] Q T (δ ft ,q)=δ ft .q+∈

[0111]

[0112] where ∈∈R u is iid (independent, identically distributed) uniform noise, where ∈ i~U(-0.5, 0.5). If we learn a quantization scale q for each image, we should use 16 bits to include q as part of C in the bitstream.

[0113] Surprisingly, the bias updates generally follow a Gaussian distribution. Since we quantize the scaled updates to the nearest integer value, the quantization bin width is 1. Therefore, given the scaled and quantized update vector The expected probability of can be calculated during fine-tuning as follows:

[0114]

[0115] in is a vector The i-th element of N(.; μ, σ) is the probability density function of the Gaussian distribution, which takes μ and σ as parameters, where μ and σ are vectors The mean and standard deviation of the Gaussian probability model are fitted to a given vector In the test phase, in order to compress the bias updates using entropy coding, a truncated Gaussian distribution is fitted to the quantized scaled bias updates, whose support is determined by the minimum symbol s min To the maximum symbol s max If these parameters are trained for each image, then C must include the fitted truncated Gaussian parameters μ, σ, and s, which are each 16 bits long and 8 bits long, in addition to the scale parameter q, which is quantized using 16 bits. min 、s max This 64-bit long information is the updated encoding information we need to add to the bitstream, and the bit length of the bitstream is represented by len(C) in the loss function. The format 1100 proposed in the figure illustrates one possibility for encoding C in the bitstream for this particular example.

[0116] The figure below shows a typical experimental result of this principle on the Kodak test set. The neural network architecture used is the cheng2020-anchor architecture, as described in "Learned image compression with discretized gaussian mixture likelihoods and attention modules" (CVPR, 2020) by Z. Cheng, H. Sun, M. Takeuchi, and J. Katto. Six different training models M are used as baselines. Different subsets of parameters are fine-tuned and evaluated: the bias of the last k convolutional layers of each model M, where k can vary. Unless otherwise stated, new training losses and trainable weight quantization are used, and the results are the average of all images in the test set.

[0117] Fig.12 The effect of the layer values ​​of the last few layers to be updated is shown. More specifically, it shows the effect of k values ​​from 1 to 10 on the BD rate gain (according to our scheme and relative to the baseline M) based on PSNR. Each data point corresponds to a baseline model M. In this case, the average value of k (e.g. k=5) is optimal, where lower values ​​deteriorate significantly. The baseline is represented by line 1210. Curves 1211 to 1221 represent k values ​​increasing from 1 to 11, respectively.

[0118] Fig.13 The average performance for different values ​​of k is shown. It summarizes Fig.12 For each k value (x-axis), it shows Fig.12 This corresponds to the average performance for each value of k from 1 to 10 across all baseline models M. In other words, the curve represents the savings relative to the baseline as the number of final convolutional bias layers increases.

[0119] Fig.14 The performance achieved when using the optimal k value for each baseline model M is shown. This better demonstrates the performance that can be achieved in practice, where the number of layers can be selected independently for each baseline model M. The baseline is represented by line 1410. Curve 1420 represents the proposed solution.

[0120] Fig.15PSNR vs. bits per pixel of our scheme on two different baselines, with six trained models used on each baseline. Curve 1510 represents a baseline based on the cheng2020-anchor architecture, and curve 1520 represents the proposed scheme applied to that baseline. Curve 1530 represents a baseline based on the bmshj2018_factorized architecture, as described in J. Ballé, D. Minnen, S. Singh, SJ Hwang, and N. Johnston, “Variational image compression with a scale hyperprior” (International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30-May 3, 2018, Conference Track Proceedings, 2018). Curve 1540 represents the application of the proposed scheme to the second baseline. For the proposed solution, only the best value of k is shown. Other values ​​of k will be between the proposed solution and the corresponding baseline.

[0121] Fig.16 The effect of the new training procedure (new loss vs. old loss) and trainable weight quantization (learnable Q vs. non-learnable Q) on the 14th image of the test set is shown, with one quality selected. This quality and image are selected as the most representative results, with values ​​corresponding to the BDrate gain relative to the baseline for different values ​​of k. Curve 1610 represents the old loss for non-learnable quantization, curve 1620 represents the new loss for non-learnable quantization, curve 1630 represents the old loss for learnable quantization, and curve 1640 represents the new loss for learnable quantization. The combination of the new loss and trainable quantization consistently achieves the best or near-best results at higher values ​​of k (x-axis), but results are slightly worse when k < 4.

[0122] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations appearing in various places in the specification do not necessarily all refer to the same embodiment.

[0123] Furthermore, the application or its claims may involve "determining" various types of information. Determining information may include one or more of the following, such as estimating information, calculating information, predicting information, or retrieving information from a memory.

[0124] Furthermore, the application or its claims may involve "accessing" various types of information. Accessing information may include one or more operations such as receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, predicting information, or estimating information.

[0125] Furthermore, this application or its claims may involve "receiving" a variety of information. Receiving, like "accessing," is intended to be a broad term. Receiving information can include one or more operations such as accessing the information or retrieving the information (e.g., from memory or optical media storage). Furthermore, "receiving" often involves being in some way during an operation such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0126] It should be understood that the use of any of the following " / ", "and / or", and "at least one of", such as in the case of "A / B", "A and / or B", and "at least one of A and B", is intended to cover selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C", such wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This can be extended to as many items as listed as will be apparent to one of ordinary skill in this and related arts.

[0127] As will be appreciated by those of ordinary skill in the art, embodiments may generate a variety of formatted signals to carry information that may be, for example, stored or transmitted. The information may include, for example, instructions for executing a method or data generated by one of the embodiments. For example, a signal may be formatted to carry a bitstream of the embodiment. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, a signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

Claims

1. A method for encoding an input image, the method comprising: - determining an embedding representing the input image using a deep neural network based on a first model including a selected subset of parameters; - determining a parameter update to fine-tune a second model based on the first model, wherein the fine-tuning is based on a decoded version of the input image and the embedding decoded using a deep neural network based on the second model; as well as - generating encoded data comprising at least an encoded quantization embedding and an encoded quantization parameter update.

2. The method of claim 1, wherein the selected subset of parameters is independent of the input image.

3. The method of claim 1, wherein the selected subset of parameters is selected based on the input image, and wherein the encoded data further includes information indicative of the selection.

4. The method of any of the preceding claims, further comprising quantizing the parameter update based on a trained quantization having a quantization parameter, and wherein the encoded data further comprises information representative of the quantization parameter.

5. The method according to any of the preceding claims, wherein the fine-tuning is based on a loss function to minimize a distortion measure between the input image and an image reconstructed using a deep neural network based on the second model with updated parameters.

6. A method according to any of the preceding claims, wherein the selected parameter subset is selected from a set including biases, weights, parameters of a nonlinear function of the model, a subset of layers of the model, a specific layer of the model, biases of a specific layer of the model, and a subset of neurons of the model.

7. A method for decoding an image represented by encoded data, the method comprising: - obtaining a decoded embedding and a decoded parameter update from said encoded data; -updating the parameters of the deep neural network model through the obtained parameter update; as well as - Using the deep neural network with updated parameters, determining a decoded image based on the obtained decoded embedding.

8. The method of claim 7, wherein the selected subset of parameters is contained in a set comprising biases, weights, parameters of a nonlinear function of a model, a subset of layers of a model, a specific layer of a model, biases of a specific layer of a model, and a subset of neurons of a model.

9. An apparatus comprising an encoder for encoding an image, the encoder being configured to: - determining an embedding representing the input image using a deep neural network based on a first model including a selected subset of parameters; - determining a parameter update to fine-tune a second model based on the first model, wherein the fine-tuning is based on a decoded version of the input image and the embedding decoded using a deep neural network based on the second model; as well as - generating encoded data comprising at least an encoded quantization embedding and an encoded quantization parameter update.

10. The device of claim 9, wherein the selected subset of parameters is independent of the input image.

11. The apparatus of claim 9, wherein the selected subset of parameters is selected based on the input image, and wherein the encoded data further includes information indicative of the selection.

12. The apparatus of any one of claims 9 to 11, further comprising quantizing the parameter update based on a trained quantization having a quantization parameter, and wherein the encoded data further comprises information representative of the quantization parameter.

13. The apparatus according to any one of claims 9 to 12, wherein the fine-tuning is based on a loss function to minimize a distortion measure between the input image and an image reconstructed using a deep neural network based on the second model with updated parameters.

14. An apparatus according to any one of claims 9 to 13, wherein the selected parameter subset is selected from a set including biases, weights, parameters of a nonlinear function of a model, a subset of layers of a model, a specific layer of a model, biases of a specific layer of a model, and a subset of neurons of a model.

15. An apparatus comprising a decoder for decoding an image, the decoder being configured to: - obtaining a decoded embedding and a decoded parameter update from said encoded data; -updating the parameters of the deep neural network model through the obtained parameter update; as well as - Using the deep neural network with updated parameters, determining a decoded image based on the obtained decoded embedding.

16. The device of claim 15, wherein the selected subset of parameters is included in a set comprising biases, weights, parameters of a nonlinear function of a model, a subset of layers of a model, a specific layer of a model, biases of a specific layer of a model, and a subset of neurons of a model.

17. A computer program comprising program code instructions for implementing the method according to at least one of claims 1 to 8 when executed by a processor.

18. A non-transitory computer-readable medium comprising program code instructions for implementing the method according to at least one of claims 1 to 8 when executed by a processor.

Images