Online training-based encoder tuning in neural image compression

Abstract

An apparatus for image / video encoding includes a processing circuit. The processing circuit performs online training of a neural image compression (NIC) framework based on one or more input images. The NIC framework is an end-to-end framework with one or more first neural networks in an encoding portion and one or more second neural networks in a decoding portion. The online training determines updates to one or more tunable parameters in the one or more first neural networks with the one or more second neural networks having fixed parameters. The processing circuit updates the one or more tunable parameters in the one or more first neural networks according to the updates, and encodes the one or more input images into a bitstream by the encoding portion of the NIC framework with the one or more tunable parameters in the one or more first neural networks that are updated.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims benefit of priority to U.S. Provisional Application No. 63 / 323,878, entitled "Online Training-based Encoder Tuning in Neural Image Compression," filed March 25, 2022, which claims benefit of priority to U.S. Provisional Application No. 18 / 122,565, entitled "ONLINE TRAINING-BASED ENCODER TUNING IN NEURAL IMAGE COMPRESSION," filed March 16, 2023. The disclosures of the prior applications are incorporated herein by reference in their entireties.

[0002] This disclosure describes embodiments generally related to image / video processing. [Background technology]

[0003] The background discussion provided herein is intended to generally present the context of the present disclosure. The inventors' work, to the extent that it is described in this background section, and aspects of the description that may not be admitted as prior art at the time of filing, are not admitted expressly or impliedly as prior art to the present disclosure.

[0004] Image / video compression can help transmit image / video files across different devices, storage, and networks with minimal quality loss. Improving image / video compression tools can require a lot of expertise, effort, and time. Machine learning techniques can be applied to image / video compression to simplify and accelerate the improvement of compression tools. Summary of the Invention [Means for solving the problem]

[0005] Aspects of the present disclosure provide a method and apparatus for image / video encoding and decoding. In some examples, the apparatus for image / video encoding includes a processing circuit. The processing circuit performs online training of a neural image compression (NIC) framework based on one or more input images. The NIC framework is an end-to-end framework including both (i) one or more first neural networks in an encoding portion, and (ii) one or more second neural networks in a decoding portion. The online training determines updates (e.g., multiple update values) for one or more tunable parameters in one or more first neural networks having fixed parameters (e.g., one or more second neural networks having non-tunable parameters). The processing circuit updates one or more tunable parameters in the one or more first neural networks according to the updates, and encodes the one or more input images into a bitstream by the encoding portion of the NIC framework having one or more tunable parameters in the one or more first neural networks that are updated.

[0006] In some examples, the fixed parameters of the one or more second neural networks are fixed at pre-trained values ​​from offline training of the NIC framework.

[0007] In some examples, the NIC framework includes a specific neural network in both the encoding portion and the decoding portion, the specific neural network including a first parameter that is fixed during online training. In one example, the specific neural network includes a hyperdecoder network.

[0008] In some examples, online training is performed with parameters within the main encoder network and the hyper-encoder network of the NIC framework being tunable.

[0009] In some examples, online training is performed with a subset of parameters within the main encoder network and the hyper-encoder network of the NIC framework being adjustable.

[0010] In some examples, online training is performed with parameters of layers within the main encoder network or the hyper-encoder network of the NIC framework being tunable.

[0011] In some examples, online training is performed with parameters of channels within the main encoder network or layers of the hyper-encoder network of the NIC framework being tunable.

[0012] In some examples, the processing circuit divides the input image into a plurality of blocks, assigns respective step sizes to the plurality of blocks, and performs online training of the NIC framework according to the plurality of blocks having the respective step sizes.

[0013] In some examples, the processing circuitry assigns a step size to the input image based on a type of content in the input image, and performs online training of the NIC framework according to the input image having the step size.

[0014] Aspects of the present disclosure also provide a non-transitory computer-readable storage medium storing a program executable by at least one processor to perform a method for image / video encoding and / or decoding.

[0015] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 illustrates a neural image compression (NIC) framework in some examples. [Diagram 2] FIG. 2 illustrates an example of a main encoder network in some examples. [Diagram 3] FIG. 2 illustrates an example of a main decoder network in some examples. [Figure 4] FIG. 1 illustrates an example of a hyper-encoder network in some examples. [Diagram 5] FIG. 2 illustrates an example of a hyperdecoder network in some examples. [Figure 6] FIG. 2 illustrates an example of a context model neural network in some examples. [Figure 7] FIG. 2 illustrates an example of an entropy parameter neural network in some examples. [Figure 8] FIG. 2 illustrates an image encoder in some examples. [Figure 9] FIG. 2 illustrates an image decoder in some examples. [Figure 10] 1 illustrates an image encoder and a corresponding image decoder in some examples. [Figure 11] FIG. 2 is another diagram illustrating an image encoder and a corresponding image decoder in some examples. [Figure 12] FIG. 2 illustrates an example of block-based image coding in some examples. [Figure 13A] 1 is a block diagram illustrating an electronic device in some examples. [Figure 13B] FIG. 2 is another block diagram illustrating an electronic device in some examples. [Figure 14] FIG. 1 illustrates an electronic device in some examples. [Figure 15] 1 is a flowchart outlining a process in some examples. [Figure 16] FIG. 1 is a schematic diagram of a computer system according to some examples. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] According to one aspect of the present disclosure, some video codecs may be difficult to optimize as a whole. For example, the improvement of a single module (e.g., an encoder) in a video codec may not result in coding improvement in overall performance. In contrast, in an artificial neural network (ANN)-based video / image coding framework, a machine learning process may be performed, and then different modules of the ANN-based video / image coding framework may be jointly optimized from input to output to improve the final objective (e.g., rate-distortion performance such as the rate-distortion loss L described in the present disclosure). For example, a learning or training process (e.g., a machine learning process) may be performed on the ANN-based video / image coding framework to jointly optimize modules of the ANN-based video / image coding framework to achieve an overall optimized rate-distortion performance, and thus the optimization result may be an end-to-end (E2E) optimized neural image compression (NIC).

[0018] In the following description, the ANN-based video / image coding framework is illustrated by a Neural Image Compression (NIC) framework. Note that, although the following description illustrates image compression (e.g., encoding and decoding), the techniques of image compression are suitably applicable to video compression.

[0019] According to some aspects of the present disclosure, the NIC framework can be trained in an offline training process and / or an online training process. In the offline training process, a set of previously collected training images can be used to train the NIC framework and optimize the NIC framework. In some examples, the parameters of the NIC framework determined by the offline training process can be referred to as pre-trained parameters, and the NIC framework with the pre-trained parameters can be referred to as a pre-trained NIC framework. The pre-trained NIC framework can be used for image compression operations.

[0020] In some examples, when one or more images (also referred to as one or more target images) are available for image compression operations, the pre-trained NIC framework is further trained based on the one or more target images in an online training process to adjust parameters of the NIC framework. The adjusted parameters of the NIC framework by the online training process can be referred to as online trained parameters, and the NIC framework with the online trained parameters can be referred to as an online trained NIC framework. The online trained NIC framework can then perform image compression operations on the one or more target images. Some aspects of the present disclosure provide techniques for online training-based encoder adjustment in neural image compression.

[0021] Neural networks refer to a computational architecture that models the biological brain. A neural network can be a model implemented in software or hardware that emulates the computational capabilities of biological systems with a large number of artificial neurons connected through connection lines. The artificial neurons, called nodes, are connected to each other and work together to process input data. Neural networks (NNs) are also known as artificial neural networks (ANNs).

[0022] The nodes in an ANN may be organized in any suitable architecture. In some embodiments, the nodes in an ANN are organized in layers including an input layer that receives input signal(s) to the ANN and an output layer that outputs output signal(s) from the ANN. In one embodiment, the ANN further includes a layer(s) that may be referred to as hidden layer(s) between the input layer and the output layer. Different layers may perform different types of transformations on the inputs of each of the different layers. Signals may proceed from the input layer to the output layer.

[0023] An ANN with multiple layers between the input layer and the output layer can be referred to as a deep neural network (DNN). The DNN can have any suitable structure. In some examples, the DNN is configured with a feed-forward network structure where data flows from the input layer to the output layer without looping back. In some examples, the DNN is configured with a fully connected network structure where each node in one layer is connected to every node in the next layer. In some examples, the DNN is configured with a recurrent neural network (RNN) structure where data can flow in any direction.

[0024] An ANN having at least a convolution layer that performs a convolution operation can be called a convolutional neural network (CNN). A CNN can include an input layer, an output layer, and a hidden layer(s) between the input layer and the output layer. The hidden layer(s) can include a convolution layer(s) (e.g., used in an encoder) that performs a convolution, such as a two-dimensional (2D) convolution. In one embodiment, the 2D convolution performed in the convolution layer is between a convolution kernel (also called a filter or channel, such as a 5×5 matrix) and an input signal to the convolution layer (e.g., a 2D matrix, such as a 2D block, a 256×256 matrix). The dimension of the convolution kernel (e.g., 5×5) is smaller than the dimension of the input signal (e.g., 256×256). During a convolution operation, a dot product is performed between the convolution kernel and a patch (e.g., a 5×5 region) in the input signal (e.g., a 256×256 matrix) of the same size as the convolution kernel to generate an output signal for input to the next layer. The patch (e.g., a 5×5 region) in the input signal (e.g., a 256×256 matrix) of the size of the convolution kernel can be called the receiving field of each node in the next layer.

[0025] During convolution, the dot product of the convolution kernel and the corresponding receptive field in the input signal is calculated. The convolution kernel contains weights as elements, and each element of the convolution kernel is a weight applied to the corresponding sample in the receptive field. For example, a convolution kernel represented by a 5 × 5 matrix has 25 weights. In some examples, a bias is applied to the output signal of the convolution layer, and the output signal is based on the sum of the dot product and the bias.

[0026] In some examples, the convolution kernel can shift along the input signal (e.g., a 2D matrix) by a size called the stride, so that the convolution operation generates a feature map or activation map (e.g., another 2D matrix) that contributes to the input of the next layer in the CNN. For example, the input signal is a 2D block with 256×256 samples, and the stride is 2 samples (e.g., a stride of 2). For a stride of 2, the convolution kernel shifts by 2 samples along the X direction (e.g., horizontal) and / or Y direction (e.g., vertical).

[0027] In some examples, multiple convolution kernels can be applied to the input signal in the same convolution layer to generate multiple feature maps, each of which can represent a particular feature of the input signal. In some examples, the convolution kernels can correspond to the feature maps. A convolution layer having N convolution kernels (or N channels), each of which has M×M samples, and a stride S, can be specified as Conv:M×M cN sS. For example, a convolution layer having 192 convolution kernels (or 192 channels), each of which has 5×5 samples, and a stride of 2, is specified as Conv:5×5 c192 s2. The hidden layer(s) can include a deconvolution layer(s) that performs deconvolution, such as 2D deconvolution (e.g., used in a decoder). Deconvolution is the inverse of convolution. A deconvolution layer with 192 convolution kernels (or 192 channels), each deconvolution kernel having 5 × 5 samples, and a stride of 2 is specified as DeConv:5×5 c192 s2.

[0028] In a CNN, a relatively large number of nodes can share the same filter (e.g., the same weight) and the same bias (if a bias is used), so that a single bias and a single weight vector can be used across all receptive fields that share the same filter, thereby reducing the memory footprint. For example, for an input signal with 100×100 samples, a convolution layer with a convolution kernel with 5×5 samples has 25 learnable parameters (e.g., weights). If a bias is used, one channel uses 26 learnable parameters (e.g., 25 weights and one bias). If a convolution layer has N convolution kernels, the total number of learnable parameters is 26×N. The number of learnable parameters is relatively small compared to a fully connected feedforward neural network layer. For example, for a fully connected feedforward layer, 100×100 (i.e., 10000) weights are used to generate a result signal for input to each node of the next layer. If the next layer has L nodes, the total number of learnable parameters is 10000×L.

[0029] A CNN may further include one or more other layers, such as a pooling layer(s), a fully connected layer(s) that may connect all nodes in one layer to all nodes in other layers, a normalization layer(s), etc. The layers in a CNN may be arranged in any suitable order and in any suitable architecture (e.g., feed-forward architecture, recurrent architecture). In one example, a convolutional layer is followed by other layers, such as a pooling layer(s), a fully connected layer(s), and a normalization layer(s).

[0030] Pooling layers can be used to reduce the dimensionality of data by combining the outputs from multiple nodes in one layer into a single node in the next layer. The pooling operation for a pooling layer with a feature map as input is described below. This description can be adapted appropriately to other input signals. The feature map can be divided into subregions (e.g., rectangular subregions), and the features in each subregion can be singly downsampled (or pooled) to a single value, for example, by taking the average value in average pooling or the maximum value in max pooling.

[0031] The pooling layer can perform pooling such as local pooling, global pooling, max pooling, average pooling, etc. Pooling is a form of nonlinear downsampling. Local pooling combines a small number of nodes in the feature map (e.g., a local cluster of nodes, 2x2 nodes, etc.). Global pooling can, for example, combine all the nodes in the feature map.

[0032] The pooling layer can reduce the size of the representation, and thus the number of parameters, memory footprint, and computational complexity in the CNN. In one example, the pooling layer is inserted between successive convolutional layers in the CNN. In one example, the pooling layer is followed by an activation function, such as a rectified linear unit (ReLU) layer. In one example, the pooling layer is omitted between successive convolutional layers in the CNN.

[0033] The normalization layer can be a ReLU, a leaky ReLU, a generalized partitioned normalization (GDN), an inverse GDN (IGDN), etc. A ReLU can apply a non-saturating activation function to remove negative values ​​from an input signal such as a feature map by setting negative values ​​to 0. A leaky ReLU can have a small slope (e.g., 0.01) for negative values ​​instead of a flat slope (e.g., 0). Thus, if the value x is greater than 0, the output from the leaky ReLU is x. Otherwise, the output from the leaky ReLU is the value x multiplied by the small slope (e.g., 0.01). In one example, the slope is determined before training and is thus not learned during training.

[0034] The NIC framework can support a compression model for image compression. The NIC framework receives an input image x and generates a decompressed image corresponding to the input image x.

number

number

number

number

number

number

number

[0035] In some examples, the NIC framework can use a variational autoencoder (VAE) structure. In a VAE structure, the entire input image x can be fed into a neural network encoder. The entire input image x acts as a black box to generate the compressed representation

number

number

number

number

number

number

number

number

number

number

[0036] Neural networks (e.g., ANNs) can learn to perform tasks from examples without task-specific programming. ANNs can be composed of connected nodes or artificial neurons. The connections between the nodes can send a signal from a first node to a second node (e.g., a receiving node), and the signal can be modified by a weight, which can be denoted by a weight coefficient of the connection. The receiving node can process the signal(s) from the node(s) that transmits the signal(s) to the receiving node (i.e., the input signal(s) intended for the receiving node) and then generate an output signal by applying a function to the input signal. The function can be a linear function. In one example, the output signal is a weighted sum of the input signal(s). In one example, the output signal is further modified by a bias, which can be denoted by a bias term, such that the output signal is a sum of the bias and the weighted sum of the input signal(s). The function can include, for example, a nonlinear operation on the weighted sum or sum of the bias and the weighted sum of the input signal(s). The output signal can be sent to the node(s) (the downstream node(s) connected to the receiving node). The ANN can be represented or configured by parameters (e.g., weights and / or biases of the connections). The weights and / or biases can be obtained by training (e.g., offline training, online training, etc.) the ANN with examples, where the weights and / or biases can be iteratively adjusted. The trained ANN configured with the determined weights and / or determined biases can be used to perform a task.

[0037] 1 illustrates a NIC framework (100) (e.g., a NIC system) in some examples. The NIC framework (100) can be based on neural networks such as DNNs and / or CNNs. The NIC framework (100) can be used to compress (e.g., encode) images and to decompress (e.g., decode or restore) compressed images (e.g., encoded images).

[0038] Specifically, in the example of Figure 1, the compressed model in the NIC framework (100) includes two levels, called the compressed model main level and the compressed model hyper level. The compressed model main level and the compressed model hyper level can be implemented using neural networks. In Figure 1, the compressed model main level neural network is shown as a first sub-NN (151), and the compressed model hyper level is shown as a second sub-NN (152).

[0039] The first sub-NN (151) can resemble an autoencoder, generating a compressed image of the input image x.

number

number

number

[0040] Referring to Figure 1, the main encoder network (111) can generate a latent image or latent representation y from an input image x (e.g., an image to be compressed or encoded). In one example, the main encoder network (111) is implemented using a CNN. The relationship between the latent representation y and the input image x can be described using Equation 2 as follows: y=f1(x;θ1) Equation 2 In the equation, the parameter θ1 represents parameters such as weights and biases (if biases are used in the main encoder network (111)) used in the convolution kernels in the main encoder network (111).

[0041] The latent representation y is the quantized latent image

number

number

number

number

[0042] The coded image (131) may be decompressed (e.g., entropy decoded) by an entropy decoder (114) to generate an output. The entropy decoder (114) may use an entropy coding technique, such as Huffman coding, arithmetic coding, or the like, that corresponds to the entropy coding technique used by the entropy encoder (113). In one example, the entropy decoder (114) uses arithmetic decoding and is an arithmetic decoder. In one example, lossless compression is used by the entropy encoder (113) and lossless decompression is used by the entropy decoder (114), where noise from, e.g., the transmission of the coded image (131) may be omitted, and the output from the entropy decoder (114) may be a quantized latent image.

number

[0043] The main decoder network (115) processes the quantized latent image

number

number

number

number

number

number

number

[0044] In some examples, the second sub-NN (152) may generate a quantized latent image used for entropy coding.

number

[0045] In some examples, the second sub-NN (152) can include a context model NN (116), an entropy parameter NN (117), a hyper-encoder network (121), a quantizer (122), an entropy encoder (123), an entropy decoder (124), and a hyper-decoder network (125). The entropy model used in the context model NN (116) can be a latent image (e.g., a quantized latent image

number

[0046] Referring to FIG. 1, at the encoder side, the quantized latent image from the quantizer (112) is

number

number

number

number

number

number

number

[0047] Output from the context model NN (116) cm,i and the output from the hyperdecoder network (125) hc will output ep The entropy parameter NN (117) can be implemented using a neural network such as a CNN. ep and the input of the entropy parameter NN(117) (e.g., o cm,i Yao hc ) can be written using Equation 5 as follows: o ep =f4(o cm,i ,o hc ;θ4) Equation 5 where the parameter θ4 represents parameters such as weights and biases (if biases are used in the entropy parameter NN (117)) used in the convolution kernels in the entropy parameter NN (117). The output o epThus, the conditional entropy model can be used, for example, to model the output o from the hyperdecoder network (125). hc In one example, the output o ep includes parameters such as mean and scale parameters that are used to condition an entropy model (e.g., a GMM). Referring to Figure 1, the entropy model (e.g., a conditional entropy model) can be used by the entropy encoder (113) and the entropy decoder (114) in entropy coding and entropy decoding, respectively.

[0048] The second sub-NN (152) can be described as follows: The latent image y can be fed to the hyper-encoder network (121) to generate a hyper-latent image z. In one example, the hyper-encoder network (121) is implemented using a neural network such as a CNN. The relationship between the hyper-latent image z and the latent image y can be described using Equation 6 as follows: z=f5(y;θ5) Equation 6 In the equation, the parameter θ5 represents parameters such as weights and biases (if biases are used in the hyperencoder network (121)) used in the convolution kernels in the hyperencoder network (121).

[0049] The hyper latent image z is a quantized latent image

number

number

[0050] The side information, such as the coded bits (132), can be decompressed (e.g., entropy decoded) by an entropy decoder (124) to generate an output. The entropy decoder (124) can use entropy coding techniques, such as Huffman coding, arithmetic coding, etc. In one example, the entropy decoder (124) uses arithmetic decoding and is an arithmetic decoder. In one example, lossless compression is used in the entropy encoder (123) and lossless decompression is used in the entropy decoder (124), where noise, such as that due to the transmission of side information, can be omitted, and the output from the entropy decoder (124) is a quantized latent image.

number

number

number

number

[0051] As mentioned above, the compressed or encoded bits (132) can be added to the coded bitstream as side information, allowing the entropy decoder (114) to use a conditional entropy model. Thus, the entropy model can be image-dependent and spatially adaptive, and therefore more accurate than a fixed entropy model.

[0052] The NIC framework (100) may be suitably adapted, for example, to omit one or more components shown in Figure 1, to modify one or more components shown in Figure 1, and / or to include one or more components not shown in Figure 1. In one example, the NIC framework using a fixed entropy model includes a first sub-NN (151) and does not include a second sub-NN (152). In one example, the NIC framework includes the components in the NIC framework (100) except for the entropy encoder (123) and the entropy decoder (124).

[0053] In one embodiment, one or more components in the NIC framework (100) shown in Figure 1 are implemented using neural network(s), such as CNN(s). Each NN-based component (e.g., the main encoder network (111), the main decoder network (115), the context model NN (116), the entropy parameter NN (117), the hyper-encoder network (121), or the hyper-decoder network (125)) in the NIC framework (e.g., the NIC framework (100)) may include any suitable architecture (e.g., have any suitable combination of layers), include any suitable type of parameters (e.g., weights, biases, combinations of weights and biases, etc.), and include any suitable number of parameters.

[0054] In one embodiment, the main encoder network (111), the main decoder network (115), the context model NN (116), the entropy parameter NN (117), the hyper-encoder network (121), and the hyper-decoder network (125) are implemented using respective CNNs.

[0055] 2 illustrates an exemplary CNN for the main encoder network (111) according to one embodiment of the present disclosure. For example, the main encoder network (111) includes four sets of layers, each set of layers including a convolutional layer 5x5 c192 s2 followed by a GDN layer. One or more layers shown in FIG. 2 may be modified and / or omitted. Additional layer(s) may be added to the main encoder network (111).

[0056] FIG. 3 illustrates an exemplary CNN of the main decoder network (115) according to one embodiment of the present disclosure. For example, the main decoder network (115) includes three sets of layers, each set of layers including a deconvolutional layer 5×5 c192 s2 followed by an IGDN layer. In addition, the three sets of layers are followed by a deconvolutional layer 5×5 c3 s2 followed by an IGDN layer. One or more layers shown in FIG. 3 may be modified and / or omitted. Additional layer(s) may be added to the main decoder network (115).

[0057] 4 illustrates an exemplary CNN of a hyperencoder network (121) according to one embodiment of the present disclosure. For example, the hyperencoder network (121) includes a convolutional layer 3×3 c192 s1 followed by a leaky ReLU, a convolutional layer 5×5 c192 s2 followed by a leaky ReLU, and a convolutional layer 5×5 c192 s2. One or more layers shown in FIG. 4 may be modified and / or omitted. Additional layer(s) may be added to the hyperencoder network (121).

[0058] FIG. 5 illustrates an exemplary CNN of a hyper-decoder network (125) according to one embodiment of the present disclosure. For example, the hyper-decoder network (125) includes a deconvolution layer 5×5 c192 s2 followed by a leaky ReLU, a deconvolution layer 5×5 c288 s2 followed by a leaky ReLU, and a deconvolution layer 3×3 c384 s1. One or more layers shown in FIG. 5 may be modified and / or omitted. Additional layer(s) may be added to the hyper-decoder network (125).

[0059] 6 illustrates an exemplary CNN of the context model NN (116) according to one embodiment of the present disclosure. For example, the context model NN (116) includes a masked convolution 5×5 c384 s1 for context prediction, and thus the context

number

[0060] 7 illustrates an exemplary CNN of an entropy parameter NN (117) according to one embodiment of the present disclosure. For example, the entropy parameter NN (117) includes a convolutional layer 1×1 c640 s1 followed by a leaky ReLU, a convolutional layer 1×1 c512 s1 followed by a leaky ReLU, and a convolutional layer 1×1 c384 s1. One or more layers illustrated in FIG. 7 may be modified and / or omitted. Additional layer(s) may be added to the entropy parameter NN (117).

[0061] The NIC framework (100) may be implemented using a CNN, as described with reference to Figures 2-7. The NIC framework (100) may be suitably adapted such that one or more components within the NIC framework (100) (e.g., (111), (115), (116), (117), (121), and / or (125)) are implemented using any suitable type of neural network (e.g., a CNN or a non-CNN based neural network). One or more other components of the NIC framework (100) may be implemented using a neural network(s).

[0062] The NIC framework (100) including a neural network (e.g., a CNN) can be trained to learn parameters used in the neural network. For example, when a CNN is used, the weights and biases used in the convolution kernels in the main encoder network (111) (if biases are used in the main encoder network (111)), the weights and biases used in the convolution kernels in the main decoder network (115) (if biases are used in the main decoder network (115)), the weights and biases used in the convolution kernels in the hyper encoder network (121) (if biases are used in the hyper encoder network (121)), the weights and biases used in the convolution kernels in the hyper decoder network (125) (if biases are used in the hyper encoder network (125 ... decoder network (125)), the weights and biases used in the convolution kernels in the main encoder network (111) (if biases are used in the main decoder network (115)), the weights and biases used in the convolution kernels in the main decoder network (115) (if biases are used in the main decoder network (115)), the weights and biases used in the convolution kernels in the hyper decoder network (125) (if biases are used in the hyper encoder network (125)), the weights and biases used in the convolution kernels in the main encoder network (111) (if The parameters represented by θ1 through θ6, such as the weights and biases (if biases are used in the hyperdecoder network (125)), the weights and biases used in the convolution kernel(s) in the context model NN (116) (if biases are used in the context model NN (116)), and the weights and biases used in the convolution kernel(s) in the entropy parameter NN (117) (if biases are used in the entropy parameter NN (117)), can each be learned in a training process (e.g., an offline training process, an online training process, etc.).

[0063] In one example, referring to FIG. 2, the main encoder network (111) includes four convolution layers, each with a 5×5 convolution kernel and 192 channels. Thus, the number of weights used in the convolution kernels in the main encoder network (111) is 19200 (i.e., 4×5×5×192). The parameters used in the main encoder network (111) include the 19200 weights and optional biases. When biases and / or additional NN(s) are used in the main encoder network (111), additional parameter(s) may be included.

[0064] Referring to FIG. 1, the NIC framework (100) includes at least one component or module built on a neural network (one or more). The at least one component may include one or more of a main encoder network (111), a main decoder network (115), a hyper encoder network (121), a hyper decoder network (125), a context model NN (116), and an entropy parameter NN (117). The at least one component may be trained individually. In one example, the training process is used to learn parameters for each component separately. The at least one component may be trained together as a group. In one example, the training process is used to learn parameters for a subset of the at least one component together. In one example, the training process is used to learn all parameters of the at least one component, thus referred to as E2E optimization.

[0065] In the training process for one or more components in the NIC framework (100), the weights (or weight coefficients) of one or more components can be initialized. In one example, the weights are initialized based on the corresponding pre-trained neural network model(s) (e.g., DNN model, CNN model). In one example, the weights are initialized by setting the weights to random numbers.

[0066] For example, after the weights are initialized, a set of training images can be used to train one or more components. The set of training images can include any suitable images having any suitable size. In some examples, the set of training images includes images from raw images, natural images, computer-generated images, etc. in the spatial domain. In some examples, the set of training images includes images from residual images or residual images with residual data in the spatial domain. The residual data can be calculated by a residual calculator. In some examples, the raw images and / or residual images with residual data can be directly used to train a neural network in a NIC framework, such as the NIC framework (100). Thus, the raw images, the residual images, images from the raw images, and / or images from the residual images can be used to train a neural network in the NIC framework.

[0067] For simplicity, the following training process (e.g., offline training process, online training process, etc.) is described using a training image as an example. The description can be adapted to training blocks as appropriate. A training image t of a set of training images can be passed through the encoding process of FIG. 1 to generate a compressed representation (e.g., encoded information into a bitstream, for example). The encoded information is passed through the decoding process described in FIG. 1 to generate a decompressed image

number

[0068] In the NIC framework (100), two competing goals are balanced: reconstruction quality and bit consumption. A quality loss function (e.g., distortion or distortion loss)

number

number

[0069] In the case of neural image compression, a differentiable approximation of quantization can be used in the E2E optimization. In various examples, in the training process of neural network-based image compression, noise injection is used to simulate quantization, so that quantization is simulated by noise injection instead of being performed by a quantizer (e.g., quantizer (112)). Thus, training with noise injection can variationally approximate the quantization error. A bits-per-pixel (BPP) estimator can be used to simulate an entropy coder, so that entropy coding is simulated by the BPP estimator instead of being performed by an entropy encoder (e.g., (113)) and an entropy decoder (e.g., (114)). Thus, the rate loss R in the loss function L shown in Equation 1 during the training process can be estimated, for example, based on noise injection and the BPP estimator. In general, a higher rate R can enable lower distortion D, and a lower rate R can result in higher distortion D. Thus, the trade-off hyperparameter λ in Equation 1 can be used to optimize the joint RD loss L, optimizing L as the sum of λ and R. The training process can be used to tune parameters of one or more components (e.g., (111)(115)) in the NIC framework (100) such that the joint RD loss L is minimized or optimized. In some examples, the trade-off hyperparameter λ can be

number

[0070] Various models can be used to determine the distortion loss D and the rate loss R, and thus the joint RD loss L in Equation 1. In one example, the distortion loss

number

[0071] In one example, the goal of the training process is to train an encoding neural network (e.g., encoding DNN) such as a video encoder to be used on the encoder side and a decoding neural network (e.g., decoding DNN) such as a video decoder to be used on the decoder side. In one example, referring to FIG. 1, the encoding neural network may include a main encoder network (111), a hyper encoder network (121), a hyper decoder network (125), a context model NN (116), and an entropy parameter NN (117). The decoding neural network may include a main decoder network (115), a hyper decoder network (125), a context model NN (116), and an entropy parameter NN (117). The video encoder and / or the video decoder may include other components based on NN(s) and / or not based on NN(s).

[0072] The NIC framework (e.g., the NIC framework (100)) can be trained in an E2E manner. In one example, the encoding neural network and the decoding neural network are updated together in a training process based on backpropagation gradients in an E2E manner, for example, using a gradient descent algorithm. The gradient descent algorithm can iteratively optimize the parameters of the NIC framework to find a local minimum of a differentiable function of the NIC framework (e.g., a local minimum of the rate-distortion loss). For example, the gradient descent algorithm can take iterative steps in the opposite direction of the gradient (or approximate gradient) of the differentiable function at the current point.

[0073] After the parameters of the neural network in the NIC framework (100) are trained, one or more components in the NIC framework (100) can be used to encode and / or decode an image. In one embodiment, on the encoder side, an image encoder is configured to encode an input image x into an encoded image (131) that is transmitted in a bitstream. The image encoder can include multiple components in the NIC framework (100). In one embodiment, on the decoder side, a corresponding image decoder converts the encoded image (131) carried in the bitstream into a reconstructed image (131).

number

[0074] It should be noted that the image encoder and decoder according to the NIC framework may have corresponding structures.

[0075] 8 shows an exemplary image encoder (800) according to one embodiment of the present disclosure. The image encoder (800) includes a main encoder network (811), a quantizer (812), an entropy encoder (813), and a second sub-NN (852). The main encoder network (811) is configured similarly to the main encoder network (111), the quantizer (812) is configured similarly to the quantizer (112), the entropy encoder (813) is configured similarly to the entropy encoder (113), and the second sub-NN (852) is configured similarly to the second sub-NN (152). Descriptions are provided above with reference to FIG. 1 and are omitted here for clarity.

[0076] 9 illustrates an exemplary image decoder (900) according to one embodiment of this disclosure. The image decoder (900) may correspond to the image encoder (800). The image decoder (900) may include a main decoder network (915), an entropy decoder (914), a context model NN (916), an entropy parameter NN (917), an entropy decoder (924), and a hyper decoder network (925). The main decoder network (915) is configured similarly to the main decoder network (115), the entropy decoder (914) is configured similarly to the entropy decoder (114), the context model NN (916) is configured similarly to the context model NN (116), the entropy parameter NN (917) is configured similarly to the entropy parameter NN (117), the entropy decoder (924) is configured similarly to the entropy decoder (124), and the hyper decoder network (925) is configured similarly to the hyper decoder network (125), a description of which is provided above with reference to FIG.

[0077] 8-9, on the encoder side, an image encoder (800) can generate an encoded image (831) and encoded bits (832) that are transmitted in a bitstream. On the decoder side, an image decoder (900) can receive and decode the encoded image (931) and encoded bits (932). The encoded image (931) and encoded bits (932) can be parsed from the received bitstream.

[0078] 10-11 respectively show an exemplary image encoder (1000) and corresponding image decoder (1100) according to an embodiment of the present disclosure. Referring to FIG. 10, the image encoder (1000) includes a main encoder network (1011), a quantizer (1012), and an entropy encoder (1013). The main encoder network (1011) is configured similarly to the main encoder network (111), the quantizer (1012) is configured similarly to the quantizer (112), and the entropy encoder (1013) is configured similarly to the entropy encoder (113). Descriptions are provided above with reference to FIG. 1 and are omitted here for clarity.

[0079] Referring to Figure 11, the image decoder (1100) includes a main decoder network (1115) and an entropy decoder (1114). The main decoder network (1115) is configured similarly to the main decoder network (115), and the entropy decoder (1114) is configured similarly to the entropy decoder (114). Descriptions are provided above with reference to Figure 1 and are omitted here for clarity.

[0080] 10 and 11, an image encoder (1000) can generate an encoded image (1031) to be included in a bitstream. An image decoder (1100) can receive the bitstream and decode the encoded image (1131) carried in the bitstream.

[0081] According to one aspect of the present disclosure, in a NN-based image compression method, such as a DNN-based or CNN-based image compression method, instead of directly encoding the entire image, a block-based or block-wise coding mechanism may be effective to compress the image. The entire image may be divided into blocks of equal or different sizes, and the blocks may be compressed separately. In one embodiment, the image may be divided into blocks of equal or unequal size. The divided blocks may also be compressed instead of the image.

[0082] Figure 12 shows an example of block-based image coding. An image (1280) may be divided into blocks, for example, blocks (1281)-(1296). Blocks (1281)-(1296) may be compressed, for example, according to scan order. In the example shown in Figure 12, blocks (1281)-(1289) have already been compressed, and blocks (1290)-(1296) are to be compressed.

[0083] In one embodiment, an image is treated as blocks, where a block is the entire image, and the image is compressed without being divided into blocks. The entire image can be the input of the E2E NIC framework.

[0084] Further, some aspects of the present disclosure provide techniques for online training-based image compression with neural networks, such as artificial intelligence (AI)-based neural image compression (NIC). In some examples, the techniques for online training-based image compression are applicable to a compression model of an end-to-end (E2E) optimization framework. The E2E optimization framework includes an encoding portion and a decoding portion. The encoding portion and the decoding portion may have overlapping portions (e.g., the same neural network, the same neural network layer). In some examples, the encoding portion includes one or more pre-trained neural networks (referred to as one or more first pre-trained neural networks) that can encode one or more images into a bitstream. The decoding portion includes one or more pre-trained neural networks (referred to as one or more second pre-trained neural networks) that can decode the bitstream to generate one or more restored images. In some examples, certain pre-trained neural networks in the one or more first pre-trained neural networks are also present in the one or more second pre-trained neural networks. According to some aspects of the present disclosure, during the online training process, the decoding part is fixed, and the modules of only the encoding part can be adjusted based on one or more input images to optimize the rate-distortion performance. For example, parameters in only the encoding part (not in the decoding part) of the E2E optimization framework can be adjusted based on one or more input images to determine updated parameters that can optimize the rate-distortion performance. Then, the encoding part (also called an optimized encoder) with the updated parameters can encode one or more input images to generate a bitstream. The updated parameters are encoder-only parameters and do not need to be provided to the decoder side, thus improving the coding efficiency.

[0085] According to one aspect of the present disclosure, for each input image (also called a target image) to be compressed, an online training process is applied to find an optimized encoder for the target image, and then the target image is compressed by the optimized encoder instead of the original encoder. By using the optimized encoder, the NIC can achieve better compression performance. In some examples, the online training-based encoder tuning is used as a pre-processing step (e.g., before the formal compression of each input image) to improve the compression performance of the E2E NIC compression. In one example, the online training-based encoder tuning can be performed on a pre-trained compression model, such as a pre-trained NIC framework. According to one aspect of the present disclosure, the pre-trained compression model itself, such as the structure of the pre-trained NIC framework, does not require any training or fine-tuning. The online training-based encoder tuning does not require any additional training data other than the target image.

[0086] As mentioned above, training-based image compression can be viewed as a two-step mapping process, including a first step of encoding the mapping and a second step of decoding the mapping. In the first step, an original image x0 (e.g., a target image) in a high-dimensional space (e.g., a two-dimensional image, a three-dimensional image, a two-dimensional image with three color channels, etc.) is mapped to a bitstream of length R(x0). In the second step, the bitstream is then converted into a reconstructed image

number

number

[0087] According to one aspect of the present disclosure, when an optimized encoder is present, the optimized NIC framework (having the optimized encoder) may decompose the original image x0 (first restored image x1) into a new image x2 (second restored image x3) according to a distance measure or loss function (e.g., having a smaller loss function).

number

number

[0088] According to some aspects of the present disclosure, online training-based encoder tuning may be performed at any suitable intermediate step of the neural network on the encoder side to reduce the difference between the decoded image and the original image.

[0089] According to one aspect of the present disclosure, in the offline training process (also referred to as the model training phase), a gradient descent algorithm is used to determine the parameters of the entire compression model. In some examples, in the online training-based encoder tuning process, the decoder part of the compression model is fixed, and the encoder part of the compression model is updated using a gradient descent algorithm. It should be noted that the entire compression model can be made differentiable by replacing the non-differentiable part with a differentiable part (e.g., replacing quantization with noise injection), and thus the gradient descent algorithm can be used in the online training-based encoder tuning process to iteratively optimize the encoder part.

[0090] It should be noted that the online training-based encoder tuning process can use a first hyperparameter-step size and a second hyperparameter-step number. The step size indicates the "learning rate" of the online training-based encoder tuning process. In some embodiments, different step sizes are used during the online training-based encoder tuning process for images with different types of content to achieve the best optimization results. The step number indicates the number of updates in the online training-based encoder tuning process. The hyperparameters are used in the online training-based encoder tuning process with a loss function. In one example, the step size is used in a gradient descent algorithm or a backpropagation calculation performed in the online training-based encoder tuning process, and the iteration number can be used as a maximum iteration number threshold to control the termination of the learning process.

[0091] According to some aspects of the present disclosure, for each input image x0, three operations, such as a first operation of an online training-based encoder tuning operation, a second operation of encoding, and a third operation of decoding, may be performed according to the NIC framework. In some examples, the first operation and the second operation may be performed in an electronic device according to the NIC framework, and the third operation may be performed by the same electronic device or a different electronic device according to the NIC framework.

[0092] 13A and 13B illustrate an electronic device (1300) configured to perform online training-based encoder tuning and encoding operations on an input image x0 according to some aspects of the present disclosure. The electronic device (2100) may be any suitable device, such as a server computer, a desktop computer, a laptop computer, etc.

[0093] FIG. 13A shows a diagram of components in an electronic device (1300) for performing an online training-based encoder tuning operation. The electronic device (1300) includes components forming a NIC framework (1301) (also referred to as a compressed model) including two levels, such as a main level of the compressed model shown as a first sub-NN (1351) and a hyper level of the compressed model shown as a second sub-NN (1352). In FIG. 1, the first sub-NN (1351) is configured similarly to the first sub-NN (151), and the second sub-NN (1352) is configured similarly to the second sub-NN (152). It should be noted that the NIC framework of FIG. 13A is an example for illustrating the technique for online training-based encoder tuning, and the technique can be used in other suitable NIC frameworks, such as the NIC framework of FIG. 1, the NIC framework of FIG. 10-FIG. 11, etc.

[0094] The first sub-NN (1351) includes a main encoder network (1311), a quantizer (1312), an entropy encoder (1313), an entropy decoder (1314), and a main decoder network (1315). The main encoder network (1311) is configured similarly to the main encoder network (111), the quantizer (1312) is configured similarly to the quantizer (112), the entropy encoder (1313) is configured similarly to the entropy encoder (113), the entropy decoder (1314) is configured similarly to the entropy decoder (114), and the main decoder network (1315) is configured similarly to the main decoder network (115). Descriptions are provided above with reference to FIG. 1 and are omitted here for clarity.

[0095] The second sub-NN (1352) may include a hyper-encoder network (1321), a quantizer (1322), an entropy encoder (1323), an entropy decoder (1324), and a hyper-decoder network (1325). The hyper-encoder network (1321) is configured similarly to the hyper-encoder network (121), the quantizer (1322) is configured similarly to the quantizer (122), the entropy encoder (1323) is configured similarly to the entropy encoder (123), the entropy decoder (1324) is configured similarly to the entropy decoder (124), and the hyper-decoder network (1325) is configured similarly to the hyper-decoder network (125). Descriptions are provided above with reference to FIG. 1 and are omitted here for clarity.

[0096] In some examples, initially, the parameters in the neural network of the NIC framework (1301) are pre-trained parameters. During the online training-based encoder tuning operation, in some examples, for an input image x0, the main encoder network (1311) generates a latent representation y0 from the input image x0. The latent representation y0 is a quantized latent image

number

number

number

number

[0097] The coded image (1331) is decompressed (e.g., entropy decoded) by the entropy decoder (1314) to produce a quantized latent image.

number

number

number

number

[0098] The latent representation y0 can be fed to a hyperencoder network (1321) to generate a hyperlatent image z0. The hyperlatent image z0 can be expressed as a quantized latent image

number

number

[0099] Side information such as the coding bits (1332) are stored in the quantized latent image.

number

number

[0100] In some examples, a performance metric such as rate-distortion loss can be calculated, for example, according to Equation 1. Furthermore, the encoder-only parameters within the NIC framework can be trained. In one example, the encoder-only parameters are updated in a training process (online training-based encoder tuning process) based on back-propagated gradients in an end-to-end manner, for example, using a gradient descent algorithm. The gradient descent algorithm can iteratively optimize the encoder-only parameters to find a local minimum of a differentiable function (e.g., a local minimum of the rate-distortion loss). For example, the gradient descent algorithm can take iterative steps in the opposite direction of the gradient (or approximate gradient) of the differentiable function at the current point.

[0101] In some examples, the corresponding decoder may have an entropy decoder corresponding to the entropy decoder (1314) and the entropy decoder (1324), a main decoder network corresponding to the main decoder network (1315), and a hyper decoder network corresponding to the hyper decoder network (1325). Thus, the encoder-only portion includes the main encoder network (1311), the quantizer (1312), the entropy encoder (1313), the hyper encoder network (1321), the quantizer (1322), and the entropy encoder (1323).

[0102] In some examples, parameters in the neural networks of the main encoder network (1311) and the hyper-encoder network (1321) are adjusted during an online training-based encoder tuning operation to determine updated parameters to achieve a minimum of the rate-distortion loss for the input image x0.

[0103] FIG. 13B illustrates a diagram of a neural network-based image encoder (1302) in an electronic device (1300) for performing an encoding operation of an input image x0 according to some aspects of the present disclosure. The neural network-based image encoder (1302) is formed according to a NIC framework (1301) with updated parameters from an online training-based encoder tuning operation. The neural network-based image encoder (1302) includes a main encoder network (1311), a quantizer (1312), an entropy encoder (1313), a hyper encoder network (1321), a quantizer (1322), an entropy encoder (1323), an entropy decoder (1324), and a hyper decoder network (1325). In some examples, one or more parameters of the main encoder network (1311) and / or the hyper encoder network (1321) are updated parameters according to the online training-based encoder tuning operation.

[0104] During the encoding operation, in some examples, for an input image x0, the main encoder network (1311) generates a latent representation y0' from the input image x0. The latent representation y0' is a quantized latent image

number

number

number

number

[0105] The latent representation y0′ can be fed to a hyper-encoder network (1321) to generate a hyper-latent image z0′. The hyper-latent image z0′ can be expressed as a quantized latent image

number

number

[0106] Side information such as the coding bits (1332) are stored in the quantized latent image.

number

number

[0107] In one example, a compressed image (e.g., an encoded image)

number

[0108] 14 illustrates a diagram of components in an electronic device (1400) for performing a decoding operation of an input image x0 according to some aspects of the present disclosure. The electronic device (1400) may be any suitable device, such as a server computer, a desktop computer, a laptop computer, etc. In one example, the electronic device (1400) is the electronic device (1300). In another example, the electronic device (1400) is a different device than the electronic device (1300).

[0109] The electronic device (1400) includes a neural network-based image decoder (1403) that includes an entropy decoder (1414), a main decoder network (1415), an entropy decoder (1424), and a hyperdecoder network (1425). The entropy decoder (1414) may correspond to the entropy decoder (1314) (e.g., with the same structure and the same parameters) and is configured similarly to the entropy decoder (114), the main decoder network (1415) may correspond to the main decoder network (1315) (e.g., with the same structure and the same parameters) and is configured similarly to the main decoder network (115), the entropy decoder (1424) may correspond to the entropy decoder (1324) (e.g., with the same structure and the same parameters) and is configured similarly to the entropy decoder (124), and the hyper decoder network (1425) may correspond to the hyper decoder network (1325) (e.g., with the same structure and the same parameters) and is configured similarly to the hyper decoder network (125), a description of which has been provided above with reference to FIG. 1 and will be omitted here for clarity.

[0110] It should be noted that in some examples, the parameters in the neural network of the neural network-based image decoder (1403) are pre-trained parameters.

[0111] During the decoding operation, in some instances, a compressed representation of the input image x0

number

number

number

number

[0112] The coded bits (1432) are the quantized latent image

number

number

[0113] It should be noted that the online training-based encoder tuning operation makes changes on the encoder side and does not require any changes in the decoder-related operations.

[0114] In some embodiments, during an online training-based encoder tuning operation, all parameters in the main encoder network (1311) and the hyper-encoder network (1321) are tuned and optimized.

[0115] In some embodiments, only a portion of the parameters in the main encoder network (1311) and / or the hyper-encoder network (1321) are tuned and optimized. In some examples, parameters of several layers of the main encoder network (1311) and / or the hyper-encoder network (1321) are tuned. In some examples, parameters of one or more channels in a layer of the main encoder network (1311) and / or the hyper-encoder network (1321) are tuned.

[0116] In some examples, the input image is first divided into blocks for compression by blocks. The step size for each block may be different. In one example, different step sizes may be assigned to blocks of an image to achieve better compression results. In examples where an image is compressed without being divided into blocks, different images may have different step sizes to achieve optimized compression results.

[0117] Note that the updates from online training involve changing parameters of only the encoding part, while the parameters of the decoding part are fixed. Thus, in some examples, the encoded images can be decoded by the same image decoder with pre-trained parameters from offline training. Online training leverages optimized encoder mechanisms to improve NIC coding efficiency and can be flexible, and the general framework can accommodate various types of quality metrics.

[0118] FIG. 15 shows a flow chart outlining a process (1500) according to one embodiment of the present disclosure. The process (1500) is an encoding process including online training-based encoder tuning in a NIC framework. The process (1500) may be executed in an electronic device, such as the electronic device (1300) in one example. In some embodiments, the process (1500) is implemented in software instructions, and thus the processing circuit executes the process (1500) when the processing circuit executes the software instructions. The process starts at (S1501) and proceeds to (S1510).

[0119] At (S1510), online training of a neural image compression (NIC) framework is performed based on one or more input images. The NIC framework is an end-to-end framework that includes one or more first neural networks in an encoding part and one or more second neural networks in a decoding part. The online training determines updates (e.g., multiple update values) for one or more tunable parameters in one or more first neural networks with one or more second neural networks having fixed parameters (e.g., one or more second neural networks having non-tunable values). The updates can, for example, achieve a local minimum of rate-distortion loss.

[0120] At (S1520), one or more tunable parameters in the one or more first neural networks are updated according to the update.

[0121] At (S1530), an encoding portion of the NIC framework having one or more tunable parameters in the one or more first neural networks updated encodes one or more input images into a bitstream.

[0122] In some embodiments, the decoding portion is fixed with pre-trained parameters, for example, the fixed parameters of the one or more second neural networks are fixed with pre-trained values ​​from offline training of the NIC framework.

[0123] In some embodiments, the NIC framework includes a specific neural network in both the encoding portion and the decoding portion, the specific neural network including a first parameter that is fixed during online training. In one example, the specific neural network includes a hyperdecoder network.

[0124] In some examples, online training may be performed by making parameters within the main encoder network and / or the hyper-encoder network of the NIC framework tunable.

[0125] In some examples, online training may be performed by making a subset of parameters within the main encoder network and / or the hyper-encoder network of the NIC framework adjustable.

[0126] In some examples, online training may be performed by making the parameters of layers within the main encoder network or the hyper-encoder network of the NIC framework tunable.

[0127] In some examples, online training may be performed by allowing tuning of parameters of channels within the main encoder network or layers of the hyper-encoder network of the NIC framework.

[0128] In some examples, the input image can be divided into multiple blocks, and a step size can be assigned to each of the multiple blocks. Online training of the NIC framework can be performed according to the multiple blocks having respective step sizes.

[0129] In some examples, the step size is assigned to the input image based on the type of content in the input image. Online training of the NIC framework is performed according to the step size input image.

[0130] Next, the process (1500) proceeds to (S1599) and ends.

[0131] The process (1500) can be adapted appropriately to various scenarios, and the steps in the process (1500) can be adjusted accordingly. One or more of the steps in the process (1500) can be adapted, omitted, repeated, and / or combined. Any suitable order can be used to perform the process (1500). Additional step(s) can be added.

[0132] The techniques described above can be implemented as computer software using computer readable instructions and physically stored on one or more computer readable media. For example, Figure 16 illustrates a computer system (1600) suitable for implementing certain embodiments of the disclosed subject matter.

[0133] Computer software may be coded using any suitable machine code or computer language that can undergo mechanisms such as assembly, compilation, linking, etc. to produce code containing instructions that can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or via interpretation, microcode execution, etc.

[0134] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0135] 16 for computer system (1600) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of computer system (1600).

[0136] The computer system (1600) may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users, for example, via tactile input (such as keystrokes, swipes, data glove movements), audio input (such as voice, clapping), visual input (such as gestures), or olfactory input (not depicted). Human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (such as voice, music, ambient sounds), images (such as scanned images, photographic images obtained from still image cameras), and video (such as two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0137] The input human interface devices may include one or more (only one of each is shown) of a keyboard (1601), a mouse (1602), a trackpad (1603), a touch screen (1610), a data glove (not shown), a joystick (1605), a microphone (1606), a scanner (1607), and a camera (1608).

[0138] The computer system (1600) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the senses of a human user, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (1610), data gloves (not shown), or joystick (1605), although some haptic feedback devices may not function as input devices), audio output devices (speakers (1609), headphones (not shown), etc.), visual output devices (screens (1610), including CRT screens, LCD screens, plasma screens, OLED screens, etc., each with or without touch screen input capability, each with or without haptic feedback capability, some capable of outputting two-dimensional visual output or output in more than three dimensions by means of stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown), and printers (not shown).

[0139] The computer system (1600) may also include human accessible storage devices and their associated media, such as optical media, including CD / DVD ROM / RW (1620) with media (1621) such as CDs / DVDs, thumb drives (1622), removable hard drives or solid state drives (1623), legacy magnetic media such as tapes and floppy disks (not shown), and dedicated ROM / ASIC / PLD based devices (not shown) such as security dongles.

[0140] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0141] The computer system (1600) may also include an interface (1654) to one or more communication networks (1655). The networks may be, for example, wireless, wired, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, and the like. Examples of networks include local area networks such as Ethernet, cellular networks including WLAN, GSM, 3G, 4G, 5G, LTE, and the like, TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial including CANBus, and the like. Certain networks generally require an external network interface adapter attached to a particular general-purpose data port or peripheral bus (1649) (e.g., a USB port of the computer system (1600) or the like). Others are generally integrated into the core of the computer system (1600) by attachment to a system bus (e.g., an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system) as described below. Using any of these networks, computer system (1600) can communicate with other entities. Such communications may be one-way receive-only (e.g., broadcast TV), one-way transmit-only (e.g., CANbus to a CANbus device), or bidirectional, for example, to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used with each of these networks and network interfaces, as described above.

[0142] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to the core (1640) of the computer system (1600).

[0143] The cores (1640) may include one or more central processing units (CPUs) (1641), graphics processing units (GPUs) (1642), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (1643), hardware accelerators for specific tasks (1644), graphics adapters (1650), and the like. These devices may be connected via a system bus (1648), along with read-only memory (ROM) (1645), random access memory (1646), and internal mass storage devices (1647), such as internal hard drives, SSDs, etc. that are not user accessible. In some computer systems, the system bus (1648) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (1648) or via a peripheral bus (1649). In one example, a screen (1610) may be connected to the graphics adapter (1650). Architectures for peripheral buses include PCI, USB, etc.

[0144] The CPU (1641), GPU (1642), FPGA (1643), and accelerator (1644) can execute certain instructions that can combine to constitute the aforementioned computer code. The computer code can be stored in ROM (1645) or RAM (1646). Transient data can also be stored in RAM (1646), while persistent data can be stored, for example, in internal mass storage (1647). The use of cache memory, which can be closely associated with one or more CPUs (1641), GPUs (1642), mass storage (1647), ROM (1645), RAM (1646), etc., can enable fast storage and retrieval in any memory device.

[0145] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the type well known and available to those skilled in the computer software arts.

[0146] By way of example and not limitation, a computer system (1600) having an architecture, and specifically a core (1640), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage devices as introduced above, as well as media associated with a particular storage device of the core (1640) that is non-transitory in nature, such as a core internal mass storage device (1647) or a ROM (1645). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (1640). The computer-readable media can include one or more memory devices or chips, depending on the particular needs. The software can cause the core (1640) and specifically the processors therein (including a CPU, GPU, FPGA, etc.) to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM (1646) and modifying such data structures according to the processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (1644)) that may operate in place of or in conjunction with software to perform certain processes or certain portions of certain processes described herein. Reference to software may encompass logic, and vice versa, where appropriate. Reference to a computer-readable medium may encompass circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0147] While this disclosure has described several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that are within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of this disclosure and are therefore within the spirit and scope of this disclosure. [Explanation of symbols]

[0148] 100 NIC framework, 111 main encoder network, 112 quantizer, 113 entropy encoder, 114 entropy decoder, 115 main decoder network, 116 context model NN, 117 entropy parameter NN, 121 hyperencoder network, 122 quantizer, 123 entropy encoder, 124 entropy decoder, 125 hyperdecoder network, 131 encoded image, 132 encoded bits, 151 first sub-NN, 152 second sub-NN, 800 image encoder, 811 main encoder network, 812 quantizer, 813 entropy encoder, 816 context model NN, 817 entropy parameter NN, 821 hyperencoder network, 822 quantizer, 823 entropy encoder, 824 entropy decoder, 825 hyperdecoder network, 831 encoded image, 832 Encoded bits, 852 Second sub-NN, 900 Image decoder, 914 Entropy decoder, 915 Main decoder network, 916 Context model NN, 917 Entropy parameter NN, 924 Entropy decoder, 925 Hyper decoder network, 931 Encoded image, 932 Encoded bits, 1000 Image encoder, 1011 Main encoder network, 1012 Quantizer, 1013 Entropy encoder, 1031 Encoded image, 1100 Image decoder, 1114 Entropy decoder, 1115 Main decoder network, 1131 Encoded image, 1280 Image, 1281 to 1296 Block, 1300 Electronic device, 1301 NIC framework, 1311 Main encoder network, 1312 Quantizer, 1313 Entropy encoder, 1314 Entropy decoder, 1315 Main decoder network, 1321, hyperencoder network, 1322, quantizer, 1323, entropy encoder, 1324, entropy decoder, 1325, hyperdecoder network, 1331, encoded image, 1332, encoded bits, 1351, first sub-NN, 1352, second sub-NN, 1300, electronic device, 1302Image encoder, 1311, main encoder network, 1312, quantizer, 1313, entropy encoder, 1321, hyperencoder network, 1322, quantizer, 1323, entropy encoder, 1324, entropy decoder, 1325, hyperdecoder network, 1331, encoded image, 1332, encoded bits, 1352, second sub-NN, 1400, electronic device, 1403, image decoder, 1414, entropy decoder, 1415, main decoder network, 1424, entropy decoder, 1425, hyperdecoder network, 1431, encoded image, 1432, encoded bits, 1600, computer system, 1601, keyboard, 1602, mouse, 1603, track pad, 1605, joystick, 1606, microphone, 1607, scanner, 1608, camera, 1609 speaker, 1610 screen, 1620 CD / DVD ROM / RW, 1621 CD / DVD or other medium, 1622 thumb drive, 1623 removable hard drive or solid state drive, 1640 core, 1641 central processing unit (CPU), 1642 graphics processing unit (GPU), 1643 field programmable gate area (FPGA), 1644 accelerator, 1645 read only memory (ROM), 1646 random access memory (RAM), 1647 internal mass storage device, 1648 system bus, 1650 graphics adapter, 1654 network interface, 1655 communication network

Claims

1. 1. A method for image coding by a processing circuit, comprising: performing online training of a neural image compression (NIC) framework based on one or more input images, the NIC framework being an end-to-end framework including both (i) one or more first neural networks in an encoding portion and (ii) one or more second neural networks in a decoding portion, the online training determining a plurality of update values ​​for one or more tunable parameters in the one or more first neural networks, the one or more second neural networks having non-tunable parameters; Dividing an input image into a number of blocks; assigning respective step sizes to the plurality of blocks; and performing the online training of the NIC framework according to the plurality of blocks having the respective step sizes. Steps and updating the one or more tunable parameters in the one or more first neural networks according to the plurality of update values; encoding the one or more input images into a bitstream by the encoding portion of the NIC framework using the one or more tunable parameters in the one or more first neural networks that are updated; A method comprising:

2. 2. The method of claim 1, wherein the non-tunable parameters of the one or more second neural networks are fixed at pre-trained values ​​from offline training of the NIC framework.

3. 2. The method of claim 1, wherein the NIC framework includes a specific neural network for both the encoding portion and the decoding portion, the specific neural network including a first parameter that is fixed during the online training.

4. The method of claim 3 , wherein the particular neural network comprises a hyper-decoder network.

5. said step of performing said online training of said NIC framework further comprising: performing the online training by making each parameter of the main encoder network and the hyper encoder network of the NIC framework adjustable. The method of claim 1, further comprising:

6. said step of performing said online training of said NIC framework further comprising: performing the online training by making a subset of parameters in the main encoder network and the hyper-encoder network of the NIC framework adjustable. The method of claim 1, further comprising:

7. said step of performing said online training of said NIC framework further comprising: performing the online training by making parameters of layers in a main encoder network or a hyper encoder network of the NIC framework adjustable. The method of claim 1, further comprising:

8. said step of performing said online training of said NIC framework further comprising: performing the online training by making adjustable parameters of channels within a main encoder network or a hyper encoder network layer of the NIC framework; The method of claim 1, further comprising:

9. said step of performing said online training of said NIC framework further comprising: assigning a step size to an input image based on a type of content within the input image; performing the online training of the NIC framework according to the input image having the step size; 2. The method of claim 1, comprising:

10. An apparatus for image coding, comprising a processing circuit configured to perform the method according to any one of claims 1 to 9.

11. A computer program product for causing a processing circuit to carry out the method according to any one of claims 1 to 9.

Images