Weight quantization in neural network-based compression schemes.

Abstract

An encoding method is disclosed. First, neural network weights representing an input image are obtained. Then, at least one value representing the maximum absolute value of the weights in a layer of the neural network is obtained. The layer weights are quantized according to the at least one value. Finally, the at least one value and the quantized weights are coded into a bitstream. These coded weights can be provided to a decoder configured to reconstruct the image.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of European Application No. 22306480.9, filed October 4, 2022, which is incorporated herein by reference in its entirety.

[0002] At least one of the present embodiments generally relates to a method and apparatus for encoding (respectively decoding) neural network weights, the weights representing an image. [Background technology]

[0003] Image and video compression is a fundamental task in image processing, which has become important during times of pandemics and increased video streaming. Thanks to significant community efforts over decades, traditional methods have reached the current state-of-the-art rate-distortion performance and dominate current industrial codec solutions. End-to-end trainable deep models have recently emerged as an alternative with promising results. They currently outperform the best traditional compression methods (VVC, generic video coding) even in terms of peak signal-to-noise ratio for single-image compression. Summary of the Invention

[0004] In one embodiment, obtaining weights of a neural network, the weights representing an input image; obtaining at least one value representing the maximum absolute value of weights in a layer of said neural network; quantizing the weights of the layer in response to the at least one value; and encoding the at least one value and the quantized weight into a bitstream.

[0005] An encoding device is disclosed, comprising one or more processors and at least one memory coupled to the one or more processors, the one or more processors configured to perform the method disclosed above.

[0006] In another embodiment, obtaining a bitstream including at least one value representing the maximum absolute value of a weight in a layer of a neural network and the quantized weights of said layer; decoding the at least one value and the quantized weights of the neural network from a bitstream; dequantizing the quantized weights of the layer according to at least one value to obtain dequantized weights; and reconstructing the image using a neural network parameterized by the dequantized weights.

[0007] A decoding device is disclosed, comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to perform the method disclosed above.

[0008] Further embodiments that may be used alone or in combination are described herein.

[0009] One or more embodiments also provide a computer program comprising instructions that, when executed by one or more processors, cause the one or more processors to perform a method for encoding / decoding image or video data according to any of the embodiments described herein. One or more of the present embodiments also provide a non-transitory computer-readable medium and / or a computer-readable storage medium having stored thereon instructions for encoding / decoding image or video data according to the methods described herein.

[0010] One or more embodiments also provide a computer-readable storage medium having stored thereon encoded data, e.g., a bitstream, generated according to the methods described herein. One or more embodiments also provide methods and apparatus for transmitting or receiving encoded data, e.g., a bitstream, generated according to the methods described above. [Brief explanation of the drawings]

[0011] [Figure 1] 1 shows a block diagram of a system in which aspects of the present embodiments may be implemented; [Figure 2] 2 illustrates an example of an end-to-end neural network-based compression system 200 for encoding images using deep neural networks. [Figure 3] 1 illustrates an example of an end-to-end implicit neural network-based compression system for encoding images. [Figure 4] 1 illustrates an example flowchart of a method for encoding according to one embodiment. [Figure 5] 1 illustrates an example of an image decoder according to at least one embodiment. [Figure 6] 1 illustrates an example flowchart of a method for decoding according to one embodiment. [Figure 7] 1 illustrates a method for training a quantization-aware INR according to one embodiment. [Figure 8] 1 illustrates an example flowchart of a method for encoding according to one embodiment. [Figure 9] 1 illustrates a model used for entropy coding according to one embodiment. [Figure 10] 1 illustrates an example of an image decoder according to at least one embodiment. [Figure 11] 1 illustrates an example flowchart of a method for decoding according to one embodiment. [Figure 12] 10 illustrates experimental results obtained with a method according to at least one embodiment. [Figure 13]10 illustrates experimental results obtained with a method according to at least one embodiment. [Figure 14] 10 illustrates experimental results obtained with a method according to at least one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] This application describes various aspects, including tools, features, embodiments, models, approaches, and the like. Many of these aspects are described with specificity, often in a definitive manner, to at least illustrate their individual characteristics. However, this is for clarity of description and does not limit the application or scope of these aspects. In fact, all of the different aspects can be combined and interchanged to provide further aspects. Furthermore, these aspects can also be combined and interchanged with aspects described in previous applications.

[0013] Aspects described and contemplated in this application may be implemented in many different forms. At least one of these aspects relates generally to video encoding and decoding, and at least one other aspect relates generally to transmitting generated or encoded bitstreams. These and other aspects may be implemented as a method, an apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the described methods.

[0014] In this application, the terms "reconstructed" and "decoded" may be used interchangeably, the terms "encoded" or "coded" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image", "picture", and "frame" may be used interchangeably. Typically, but not necessarily, the term "reconstructed" is used on the encoder side, while "decoded" is used on the decoder side.

[0015] Various methods are described herein, each of which includes one or more steps or acts for achieving the described method. Unless a specific order of steps or acts is required for the successful operation of the method, the order and / or use of specific steps and / or acts may be modified or combined. Furthermore, terms such as “first,” “second,” etc. may be used in various embodiments to modify elements, components, steps, operations, etc., e.g., “first decode” and “second decode.” The use of such terms does not imply a modified order of operations unless specifically required. Thus, in this example, the first decode need not be performed before the second decode, but could occur, for example, before, during, or during an overlapping period with the second decode.

[0016] FIG. 1 illustrates a block diagram of an example system in which various aspects and embodiments can be implemented. System 100 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television sets, personal video recording systems, connected home appliances, and servers. The elements of system 100, alone or in combination, may be embodied in a single integrated circuit, multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed across multiple ICs and / or discrete components. In various embodiments, system 100 is communicatively coupled to other systems or other electronic devices, for example, via a communication bus or via dedicated input and / or output ports. In various embodiments, system 100 is configured to implement one or more of the aspects described herein.

[0017] The system 100 includes at least one processor 110 configured to execute instructions loaded therein, for example, to implement various aspects described herein. The processor 110 may include embedded memory, input / output interfaces, and the like. It may also include various other circuits as are known in the art. System 100 includes at least one memory 120 (e.g., a volatile memory device and / or a non-volatile memory device). System 100 includes a storage device 140, which may include non-volatile memory and / or volatile memory, including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drives, and / or optical disk drives. Storage device 140 may include, by way of non-limiting example, an internal storage device, a removable storage device, and / or a network-accessible storage device.

[0018] System 100 includes encoder / decoder module 130, which may include its own processor and memory, configured to process data to provide, for example, encoded or decoded video. Encoder / decoder module 130 represents a module or modules that may be included in a device that performs encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Furthermore, encoder / decoder module 130 may be implemented as a separate element of system 100 or may be incorporated within processor 110 as a combination of hardware and software, as is known to those skilled in the art.

[0019] Program code loaded into processor 110 or encoder / decoder module 130 to perform various aspects described herein may be stored in storage device 140 and subsequently loaded onto memory 120 for execution by processor 110. According to various embodiments, one or more of processor 110, memory 120, storage device 140, and encoder / decoder module 130 may store one or more of various items during execution of processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from processing equations, expressions, operations, and computational logic.

[0020] In some embodiments, memory internal to the processor 110 and / or the encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be either the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory may be the memory 120 and / or the storage device 140, and may be, for example, dynamic volatile memory and / or non-volatile flash memory. In some embodiments, external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, high-speed external dynamic volatile memory, such as RAM, is used as working memory for video coding and decoding operations.

[0021] Input to the elements of system 100 may be provided via various input devices, as shown in block 105. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives, for example, RF signals transmitted over the air by a broadcast station, (ii) a component (COMP) input terminal (or set of COMP input terminals), (iii) a Universal Serial Bus (USB) input terminal, and / or (iv) a High Definition Multimedia Interface (HDMI) input terminal. Other examples not shown in FIG. 1 include composite video.

[0022] In various embodiments, the input devices of block 105 have associated respective input processing elements as known in the art. For example, the RF section may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or bandlimiting a signal to a frequency band), (ii) downconverting the selected signal, (iii) bandlimiting again to a narrower frequency band to select a signal frequency band, which in certain embodiments may be referred to as a channel (for example), (iv) demodulating the downconverted and bandlimited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section of various embodiments includes one or more elements for performing these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF section may include, for example, a tuner that performs various of these functions, including downconverting a received signal to a lower frequency (e.g., an intermediate frequency or a frequency near baseband) or to baseband. In one set-top box embodiment, the RF section and its associated input processing elements perform frequency selection by receiving, filtering, downconverting, and re-filtering RF signals transmitted over a wired (e.g., cable) medium to a desired frequency band. In various embodiments, the order of these (and other) elements is rearranged, some of these elements are removed, and / or other elements that perform similar or different functions are added. Adding elements may include inserting elements between existing elements, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0023] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting system 100 to other electronic devices over USB and / or HDMI connections. It should be understood that various aspects of input processing, e.g., Reed-Solomon error correction, may be performed, for example, within a separate input processing IC or within processor 110, as desired. Similarly, aspects of USB or HDMI interface processing may be performed, as desired, within a separate interface IC or within processor 110. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 110 and encoder / decoder module 130, which operates in combination with memory and storage elements to process the data stream as desired for display on an output device.

[0024] The various elements of system 100 may be provided within an integrated housing in which the various elements are interconnected and may transmit data therebetween using a suitable connection arrangement 115, e.g., an internal bus as known in the art, including an I2C bus, wiring, and a printed circuit board.

[0025] System 100 includes a communication interface 150 that enables communication with other devices over a communication channel 190. Communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 190. Communication interface 150 may include, but is not limited to, a modem or a network card, and communication channel 190 may be implemented in a wired and / or wireless medium, for example.

[0026] In various embodiments, data is streamed to system 100 using a Wi-Fi network, such as IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal in these embodiments is received via communication channel 190 and communication interface 150, which are adapted for Wi-Fi communication. Communication channel 190 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. In other embodiments, streamed data is provided to system 100 using a set-top box that delivers data via an HDMI connection in input block 105. In yet other embodiments, streamed data is provided to system 100 using an RF connection in input block 105. As noted above, in various embodiments, data is provided in a non-streaming manner. Additionally, in various embodiments, wireless networks other than Wi-Fi, such as a cellular network or a Bluetooth network, are used.

[0027] System 100 may provide output signals to various output devices, including a display 165, speakers 175, and other peripheral devices 185. Display 165 in various embodiments includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 165 may be for a television, a tablet, a laptop, a mobile phone, or other device. Display 165 may also be integrated with other components (e.g., as in a smartphone) or may be separate (e.g., an external monitor for a laptop). Other peripheral devices 185, in various example embodiments, include one or more of a standalone digital video disc (or digital versatile disc) (both terms DVR), a disc player, a stereo system, and / or a lighting system. In various embodiments, one or more peripheral devices 185 are used that provide functionality based on the output of system 100. For example, a disc player performs the function of playing the output of system 100.

[0028] In various embodiments, control signals are communicated between system 100 and display 165, speaker 175, or other peripheral device 185 using signaling such as AV.Link, CEC, or other communication protocols that allow control between devices with or without user intervention. Output devices may be communicatively coupled to system 100 via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, output devices may be connected to system 100 using communication channel 190 via communication interface 150. Display 165 and speaker 175 may be integrated into a single unit with other components of system 100 in an electronic device such as a television. In various embodiments, display interface 160 includes a display driver, e.g., a timing controller (T Con) chip.

[0029] Display 165 and speakers 175 may alternatively be separate from one or more of the other components, for example, if the RF portion of input 105 is part of a separate set-top box. In various embodiments in which display 165 and speakers 175 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0030] The embodiments may be executed by the processor 110, or by computer software implemented by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 120 may be of any type appropriate to the technological environment, and may be implemented using any suitable data storage technology, such as, by way of non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 110 may be of any type appropriate to the technological environment, and may include, by way of non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0031] 2 shows an example of an end-to-end neural network-based compression system 200 for encoding an image using a deep neural network. An input image I to be encoded is first processed by a deep neural network encoder 210 (hereafter identified as a deep encoder). The output of the encoder y is called the embedding of the image. This embedding is then encoded, for example, into a bitstream 220 by passing it through a quantizer Q and then through an entropy encoder, for example, an arithmetic encoder AE. The resulting bitstream 220 is then encoded into a bitstream 220 by passing it through a quantizer Q and then through an entropy encoder, for example, an arithmetic encoder AE.

number

number

[0032] Deep encoders and decoders consist of multiple neural layers, such as convolutional layers. Each neural layer can be described as a function that first multiplies the input by a tensor, adds a vector called a bias, and then applies a nonlinear function to the resulting value. The values ​​of the tensor and bias are denoted by the term "weights." The weights, and the parameters of the nonlinear function, if applicable, are called the network's parameters. In such compression systems, the encoder and decoder are fixed based on a predetermined model that is assumed to be known at the time of encoding and decoding. The neural networks of the encoder and decoder are, for example, trained simultaneously to be compatible. In practice, to learn the weights of the encoder and decoder, the neural networks are trained on a large database D of images. Together, they are sometimes called "autoencoders," which encode the input and then reconstruct it. While some layers or their ordering may differ slightly, the decoder architecture is typically largely the inverse of the encoder.

[0033] 3 illustrates an example of an end-to-end implicit neural network (INR)-based compression system 300 for encoding an image. The system includes an encoder 312 that generates encoded data, e.g., in the form of a bitstream 320, and a decoder 332. The encoder 312 includes an INR 310, and the decoder 332 includes an INR 330. Compared to autoencoder-based image compression that uses latent points to control the rate-distortion target, in an INR-based compression system, the rate (R)-distortion (D) tradeoff is controlled by the number of weights or the size of the neural network. Thus, for different rates, the INR has different neural network architectures with different numbers of weights. As shown in FIG. 3, the INR 310 or 330 maps pixel coordinates (x, y) to pixel values, e.g., (R, G, B) values, or other values ​​such as YCbCr, YUV, or any other color values ​​in a given color space, e.g., considering the RGB color space, fθ(x, y) = (r, g, b), where fθ(┤) is the INR function. The INR is designed using a multi-layer perceptron (MLP), where "L" is the number of layers, each containing a desired number of hidden neurons. Each layer can be described as a function that first multiplies the input value by a tensor, adds a bias, and finally transforms the result by a nonlinear activation function. The values ​​of the tensor and bias are denoted by the term "weight" and denoted as θ. These weights are unknown and are estimated on the encoder side.

[0034] Compressing an image I with an INR function fθ is equivalent to determining these weights for storage or transmission. For this purpose, the image I is first processed by an INR 310, which is responsible for determining the weights θ from the image I. The weights θ are encoded into a bitstream 320, for example by passing them through a quantizer Q and then through an encoder ENC, e.g., an entropy encoder such as an arithmetic encoder. The resulting bitstream is decoded by passing it through a decoder DEC to reconstruct the quantized weights, which are then dequantized by an inverse quantizer IQ (also known as a dequantizer). The pixel coordinates of the reconstructed image are then stored in the INR 310.

number

[0035] In contrast to autoencoders, the weights θ may be determined by training on the image I to be encoded. As a result, each image to be encoded has its own associated weight. The weights θ may be determined by minimizing the following loss function:

number

[0036] For each image I, there is one specific INR function fθ that is overfitted to the given image I. The quality of the image reconstructed by fθ depends on the size of the neural network. Since the weights are used as descriptors of the image, the larger the size of the neural network, the longer the bit length. On the other hand, limiting the number of weights reduces the bit length at the expense of distortion.

[0037] Some existing methods for quantizing weights perform naive quantization of weights by quantizing 32-bit precision weights to 16-bit precision weights. Post-training quantization or primitive quantization-aware training methods can also be used. However, the compression efficiency of these methods is not optimal because INR does not recognize the distortion resulting from post-training quantization or quantization methods, and entropy models are not efficient in existing quantization-aware training procedures.

[0038] The embodiments described below aim to improve quantization and possibly entropy coding to increase compression efficiency, i.e., reduce the file size of the weights with negligible or minimal loss in reconstruction quality. This principle can also be applied to encoding / decoding images (i.e., frames) of a video sequence. Furthermore, the decoding method disclosed below allows for progressive decoding of an image by simply evaluating the function fθ at various pixel locations, e.g., one of two pixels, for example, by first decoding a portion of the image or a lower-resolution image. Partial decoding of an image is difficult with an autoencoder.

[0039] 4 shows an example flowchart of a method for encoding an image I according to one embodiment. This method may be performed by the encoder 312 of FIG. 3, for example, in the system 100 of FIG.

number

number

number

[0040] In step S110, the maximum absolute value among the weights of a certain type, e.g., among the tensor values ​​or among the biases, in the current layer of index l is obtained. The maximum absolute value is calculated over the weights of this type (e.g., for the tensor values) as follows:

number

number

number

number

number

[0041] In step S130, the maximum absolute value

number

number

[0042] In one example, the above steps S110 to S130 can be repeated for another layer. In one example, the above steps S110 to S130 can be repeated for all remaining layers, and the fixed-bit quantized weights of all layers are

number

[0043] Similarly, the above steps S110-S130 may be repeated for quantizing and encoding another type of weight, e.g., bias, of one current layer or of two or more layers, e.g., all layers. The quantized bias of layer l is

number

number

number

[0044] Encoding the maximum absolute value of the tensors and biases for the current layer requires 2×L×n bits in addition to the network weights. In this case,

number

[0045] In one example, only a subset of weights may be quantized, e.g., only biases, only tensor values, and / or only some layers. Thus, in this case, only a subset of the maximum absolute values ​​may be quantized, e.g.,

number

[0046] In one example, the quantization may be performed on all quantized weights at once, rather than an iterative process across layers.

[0047] Rather than layer-by-layer, the aforementioned iterative process may be performed over any subset of weights, e.g., weight-by-weight, neuron-by-neuron, group-by-neuron, or any combination of these subsets, including, for example, quantizing some weights of some / all layers at each iteration.

[0048] 5 illustrates an example of an image decoder 432 according to at least one embodiment. The image decoder 432 may be implemented, for example, in the system 100 of FIG. 1 and is adapted to decode coded data, e.g., configured as a bitstream 400, including coded maximum absolute value(s) 410 and coded quantized weights 420. The coded maximum absolute value(s) 410 are decoded (dec) from the bitstream. The coded quantized weights 420 are decoded (DEC) and dequantized (also called dequantization) according to the decoded maximum absolute value(s). The pixel coordinates of the reconstructed image are then decoded (DEC) and dequantized (also called dequantization).

number

[0049] 6 shows an example flowchart of a method for decoding according to one embodiment, which may be performed by decoder 332 of FIG. 3 or decoder 432 of FIG. 5, for example, in system 100 of FIG.

[0050] In step S600, the decoder obtains coded data, e.g., in the form of a bitstream, received from another device or read from a storage medium. The coded data, e.g., bitstream 400, may include at least one maximum absolute value(s) w, as shown in FIG. max 410 and quantized weights

number

[0051] In step S610, the quantized weights

number

[0052] In step S620, the dequantized weights are dequantized. As an example, for the current layer l and tensor value, the dequantized weights are obtained as follows:

number

[0053] In step S630, the pixel coordinates of the image to be reconstructed are calculated using the reconstructed image

number

[0054] 7 shows a method for training a quantization-aware INR according to one embodiment, which may be used to obtain the weights to be coded in step S100.

[0055] Quantization-aware training involves training weights θ of a model already trained with full precision (e.g., 32-bit floating-point weights). * In other words, we can start with the initial weights, e.g., the weights θ * is obtained in step S100-1. In a variant, a default random initialization of the weights may be obtained instead.

[0056] In step S100-2, these weights are converted into quantized weights by applying steps S110-S120 of the method of FIG.

number

number

[0057] In step S100-3, the quantized weights are dequantized as follows:

number

number

[0058] In step S100-4, the reconstruction loss is calculated as follows:

number

number

number

[0059] In step S100-5, the weights are updated according to the reconstruction loss using batch gradient descent or stochastic gradient descent.

[0060] These steps S100-2 to S100-5 are repeated (S100-6) until a stopping criterion is reached, which can be a convergence criterion (e.g., loss<threshold) or that a certain number of iterations K, e.g., K=10,000, has been reached.

[0061] In the first variant, the prediction of the quantized model

number

number

[0062] In the first example, T is the prediction of the quantized model.

number

number

number

number

number

number

number

[0063] Second, the quantized model

number

number

[0064] For faster coding, only the regularization term in equation (4), i.e.,

number

[0065] During the backward pass, since the nature of the quantization is non-differentiable, the gradients are calculated using a straight-through estimator (STE), and the weights are updated using any optimizer. Finally, once determined, the weights (tensor values ​​and / or biases) are quantized to q bits and coded. For this purpose, as in the previous embodiment, steps S110 to S130 are applied to the weights obtained by the above training method. Thus, a maximum absolute value is determined for each layer and for each type of weight (tensor, bias, etc.). The weights obtained by the above training method are quantized according to the obtained maximum absolute value(s). The obtained maximum absolute value(s) are coded using n bits, for example n=16 bits, and the quantized weights are coded using n bits, for example n=16 bits.

number

[0066] FIG. 8 shows an example of a flowchart of a method for encoding an image I according to another embodiment. This method may be performed by the encoder 312 of FIG. 3, for example, in the system 100 of FIG. 1. Steps identical to those of the encoding method shown in FIG. 4 are identified with the same reference numerals in FIG. 9. In particular, this method includes steps S100, S110, and S120. As described with reference to the previous embodiment, in step S130, the quantized weights may be written directly into the bitstream using q bits. However, they may also be coded using various methods, for example, entropy coding methods, more specifically, arithmetic coding methods, to obtain additional compression efficiency. Entropy coding utilizes the weight distribution shape and encodes the quantized weights with q bits.

number

number

number

number

number

number

number

number

number

number

number

[0067]

number

number

number

[0068] In another embodiment, different values

number

number

[0069] The boundary value probabilities may also include terms from a Gaussian distribution defined as follows:

number

number

[0070] If the data is adapted on a per-picture basis, for example, additional information may be included in the bitstream, for example L1, or one or more bits signaling the selection made for each picture.

[0071] 10 illustrates an example of an image decoder 532 according to at least one embodiment. This decoder 532 may be implemented, for example, in the system 100 of FIG. 1 and uses entropy model parameters 510, such as mean μ and standard deviation σ (or variance σ). 2 ), the coded maximum absolute value(s) 515, and the coded quantized weights 520. The coded maximum absolute value(s) are decoded (dec) from the bitstream. The parameters of the entropy model are decoded (D). The coded quantized weights 520 are used to determine if the probability model has parameters μ and σ 2 parameterized by a fixed probability boundary value P border The decoded quantized weights are then dequantized (also called dequantized) according to the maximum decoded absolute value(s). The pixel coordinates of the reconstructed image are then dequantized by the entropy decoder AD, which is further parameterized by

number

[0072] 11 shows an example flowchart of a method for decoding according to one embodiment, which may be performed by decoder 332 of FIG. 3 or 532 of FIG. 10, for example, in system 100 of FIG.

[0073] In step S900, the decoder obtains coded data, for example in the form of a bitstream, received from another device or read from a storage medium. The coded data includes at least one maximum absolute value(s) w max , the quantized weights

number

[0074] In step S910, the mean μ and the standard deviation σ (or variance σ 2 ) and at least one maximum absolute value w max is decoded from the bitstream.

[0075] In step S920, the entropy coded quantized weights are converted into a weight vector with parameters μ and σ 2 is defined as a truncated Gaussian distribution with a fixed probability boundary value P border , and then entropy decoded using a probability model further defined by: This step is the inverse of the entropy encoding step.

[0076] In step S930, the dequantized weights are dequantized according to the maximum decoded absolute value. As an example, for the current layer l and tensor value, the dequantized weights are obtained as follows:

number

[0077] In step S940, the pixel coordinates of the image to be reconstructed are calculated using the reconstructed image

number

[0078] The following figures show experimental results obtained using the above methods (quantization, entropy coding, and quantization-aware training) on ​​the Kodak Test Set. FIG. 12 shows the rate-distortion curves averaged over all images in the Kodak dataset, demonstrating that the proposed method 600 has significant gains over competitors known as Coin 610 and Coin++ 620. To quantify the gains in percentage, the BD-rate gains are calculated. FIG. 13 shows the average gain of the disclosed method 700 of 41.8% over Coin method 710, and FIG. 14 shows the average gain of the disclosed method 800 of 31.5% over Coin++ 810. The regularization term T provides a gain of approximately 10% over simply using 8-bit quantization with entropy coding. Furthermore, the disclosed method is general and can be applied to any INR-based image / video codec.

[0079] Unless otherwise indicated or technically excluded, the aspects described in this application can be used individually or in any combination.

[0080] Various numerical values ​​are used in this application. The particular values ​​are for illustrative purposes and the described aspects are not limited to these particular values.

[0081] Various embodiments involve decoding. As used herein, "decoding" may encompass all or part of the processes performed on a received encoded sequence to generate a final output suitable for display, for example. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding and inverse quantization. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or to the broader decoding process generally will be clear based on the context of the particular description and will be well understood by those skilled in the art.

[0082] Various implementations involve encoding. Similar to the above discussion of "decoding," "encoding" as used herein may encompass, for example, all or part of the processes performed on an input video sequence to generate an encoded bitstream.

[0083] Implementations and aspects described herein may be realized in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even when discussed only in the context of a single implementation (e.g., discussed only as a method), the implementation of the discussed features may also be realized in other forms (e.g., an apparatus or a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The method may be performed in, for example, an apparatus, e.g., a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, e.g., computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate the transfer of information between end users.

[0084] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation," as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with that embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment," or "in one implementation" or "in an implementation," as well as other variations thereof, appearing in various places throughout this specification are not necessarily all referring to the same embodiment.

[0085] Additionally, this application or its claims may refer to "determining" various pieces of information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0086] Additionally, this application or its claims may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, predicting information, or inferring information.

[0087] Also, in this application or its claims, various information may be referred to as "receiving." Receiving, like "accessing," is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from memory or optical media storage). Furthermore, "receiving" typically involves in some manner, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, judging information, predicting information, or evaluating information.

[0088] For example, in the case of "A / B," "A and / or B," and "at least one of A and B," it should be understood that the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass the selection of only the first-listed option (A), or the selection of only the second-listed option (B), or the selection of both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first-listed option (A), or the selection of only the second-listed option (B), or the selection of only the third-listed option (C), or the selection of only the first and second-listed options (A and B), or the selection of only the first and third-listed options (A and C), or the selection of only the second and third-listed options (B and C), or the selection of all three options (A, B, and C). This applies regardless of the number of items listed, as will be readily apparent to those skilled in the art.

[0089] Also, as used herein, the word "signaling" refers to, among other things, indicating something to a corresponding decoder. For example, in certain embodiments, an encoder signals a quantization matrix for dequantization, or at least one value representing the maximum absolute value of weights in a layer of the neural network described above, to quantize weights, the mean, and the standard deviation of a Gaussian distribution. In this manner, in one embodiment, the same parameters are used on both the encoder and decoder sides. Thus, for example, an encoder can transmit specific parameters to a decoder (explicit signaling), so that the decoder can use the same specific parameters. Conversely, if the decoder already has specific parameters as well as other parameters, signaling can be used without transmission (implicit signaling), allowing the decoder to easily recognize and select the specific parameters. By avoiding the transmission of any actual functions, bit savings are realized in various embodiments. It should be appreciated that signaling can be achieved in various ways. For example, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder in various embodiments. Although the above refers to the verb form of the word "signaling," the word "signaling" can also be used as a noun herein.

[0090] As will be apparent to those skilled in the art, implementations may generate various signals formatted to carry information, which may be stored or transmitted, for example. The information may include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor-readable medium.

[0091] Several embodiments have been described above, and the features of these embodiments may be provided singly or in any combination across various claim categories and types.

[0092] 1. A coding method comprising: obtaining weights of a neural network, the weights representing an input image; obtaining at least one value representing the maximum absolute value of weights in a layer of said neural network; quantizing the weights of the layer in response to the at least one value; and encoding the at least one value and the quantized weight into a bitstream.

[0093] In one example, quantizing the weights of the layer in response to the at least one value includes: dividing the weight by the at least one value to obtain a normalized weight; quantizing the normalized weights using a fixed-bit quantizer.

[0094] In one example, encoding the quantized weights includes entropy coding the quantized weights using a probability model defined by fixed probabilities for boundary symbol values ​​and Gaussian distributions for the remaining symbols.

[0095] In one example, the mean and standard deviation of the Gaussian distribution are encoded into the bitstream.

[0096] In one example, obtaining the neural network weights includes minimizing distortion between the input image and an image reconstructed from a neural network parameterized by the dequantized weights.

[0097] In one example, obtaining the neural network weights includes minimizing a loss function that is a weighted sum between a first distortion and a second distortion, where the first distortion is the distortion between an input image and an image reconstructed from a neural network parameterized with dequantized weights, and the second distortion is the distortion between an image reconstructed from a neural network parameterized with full precision with fixed weights and an image reconstructed from a neural network parameterized with dequantized weights.

[0098] In one example, obtaining the neural network weights includes minimizing a loss function that is a weighted sum between a first distortion and a second distortion, where the first distortion is the distortion between an input image and an image reconstructed from a neural network parameterized by dequantized weights, and the second distortion is the distortion between an image reconstructed from a neural network parameterized by unquantized weights and an image reconstructed from a neural network parameterized by dequantized weights.

[0099] In one example, the weights belong to a set of weights that includes biases and tensor values.

[0100] 1. A decoding method comprising: obtaining a bitstream including at least one value representing the maximum absolute value of a weight in a layer of a neural network and the quantized weights of said layer; decoding the at least one value and the quantized weights of the neural network from a bitstream; dequantizing the quantized weights of the layer according to at least one value to obtain dequantized weights; and reconstructing the image using a neural network parameterized by the dequantized weights.

[0101] In one example, dequantizing the weights of the layer in response to the at least one value includes: dequantizing the quantized weights using a fixed-bit quantizer; multiplying the inverse quantized weights by at least one value to obtain dequantized weights.

[0102] In one example, decoding the quantized weights includes entropy decoding the quantized weights using a probability model defined by fixed probabilities for boundary symbol values ​​and Gaussian distributions for the remaining symbols.

[0103] In one example, the mean and standard deviation of the Gaussian distribution are decoded from the bitstream.

[0104] In one example, the weights belong to a set of weights that includes biases and tensor values.

[0105] An encoding device is disclosed, comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to perform an encoding method according to any one of the previously disclosed examples.

[0106] A decoding device is disclosed, comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to perform a decoding method according to any one of the previously disclosed examples.

[0107] A computer program comprising program code instructions for performing an encoding method or a decoding method when executed by a processor is disclosed.

[0108] A computer-readable storage medium having stored thereon instructions for performing an encoding method or a decoding method is disclosed.

Claims

1. obtaining weights of a neural network, the weights representing an input image; obtaining at least one value representing the maximum absolute value of weights in a layer of the neural network; quantizing the weights of the layer in response to the at least one value; encoding the at least one value and the quantized weights into a bitstream; 10. An encoding method comprising:

2. quantizing the weights of the layer in response to the at least one value dividing the weight by the at least one value to obtain a normalized weight; quantizing the normalized weights using a fixed-bit quantizer.

3. 3. The method of claim 1, wherein encoding the quantized weights comprises entropy coding the quantized weights using a probability model defined by fixed probabilities for boundary symbol values ​​and Gaussian distributions for the remaining symbols.

4. The method of claim 3 , wherein the mean and standard deviation of the Gaussian distribution are encoded into the bitstream.

5. 5. The method of claim 1, wherein obtaining the neural network weights comprises minimizing distortion between the input image and an image reconstructed from a neural network parameterized by the dequantized weights.

6. 5. The method of claim 1, wherein obtaining the neural network weights comprises minimizing a loss function that is a weighted sum between a first distortion and a second distortion, the first distortion being the distortion between the input image and an image reconstructed from the neural network parameterized by dequantized weights, and the second distortion being the distortion between an image reconstructed from the neural network parameterized with full precision by fixed weights and an image reconstructed from the neural network parameterized by dequantized weights.

7. 5. The method of claim 1, wherein obtaining the neural network weights comprises minimizing a loss function that is a weighted sum between a first distortion and a second distortion, the first distortion being the distortion between the input image and an image reconstructed from the neural network parameterized by dequantized weights, and the second distortion being the distortion between the image reconstructed from the neural network parameterized by unquantized weights and an image reconstructed from the neural network parameterized by dequantized weights.

8. The method according to any one of claims 1 to 7, wherein the weights belong to a set of weights comprising biases and tensor values.

9. obtaining a bitstream including at least one value representing the maximum absolute value of weights in a layer of a neural network and quantized weights of said layer; decoding the at least one value and the quantized weights of a neural network from the bitstream; dequantizing the quantized weights of the layer according to the at least one value to obtain dequantized weights; reconstructing an image using a neural network parameterized by the dequantized weights; A decoding method comprising:

10. Inverse quantizing the weights of the layer in response to the at least one value includes: dequantizing the quantized weights using a fixed-bit quantizer; and multiplying the dequantized weight by the at least one value to obtain a dequantized weight.

11. 11. The method of claim 9 or 10, wherein decoding the quantized weights comprises entropy decoding the quantized weights using a probability model defined by fixed probabilities for boundary symbol values ​​and Gaussian distributions for the remaining symbols.

12. The method of claim 11 , wherein the mean and standard deviation of the Gaussian distribution are decoded from the bitstream.

13. The method according to any one of claims 8 to 12, wherein the weights belong to a set of weights comprising biases and tensor values.

14. 9. An encoding device comprising one or more processors and at least one memory coupled to said one or more processors, said one or more processors being configured to execute the method of any one of claims 1 to 8.

15. 14. A decoding device comprising one or more processors and at least one memory coupled to said one or more processors, said one or more processors being configured to execute the method of any one of claims 9 to 13.

16. A computer program comprising program code instructions for performing the method according to any one of claims 1 to 13 when the computer program is executed by a processor.

17. A computer readable storage medium storing instructions for performing the method of any one of claims 1 to 13 when executed by a processor.

Images