Deep neural network-based image compression using potential shift of potential entropy-based gradient

Abstract

A video compression system based on a deep neural network, wherein the compression efficiency is improved by using the gradient of entropy relative to side potential and main potential at the decoding side.

Description

Technical Field

[0001] One or more embodiments in this example generally relate to a method and device for encoding and decoding picture data using a deep neural network. Specifically, it relates to a method and device for improving compression efficiency by utilizing the gradient of latent entropy. Background Art

[0002] To achieve high compression efficiency, traditional video compression schemes typically employ prediction and transformation to exploit spatial and temporal redundancies in video content. During encoding, pictures of video content are divided into sample blocks (i.e., pixels), and these blocks are then divided into one or more sub-blocks, hereinafter referred to as original sub-blocks. Then, intra-frame or inter-frame prediction is applied to each sub-block to utilize intra-frame or inter-frame image correlations. Regardless of the prediction method used (intra-frame or inter-frame), a predicted sub-block is determined for each original sub-block. Then, the sub-block representing the difference between the original sub-block and the predicted sub-block (usually denoted as the prediction error sub-block, prediction residual sub-block, or simply the residual sub-block) is transformed, quantized, and entropy-encoded to generate an encoded video stream. To reconstruct the video, the compressed data is decoded through the inverse processes corresponding to transformation, quantization, and entropy encoding.

[0003] In recently explored video coding solutions, neural network (NN)-based processing has been studied. Several NN-based solutions have been proposed from hybrid solutions, where the NN is used to implement some of the tools of the above traditional video compression scheme to achieve a fully NN-based compression solution.

[0004] A key aspect of any video (or image) compression solution is to use as much information available on the decoder side as possible to improve compression efficiency. The information available on the decoder side of course includes the data explicitly encoded in the bitstream (i.e., the encoded video data), but also includes data that is not explicitly encoded but can be inferred from the explicitly encoded video data. This inferred data is mainly used in traditional video compression schemes. However, fully NN-based compression methods are not as mature as methods based on traditional video compression schemes. Therefore, the possibility of using explicitly encoded data to infer other data remains an aspect to be explored.

[0005] It is desirable to determine which data can be inferred from the explicitly encoded data available on the decoder side in a fully NN-based video (or image) compression method, and how to use this inferred data to improve the compression efficiency of the fully NN-based video compression method. Summary of the Invention

[0006] In a first aspect, one or more of the embodiments in this example provide a method, including: Obtain an input picture; apply an encoder based on a deep neural network to the input picture to obtain a main latent; apply an encoder based on a deep hyperprior neural network to the input picture to obtain a side latent; use a first quantization method to quantize the side latent to obtain a side code; apply a factorized entropy model to the side code to determine a first probability mass function of the side code under the first quantization method; perform arithmetic coding on the side code in video data based on the first probability mass function; perform inverse quantization on the side code to obtain a reconstructed side latent; determine a first step size based on the gradient of the entropy of the side information with respect to the reconstructed side latent; encode information representing the first step size in the video data; shift the reconstructed side latent using the first step size and the gradient of the entropy of the side information with respect to the reconstructed side latent; apply a deep hyperprior decoder to the shifted reconstructed side latent to obtain information representing a probability distribution for modeling the main latent; obtain a second probability mass function of the main code using the probability distribution for modeling the main latent; quantize the main latent to obtain the main code; and perform arithmetic coding on the main code in the video data based on the second probability mass function.

[0007] In one embodiment, the method further includes: performing inverse quantization on the main code to obtain a reconstructed main code; determining a second step size based on the gradient of the entropy of the main information with respect to the entropy of the main latent; and encoding information representing the second step size in the video data.

[0008] In a second aspect, one or more of the present embodiments provide a method, including: Obtain an input picture; apply an encoder based on a deep neural network to the input picture to obtain a main latent; apply an encoder based on a deep hyperprior neural network to the input picture to obtain a side latent; use a first quantization method to quantize the side latent to obtain a side code; apply a factorized entropy model to the side code to determine a first probability mass function of the side code under the first quantization method; perform arithmetic coding on the side code in video data based on the first probability mass function; perform inverse quantization on the side code to obtain a reconstructed side latent; apply a deep hyperprior decoder to the shifted reconstructed side latent to obtain information representing a probability distribution for modeling the main latent; obtain a second probability mass function of the main code using the probability distribution for modeling the main latent; perform arithmetic coding on the main code in the video data based on the second probability mass function; perform inverse quantization on the main code to obtain a reconstructed main latent; determine a second step size based on the gradient of the entropy of the main information with respect to the main latent; and encode information representing the second step size in the video data.

[0009] In a third aspect, one or more of the present embodiments provide a method, including: Apply arithmetic decoding to side information included in video data to obtain a side code using a first probability mass function provided by a factorized entropy model; inverse-quantize the side code to obtain a reconstructed side latent; decode information representing a first step size based on the gradient of the side information entropy with respect to the reconstructed side latent; shift the reconstructed side latent using the first step size and the gradient of the side information entropy with respect to the reconstructed side latent; apply a deep hyperprior decoder to the shifted reconstructed side latent to obtain information representing a probability distribution modeling the main latent; obtain a second probability mass function for the main code using the probability distribution modeling the main latent; arithmetically decode the main code from the video data using the second probability mass function; inverse-quantize the main code to obtain a reconstructed main latent; and apply a deep neural network-based decoder to the reconstructed main latent image to obtain an output picture.

[0010] In one embodiment, the method further includes: decoding information representing a second step size from the video data based on the gradient of the main information entropy with respect to the main latent; and shifting the reconstructed main latent using the second step size and the gradient of the main information entropy with respect to the main latent, and applying a deep neural network-based decoder to the shifted reconstructed main latent.

[0011] In a fourth aspect, one or more of the present embodiments provide a method, including: Apply arithmetic decoding to side information included in video data to obtain a side code using a first probability mass function provided by a factorized entropy model; inverse-quantize the side code to obtain a reconstructed side latent; apply a deep hyperprior decoder to the reconstructed side latent to obtain information representing a probability distribution modeling the main latent; obtain a second probability mass function for the main code using the probability distribution modeling the main latent; arithmetically decode the main code from the video data using the second probability mass function; inverse-quantize the main code to obtain a reconstructed main latent; decode information representing a second step size from the video data based on the gradient of the main information entropy with respect to the main latent; shift the reconstructed main latent using the second step size and the gradient of the main information entropy with respect to the main latent; and apply a deep neural network-based decoder to the shifted reconstructed main latent to obtain an output picture.

[0012] In a fifth aspect, one or more of the present embodiments provide a device including an electronic circuit configured to: Obtain an input picture; Apply a deep neural network-based encoder to the input picture to obtain a main latent; Apply a deep hyperprior neural network-based encoder to the input picture to obtain a side latent; Quantize the side latent using a first quantization method to obtain a side code; Apply the decomposed entropy model to the side code to determine the first probability mass function of the side code under the first quantization method; Arithmetically encode the side code in the video data based on the first probability mass function; Inverse quantize the side code to obtain the reconstructed side latent; Determine the first step size based on the gradient of the entropy of the side information with respect to the reconstructed side latent; Encode information representing the first step size in the video data; Shift the reconstructed side latent using the first step size and the gradient of the side information entropy with respect to the reconstructed side latent; Apply the deep hyperprior decoder to the shifted reconstructed side latent to obtain information representing the probability distribution for modeling the main latent; Obtain the second probability mass function of the main code using the probability distribution for modeling the main latent; Quantize the main latent to obtain the main code; and, Arithmetically encode the main code in the video data based on the second probability mass function.

[0013] In one embodiment, the electronic circuit is further configured to: Inverse quantize the main code to obtain the reconstructed main latent; Determine the second step size based on the gradient of the main information entropy with respect to the main latent; and, Encode information representing the second step size in the video data.

[0014] In a sixth aspect, one or more of the present embodiments provide an apparatus including an electronic circuit configured to: Obtain an input picture; Apply an encoder based on a deep neural network to the input picture to obtain a main latent; Apply an encoder based on a deep hyperprior neural network to the input picture to obtain a side latent; Quantize the side latent using the first quantization method to obtain a side code; Apply the decomposed entropy model to the side code to determine the first probability mass function of the side code under the first quantization method; Arithmetically encode the side code in the video data based on the first probability mass function; Inverse quantize the side code to obtain the reconstructed side latent; Apply the deep hyperprior decoder to the shifted reconstructed side latent to obtain information representing the probability distribution for modeling the main latent; Obtain the second probability mass function of the main code using the probability distribution for modeling the main latent; Arithmetically encode a primary code in video data based on a second probability mass function; Inverse quantize the primary code to obtain a reconstructed primary latent; Determine a second step size based on the gradient of the primary information entropy with respect to the primary latent; and, Encode information representing the second step size in the video data.

[0015] In a seventh aspect, one or more of the present embodiments provide a device including an electronic circuit configured to: Apply arithmetic decoding to side information included in video data to obtain a side code using a first probability mass function provided by a factored entropy model; Inverse quantize the side code to obtain a reconstructed side latent; Decode information representing a first step size based on the gradient of the side information entropy with respect to the reconstructed side latent; Shift the reconstructed side latent using the first step size and the gradient of the side information entropy with respect to the reconstructed side latent; Apply a deep hyperprior decoder to the shifted reconstructed side latent to obtain information representing a probability distribution modeling the primary latent; Obtain a second probability mass function of the primary code using the probability distribution modeling the primary latent; Arithmetically decode the primary code from the video data using the second probability mass function; Inverse quantize the primary code to obtain a reconstructed primary latent; and, Apply a deep neural network-based decoder to the reconstructed primary latent image to obtain an output picture.

[0016] In one embodiment, the electronic circuit is further configured to: Decode information representing the second step size from the video data based on the gradient of the primary information entropy with respect to the primary latent value; and, Shift the reconstructed primary latent using the second step size and the gradient of the primary information entropy with respect to the primary latent, and apply the deep neural network-based decoder to the shifted reconstructed primary latent.

[0017] In an eighth aspect, one or more of the present embodiments provide a device including an electronic circuit configured to: Apply arithmetic decoding to side information included in video data to obtain a side code using a first probability mass function provided by a factored entropy model; Inverse quantize the side code to obtain a reconstructed side latent; Apply the depth hyperprior decoder to the reconstructed side latent to obtain information representing the probability distribution modeling the main latent; obtain a second probability mass function of the main code using the probability distribution modeling the main latent; Arithmetically decode the main code from the video data using the second probability mass function; Inverse quantize the main code to obtain the reconstructed main latent; Decode information representing the second step size from the video data based on the gradient of the main information entropy with respect to the main latent; Use the second step size and the gradient of the main information entropy with respect to the main latent to shift the reconstructed main latent; and apply a decoder based on a deep neural network to the shifted reconstructed main latent to obtain an output picture.

[0018] In a ninth aspect, one or more of the present embodiments provide a computer program comprising program code instructions for implementing the method according to the first, second, third, fourth, fifth, or sixth aspect.

[0019] In a tenth aspect, one or more of the present embodiments provide a non-transitory information storage medium storing program code instructions for implementing the method according to the first, second, third, fourth, fifth, or sixth aspect.

[0020] In an eleventh aspect, one or more of the present embodiments provide a signal generated by the method according to the first, second, or third aspect or by the device according to the seventh, eighth, or ninth aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Schematically shows the context in which the embodiments are implemented; Figure 2A Schematically shows an example of the hardware architecture of a processing module capable of implementing an encoding module or a decoding module in which various aspects and embodiments are implemented; Figure 2B Shows a block diagram of an example of a first system in which various aspects and embodiments are implemented; Figure 2C Shows a block diagram of an example of a second system in which various aspects and embodiments are implemented; Figure 3 Shows the training phase of a video compression system based on a deep NN; Figure 4 Shows a video encoder of a video compression system based on a trained deep NN; Figure 5 Shows a video decoder of a video compression system based on a trained deep NN; Figure 6Illustrates the video encoding process of a video compression system based on a trained deep NN according to an embodiment; and, Figure 7 Illustrates the video decoding process of a video compression system based on a trained deep NN according to an embodiment. Detailed implementation

[0022] Figure 1 Schematically shows the context in which the embodiment is implemented.

[0023] In Figure 1 System 11, which may be a camera, storage device, computer, server, or any device capable of delivering video data, uses communication channel 12 to transmit video data to system 13. The video data is encoded and sent by system 11, or received and / or stored by system 11 and then sent. Communication channel 12 is a wired (such as the Internet or Ethernet) or wireless (such as WiFi, 3G, 4G, or 5G) network link.

[0024] System 13, which may be a set-top box for example, receives and decodes the video data to generate a sequence of decoded pictures.

[0025] Then, the obtained sequence of decoded images is transmitted to display system 15 using communication channel 14, which may be a wired or wireless network. Display system 15 then displays the pictures.

[0026] In one embodiment, system 13 is included in display system 15. In this case, system 13 and display 15 are included in a TV, computer, tablet, smartphone, head-mounted display, etc.

[0027] Figure 2A 、 2B And 2C describe examples of devices, apparatuses, and / or systems that allow the implementation of various embodiments.

[0028] Figure 2A Schematically shows an example of the hardware architecture of processing module 200 capable of implementing an encoding module or a decoding module, which can respectively implement the Figure 6 encoding method and Figure 7 decoding method. For example, when system 11 is responsible for encoding video data, the encoding module is included in this system. The decoding module is included in system 13 for example.

[0029] The processing module 200 connected through the communication bus 2005 includes: a processor or CPU (Central Processing Unit) 2000, which, by way of non-limiting example, includes one or more microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures; a random access memory (RAM) 2001; a read-only memory (ROM) 2002; a storage unit 2003, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives, or a storage medium reader, such as an SD (Secure Digital) card reader and / or a hard disk drive (HDD), and / or a network-accessible storage device; at least one communication interface 2004 for exchanging data with other modules, devices, or systems. The communication interface 2004 may include, but is not limited to, a transceiver configured to send and receive data through a communication channel. The communication interface 2004 may include, but is not limited to, a modem or a network card.

[0030] If the processing module 200 implements a decoding module, the communication interface 2004 enables the processing module 200, for example, to receive encoded video data and provide a decoded picture sequence. If the processing module 200 implements an encoding module, the communication interface 2004 enables the processing module 200, for example, to receive an original picture sequence for encoding and provide encoded video data.

[0031] The processor 2000 is capable of executing instructions loaded into the RAM 2001 from the ROM 2002, an external memory (not shown), a storage medium, or a communication network. When the processing module 200 is powered on, the processor 2000 is capable of reading instructions from the RAM 2001 and executing them. These instructions form a computer program that causes the processor 2000, for example, to implement the decoding method described in conjunction with Figure 7 and / or the encoding method described in conjunction with Figure 6 The all or some of the algorithms and steps of the described method.

[0032] Figure 6 and 7 can be implemented in software form by a programmable machine, such as a DSP (Digital Signal Processor) or a microcontroller, executing a set of instructions, or in hardware form by a machine or a dedicated component, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0033] It can be seen that microprocessors, general-purpose computers, special-purpose computers, processors based on or not based on multi-core architectures, DSPs, microcontrollers, FPGAs, and ASICs are suitable for at least partially implementing Figure 6 and 7 the methods described herein.

[0034] Figure 2C FIG. shows a block diagram of an example of a system 13 in which various aspects and embodiments are implemented. System 13 may be implemented as a device including the various components described below and is configured to perform one or more aspects and embodiments described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances, and head-mounted displays. The elements of system 13 may be embodied singly or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, system 13 includes a processing module 200 that implements a decoding module. In various embodiments, system 13 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 13 is configured to implement one or more aspects described herein.

[0035] As shown in block 231, inputs may be provided to the processing module 200 through various input modules. Such input modules include, but are not limited to, (i) a radio frequency (RF) module that receives, for example, RF signals transmitted over the air by a broadcaster, (ii) a component (COMP) input module (or a set of COMP input modules), (iii) a universal serial bus (USB) input module, and / or (iv) a high-definition multimedia interface (HDMI) input module. Figure 2C Other examples not shown include composite video.

[0036] In various embodiments, the input module of block 231 has corresponding input processing elements associated therewith, as known in the art. For example, an RF module may be associated with elements suitable for: (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal band to a frequency band), (ii) down-converting the selected signal, (iii) again band-limiting the narrower frequency band to select a signal band that may be referred to as a channel in certain embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) de-multiplexing to select a desired data packet stream. The RF module of various embodiments includes one or more elements that perform these functions, such as, for example, a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a de-multiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or to baseband. In one set-top box embodiment, the RF module and its associated input processing elements receive an RF signal transmitted through a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and again filtering to a desired frequency band. Various embodiments rearrange the order of the above (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as, for example, inserting an amplifier and an analog-to-digital converter. In various embodiments, the RF module includes an antenna.

[0037] In addition, the USB and / or HDMI module may include respective interface processors for connecting system 13 to other electronic devices via a USB and / or HDMI connection. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented, as needed, within, for example, a separate input processing IC or within processing module 200. Similarly, various aspects of USB or HDMI interface processing may be implemented, as needed, within a separate interface IC or within processing module 200. The streams of demodulation, error correction, and de-multiplexing are provided to processing module 200.

[0038] The various elements of system 13 may be provided within an integrated housing. Within the integrated housing, the various elements may be interconnected using suitable connection means and data may be transmitted therebetween, such as, for example, internal buses known in the art, including an inter-integrated circuit (I2C) bus, wiring, and printed circuit boards. For example, in system 13, processing module 200 is interconnected with the other elements of system 13 via bus 2005.

[0039] The communication interface 2004 of processing module 200 allows system 13 to communicate over communication channel 12. As described above, communication channel 12 may be implemented in, for example, a wired and / or wireless medium.

[0040] In various embodiments, a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers), is used to stream or otherwise provide data to system 13. The Wi-Fi signals of these embodiments are received via communication channel 12 and communication interface 2004 adapted for Wi-Fi communication. The communication channel 12 of these embodiments is typically connected to an access point or router that provides access to an external network, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use an RF connection of input block 231 to provide streaming data to system 13. As described above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0041] System 13 can provide output signals to various output devices, including display system 15, speaker 235, and other peripheral devices 236. The display system 15 of various embodiments includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display system 15 can be used in a television, a tablet, a laptop, a cell phone (mobile phone), a head-mounted display, or other devices. The display system 15 can also be integrated with other components (e.g., in a smart phone) or be standalone (e.g., an external monitor for a laptop). In various examples of embodiments, other peripheral devices 236 include one or more of a standalone digital video disc (or digital versatile disc) (DVR, for both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 236 that provide functions based on the output of system 13. For example, the disc player performs the function of playing the output of system 13.

[0042] In various embodiments, control signals are transmitted between system 13 and display system 15, speaker 235, or other peripheral devices 236 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols capable of enabling device-to-device control with or without user intervention. The output devices can be communicatively coupled to system 13 via dedicated connections through corresponding interfaces 232, 233, and 234. Alternatively, the output devices can use communication channel 12 via communication interface 2004 or use a corresponding one via communication interface 2004 corresponding to Figure 2CThe dedicated communication channel of communication channel 12 therein is connected to system 13. The display system 15 and the speaker 235 can be integrated into a single unit with other components of system 13 in an electronic device (such as, for example, a television). In various embodiments, the display interface 232 includes a display driver, such as, for example, a timing controller (TCon) chip.

[0043] Alternatively, the display system 15 and the speaker 235 can be separated from one or more of the other components. In various embodiments where the display system 15 and the speaker 235 are external components, the output signal can be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0044] Figure 2B A block diagram of an example of system 11 in which various aspects and embodiments are implemented is shown. System 11 is very similar to system 13. System 11 can be implemented as a device including the various components described below and is configured to perform one or more aspects and embodiments described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, cameras, and servers. The elements of system 11 can be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, system 11 includes a processing module 200 that implements an encoding module. In various embodiments, system 11 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 11 is configured to implement one or more aspects described herein.

[0045] The input to the processing module 200 can be provided through various input modules, as shown in block 231 already referenced Figure 2C as described.

[0046] The various elements of system 11 can be provided within an integrated housing. Within the integrated housing, the various elements can be interconnected using suitable connection means and data can be transmitted between them, for example, internal buses known in the art, including the inter-integrated circuit (I2C) bus, wiring, and printed circuit boards. For example, in system 11, the processing module 200 is interconnected with the other elements of system 11 through bus 2005.

[0047] The communication interface 2004 of the processing module 200 allows system 11 to communicate on communication channel 12.

[0048] In various embodiments, a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers), is used to stream or otherwise provide data to system 11. The Wi-Fi signals of these embodiments are received via a communication channel 12 and a communication interface 2004 adapted for Wi-Fi communication. The communication channel 12 of these embodiments is typically connected to an access point or a router that provides access to an external network (including the Internet) to allow streaming applications and other over-the-top communications. Other embodiments use an RF connection of input block 231 to provide streaming data to system 11.

[0049] As described above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0050] The data provided to system 11 can be provided in different formats. In various embodiments, this data is encoded and conforms to known video compression formats, such as AV1, VP9, VVC, HEVC, AVC, etc. In various embodiments, this data is, for example, raw data provided by a picture and / or audio acquisition module connected to or included in system 11. In such a case, the processing module 200 is responsible for encoding this data.

[0051] System 11 can provide an output signal to various output devices such as system 13 that are capable of storing and / or decoding the output signal.

[0052] Various implementations involve decoding. "Decoding" as used in this application can include, for example, all or part of a process performed on received video data to produce a final output suitable for display. In various embodiments, such a process includes a decoding process based on a deep neural network and a decoding process based on a deep superprior neural network.

[0053] Based on the context of the specific description, whether the phrase "decoding process" is intended to specifically refer to a subset of operations or to a more general decoding process will be clear and is considered well understood by those skilled in the art.

[0054] Various implementations involve encoding. In a manner similar to the discussion above regarding "decoding", "encoding" as used in this application can include, for example, all or part of a process performed on an input video sequence to produce encoded video data. In various embodiments, such a process includes an encoding process based on a deep neural network and an encoding process based on a deep superprior neural network.

[0055] Based on the context of the specific description, it will be clear whether the phrase "encoding process" is intended to specifically refer to a subset of operations or to a more general encoding process, and is considered well understood by those skilled in the art.

[0056] When a figure is represented as a flowchart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flowchart of the corresponding method / process.

[0057] Various embodiments relate to rate-distortion tradeoffs. In particular, during the training of deep-NN-based compression methods, a balance or tradeoff between rate and distortion is typically considered. Rate-distortion optimization is typically formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. NN-based compression methods have trainable parameters found by gradient-based optimization and initially defined hyperparameters. There are different ways to find these hyperparameters to minimize the rate-distortion optimization problem. For example, the method can be based on extensive testing of all deep neural network parameter values and a complete evaluation of the encoding cost and the associated distortion of the reconstructed signal after encoding and decoding. Faster methods can also be used to save computational complexity, especially by calculating an approximate distortion based on the predicted or prediction residual signal rather than the reconstructed signal. A hybrid of these two methods can also be used, such as by using approximate distortion for only some possible deep-NN network parameters and full distortion for other deep-NN network parameters. Other methods only evaluate a subset of the possible encoding options. More generally, many methods employ any of a variety of techniques to perform the optimization, but the optimization does not necessarily involve a complete evaluation of both the encoding cost and the associated distortion.

[0058] The implementations and aspects described herein can be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even when discussed in the context of only a single implementation form (e.g., only as a method), the implementation of the discussed features can be implemented in other forms (e.g., an apparatus or a program). The apparatus can be implemented in, for example, appropriate hardware, software, and firmware. These methods can be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes communication devices, such as, for example, a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate information communication between end users.

[0059] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" and other variations thereof mean that the particular features, structures, characteristics, etc. described in connection with that embodiment are included in at least one embodiment. Thus, the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations that occur throughout the application do not necessarily all refer to the same embodiment.

[0060] In addition, this application may refer to "determining" various pieces of information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, retrieving information from a memory, or obtaining information from another device, module, or user.

[0061] In addition, this application may refer to "accessing" various pieces of information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0062] In addition, this application may refer to "receiving" various pieces of information. Like "accessing", receiving is a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). In addition, "receiving" is typically involved in one way or another during operations such as storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0063] It should be understood that the use of any of the following, " / ", "and / or", and "at least one of...", "one or more of...", for example, in the cases of "A / B", "A and / or B", and "at least one of A and B", "one or more of A and B", is intended to include only the first-listed option (A), or only the second-listed option (B), or both options (A and B). As a further example, in the cases of "A, B, and / or C" and "at least one of A, B, and C", "one or more of A, B, and C", such language is intended to include only the first-listed option (A), or only the second-listed option (B), or only the third-listed option (C), or only the first and second-listed options (A and B), or only the first and third-listed options (A and C), or only the second and third-listed options (B and C), or all three options (A and B and C). As will be clear to those of ordinary skill in the art and related fields, this can be extended to any number of items listed.

[0064] In addition, as used herein, the word "signal" particularly refers to indicating something to the corresponding decoder. For example, in some embodiments, the encoder signals a parameter or a step size. In this way, in one embodiment, the same parameters can be used on the encoder side and the decoder side. Thus, for example, the encoder can send (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. Conversely, if the decoder already has specific parameters as well as other parameters, signaling can be used without sending (implicit signaling) to simply allow the decoder to know and select the specific parameters. By avoiding the transmission of any actual functionality, bit savings are achieved in various embodiments. It should be understood that signaling can be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to the corresponding decoder. Although the foregoing relates to the verb form of the word "signal", the word "signal" can also be used as a noun herein.

[0065] It will be apparent to those of ordinary skill in the art that embodiments can generate a variety of signals that are formatted to carry information such as can be stored or transmitted. The information can include, for example, instructions for performing a method or data generated by one of the described embodiments. For example, a signal can be formatted to carry the encoded video data of the described embodiments. Such a signal can be formatted as, for example, an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or a baseband signal. Formatting can include, for example, encoding the video data and modulating a carrier with the encoded video data. The information carried by the signal can be, for example, analog or digital information. The signal can be transmitted over various known wired or wireless links. The signal can be stored on a processor-readable medium.

[0066] Figure 3 The training phase of a deep-NN-based video compression system is shown.

[0067] Figure 3 The training phase is performed, for example, by the processing module 200.

[0068] In step 301, the processing module 200 applies a deep-NN-based encoding g a to an input picture x ∈ R of size n×n with three components n×n×3 . The output y of the deep-NN-based encoding (where y ∈ R m×m×o ) is referred to as the main latent (or main embedding) of the input image x.

[0069] In step 303, the processing module 200 applies quantization Q(.) to the main latent y to obtain the main code of the input picture x For example, during Figure 3 the quantization is scalar quantization.

[0070] In step 304, the processing module 200 applies inverse quantization (Q -1 ()) to the main code to obtain the reconstructed main latent

[0071] In step 302, the processing module 200 applies the encoding h a based on the deep hyperprior NN to the main latent y to obtain the side latent (or side embedding) z, where z ∈ R k×k×f . The encoding h a based on the deep hyperprior NN is h based on the hyperprior entropy model P

[0072] In step 305, the processing module 200 applies quantization (Q()) to the side latent z to obtain the side code

[0073] In step 306, the processing module 200 applies inverse quantization (Q -1 ()) to the side code to obtain the reconstructed side latent

[0074] The reconstructed side latent can be used to learn the probability model of the reconstructed main latent . Typically, the reconstructed main latent can be modeled by a Gaussian distribution, and the parameters of the Gaussian distribution (i.e., the mean μ and the standard deviation σ) are obtained by the decoder h s based on the deep hyperprior NN

[0075] In step 307, the processing module 200 applies the deep hyperprior decoder h s to the reconstructed side latent to obtain the parameters of the Gaussian distribution that models the main latent .

[0076] In step 308, the processing module 200 applies the decoder g s based on the deep NN to the reconstructed main latent to obtain the reconstructed picture

[0077] In Figure 3 the training phase shown, the factored entropy model p f can be used to calculate the lower bound of the bit length of the side code . The factored entropy model allows learning the side code from the reconstructed side latent The probability density function (PDF). Using this PDF, the probability mass function (PMF) of each side code can be calculated under a known quantization method (here scalar quantization). It can be noted that the PMF is the integral of the PDF from one boundary of the quantization range to the other boundary of the quantization range (assuming scalar quantization). The PMF values are sufficient to calculate the lower bound of the bit length of the side code. It can be noted that the PMF can be represented by a table called the PMF table. The probability mass function (PMF). It can be noted that the PMF is the integral of the PDF from one boundary of the quantization range to the other boundary of the quantization range (assuming scalar quantization). The PMF values are sufficient to calculate the lower bound of the bit length of the side code. The lower bound of the bit length.

[0078] In step 309, the processing module 200 applies the decomposed entropy model p f to the reconstructed side latent to determine the lower bound of the bit length of the side code which is directly obtained from the PMF (or PMF table). We show below that the lower bound of the bit length of the side code is used as the first rate information in the rate-distortion optimization that allows the training of the model parameters. It can be noted that the PMF obtained from the decomposed entropy model p is considered the learned PMF as they depend on the learned model. is used as the first rate information in the rate-distortion optimization that allows the training of the model parameters. It can be noted that the PMF obtained from the decomposed entropy model p f is considered the learned PMF as they depend on the learned model.

[0079] On the other hand, the hyperprior entropy model P h can be used to calculate the lower bound of the bit length of the main code . This model is typically implemented by a Gaussian distribution. Basically, it is assumed that each reconstructed main latent follows a Gaussian distribution, where the parameters (μ i , σ i ) are obtained using the side information in the previous step (or because and are directly related). The PMF of the Gaussian distribution can be obtained based on these parameters (μ i , σ i ) and the known quantization method. The parameters (μ i , σ i ) are considered learned parameters as they depend on the hyperprior entropy model P h which is a learned model. Similarly, the PMF of the Gaussian distribution is considered the learned PMF. Therefore, these Gaussian PMFs are sufficient to calculate the lower bound of the bit length of the main code.

[0080] In step 310, the processing module 200 uses the learned Gaussian PMF to determine the lower bound of the bit length of the main code which is As follows:

[0081] In fact, is directly obtained from the learned Gaussian PMF.

[0082] We show below that, in the rate-distortion optimization that allows training of the model parameters, the lower bound on the bit length of the main code is used as the second rate information.

[0083] During Figure 3 the process, the deep-NN-based encoder g a (or (g a (.; φ))), the deep-NN-based decoder g s (or (g s (.; θ))), the deep hyperprior-NN-based encoder h a (or (h a (.; Φ))), the deep hyperprior-NN-based decoder h s (or h s (.; Θ)) and the factorized entropy model p f (or p f (.; ω)) consist of multiple NN layers, such as convolutional layers. Each NN layer can be described as a function that first multiplies the input by a tensor, adds a vector called the bias, and then applies a non-linear function to the resulting value. The shape (and other properties) of the tensor and the type of non-linear function are referred to as the architecture of the network. We use the term weights to denote the values of the tensor and the bias. The weights and parameters (if applicable) of the non-linear function are called the parameters. The architecture and parameters define the model. In the deep-NN-based encoder g a (or (g a (.; φ))), the deep-NN-based decoder g s (or (g s (.; θ))), the deep hyperprior-NN-based encoder h a (or (h a (.; Φ))), the deep hyperprior-NN-based decoder h s (or h s (.; Θ)) and the factorized entropy model p f (or p f (.; ω)), the parameters of the models used are denoted by φ, θ, Φ, Θ, and ω, respectively.

[0084] The model must be trained on a huge picture database D to learn its parameters. Typically, the parameters of the model are optimized to minimize the training loss LOSS represented in Equation (Equation 1). where d(.,.) measures the distortion between the input picture and the reconstructed picture (e.g., d(.,.) is the mean squared error). In Equation 1, the rate term is the lower bound of the bit length of the side information (i.e., ) and the lower bound of the bit length of the main information (i.e., ). The hyperparameter λ controls the trade-off between the rate term and the distortion term.

[0085] Figure 4 illustrates the video encoding process of a trained deep-NN-based video compression system.

[0086] For example, the trained deep-NN-based video compression system is Figure 3 the system (i.e., a deep-NN-based video compression system with parameters φ, θ, Φ, Θ, and ω of the method using Figure 3 ). The video encoding process is executed, for example, by the processing module of System 11. In Figure 4 , for the steps already described in connection with Figure 3 , we keep the same reference numbers.

[0087] In step 301, the processing module 200 applies the deep-NN-based encoding g a to the input picture x.

[0088] In step 303, the processing module 200 applies the quantization Q(.) to the main latent y to obtain the main code

[0089] In step 302, the processing module 200 applies the deep hyperprior-NN-based encoding h a to the main latent y to obtain the side latent z.

[0090] In step 305, the processing module 200 applies quantization to the side latent z to obtain the side code

[0091] In step 306, the processing module 200 applies inverse quantization to the side code to obtain the reconstructed side latent

[0092] As described above, the reconstructed side latent is used to obtain the (Gaussian) probability model of the reconstructed main latent . In step 307, the processing module 200 applies the deep hyperprior decoder h s to the reconstructed side latent to obtain the parameters (μ of the Gaussian distribution that models the main latent i , σi )。The learned PMF (represented by the learned PMF table) is then derived from a Gaussian distribution that models the primary latent component.

[0093] In step 401, the processing module 200 applies arithmetic coding (AE) to the primary code to obtain the encoded primary information and inserts the encoded primary information into the video data (i.e., in the encoded video stream). In step 307, the AE uses the learned PMF table provided by the decoder based on the deep hyperprior NN.

[0094] In step 309, the processing module 200 applies the factorized entropy model p f to the side code to determine the learned PMF table for each side code under a known quantization method.

[0095] In step 402, the processing module 200 applies AE to the side code to obtain the encoded side information and inserts the encoded side information into the video data (i.e., in the encoded video stream). The AE in step 402 is driven by the learned PMF table calculated in step 309.

[0096] Figure 5 FIG. shows a video decoder of a trained deep-NN-based video compression system.

[0097] For example, the trained deep-NN-based video compression system is Figure 3 the system (i.e., the deep-NN-based video compression system with parameters φ, θ, Φ, Θ, and ω of the method that utilizes Figure 3 ). The video decoding process is performed, for example, by the processing module of system 13. In Figure 5 , for the steps already described in connection with Figure 3 , we keep the same reference numerals.

[0098] In step 501, the processing module 200 applies arithmetic decoding (AD) to the side information included in the video data (i.e., in the encoded video stream) to obtain the side code using the learned PMF table provided by the factorized entropy model in step 309.

[0099] In step 306, the processing module 200 applies inverse quantization to the side code to obtain the reconstructed side latent

[0100] In step 307, the processing module 200 applies the deep hyperprior decoder h s to the reconstructed side latent To obtain the main potential The parameters of the Gaussian distribution being modeled (μ i , σ i ), and derive the learned PMFs (in the form of learned PMF tables) from these Gaussian distributions.

[0101] In step 502, processing module 200 applies AD to the encoded main information contained in the video data using the learned PMF determined in step 307. These distributions (and the learned PMF table) instruct the AD how to read the main code from the bitstream.

[0102] In step 304, the processing module 200 applies inverse quantization to the main code To obtain the main potential of reconstruction

[0103] In step 308, the processing module 200 decodes the deep NN based on the s The main potential used for reconstruction to obtain the reconstructed image.

[0104] As stated in the introduction of this document, the purpose of the following embodiments is to study which data can be inferred from the explicitly encoded data available on the decoding side in a deep NN-based video compression method (also known as a fully NN-based video compression method), and how to use these inferred data to improve the compression efficiency of the deep NN-based video compression method.

[0105] A first example of data that can be inferred from explicitly encoded data contained in the video data is the entropy of the side information relative to the reconstructed side potential. The gradient of . This gradient can be easily calculated on the decoding side and is expressed as This gradient can give an idea of ​​how to change the potential of the reconstruction In order to increase or decrease their entropy.

[0106] A second example of data that can be inferred from explicitly encoded data contained in the video data is the entropy of the primary information relative to the reconstructed primary potential. The gradient of . This gradient can also be easily calculated on the decoding side and is expressed as This gradient can give how to change the main potential In order to increase or decrease their entropy thoughts.

[0107] So far, these two gradients have still not been used in the literature of deep NN based video compression methods.

[0108] In the following embodiments, these gradients of entropy with respect to the side latent and the main latent are used on the decoding side to improve compression efficiency. These embodiments utilize the correlation that exists between these two gradients and other useful gradients.

[0109] More specifically, in these embodiments, it is shown that: · The gradient of the entropy of the side information with respect to (w.r.t) the reconstructed side latent is correlated with the gradient of the entropy of the main information with respect to . Therefore, after obtaining the reconstructed side latent , an approximation of the gradient of the entropy of the main information with respect to can be derived, and thus the side latent can be shifted by a first step size according to the approximated gradient to reduce the bit length of the main information. · The gradient of the main information entropy with respect to the reconstructed main latent is correlated with the reconstruction quality (measured by ) between the input picture x and the output picture with respect to the reconstructed main latent . Therefore, after obtaining the reconstructed main latent , an approximation of the gradient of the reconstruction quality with respect to the reconstructed main latent can be derived. Thus, the main latent can be shifted by a second step size according to this approximated gradient to improve the reconstruction quality.

[0110] In the following, the optimal first and second step sizes are found on the encoding side, and these two step sizes are encoded in the video data so that they can be used in the decoding process.

[0111] In the following, we give some definitions, remarks, and theorems that allow the clarification of the various embodiments.

[0112] Prior art deep NN-based video compression methods typically use a loss function represented by Equation (Equation 1). This loss function can be regarded as an unconstrained multi-objective optimization problem, where the objectives are the minimum bit length of the side information the minimum bit length of the main information and the minimum reconstruction error

[0113] The following definition gives the concept of the optimal solution for multi-objective optimization: Definition: A Pareto optimal solution is an optimal solution where no objective can be made better without worsening at least one other objective. By using different importance weights for the objectives, we can obtain different solutions. If these solutions are Pareto optimal, they create a curve called the Pareto boundary curve.

[0114] Therefore, the aim of multi-objective optimization is to find Pareto optimal points on the Pareto boundary curve, where different points on the curve are obtained by a given weighting term λ. In Equation (Equation 1), there is no reason to privilege the first bit length term (i.e., the minimum bit length of the side information or the second bit length term (i.e., the minimum bit length of the main information . These two bit length terms are of equal importance, and their coefficients can be regarded as 1. However, λ will cause the optimizer to produce different solutions. If all these solutions are Pareto optimal, then these solutions lie on the Pareto boundary curve. In the field of image / video compression, this curve is usually called the rate-distortion curve.

[0115] The following remarks show useful properties of the solutions to unconstrained multi-objective optimization problems: Remark: The solution to a multi-objective optimization problem is Pareto optimal if and only if it satisfies the Karush-Kuhn-Tucker (KKT) conditions. More specifically, in the case of an unconstrained multi-objective optimization problem, if the objective of the problem is; where α i ≥ 0 and ∑ i α i = 1 and the objective function, then the solution w * is Pareto optimal if and only if the following conditions are satisfied:

[0116] In other words, this remark shows that at the optimal solution point, the gradients of all objectives should be in a fight with each other and no one can win more. All the forces driven by the gradients cancel each other out, and the solution reaches a saddle point. This property is used to test the optimality of candidate solutions.

[0117] This remark also applies to end-to-end compression models. The following theorem shows how to use the KKT conditions for end-to-end video compression systems, such as Figure 3 and Figure 4 deep NN-based compression systems.

[0118] Theorem: An end-to-end compression model optimized with λ is Pareto optimal if and only if the following two conditions are satisfied:

[0119] Proof: To show the similarity between the end-to-end loss and the unconstrained multi-objective optimization loss, we can rewrite the loss as follows: Here, And Since, λ > 0, Vi, α i > 0 and Therefore α i Satisfies the coefficients for unconstrained multi-objective optimization. Since our goal is not to change the parameters of the end-to-end model, but only to adjust the main latent and side latent So we keep all the parameters fixed, but As variables. Therefore, we can use To write the bit length objective of the side information, and use To write the bit length objective of the main information, and finally use To write the distortion objective. Therefore, the model can be written as an unconstrained multi-objective optimization problem: Since this problem has two sets of variables to optimize, both variables should satisfy the KKT conditions. Therefore, the variable For Has no effect on anything. Therefore We can write the KKT conditions of As follows: Since α1 = α2, they cancel each other out, and we reach the first condition in the theorem in equation (Equation 2) if we substitute their definitions for And Therefore, the second variable For Has no effect on anything, so We can write the KKT conditions of As follows: If we substitute the definitions of the objectives for the objectives and divide both sides by α2, and We reach the second condition in the theorem in equation (Equation 3).

[0120] The proof is straightforward and almost trivial. But the result of this theorem is meaningful. The above theorem can be interpreted as, if one of the existing end-to-end models is optimal (at least in terms of the training process, not in terms of compression performance), then the gradient of the entropy of the side information with respect to the side latent And the gradient of the entropy of the main information with respect to the side latent Can be expected to cancel each other out. This also holds for the second condition. We can state that the entropy of the main information with respect to the main latent And the weighted gradient of the reconstruction error with respect to the main latent They are expected to cancel each other out. However, equations (Equation 2) and (Equation 3) will be zero in all training sets rather than the average of any given single specific picture. Therefore, for a given single picture, their sum may not be exactly zero.

[0121] The following hypothesis is important for the following embodiments: Hypothesis: We assume that all existing end-to-end video compression models are well-trained and their solutions are Pareto optimal. The sum of the gradients in the theorem is zero for the training set and they are relevant for a given single picture.

[0122] To verify the above hypothesis, the correlation of the gradient pairs in the theorem was calculated. Tests have shown that the correlation between the gradients in equation (Equation 2) is not so clear for a given single picture, and thus has little impact on performance. However, for the latter case of equation (Equation 3), the correlation is quite clear. These test results motivate using the gradients available on the decoding side as a proxy for the useful gradients unknown on the decoding side.

[0123] If the side latent is shifted then it reduces the bit length of the main information because the gradient indicates how to shift so as to reduce by definition However, in the decoding device it is not available. Since we have shown that and are relevant, we use the available gradient as a proxy for the gradient : Proposal 1. After coding on the decoding side and obtaining the reconstructed side latent if we shift then there is a real step size that reduces the bit length of the main information such that: such that: ρ f can be found by forcing to find the optimal one from a few candidates or by any optimization method such that:

[0124] Furthermore, if we shift the main latent then the reconstruction error is reduced because the gradient indicates how to shift so as to reduce by definition However, it is not available in the decoding device. Since we have shown and is relevant, so we reuse the available gradient in the second proposal as the gradient proxy: Proposal 2. After decoding the main code and obtaining the reconstructed main latent if we shift there is a real step size that reduces the reconstruction error such that: ρ h can be found by forcing it from a few candidates, or by finding the optimal one through any optimization method, such that:

[0125] In the following embodiments, Proposal 1 and Proposal 2 can be implemented together or separately.

[0126] In one embodiment, finding the optimal step size is done by identifying the optimal step size from some predefined candidate list of step sizes, or by running non-linear optimization starting with p h = p f = 0 to find the optimal step size.

[0127] Figure 6 illustrates the video encoding process of a video compression system based on a trained deep NN in one embodiment.

[0128] Figure 6 The process is based on Proposals 1 and 2. The process is executed, for example, by the processing module 200 of the system 11.

[0129] In step 601, the processing module 200 obtains the input picture x.

[0130] In step 602, similar to step 301, the processing module 200 applies the deep-NN-based encoding g a to the input picture x to obtain the main latent y.

[0131] In step 603, similar to step 302, the processing module 200 applies the deep hyperprior-NN-based encoding h a to the main latent y to obtain the side latent z.

[0132] In step 604, similar to step 305, the processing module 200 applies quantization to the side latent z to obtain the side code

[0133] ​In step 605, similar to step 309, processing module 200 applies the decomposed entropy model p f to the side code to determine the learned PMF table.

[0134] In step 606, similar to step 402, processing module 200 applies AE to the side code to obtain the encoded side information and inserts the encoded side information into the video data. The AE in step 402 is driven by the learned PMF table calculated in step 309.

[0135] In step 607, similar to step 306, processing module 200 applies inverse quantization to the side code to obtain the reconstructed side latent

[0136] In step 608, processing module 200 determines the optimal first step size based on the gradient of the entropy of the side information with respect to the reconstructed side latent as follows:

[0137] In step 609, processing module 200 encodes the information representing the first step size in the video data. The first step size indicates how to shift the side latent.

[0138] In step 610, processing module 200 uses the first step size and the gradient of the entropy of the side information with respect to the reconstructed side latent to shift the side latent as follows:

[0139] In step 611, processing module 200 applies the deep hyperprior decoder h s to the shifted reconstructed side latent to obtain the parameters of the Gaussian distribution that models the main latent

[0140] In step 612, processing module 200 obtains the learned PMF of the main code using the Gaussian distribution that models the main latent

[0141] In step 613, similar to step 303, processing module 200 applies quantization Q(.) to the main latent y to obtain the main code

[0142] In step 614, processing module 200 applies AE to the main code ​​To obtain the encoded main information and insert the encoded main information into the video data. In step 612, the AE uses the learned PMF table provided by the decoder based on the deep hyperprior NN.

[0143] In step 615, similar to step 304, the processing module 200 applies inverse quantization to the main code to obtain the reconstructed main latent

[0144] In step 616, the processing module 200 determines the optimal second step size based on the gradient of the entropy of the main information with respect to the main latent as follows: as follows:

[0145] In step 617, the processing module 200 encodes the information representing the second step size in the video data The second step size indicates how to shift the main latent.

[0146] In Figure 6 the steps related to the first proposal are steps 608, 609, and 610, and the steps related to the second proposal are steps 615, 616, and 617. Figure 6 represents an embodiment implementing proposals 1 and 2.

[0147] If only proposal 1 is implemented, steps 615, 616, and 617 are skipped.

[0148] If only proposal 2 is implemented, steps 608, 609, and 610 are skipped, and in step 611, the reconstructed side latent is used instead of the shifted reconstructed side latent

[0149] Figure 7 illustrates the video decoding process of a video compression system based on a trained deep NN for an embodiment.

[0150] Figure 7 The process is based on proposals 1 and 2. The process is performed, for example, by the processing module 200 of system 13.

[0151] In step 701, similar to step 501, the processing module 200 applies arithmetic decoding (AD) to the side information included in the video data to obtain the side code using the learned PMF table provided by the decomposed entropy model in step 309.

[0152] In step 702, similar to step 306, the processing module 200 applies inverse quantization to the side code to obtain the reconstructed side latent

[0153] In step 703, the processing module 200 decodes information representing the first step size from the video data. from the video data.

[0154] In step 704, the processing module 200 uses the first step size and the gradient of the side information entropy with respect to the side latent to shift the side latent as follows:

[0155] In step 705, the processing module 200 applies the depth hyper prior decoder h s to the shifted side latent to obtain the parameters of the Gaussian distribution that models the main latent

[0156] In step 706, the processing module 200 uses the Gaussian distribution that models the main latent to obtain the learned PMF of the main code (in the form of a learned PMF table).

[0157] In step 707, similar to step 502, the processing module 200 applies the AD to the encoded main information included in the video data using the learned PMF table determined in step 706.

[0158] In step 708, similar to step 304, the processing module 200 applies inverse quantization to the main code to obtain the reconstructed main latent

[0159] In step 709, the processing module decodes information representing the second step size from the video data. from the video data.

[0160] In step 710, the processing module 200 uses the second step size and the gradient of the main information entropy with respect to the main latent to shift the main latent as follows:

[0161] In step 711, the processing module 200 applies the depth NN-based decoder g s to the shifted reconstructed main latent to obtain the reconstructed image.

[0162] In Figure 6 , the steps related to the first proposal are steps 703 and 704, and the steps related to the second proposal are steps 709 and 710. Figure 7 ​Represents an embodiment implementing Proposal 1 and 2.

[0163] If only Proposal 1 is implemented, steps 709 and 710 are skipped, and step 711 uses the reconstructed main latent instead of the shifted reconstructed main latent

[0164] If only Embodiment 2 is implemented, steps 703 and 704 are skipped, and step 705 uses the reconstructed side latent instead of the shifted reconstructed side latent

[0165] We have described many embodiments above. The features of these embodiments can be provided individually or in any combination. Additionally, the embodiments can include one or more of the following features, devices, or aspects in a separate or any combination across various claim categories and types: · A bitstream or signal including one or more of the main information, side information, and the first step size and / or the second step size or variations thereof. · Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal that includes one or more of the main information, side information, and the first step size and / or the second step size or variations thereof. · A TV, set-top box, mobile phone, tablet, or other electronic device that executes at least one of the embodiments. · A TV, set-top box, mobile phone, tablet, or other electronic device that executes at least one of the embodiments and displays (e.g., using a monitor, screen, or other type of display) the resulting picture. · A TV, set-top box, mobile phone, tablet, or other electronic device that tunes (e.g., using a tuner) a channel to receive a signal including an encoded video stream and executes at least one of the described embodiments. · A TV, set-top box, mobile phone, tablet, or other electronic device that receives (e.g., using an antenna) a signal including an encoded video stream over the air and executes at least one of the described embodiments. · A server, camera, mobile phone, tablet, or other electronic device that transmits (e.g., using an antenna) a signal including an encoded video stream over the air and executes at least one of the described embodiments. · A server, camera, mobile phone, tablet, or other electronic device that tunes (e.g., using a tuner) a channel to send a signal including an encoded video stream and executes at least one of the described embodiments.

Claims

1. A method, comprising: Obtain (601) an input picture; Apply (602) an encoder based on a deep neural network to the input picture to obtain a main latent; Apply (603) an encoder based on a deep hyperprior neural network to the main latent to obtain a side latent; Quantize (604) the side latent using a first quantization method to obtain a side code; Apply (605) a decomposed entropy model to the side code to determine a first probability mass function of the side code under the first quantization method; Arithmetic-code (606) the side code in the video data based on the first probability mass function; Inverse-quantize (607) the side code to obtain a reconstructed side latent; Determine (608) a first step size based on the gradient of the entropy of the side information with respect to the reconstructed side latent; Encode (609) information representing the first step size in the video data; Shift (610) the reconstructed side latent using the first step size and the gradient of the entropy of the side information with respect to the reconstructed side latent; Apply (611) a deep hyperprior decoder to the shifted reconstructed side latent to obtain information representing a probability distribution for modeling the main latent; Obtain (612) a second probability mass function of the main code using the probability distribution for modeling the main latent; Quantize (613) the main latent to obtain a main code; And, Arithmetic-code (614) the main code in the video data based on the second probability mass function.

2. The method according to claim 1, further comprising: Inverse-quantize (615) the main code to obtain a reconstructed main latent; Determine (616) a second step size based on the gradient of the main information entropy with respect to the main latent; And, Encode (617) information representing the second step size in the video data.

3. A method, comprising: Obtain (601) an input picture; Apply (602) an encoder based on a deep neural network to the input picture to obtain a main latent; Apply (603) an encoder based on a deep hyperprior neural network to the main latent to obtain a side latent; Quantize (604) the side latent using a first quantization method to obtain a side code; Apply (605) a decomposed entropy model to the side code to determine a first probability mass function of the side code under the first quantization method; Arithmetic-code (606) the side code in the video data based on the first probability mass function; Inverse-quantize (607) the side code to obtain a reconstructed side latent; Apply (611) a deep hyperprior decoder to the reconstructed side latent to obtain information representing a probability distribution for modeling the main latent; Obtain (612) a second probability mass function of the main code using the probability distribution for modeling the main latent; Arithmetic-code (614) the main code in the video data based on the second probability mass function; Inverse-quantize (615) the main code to obtain a reconstructed main latent; Determine (616) a second step size based on the gradient of the main information entropy with respect to the main latent; and, Encode (617) information representing the second step size in the video data.

4. A method, comprising: Apply (701) arithmetic decoding to side information included in the video data to obtain a side code using the first probability mass function provided by the decomposed entropy model; Inverse-quantize (702) the side code to obtain a reconstructed side latent; The entropy of the side information with respect to the gradient of the reconstructed side latent decodes (703) the information representing the first step size; Shift (704) the reconstructed side latent using the first step size and the entropy of the side information with respect to the gradient of the reconstructed side latent; Apply (705) the deep hyperprior decoder to the shifted reconstructed side latent to obtain the information representing the probability distribution modeling the main latent; Obtain (706) the second probability mass function of the main code using the probability distribution modeling the main latent; Arithmetically decode (707) the main code from the video data using the second probability mass function; Inverse quantize (708) the main code to obtain the reconstructed main latent; And, Apply (711) the deep neural network-based decoder to the reconstructed main latent to obtain the output picture.

5. The method according to claim 4, comprising: Decode (709) the information representing the second step size from the video data based on the entropy of the main information with respect to the gradient of the main latent; And, Shift (710) the reconstructed main latent using the second step size and the entropy of the main information with respect to the gradient of the main latent, and apply the deep neural network-based decoder to the shifted reconstructed main latent.

6. A method, comprising: Apply (701) arithmetic decoding to the side information included in the video data to obtain the side code using the first probability mass function provided by the factorized entropy model; Inverse quantize (702) the side code to obtain the reconstructed side latent; Apply (705) the deep hyperprior decoder to the reconstructed side latent to obtain the information representing the probability distribution modeling the main latent; Obtain (706) the second probability mass function of the main code using the probability distribution modeling the main latent; Arithmetically decode (707) the main code from the video data using the second probability mass function; Inverse quantize (708) the main code to obtain the reconstructed main latent; Decode (709) the information representing the second step size from the video data based on the entropy of the main information with respect to the gradient of the main latent; Shift (710) the reconstructed main latent using the second step size and the entropy of the main information with respect to the gradient of the main latent; And, Apply (711) the deep neural network-based decoder to the shifted reconstructed main latent to obtain the output picture.

7. An apparatus comprising an electronic circuit, the electronic circuit being configured to: Obtain (601) an input picture; Apply (602) an encoder based on a deep neural network to the input picture to obtain a main latent; Apply (603) an encoder based on a deep hyperprior neural network to the main latent to obtain a side latent; Quantize (604) the side latent using the first quantization method to obtain the side code; Apply (605) the factorized entropy model to the side code to determine the first probability mass function of the side code under the first quantization method; Arithmetically encode (606) the side code in the video data based on the first probability mass function; Inverse quantize (607) the side code to obtain the reconstructed side latent; Determine (608) the first step size based on the entropy of the side information with respect to the gradient of the reconstructed side latent; Encode (609) the information representing the first step size in the video data; Shift (610) the reconstructed side latent using the first step size and the entropy of the side information with respect to the gradient of the reconstructed side latent; Apply (611) the deep hyperprior decoder to the shifted reconstructed side latent to obtain the information representing the probability distribution modeling the main latent; Obtain (612) the second probability mass function of the main code using the probability distribution modeling the main latent; Quantize (613) the primary latent to obtain a primary code; And, Arithmetically encode (614) the primary code in the video data based on a second probability mass function.

8. The apparatus according to claim 7, wherein,The electronic circuit is further configured to: Inverse quantize (615) the primary code to obtain a reconstructed primary latent; Determine (616) a second step size based on the gradient of the primary information entropy with respect to the primary latent; and, Encode (617) information representing the second step size in the video data.

9. An apparatus comprising an electronic circuit, the electronic circuit being configured to: Obtain (601) an input picture; Apply (602) an encoder based on a deep neural network to the input picture to obtain a main latent; Apply (603) an encoder based on a deep hyperprior neural network to the main latent to obtain a side latent; Quantize (604) the side latent using a first quantization method to obtain a side code; Apply (605) a decomposed entropy model to the side code to determine a first probability mass function of the side code under the first quantization method; Arithmetically encode (606) the side code in the video data based on the first probability mass function; Inverse quantize (607) the side code to obtain a reconstructed side latent; Apply (611) a deep hyperprior decoder to the shifted reconstructed side latent to obtain information representing the probability distribution for modeling the primary latent; Obtain (612) a second probability mass function of the primary code using the probability distribution for modeling the primary latent; Arithmetically encode (614) the primary code in the video data based on the second probability mass function; Inverse quantize (615) the primary code to obtain a reconstructed primary latent; Determine (616) a second step size based on the gradient of the primary information entropy with respect to the primary latent; and, Encode (617) information representing the second step size in the video data.

10. An apparatus comprising an electronic circuit, the electronic circuit being configured to: Apply arithmetic decoding (701) to side information included in video data to obtain a side code using a first probability mass function provided by a factored entropy model; Inverse quantize (702) the side code to obtain a reconstructed side latent; Decode (703) information representing the first step size based on the gradient of the side information entropy with respect to the reconstructed side latent; Shift (704) the reconstructed side latent using the first step size and the gradient of the side information entropy with respect to the reconstructed side latent; Apply (705) a deep hyperprior decoder to the shifted reconstructed side latent to obtain information representing the probability distribution for modeling the primary latent; Obtain (706) a second probability mass function of the primary code using the probability distribution for modeling the primary latent; Arithmetically decode (707) the primary code from the video data using the second probability mass function; Inverse quantize (708) the primary code to obtain a reconstructed primary latent; And, Apply (711) a decoder based on a deep neural network to the reconstructed primary latent to obtain an output picture.

11. The apparatus according to claim 10, wherein, The electronic circuit is further configured to: Decode (709) information representing the second step size from the video data based on the gradient of the primary information entropy with respect to the primary latent; and, Shift (710) the reconstructed primary latent using the second step size and the gradient of the primary information entropy with respect to the primary latent, and apply the decoder based on the deep neural network to the shifted reconstructed primary latent.

12. An apparatus comprising an electronic circuit, the electronic circuit being configured to: Apply arithmetic decoding (701) to side information included in video data to obtain a side code using a first probability mass function provided by a factorized entropy model; Inverse quantize (702) the side code to obtain a reconstructed side latent; Apply (705) a deep hyperprior decoder to the reconstructed side latent to obtain information representing the probability distribution for modeling the primary latent; Obtain (706) a second probability mass function of the primary code using the probability distribution for modeling the primary latent; Arithmetically decode (707) the primary code from the video data using the second probability mass function; Inverse quantize (708) the main code to obtain the reconstructed main latent; Decode (709) information representing the second step size from the video data based on the gradient of the entropy of the main information with respect to the main latent; Shift (710) the reconstructed main latent using the second step size and the gradient of the main information entropy with respect to the main latent; And, Apply (711) a deep neural network-based decoder to the reconstructed main latent to obtain the output picture.

13. A computer program comprising program code instructions for implementing the method according to any one of the preceding claims 1 to 6.

14. A non-transitory information storage medium storing program code instructions for implementing the method according to any one of the preceding claims 1 to 6.

15. A signal produced by the method according to any one of the preceding claims 1 to 3 or by the apparatus according to any one of the preceding claims 7 to 9.

Images