Deep neural network-based image compression using latent shift based on latent entropy gradients

Abstract

A deep neural network based video compression system in which the gradient of entropy with respect to the side latent and the main latent is used at the decoding side to improve compression efficiency.

Description

[Technical Field]

[0001] At least one of these embodiments relates in general to a method and device for encoding and decoding picture data using a deep neural network, and more particularly to a method and device for utilizing the gradient of latent entropy to improve compression efficiency. [Background technology]

[0002] To achieve high compression efficiency, conventional video compression methods typically employ prediction and transformation techniques that leverage the spatial and temporal redundancy of video content. During encoding, the video content picture is divided into blocks of samples (i.e., pixels), which are then divided into one or more subblocks, hereafter referred to as original subblocks. Intra-prediction or inter-prediction is then applied to each subblock to utilize intra-image or inter-image correlations. Whatever prediction method (intra- or inter-) is used, a predictor subblock is determined for each original subblock. The subblock representing the difference between the original subblock and the predictor subblock is then transformed, quantized, and entropy-encoded to produce the encoded video stream, often referred to as the prediction error subblock, prediction residual subblock, or simply residual subblock. To reconstruct the video, the compressed data is decoded by the reverse process corresponding to the transformation, quantization, and entropy encoding.

[0003] In recently explored video encoding solutions, neural network (NN)-based processing has been studied. Several NN-based solutions have been proposed, ranging from hybrid solutions where NNs are used to implement some of the tools of the conventional video compression methods mentioned above, to fully NN-based compression solutions.

[0004] An important aspect of any video (or image) compression solution is to use as much information as possible available on the decoder side to improve compression efficiency. The information available on the decoder side naturally includes data explicitly encoded in the bitstream (i.e., in the encoded video data), but also includes data that is not explicitly encoded but can be inferred from the explicitly encoded video data. Such inferred data has been mainly used in conventional video compression schemes. However, fully NN-based compression methods are not as mature as methods based on conventional video compression schemes. Therefore, the possibility of inferring other data using explicitly encoded data remains an aspect to be explored.

[0005] It is desirable to determine which data can be inferred from the explicitly encoded data available at the decoder side in a fully NN-based video (or image) compression method, and how to use these inferred data to improve the compression efficiency of the fully NN-based video compression method. Summary of the Invention

[0006] In a first aspect, one or more of the present embodiments provide a method, the method comprising: obtaining an input picture, applying a deep neural network-based encoder to the input picture to obtain a main latent, applying a deep hyper prior neural network-based encoder to the input picture to obtain a side latent, quantizing the side latent using a first quantization method to obtain a side code, and determining a first probability mass function of the side code under the first quantization method, Factorized entropy modelapplying to a side code, arithmetically encoding the side code based on a first probability mass function in video data, dequantizing the side code to obtain a reconstructed side latent, determining a first step size based on the gradient of the entropy of side information with respect to the reconstructed side latent, encoding information representing the first step size in the video data, shifting the reconstructed side latent using the first step size and the gradient of the entropy of side information with respect to the reconstructed side latent, applying a deep hyper prior decoder to the reconstructed side latent to obtain information representing a probability distribution modeling a main latent, obtaining a second probability mass function for a main code using the probability distribution modeling the main latent, quantizing the main latent to obtain the main code, and arithmetically encoding the main code based on the second probability mass function in the video data.

[0007] In one embodiment, the method further comprises dequantizing the main code to obtain a reconstructed main latent, determining a second step size based on the gradient of the entropy of main information with respect to the main latent, and encoding information representing the second step size in the video data.

[0008] In a second aspect, one or more of the present embodiments provide a method, the method comprising, obtaining an input picture, applying a deep neural network-based encoder to the input picture to obtain a main latent, applying a deep hyper prior neural network-based encoder to the input picture to obtain a side latent, quantizing the side latent to obtain a side code using a first quantization method, in order to determine a first probability mass function of the side code under the first quantization method, Factorized Entropy ModelThe method further includes applying a side code, arithmetic coding the side code in video data based on a first probability mass function, inverse quantizing the side code to obtain a reconstructed side latent, applying a deep hyper pre-decoder to the reconstructed side latent to obtain information representing a probability distribution that models the main latent, obtaining a second probability mass function of the main code using the probability distribution that models the main latent, arithmetic coding the main code in video data based on a second probability mass function, inverse quantizing the main code to obtain a reconstructed main latent, determining a second step size based on the entropy gradient of the main information with respect to the main latent, and encoding information representing the second step size in video data.

[0009] In a third aspect, one or more of these embodiments provide a method, and the method is Factorized Entropy Model This includes: applying arithmetic decoding to side information contained in video data to obtain a side code using a first probability mass function provided by; inverse quantizing the side code to obtain a reconstructed side latent; decoding information representing a first step size based on the entropy gradient of the side information with respect to the reconstructed side latent; shifting the reconstructed side latent using the first step size and the entropy gradient of the side information with respect to the reconstructed side latent; applying a deep hyperpre-decoder to the reconstructed side latent to obtain information representing a probability distribution that models the main latent; obtaining a second probability mass function of the main code using the probability distribution that models the main latent; arithmetic decoding the main code from video data using the second probability mass function; inverse quantizing the main code to obtain a reconstructed main latent; and applying a deep neural network-based decoder to the reconstructed main latent to obtain an output picture.

[0010] In one embodiment, the method further comprises decoding information representing a second step size based on the entropy gradient of the main information to the main latent from video data, and shifting the reconstructed main latent using the second step size and the entropy gradient of the main information to the main latent, wherein a deep neural network-based decoder is applied to the shifted reconstructed main latent.

[0011] In the fourth aspect, one or more of these embodiments provide a method, and the method is Factorized Entropy Model The process includes: applying arithmetic decoding to side information contained in video data to obtain a side code using a first probability mass function provided by; inverse quantizing the side code to obtain a reconstructed side latent; applying a deep hyper pre-decoder to the reconstructed side latent to obtain information representing a probability distribution that models the main latent; obtaining a second probability mass function of the main code using the probability distribution that models the main latent; arithmetic decoding the main code from video data using the second probability mass function; inverse quantizing the main code to obtain a reconstructed main latent; decoding information representing a second step size based on the entropy gradient of the main information to the main latent from video data; shifting the reconstructed main latent using the second step size and the entropy gradient of the main information to the main latent; and applying a deep neural network-based decoder to the shifted reconstructed main latent to obtain an output picture.

[0012] In a fifth aspect, one or more of these embodiments provide a device comprising an electronic circuit, the electronic circuit is Get the input picture, To obtain the main latent, a deep neural network-based encoder is applied to the input picture. To obtain the side latent, a deep hyperpre-neural network-based encoder is applied to the input picture. Using the first quantization method, the side latent is quantized in order to obtain the side code. To determine the first probability mass function of the side code under the first quantization method, Factorized Entropy Model Apply this to the side code, In the video data, the side code is arithmetically coded based on the first probability mass function. To obtain the reconstructed side latent, we dequantize the side code, Based on the entropy gradient of the side information relative to the reconstructed side latent, the first step size is determined. Encode information representing the first step size in the video data. Using the first step size and the entropy gradient of the side information relative to the reconstructed side latent, the side latent is shifted. To obtain information representing the probability distribution that models the main latent, we apply a deep hyperpredecoder to the reconstructed side latent, Using the probability distribution that models the main latent, we obtain the second probability mass function of the main code. To obtain the main code, we quantize the main latent, It is configured to arithmetic encode the main code based on a second probability mass function in the video data.

[0013] In one embodiment, the electronic circuit is To obtain the reconstructed main latent, dequantize the main code, Based on the entropy gradient of the main information relative to the main latent, the second step size is determined. Encoding information representing the second step size in video data, It is further composed for this purpose.

[0014] In the sixth aspect, one or more of these embodiments provide a device comprising an electronic circuit, the electronic circuit is Get the input picture, To obtain the main latent, a deep neural network-based encoder is applied to the input picture. To obtain the side latent, a deep hyperpre-neural network-based encoder is applied to the input picture. Using the first quantization method, the side latent is quantized in order to obtain the side code. To determine the first probability mass function of the side code under the first quantization method, Factorized Entropy Model Apply this to the side code, In the video data, the side code is arithmetically coded based on the first probability mass function. To obtain the reconstructed side latent, we dequantize the side code, To obtain information representing the probability distribution that models the main latent, we apply a deep hyperpredecoder to the reconstructed side latent, Using the probability distribution that models the main latent, we obtain the second probability mass function of the main code. In the video data, the main code is arithmetically coded based on a second probability mass function. 、 To obtain the reconstructed main latent, dequantize the main code, Based on the entropy gradient of the main information relative to the main latent, the second step size is determined. Encoding information representing the second step size in video data, It is configured for that purpose.

[0015] In the seventh aspect, one or more of these embodiments provide a device comprising an electronic circuit, the electronic circuit is Factorized Entropy Model Using the first probability mass function provided by, arithmetic decoding is applied to the side information contained in the video data in order to obtain the side code. To obtain the reconstructed side latent, we dequantize the side code, Based on the entropy gradient of the side information relative to the reconstructed side latent, the information representing the first step size is decoded. Using the first step size and the entropy gradient of the side information relative to the reconstructed side latent, the reconstructed side latent is shifted. To obtain information representing the probability distribution that models the main latent, we apply a deep hyperpredecoder to the reconstructed side latent, Using the probability distribution that models the main latent, we obtain the second probability mass function of the main code. Using a second probability mass function, the main code is arithmetically decoded from the video data. To obtain the reconstructed main latent, dequantize the main code, To obtain the output picture, a deep neural network-based decoder is applied to the reconfigured main latent. It is configured for that purpose.

[0016] In one embodiment, the electronic circuit is Based on the entropy gradient of the main information relative to the main latent from the video data, information representing the second step size is decoded. The reconstructed main latent is shifted using the second step size and the entropy gradient of the main information relative to the main latent. Furthermore, a deep neural network-based decoder is applied to the shifted and reconstructed main latent.

[0017] In the eighth aspect, one or more of these embodiments provide a device comprising an electronic circuit, the electronic circuit is Factorized Entropy Model Using the first probability mass function provided by, arithmetic decoding is applied to the side information contained in the video data in order to obtain the side code. To obtain the reconstructed side latent, we dequantize the side code, To obtain information representing the probability distribution that models the main latent, we apply a deep hyperpredecoder to the reconstructed side latent, Using the probability distribution that models the main latent, we obtain the second probability mass function of the main code. Using a second probability mass function, the main code is arithmetically decoded from the video data. To obtain the reconstructed main latent, dequantize the main code, Based on the entropy gradient of the main information relative to the main latent from the video data, information representing the second step size is decoded. Using the second step size and the entropy gradient of the main information relative to the main latent, the reconstructed main latent is shifted. To obtain the output picture, a deep neural network-based decoder is applied to the shifted and reconstructed main latent. It is configured for that purpose.

[0018] In the ninth aspect, one or more of these embodiments provide a computer program including program code instructions for implementing the method according to the first, second, third, fourth, fifth, or sixth aspect.

[0019] In the tenth aspect, one or more of these embodiments provide a non-temporary information storage medium for storing program code instructions for implementing the method according to the first, second, third, fourth, fifth, or sixth aspect.

[0020] In the eleventh embodiment, one or more of these embodiments provide a signal generated by the method of the first, second, or third embodiment, or by the device of the seventh, eighth, or ninth embodiment. [Brief explanation of the drawing]

[0021] [Figure 1] This section provides a schematic example of the context in which the embodiment is implemented. [Figure 2A]This paper schematically illustrates an example of a hardware architecture for a processing module that can implement an encoding module or a decoding module, which can be implemented in various forms and embodiments. [Figure 2B] A block diagram illustrating an example of a first system in which various aspects and embodiments are implemented is provided. [Figure 2C] A block diagram illustrating an example of a second system in which various aspects and embodiments are implemented is provided. [Figure 3] This example illustrates the training phase of a deep neural network-based video compression system. [Figure 4] This example illustrates a video encoder based on a pre-trained deep neural network-based video compression system. [Figure 5] This example illustrates a video decoder based on a pre-trained deep neural network-based video compression system. [Figure 6] This document illustrates a video encoding process based on a pre-trained deep neural network-based video compression system of one embodiment. [Figure 7] This document illustrates a video decoding process based on a pre-trained deep neural network-based video compression system of one embodiment. [Modes for carrying out the invention]

[0022] Figure 1 schematically illustrates an example of a context in which the embodiment is implemented.

[0023] In Figure 1, System 11, which may be a camera, a storage device, a computer, a server, or any device capable of distributing video data, transmits video data to System 13 using a communication channel 12. The video data is encoded and transmitted by System 11, or received and / or stored by System 11 and then transmitted. The communication channel 12 is a wired (e.g., Internet or Ethernet) or wireless (e.g., WiFi, 3G, 4G, or 5G) network link.

[0024] System 13, which may be a set-top box for example, receives and decodes video data and generates a sequence of decoded pictures.

[0025] Next, the acquired and decoded sequence of pictures is transmitted to the display system 15 using a communication channel 14, which may be a wired or wireless network. The display system 15 then displays the pictures.

[0026] In this embodiment, system 13 is included in display system 15. In this case, system 13 and display 15 include TVs, computers, tablets, smartphones, head-mounted displays, etc.

[0027] Figures 2A, 2B, and 2C illustrate examples of devices, apparatus, and / or systems that enable the implementation of various embodiments.

[0028] Figure 2A schematically illustrates an example of a hardware architecture for a processing module 200 that can implement an encoding module or a decoding module, respectively, which can implement the encoding method shown in Figure 6 and the decoding method shown in Figure 7. The encoding module is included in system 11, for example, when this system is responsible for encoding video data. The decoding module is included in system 13, for example.

[0029] The processing module 200 includes, as a non-limiting example, one or more microprocessors, general-purpose computers, dedicated computers, and processors based on multi-core architectures connected by a communication bus 2005, a processor or CPU (central processing unit) 2000, random access memory (RAM) 2001, read-only memory (ROM) 2002, electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, magnetic disk drives, and / or optical disk drives, or SD (secure digital) card readers and / or hard disk drives. The system comprises a storage unit 2003 that may include non-volatile memory and / or volatile memory, including but not limited to a storage medium reader such as a drive, HDD, etc., and / or a network-accessible storage device, and at least one communication interface 2004 for exchanging data with other modules, devices, or systems. The communication interface 2004 may include, but is not limited to, a transceiver configured to transmit and receive data over a communication channel. The communication interface 2004 may include, but is not limited to, a modem or a network card.

[0030] If the processing module 200 implements a decoding module, the communication interface 2004 enables, for example, the processing module 200 to receive encoded video data and provide a sequence of decoded pictures. If the processing module 200 implements an encoding module, the communication interface 2004 enables, for example, the processing module 200 to receive and encode a sequence of original pictures and provide encoded video data.

[0031] The processor 2000 can execute instructions loaded into RAM 2001 from ROM 2002, external memory (not shown), a storage medium, or a communication network. When power is turned on to the processing module 200, the processor 2000 can read instructions from RAM 2001 and execute them. These instructions form a computer program to be executed by the processor 2000, for example, the decoding method described in relation to Figure 7 or the encoding method described in relation to Figure 6.

[0032] All or part of the algorithms and steps of the methods shown in Figures 6 and 7 can be implemented in software form by executing instruction sets by a programmable machine such as a DSP (digital signal processor) or microcontroller, or in hardware form by a machine or dedicated component such as an FPGA (field-programmable gate array) or ASIC (application-specific integrated circuit).

[0033] As can be seen, microprocessors, general-purpose computers, dedicated computers, processors based on or not based on multicore architectures, DSPs, microcontrollers, FPGAs, and ASICs are electronic circuits adapted to at least partially implement the methods shown in Figures 6 and 7.

[0034] Figure 2C illustrates a block diagram of an example of a system 13 in which various aspects and embodiments are implemented. System 13 can be embodied as a device including various components described later and is configured to perform one or more aspects and embodiments described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and head-mounted displays. The elements of system 13 can be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or separate components. For example, in at least one embodiment, system 13 comprises a processing module 200 that implements a decoding module. In various embodiments, system 13 is communicably coupled to one or more other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, system 13 is configured to implement one or more aspects described herein.

[0035] Inputs to the processing module 200 can be provided through various input modules as shown in block 231. Such input modules include, but are not limited to, (i) a radio frequency (RF) module for receiving RF signals wirelessly transmitted from a broadcasting station, for example, (ii) a component (COMP) input module (or a set of COMP input modules), (iii) a Universal Serial Bus (USB) input module, and / or (iv) a High Definition Multimedia Interface (HDMI) input module. Another example, although not shown in Figure 2C, is composite video.

[0036] In various embodiments, the input module of block 231 has associated input processing elements, as known in the art. For example, the RF module may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) in a particular embodiment, band-limiting again to a narrower frequency band in order to select a signal frequency band which may be referred to as a channel, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) multiplexing to select a desired stream of data packets. The RF module of various embodiments includes one or more elements that perform these functions, e.g., frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion may include a tuner that performs various functions of these functions, e.g., down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a frequency close to the baseband) or to the baseband. In one embodiment of a set-top box, an RF module and its associated input processing elements receive an RF signal transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering it to a desired frequency band. Various embodiments may rearrange the order of the above (and other) elements, omit some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as inserting an amplifier and an analog-to-digital converter. In various embodiments, the RF module includes an antenna.

[0037] Furthermore, the USB module and / or HDMI module may include their respective interface processors for connecting the system 13 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, can be implemented, for example, in a separate input processing IC or, if necessary, in the processing module 200. Similarly, aspects of USB or HDMI interface processing can be implemented, if necessary, in a separate interface IC or within the processing module 200. Demodulation, error correction, and multiplexed streams are provided to the processing module 200.

[0038] Various elements of system 13 can be provided within an integrated housing. Within the integrated housing, the various elements can be interconnected using internal buses known in the art, such as inter-IC (I2C) buses, wiring, and printed circuit boards, and data can be transmitted between them. For example, in system 13, processing module 200 is interconnected with other elements of system 13 by bus 2005.

[0039] The communication interface 2004 of the processing module 200 enables the system 13 to communicate over the communication channel 12. As already mentioned above, the communication channel 12 can be implemented, for example, within a wired and / or wireless medium.

[0040] In various embodiments, data is streamed to system 13 or otherwise provided using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signals in these embodiments are received via a communication channel 12 and a communication interface 2004 adapted for Wi-Fi communication. The communication channel 12 in these embodiments is typically connected to an access point or router that provides access to an external network, including the Internet, to enable streaming applications and other over-the-top communications. In other embodiments, streaming data is provided to system 13 using the RF connection of input block 231. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, e.g., cellular networks or Bluetooth networks.

[0041] System 13 can provide output signals to various output devices, including a display system 15, a speaker 235, and other peripheral devices 236. In various embodiments, the display system 15 includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display system 15 may be for a television, tablet, laptop, mobile phone, head-mounted display, or other device. The display system 15 may also be integrated with other components (e.g., as in a smartphone) or separate (e.g., an external monitor for a laptop). In various examples of embodiments, the other peripheral devices 236 include one or more of a standalone digital video disc (or digital multi-purpose disc) (both terms, DVR), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 236 that provide functions based on the output of System 13. For example, a disc player performs the function of playing back the output of System 13.

[0042] In various embodiments, control signals are communicated between the system 13 and the display system 15, speaker 235, or other peripheral devices 236 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable control between devices with or without user intervention. Output devices can be coupled to the system 13 communicably via dedicated connections through their respective interfaces 232, 233, and 234. Alternatively, output devices can be connected to the system 13 using communication channel 12 via communication interface 2004, or using a dedicated communication channel corresponding to communication channel 12 in Figure 2C via communication interface 2004. The display system 15 and speaker 235 can be integrated into a single unit with other components of the system 13 in an electronic device such as a television. In various embodiments, the display interface 232 includes a display driver, such as a timing controller (TCon) chip.

[0043] The display system 15 and speaker 235 can, alternatively, be separated from one or more of the other components. In various embodiments where the display system 15 and speaker 235 are external components, output signals can be provided via dedicated output connections, such as an HDMI port, a USB port, or a COMP output.

[0044] Figure 2B illustrates a block diagram of an example of system 11 in which various aspects and embodiments are implemented. System 11 is very similar to system 13. System 11 can be embodied as a device including various components described later and is configured to perform one or more aspects and embodiments described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, cameras, and servers. The elements of system 11 can be embodied individually or in combination as a single integrated circuit (IC), multiple ICs, and / or separate components. For example, in at least one embodiment, system 11 comprises one processing module 200 that implements an encoding module. In various embodiments, system 11 is communicably coupled to one or more other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, system 11 is configured to implement one or more aspects described herein.

[0045] The input to the processing module 200 can be provided through various input modules, as shown in block 231, which has already been described with respect to Figure 2C.

[0046] Various elements of system 11 can be provided within an integrated housing. Within the integrated housing, the various elements can be interconnected using internal buses known in the art, such as inter-IC (I2C) buses, wiring, and printed circuit boards, and data can be transmitted between them. For example, in system 11, processing module 200 is interconnected to other elements of system 11 by bus 2005.

[0047] The communication interface 2004 of the processing module 200 enables the system 11 to communicate over the communication channel 12.

[0048] In various embodiments, data is streamed to system 11 or otherwise provided using a wireless network such as a Wi-Fi network, for example, IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). In these embodiments, the Wi-Fi signal is received via a communication channel 12 and a communication interface 2004 adapted for Wi-Fi communication. In these embodiments, the communication channel 12 is typically connected to an access point or router that provides access to an external network, including the Internet, to enable streaming applications and other over-the-top communications. In other embodiments, the RF connection of input block 231 is used to provide streaming data to system 11.

[0049] As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0050] The data provided to system 11 can be provided in different formats. In various embodiments, this data is encoded and conforms to known video compression formats such as AV1, VP9, ​​VVC, HEVC, and AVC. In various embodiments, this data is raw data provided, for example, by a picture and / or audio acquisition module connected to or included in system 11. In this case, the processing module 200 is responsible for encoding this data.

[0051] System 11 can provide output signals to various output devices, such as System 13, which can store and / or decode output signals.

[0052] Various implementations involve decoding. As used in this application, “decoding” can encompass all or part of the processing performed on the received video data to produce a final output suitable for display. In various embodiments, such processes include a deep neural network-based decoding process and a deep hyper-pre-neural network-based decoding process.

[0053] Whether the phrase "decryption process" is intended to refer to a specific subset of operations or to a broader decoding process as a whole will become clear from the context of the specific description and should be well understood by those skilled in the art.

[0054] Various implementations involve encoding. As with the above considerations regarding "decoding," when used in this application, "encoding" can encompass all or part of the processes performed on an input video sequence to generate encoded video data. In various embodiments, such processes include deep neural network-based encoding processes and deep hyperpre-neural network-based encoding processes.

[0055] Whether the phrase "encoding process" is intended to refer to a specific subset of operations or to a broader encoding process as a whole will become clear from the context of the specific description and should be well understood by those skilled in the art.

[0056] When a diagram is presented as a flowchart, it should be understood that the diagram also provides a block diagram of the corresponding device. Similarly, when a diagram is presented as a block diagram, it should be understood that the diagram also provides a flowchart of the corresponding method / process.

[0057] Various embodiments refer to rate-distortion tradeoffs. In particular, a balance or tradeoff between rate and distortion is usually considered during training of deep NN-based compression methods. Rate-distortion optimization is typically formulated to minimize a rate-distortion function, which is a weighted sum of rate and distortion. An NN-based compression method has trainable parameters found by gradient-based optimization and initially defined hyperparameters. To minimize the rate-distortion optimization problem, there are various approaches to finding these hyperparameters. For example, these approaches are obtained based on extensive testing of all deep neural network parameter values, involving a complete evaluation of their coding costs, as well as the associated distortions of the reconstructed signals after coding and decoding. Alternatively, to reduce computational complexity, faster approaches can be used, particularly the calculation of approximate distortion based on prediction or prediction residual signals rather than the reconstructed signals. A mixture of these two approaches can also be used, such as using approximate distortion for only some of the possible deep NN network parameters and full distortion for other deep NN network parameters. Other approaches evaluate only a subset of the possible coding options. More generally, many approaches employ one of several techniques to perform optimization, but optimization is not necessarily a complete evaluation of both the encoding cost and the associated distortions.

[0058] The implementations and embodiments described herein may be implemented, for example, as methods or processes, apparatus, software programs, data streams, or signals. Even if considered only in the context of a single form of implementation (for example, considered only as a method), the implementation of the considered feature may also be implemented in other forms (for example, apparatus or programs). For example, an apparatus may be implemented in appropriate hardware, software, and firmware. A method may be implemented, for example, in a processor, where a processor refers to a general processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. A processor also includes communication devices such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate the communication of information between end users.

[0059] The phrases "one embodiment" or "an embodiment," or "one implementation" or "an implementation," and references to other variations thereof, mean that the specific features, structures, characteristics, etc. described in relation to that embodiment are included in at least one embodiment. Therefore, where the phrases "in one embodiment" or "in an embodiment," or "in one implementation" or "in an implementation," and other variations appear in various places throughout this application, they do not necessarily all refer to the same embodiment.

[0060] In addition, this application may refer to "determining" various types of information. Determining information may include, for example, estimating information, calculating information, predicting information, retrieving information from memory, or retrieving information from another device, module, or user, for example.

[0061] Furthermore, this application may refer to "accessing" various types of information. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0062] In addition, this application may refer to "receiving" various types of information. Receiving is intended to be a broad term, similar to "accessing." Receiving information may include, for example, accessing information or retrieving information (for example, from memory). Furthermore, "receiving" is typically involved in some way during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0063] The use of any of the following phrases, for example, "A / B", "A and / or B", "at least one of A and B", or "one or more of A and B", should be understood as being intended to include the selection of only the first listed option (A), or only the second listed option (B), or both options (A and B). As further examples, in the case of "A, B, and / or C" and "at least one of A, B, and C", or "one or more of A, B, and C", such phrases are intended to include the selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). This can be extended to the number of items listed, as will be obvious to those skilled in the art in the relevant and related fields.

[0064] Furthermore, as used herein, the term “signaling” specifically means indicating something to the corresponding decoder. For example, in certain embodiments, the encoder signals a parameter or step size. In this way, in embodiments, the same parameter can be used on both the encoder and decoder sides. Thus, for example, the encoder can transmit a specific parameter to the decoder so that the decoder can use the same specific parameter (explicit signaling). On the other hand, if the decoder already has other parameters along with that specific parameter, it can use signaling without transmission (implicit signaling) so that the decoder simply knows and can select that specific parameter. Bit saving is achieved in various embodiments by avoiding the transmission of any actual function. It will be understood that signaling can be achieved in various ways. For example, one or more syntactic elements, flags, etc., are used in various embodiments to signal information to the corresponding decoder. The above relates to the verb form of the term “signal,” but the term “signal” may also be used as a noun herein.

[0065] As will be apparent to those skilled in the art, implementations can generate a variety of signals formatted to carry information that can be stored or transmitted. This information may include, for example, instructions for carrying out a method, or data generated by one of the implementations described. For example, a signal can be formatted to carry encoded video data of the embodiment described. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding the video data and modulating the carrier wave with the encoded video data. The information carried by the signal may be, for example, analog or digital information. As is well known, signals can be transmitted over a variety of different wired or wireless links. The signals can be stored in a processor-readable medium.

[0066] Figure 3 illustrates the training phase of a deep neural network-based video compression system.

[0067] The training phase shown in Figure 3 is performed, for example, by the processing module 200.

[0068] In step 301, the processing module 200 performs deep NN-based coding g a This is an input picture x ∈ R of size n × n having three components. n×n×3 This applies to the output y (y ∈ R) of the deep NN-based coding. m×m×o The input picture x is called the main latent (or main embedding).

[0069] In step 303, the processing module 200 applies the quantization Q(.) to the main latent y and the main code of the input picture x.

[0070]

number

[0071] [Math] For example, in the process of Figure 3, the quantization is scalar quantization.

[0072] In step 304, the processing module 200 performs dequantization(Q -1 ()) on the main code

[0073] [Math] to obtain the reconstructed main latent

[0074] [Math] is obtained

[0075] [Math]

[0076] In step 302, the processing module 200 applies deep hyper-prior NN-based encoding h a to the main latent y to obtain the side latent (or side embedding) z, where z ∈ R k×k×f . The deep hyper-prior NN-based encoding h a is based on the hyper-prior entropy model P h .

[0077] In step 305, the processing module 200 applies quantization(Q()) to the side latent z to obtain the side code

[0078] [Math] is obtained.

[0079] In step 306, the processing module 200 performs dequantization(Q -1()) side chord

[0080]

number

[0081]

number

[0082] Reconstructed side potential

[0083]

number

[0084]

number

[0085]

number

[0086] In step 307, the processing module 200 performs a deep hyper predecoder h s Reconfigured side potential

[0087]

number

[0088]

number

[0089] In step 308, the processing module 200 uses a deep NN-based decoder g s Reconfigured main potential

[0090]

number

[0091]

number

[0092] In the training phase illustrated in Figure 3, side code

[0093]

number

[0094]

number

[0095]

number

[0096]

number

[0097]

number

[0098] In step 309, the processing module 200, Factorized Entropy Model p f reconstructed side potential

[0099]

number

[0100]

number

[0101]

number

[0102]

number

[0103]

number

[0104]

number

[0105] Meanwhile, the main code

[0106]

number

[0107]

number

[0108]

number

[0109]

number

[0110] In step 310, the processing module 200 uses the trained Gaussian PMF to execute the main code

[0111]

number

[0112]

number

[0113]

number

[0114] actually,

[0115]

number

[0116] Main Code

[0117]

number

[0118]

number

[0119] In the process shown in Figure 3, deep NN-based coding g a (or (g a (.;φ)))), deep NN-based decoding s (or (g s(.;θ))) Deep hyper pre-NN based coding h a (or (h a (.;φ))) Deep hyper pre-NN based decoding h s (or h s (.;Θ)), and Factorized Entropy Model p f (or p f (.;ω)) is composed of multiple NN layers, such as convolutional layers. Each NN layer can be described as a function that first multiplies the input by a tensor, adds a vector called a bias, and then applies a nonlinear function to the resulting value. The shape (and other properties) of the tensor and the type of nonlinear function are called the network architecture. The values ​​of the tensor and bias are represented in the weights term. The weights and, if applicable, the parameters of the nonlinear function are called parameters. The architecture and parameters define the model. Deep NN-based coding g a (or (g a (.;φ))), deep NN-based decoding s (or (g s (.;θ))) Deep hyper pre-NN based coding h a (or (h a (.;φ))) Deep hyper pre-NN based decoding h s (or h s (.;Θ)), and Factorized Entropy Model p f or p f p represented by (.;ω) f (or p f The parameters of the model used in (.;ω)) are denoted by φ, θ, φ, Θ, and ω, respectively.

[0120] The model must be trained on a large database D of pictures to learn its parameters. Typically, the model's parameters are optimized to minimize the training loss, expressed by equation (Equation 1).

[0121]

number

[0122]

number

[0123]

number

[0124]

number

[0125]

number

[0126] Figure 4 illustrates a video encoding process based on a trained deep neural network-based video compression system.

[0127] For example, a trained deep NN-based video compression system is shown in Figure 3 (i.e., a deep NN-based video compression system with parameters φ, θ, φ, Θ, and ω, according to the method in Figure 3). The video coding process is performed, for example, by a processing module in system 11. In Figure 4, the same reference numbers are used for steps already described in relation to Figure 3.

[0128] In step 301, the processing module 200 performs deep NN-based coding g a Apply to input picture x.

[0129] In step 303, the processing module 200 applies the quantized Q(.) to the main latent y, and the main code

[0130]

number

[0131] In step 302, the processing module 200 performs deep hyper pre-NN-based coding h a Apply this to the main latent y to obtain the side latent z.

[0132] In step 305, the processing module 200 applies quantization to the side latent z and the side code

[0133]

number

[0134] In step 306, the processing module 200 performs inverse quantization side code

[0135]

number

[0136]

number

[0137] As already mentioned above, reconstructed side potential

[0138]

number

[0139]

number

[0140]

number

[0141]

number

[0142]

number

[0143]

number

[0144] In step 401, the processing module 200 performs arithmetic encoding (AE) on the main code.

[0145]

number

[0146] In step 309, the processing module 200, Factorized Entropy Model p f Side code

[0147]

number

[0148]

number

[0149] In step 402, the processing module 200 sidecodes the AE.

[0150]

number

[0151] Figure 5 illustrates a video decoder based on a trained deep neural network-based video compression system.

[0152] For example, a trained deep NN-based video compression system is shown in Figure 3 (i.e., a deep NN-based video compression system with parameters φ, θ, φ, Θ, and ω, according to the method in Figure 3). The video decoding process is performed, for example, by the processing module of system 13. In Figure 5, the same reference numbers are used for steps already described in relation to Figure 3.

[0153] In step 501, the processing module 200 applies arithmetic decoding (AD) to the side information contained in the video data (i.e., the encoded video stream) in step 309. Factorized Entropy Model Use the pre-trained PMF table provided by to obtain the side code.

[0154] In step 306, the processing module 200 performs inverse quantization side code

[0155]

number

[0156]

number

[0157] In step 307, the processing module 200 performs a deep hyper predecoder h s Reconfigured side potential

[0158]

number

[0159]

number

[0160] In step 502, the processing module 200 applies AD to the encoded main information contained in the video data using the trained PMF determined in step 307. These distributions (and trained PMF tables) indicate to AD how to read the main code from the bitstream.

[0161] In step 304, the processing module 200 performs inverse quantization in the main code

[0162]

number

[0163] [Formula] is obtained.

[0164] In step 308, the processing module 200 applies the deep NN-based decoding g s to the reconstructed main latent

[0165] [Formula] to obtain a reconstructed picture.

[0166] As recalled in the introduction of this specification, the purpose of the following embodiments is to study which data can be inferred from the explicitly encoded data available at the decoding side in a deep NN-based video compression method (also referred to as a full NN-based video compression method), and how to use these inferred data to improve the compression efficiency of the deep NN-based video compression method.

[0167] A first example of data that can be inferred from the explicitly encoded data contained in video data is the reconstructed side latent

[0168] [Formula] the entropy gradient of the side information for. This gradient of interest

[0169] [Formula] can be easily calculated on the decoding side. This gradient is used to increase or decrease the entropy for the reconstructed side latent

[0170] [Formula] can give an idea of how to make changes.

[0171] A second example of data that can be inferred from explicitly encoded data included in video data is the reconstructed main latent

[0172] [Formula] is the entropy gradient of the main information with respect to. This gradient of interest

[0173] [Formula] can also be easily calculated on the decoding side. This gradient can give an idea of how to modify the main latent to increase or decrease the entropy

[0174] [Formula] can give an idea of how to make changes.

[0175] Hitherto, these two gradients have remained unused in the literature of deep NN-based video compression methods.

[0176] In the following embodiments, these gradients of entropy for the side latent and the main latent are used on the decoding side to improve compression efficiency. These embodiments exploit the correlations that exist between these two gradients and other useful gradients.

[0177] More specifically, in these embodiments, the following is shown. · Reconstructed side latent

[0178] [Formula] The entropy gradient of the side information with respect to is

[0179]

number

[0180]

number

[0181]

number

[0182]

number

[0183]

number

[0184]

number

[0185]

number

[0186] In the following, the best first and second step sizes are found on the encoding side, and the two step sizes are encoded in the video data so that they can be used in the decoding process.

[0187] In the following, the inventors provide several definitions, views, and theorems that will enable the clarification of various embodiments.

[0188] State-of-the-art deep neural network-based video compression methods generally use a loss function represented by equation (Equation 1). This loss function can be viewed as an unconstrained multi-objective optimization problem, the objective being the minimum bit length of the side information.

[0189]

number

[0190]

number

[0191]

number

[0192] The following definitions give an idea of ​​what the optimal solution is for multi-objective optimization.

[0193] Definition: A Pareto optimal solution is an optimal solution in which at least one objective cannot be improved without worsening it. Different solutions can be obtained by using different importance weights for the objectives. If these solutions are Pareto optimal, they form a curve called the Pareto frontier curve.

[0194] Therefore, the goal of multi-objective optimization is to find the Pareto optimal point on the Pareto frontier curve, where different points on the curve are obtained by a given weighted term λ. In equation (Equation 1), the first bit length term (i.e., the minimum bit length of the side information)

[0195]

number

[0196]

number

[0197] The following observations demonstrate useful properties of solutions to unconstrained multi-objective optimization problems.

[0198] Viewpoint: A solution to a multi-objective optimization problem is Pareto optimal only if it satisfies the Karush-Kuhn-Tucker (KKT) condition. More specifically, in the case of an unconstrained multi-objective optimization problem, if the problem's objective is as follows:

[0199]

number

[0200]

number

[0201]

number

[0202] In other words, this view shows that at the optimal solution point, all objective gradients compete, and no one can win any further. All forces driven by gradients cancel each other out, and the solution reaches a saddle point. This property is used to test the optimality of candidate solutions.

[0203] This view is also valid for end-to-end compression models. The following theorem shows how to use the KKT condition for end-to-end video compression systems, such as the deep NN-based compression systems in Figures 3 and 4.

[0204] Theorem: An end-to-end compression model optimized with a lambda tradeoff is Pareto optimal only if the following two conditions are met.

[0205]

number

[0206] Proof: To demonstrate the similarity between end-to-end loss and unconstrained multi-objective optimization loss, the loss can be rewritten as follows:

[0207]

number

[0208]

number

[0209]

number

[0210]

number

[0211]

number

[0212]

number

[0213]

number

[0214]

number

[0215]

number

[0216] This problem has two sets of variables to be optimized, so both sets of variables should satisfy the KKT condition.

[0217]

number

[0218]

number

[0219]

number

[0220]

number

[0221]

number

[0222] Since α1 = α2, these cancel each other out.

[0223]

number

[0224] Second variable

[0225]

number

[0226]

number

[0227]

number

[0228]

number

[0229]

number

[0230] Replace the objective with those definitions, and define the two edges as α2,

[0231]

number

[0232] The proof is simple and almost trivial. However, the result of this theorem is important. The above theorem states that if one of the existing end-to-end models is optimal (not in terms of compression performance, but at least in terms of the training procedure), then the side latent

[0233]

number

[0234]

number

[0235]

number

[0236]

number

[0237] The following hypothesis is important for the following embodiments. Hypothesis: Assume that all existing end-to-end video compression models are well trained and their solutions are Pareto optimal. The sum of gradients in the theorem is zero for the training set and they are correlated for a given single picture.

[0238] To test the above hypothesis, the correlation between pairs of gradients in the theorem is calculated. The tests show that the correlation between gradients in equation (Equation 2) is not very clear for a given single picture and therefore has a weak impact on performance. However, in the latter case of equation (Equation 3), the correlation is very clear. These test results provide an incentive to use gradients available on the decoding side as a substitute for useful gradients that are unknown on the decoding side.

[0239] Side potential

[0240]

number

[0241]

number

[0242]

number

[0243]

number

[0244]

number

[0245]

number

[0246]

number

[0247]

number

[0248] Proposal 1. Decode the side code and reconstruct the side latent

[0249]

number

[0250]

number

[0251]

number

[0252]

number

[0253]

number

[0254] ρ f The optimal solution can be found from a small number of candidates by obvious force or by any optimization method, and the following can be found:

[0255]

number

[0256] In addition, main potential

[0257]

number

[0258]

number

[0259]

number

[0260]

number

[0261]

number

[0262]

number

[0263]

number

[0264]

number

[0265] Proposal 2. Decode the main code and reconstruct the main latent

[0266]

number

[0267]

number

[0268]

number

[0269]

number

[0270]

number

[0271] ρ h The optimal solution can be found from a small number of candidates by obvious force or by any optimization method, and the following can be found:

[0272]

number

[0273] In the following embodiments, proposals 1 and 2 can be implemented together or individually.

[0274] In one embodiment, finding the best step size is done by identifying the best step size from a predefined list of step sizes, or by performing nonlinear optimization p h =p f This is done by finding the best step size starting with =0.

[0275] Figure 6 illustrates a video encoding process based on one embodiment of a trained deep neural network-based video compression system.

[0276] The process in Figure 6 is based on proposals 1 and 2. The process in Figure 6 is executed, for example, by the processing module 200 of system 11.

[0277] In step 601, the processing module 200 obtains the input picture x.

[0278] In step 602, which is the same as step 301, the processing module 200 performs deep NN-based coding g a Apply this to input picture x to obtain the main potential y.

[0279] In step 603, which is the same as step 302, the processing module 200 performs deep hyper pre-NN-based coding h aApply this to the main latent y to obtain the side latent z.

[0280] In step 604, which is the same as step 305, the processing module 200 applies quantization to the side latent z and the side code

[0281]

number

[0282] In step 605, which is the same as step 309, the processing module 200, Factorized Entropy Model p f Side code

[0283]

number

[0284] In step 606, which is the same as step 402, the processing module 200 sidecodes the AE.

[0285]

number

[0286] In step 607, which is the same as step 306, the processing module 200 performs inverse quantization side code

[0287]

number

[0288]

number

[0289] In step 608, the processing module 200 processes the reconfigured side latent as follows:

[0290]

number

[0291]

number

[0292]

number

[0293] In step 609, the processing module 200 determines the first step size in the video data.

[0294]

number

[0295] In step 610, the processing module 200 performs the first step size as follows:

[0296]

number

[0297]

number

[0298]

number

[0299] In step 611, the processing module 200 performs a deep hyper pre-decoder h s Shifted and reconfigured side potential

[0300]

number

[0301]

number

[0302] In step 612, the processing module 200 is the main latent

[0303]

number

[0304] In step 613, which is the same as step 303, the processing module 200 applies the quantized Q(.) to the main latent y and the main code

[0305]

number

[0306] In step 614, the processing module 200 sends the AE to the main code

[0307]

number

[0308] In step 615, which is the same as step 304, the processing module 200 performs inverse quantization in the main code

[0309]

number

[0310]

number

[0311] In step 616, the processing module 200 performs the following main side latent

[0312]

number

[0313]

number

[0314]

number

[0315] In step 617, the processing module 200 determines the second step size in the video data.

[0316]

number

[0317] In Figure 6, the steps related to the first proposal are steps 608, 609, and 610, and the steps related to the second proposal are steps 615, 616, and 617. Figure 6 shows embodiments in which proposals 1 and 2 are implemented.

[0318] If only Proposal 1 is implemented, steps 615, 616, and 617 are skipped.

[0319] If only Proposal 2 is implemented, steps 608, 609, and 610 are skipped, and step 611 is performed on the reconfigured side potential.

[0320]

number

[0321]

number

[0322] Figure 7 illustrates a video decoding process based on a trained deep neural network-based video compression system of one embodiment.

[0323] The process shown in Figure 7 is based on proposals 1 and 2. This process is executed, for example, by the processing module 200 of system 13.

[0324] In step 701, which is the same as step 501, the processing module 200 applies arithmetic decoding (AD) to the side information contained in the video data, in step 309 Factorized Entropy Model Use the pre-trained PMF table provided by to obtain the side code.

[0325] In step 702, which is the same as step 306, the processing module 200 performs inverse quantization side code

[0326]

number

[0327]

number

[0328] In step 703, the processing module 200 processes the video data using a first step size

[0329]

number

[0330] In step 704, the processing module 200 performs the first step size as follows:

[0331]

number

[0332]

number

[0333]

number

[0334] In step 705, the processing module 200 performs a deep hyper predecoder h s Shifted side potential

[0335]

number

[0336]

number

[0337] In step 706, the processing module 200 is the main latent

[0338]

number

[0339] In step 707, which is the same as step 502, the processing module 200 applies AD to the encoded main information contained in the video data using the learned PMF table determined in step 706.

[0340] In step 708, which is the same as step 304, the processing module 200 performs inverse quantization in the main code

[0341]

number

[0342]

number

[0343] In step 709, the processing module processes the video data using a second step size.

[0344]

number

[0345] In step 710, the processing module 200 performs the second step size as follows:

[0346]

number

[0347]

number

[0348]

number

[0349] In step 711, the processing module 200 performs deep NN-based decoding. s Shifted and reconfigured main potential

[0350]

number

[0351] In Figure 6, the steps related to the first proposal are steps 703 and 704, and the steps related to the second proposal are steps 709 and 710. Figure 7 shows embodiments in which proposals 1 and 2 are implemented.

[0352] If only Proposal 1 is implemented, steps 709 and 710 are skipped, and step 711 is performed on the reconfigured main potential.

[0353]

number

[0354]

number

[0355] If only Proposal 2 is implemented, steps 703 and 704 are skipped, and step 705 is performed on the reconfigured side potential.

[0356]

number

[0357]

number

[0358] Several embodiments have been described above. The features of these embodiments can be provided individually or in any combination. Furthermore, embodiments may include, individually or in any combination, one or more of the following features, devices, or aspects, spanning various categories and types of claims. A bitstream or signal containing the main information, side information, and one or more of the first step size and / or second step size, or variations thereof. To create and / or transmit and / or receive and / or decode a bitstream or signal containing the main information, side information, and one or more of the first step size and / or second step size, or variations thereof. A television, set-top box, mobile phone, tablet, or other electronic device that performs at least one of the embodiments described. A television, set-top box, mobile phone, tablet, or other electronic device that performs at least one of the embodiments described and displays the resulting picture (for example, using a monitor, screen, or other type of display). A television, set-top box, mobile phone, tablet, or other electronic device that tunes a channel (for example, using a tuner) to receive a signal containing an encoded video stream and performs at least one of the embodiments described. A television, set-top box, mobile phone, tablet, or other electronic device that wirelessly receives a signal containing an encoded video stream (for example, using an antenna) and performs at least one of the embodiments described. A server, camera, mobile phone, tablet, or other electronic device that wirelessly receives a signal containing an encoded video stream (for example, using an antenna) and performs at least one of the embodiments described. A server, camera, mobile phone, tablet, or other electronic device that tunes a channel (for example, using a tuner) to transmit a signal containing an encoded video stream and performs at least one of the embodiments described.

Claims

1. It is a method, To obtain the input picture, In order to obtain the main latent, a deep neural network-based encoder is applied to the input picture, In order to obtain the side latent, a deep hyperpre-neural network-based encoder is applied to the main latent, Using the first quantization method, the side latent is quantized in order to obtain the side code, In order to determine the first probability mass function of the side code under the first quantization method, a factorized entropy model is applied to the side code, In order to obtain the side information encoded in the video data, the side code is arithmetically encoded based on the first probability mass function, In order to obtain the reconstructed side latent, the side code is dequantized, The first step size is determined based on the entropy gradient of the side information relative to the reconstructed side latent, The video data includes encoding information representing the first step size, Shifting the reconstructed side latent using the first step size and the gradient of the entropy of the side information with respect to the reconstructed side latent, In order to obtain information representing the probability distribution that models the main latent, a deep hyperpredecoder is applied to the shifted and reconstructed side latent, Using the probability distribution that models the main latent, the second probability mass function of the main code is obtained, In order to obtain the aforementioned main code, the main latent is quantized, In order to obtain the main information encoded in the video data, the main code is arithmetically encoded based on the second probability mass function, Methods that include...

2. In order to obtain the reconstructed main latent, the main code is dequantized, The second step size is determined based on the entropy gradient of the main information relative to the main latent, The video data includes encoding information representing the second step size, The method according to claim 1, further comprising:

3. It is a method, To obtain the input picture, In order to obtain the main latent, a deep neural network-based encoder is applied to the input picture, In order to obtain the side latent, a deep hyperpre-neural network-based encoder is applied to the main latent, Using the first quantization method, the side latent is quantized in order to obtain the side code, In order to determine the first probability mass function of the side code under the first quantization method, a factorized entropy model is applied to the side code, In order to obtain the side information encoded in the video data, the side code is arithmetically encoded based on the first probability mass function, In order to obtain the reconstructed side latent, the side code is dequantized, In order to obtain information representing the probability distribution that models the main latent, a deep hyperpredecoder is applied to the reconstructed side latent, Using the probability distribution that models the main latent, the second probability mass function of the main code is obtained, In order to obtain the main information encoded in the video data, the main code is arithmetically encoded based on the second probability mass function, In order to obtain the reconstructed main latent, the main code is dequantized, The second step size is determined based on the entropy gradient of the main information relative to the main latent, The video data includes encoding information representing the second step size, Methods that include...

4. It is a method, Using the first probability mass function provided by the factorized entropy model, arithmetic decoding is applied to the side information contained in the video data in order to obtain the side code, In order to obtain the reconstructed side latent, the side code is dequantized, Decoding information representing a first step size from the video data, wherein the first step size is based on the gradient of the entropy of the side information relative to the reconstructed side latent, Shifting the reconstructed side latent using the first step size and the gradient of the entropy of the side information with respect to the reconstructed side latent, To obtain information representing the probability distribution that models the main latent, a deep hyperpredecoder is applied to the shifted and reconstructed side latent, Using the probability distribution that models the main latent, the second probability mass function of the main code is obtained, Using the second probability mass function described above, the main code is arithmetically decoded from the video data, To obtain the reconstructed main latent, we dequantize the main code, To obtain the output picture, a deep neural network-based decoder is applied to the reconstructed main latent, Methods that include...

5. Decoding information representing a second step size from the video data, wherein the second step size is based on the gradient of the entropy of the main information relative to the main latent. The reconstructed main latent is shifted using the second step size and the gradient of the entropy of the main information with respect to the main latent, wherein the deep neural network-based decoder is applied to the shifted reconstructed main latent. The method according to claim 4, including the method described in claim 4.

6. It is a method, Using the first probability mass function provided by the factorized entropy model, arithmetic decoding is applied to the side information contained in the video data in order to obtain the side code, In order to obtain the reconstructed side latent, the side code is dequantized, In order to obtain information representing the probability distribution that models the main latent, a deep hyperpredecoder is applied to the reconstructed side latent, Using the probability distribution that models the main latent, the second probability mass function of the main code is obtained, Using the second probability mass function described above, the main code is arithmetically decoded from the video data, In order to obtain the reconstructed main latent, the main code is dequantized, Decoding information representing a second step size from the video data, wherein the second step size is based on the gradient of the entropy of the main information relative to the main latent. Shifting the reconstructed main latent using the second step size and the gradient of the entropy of the main information with respect to the main latent, To obtain the output picture, a deep neural network-based decoder is applied to the shifted and reconstructed main latent, Methods that include...

7. A device comprising an electronic circuit, wherein the electronic circuit is Get the input picture, To obtain the main latent, a deep neural network-based encoder is applied to the input picture. To obtain the side latent, a deep hyperpre-neural network-based encoder is applied to the main latent. Using the first quantization method, the side latent is quantized in order to obtain the side code. To determine the first probability mass function of the side code under the first quantization method, a factorized entropy model is applied to the side code. In order to obtain the side information encoded in the video data, the side code is arithmetically encoded based on the first probability mass function, To obtain the reconstructed side latent, the side code is dequantized, Based on the entropy gradient of the side information relative to the reconstructed side latent, the first step size is determined. Information representing the first step size is encoded within the video data. The reconstructed side latent is shifted using the first step size and the gradient of the entropy of the side information relative to the reconstructed side latent. To obtain information representing the probability distribution that models the main latent, a deep hyperpredecoder is applied to the shifted and reconstructed side latent. Using the probability distribution that models the main latent, we obtain the second probability mass function of the main code. In order to obtain the aforementioned main code, the main latent is quantized, In order to obtain the main information encoded in the video data, the main code is arithmetically encoded based on the second probability mass function. A device configured for this purpose.

8. The aforementioned electronic circuit is To obtain the reconstructed main latent, the main code is dequantized, Based on the entropy gradient of the main information relative to the main latent, the second step size is determined. The video data is encoded with information representing the second step size. The device according to claim 7, further configured for the purpose of

9. A device comprising an electronic circuit, wherein the electronic circuit is Get the input picture, To obtain the main latent, a deep neural network-based encoder is applied to the input picture. To obtain the side latent, a deep hyperpre-neural network-based encoder is applied to the main latent. Using the first quantization method, the side latent is quantized in order to obtain the side code. To determine the first probability mass function of the side code under the first quantization method, a factorized entropy model is applied to the side code. In order to obtain the side information encoded in the video data, the side code is arithmetically encoded based on the first probability mass function, To obtain the reconstructed side latent, the side code is dequantized, To obtain information representing the probability distribution that models the main latent, a deep hyperpredecoder is applied to the reconstructed side latent. Using the probability distribution that models the main latent, we obtain the second probability mass function of the main code. In order to obtain the main information encoded in the video data, the main code is arithmetically encoded based on the second probability mass function, To obtain the reconstructed main latent, the main code is dequantized, Based on the entropy gradient of the main information relative to the main latent, the second step size is determined. The video data is encoded with information representing the second step size. A device configured for this purpose.

10. A device comprising an electronic circuit, wherein the electronic circuit is Using the first probability mass function provided by the factorized entropy model, arithmetic decoding is applied to the side information contained in the video data to obtain the side code. To obtain the reconstructed side latent, the side code is dequantized, Information representing the first step size is decoded from the video data, and the first step size is determined based on the gradient of the entropy of the side information relative to the reconstructed side latent. The reconstructed side latent is shifted using the first step size and the gradient of the entropy of the side information relative to the reconstructed side latent. To obtain information representing the probability distribution that models the main latent, a deep hyperpredecoder is applied to the shifted and reconstructed side latent. Using the probability distribution that models the main latent, we obtain the second probability mass function of the main code. Using the second probability mass function described above, the main code is arithmetically decoded from the video data. To obtain the reconstructed main latent, dequantize the main code, To obtain the output picture, a deep neural network-based decoder is applied to the reconstructed main latent. A device configured for this purpose.

11. The aforementioned electronic circuit is Information representing the second step size is decoded from the video data, and the second step size is determined based on the gradient of the entropy of the main information relative to the main latent. Using the second step size and the gradient of the entropy of the main information relative to the main latent, the reconstructed main latent is shifted, and the deep neural network-based decoder applies to the shifted reconstructed main latent. The device according to claim 10, further configured for the purpose of

12. A device comprising an electronic circuit, wherein the electronic circuit is Using the first probability mass function provided by the factorized entropy model, arithmetic decoding is applied to the side information contained in the video data to obtain the side code. To obtain the reconstructed side latent, the side code is dequantized, To obtain information representing the probability distribution that models the main latent, a deep hyperpredecoder is applied to the reconstructed side latent. Using the probability distribution that models the main latent, we obtain the second probability mass function of the main code. Using the second probability mass function described above, the main code is arithmetically decoded from the video data. To obtain the reconstructed main latent, the main code is dequantized, Information representing the second step size is decoded from the video data, and the second step size is determined based on the gradient of the entropy of the main information relative to the main latent from the video data. Using the second step size and the gradient of the entropy of the main information with respect to the main latent, the reconstructed main latent is shifted. To obtain the output picture, a deep neural network-based decoder is applied to the reconstructed main latent. A device configured for this purpose.

13. A non-temporary information storage medium for storing program code instructions for implementing the method according to claim 1 or 2.

14. A non-temporary information storage medium for storing program code instructions for implementing the method described in claim 3.

15. A non-temporary information storage medium for storing program code instructions for implementing the method according to claim 4 or 5.

16. A non-temporary information storage medium for storing program code instructions for implementing the method described in claim 6.

Images