Neural codec using hybrid entropy model and flexible quantization

Abstract

Innovations in systems, methods, and software for features of neural images or video codecs are described herein. For example, a neural video encoder may receive a current video frame, encode the current video frame to produce encoded data, and output the encoded data as part of a bitstream. As part of encoding, an encoder may determine a current implicit representation for a current video frame and encode the current implicit representation using an entropy model network comprising one or more convolutional layers. As part of encoding the current implicit representation, the encoder may estimate a statistical characteristic of a quantized version of the current implicit representation based at least in part on a previous implicit representation for a previous video frame, and entropy decode the quantized version of the current implicit representation based at least in part on the estimated statistical characteristic.

Description

Background Art

[0001] Engineers use compression (also called source codecs or source encoding) to reduce the bit rate of digital video. Compression reduces the cost of storing and transmitting video information by converting the information into a lower bit rate form. Decompression (also called decoding) reconstructs a version of the original information from the compressed form. A "codec" is an encoder / decoder system.

[0002] Over the past few decades, various video codec standards have been adopted, including ITU-TH.261, H.262 (MPEG-2 or ISO / IEC 13818-2), H.263, H.264 (MPEG-4 AVC or ISO / IEC 14496-10), H.265 / HEVC, H.266 / VVC (ISO / IEC 23090-3 or MPEG-IPart 3) standards, MPEG-1 (ISO / IEC 11172-2) and MPEG-4 Visual (ISO / IEC 14496-2) standards, and SMPTE 421M (VC-1) standards. Such video codec standards typically define options for the syntax of the encoded video bitstream, detailing the parameters in the bitstream when using specific features during encoding and decoding. In many cases, the video codec standard also provides detailed information on the decoding operations that the video decoder should perform to achieve a decoding result that complies with the specification. In addition to codec standards, various proprietary codec formats define additional options for the syntax of the encoded video bitstream and the corresponding decoding operations.

[0003] Recently, some codecs have used neural networks and other machine learning methods for data compression. For example, a neural image codec has been developed to compress / decompress images using an entropy model neural network (or "entropy model network", or simply "entropy model"), which is designed to predict the probability distribution of a quantized latent representation of an image. Based on similar concepts, a neural video codec has been developed to compress / decompress video frames using an entropy model. Despite the recent success of neural video codecs over traditional video compression / decompression techniques, there is still room for improvement in terms of improving compression quality and / or efficiency. Summary of the invention

[0004] In summary, innovations in efficient and high-quality codec technology are described herein. Some of the innovations described herein use an improved entropy model for a neural codec that can efficiently exploit spatial and temporal dependencies between video frames. Other innovations described herein provide a flexible quantization approach in a neural codec. As described more fully below, the innovations described herein include, but are not limited to, the following: incorporating previous implicit representations ("implicit priors") of previous video frames into an entropy model to exploit the correlation between implicit representations; incorporating cross-channel, cross-region predictions ("dual spatial priors") into an entropy model to exploit spatial redundancy in a parallel-friendly manner; incorporating a flexible quantization mechanism that supports multiple rates in a single neural codec system and improving rate-distortion ("RD") performance through dynamic bit allocation. The innovations described herein can be implemented in a neural video codec, and in some cases, in a neural image codec. The innovations described herein can be implemented for future codec standards or formats.

[0005] According to one aspect of the innovation described herein, a neural video encoder may receive a current video frame, encode the current video frame to produce encoded data, and output the encoded data as part of a bitstream. As part of the encoding, the encoder may determine a current implicit representation for the current video frame, and encode the current implicit representation using an entropy model network comprising one or more convolutional layers. As part of encoding the current implicit representation using the entropy model network, the encoder may estimate the statistical properties of a quantized version of the current implicit representation based at least in part on a previous implicit representation for a previous video frame, and entropy encode and decode the quantized version of the current implicit representation based at least in part on the estimated statistical properties. In some cases, using the previous implicit representation as input to the entropy model network helps to exploit temporal redundancy to improve the RD performance of the neural video encoder.

[0006] A corresponding neural video decoder may receive encoded data as part of a bitstream, decode the encoded data to reconstruct a current video frame, and output the reconstructed current video frame. As part of decoding, the decoder may reconstruct a current latent representation for the current video frame using an entropy model network comprising one or more convolutional layers. As part of reconstructing the current latent representation, the decoder may estimate statistical properties of a quantized version of the current latent representation based at least in part on a previous latent representation for a previous video frame, and entropy decode the quantized version of the current latent representation based at least in part on the estimated statistical properties.

[0007] According to another aspect of the innovation described herein, a neural image encoder or a neural video encoder may receive a current frame, encode the current frame to generate encoded data, and output the encoded data as part of a bitstream. As part of the encoding, the encoder may determine a current implicit representation for the current frame, and encode the current implicit representation using an entropy model network including one or more convolutional layers. The elements of the current implicit representation may be logically organized along a channel dimension and two spatial dimensions. As part of encoding the current implicit representation, the encoder may divide the elements of the current implicit representation into different channel sets along the channel dimension, and divide the elements of the current implicit representation into different spatial position sets along the two spatial dimensions to form a plurality of element sets, wherein each of the plurality of element sets has a different combination of a channel set in a different channel set and a spatial position set in a different spatial position set. Then, the encoder may estimate statistical properties of quantized versions of the plurality of element sets, respectively, including estimating statistical properties of quantized versions of the second element set among the plurality of element sets based at least in part on a quantized version of a first element set among the plurality of element sets. Additionally, the encoder may entropy encode and decode the quantized versions of the plurality of element sets, respectively, based at least in part on the estimated statistical properties. In some cases, using intersection-set estimates in entropy model networks helps exploit spatial redundancy (and potentially channel redundancy) to improve the RD performance of the neural encoder.

[0008] A corresponding neural image decoder or a neural video decoder may receive the encoded data as part of a bitstream, decode the encoded data to reconstruct a current frame, and output the reconstructed current frame. As part of the decoding, the decoder may reconstruct a current implicit representation for the current frame using an entropy model network including one or more convolutional layers. The elements of the current implicit representation may be logically organized along a channel dimension and two spatial dimensions. The elements of the current implicit representation have been divided into different channel sets along the channel dimension and divided into different spatial position sets along the two spatial dimensions to form a plurality of element sets. Each element set in the plurality of element sets has a different combination of one of the different channel sets and one of the different spatial position sets. As part of reconstructing the current implicit representation, the decoder may estimate statistical properties of quantized versions of the plurality of element sets, respectively, including estimating statistical properties of quantized versions of the second element set among the plurality of element sets, based at least in part on a quantized version of the first element set among the plurality of element sets. Additionally, the decoder may entropy decode the quantized versions of the plurality of element sets, respectively, based at least in part on the estimated statistical properties.

[0009] According to another aspect of the innovation described herein, a neural image encoder or a neural video encoder can receive a current frame, encode the current frame to produce encoded data, and output the encoded data as part of a bitstream. As part of the encoding, the encoder can determine a current implicit representation for the current frame. The elements of the current implicit representation are logically organized along a channel dimension and two spatial dimensions. As part of the encoding, the encoder can quantize the current implicit representation in multiple stages using different quantization step ("QS") values ​​in multiple stages, thereby producing a quantized version of the current implicit representation. Additionally, the encoder can entropy encode and decode the quantized version of the current implicit representation. In some cases, using multiple stages of quantization helps provide flexibility so that the neural encoder can be used across a range of QS values ​​for different quality and bitrate levels.

[0010] A corresponding neural image decoder or neural video decoder may receive the encoded data as part of a bitstream, decode the encoded data to reconstruct a current frame, and output the reconstructed current frame. As part of the decoding, the decoder may reconstruct a current implicit representation for the current frame. The elements of the current implicit representation are logically organized along a channel dimension and two spatial dimensions. As part of reconstructing the current implicit representation, the decoder may entropy decode a quantized version of the current implicit representation, and inverse quantize the quantized version of the current implicit representation in multiple stages using different QS values ​​in multiple stages, respectively.

[0011] These innovations can be implemented as part of a method, as part of a computer system configured to perform the operation of the method, or as part of one or more computer-readable media storing computer-executable instructions for causing a computer system to perform the operation of the method. Various innovations can be used in combination or alone. The present disclosure is provided to introduce the selection of concepts further described below in the specific embodiments in a simplified form. The present disclosure is not intended to identify the key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. The foregoing and other objects, features and advantages of the present invention will become more apparent through the following detailed description with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a diagram illustrating an example computer system in which some described embodiments may be implemented.

[0013] 2a and 2b are diagrams illustrating example network environments in which some described embodiments may be implemented.

[0014] Figure 3is a diagram illustrating an example neural video encoder in conjunction with which some described embodiments may be implemented, and an example neural video decoder in conjunction with which some described embodiments may be implemented.

[0015] Figure 4 is a diagram showing an example entropy model network for a neural codec, as well as segmentation, quantization, inverse quantization, and concatenation operations, which in conjunction with these operations may implement some of the described embodiments.

[0016] Figure 5A is a diagram showing features of an example entropy model network that accepts a prior latent representation as input.

[0017] Figure 5B is a diagram showing features of a hyper-prior for the current latent representation processed by an example entropy model network.

[0018] Figure 6 is a diagram illustrating features of multi-stage quantization and corresponding inverse quantization in a neural codec system in conjunction with which some described embodiments may be implemented.

[0019] Figure 7 is a set of screenshots illustrating the flexible quantization feature in the neural codec system.

[0020] Fig. 8A and Figure 8B are diagrams showing example network structures of a context encoder and a context decoder, respectively, for motion vector information in an example neural video codec system.

[0021] Fig.9A and Fig. 9B are diagrams showing example network structures of a context encoder and a context decoder, respectively, for sample value information in an example neural video codec system.

[0022] Fig. 10A and Fig. 10B are diagrams showing example network structures of a frame generator and a residual block with attention in an example neural video codec system, respectively.

[0023] Fig.11 is a diagram showing an example network structure of an entropy model network with prior latent representations as input and with cross-set estimation in an example neural video codec system.

[0024] Fig.12 is a diagram showing an example network structure for a temporal content encoder.

[0025] Fig.13A and Fig. 13Bare diagrams showing example network structures of a super-prior encoder and a super-prior decoder, respectively, for a current latent representation in an example neural video codec system.

[0026] Fig.14A and Fig. 14B are diagrams showing example network structures of a context encoder and a context decoder, respectively, for latent representation in an example neural image codec system.

[0027] Fig.15A and Fig. 15B are flow charts illustrating generalized techniques for encoding and decoding, respectively, for implementing one or more innovations described herein.

[0028] Fig.16A and Fig. 16B is a flow chart illustrating generalized techniques for determining and reconstructing a current latent representation, respectively, in some example embodiments.

[0029] Fig.17A and Fig. 17B is a flow chart illustrating a generalized technique for using a previous latent representation as input to an entropy model network during encoding and decoding, respectively, in some example embodiments.

[0030] Fig.18A and Fig.18B is a flow chart illustrating a generalized technique for using intersection set estimation in an entropy model network during encoding and decoding, respectively, in some example embodiments.

[0031] Fig.19A and Fig.19B are flow charts illustrating generalized techniques for multi-stage quantization and inverse quantization, respectively, in some example embodiments. DETAILED DESCRIPTION

[0032] This specific embodiment introduces innovations in efficient and high-quality codec technology using an improved entropy model and using flexible quantization, which can efficiently exploit both spatial and temporal dependencies between frames. As described more fully below, the innovations described herein include, but are not limited to, the following: incorporating implicit priors (e.g., previous implicit representations of sample value information or motion vector information) into the entropy model to exploit the correlation between implicit representations, thereby improving the RD performance of the neural codec system; incorporating dual spatial priors (e.g., dividing the elements of the implicit representation into multiple element sets for cross-set prediction / estimation pipelines) into the entropy model, thereby exploiting spatial redundancy between element sets in a parallel-friendly manner, thereby improving the RD performance of the neural codec system; incorporating a flexible quantization mechanism to achieve multi-rates in a single neural codec system and improve RD performance through dynamic bit allocation. The innovations described herein can be used in future video codec standards or formats.

[0033] In the examples described herein, the same reference numbers in different figures represent the same components, modules or operations. Depending on the context, a given component or module may accept different types of information as input and / or generate different types of information as output, or be processed in different ways.

[0034] More generally, there are various alternatives to the examples described herein. For example, some of the methods described herein can be modified by changing the order of the described method behaviors, splitting, repeating, or omitting certain method behaviors. Various aspects of the disclosed technology can be used in combination or alone. Different embodiments use one or more of the innovations described. Some of the innovations described herein solve one or more of the problems mentioned in the background technology. Generally, a given technology / tool ​​cannot solve all of these problems.

[0035] I. Overview of Neural Video Codec

[0036] Recent years have witnessed the development of neural image codec technology. Most neural image codec technologies focus on designing entropy models to predict the probability distribution of the quantized latent representation of an image, for example by using factorization models, hyper-priors, autoregressive priors, mixture Gaussian models, transformer-based models, etc. Benefiting from these continuously improving entropy models, the compression ratio of neural image codecs has been shown to be superior to more traditional image codec technologies, such as H.266 intra-frame coding. Inspired by the success of neural image codec technology, neural video codec technology has received increasing attention recently.

[0037] Most of the existing work on neural video codecs can be roughly divided into three categories: residual codec-based, conditional codec-based, and 3D autoencoder-based solutions. The residual codec method is derived from the traditional hybrid video codec architecture. Specifically, when encoding the current frame, a motion compensated prediction is first generated, and then the residual with the current frame is encoded and decoded. For solutions based on conditional codecs, the temporal frame or feature set for the previous frame serves as a condition for encoding and decoding the current frame. Compared with residual codecs, conditional codecs have been shown to have a lower or equal entropy upper limit. Solutions based on 3D autoencoders are a natural extension of neural image codec technology by expanding the input dimension. However, solutions based on 3D autoencoders may be associated with increased encoding delays and significantly increase memory costs. In general, most of these existing works focus on how to generate hidden representations of video frames by exploring different data streams or network structures. As for entropy models, most of these existing methods directly use off-the-shelf solutions (e.g., super priors, autoregressive priors, etc.) borrowed from neural image codec technology to encode and decode hidden representations for the current frame. Spatial-temporal correlations have not been fully exploited in the design of entropy models for neural video codec techniques. As a result, previous neural video codec techniques have limited RD performance and have been shown to only slightly outperform H.265 encoding.

[0038] II. Overview of the Improved Neural Video Codec with Hybrid Entropy Model and Flexible Quantization

[0039] The techniques described herein improve neural video codecs by incorporating a hybrid entropy model that can efficiently exploit both spatial and temporal correlations between and / or within video frames. Some aspects of the techniques described herein can also be used in neural image codecs.

[0040] According to one aspect of the disclosed technology, a previous implicit representation for a previous video frame (also referred to as a "hidden prior" hereinafter) is included in the entropy model. The use of the implicit prior can help exploit the temporal correlation of the implicit representations across video frames. As described more fully below, the quantized implicit representations of the previous video frames can be used to predict the distribution of the quantized implicit representations for the current video frame. Via a cascade training strategy, a propagation chain of implicit representations is formed. In this way, an implicit connection between the implicit representation of the current video frame and the implicit representation of the long-distance reference frame can be established. Such a connection can help the neural codec further exploit the temporal redundancy between the implicit representations.

[0041] According to another aspect of the disclosed technology, for a neural video codec or a neural image codec, a dual spatial prior feature is included in the entropy model to exploit spatial redundancy within a frame. Most existing neural codecs rely on "autoregressive priors" to exploit spatial correlation. However, the autoregressive prior is a serialized solution and follows a strict scanning order. As a result, neural codecs based on autoregressive priors are parallel unfriendly and tend to have very slow speeds. In contrast, the dual spatial prior described in this article is a two-step encoding and decoding scheme based on an improved checkerboard context model, which is more time efficient. Previously, He et al. presented a checkerboard context model in "Checkerboard context model for efficient learned image compression" (Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 14771-14780, 2021). In this model, all channels follow the same encoding and decoding order (for example, elements in even positions are always encoded and decoded first, and then used as context for encoding and decoding elements in odd positions). This approach cannot efficiently handle certain video content because sometimes encoding and decoding the even positions first has worse RD performance than encoding and decoding the odd positions first. In contrast, as described more fully below, the dual spatial prior introduces a mechanism that first encodes and decodes half of the implicit representation for elements in both odd and even positions, and then encodes and decodes the other half of the implicit representation, which can benefit from the context of elements in all (odd and even) positions. In addition, the correlation across multiple channels of the implicit representation can also be exploited during the two-step encoding and decoding. Without introducing additional codec dependencies, the dual spatial prior method increases the range or dimension of the spatial context and exploits the channel context. As a result, a more accurate prediction of the probability distribution of the quantized implicit representation can be achieved.

[0042] According to another aspect of the present disclosure, for a neural video codec or a neural image codec, the entropy model is configured to support an adaptive quantization mechanism. For a neural codec, a challenge is how to achieve smooth rate adjustment in a single trained neural codec model. In traditional (non-neural) codecs, smooth rate adjustment can be achieved by adjusting the quantization parameters. However, conventional neural codecs lack this capability and typically use a fixed quantization step size ("QS"). In order to achieve different rates, it is necessary to retrain such conventional neural codecs, which increases the burden of model training and model storage. In contrast, the adaptive quantization mechanism driven by the improved entropy model described herein allows quantization at multiple granularity levels. For example, as described more fully below, the entire (collective) QS can be determined at three different granularities. First, the user can set a global QS value for a specific target rate. Then, the global QS can be multiplied by a channel-by-channel (or per-channel) QS value, because different channels can contain information of different importance. Then, the product of the global QS value and the channel-by-channel QS value can be further multiplied by a spatial channel-by-channel (or per-region) QS value generated by the entropy model. This adaptive quantization mechanism can help the neural codec cope with various types of content and achieve accurate rate adjustment at each position of the global QS. In addition, the adaptive quantization mechanism can train the entropy model to learn QS (especially the spatial channel / region QS value), thereby not only achieving smooth rate adjustment in a single model for different global QS values, but also improving RD performance. This is because, using the adaptive quantization mechanism, the entropy model can learn to allocate more bits to more important content (through spatial channel / region QS value), which is crucial for the reconstruction of the current and subsequent video frames. This content-adaptive quantization mechanism enables dynamic bit allocation to improve the final compression ratio.

[0043] III. Example Computer System

[0044] Figure 1 A generalized example of a suitable computer system (100) is shown in which several of the described innovations may be implemented. The computer system (100) is not intended to suggest any limitation as to scope of use or functionality, as the innovations may be implemented in a variety of general-purpose or special-purpose computer systems.

[0045] refer to Figure 1, the computer system (100) includes one or more processing units (110, 115) and memory (120, 125). The processing units (110, 115) execute computer-executable instructions. The processing units may be general-purpose central processing units ("CPUs"), processors in application-specific integrated circuits ("ASICs"), or any other type of processor. In a multi-processing system, multiple processing units execute computer-executable instructions to increase processing power. For example, Figure 1 A CPU (110) and a graphics processing unit or co-processing unit (115) are shown. Tangible memory (120, 125) may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of both accessible by the processing unit(s). The memory (120, 125) stores software (180) implementing one or more innovations of a neural codec utilizing a hybrid entropy model and / or flexible quantization in the form of computer executable instructions suitable for execution by the processing unit(s).

[0046] The computer system may have additional features. For example, the computer system (100) includes storage (140), one or more input devices (150), one or more output devices (160), and one or more communication connections (170). An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the computer system (100). Typically, operating system software (not shown) provides an operating environment for other software executed in the computer system (100) and coordinates the activities of the components of the computer system (100).

[0047] The tangible storage device (140) may be removable or non-removable and includes magnetic media (such as disks, tapes, or cassettes), optical media (such as CD-ROMs or DVDs), or any other media that can be used to store information and is accessible within the computer system (100). The storage device (140) stores instructions for implementing the software (180) for one or more innovations utilizing a hybrid entropy model and / or a flexible quantized neural codec.

[0048] The input device(s) (150) may be a touch input device such as a keyboard, mouse, pen, or trackball, a voice input device, a scanning device, or another device that provides input to the computer system (100). For video, the input device(s) (150) may be a camera, video card, screen capture module, TV tuner card, or similar device that accepts video input in analog or digital form, or a CD-ROM or CD-RW that reads video input into the computer system (100). The output device(s) (160) may be a display, printer, speaker, CD writer, or other device that provides output from the computer system (100).

[0049] The communication connection(s) (170) enable communication with another computing entity over a communication medium. The communication medium transmits information, such as computer executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal that has one or more of its characteristics set or changed to encode the information in the signal. By way of example and not limitation, the communication medium may employ an electrical, optical, RF or other carrier wave.

[0050] The innovation may be described in the general context of computer-readable media. Computer-readable media is any available tangible media that can be accessed within a computing environment. By way of example and not limitation, for computer system (100), computer-readable media include memory (120, 125), storage device (140), and combinations thereof. Thus, computer-readable media may be, for example, volatile memory, non-volatile memory, optical media, or magnetic media. As used herein, the term computer-readable media does not include transient signals or propagating carrier waves.

[0051] The innovation may be described in the general context of computer executable instructions (such as instructions included in program modules) executed in a computer system on a target real or virtual processor. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules may be combined or split between program modules as needed. Computer executable instructions for program modules may be executed within a local or distributed computer system.

[0052] The terms "system" and "device" are used interchangeably herein. Unless the context clearly indicates otherwise, neither term implies any limitation on the type of computer system or computing device. In general, a computer system or computing device may be local or distributed and may include any combination of special-purpose hardware and / or general-purpose hardware with software that implements the functionality described herein.

[0053] The disclosed methods may also be implemented using dedicated computing hardware configured to perform any of the disclosed methods. For example, the disclosed methods may be implemented by an integrated circuit (e.g., an ASIC such as an ASIC digital signal processor ("DSP"), a graphics processing unit ("GPU"), or a programmable logic device ("PLD") such as a field programmable gate array ("FPGA")) that is specifically designed or configured to implement any of the disclosed methods.

[0054] For presentation purposes, this detailed description uses terms such as "select" and "determine" to describe computer operations in a computer system. These terms are high-level abstractions for operations performed by a computer and should not be confused with actions performed by a human. The actual computer operations corresponding to these terms vary depending on the implementation.

[0055] IV. Sample Network Environment

[0056] Figure 2A and Figure 2B An example network environment (201, 202) is shown that includes a video encoder (220) and a video decoder (270). The encoder (220) and the decoder (270) are connected via a network (250) using an appropriate communication protocol. The network (250) may include the Internet or another computer network.

[0057] exist Figure 2A In the illustrated network environment (201), each real-time communication ("RTC") tool (210) includes an encoder (220) and a decoder (270) for two-way communication. A given encoder (220) may output encoded data as part of a bitstream, wherein a corresponding decoder (270) accepts the encoded data from the encoder (220). The two-way communication may be part of a video conference, a video phone call, or other two-party or multi-party communication scenario. Although Figure 2A The network environment (201) includes two real-time communication tools (210), but the network environment (201) may alternatively include three or more real-time communication tools (210) participating in multi-party communication.

[0058] The real-time communication tool (210) manages encoding by the encoder (220). Figure 3 An example encoder (340) is shown that may be included in the real-time communication tool (210). The real-time communication tool (210) also manages decoding by the decoder (270). Figure 3 Also shown is an example decoder (350) that may be included in the real-time communication tool (210).

[0059] exist Figure 2BIn the illustrated network environment (202), an encoding tool (212) includes an encoder (220) that encodes video for delivery to a plurality of playback tools (214) including a decoder (270). One-way communication may be provided for video surveillance systems, web camera surveillance systems, remote desktop conferencing presentation or sharing, wireless screen projection, cloud computing or gaming, or other scenarios in which video is encoded and sent from one location to one or more other locations. Although Figure 2B The network environment (202) in FIG. 1 includes two playback tools (214), but the network environment (202) may include more or fewer playback tools (214). Generally, the playback tool (214) communicates with the encoding tool (212) to determine the video stream that the playback tool (214) is to receive. The playback tool (214) receives the stream, buffers the received encoded data for an appropriate period of time, and begins decoding and playback.

[0060] Figure 3 An example encoder (340) is shown that may be included in an encoding tool (212). The encoding tool (212) may also include server-side controller logic for managing connections with one or more playback tools (214). The playback tool (214) may include client-side controller logic for managing connections with the encoding tool (212). Figure 3 Also shown is an example decoder (350) that may be included in the playback tool (214).

[0061] V. Example Improved Neural Video Codec System

[0062] Figure 3 An example neural video codec system (300) is shown in conjunction with which some of the described embodiments may be implemented. The neural video codec system (300) includes a neural video encoder (340) configured to encode video frames into encoded data using a hybrid entropy model. The neural video encoder (340) may be Figure 2A-2B The neural video codec system (300) further includes a neural video decoder (350) configured to reconstruct video frames from the encoded data using a hybrid entropy model. The neural video decoder (350) may be Figure 2A-2B . As shown, the neural video encoder (340) can include a neural video decoder (350). In some examples, the neural video decoder (350) can be a stand-alone system. For example, when the computer system only includes the decoder, the neural video decoder (350) is separate. Figure 4 An example mixed entropy model is further described in detail in .

[0063] The neural video codec system (300) or portions of the neural video codec system (300) (e.g., the neural video encoder (340) and / or the neural video decoder (350)) can be implemented as part of an operating system module, as part of an application library, as part of a stand-alone application, or using dedicated hardware. In general, the neural video encoder (340) receives a sequence of source video frames from a video source (e.g., a camera, a tuner card, a storage medium, a screen capture module, or other digital video source) and generates encoded data as output to an output channel (338). The encoded data output to the output channel (338) may include content encoded using one or more innovations described herein. When separated, the neural video decoder (350) receives encoded data from the output channel (338) and generates a reconstructed video frame (320) as output for an output destination (e.g., a video display device, a storage medium, etc.). The received encoded data may include content encoded using one or more innovations described herein. As used herein, the term "frame" generally refers to source, encoded, or reconstructed image data.

[0064] The neural video encoder (340) receives a current video frame (302), encodes the current video frame (302) to produce encoded data, and outputs the encoded data as part of a bitstream fed to an output channel (338). As part of the encoding, the neural video encoder (340) uses one or more features of a mixed entropy model as described herein in some cases. As shown, the neural video encoder (340) also includes at least some components of a neural video decoder (350) in a reconstruction loop, including components for inverse quantization, context decoding, frame generation, buffering, temporal context mining, and motion vector decoding. The neural video decoder (350) can receive the encoded data as part of a bitstream, decode the encoded data to reconstruct the current video frame, and output the reconstructed current video frame (320). As part of the decoding, the neural video decoder (350) uses one or more features of a mixed entropy model as described herein in some cases. In Figure 3 In the example, the current video frame (302) is represented as x t , where t is the frame index, and the reconstructed current video frame (320) is represented as

[0065] As described herein, the neural video encoder (340) can be configured to generate temporal context parameters associated with a current video frame, perform context encoding and context decoding for the current video frame, and reconstruct the current video frame, as described below.

[0066] A. Example Generation of Temporal Context Parameters

[0067] Several modules are involved in generating temporal context parameters, including a motion estimator (326), a motion vector ("MV") encoder (328), a MV decoder (330), a temporal context mining network (324), and a frame and feature buffer (322).

[0068] Current video frame x t and the reconstructed previous video frame (retrieved from the frame and feature buffer (322)) is fed into the motion estimator (326) to generate a set of MV values ​​v for the current video frame t The MV value set v t Including values ​​representing or characterizing the transformation from a previous video frame to a current video frame. In some examples, the motion estimator (326) can be implemented based on a pre-trained Spynet, as described by Ranjan and Black in "Optical flow estimation using a spatialpyramid network (published in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 4161-4170, 2017)". Alternatively, the motion estimator (326) can be implemented in some other way.

[0069] The generated MV value set v t may be compressed by an MV encoder (328) and then decompressed by an MV decoder (330) to produce a set of reconstructed MV values The MV encoder (328) and the MV decoder (330) may also be collectively referred to as an MV codec. In some cases, the MV encoder (328) includes one or more convolutional layers and is configured to extract the MV from the MV set. Generate a current latent MV representation. Therefore, the MV decoder (330) also includes one or more convolutional layers and is configured to reconstruct a set of MV values ​​from the current latent MV representation although Figure 3The internal details of the MV encoder (328) and MV decoder (330) are not shown, but the MV encoder (328) and MV decoder (330) may include components configured to encode / decode MV information. For example, for encoding of MV information, the MV encoder (328) may include its own context encoder, super-prior encoder, super-prior decoder, entropy model network, quantizer, hidden buffer (for previous hidden MV representation), inverse quantizer, context decoder, and arithmetic encoder (and if the encoding is not bypassed internally, an arithmetic decoder). Similarly, when as part of a separate decoder, the MV decoder (330) may include its own super-prior decoder, entropy model network, hidden buffer (for previous hidden MV representation), inverse quantizer, context decoder, and arithmetic decoder. Refer to the following respectively. Fig. 8A and Figure 8B An example network structure for a context encoder for an MV encoder (328) and a context decoder for an MV decoder (330) is further described. Alternatively, different network structures may be used to implement a content encoder and context decoder for MV information. The super-a priori encoder of the MV encoder (328) determines a highly parameterized version of the current hidden MV representation from the context encoder, and the super-a priori decoder reconstructs a version of the current hidden MV representation. An example network structure for a super-a priori encoder and a super-a priori decoder for MV information is described below. Alternatively, different network structures may be used to implement a super-a priori encoder and a super-a priori decoder for MV information. The entropy model network of the MV encoder (328) / MV decoder (330) may determine statistical information (arithmetic encoding and decoding for MV information) based on a reconstructed version of the current hidden MV representation and a reconstructed version of a previous hidden MV representation. An example network structure for an entropy model network for MV information is described below. Alternatively, different network structures may be used to implement an entropy model network for MV information.

[0070] Reconstructed MV value set is fed to the temporal context mining network (324), which also receives the previous feature parameter set F retrieved from the frame and feature buffer (322). t-1 Input. The previous feature parameter set F t-1 Associated with the previous video frame and generated by the frame generator (318), as further described below.

[0071] The temporal context mining network (324) includes one or more convolutional layers and is configured to explore or capture temporal correlations present in video frames. An example temporal context mining network (324) including an example network structure is described in more detail in Sheng et al. (“Temporal Context Mining for Learned Video Compression,” arXiv preprint arXiv:2111.13850, 2021, hereinafter referred to as “Sheng 2021”). In general, the temporal context mining network (324) can be configured to be based on and F t-1 To generate one or more temporal context parameter sets with different scales, e.g. Multiscale temporal context parameter set and With different spatial resolutions (e.g. With F t-1 The same spatial resolution, and have a gradually decreasing spatial resolution as they are derived from F t-1 ), and they can help represent spatiotemporal non-uniform motion and texture information of video frames. Figure 3 Three temporal context parameter sets at different scales are shown However, in some cases, the temporal context mining network (324) may be configured to generate more than three (eg, 4, 5, etc.) or less than three (eg, 1, 2) temporal context parameter sets.

[0072] The generated one or more temporal context parameter sets (e.g., ) can be fed to other modules of the neural video codec system (300), such as the context encoder (304), the context decoder (316), and the entropy model network (310), as described below.

[0073] B. Example Context Encoding and Decoding

[0074] In the multi-scale temporal context parameter set (e.g., ), the context encoder (304) may be configured to generate a t The current implicit representation y t As described in this paper, the current implicit representation y t The elements of are logically organized in three dimensions, including two spatial dimensions (corresponding to the current video frame x tThe height and width of the neural network) and a channel dimension. Typically, the channel dimension organizes the parameters (basic features or coefficients) output from the neural network layer or network structure. The number of channels depends on the implementation. Generally, as the number of channels increases, the complexity of the neural model network increases, and the accuracy of the neural model network also increases (at least before reaching the point of diminishing returns). Since the current implicit representation y t is the value from the current video frame x t Generated, so the current hidden representation y t It may also be referred to as the current latent sample value (SV) representation (to distinguish it from the current latent MV representation, as described above). The context encoder (304) includes one or more convolutional layers. Fig.9A An example network structure of a context encoder (304) is described. Additional explanation of the operation of a similar context encoder is described in Sheng 2021. Alternatively, a different network structure may be used to implement the context encoder (304).

[0075] In order to achieve bit rate saving, the current implicit SV represents y t may be quantized into a quantized version by a quantizer (306) before being sent to an arithmetic encoder ("AE" 308) The arithmetic encoder generates a bitstream containing the encoded data for the current latent SV representation. Similarly, a quantized version of the current latent representation The elements in are logically organized along two spatial dimensions and the channel dimension. During decoding, a quantized version of the current latent representation Decoded from the bitstream by the arithmetic decoder ("AD" 312) and inverse quantized by the inverse quantizer (314) into a decoded (reconstructed) version of the current latent representation In fact, since arithmetic encoding / decoding is lossless, if the quantized version of the current latent representation is passed directly to the inverse quantizer (314) within the encoder (340), then the AD (312) can be omitted during decoding.

[0076] As described herein, the AE (308) and AD (312) work in conjunction with the entropy model network (310) to provide entropy encoding and entropy decoding, respectively. As further described below, the entropy model network (310) can be configured to estimate a quantized version of the current latent SV representation Example statistical properties include a quantized version of the current latent SV representation The mean or average value, standard deviation, scale parameter, variance, median, etc. of the probability distribution function of . For example, for a Laplace or Gaussian distribution, the statistical properties may include the mean (μ t ) and scale parameter (σ t ).

[0077] In addition, the entropy model network (310) can generate multiple per-region QS values ​​(denoted as qs sc ). The generated qs sc The values ​​may be fed to the quantizer (306) and the inverse quantizer (314). The quantizer (306) and the inverse quantizer (314) may also receive global QS values ​​(336, denoted as qs global ) and multiple per-channel QS values ​​(334, denoted as qs ch ). Based on qs global 、qs ch and qs sc , the quantizer (306) and the inverse quantizer (314) can perform multi-granularity quantization and inverse quantization, respectively, as described more fully below.

[0078] Also with a time context parameter set (e.g. ) is the condition, the context decoder (316) is configured to Decode the estimated current set of feature parameters Similar to the context encoder (304), the context decoder (316) includes one or more convolutional layers. Fig. 9B An example network structure for a context decoder is described. Additional explanation of the operation of a similar context decoder is described in Sheng 2021. Alternatively, a different network structure may be used to implement the context decoder (316).

[0079] In the encoding / decoding process, the entropy model network (310) performs statistical analysis on the statistical characteristics. Accurate estimation of is crucial for bitrate reduction. To this end, a hybrid entropy model is used to efficiently capture the spatial and temporal dependencies between video frames. In addition, the hybrid entropy model is configured to support multi-granularity quantization and inverse quantization in a single model. Figure 4 An example mixed entropy model is described more fully.

[0080] C. Example reconstruction of the current video frame

[0081] like Figure 3 As shown, the frame generator (318) is configured to generate a frame from the estimated current feature parameter set Generate a reconstructed current video frame (320). Usually, Indicates a refactored version of something. For example, in Figure 3 middle, Represents the reconstructed feature parameter set from the context decoder (316). Reconstructed video frame (320) can be stored in a frame and feature buffer (322) similar to a decoder picture buffer ("DPB") and used by a motion estimator (326) to generate a set of MV values ​​for subsequent video frames. Additionally, the frame generator (318) also generates a current feature parameter set F t The current feature parameter set F t It can also be stored in the frame and feature buffer (322) and used by the temporal context mining network (324) to generate a set of temporal context parameters for subsequent video frames. In some cases, the frame generator (318) includes one or more convolutional layers. Fig. 10A An example network structure for the frame generator (318) is further described. Alternatively, the frame generator (318) may be implemented using a different network structure.

[0082] D. Example variants of the Neural Video Codec system

[0083] Depending on the implementation and the type of compression / decompression desired, modules of the neural video codec system (300) may be added, omitted, separated into multiple modules, combined with other modules, and / or replaced with similar modules. The relationships shown between modules within the neural video encoder (340) and the neural video decoder (350) indicate the general information flow in the neural video encoder (340) and the neural video decoder (350), respectively; other relationships are not shown for simplicity. In general, a given module of the neural video codec system (300) may be implemented by software capable of executing on a CPU, by software controlling dedicated hardware (e.g., graphics hardware for video acceleration), or by dedicated hardware (e.g., in an ASIC).

[0084] VI. Example Mixed Entropy Model

[0085] As described above, both AE (308) and AD (312) work in conjunction with the entropy model network (310) to provide entropy encoding and entropy decoding, respectively. To this end, both entropy encoding and entropy decoding utilize a quantized version of the current latent SV representation The probability mass function (“PMF”) of The true PMF (or ) is unknown, so the entropy model network (310) uses the estimated PMF To approximate the true PMF Cross Entropy Captures the average number of bits used by arithmetic coding, ignoring negligible overhead. In some examples, a quantized version of the current latent SV representation can be considered to follow a Laplace or Gaussian distribution.

[0086] Figure 4A mixed entropy model network (440) is depicted, which can accurately estimate the statistical properties To reduce the cross entropy. Specifically, the hybrid entropy model network (440) incorporates latent priors as model inputs to exploit the temporal correlation of latent representations across video frames. Figure 4 In , the hybrid entropy model network (440) also incorporates dual spatial prior features to exploit spatial redundancy within the video frame, but alternatively, the hybrid entropy model network can be implemented without dual spatial prior features. Figure 4 In the embodiment, the hybrid entropy model network (440) is configured to support an adaptive quantization mechanism that allows quantization at multiple granularity levels, but alternatively, the hybrid entropy model network can be implemented without an adaptive quantization mechanism.

[0087] In the depicted example, the hybrid entropy model network (440) has an input unit (406), a first fusion unit (408), a first statistic / parameter estimator (410), a second fusion unit (412), and a second statistic estimator (414). Each of the first statistic / parameter estimator (410) and the second statistic estimator (414) may include one or more convolutional layers. An example network structure of the hybrid entropy model network (440) is shown in FIG. Fig.11 Alternatively, a different network structure may be used to implement the hybrid entropy model network (440).

[0088] A. Implicit Prior

[0089] To improve the estimation of statistical information by the entropy model network, the temporal correlation of the latent representations across video frames can be exploited. For MV encoding / decoding, the latent prior can be the previous latent MV representation. For SV encoding / decoding, the latent prior can be the previous latent SV representation. Figure 4 , as mentioned above, The elements in are logically organized in three dimensions, including one channel dimension and two space dimensions. (where i, j, and k are the height, width, and channel index, respectively) is represented as Theoretically, each Can be used with the previous video frame The corresponding element in the decoded latent representation of In addition, each With the time context parameter set (e.g., ), these temporal context parameter sets are based in part on previous feature parameter sets F associated with previous video frames. t-1 and based in part on the reconstructed MV value set for the current video frame Due to the large space requirements, traditional codecs cannot explicitly exploit such correlations and therefore rely on simple handcrafted rules to use context from a few neighboring locations. In contrast, deep learning neural network architectures enable the ability to automatically mine correlations in large spaces. For example, Figure 4 As shown, the hybrid entropy model network (440) can receive multimodal input via the input unit (406). As a result, the hybrid entropy model network (440) can extract complementary information from rich high-dimensional input.

[0090] Figure 5A Shown is the estimate An example input of the mixed entropy model network (440) of the statistical characteristics of . The fusion unit (530) depicted in Figure 5 can be Figure 4 As shown in the figure, the hybrid entropy model network can receive three different inputs: the hyper-prior parameters associated with the latent SV representation of the current video frame A set of temporal context parameters and for the previous video frame The decoded previous latent SV representation (i.e., latent prior) of . Before being fused by the fusion unit (530), the super prior information First, it is decoded by the super-a priori decoder (510) to generate decoded super-a priori parameters. Fig. 13B An example network structure of a super prior decoder for implicit SV representation is described. Alternatively, the super prior decoder can be implemented using a different network structure. Additionally, before being fused by the fusion unit (530), the temporal context parameter set First, the temporal context encoder (520) is used to generate a temporal context prior. The temporal context encoder (520) may include one or more convolutional layers. Fig.12 An example network structure of a temporal context encoder is described. Alternatively, a different network structure may be used to implement a temporal context encoder. Fig.11 An example network structure of the fusion unit (530) is shown ( Fig.11 Alternatively, the fusion unit (530) can be implemented using a different network structure.

[0091] Typically, during encoding, the current latent SV representation y t Get the hyper-prior parameter z t , for the hyper-prior parameter z t quantized, entropy encoded, and output as part of the encoded data. During decoding or as part of reconstruction during encoding, the hyper-prior parameters Reconstructed by entropy decoding (as needed) and super-prior decoder. Reconstructed super-prior parameters This is then used by the entropy model network. For example, Figure 5B As shown in We can express y from the current hidden SV t Specifically, the current latent SV represents y t It may first be encoded by a super priori encoder (502). The encoded y t quantized by a quantizer (504) and then further encoded by an arithmetic encoder (506). The encoded output of the arithmetic encoder (506) can be decoded by an arithmetic decoder (508) to generate a super prior Each of the super-a priori encoder (502) and the super-a priori decoder (510) may include one or more convolutional layers. Fig.13A and Fig. 13B Example network structures for a hyper-prior encoder and a hyper-prior decoder for SV information are further described. Additional details about the hyper-prior are described in "Variational image compression with a scale hyperprior" by Ballé et al. (Sixth International Conference on Learning Representations, ICLR (2018)). Alternatively, a different network structure can be used to implement a hyper-prior encoder and a hyper-prior decoder for SV information.

[0092] Despite Figure 5A Only one time context parameter set in , but in some cases, the input to the hybrid entropy model network may include different sets of temporal context parameters (e.g., or ). In some examples, the input to the hybrid entropy model network may include two or more sets of temporal context parameters (e.g., and ).

[0093] like Figure 3 As shown, implicit prior may be generated by the inverse quantizer (314) and stored in the implicit buffer (332). Although the temporal context parameter set (e.g. ) and the reconstructed implicit prior Both contain time information, but they have different characteristics. For example, in some example implementations, The 4x downsampled resolution typically contains a lot of motion information. In contrast, in the example implementation, In the latent representation domain with 16 times downsampled resolution, and with the same latent SV representation y t More similar characteristics. Therefore, and Can provide supplementary and ancillary information to improve estimates The accuracy of the statistical characteristics.

[0094] In addition, in some implementations, a cascade training strategy can be adopted so that gradients can be back-propagated to multiple video frames. Further details on this cascade training strategy in different contexts are described in "BasicVSR: The search for essential components in video super-resolution and beyond" by Chan et al., published in the Proceedings of the 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 4947-4956) and Sheng 2021. Under such a training strategy, a propagation chain of latent representations can be formed. As a result, a connection between the latent representation of the current video frame and the latent representation of the long-distance reference video frame is also established. This connection can be very helpful in extracting correlations in the latent representations across multiple video frames, leading to the More accurate predictions of the statistical distribution of .

[0095] During decoding, based on the same inputs as during encoding, the hybrid entropy model network (440) performs operations in the same order to determine the statistical characteristics of the corresponding set of elements used in entropy decoding and the per-spatial region (and per-channel) QS values ​​used in inverse quantization.

[0096] Figure 4 , Figure 5A and Figure 5B Depicted is a hybrid entropy model network for encoding / decoding of latent SV representations. The hybrid entropy model network can also be used for encoding / decoding of latent MV representations. For encoding / decoding of MV information, the input to the hybrid entropy model network includes the hyper-prior parameters of the current latent MV representation for the current video frame and the decoded previous implicit MV representation for the previous video frame (i.e., implicit prior). Based on the MV value for the current video frame The above temporal context parameter set is not input into the hybrid entropy model network for MV encoding / decoding. Before being fused by the fusion unit, the super prior information for the hidden MV representation is It is first decoded by a super-prior decoder to generate decoded super-prior parameters for MV information. The hidden buffer can store decoded previous hidden MV representations for previous video frames. (i.e., implicit prior).

[0097] B. Dual-space priors

[0098] Whether or not to use implicit priors (or ) to enrich the input, the hybrid entropy model network can also be configured to implement dual spatial prior features to exploit spatial correlation within the video frame. The dual spatial prior features can be used when encoding / decoding the current latent SV representation or the current latent MV representation.

[0099] For example, to improve the operation efficiency, dual spatial prior features can be implemented in a two-stage estimation process based on the segmented chessboard context model, such as Figure 4 For simplicity, Figure 4 The arithmetic encoder and decoder are omitted in the two-stage estimation of Figure 3 308 and 312 in ). More generally, the dual spatial prior features can be implemented in a multi-stage estimation process with more than two stages. In the corresponding decoding, the dual spatial prior features are implemented in a two-stage estimation process (more generally, a multi-stage estimation process).

[0100] Figure 4 The encoding / decoding of the current latent SV representation using the dual spatial prior features is shown. As mentioned above, the current latent SV representation y t The elements of are logically organized in three dimensions, including two spatial dimensions and one channel dimension. Figure 4 As shown, the current hidden SV represents y t (402) can be split into two blocks of elements (422, 424) along the channel dimension by a splitter (404). Each of the two blocks (422, 424) has the same t (404) has the same spatial dimensions, but only has a subset of the channels. For example, suppose y t The total number of channels in the first block (422) is C, then the first block (422) may include elements in the next half of the channels (e.g., y t,k<C / 2 ), and the second block (424) may include elements in the upper half of the channel (e.g., y t,k≥C / 2 ).

[0101] like Figure 4 As shown, two blocks (422, 424) of elements may be quantized by a quantizer (416) to generate quantized versions of the two blocks, respectively (e.g., and ). Figure 4 The quantizer (416) depicted in FIG. 4 may be Figure 3 Quantizer (306). Figure 6 As further described, the quantizer (416) may be configured to implement an adaptive quantization mechanism that allows for multiple levels of granularity of quantization.

[0102] Each quantization block (e.g. and ) The elements in can be further divided into two sets along two spatial dimensions: an odd set containing the elements at odd positions, and an even set containing the elements at even positions. As described herein, if the sum of the element indices in two spatial dimensions is even (e.g., (i + j) % 2 == 0), the element is at an even position, and if the sum of the element indices in two spatial dimensions is odd (e.g., (i + j) % 2 == 1), the element is at an odd position. For example, the first quantization block The elements in can be partitioned into a first even set (426) and a first odd set (430), and the second quantization block The elements in can be partitioned into a second odd set (428) and a second even set (432).

[0103] During the first stage of the two - spatial - prior - estimation process, the first statistics / parameter estimator (410) can be configured to encode the elements in the first even set (426) while setting the elements in the first odd set (430) to zero. Additionally, the first statistics / parameter estimator (410) can be configured to encode the elements in the second odd set (428) while setting the elements in the second even set (432) to zero. Encoding the elements in the first even set (426) and encoding the elements in the second odd set (428) can be performed simultaneously or substantially simultaneously (e.g., via parallel computing).

[0104] During the first - stage estimation process, the first statistics / parameter estimator (410) can estimate the statistical characteristics (e.g., μ idx and σ idx , where idx = {t, (i + j) % 2 == 0, k < C / 2}) of the elements in the first even set (426) and the statistical characteristics (e.g., μ idx and σ idx , where idx = {t, (i + j) % 2 == 1, k ≥ C / 2}) of the elements in the second odd set (428). During the first - stage estimation process, the first statistics / parameter estimator (410) can also determine the QS parameters for each spatial region (and each channel) provided to the quantizer (416).

[0105] Fig.11 shows an example network structure of the first statistics / parameter estimator (410) ( Fig.11 the left "parameter estimation" structure in). Alternatively, the first statistics / parameter estimator (410) can be implemented using a different network structure.

[0106] After the first - stage encoding and decoding, the first set of even numbers (426) and the second set of odd numbers (428) that have been quantized can be fused together by the second fusion unit (412). The second fusion unit (412) also receives at least some of the output channels from the first statistical information / parameter estimator (410) as inputs, and then further generates contexts for the second - stage estimation. Fig.11 An example network structure of the second fusion unit (412) is shown ( Fig.11 the right - hand "previous fusion" layer in

[0107] During the second stage of the dual - space prior estimation process, the second statistical information estimator (414) can be configured to encode the elements in the first set of odd numbers (430) and the elements in the second set of even numbers (432). Similarly, encoding the elements in the first set of odd numbers (430) and encoding the elements in the second set of even numbers (432) can be performed simultaneously or substantially simultaneously (e.g., via parallel computing).

[0108] Fig.11 An example network structure of the second statistical information estimator (414) is shown ( Fig.11 the right - hand "parameter estimation" structure in

[0109] During the second - stage estimation process, the second statistical information estimator (414) can estimate the statistical characteristics (e.g., μ idx and σ idx , where idx = {t, (i + j)%2 == 1, k < C / 2}) of the elements in the first set of odd numbers (430) and the statistical characteristics (e.g., μ idx and σ idx , where idx = {t, (i + j)%2 == 0, k ≥ C / 2}) of the elements in the second set of even numbers (432). Since the second fusion unit (412) fuses at least some of the results of the first statistical information estimator (410), the second - stage estimation process can benefit from the contexts from all spatial positions. Therefore, by leveraging the estimation results obtained by the first statistical information estimator (410), the estimation of the statistical characteristics by the second statistical information estimator (414) can be more accurate.

[0110] Entropy encoding for the first set of even numbers (426), the second set of odd numbers (428), the first set of odd numbers (430), and the second set of even numbers (432) can occur simultaneously using the corresponding statistical characteristics for different sets of elements.

[0111] During decoding, based on the same inputs, the entropy model network (440) performs operations in the same order to determine statistical properties and QS values ​​per spatial region (and per channel) for corresponding sets of elements. When reconstructing the current latent SV representation during decoding, the statistical properties estimated by the first statistical information / parameter estimator (410) can be used for entropy decoding of elements in the first even set (426) and the second odd set (428), and the statistical properties estimated by the second statistical information estimator (414) can be used for entropy decoding of elements in the first odd set (430) and the second even set (432). During decoding and as part of a reconstruction loop during encoding, the decoded elements in the first even set (426), the second odd set (428), the first odd set (430), and the second even set (432) can be inverse quantized by the inverse quantizer (418) using the QS values ​​per spatial region (and per channel) from the entropy model network (440) to generate a reconstructed first block (434, e.g., respectively). ) and the reconstructed second block (436, e.g. ). Figure 4 The inverse quantizer (418) depicted in FIG. 4 may be Figure 3 The inverse quantizer (314) of Figure 6 As further described, the inverse quantizer (418) can be configured to allow for multiple levels of granularity of inverse quantization. Finally, the reconstructed first block (434) and the reconstructed second block (436) can be concatenated by a concatenator (420) to generate a reconstructed current implicit representation (438).

[0112] Compared to the conventional checkerboard context model described by He et al. in “Checkerboard context model for efficient learned image compression” (Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 14771-14780, 2021), the segmented checkerboard context model described in this article increases the range of spatial context through channel segmentation. In other words, the elements in the quantized version of the implicit representation are divided not only in two spatial dimensions (e.g., based on the odd or even positions of the elements), but also along the channel dimension. As a result, a more accurate estimate of statistical characteristics can be achieved. Additionally, for both the first estimation stage and the second estimation stage, a first quantization block and a second quantization block can be added and sent to the arithmetic encoder (e.g., Figure 3308). Therefore, compared with the conventional checkerboard model described by He et al., the dual spatial prior features described in this paper do not introduce any additional encoding and decoding delay.

[0113] Importantly, the dual spatial prior features described herein can exploit cross-channel correlations. For example, the quantized first even set (426) used for the first stage estimation process can also be used as a condition for encoding the second even set (432) during the second stage estimation process. Similarly, the quantized second odd set (428) used for the first stage estimation process can also be used as a condition for encoding the first odd set (430) during the second stage estimation process. As a result, the dual spatial prior can further squeeze redundancy by more efficiently exploiting correlations across spatial locations and channel dimensions.

[0114] It should be understood that Figure 4 Depicted will The division of the elements of into four sets (426, 428, 430, 432) is only a specific example. Alternatively, the elements of For example, the splitter (404) may partition y along the channel dimension differently. t (402), resulting in two different blocks. In a specific example, one block may include elements in odd channels, and the other block may include elements in even channels. Figure 4 In the example depicted in , the two blocks (422, 424) have an equal number of channels (assuming C is an even number). In other examples, the two blocks partitioned by the partitioner (404) may have different numbers of channels.

[0115] In some cases, The elements of can also be partitioned into sets along two spatial dimensions that do not follow a checkerboard pattern. For example, The elements of can be divided into different quadrants. In a specific example, the first set may include elements in the upper left quadrant, the second set may include elements in the lower right quadrant, and the third set may include elements in the upper right quadrant and the lower left quadrant. In this case, instead of using a two-stage estimation as described above, a three-stage estimation process (which can be viewed as a triple spatial prior, or more generally, a multi-stage spatial prior) can be performed based on the same principles described herein.

[0116] More generally, the current latent SV represents y t (and the resulting quantized version ) can be split into different channel sets along the channel dimension and into different spatial position sets along the two spatial dimensions to form multiple element sets. Each of the multiple element sets can have a different combination of one of the different channel sets and one of the different spatial position sets. For example, Figure 4 It is shown that the channel dimension is divided into two channel sets (upper channel and lower channel), and the two spatial dimensions are divided into two spatial position sets (odd positions and even positions). Thus, the first even set (426) includes elements in the lower half of the channel and at even positions, and the second odd set (428) includes elements in the upper half of the channel and at odd positions.

[0117] Based at least in part on the quantized version of the first set of elements in the plurality of sets of elements, a statistical characteristic of the quantized version of the second set of elements in the plurality of sets of elements may be estimated. Figure 4 In the embodiment of the present invention, the statistical characteristics of the first set of odd numbers (430) and / or the second set of even numbers (432) can be estimated based in part on the first set of even numbers (426) and / or the second set of odd numbers (428). When the estimation process is divided into multiple stages, the encoding (quantization) results from the previous stages can be fused to provide input to the subsequent estimation stage, as described above.

[0118] Figure 4 and Fig.11 Depicted is an entropy model network for encoding / decoding the current latent SV representation using dual spatial prior features. The entropy model network using dual spatial prior features can also be used for encoding / decoding the current latent MV representation. For encoding / decoding of MV information, the input to the entropy model network includes the hyper-prior parameters of the current latent MV representation for the current video frame If the entropy model network is a hybrid entropy model network that uses latent priors as input (as described in the previous section), the input also includes the decoded previous latent MV representation for the previous video frame (i.e., implicit priors). The statistical properties of the set of elements of the current latent MV representation can be estimated in two or more stages. For example, during the first stage of estimation, the first statistical information / parameter estimator of the entropy model network estimates statistical properties for one or more sets of elements of the current latent MV representation, and also determines QS parameters per spatial region (and per channel) for these sets of elements of the current latent MV representation, which are provided to the quantizer. For the second stage of estimation, the (one or more) quantized element sets of the current latent MV representation from the first stage estimation are fused with (at least some of) the outputs of the first statistical information estimator. During the second stage of estimation, the second statistical information estimator of the entropy model network estimates the statistical properties of (one or more) other sets of elements of the current latent MV representation. The statistical properties of the corresponding quantized element sets of the current latent MV representation are used in entropy encoding and decoding of the corresponding element sets.

[0119] During decoding, based on the same input(s), the entropy model network performs operations in the same order to determine statistical properties for the corresponding set of elements and QS values ​​per spatial region (and per channel) for the hidden MV representation. When reconstructing the current hidden MV representation during decoding, the statistical properties estimated by the first statistical information / parameter estimator can be used for entropy decoding of elements in the first stage set(s), and the statistical properties estimated by the second statistical information estimator can be used for entropy decoding of elements in the second stage set(s). During decoding and as part of the reconstruction loop during encoding, the decoded elements in the corresponding set of elements can be inverse quantized by an inverse quantizer using the QS values ​​per spatial region (and per channel) from the entropy model network to generate a reconstructed set of MV values. As further described below, the inverse quantizer can be configured to allow inverse quantization at multiple levels of granularity. Finally, the reconstructed set of MV values ​​can be cascaded by a cascader to generate a reconstructed current hidden MV representation.

[0120] As described above, the entropy model network implementing the multi-stage estimation process can be a hybrid entropy model network that accepts implicit priors as input. Alternatively, the entropy model network implementing the multi-stage estimation process can operate without accepting implicit priors as input.

[0121] C. Multi-granularity quantization

[0122] Conventional neural codecs cannot handle rate adjustment in a single trained neural codec model. In order to achieve different rates, the entropy model of the conventional neural codec needs to be retrained by adjusting the weights according to the RD criteria for different rates respectively. Such an approach significantly increases the training overhead and the model storage burden of the neural codec. In order to achieve a wide rate range in a single trained neural codec model, an adaptive quantization mechanism can be integrated with an entropy model network (e.g., a hybrid entropy model network (440) that accepts hidden priors as input and implements dual spatial prior features). As described below, an adaptive quantization mechanism can be used when encoding / decoding a current hidden SV representation or a current hidden MV representation. Similarly, the adaptive quantization mechanism can be used with a hybrid entropy model network that accepts hidden priors as input or an entropy model network that does not accept hidden priors as input.

[0123] like Figure 4 As shown, the current latent SV representation y including the divided first block and the second block (422, 424) t (402) may be quantized by a quantizer (416) using three sets of quantization step size ("QS") values: a global QS value (denoted as qs) used to adjust the bit rate and overall quality global ), multiple per-channel QS values ​​(expressed as qs ch , which can also be called “channel-by-channel quantization step value”), and multiple per-region QS values ​​for different spatial regions represented by the current hidden SC (expressed as qs sc , which may also be referred to as "spatial channel-by-spatial channel quantization step value"). As described herein, different spatial regions are associated with different positions or regions / blocks (e.g., determined by height, width, and channel indices i, j, and k) of the current latent SV representation, and different per-region QS values ​​are channel-specific (but alternatively, different per-region QS values ​​may be channel-independent). For example, two different spatial regions in the same channel (e.g., with different i or j indices) may have two different per-region QS values, and two identical spatial regions in two different channels (e.g., with the same i and j indices) may also have different per-region QS values.

[0124] Global QS value qs global It can be a fixed value predefined by the user as part of the overall setting of quality and bitrate. For the current latent SV representation and the current latent MV representation, the global QS value qs global It can be the same parameter, or the current latent SV representation and the current latent MV representation can have different global QS values.

[0125] QS value qs for each channel for different channels chThe per-channel QS values ​​may be configured as part of a neural video codec system (e.g., 300) based on the importance of the corresponding channels and may be learned during the model training process. Each per-channel QS value may apply to a specific channel. The per-channel QS values ​​for different channels may be the same or different. For example, the per-channel QS value for the lower half of the channels may be Any of can be greater than, less than, or equal to the per-channel QS value for the upper half of the channels Alternatively, a group of channels or a band of channels can share a per-channel QS value, such as in the quantization matrix. ch The per-channel QS value qs may be the same parameter for the current latent SV representation and the current latent MV representation, or the current latent SV representation and the current latent MV representation may have different per-channel QS values ​​(e.g., because the number of channels is different for the current latent SV representation and the current latent MV representation, or because the relative importance of the channels is different in the two latent representations). In some example implementations, the per-channel QS value qs ch does not change over time. In this case, a per-channel QS value qs can be defined at the encoder and decoder ch Alternatively, the per-channel QS value may change over time (e.g., from one video sequence to another, from one set of video frames to another set of video frames within a video sequence, or from one video frame to another, etc.), in which case the encoder may set the per-channel QS value qs ch encodes and outputs them as part of the encoded data, and the decoder can reconstruct the per-channel QS value qs ch .

[0126] QS value qs for each area in different spatial areas sc It can be generated by the hybrid entropy model network (440) as described above or other entropy model networks. The current latent SV representation and the current latent MV representation can have different per-region QS values ​​qs sc In some example implementations, since the per-region QS value qs sc is generated by the entropy model network during encoding and during decoding from the same input, so the per-region QS value qs sc is not encoded and transmitted in the encoded data. For example, in one specific example, the per-region QS values ​​(qs sc ) can be generated by the first statistical information estimator (410).

[0127] exist Figure 4 In the example, the hybrid entropy model network (440) generates a per-region QS value for all sets of elements in the current latent representation. In some other implementations, the per-region QS value (qssc ) can be generated by the first statistics / parameter estimator (410), and the per-region QS values ​​(qs sc ) may be generated by the second statistical information estimator (414), for example, based on the encoded elements for the first odd set (430) and the second even set (432). In this case, in addition to the statistical characteristics for the first odd set (430) and the second even set (432), the second statistical information estimator (414) also outputs per-region QS values ​​(qs sc ). Alternatively, the per-region QS value qs for all element sets of the current implicit representation is sc May be generated by the second statistical information estimator (414).

[0128] During decoding and as part of the reconstruction loop during encoding, a corresponding inverse quantization is applied. Typically, the inverse quantizer (418) may use the same global QS value qs for inverse quantization. global , multiple QS values ​​per channel qs ch and multiple QS values ​​per region qs sc . Global QS value qs global It can be transmitted to the decoder along with other encoded data in a bit stream. global The overhead is negligible since only a single number is transmitted for each frame or video (or two values ​​if different values ​​are used for MV information and SV information).

[0129] Figure 6 An example implementation of multi-granularity quantization and inverse quantization is depicted. As shown, quantization can be performed in three consecutive stages. At the first quantization stage (610), the global QS value qs global To quantize the current latent SV representation. At the second quantization stage (620), the per-channel QS value qs ch The output of the first quantization stage (610) is further quantized (e.g., each channel is quantized by a QS value specific to that channel). At the third quantization stage (630), the per-region QS value qs sc The output of the second quantization stage (620) may be further quantized by a rounding unit (640) to generate a final quantized version of the current latent SV representation. The final quantitative version is sent to the arithmetic encoder (308) to generate a bit stream. Figure 6Depicts the current latent SV representation y t quantization of the elements, but the current hidden MV represents mv_y t The elements of can be quantized in a similar manner within the MV encoder.

[0130] Similarly, inverse quantization can be performed in three consecutive stages, but in the opposite order. As shown in the figure, the quantized version of the current latent SV representation Decoded from the bitstream by the arithmetic decoder (312) (or, in the reconstruction loop during encoding, passed from the quantizer). In the first inverse quantization stage (650), the per-region QS value qs sc To inverse quantization (E.g., each element is inverse quantized by a QS value specific to the spatial position and channel of that element.) At the second inverse quantization stage (660), the per-channel QS value qs ch The output of the first inverse quantization stage (650) is further inverse quantized (e.g., each channel is inverse quantized by a QS value specific to that channel). At the third inverse quantization stage (670), the global QS value qs global The output of the second inverse quantization stage (660) is further inverse quantized. The output of the third inverse quantization stage (670) produces the final reconstructed current latent SV representation although Figure 6 Depicts the quantization elements for the current latent SV representation Inverse quantization, but the current hidden MV represents The elements of can be inverse quantized in a similar manner within the MV encoder or MV decoder.

[0131] Figure 6 shows the method for applying the global QS value qs during quantization and inverse quantization. global , multiple QS values ​​per channel qs ch and multiple QS values ​​per region qs sc Alternatively, the global QS value qs global , multiple QS values ​​per channel qs ch and multiple QS values ​​per region qs sc Can be applied in different orders during quantization and inverse quantization.

[0132] Because the global QS value is a single value applied to elements in all spatial locations and all channels of a given latent representation, qs globalA coarse quantization effect can be brought to control the target rate. Because different channels carry information with different importance, the per-channel QS value can scale or modulate the quantization step size at different channels. In addition, due to various image or video content, different spatial locations of each channel can also have different characteristics. Therefore, the per-region QS value can be used for more precise adjustment of the quantization step size at each location in each channel.

[0133] As mentioned above, the QS value per region is qs sc is generated by an entropy model network (e.g., a hybrid entropy model network (440)). Therefore, for each image or video frame, qs is dynamically changed sc to adapt to the image or video content. This content adaptation is not only useful for achieving smooth bitrate adjustment, but also can improve the final rate-distortion performance through content-adaptive bit allocation. Specifically, more important information that is crucial for reconstruction and / or referenced by the codec of subsequent video frames will be assigned smaller quantization values, and vice versa.

[0134] exist Figure 7 A visualization example is shown in . In this example, the upper left panel shows the input video frame. The per-region QS value qs generated by the hybrid entropy model sc The upper right panel shows the case where the per-region QS value qs is not used. sc The lower right panel shows the quantized implicit representation of the input video frame using the per-region QS value qs sc quantized implicit representation of the input video frame. In this example, the hybrid entropy model learns that the moving player is more important and produces smaller per-area QS values ​​for these areas. In contrast, background areas (e.g., marked by the horizontal and vertical lines in the lower right corner of the panel) are associated with larger per-area QS values, thus resulting in significant savings in bitrate. For example, without using per-area QS values, the bitrate is 0.065 bits per pixel (BPP), but is reduced to 0.056 BPP when using per-area QS values, thus resulting in approximately 13.8% BPP reduction at similar image quality.

[0135] In many of the foregoing examples, each spatial position (or each combination of spatial position and channel) has its own per-region QS value. Alternatively, per-region QS values ​​may be shared between multiple spatially neighboring positions (eg, for blocks or windows).

[0136] In many of the foregoing examples, there are three quantization stages and three corresponding inverse quantization stages. Alternatively, there may be fewer stages or more stages. For example, there are two quantization stages and two corresponding inverse quantization stages, where a global QS value is applied in one stage and a per-region, per-channel QS value is applied in another stage. Or, as another example, there are four quantization stages and four corresponding inverse quantization stages, where a global QS value is applied in one stage, a per-channel QS value is applied in another stage, and hierarchical QS values ​​are applied in the remaining stages for different levels of spatial granularity.

[0137] VII. Example Neural Network Architecture

[0138] Convolutional neural networks ("CNNs") are used in several components of the neural video codec system (300) and the neural image codec system described herein. Typically, a CNN includes one or more convolutional layers. A convolutional layer includes a set of filters (also called kernels), the parameters of which can be learned through a training process. The convolutional layer uses the kernel to perform a convolution operation on the input values ​​of the input image or video frame (e.g., sample values, MV values ​​for the first layer; outputs of the previous layer for subsequent layers) to extract basic features embedded in the image or video frame. The size of the kernel is typically smaller than the input image or video frame. Each kernel is convolved with the image or video frame and creates an activation map (also called a "feature map") consisting of neurons. The output volume of the convolutional layer is obtained by stacking the activation maps of all kernels along the depth dimension (an example of the channel dimension). In addition to the convolutional layers, some CNNs may also include one or more sub-pixel convolutional layers, one or more pooling layers, and / or one or more rectified linear unit ("ReLU") correction layers. The sub-pixel convolutional layer performs a standard convolution operation followed by a pixel shuffle operation. The pooling layer placed between two convolutional layers receives multiple activation maps and applies a pooling operation to each of them in order to reduce the spatial dimension while retaining the important features of the activation maps. The ReLU rectification layer acts as an activation function by replacing all negative values ​​received as input with zero.

[0139] This section describes example network structures of selected components of the neural video codec system (300) and the neural image codec system. For the convolutional layers and sub-pixel convolutional layers depicted in the examples, the symbols (K, C in , C out , S) indicate kernel size, input channel number, output channel number, and stride, respectively. Typically, stride is a kernel parameter that modifies the amount of movement on an image or video frame.

[0140] Fig.9A and Fig. 9B The current video frame x is shown respectively. t, and an example network structure of a context encoder (e.g., 304) and a context decoder (e.g., 316) for latent SV representations. In the depicted example, the input to the context encoder includes the current video frame x t (with three input channels, corresponding to the three color components at the corresponding spatial positions of the frame) and multi-scale temporal context parameter sets at different spatial resolutions (original, 2x downsampled, 4x downsampled) Each multi-scale temporal context parameter set has a total of 64 channels. The output of the context encoder is the current latent SV representation y with a total of 96 channels at a 16-fold downsampled spatial resolution. t The input to the context decoder consists of the decoded (reconstructed) current latent SV representation exist and Under the condition of , the current feature parameter set F with high resolution of 32 channels can be decoded t . As shown in the figure, the context encoder may include four convolutional layers, and the context decoder may include four sub-pixel convolutional layers. Additionally, each of the context encoder and the context decoder may include two bottleneck residual blocks for reducing the complexity of the intermediate layers, as described in Sheng 2021. The kernel size and stride of the convolutional layer of the bottleneck residual block are set to 3 and 1, respectively. Therefore, for simplicity, only the input channel number and output channel number for the bottleneck residual block are shown.

[0141] Fig.11 An example network structure of a hybrid entropy model network (e.g., 440) is shown in FIG. The input to the hybrid entropy model network includes decoded hyper-prior parameters with 192 channels, temporal context priors with 192 channels, and latent priors with 96 channels (i.e., decoded prior latent SV representations for previous video frames). ).

[0142] The decoded hyper-prior parameters can be obtained by Figure 5A The super prior decoder (510) is constructed by transforming the previous implicit representation y t Generative Hyper-Prior Decode to generate, such as Figure 5B As shown in the above reference Figure 5A As described above, the temporal context prior may be generated by a temporal context encoder (520). Specifically, the input to the temporal context encoder includes a temporal context parameter set And the output is the temporal context prior. Fig.12 An example network structure of a temporal context encoder is shown in . As shown, the temporal context encoder may include two convolutional layers separated by a leaky ReLU layer. Fig.13A and Fig. 13B , respectively, an example network structure of a super-a priori encoder (e.g., 502) and a super-a priori decoder (e.g., 510) is shown in FIG. As shown, the super-a priori encoder may include five convolutional layers separated by four leaky ReLU layers, while the super-a priori decoder may include three convolutional layers separated by four leaky ReLU layers and two sub-pixel convolutional layers.

[0143] Return to reference Fig.11 , the hybrid entropy model network is configured to perform the estimation process in two stages. The first stage has one convolutional layer used as a first fusion unit (e.g., 408) for prior fusion, followed by two convolutional layers and two leaky ReLU layers used together as a first statistics / parameter estimator (e.g., 410). In the first stage estimation, the hybrid entropy model network estimates not only two parts of the quantized latent current SV representation: and (e.g., the mean and scale parameters of the probability distribution of the first even number set (426) and the second odd number set (428)), and also generate a per-region QS value qs sc The second stage has one convolutional layer acting as a second fusion unit (e.g., 412), followed by two convolutional layers and two leaky ReLU layers that together act as a second statistics estimator (e.g., 414). In the second stage estimation, the quantized latent SV representation from the first stage part is fused to the input, and then the mixed entropy model network estimates the other two parts of the quantized latent current latent SV representation: and (eg, the mean and scale parameter of the probability distribution of the first set of odd numbers (430) and the second set of even numbers (432)).

[0144] Fig. 8A and Figure 8B 3 and 4 show example network structures of an MV context encoder (e.g., in MV encoder 328) and an MV context decoder (e.g., in MV decoder 330), respectively. The input to the MV context encoder is a current MV value set v having two channels for horizontal and vertical MV components, respectively. t , and the output of the MV context encoder is the current latent MV representation for the current MV value, denoted as mv_y t , with 16x downsampled spatial resolution and 64 channels. The MV context decoder generally follows the opposite structure. As shown in the figure, the MV context encoder has one convolutional layer and the MV context decoder has one sub-pixel convolutional layer. Both the MV context encoder and the MV context decoder include multiple residual blocks, including downsampled residual blocks in the MV context encoder and upsampled residual blocks in the MV context decoder. More details about these residual blocks can be found in Sheng 2021.

[0145] In some example embodiments, the MV encoder (328) and the MV decoder (330) include an entropy model network similar to one of the entropy model networks described above for SV information. For example, the MV encoder (328) and the MV decoder (330) may include a hybrid entropy model network having a structure similar to Fig.11 The network structure of the network structure shown in , but without the temporal context prior (based on (one or more) temporal context parameter sets) as input. Instead, the input includes a hyper-prior and a latent prior (a previous latent MV representation) for the current latent MV representation. The number of input channels of the first prior fusion layer is different (e.g., 192 channels lower), and the number of output channels and input channels of the corresponding layers of the network structure can be reduced accordingly, but the overall organization can be the same.

[0146] In some example embodiments, the MV encoder (328) includes a super-a priori encoder and a super-a priori decoder, and the MV decoder (330) includes a super-a priori decoder, similar to those described above for the SV information. For example, the super-a priori encoder and super-a priori decoder for the current hidden MV representation may have a similar Fig.13A and Fig. 13B The network structure of the super-prior encoder and super-prior decoder shown in .

[0147] Fig. 10A An example network structure of a frame generator (e.g., 318) is shown in . In the depicted example, the frame generator has a structure based on W-Net. For more details on the W-Net structure, in another context, reference can be made to "W-net: A deep model for fully unsupervised image segmentation" by Xia and Kulis, arXiv preprint arXiv:1711.08506 (2017). Such a network design can effectively expand the receptive field of the model while maintaining acceptable complexity and improving the generation capability of the model. As shown in the figure, the input to the frame generator includes a high-resolution current feature parameter set with 32 channels (output from the context decoder) and a temporal context parameter set with the original spatial resolution as the input video frame and 64 channels The output of the frame generator includes the final reconstructed current video frame and a set of feature parameters that can be used to encode / decode subsequent video frames. Fig. 10A The W-Net depicted in Figure 1 is formed by bridging two U-Nets and includes one convolutional layer and two sub-pixel convolutional layers, as well as multiple polling layers and residual blocks, including residual blocks with attention. Fig. 10BAn example network structure of a residual block with attention is further shown in . In the depicted example, the residual block with attention includes three convolutional layers, one polling layer, and several activation layers, including two ReLU layers, two linear activation layers, and a sigmoid activation layer.

[0148] As described above, the hybrid entropy model network described in this article supports content-adaptive quantization that allows multiple rates to be handled in a single model. Certain aspects of the entropy model network can be used for image encoding / decoding as well as video encoding / decoding. Therefore, a neural image codec system that supports this capability can be implemented for intra-frame encoding / decoding. Figure 14A-14B Depicted are example network structures of a neural image context encoder and a neural image context decoder, respectively. As shown, the neural image context encoder includes a convolutional layer and multiple residual blocks, including a downsampling residual block. The neural image context decoder includes a convolutional layer, a subpixel convolutional layer, a U-Net, and multiple residual blocks, including an upsampling residual block. The U-Net (which may have Fig. 10A A network structure similar to the one depicted in ) is incorporated into the neural image context decoder to improve the generation capabilities of the neural image encoder-decoder system.

[0149] In some examples, the same multi-granular quantization / inverse quantization described above can also be used in a neural image encoder / decoder, for example, to generate a t The implicit SV in the frame represents intra_y t quantized version of , and reconstructs the intra-frame SV representation

[0150] In some examples, a similar entropy model network can be used to determine the intra-frame implicit SV representation intra_y t The statistical properties of the quantized version of , and generate QS values. For example, the only difference may be the input to the entropy model. For a neural image codec, the input to the entropy model network may include only the corresponding hyper-prior for the latent SV representation within the frame, without the latent prior and the temporal context prior (because there are no previous frames to use in encoding / decoding).

[0151] VIII. Example Methods for Neural Encoding and Neural Decoding

[0152] This section describes example methods of neural encoding and neural decoding. The methods described herein may be performed by computer-executable instructions stored in one or more computer-readable media (e.g., storage devices or other tangible media) or stored in one or more computer-readable storage devices (e.g., causing a computing system to perform the method). Such methods may be performed in software, firmware, hardware, or a combination thereof. Such methods may be performed at least in part by a computing system (e.g., one or more computing devices). While still implementing these techniques, the actions shown may be described from alternative perspectives. For example, from a different perspective, "receiving" may also be described as "sending."

[0153] Figure 15A-15B are flow charts showing overall methods 1500, 1550 for neural encoding and neural decoding, respectively, and may be, for example, represented by Figure 3 The invention is implemented by a video codec system (300) or another neural video codec system or a neural image codec system.

[0154] like Fig.15A As shown in , the neural encoding method 1500 begins at 1510, where a current frame is received by a neural encoder. In some examples, the neural encoder can be a neural video encoder (e.g., 340) configured to encode a video frame (e.g., 302), such as Figure 3 As shown. In some examples, the neural encoder can be a neural image encoder configured to encode a single image. For example, the neural encoder can be a motion vector encoder (328) configured to encode a set of MV values. Or, as another example, the neural encoder can be configured to encode sample values. At 1520, the neural encoder can encode the current frame to produce encoded data. Then at 1530, the neural encoder can output the encoded data as part of a bitstream.

[0155] like Fig. 15B As shown, the neural decoding method 1550 begins at 1560, where the encoded data as part of a bitstream can be received by a neural decoder. In some examples, the neural decoder can be a neural video decoder (e.g., 350) configured to generate a decoded video frame (e.g., 320), such as Figure 3 In some examples, the neural decoder may be a neural image decoder configured to decode a single image. For example, the neural decoder may be a neural image decoder configured to generate a decoded MV value set. The neural decoder may be configured to decode the sample values. At 1570, the neural decoder may decode the encoded data to reconstruct the current frame. Then, at 1580, the neural decoder may output the reconstructed current frame.

[0156] Fig.16A An example method 1600 for encoding a current frame is shown, which may be used with Fig.17A , Fig.18A and / or Fig.19A The methods shown are used in combination as described below. Fig. 16B An example method 1650 for decoding encoded data is shown, which can be used with Fig. 17B , Fig.18B and / or Fig.19B The methods shown are used in combination as described below.

[0157] like Fig.16A As shown, at 1610, the neural encoder can determine a current latent representation (e.g., y t ,mv_y t ). Then at 1620, the neural encoder may decode the current latent representation using an entropy model network (e.g., 310) comprising one or more convolutional layers.

[0158] like Fig. 16B As shown, at 1660, the neural decoder may use an entropy model network (e.g., 310) including one or more convolutional layers to reconstruct a current latent representation for the current frame. At 1670, the neural decoder may use a context decoder (e.g., 316) including one or more convolutional layers to estimate a current feature parameter set (e.g., ). Then, at 1680, the neural decoder can reconstruct the current frame from the estimated current set of feature parameters.

[0159] Fig.17A An example method 1700 is shown for encoding a current latent representation by using a latent prior as input to an entropy model network. Accordingly, Fig. 17B An example method 1750 is shown for reconstructing a current latent representation for a current frame by using latent priors as input to an entropy model network. The neural encoder implementing method 1700 is a neural video encoder, and the neural decoder implementing method 1750 is a neural video decoder.

[0160] like Fig.17A As shown, at 1710, the neural encoder can (as part of the entropy model network) be based at least in part on a previous implicit representation for a previous video frame (e.g., ) to estimate the statistical characteristics (e.g., the mean value, scale value of the probability distribution function) of the quantized version of the current latent representation. Then, at 1720, the neural encoder may entropy encode and decode the quantized version of the current latent representation based at least in part on the estimated statistical characteristics.

[0161] For example, the current latent representation is a current latent SV representation for a current video frame, and the previous latent representation is a previous latent SV representation for a previous video frame. In this case, when the neural encoder determines the current latent representation, the neural encoder uses a context encoder including one or more convolutional layers to determine the current latent SV representation.

[0162] Or, as another example, the current latent representation is a current MV representation for a current video frame, and the previous latent representation is a previous latent MV representation for a previous video frame. In this case, when the neural encoder determines the current latent representation, the neural encoder uses motion estimation to determine the MV value for the current video frame relative to the previous video frame, and uses the MV context encoder to determine the current latent MV representation from the MV value.

[0163] The neural encoder may quantize the current latent representation, thereby generating a quantized version of the current latent representation. In doing so, the neural encoder may apply at least some QS values ​​(such as per-region QS values) determined at least in part based on the previous latent representation using the entropy model network.

[0164] like Fig. 17B As shown, at 1760, the neural decoder can (using the entropy model network) at least in part based on a previous implicit representation for a previous video frame (e.g., ) to estimate the quantized version of the current latent representation (e.g. ) is estimated (e.g., a mean value, a scale value of a probability distribution function). Then, at 1770, the neural decoder can entropy decode a quantized version of the current latent representation based at least in part on the estimated statistical characteristics.

[0165] For example, the current latent representation is a current latent SV representation for a current video frame, and the previous latent representation is a previous latent SV representation for a previous video frame. In this case, the neural decoder can estimate a current feature parameter set for the current video frame from the current latent SV representation using the context decoder, reconstruct the current video frame from the estimated current feature parameter set, and output the reconstructed current video frame.

[0166] Or, as another example, the current latent representation is the current latent MV representation for the current video frame, and the previous latent representation is the previous latent MV representation for the previous video frame. In this case, the neural decoder can use the MV context decoder to determine the MV value for the current video frame from the current latent MV representation.

[0167] The neural decoder may inverse quantize the quantized version of the current latent representation. In doing so, the neural decoder may apply at least some QS values ​​(such as per-region QS values) determined at least in part based on the previous latent representation using the entropy model network.

[0168] During encoding or decoding, the estimation of statistical properties using the entropy model network may also be based at least in part on other inputs, such as super-prior parameters for the current video frame (generated from the current latent representation using a super-prior encoder) and / or (one or more) temporal context parameter sets for the current video frame (generated from previous feature parameter sets for previous video frames and MV values ​​for the current video frame using a temporal context mining network).

[0169] Fig.18A An example method 1800 for encoding a current latent representation using a dual-space prior in an entropy model network is shown. Accordingly, Fig.18B An example method 1850 is shown for reconstructing a current latent representation using a dual spatial prior in an entropy model network. The neural encoder implementing method 1800 may be a neural video encoder or a neural image encoder. The neural decoder implementing method 1850 may be a neural video decoder or a neural image decoder.

[0170] like Fig.18A As shown, at 1810, the neural encoder can transform the current latent representation (y) along the channel dimension (before estimating using the entropy model network). t ,mv_y t ) are divided into different channel sets, and the current latent representation (y t ,mv_y t ) are divided into different sets of spatial locations to form multiple sets of elements. At 1820, the neural encoder can (as part of the entropy model network) respectively estimate quantized versions of the multiple sets of elements (e.g., ) (e.g., an average value, a scale value of a probability distribution function). Specifically, based at least in part on a quantized version of a first element set among the plurality of element sets, the neural encoder may estimate a statistical characteristic of a quantized version of a second element set among the plurality of element sets. Then, at 1830, the neural encoder may perform entropy encoding and decoding on the quantized versions of the plurality of element sets, respectively, based at least in part on the estimated statistical characteristic.

[0171] For example, the current latent representation is the current latent SV representation for the current frame. In this case, when the neural encoder determines the current latent representation, the neural encoder uses the context encoder to determine the current latent SV representation.

[0172] Or, as another example, the current latent representation is a current latent MV representation for the current frame. In this case, when the neural encoder determines the current latent representation, the neural encoder uses motion estimation to determine an MV value for the current frame relative to a previous frame, and uses an MV context encoder to determine the current latent MV representation from the MV value.

[0173] The neural encoder may quantize the current latent representation, thereby generating a quantized version of a plurality of sets of elements for the current latent representation. In doing so, the neural encoder may apply at least some QS values ​​(such as per-region QS values) determined using the entropy model network.

[0174] like Fig.18B As shown, at 1860, the neural decoder may receive a quantized version of the current latent representation (e.g., ). The elements of the current latent representation are logically organized along the channel dimension and the two spatial dimensions. The elements of the quantized version of the current latent representation have been split into different sets of channels along the channel dimension and into different sets of spatial locations along the two spatial dimensions to form a plurality of sets of elements. Each set of elements in the plurality of sets of elements has a different combination of one of the different sets of channels and one of the different sets of spatial locations. At 1870, the neural decoder may estimate (using an entropy model network) the quantized versions of the plurality of sets of elements, respectively (e.g., ) (e.g., a mean value, a scale value of a probability distribution function). Specifically, based at least in part on a quantized version of a first set of elements among the plurality of sets of elements, the neural decoder may estimate a statistical characteristic of a quantized version of a second set of elements among the plurality of sets of elements. Then, at 1880, the neural decoder may entropy decode the quantized versions of the plurality of sets of elements, respectively, based at least in part on the estimated statistical characteristic.

[0175] For example, the current latent representation is a current latent SV representation for the current frame. In this case, the neural decoder can use the context decoder to estimate a current feature parameter set for the current frame from the current latent SV representation, reconstruct the current frame from the estimated current feature parameter set, and output the reconstructed current frame.

[0176] Or, as another example, the current latent representation is a current latent MV representation for the current frame. In this case, the neural decoder can use the MV context decoder to determine the MV value for the current frame from the current latent MV representation.

[0177] The neural decoder may inverse quantize the quantized versions of the plurality of sets of elements for the current latent representation. In doing so, the neural decoder may apply at least some QS values ​​(such as per-region QS values) determined using the entropy model network.

[0178] The number of element sets depends on the implementation. In some example implementations, for dual spatial prior features, multiple element sets include: (a) a first element set having elements in a first channel set among different channel sets and in a first spatial position set among different spatial position sets; (b) a second element set having elements in a first channel set and in a second spatial position set among different spatial position sets; (c) a third element set having elements in a second channel set among different channel sets and in a second spatial position set; and (d) a fourth element set having elements in a second channel set and in a first spatial position set. For example, the first channel set includes the lower half of the channels, and the second channel set includes the upper half of the channels. Alternatively, the first channel set includes even channels, and the second channel set includes odd channels. The first spatial position set may include even positions, while the second spatial position set includes odd positions, and vice versa. Alternatively, the elements of the current implicit representation are divided into more or fewer element sets.

[0179] Fig.19A An example method 1900 for performing multi-granularity quantization is shown. Fig.19B An example method 1950 of performing multi-granularity inverse quantization is shown. A neural encoder implementing the method 1900 may be a neural video encoder or a neural image encoder. A neural decoder implementing the method 1950 may be a neural video decoder or a neural image decoder.

[0180] like Fig.19A As shown, at 1910, the neural encoder can determine the current latent representation (e.g., y t ,mv_y t ). The elements of the current latent representation are logically organized along the channel dimension and the two spatial dimensions. At 1920, the neural encoder may use different quantization step values ​​(e.g., qs ) in multiple stages, respectively. global ,qs ch and qs sc ) to quantize the current latent representation, thereby generating a quantized version of the current latent representation (e.g., ). Then, at 1930, the neural encoder may entropy encode and decode the quantized version of the current latent representation.

[0181] For example, the current latent representation is the current latent SV representation for the current frame. In this case, when the neural encoder determines the current latent representation, the neural encoder uses the context encoder to determine the current latent SV representation.

[0182] Or, as another example, the current latent representation is a current latent MV representation for the current frame. In this case, when the neural encoder determines the current latent representation, the neural encoder uses motion estimation to determine an MV value for the current frame relative to a previous frame, and uses an MV context encoder to determine the current latent MV representation from the MV value.

[0183] The neural encoder may use an entropy model network to estimate statistical properties of a quantized version of the current latent representation, and entropy encoding may use such statistical properties. The neural encoder may also use an entropy model network to determine at least some of the QS values.

[0184] like Fig.19B As shown, at 1960, the neural decoder may receive a quantized version of the current latent representation (e.g., ), the elements of the current latent representation are logically organized along the channel dimension and the two spatial dimensions. At 1970, the neural decoder can entropy decode the quantized version of the current latent representation. Then, at 1980, the neural decoder can use different quantization step values ​​(e.g., qs ) in multiple stages, respectively. global ,qs ch and qs sc ) performs inverse quantization on the quantized version of the current latent representation.

[0185] For example, the current latent representation is a current latent SV representation for the current frame. In this case, the neural decoder can use the context decoder to estimate a current feature parameter set for the current frame from the current latent SV representation, reconstruct the current frame from the estimated current feature parameter set, and output the reconstructed current frame.

[0186] Or, as another example, the current latent representation is a current latent MV representation for the current frame. In this case, the neural decoder can use the MV context decoder to determine the MV value for the current frame from the current latent MV representation.

[0187] The neural decoder may use an entropy model network to estimate statistical properties of a quantized version of the current latent representation, and entropy decoding may use such statistical properties. The neural decoder may also use an entropy model network to determine at least some of the QS values.

[0188] During encoding or decoding, different QS values ​​for quantization (encoding) or inverse quantization (encoding / reconstruction or decoding) may include a global QS value for adjusting the bit rate and overall quality. The encoded data may include one or more syntax elements indicating a global QS value that is allowed to vary within a certain range. Different QS values ​​may also include multiple per-channel QS values ​​for different channels of the current implicit representation. Multiple per-channel QS values ​​may be predefined. Alternatively, multiple per-channel QS values ​​may vary over time, in which case the encoded data may include syntax elements indicating multiple per-channel QS values. Different QS values ​​may also include multiple per-region QS values ​​for different spatial regions of the current implicit representation. Different spatial regions may be associated with different locations or regions of the current implicit representation. Different per-region QS values ​​may be channel-specific or channel-independent. Alternatively, different QS values ​​include other and / or additional QS values.

[0189] In some example implementations, for quantization, the multiple stages include (a) a first stage including quantizing corresponding elements of a current implicit representation using a global QS value, (b) a second stage including quantizing corresponding elements of the current implicit representation in different channels using multiple per-channel QS values, and (c) a third stage including quantizing corresponding elements of the current representation in different spatial regions for different channels using per-region QS values. For inverse quantization, the multiple stages include (c') a first stage including inverse quantizing corresponding elements of the current representation in different spatial regions for different channels using per-region QS values, (b') a second stage including inverse quantizing corresponding elements of the current implicit representation in different channels using multiple per-channel QS values, and (a) a third stage including inverse quantizing corresponding elements of the current implicit representation using a global QS value. Alternatively, the stages of quantization and inverse quantization may be performed in a different order.

[0190] IX. Example Experimental Results

[0191] An experimental study has been conducted on the neural network video encoding and decoding technique described in this paper to evaluate its performance.

[0192] When training the neural network video codec in the experimental scenario, the training data comes from Vimeo-90k (see Xue et al., “Video Enhancement with Task-Oriented Flow,” International Journal of Computer Vision (IJCV) 127, 8:1106–1125, 2019). The videos are randomly cropped into 256×256 image patches. The same test sequences described in Sheng 2021 are used in the testing phase. All of these sequences are widely used in traditional and neural network video codecs, including HEVC B, C, D, E classes and RGB classes. In addition, 1080p videos from UVG and MCL-JCV datasets are also tested. The UVG dataset is described in Mercat et al., “UVG dataset: 50 / 120fps 4K sequences for video codec analysis and development,” Proceedings of the 11th ACM Multimedia Systems Conference. 297–302, 2020; the MCL-JCV dataset is described in Wang et al., “MCL-JCV: a JND-based H.264 / AVC video quality assessment dataset,” 2016 IEEE International Conference on Image Processing (ICIP). IEEE, 1509–1513 (2016).

[0193] In the experimental scenario, 96 frames were tested for each video when training the neural network video codec. To be closer to the actual application scenario, the intra frame period was set to 32. The training used a low-latency encoding setting, similar to most existing works. The compression rate is measured by BD-Rate (see Bjontegaard, "Calculation of average PSNRdifferences between RDcurves," VCEG-M33 (2001)), where negative numbers represent bit rate savings and positive numbers represent bit rate increases. In addition to the x265 encoder with the veryslow preset, HM-16.20 and VTM-13.2 are also used as comparison benchmarks, which represent the optimal encoders under the H.265 and H.266 standards, respectively. For HM and VTM, the configuration with the highest compression rate is used. The experimental results are also compared with existing state-of-the-art neural network video codecs, including DVC_Pro, MLVC, RLVC, DCVC, and Sheng2021. The DVC_Pro codec is described in Lu et al., “An end-to-end learning framework for video compression,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(10): 3292-3308 (2020); MLVC is described in Lin et al., “M-LVC: multiple frames prediction for learned video compression,” Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2020; RLVC is described in Yang et al., “Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model,” IEEE Journal of Selected Topics in Signal Processing 15(2): 388–401, 2021; DCVC is described in Li et al., “Deep contextual video compression,” Advances in Neural Information Processing Systems 34 (2021).

[0194] When training a neural network video codec, for the MV value v t The entropy model and quantization of the current latent representation follow the current latent SV representation y t The main difference lies in the input of the entropy model. t When encoding and decoding the current latent representation of , the input is the corresponding super prior and hidden prior, i.e., the quantized latent MV representation of the MV value from the previous frame. There is no temporal context prior here, because the generation of temporal context depends on the decoded MV value. In addition, the neural image codec is trained to support the ability to have multiple bitrates for intra-frame coding in a single trained model, as mentioned above in conjunction with Figure 14A-14B described.

[0195] During training, the loss function includes distortion and bitrate: Loss = λ·D + R, where D represents the distortion between the input current video frame and the reconstructed current video frame. For different visual targets, the distortion can be, for example, L2Loss or MS-SSIM (described in Wang et al., “Multiscale structural similarity for image quality assessment,” in the Thirty-Seventh Asilomar Conference on Signals, Systems & Computers, Vol. 2. IEEE, 1398–1402, 2003). R represents the quantized latent SV representation used to encode and the bits of the quantized latent representation of the MV value, each associated with its own hyper-prior.

[0196] In the experimental scenario, the neural video codec was trained using a multi-stage training approach as described in Sheng2021. In addition, to support multiple bitrates in a single model, different λ values ​​were used in different optimization steps. To simplify training, four λ values ​​(85, 170, 380, 840) were used. Four random qs were set and learned using the RD loss with each corresponding λ value. global It should be noted that although only four values ​​of λ were used in training, during testing, by adjusting qs global A wide range of bit rates can still be achieved.

[0197] The result of the training is the following neural encoder / decoder, whose parameters are set to the convolutional layers and other layers in the network structure of each component in the neural encoder and neural decoder. In some example implementations, per-channel QS values ​​are also defined.

[0198] Experimental results show that the neural video codec technology described in this paper has improved performance compared to existing neural video codec technologies, including encoders for the latest legacy standard H.266 standard. For example, experimental results show that on the UVG dataset, the neural video codec technology achieves 67.4% and 57.1% bitrate savings relative to DVCPro and DCVC, respectively. In addition, the neural video codec technology saves an average of 4.7% bitrate compared to VTM. This marks the first neural video codec that surpasses VTM using the highest compression ratio configuration. In particular, for 1080p video (HEVC B, HEVC RGB, UVG, MCL-JCV), the neural video codec technology described in this paper performs better. These results demonstrate the effectiveness of hybrid entropy models in exploiting a large number of data correlations. When facing MS-SSIM, experimental results show that the neural video codec technology also brings significant improvements, such as a 46.4% bitrate savings relative to VTM.

[0199] As mentioned above, the neural network video codec technology in this paper can achieve multiple bitrates in a single model. The global QS value qs can be flexibly adjusted during testing global Its role is similar to the quantization parameter in traditional video codecs. During training, qs global Guided by RD loss. In the experiment, by learning the maximum and minimum qs global Interpolate between to get 30 qs global The experimental results confirm that a single model can achieve refined bitrate control without outliers. In contrast, previous methods such as DCVC and Sheng 2021 require training a model for each bitrate point.

[0200] The complexity of the models can be compared in terms of model size, MACs (number of multiply-accumulate operations), peak feature usage, encoding time, and decoding time. 1080p video frames are used as input to measure these metrics. In terms of encoding / decoding time, the time on the V100 GPU is measured, including the time to read and write the bitstream from the bitstream. Since the neural network video codec technology described in this article supports multiple bitrates in a single model, the burden of model training and storage is greatly reduced. Specifically, compared with DCVC, which uses a parallel-unfriendly autoregressive prior model, the neural video codec technology described in this article also has a significant reduction in encoding / decoding time.

[0201] V. Features

[0202] Various embodiments may include one or more of the inventive features shown in the following Features List.

[0203]

[0204]

[0205]

[0206]

[0207]

[0208]

[0209]

[0210]

[0211]

[0212]

[0213]

[0214]

[0215]

[0216]

[0217]

[0218]

[0219]

[0220]

[0221] In view of the many possible embodiments to which the principles of the disclosed invention may be applied, it should be appreciated that the illustrated embodiments are merely preferred examples of the invention and should not be considered to limit the scope of the invention. Instead, the scope of the invention is defined by the following claims. Therefore, we claim protection for the invention that falls within the scope and spirit of these claims.

Claims

1. In a computer system implementing a neural video encoder, a method includes: Receive the current video frame; Encoding the current video frame to generate encoded data, wherein encoding the current video frame comprises: determining a current implicit representation for the current video frame; and Encoding the current latent representation using an entropy model network including one or more convolutional layers, wherein encoding the current latent representation using the entropy model network includes: The current implicit representation is estimated based at least in part on a previous implicit representation for a previous video frame. Statistical properties of the quantized version of the latent representation; and Based at least in part on the estimated statistical characteristic, The quantized version is entropy encoded and decoded; and The encoded data is output as part of a bitstream.

2. The method according to claim 1, further comprising: The current latent representation is quantized to generate the quantized version of the current latent representation.

3. The method according to claim 2, wherein encoding the current latent representation using the entropy model network further comprises: Based at least in part on the previous implicit representation, at least some quantization step size (QS) values ​​are determined for the current implicit representation, wherein the quantization uses the at least some QS values.

4. The method of claim 1 , wherein the current latent representation is a current latent sample value (“SV”) representation for the current video frame, wherein the previous latent representation is a previous latent SV representation for the previous video frame, and wherein determining the current latent representation comprises determining the current latent SV representation using a context encoder comprising one or more convolutional layers.

5. The method of claim 1 , wherein the current implicit representation is a current implicit motion vector (“MV”) representation for the current video frame, wherein the previous implicit representation is a previous implicit MV representation for the previous video frame, and wherein determining the current implicit representation comprises: determining, using motion estimation, an MV value for the current video frame relative to the previous video frame; as well as The current latent MV representation is determined from the MV value using an MV context encoder comprising one or more convolutional layers.

6. In a computer system implementing a neural video decoder, a method comprises: Receiving encoded data as part of a bitstream; as well as Decoding the encoded data to reconstruct a current video frame, wherein decoding the encoded data comprises: Reconstructing a current latent representation for the current video frame using an entropy model network including one or more convolutional layers, wherein reconstructing the current latent representation includes: The current implicit representation is estimated based at least in part on a previous implicit representation for a previous video frame. Statistical properties of the quantized version of the latent representation; Based at least in part on the estimated statistical characteristic, The quantized version is entropy decoded.

7. The method according to claim 6, further comprising: The quantized version of the current implicit representation is inverse quantized.

8. The method of claim 7, wherein reconstructing the current latent representation using the entropy model network further comprises: Based at least in part on the previous implicit representation, at least some quantization step size (QS) values ​​are determined for the current implicit representation, wherein the inverse quantization uses the at least some QS values.

9. The method of claim 6, wherein the current implicit representation is a current implicit sample value ("SV") representation for the current video frame, wherein the previous implicit representation is a previous implicit SV representation for the previous video frame, and wherein the method further comprises: estimating a current set of feature parameters for the current video frame from the current latent SV representation using a context decoder comprising one or more convolutional layers; reconstructing the current video frame from the estimated current feature parameter set; as well as Output the reconstructed current video frame.

10. The method of claim 6, wherein the current implicit representation is a current implicit motion vector ("MV") representation for the current video frame, wherein the previous implicit representation is a previous implicit MV representation for the previous video frame, and wherein the method further comprises: An MV context decoder comprising one or more convolutional layers is used to determine an MV value for the current video frame from the current latent MV representation.

11. A method according to any one of claims 1-10, wherein estimating the statistical characteristics of the quantized version of the current implicit representation is also based at least in part on super-prior parameters for the current video frame, which have been generated from the current implicit representation using a super-prior encoder comprising one or more convolutional layers.

12. A method according to any one of claims 1-4 and 6-9, wherein estimating the statistical characteristics of the quantized version of the current implicit representation is also based at least in part on one or more temporal context parameter sets for the current video frame, which have been generated from previous feature parameter sets for the previous video frame and motion vector ("MV") values ​​for the current video frame using a temporal context mining network comprising one or more convolutional layers.

13. The method of any one of claims 1-10, wherein the statistical characteristics include one or more means and one or more scale parameters of a probability distribution function for the quantized version of the current latent representation.

14. A method according to any one of claims 1-13, wherein the elements of the current latent representation are logically organized along a channel dimension and two spatial dimensions.

15. The method of claim 14, wherein estimating the statistical property of the quantized version of the current implicit representation comprises: Dividing the elements of the current latent representation into different channel sets along the channel dimension, and dividing the elements of the current latent representation into different spatial position sets along the two spatial dimensions to form a plurality of element sets, each element set in the plurality of element sets having a different combination of a channel set in the different channel sets and a spatial position set in the different spatial position sets; as well as The statistical characteristic of a quantized version of a second set of elements among the plurality of sets of elements is estimated based at least in part on a quantized version of a first set of elements among the plurality of sets of elements.

16. The method of claim 15, wherein the plurality of sets of elements comprises: the first set of elements having elements in a first set of channels among the different sets of channels and in a first set of spatial positions among the different sets of spatial positions; the second set of elements having elements that are in the first set of channels and in a second set of spatial locations among the different sets of spatial locations; a third set of elements having elements in a second set of channels among the different sets of channels and in the second set of spatial locations; as well as A fourth set of elements having elements that are in the second set of channels and in the first set of spatial locations.

17. The method of claim 16, wherein estimating the statistical property of the quantized version of the current implicit representation comprises: estimating a statistical property of the quantized version of the first set of elements; estimating a statistical property of a quantized version of the third set of elements; fusing the quantized version of the first set of elements and the quantized version of the third set of elements with other inputs; as well as using the fused result to estimate statistical properties of a quantized version of the fourth set of elements; Wherein estimating the statistical property of the quantized version of the second set of elements also uses the result of the fusion.

18. The method of claim 1, further comprising: The current implicit representation is quantized in multiple stages using different quantization step size ("QS") values ​​in the multiple stages respectively, thereby generating the quantized version of the current implicit representation.

19. The method according to claim 6, further comprising: The quantized version of the current implicit representation is inverse quantized in multiple stages using different quantization step size ("QS") values ​​in the multiple stages respectively.

20. The method of claim 18 or 19, wherein the different QS values ​​include: A global QS value for adjusting bitrate and overall quality; a plurality of per-channel QS values ​​for different channels of the current implicit representation; as well as A plurality of per-region QS values ​​for different spatial regions of the current implicit representation, the different spatial regions being associated with different positions or different regions of the current implicit representation, and the different per-region QS values ​​being channel-specific or channel-independent.

21. In a computer system implementing a neural image encoder or a neural video encoder, a method comprising: Receive the current frame; Encoding the current frame to generate encoded data, wherein encoding the current frame comprises: determining a current latent representation for the current frame, wherein elements of the current latent representation are logically organized along a channel dimension and two spatial dimensions; and Encoding the current latent representation using an entropy model network including one or more convolutional layers, wherein encoding the current latent representation using the entropy model network includes: Dividing the elements of the current latent representation into different channel sets along the channel dimension, and dividing the elements of the current latent representation into different spatial position sets along the two spatial dimensions to form a plurality of element sets, each element set in the plurality of element sets having a different combination of a channel set in the different channel sets and a spatial position set in the different spatial position sets; respectively estimating statistical properties of quantized versions of the plurality of element sets, including at least Based in part on the quantized version of a first set of elements among the plurality of sets of elements estimating the statistical characteristic of the quantized version of a second set of elements among the plurality of sets of elements; and Based at least in part on the estimated statistical characteristic, entropy encoding and decoding the quantized versions of the plurality of sets of elements, respectively; and The encoded data is output as part of a bitstream.

22. The method according to claim 21, further comprising: The current latent representation is quantized to generate the quantized versions of the plurality of sets of elements for the current latent representation.

23. The method of claim 22, wherein encoding the current latent representation using the entropy model network further comprises: At least some quantization step size (QS) values ​​are determined for the current implicit representation, wherein the quantization uses the at least some QS values.

24. The method of claim 21, wherein the current latent representation is a current latent sample value ("SV") representation for the current frame, and wherein determining the current latent representation comprises determining the current latent SV representation using a context encoder comprising one or more convolutional layers.

25. The method of claim 21, wherein the current implicit representation is a current implicit motion vector ("MV") representation for the current frame, and wherein determining the current implicit representation comprises: determining, using motion estimation, a MV value for the current frame relative to a previous frame; as well as The current latent MV representation is determined from the MV values ​​using an MV context encoder comprising one or more convolutional layers.

26. In a computer system implementing a neural image decoder or a neural video decoder, a method comprising: Receiving encoded data as part of a bitstream; as well as Decoding the encoded data to reconstruct a current frame, wherein decoding the encoded data comprises: Reconstructing a current latent representation for the current frame using an entropy model network including one or more convolutional layers, wherein elements of the current latent representation are logically organized along a channel dimension and two spatial dimensions, the elements of the current latent representation have been divided into different channel sets along the channel dimension and divided into different spatial position sets along the two spatial dimensions to form a plurality of element sets, each element set in the plurality of element sets having a different combination of one channel set in the different channel sets and one spatial position set in the different spatial position sets, and wherein reconstructing the current latent representation comprises: respectively estimating statistical properties of quantized versions of the plurality of element sets, including at least Based in part on the quantized version of a first set of elements among the plurality of sets of elements estimating a statistical property of the quantized version of a second set of elements among the plurality of sets of elements; and Based at least in part on the estimated statistical characteristic, the quantized versions of the plurality of sets of elements are respectively entropy decoded.

27. The method according to claim 26, further comprising: The quantized versions of the plurality of sets of elements for the current implicit representation are inverse quantized.

28. The method of claim 27, wherein reconstructing the current latent representation using the entropy model network further comprises: At least some quantization step size (QS) values ​​for the current implicit representation are determined, wherein the inverse quantization uses the at least some QS values.

29. The method of claim 26, wherein the current implicit representation is a current implicit sample value ("SV") representation for the current frame, and wherein the method further comprises: estimating a current set of feature parameters for a current frame from the current latent SV representation using a context decoder comprising one or more convolutional layers; reconstructing the current frame from the estimated set of current feature parameters; as well as Output the reconstructed current frame.

30. The method of claim 26, wherein the current implicit representation is a current implicit motion vector ("MV") representation for the current frame, and wherein the method further comprises: An MV value for the current frame is determined from the current latent MV representation using an MV context decoder comprising one or more convolutional layers.

31. A method according to any one of claims 21-30, wherein the statistical characteristics include one or more mean values ​​and one or more scale parameters, and the one or more mean values ​​and one or more scale parameters are probability distribution functions for each quantized version of the quantized versions of the multiple element sets respectively.

32. The method according to any one of claims 21 to 30, wherein the plurality of sets of elements comprises: the first set of elements having elements in a first set of channels among the different sets of channels and in a first set of spatial positions among the different sets of spatial positions; the second set of elements having elements that are in the first set of channels and in a second set of spatial locations among the different sets of spatial locations; a third set of elements having elements in a second set of channels among the different sets of channels and in the second set of spatial locations; as well as A fourth set of elements having elements that are in the second set of channels and in the first set of spatial locations.

33. The method of claim 32, wherein respectively estimating the statistical properties of the quantized versions of the plurality of sets of elements comprises: estimating the statistical property of the quantized version of the first set of elements; estimating the statistical property of the quantized version of the third set of elements; fusing the quantized version of the first set of elements and the quantized version of the third set of elements with other inputs; as well as using a result of the fusion to estimate the statistical property of the quantized version of the fourth set of elements; Wherein estimating the statistical property of the quantized version of the second set of elements also uses the result of the fusion.

34. The method of claim 32, wherein: The first set of channels includes a first half of the channels; The second set of channels includes a second half of the channels; The first set of spatial positions includes one of even positions and odd positions; and The second set of spatial positions includes the other of the even positions and the odd positions.

35. The method of claim 32, wherein the entropy model network comprises: A first fusion stage, wherein the first fusion stage fuses the input; a first estimation stage, the first estimation stage using output from the first fusion stage to estimate the statistical properties of the quantized version of the first set of elements and the statistical properties of the quantized version of the third set of elements; a second fusion stage, the second fusion stage fusing the quantized version of the first set of elements and the quantized version of the third set of elements with other inputs, wherein the other inputs include some outputs from the first estimation stage; as well as A second estimation stage uses the output from the second fusion stage to estimate the statistical properties of the quantized version of the second set of elements and the statistical properties of the quantized version of the fourth set of elements.

36. The method of any one of claims 21-30, wherein estimating the statistical characteristic of the quantized version of the plurality of sets of elements is further based at least in part on: super-prior parameters for the current frame, the super-prior parameters having been generated from the current latent representation using a super-prior encoder comprising one or more convolutional layers; and If the current frame is a current video frame, it is also based at least in part on a previous implicit representation for a previous video frame.

37. The method of any one of claims 21-24 and 26-29, wherein estimating the statistical characteristic of the quantized version of the plurality of sets of elements is further based at least in part on: super-prior parameters for the current frame, the super-prior parameters having been generated from the current latent representation using a super-prior encoder comprising one or more convolutional layers; and If the current frame is a current video frame, then based at least in part on: a previous implicit representation for a previous video frame; and One or more temporal context parameter sets for the current video frame, the one or more temporal context parameter sets having been generated from previous feature parameter sets for the previous video frame and a motion vector ("MV") value for the current video frame using a temporal context mining network comprising one or more convolutional layers.

38. The method of claim 21, further comprising: The current implicit representation is quantized in multiple stages using different quantization step size ("QS") values ​​in the multiple stages respectively, thereby generating the quantized version of the current implicit representation.

39. The method of claim 26, further comprising: The quantized version of the current implicit representation is inverse quantized in multiple stages using different quantization step size ("QS") values ​​in the multiple stages respectively.

40. The method of claim 38 or 39, wherein the different QS values ​​include: A global QS value for adjusting bitrate and overall quality; a plurality of per-channel QS values ​​for different channels of the current implicit representation; as well as A plurality of per-region QS values ​​for different spatial regions of the current implicit representation, the different spatial regions being associated with different positions or different regions of the current implicit representation, and the different per-region QS values ​​being channel-specific or channel-independent.

41. In a computer system implementing a neural image encoder or a neural video encoder, a method comprising: Receive the current frame; Encoding the current frame to generate encoded data, wherein encoding the current frame comprises: determining a current latent representation for the current frame, wherein elements of the current latent representation are logically organized along a channel dimension and two spatial dimensions; quantizing the current implicit representation in a plurality of stages using different quantization step size ("QS") values ​​in the plurality of stages, thereby generating quantized versions of the current implicit representation; and performing entropy coding and decoding on the quantized version of the current latent representation; and The encoded data is output as part of a bitstream.

42. The method of claim 41, wherein encoding the current frame further comprises: Statistical properties of the quantized version of the current latent representation are estimated using an entropy model network comprising one or more convolutional layers, wherein the entropy encoding and decoding is based at least in part on the estimated statistical properties.

43. The method of claim 42, wherein encoding the current frame further comprises: At least some of the QS values ​​are determined using the entropy model network.

44. The method of claim 41, wherein the current latent representation is a current latent sample value ("SV") representation for the current frame, and wherein determining the current latent representation comprises determining the current latent SV representation using a context encoder comprising one or more convolutional layers.

45. The method of claim 41, wherein the current implicit representation is a current implicit motion vector ("MV") representation for the current frame, and wherein determining the current implicit representation comprises: determining, using motion estimation, a MV value for the current frame relative to a previous frame; as well as The current latent MV representation is determined from the MV value using an MV context encoder comprising one or more convolutional layers.

46. ​​In a computer system implementing an image decoder or a video decoder, a method comprising: Receiving encoded data as part of a bitstream; Decoding the encoded data to reconstruct a current frame, wherein decoding the encoded data comprises: Reconstructing a current latent representation for the current frame, wherein elements of the current latent representation are logically organized along a channel dimension and two spatial dimensions, wherein reconstructing the current latent representation comprises: entropy decoding a quantized version of the current latent representation; and Different quantization step sizes in the multiple stages are used respectively in the multiple stages ("QS") value to inverse quantize the quantized version of the current implicit representation.

47. The method of claim 46, wherein decoding the current frame further comprises: Statistical properties of the quantized version of the current latent representation are estimated using an entropy model network comprising one or more convolutional layers, wherein the entropy decoding is based at least in part on the estimated statistical properties.

48. The method of claim 47, wherein decoding the current frame further comprises: At least some of the QS values ​​are determined using the entropy model network.

49. The method of claim 46, wherein the current implicit representation is a current implicit sample value ("SV") representation for the current frame, and wherein the method further comprises: estimating a current set of feature parameters for the current frame from the current latent SV representation using a context decoder comprising one or more convolutional layers; reconstructing the current frame from the estimated set of current feature parameters; as well as Output the reconstructed current frame.

50. The method of claim 46, wherein the current implicit representation is a current implicit motion vector ("MV") representation for the current frame, and wherein the method further comprises: An MV context decoder including one or more convolutional layers is used to determine an MV value for the current frame from the current latent MV representation.

51. The method of any one of claims 41-50, wherein the different QS values ​​include: A global QS value for adjusting bitrate and overall quality; a plurality of per-channel QS values ​​for different channels of the current implicit representation; as well as A plurality of per-region QS values ​​for different spatial regions of the current implicit representation, the different spatial regions being associated with different positions or different regions of the current implicit representation, and the different per-region QS values ​​being channel-specific or channel-independent.

52. The method according to any one of claims 41 to 45, wherein: The multiple stages include: A first stage, the first stage comprising quantizing corresponding elements of the current latent representation using a global QS value among the different QS values; a second stage, the second stage comprising quantizing the corresponding elements in different channels of the current implicit representation using a plurality of per-channel QS values ​​among the different QS values; and A third stage comprises quantizing the corresponding elements of the current representation in different spatial regions for the different channels using per-region QS values ​​among the different QS values.

53. The method of any one of claims 46-50, wherein the plurality of stages comprises: a first stage comprising inverse quantizing corresponding elements of the current representation in different spatial regions for different channels using per-region QS values ​​among the different QS values; a second stage, the second stage comprising inverse quantizing the corresponding elements of the current implicit representation in the different channels using a plurality of per-channel QS values ​​among the different QS values; as well as The third stage includes using the global QS value among the different QS values ​​to inverse quantize the corresponding elements of the current implicit representation.

54. The method of any one of claims 41-50, wherein the different QS values ​​comprise a global QS value, and wherein the encoded data comprises one or more syntax elements indicating the global QS value, the global QS value being allowed to vary within a range.

55. The method of any one of claims 41-50, wherein the different QS values ​​comprise a plurality of per-channel QS values, and wherein: The plurality of per-channel QS values ​​are predefined; or The encoded data includes syntax elements indicating the plurality of per-channel QS values.

56. The method of any one of claims 41-50, further comprising estimating a statistical property of the quantized version of the current implicit representation based at least in part on: super-prior parameters for the current frame, the super-prior parameters having been generated from the current latent representation using a super-prior encoder comprising one or more convolutional layers; and If the current frame is a current video frame, statistical properties of the quantized version of the current latent representation are also estimated based at least in part on a previous latent representation for a previous video frame.

57. The method of any one of claims 41-44 and 46-49, further comprising estimating a statistical property of the quantized version of the current implicit representation based at least in part on: super-prior parameters for the current frame, the super-prior parameters having been generated from the current latent representation using a super-prior encoder comprising one or more convolutional layers; and If the current frame is a current video frame, estimating a statistical characteristic of the quantized version of the current implicit representation is also based at least in part on: a previous implicit representation for a previous video frame; and One or more temporal context parameter sets for the current video frame, the one or more temporal context parameter sets having been generated from previous feature parameter sets for the previous video frame and a motion vector ("MV") value for the current video frame using a temporal context mining network comprising one or more convolutional layers.

58. The method of any one of claims 41-50, further comprising estimating statistical properties of the quantized version of the current implicit representation, comprising: Dividing the elements of the current latent representation into different channel sets along the channel dimension, and dividing the elements of the current latent representation into different spatial position sets along the two spatial dimensions to form a plurality of element sets, each element set in the plurality of element sets having a different combination of a channel set in the different channel sets and a spatial position set in the different spatial position sets; as well as The statistical characteristic of a quantized version of a second set of elements among the plurality of sets of elements is estimated based at least in part on a quantized version of a first set of elements among the plurality of sets of elements.

59. The method of claim 58, wherein the plurality of sets of elements comprises: the first set of elements having elements in a first set of channels among the different sets of channels and in a first set of spatial positions among the different sets of spatial positions; the second set of elements having elements that are in the first set of channels and in a second set of spatial locations among the different sets of spatial locations; a third set of elements having elements in a second set of channels among the different sets of channels and in the second set of spatial locations; as well as A fourth set of elements having elements that are in the second set of channels and in the first set of spatial locations.

60. The method of claim 59, wherein estimating the statistical property of the quantized version of the current implicit representation comprises: estimating a statistical property of the quantized version of the first set of elements; estimating a statistical property of a quantized version of the third set of elements; fusing the quantized version of the first set of elements and the quantized version of the third set of elements with other inputs; as well as using the fused result to estimate statistical properties of a quantized version of the fourth set of elements; Wherein estimating the statistical property of the quantized version of the second set of elements also uses the result of the fusion.

61. One or more non-transitory computer-readable media storing computer-executable instructions for causing a computer system to perform the operations of the method of any one of claims 1-60 when programmed by the computer-executable instructions.

62. A computer system configured to perform the operations of the method according to any one of claims 1-5, 11-25, 30-45 and 51-60, the computer system comprising: a frame buffer configured to store the current frame or current video frame; a video encoder configured to perform said encoding; as well as A coded data buffer is configured to store the coded data for output.

63. A computer system configured to perform the operations of the method according to any one of claims 6-20, 26-40 and 46-60, the computer system comprising: a coded data buffer configured to store the coded data; a video decoder configured to perform said decoding; as well as A frame buffer is configured to store the reconstructed current video frame for output.

Images