Quality propagation with corruption for low-delay video coders

Abstract

Example techniques and systems for training a P-frame model are disclosed. An example system includes one or more memories configured to store media data and one or more processors implemented in circuitry and coupled to the one or more memories. The one or more processors are configured to obtain a corrupted I-frame. The corrupted I-frame has at least one of: a) a peak signal-to-noise ratio that satisfies a threshold, or b) one or more areas with insertion errors. The one or more processors are configured to train a P-frame model using the corrupted I-frame.

Description

[0001] This application claims priority to U.S. Patent Application No. 18 / 410,581, filed January 11, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to media encoding and decoding, including encoding and decoding image and video data. Background Technology

[0003] Digital media capabilities can be incorporated into a wide variety of devices, including digital televisions, digital direct broadcast systems, wireless direct broadcast systems, personal digital assistants (PDAs), laptops or desktop computers, tablets, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite wireless phones (so-called "smartphones"), video conferencing equipment, video streaming devices, and so on. Digital video devices implement video decoding technologies, such as those defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4 (Part 10, Advanced Video Decoding (AVC)), ITU-T H.265 / High-Efficiency Video Decoding (HEVC), ITU-T H.266 / Variety Video Decoding (VVC) and extensions to such standards, as well as proprietary video decoders / formats such as AOMedia Video 1 (AV1) developed by the Open Media Consortium. By implementing such video decoding technologies, video devices can more efficiently send, receive, encode, decode, and / or store digital video information. Summary of the Invention

[0004] Generally, this disclosure describes techniques for media compression, including techniques for video and / or image encoding and decoding. Neural network-based media (e.g., image and / or video) compression methods can outperform current standards and offer several additional advantages. A neural network-based video decoder (which may be a media decoder) may be referred to as a neural video decoder (NVC).

[0005] This disclosure addresses techniques for improving decoding using neural media decoders (such as NVCs). Some example NVCs utilize nonparametric methods for univariate density estimation, thereby amortizing learning across channels, rather than using over-prior variables. However, the output variable can retain much information from the over-prior variable. Therefore, it may be desirable to utilize the over-prior variable when modeling the output variable.

[0006] This disclosure also describes improved training techniques. Training NVC can lead to the accumulation of temporal errors over larger test-time group of pictures (GOP) sizes. Therefore, some NVCs attempt to correct this accumulation of temporal errors by implementing a weighted objective, where the importance of distortion increases with depth in the training sequence. While this technique can delay degradation, it still struggles to maintain quality over longer image sequences. Therefore, it may be desirable to train the NVC such that it can accumulate improved reconstructions across subsequent training frames.

[0007] While the decoding capabilities of NVC have improved, work remains to be done to make NVC competitive with conventional or manually designed methods, such as codecs implementing VVC. It should be noted that the terms codec and decoder are used interchangeably in this disclosure. This disclosure discusses several techniques for both neural image and video compression (e.g., media compression). These techniques are generally simple, model-independent, and readily incorporable into other techniques. This disclosure also identifies current evaluation protocols that are significantly disadvantageous to conventional, manually designed decoders and provides a basis for encouraging future comparisons using test conditions from video standards organizations. Decoders built on conditional decoding frameworks based on the techniques of this disclosure can outperform disclosed methods—with a significant margin of advantage over many disclosed methods. Neural decoders utilizing such techniques can outperform VVC on the metrics expected by VVC (weighted YUV-PSNR).

[0008] In one example, a method for training a P-frame model for a neural media decoder includes: acquiring a corrupted I-frame having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold, or b) having one or more regions with insertion errors; and using the corrupted I-frame to train the P-frame model.

[0009] In another example, a method for decoding media data includes: applying a pre-trained P-frame model to the media data, the pre-trained P-frame model being trained using corrupted I-frames having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold, or b) one or more regions with insertion errors; and decoding the media data based on applying the pre-trained P-frame model to the media data.

[0010] In another example, a device includes: one or more memories configured to store media data and a P-frame model; and one or more processors implemented in a circuit and coupled to the one or more memories, the one or more processors being configured to: acquire a corrupted I-frame having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold, or b) having one or more regions with insertion errors; and use the corrupted I-frame to train the P-frame model.

[0011] In another example, a device includes: one or more memories configured to store media data; and one or more processors implemented in a circuit and coupled to the one or more memories, the one or more processors being configured to: apply a pre-trained P-frame model to the media data, the pre-trained P-frame model being trained using corrupted I-frames having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold, or b) one or more regions with insertion errors; and decode the media data based on the application of the pre-trained P-frame model to the media data.

[0012] Details of one or more examples are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the description, drawings, and claims. Attached Figure Description

[0013] Figure 1 This is a block diagram illustrating an example media encoding and decoding system that can perform the techniques of this disclosure.

[0014] Figure 2 This is a conceptual diagram illustrating the differences between conventional media decoding and neural-based media decoding.

[0015] Figure 3 This is a block diagram of an example neural image decoder that uses a variational autoencoder and a super-prior network for entropy decoding according to one or more aspects of this disclosure.

[0016] Figure 4 This is a conceptual diagram illustrating an example of adaptive reference selection.

[0017] Figure 5 This is a conceptual diagram illustrating an example causal inference structure for an uneven chessboard context module.

[0018] Figure 6 This is a conceptual diagram illustrating example channels of y, z^, and z^ after DCT transformation.

[0019] Figure 7AThis is a block diagram illustrating an example of a conditional P-frame model.

[0020] Figure 7B It is a block diagram of an example architecture for conditional analysis transformations and synthetic transformations based on one or more aspects of this disclosure.

[0021] Figure 8 This is a conceptual diagram illustrating an example of a three-frame verification sequence obtained during training.

[0022] Figure 9 This is a graphical representation of example peak signal-to-noise ratio (PSNR) and bits per pixel across 125 evaluation frames, with and without forced quality enhancement (FQE).

[0023] Figure 10 This is a graphical representation of example RGB rate distortion curves for the UVG and MCL-JCV datasets.

[0024] Figure 11 This is a graphical representation of the example RGB-PSNR difference between verification sequences with and without I-frames.

[0025] Figure 12 This is a graphical example illustrating kernel density estimation of channels across the Kodak dataset in RGB and YUV formats.

[0026] Figure 13 This is a graphical representation of an example RGB rate distortion curve for the Kodak dataset.

[0027] Figure 14 This is a block diagram illustrating an example system for training a P-frame model according to one or more aspects of this disclosure.

[0028] Figure 15 This is a flowchart illustrating example techniques for encoding media data according to one or more aspects of this disclosure.

[0029] Figure 16 This is a flowchart illustrating example techniques for decoding media data according to one or more aspects of this disclosure.

[0030] Figure 17 This is a flowchart illustrating an example technique for training a P-frame model according to one or more aspects of this disclosure.

[0031] Figure 18 This is a flowchart illustrating an example technique for decoding media data using a trained P-frame model, according to one or more aspects of this disclosure. Detailed Implementation

[0032] This disclosure describes techniques for encoding and decoding media data (e.g., images or videos, such as video data) using neural network-based media decoding techniques. Specifically, this disclosure describes techniques for utilizing prior variables when modeling output variables, for example, by applying a discrete cosine transform (DCT) to the prior variables. This disclosure also describes techniques for training neural media decoders (such as NVC) and using such trained neural media decoders to avoid or reduce the accumulation of temporal errors, for example, by using corrupted I-frames. A corrupted I-frame can be an I-frame with a peak signal-to-noise ratio (PSNR) that meets a threshold and / or an I-frame with one or more regions containing insertion errors. Such techniques, individually or together, can improve the quality of video decoding, such as PSNR, thereby providing a better viewing experience for viewers of the decoded video data.

[0033] Figure 1 This is a block diagram illustrating an example media encoding and decoding system 100 capable of performing the techniques of this disclosure. In the context of this disclosure, media may include any digital file to be compressed, including video data and / or images. The example techniques of this disclosure generally involve decoding (encoding and / or decoding) video data and / or image data. Although the techniques will be described with reference to media encoding and decoding... Figure 1 While this is just one example, the technology of this application is equally applicable to encoding and decoding any type of data file using neural-based compression techniques.

[0034] like Figure 1 As shown, in this example, system 100 includes a source device 102 that provides encoded media data to be decoded and displayed by a destination device 116. Specifically, source device 102 provides the media data to destination device 116 via computer-readable medium 110. Source device 102 and destination device 116 can include any of a wide variety of devices, including desktop computers, laptop computers, mobile devices, tablet computers, set-top boxes, mobile phones such as smartphones, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, broadcast receiver devices, etc. In some cases, source device 102 and destination device 116 may be configured for wireless communication and are therefore referred to as wireless communication devices.

[0035] exist Figure 1In the example, source device 102 includes a media source 104, memory 106, media encoder 200, and output interface 108. Destination device 116 includes an input interface 122, media decoder 300, memory 120, and display device 118. According to this disclosure, the media encoder 200 of source device 102 and the media decoder 300 of destination device 116 can be configured to apply techniques for media decoding in a neural-based media compression system. Thus, source device 102 represents an example of a media encoding device, while destination device 116 represents an example of a media decoding device. In other examples, the source device and destination device may include other components or arrangements. For example, source device 102 may receive media data from an external media source such as an external camera. Similarly, destination device 116 may interface with an external display device, rather than including an integrated display device.

[0036] like Figure 1 The system 100 shown is merely an example. Generally, any digital media encoding and / or decoding device can perform techniques for media decoding in a neural-based media compression system. Source device 102 and destination device 116 are merely examples of such decoding devices, where source device 102 generates decoded media data for transmission to destination device 116. This disclosure refers to a “decoding” device as a device that performs the decoding (e.g., encoding and / or decoding) of data. Thus, media encoder 200 and media decoder 300 represent examples of decoding devices, specifically, media encoder and media decoder, respectively. In some examples, media encoder 200 and media decoder 300 may be examples of NVCs. In some examples, source device 102 and destination device 116 may operate in a substantially symmetrical manner, such that each of source device 102 and destination device 116 includes both a media encoding component and a decoding component. Therefore, system 100 may support unidirectional or bidirectional media transmission between source device 102 and destination device 116, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0037] Generally, media source 104 represents the source of media data (i.e., unencoded raw media data) and provides a sequential series of pictures (also referred to as "frames") of the media data to media encoder 200, which encodes the data of the pictures. Media source 104 of source device 102 may include video capture devices, such as cameras, video archives containing previously captured raw video, and / or video feed interfaces for receiving video from video content providers. Alternatively, media source 104 may generate computer graphics-based data as source media, or a combination of live media, archived media, and computer-generated media. In each case, media encoder 200 encodes the captured, pre-captured, or computer-generated media data. Media encoder 200 may rearrange the pictures from the received order (sometimes referred to as "display order") to a decoding order for decoding. Media encoder 200 may generate a bitstream including the encoded media data. Then, the source device 102 can output the encoded media data to the computer-readable medium 110 via the output interface 108 for reception and / or retrieval by, for example, the input interface 122 of the destination device 116.

[0038] The memory 106 of source device 102 and the memory 120 of destination device 116 represent general-purpose memory. In some examples, memories 106 and 120 may store raw media data, such as raw media from media source 104 and decoded raw media data from media decoder 300. Additionally or alternatively, memories 106 and 120 may store software instructions executable by, for example, media encoder 200 and media decoder 300. Although memories 106 and 120 are shown separately from media encoder 200 and media decoder 300 in this example, it should be understood that media encoder 200 and media decoder 300 may also include internal memory for functionally similar or equivalent purposes. Furthermore, memories 106 and 120 may store encoded media data, such as output from media encoder 200 and input to media decoder 300. In some examples, portions of memories 106 and 120 may be allocated as one or more buffers, for example, to store decoded and / or encoded raw media data.

[0039] Computer-readable medium 110 may represent any type of medium or device capable of transmitting encoded media data from source device 102 to destination device 116. In one example, computer-readable medium 110 represents a communication medium enabling source device 102 to transmit encoded media data directly to destination device 116 in real time, for example, via a radio frequency network or a computer-based network. According to a communication standard such as a wireless communication protocol, output interface 108 may modulate the transmitted signal including the encoded media data, and input interface 122 may demodulate the received transmitted signal. The communication medium may include any wireless or wired communication medium, such as radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network such as a local area network, a wide area network, or a global network (such as the Internet). The communication medium may include a router, switch, base station, or any other equipment that may be useful for facilitating communication from source device 102 to destination device 116.

[0040] In some examples, source device 102 can output encoded data from output interface 108 to storage device 112. Similarly, destination device 116 can access encoded data from storage device 112 via input interface 122. Storage device 112 may include any of a variety of distributed or locally accessed data storage media, such as hard disk drives, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded media data.

[0041] In some examples, source device 102 may output encoded media data to file server 114 or another intermediate storage device that may store the encoded media data generated by source device 102. Destination device 116 may access the stored media data from file server 114 via streaming or downloading.

[0042] File server 114 can be any type of server device capable of storing encoded media data and sending that encoded media data to destination device 116. File server 114 can represent a web server (e.g., for a website), a server configured to provide file transfer protocol services (such as File Transfer Protocol (FTP) or FLUTE-based file delivery protocol), a Content Delivery Network (CDN) device, a Hypertext Transfer Protocol (HTTP) server, a Multimedia Broadcast Multicast Service (MBMS) or Enhanced MBMS (eMBMS) server, and / or a Network Attached Storage (NAS) device. File server 114 may additionally or alternatively implement one or more HTTP streaming protocols, such as HTTP-based Dynamic Adaptive Streaming (DASH), HTTP Live Streaming (HLS), Real-Time Streaming Protocol (RTSP), HTTP Dynamic Streaming, etc.

[0043] Destination device 116 can access encoded media data from file server 114 via any standard data connection, including an internet connection. This standard data connection may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem, etc.), or a combination of both suitable for accessing encoded media data stored on file server 114. Input interface 122 can be configured to operate according to any or more of the various protocols discussed above for retrieving or receiving media data from file server 114, or other such protocols for retrieving media data.

[0044] Output interface 108 and input interface 122 can represent a wireless transmitter / receiver, a modem, a wired networking component (e.g., an Ethernet card), a wireless communication component operating according to any of the various IEEE 802.11 standards, or other physical components. In examples where output interface 108 and input interface 122 include wireless components, output interface 108 and input interface 122 can be configured to transmit data (such as encoded media data) according to cellular communication standards (such as 4G, 4G-LTE (Long Term Evolution), Advanced LTE, 5G, etc.). In some examples where output interface 108 includes a wireless transmitter, output interface 108 and input interface 122 can be configured to comply with other wireless standards (such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee)). ™ ),Bluetooth ™The source device 102 and / or destination device 116 may include corresponding system-on-chip (SoC) devices. For example, source device 102 may include an SoC device for performing functions belonging to media encoder 200 and / or output interface 108, and destination device 116 may include an SoC device for performing functions belonging to media decoder 300 and / or input interface 122.

[0045] The technology disclosed herein can be applied to media decoding to support any multimedia application in a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, Internet streaming media transmission (such as HTTP-based Dynamic Adaptive Streaming (DASH)), digital media encoded onto data storage media, decoding of digital media stored on data storage media, or other applications.

[0046] The input interface 122 of the destination device 116 receives an encoded media bitstream from a computer-readable medium 110 (e.g., a communication medium, storage device 112, file server 114, etc.). The encoded media bitstream may include signaling information defined by the media encoder 200, which is also used by the media decoder 300. The display device 118 displays a decoded image of the decoded media data to the user. The display device 118 may represent any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0047] Despite Figure 1 Not shown, but in some examples, both the media encoder 200 and the media decoder 300 may be integrated with the audio encoder and / or audio decoder, and may include appropriate MUX-DEMUX units or other hardware and / or software to process multiplexed streams that include both audio and media in a common data stream.

[0048] Both the media encoder 200 and the media decoder 300 can be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the technology is partially implemented in software, the device may store instructions for the software in a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the technology of this disclosure. Each of the media encoder 200 and the media decoder 300 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined encoder / decoder (decoder) in the respective device. Devices including the media encoder 200 and / or the media decoder 300 may include integrated circuits, microprocessors, and / or wireless communication devices (such as cellular phones).

[0049] Neural network-based image and video (e.g., media) compression methods can compete with current standards and offer several additional advantages.

[0050] This disclosure describes techniques for improving decoding using neural media decoders (such as NVCs). Most NVCs utilize nonparametric methods for univariate density estimation, thereby amortizing learning across channels, rather than using over-prior variables. However, output variables can retain much information from over-prior variables. Therefore, it may be desirable to utilize over-prior variables when modeling output variables, such as by applying DCT to over-prior variables.

[0051] This disclosure also describes improved training techniques for NVC. Training NVC can lead to the accumulation of temporal errors over large test-time group of pictures (GOP) sizes. Therefore, some NVCs attempt to correct this accumulation by implementing a weighted objective, where the importance of distortion increases with depth in the training sequence. While this technique can delay degradation, it still struggles to maintain quality over longer sequences. Therefore, it may be desirable to train the NVC such that it can accumulate improved reconstructions across subsequent training frames. Techniques disclosed herein include using corrupted I-frames to train P-frame models, which can reduce or improve the NVC's accumulation of temporal errors.

[0052] Figure 2 This is a conceptual diagram illustrating an example of the differences between conventional media decoding and neural-based media decoding. Because conventional (e.g., manually designed) decoders and neural-based decoders use very different forms of compression, conventional decoders and neural-based decoders employ entropy decoding in significantly different ways. Figure 2The paper outlines the main differences, showing that even when the two methods use the same decoding method, such as arithmetic decoding, all other aspects differ. This implies that developing new techniques to optimize the performance of neural-based decoders is beneficial.

[0053] like Figure 2 As shown, the design phases for both conventional decoding and neural-based decoding technologies include a data organization process 201 and a statistical data modeling process 202. For conventional media decoding technologies such as H.265, the data organization process 201 includes defining decoding elements (e.g., syntax elements and decoding modes). For neural-based decoding, the data organization process 201 includes training and developing an image / video (e.g., media) autoencoder network. For the statistical data modeling process 202, conventional decoding technologies include context-defined decoding, such as context-adaptive binary arithmetic decoding used for entropy decoding. For neural-based decoding, the statistical data modeling process 202 includes developing and training a hyper-prior neural network.

[0054] Both conventional decoding and neural-based decoding techniques include an implementation phase that includes a probability determination process 204 and a bit-conversion process 206. Both the probability determination process 204 and the bit-conversion process 206 are based on the data to be decoded. For conventional decoding, the probability determination process 204 (e.g., a bit rate assignment process) includes adaptive context probability estimation. For neural-based decoding, the probability determination process 204 includes the determination of probability distribution parameters, as described in more detail below. Both conventional decoding and neural-based decoding techniques use an arithmetic decoder and / or an entropy decoder (e.g., an arithmetic or entropy encoder or decoder) to perform the bit-conversion process 206. The arithmetic decoder can be a Huffman decoder, a range decoder, or an asymmetric digital system (ANS) decoder.

[0055] The data stream management phase involves managing the compressed data 208. In some example conventional decoding systems, the compressed data is in a single bit stream. Some example neural-based decoding systems use multiple streams for concurrent decoding.

[0056] An important class of methods proposed for neural-based image and video compression employs a variational autoencoder architecture, where latent variables are quantized and entropy decoding is performed using a fixed learned distribution. In some examples, this architecture has been extended by adding a smaller neural network called a “super-prior,” which uses latent variables generated from the autoencoder to define the probability distribution to be used for entropy decoding of those autoencoder latent variables. This super-prior network also uses quantization and entropy decoding, but with predefined (e.g., learned) PDF parameters.

[0057] Figure 3 A is a block diagram of an example neural image decoder that uses a variational autoencoder and a super-prior network for entropy decoding according to one or more aspects of this disclosure. Figure 3 The process shown in A involves a neural network used for image analysis and synthesis that implements a variational autoencoder, and the entropy decoding of its variables is defined by a hyper-prior decoder network (used at both the transmitter and receiver). Since "encoder" and "decoder" (e.g., media encoder 200 and media decoder 300) have already been used to denote the autoencoder portion, in neural network literature, the systems conventionally referred to by those names are typically called "transmitter" and "receiver," respectively. The input image 400 and output image 428 may comprise pictures or frames of video data.

[0058] exist Figure 3 In A, media encoder 200 may include an image analysis neural network 402, a hyper-prior encoder neural network 404, a DCT process 405, a quantization process 406, an entropy encoding process 408, a quantization process 416, and an entropy encoding process 418. Media encoder 200 may also include an entropy decoding process 412, a hyper-prior decoder neural network 424, an inverse DCT (IDCT) process 423, an entropy decoding process 422, and an image synthesis neural network 426. Generally, these components or functions may resemble the reconstruction loop or decoding loop in a hybrid video decoder. Although not shown, media decoder 300 may include a structure similar to entropy decoding process 412, hyper-prior decoder neural network 424, IDCT process 423, entropy decoding process 422, and image synthesis neural network 426 to reconstruct the output image. For example, media encoder 200 may output bitstream 1 410 and bitstream 2 420. The media decoder 300 can obtain bitstream 1 410 and bitstream 2 420, and similar to the media encoder 200, generates an output image, such as output image 428, by using a structure similar to that of the entropy decoding process 412, the super-prior decoder neural network 424, the IDCT process 423, the entropy decoding process 422, and the image synthesis neural network 426.

[0059] Image analysis neural network 402 is a neural network configured to encode and / or compress an input image 400. The compressed data stream created by image analysis neural network 402 is then processed by a super-prior encoder neural network 404. The output of super-prior encoder neural network 404 (which may include super-prior variables) then undergoes a DCT process 405. DCT process 405 may include applying a DCT to the output of super-prior encoder neural network 404 to generate DCT coefficients for the super-prior variables. The output of DCT process 405 (e.g., DCT coefficients) is quantized by quantization process 406 and entropy decoded by entropy encoding process 408 to create bit stream 1 410. Super-prior encoder neural network 404 is part of a process configured to determine parameters (e.g., standard deviation, variance, mean, etc.) of the PDF of the data stream created by image analysis neural network 402. The parameters of the PDF can then be used to entropy encode the data stream from image analysis neural network 402.

[0060] The entropy encoding process 408 itself uses learned parameters 414 learned from multiple images to be encoded. Learned parameters 414 may include learned motion parameters. For a P-frame model, corrupted I-frames can be used to learn such learned parameters 414, as discussed further later in this disclosure. The entropy decoding process 412 uses the same PDF parameters to entropy decode bitstream 1 410. The entropy-decoded bitstream may correspond to quantized DCT coefficients. This entropy-decoded bitstream is input to the IDCT process 423, where an inverse DCT is applied to the entropy-decoded bitstream. The output of the IDCT process 423 is processed by a super-prior decoder neural network 424 to generate PDF parameters for the specific image being decoded (e.g., input image 400). The PDF parameters may include variance information associated with the super-prior, such as standard deviation, mean, etc.

[0061] Then, the PDF parameters generated by the super-prior decoder neural network 424 are used by the entropy encoding process 418 to encode the data stream quantized by the quantization process 416 generated by the image analysis neural network 402. The entropy encoding process 418 creates bit stream 2 420. Then, the entropy decoding process 422 entropy decodes bit stream 2 420 using the same PDF parameters from the super-prior decoder neural network 424. After entropy decoding, the image synthesis neural network 426 processes the decoded data to produce the output image 428.

[0062] By applying the DCT procedure 405 to the output (e.g., the hyperprior variable) of the hyperprior encoder neural network 404, the media encoder 200 can further compress the hyperprior information than existing NVCs. The media encoder 200 and the media decoder 300 can use the corresponding IDCT procedure 423 to generate a quantized version of the output of the hyperprior encoder neural network 404.

[0063] Image analysis neural network 402, super-prior encoder neural network 404, super-prior decoder neural network 424 and / or image synthesis neural network 426, or parts thereof, may be connected to... Figure 7A and Figure 7B The example architecture described herein is implemented in a similar manner.

[0064] In some examples, an NVC such as a media encoder 200 can encode media data (e.g., an input image 400). The media encoder 200 can determine a priori variables for the NVC (e.g., a media decoder 200) based on the media data. For example, the priori variables could be the output of a priori encoder neural network 404. The media encoder 200 can apply a DCT (e.g., DCT 405) to the priori variables to generate DCT coefficients. The media encoder 200 can encode the DCT coefficients, for example, by applying quantization 406 and entropy coding 408.

[0065] In some examples, an NVC (such as media decoder 300) can decode media data (e.g., encoded media data from bitstream 1 410). Media decoder 300 can determine the DCT coefficients of the super-prior variables of the NVC (e.g., media encoder 200). For example, media decoder 300 can apply entropy decoding 412 to bitstream 1 410 to determine the DCT coefficients. Media decoder 300 can apply inverse DCT 423 to the DCT coefficients to generate the super-prior variables. Media decoder 300 can decode media data based on the super-prior variables. For example, media decoder 300 can utilize a super-prior decoder neural network 424, entropy decoding 422, and an image synthesis neural network 426 (or a similar structure) to process the super-prior variables to generate an output image 428.

[0066] The media encoder 200 determines the prior variables of the NVC (e.g., media decoder 200) based on media data. For example, the prior variables could be the output of a prior encoder neural network 404. The media encoder 200 can apply a DCT (e.g., DCT 405) to the prior variables to generate DCT coefficients. The media encoder 200 can encode the DCT coefficients, for example, by applying quantization 406 and entropy coding 408.

[0067] Conventional video decoders based on predictive decoding frameworks, such as those described in the VVC and HEVC standards (Bross et al., 2021; Sullivan et al., 2012), have long been prevalent for lossy video compression. Developed by standards organizations over the past 30 years, these decoders offer improved capabilities for efficient data storage and transmission compared to previous decoders, and have potentially supported significant economic growth. However, despite the success of such conventional approaches (which may be limited by manually designed linear submodules), improvements have slowed, with diminishing returns on gains.

[0068] In pursuit of improved performance, interest has grown in the emerging field of neural video decoders (NVCs) (Lu et al., 2019; Agustsson et al., 2020; Hu et al., 2021; Li et al., 2021; Rippel et al., 2021; Li et al., 2022a; Mentzer et al., 2022). Media encoder 200 and media decoder 300 ( Figure 1 DVC can be an example. Using DVC, the technique allows for end-to-end training of decoders by replacing (or completely redesigning) manually designed submodules with differentiable neural operations. However, despite offering flexibility, NVC has so far struggled to produce competitive decoding results at real-world decoding speeds. Nevertheless, progress has been relatively rapid, with many open questions specific to these NVC approaches remaining unanswered.

[0069] This disclosure describes techniques related to the use of decoded information. Unlike the predictive decoding paradigm of conventional approaches, those working in the NVC field have not yet reached a consensus on a unified approach to using decoded information. Example methods can be broadly categorized into several frameworks that define the use of past information: predictive decoding (where the decoder decodes the residual difference between the motion-compensated frame and the target frame) (Lu et al., 2019; Agustsson et al., 2020; Hu et al., 2021); conditional decoding (where the decoder conditions the target frame with motion-compensated information) (Li et al., 2022a; 2021; Ladune et al., 2021; Ladune and Philippe, 2022); and techniques where the decoder does not use explicit motion compensation (such as 3D autoencoders) (Habibian et al., 2019; Pessoa et al., 2020).

[0070] This disclosure also describes techniques related to minimizing or reducing the accumulation of temporal errors. Unlike conventional methods, NVC utilization distortion optimization (RDO) (Sullivan and Wiegand, 1998) is not commonly used when performing tests. Conventional decoders allocate bits across groups of pictures (GOPs) via a search process to maintain frame quality and minimize bit rate. Instead of this allocation, most NVCs will experience quality degradation across larger GOP sizes, which the decoder cannot train on due to time and memory constraints. Although there are some specific solutions to this problem (Chan et al., 2021; Sheng et al., 2022), the standard approach still inserts high-bit still images (e.g., I-frames) at a higher frequency than the low-latency mode of conventional decoders. This can introduce significant disadvantages compared to conventional decoders. For example, Sheng et al. (2022) pointed out that in HEVC, moving the GOP from 12 (commonly used in NVC) to 32 increased the Bjontegaard-delta (BD) rate (Bjontegaard, 2001) by an average of 23.8%.

[0071] According to the techniques disclosed herein, the conditional decoding framework can operate in an LDP (Low-Latency P-Frame) configuration (e.g., using predictions from a single reference). This technique is supported by information theory results, namely that the (Shannon) entropy of the residual is greater than or equal to the entropy of the conditional entropy (Ladune et al., 2021).

[0072] ,

[0073] Where xt and These are the current frame and the motion-compensated frame, respectively. Unlike conventional decoding techniques, neural networks can freely explore complex nonlocal pixel dependencies and learn arbitrarily complex conditional relationships through nonlinear estimators. Therefore, NVC can use conditional decoding (Li et al., 2022a).

[0074] According to the technology disclosed herein, the media encoder 200 or media decoder 300 can apply conditional decoding paradigms to motion estimation. The media encoder 200 or media decoder 300 can also apply buffering ideas from conventional decoders to model long-range dependencies in motion estimation, thereby mitigating visual challenges such as occlusion.

[0075] According to the technology disclosed herein, the media encoder 200 or media decoder 300 can mitigate or circumvent the propagation of timing errors that are inherent to low-latency NVCs. For example, like conventional decoders, such a media encoder 200 or media decoder 300 can achieve better results (compared to other NVCs) across standard verification tasks in “infinite” intra-frame cycles (e.g., all P frames after an I-frame).

[0076] This disclosure discusses variations of probabilistic models for autoencoders that might otherwise be common in image and video decoders.

[0077] This disclosure discusses an analysis of common evaluation protocols discussed in the literature and provides a basis for encouraging future comparisons of weighted YUV-PSNR metrics with conventional methods.

[0078] According to the techniques disclosed herein, the resulting NVC decoder (e.g., media encoder 200 or media decoder 300) achieves improved results when compared with other NVC decoders. This NVC decoder can outperform the VVC test model (VTM) on YUV-PSNR across VTM Common Test Conditions (CTC) (Bossen et al., 2022).

[0079] Now let's discuss related work. Some examples in neural image compression have been based on mean-scale hyperprior (Ball'e et al., 2018). Broadly considered as variational autoencoders (VAEs), these methods most often assume a two-level hierarchical structure of latent variables, such that the rate-distortion objective is given by:

[0080]

[0081] in and These are the first-order and second-order latent variables quantified, respectively. It's about reconstruction. Here, and These are the so-called analyzer, superanalyzer, synthesizer, and supersynthesizer functions, respectively. Rate estimation. and distortion function The balance between them is achieved through Lagrange multipliers accomplish.

[0082] Extending this formula, methods have been developed that consider: improving rate estimation using autoregressive models (Minnen et al., 2018; He et al., 2021; 2022a; Minnen and Singh, 2020; Koyuncu et al., 2022; Lee et al., 2018), more complex transformations (Cheng et al., 2020; Zhu et al., 2021), and improved gradient estimators for non-differentiable quantization operators (Agustsson and Theis, 2020; Guo et al., 2021b). These advances have yielded significant improvements, with many techniques outperforming VVC on RGB-PSNR and MS-SSIM (Wang et al., 2003).

[0083] Following the success of image compression, many have applied these techniques to video. Most research here focuses on low-latency (LD) configurations: using temporally referenced previous frames to decode the current frame (e.g., a P-frame). In contrast to random access decoders, LD codecs can be codecs that decode video data in temporal frame order (F0, F1, F2, ...). It should be noted that there is a distinction between LDP (Low-Latency P-frames) and LDB (Low-Latency B-frames): for a given pixel in the current frame, B can use any number of previous frames; P can use only one previous frame. In an alternative to LD, the random access (RA) configuration is free to use both past and future frames for decoding.

[0084] The causal inference structure of RA is best suited for applications without latency constraints (e.g., streaming services), typically reducing the BD rate by about 30%. Although many decoders utilize frame interpolation and bidirectional motion estimation to extend NVC to RA (Pourreza and Cohen, 2021; Wu et al., 2018; Djelouah et al., 2019; Yang et al., 2020), these techniques are still less studied compared to LD NVC.

[0085] In optical flow estimation (LD), most work has adopted the predictive decoding approach of conventional methods (Rippel et al., 2021; Agustsson et al., 2020; Hu et al., 2021; 2020; Lin et al., 2020). In this case, compressed motion information can be used to derive motion-compensated frames. Methods for motion estimation include using Gaussian blur to enhance optical flow prediction (Agustsson et al., 2020), learning offset maps for deformable convolution (Hu et al., 2021), etc. Given a motion-compensated frame, decoders using these techniques will decode the sparse differences between the motion-compensated frame and the target frame. Recently, many proposals have extended the predictive decoding framework to conditional decoding (Li et al., 2022a; 2021; Ladune et al., 2021; Ladune and Philippe, 2022). Decoders vary widely in their use of motion; they can conditionally decode the target frame directly with information from the motion-compensated decoding, rather than decoding the residuals. In practice, this conditionalization manifests as cascading conditional information into both analytic and synthetic transformations. As discussed, this result can take advantage of many of the benefits of neural formulation.

[0086] Beyond inductive biases in explicit motion estimation, many researchers have considered 3D autoencoders (Habibian et al., 2019; Pessoa et al., 2020). As a natural extension of image compression, such techniques expand the input dimension to account for convolutions in both space and time. Recently, Mentzer et al. (2022) proposed an enhanced image decoder to include a transformer-based entropy model conditioned on previously decoded latent variables. Assuming independence between frame reconstructions, this approach avoids quality degradation while remaining competitive with predictive decoding techniques (e.g., computationally competitive).

[0087] Finally, neural representation techniques, which are not suitable for low-latency applications, have emerged as a potential alternative with relatively fast decoding speeds (Lee et al., 2022; Chen et al., 2021; Li et al., 2022b). Despite such relatively fast decoding speeds, the results are currently not competitive with "state-of-the-art" conventional decoders, and the cost of amortizing the transmission network weights across sequences is only effective when looking at the entire sequence, which is not guaranteed.

[0088] Figure 4 This is a conceptual diagram illustrating an example of adaptive reference selection. Figure 4 In the example, one or more hard masks are used to select regions from the previous reconstruction (see panels 450 and 460) to use as the basis for prediction, such as for predicting frame 470. The hard masks in panels 450 and 460 are represented as black areas that occlude portions of the panels. Such hard masks can be used to isolate hard samples representing specific types, such as faces.

[0089] The compression backbone for media encoder 200 and / or media decoder 300 is now described. According to the techniques of this disclosure, NVCs (such as media encoder 200 and / or media decoder 300) may use three autoencoder models: one for I-frames and two for P-frames. The P-frame model may include a (conditional) motion estimation module and a conditional decoder module. The compression backbone for each model may be based on a mean-scaled prior. However, since the performance of this approach may not be comparable to other conventional decoders, media encoder 200 and / or media decoder 300 may apply extensions to the model to achieve competitive performance.

[0090] Now, we describe contextual modeling. The constraint of full factorization beyond earlier entropy modeling remains important for producing competitive results. Unlike the earliest attempts (Minnen et al., 2018), such an extension should not significantly compromise time complexity. To avoid sacrificing time complexity, some have considered group-based autoregressive models, such as chessboard models and channel-by-channel models (He et al., 2021; Minnen and Singh, 2020). He et al. (2022a) proposed a non-uniform channel-by-channel grouping strategy.

[0091] According to the technology disclosed herein, the media encoder 200 or media decoder 300 may employ a non-uniform spatial subsampling chessboard. For example, the media encoder 200 and / or media decoder 300 may divide the latent variables into three groups: the first two groups each decode 25% of the data; and the last group decodes the remaining 50%. The media encoder 200 and / or media decoder 300 may construct the chessboard by stacking shifted copies of a 4×4 mask. In this way, the autoregression alternately accesses spatial locations between channels, thereby generating a complete spatial context at each autoregressive step. Figure 5 The diagram depicts this causal structure.

[0092] Figure 5 This is a conceptual diagram illustrating an example causal inference structure for an uneven chessboard context module. Figure 5 In the example, the causal inference structure proceeds from left to right. Figure 5 In this representation, three pixels are used. For example, the shaded pixels marked 500 represent the pixels to be inferred. The shaded pixels marked 502 represent the pixels to be decoded. Transparent pixels (such as those marked 504) represent unusable pixels.

[0093] Now we will discuss applying DCT to the prior. Although for Using powerful density estimation techniques is common, but beyond prior knowledge is crucial. This same flexibility is not typically offered. Following Ballé et al. (2018), most NVCs utilize their nonparametric methods for univariate density estimation. Because they “have no prior assumptions about the super-prior,” this approach attempts to amortize the effects across channels. This learning avoids the expression of spatial and / or frequency information.

[0094] Figure 6 is an example of y, and DCT transformation A conceptual diagram of an example channel. y 600 could be... Figure 3 Example of the output of image analysis neural network 402 (A). z in Figure 6Not shown, it can be a priori variable, for example, the output of the priori encoder neural network 404. If the priori variable z (e.g., the output of the priori encoder neural network 404) does not undergo the DCT process 405, but is directly fed into the quantization process 406 (all from... Figure 3 A), then 602 can be an example of the output of quantization process 406. After DCT transformation 604 can be Figure 3 An example of the output of DCT procedure 405 of A (e.g., the super-prior variable after DCT transformation).

[0095] right Figure 6 Visual observation shows 602 retains most of the structure of the input y 600. In this case, there is a strong basis for forming the prior hypothesis: The statistics of 602 are no different from those of natural images. To take advantage of this similarity, the media encoder 200 and / or media decoder 300 can use discrete cosine transform (DCT) to model the hyperprior. For example, the media encoder 200 and / or media decoder 300 can use DCT coefficients (e.g., DCT-transformed coefficients). 604) channel-by-channel Based on a block-based parametric hybrid model, the media encoder 200 or media decoder can perform quantization and entropy decoding. Compared to earlier methods, the media encoder 200 and / or media decoder 300 allow for the description of frequency information, thereby decoupling the latent representation. This technique improves performance with minimal inference complexity, as discussed further below.

[0096] In some examples, the media encoder 200 and media decoder 300 can be symmetric encoders and decoders, and can include multiple residual blocks, such as five or six residual blocks, omitting common ideas such as non-local attention blocks. The media encoder 200 and / or media decoder 300 can follow the approach of Li et al. (2022a) and utilize multi-granularity quantization. While their work focuses on the learned quantization chamber size in multi-rate models, their work improves rate-distortion performance in single-rate models. Therefore, in addition to predicting the mean and variance... and In addition, the media encoder 200 and / or media decoder 300 can predict This can be used to provide adaptive scaling of the content of variables before and after quantization. For example, the media encoder 200 and media decoder 300 can be built from a symmetric architecture of 5 residual blocks with 192 channels. For example, the supernetwork can be built from 2 residual blocks, both with 256 channels. A kernel of size 3 can be used anywhere, and upsampling can be accomplished using transposed convolutions.

[0097] For example, compared to the usual four downsampling operations, the first-level latent variable y can be downsampled a total of three times, for example, by quantization 416. This choice, combined with a content-adaptive quantization bin size, can present a good balance between utilizing spatial redundancy and preserving high-frequency information.

[0098] The application of this compressed backbone, as discussed above, to P-frame decoding is now discussed. As mentioned above, this disclosure discusses the use of information for decoding and efforts to limit quality degradation. P-frames of the media encoder 200 and / or media decoder 300 can be processed at a high level as follows. A pre-trained optical flow model (Ranjan and Black, 2017) can estimate the motion vectors between the current frame and the reference frame. Motion vectors can be used with a decoder buffer. Compressed and reconstructed The following section further discusses the extension of conditional decoding to motion vector compression and the use of decoder buffers. Media encoder 200 and / or media decoder 300 can distort reference features And this distorted reference feature is used as the conditional input to the conditional autoencoder. This returns the reconstructed features of the target frame, i.e. and .

[0099] Figure 7A This is a block diagram illustrating an example of a conditional P-frame model. Figure 7A In the example, the box labeled 700 represents the conditional encoder, the boxes labeled 702A and 702B represent the synthesis transform and the analysis transform, and the box labeled 704 represents the probability estimation and entropy decoding.

[0100] The conditional P-frame model 730 can be an example of some or all of the image analysis neural networks in the image analysis neural network 402, which may include a motion decoder 710 and a conditional decoder 720. The motion decoder 710 may include an autoencoder 730 and can decode motion associated with the input video sequence. For example, the motion decoder 710 may be based on motion signal m t and input video sequence x t Motion is encoded. The input video sequence x t May include Figure 3 The input image is 400. The autoencoder 740 can generate a quantized version of the input video sequence. .

[0101] Box 706 allows the Gubel-Softmax function to be applied to samples from the decoder buffer. For example, in low-latency decoding, the reference frame is selected from the decoder buffer of a previously decoded frame. The NVC is used for prediction. In low-latency P, typically only one reference block / pixel / frame can be selected for prediction. To allow the NVC to select one reference block from many reference blocks, the NVC generates a sampling probability for each pixel. For example, if the NVC is decoding a frame at time t=4, the reference buffer (e.g., the decoder buffer) has frames t=1, 2, 3 that have been decoded. The NVC can output the selection probability for each pixel in each of those potential reference frames from the reference buffer. For example, for any pixel in any color channel, P can be equal to [0.2, 0.3, 0.5], e.g., 50% probability that the NVC should include t=3, 30% probability that the NVC should include t=2, and so on. However, since the NVC is a low-latency NVC, the NVC can only select one of these three probabilities. Therefore, the NVC can convert the P vector into one-hot probabilities (e.g., one bit is high (1) and all other bits are low (0)). To achieve this, the NVC can apply the Gumbel-Softmax function. Therefore, the NVC determines the maximum value of the probability vector in a differentiable manner. In the example of P=[0.2, 0.3, 0.5], the output of box 706 (the application of the Gubel-Softmax function) will be gumbel_softmax(p)=[0, 0, 1]. In this case, the NVC can select a reference frame t=3 for a specific position in the final reference frame.

[0102] The same architecture can be used for conditional autoencoders of P-frame and I-frame models (e.g., media encoder 200 and media decoder 300). This technique is discussed further later in this disclosure. The conditional encoder (e.g., conditional decoder 720) may include small residual blocks and can produce a spatial pyramid structure of features for cascading to both analytical and synthetic transforms. Three features (or "context") can be extracted: Highest resolution features The output of the conditional synthesis transform can be combined with U-Net to produce the final output. In some examples, the autoencoder (such as motion decoder 710) according to the techniques of this disclosure may not include U-Net. Such a conditional architecture may include a simplification of Li et al. (2022a), who disclosed W-Net and allowed the downsampled motion vectors to be warped independently at each feature scale. In the method of Li et al. (2022a), a top-down “feature fusion” network is used to update the warped features, i.e., using... renew and use renew ,in It is a scale The distortion features at the location. In some examples, the media encoder 200 and / or media decoder 300 may not utilize or avoid this update of the distortion features using a top-down feature fusion network.

[0103] Figure 7B This is a block diagram of an example architecture for conditional analysis transformations and synthetic transformations according to one or more aspects of this disclosure. For example, conditional analysis transformation 750 (which may be...) Figure 7A The conditional analysis transform 702A example can obtain inputs, such as input video sequence x. t And generate an output (e.g., a latent variable), which can be fed into a hyperprior encoder neural network 404 ( Figure 3 The conditional analysis transform 750 can also send its output to an entropy decoder 756 for entropy decoding, in which case the entropy decoder 756 can be entropy-coded 418 (…). Figure 3 Example of a conditional encoder 752 (which can be...) Figure 7A The probability estimation and entropy decoding 704 (example) can obtain a conditional input (e.g., a motion vector) and output the conditionally decoded data to the entropy decoder 756. The conditional analysis transform 754 can obtain the output of the entropy decoder 756 and output it to, for example, a U-Net 758. The U-Net 758 can generate an output, such as a quantized version of the input video sequence. .

[0104] Now we will discuss conditional motion estimation. While temporal redundancy may exist across frames, there are many types of motion for which additional temporal redundancy exists across motion vectors. This observation has been successfully utilized in previous work—methods have solved the problem of predicting motion vectors before decoding sparse updates (Rippel et al., 2021; Pourreza et al., 2023). These techniques can enhance motion prediction without transmitting additional bits.

[0105] Building upon the theoretical superiority of conditional decoding, this idea can be applied to motion estimation. It can be noted that Ladune and Philippe (2022) apply a technique that conditions only the conditional frame, rather than the reference motion vector. Here, the media encoder 200 and / or media decoder 300 can use a previously transmitted reference frame to derive the motion vector as a conditional input for use by the motion autoencoder. That is, given a reference frame buffer... The media encoder 200 and / or media decoder 300 can use optical flow to re-estimate the vector. It can be noted that the reconstructed motion vectors Compared to those recalculated for conditionalization The differences between them are also noted. It can also be observed that using previously reconstructed data (e.g., the output of a conditional autoencoder) can make evaluation unstable for larger GOP sizes. Furthermore, since training on sequence lengths that periodically fill multi-frame buffers during normal operation may be impractical, the media encoder 200 and / or media decoder 300 can use an I-frame model to initialize the pseudo-buffer of decoded P-frames. This method allows the media encoder 200 and / or media decoder 300 to be trained on relatively short sequences.

[0106] Now we discuss adaptive reference selection. Conventional decoders select a reference block from the buffer of the decoded frame for motion prediction (e.g., Wien (2015)). Such encoders select an optimal block from the buffer—and the resulting motion prediction—that minimizes the error between the motion-compensated frame and the current frame. This technique allows the decoder to model long-range dependencies, thereby overcoming visual challenges such as occlusion.

[0107] Therefore, the NVC (e.g., media encoder 200 or media decoder 300) can use such a buffer to model long-range dependencies. This technique can be called Adaptive Reference Selection (ARS). That is, given a reference frame buffer and its associated motion vector, the media encoder 200 and / or media decoder 300 can return a per-pixel sampling probability tensor along with the reconstructed motion vector and a distorted skip mode. Using the Gumbel-Softmax operator, the media encoder 200 and / or media decoder 300 can use this probability tensor to sample from the reference buffer and then distort the result using the decoded motion vector. Finally, the media encoder 200 and / or media decoder 300 can use the skip mode as a low-cost (e.g., computationally inexpensive) mechanism to decode the identity transformation between the current frame and any reference frame in the buffer. For example, such motion can be defined as:

[0108]

[0109] in It is motion prediction. It is a reference frame sampled using the Gumbel-Softmax operator. It is a skip mode, and It is a bilinear twisting function.

[0110] Figure 8This is a conceptual diagram illustrating an example of a three-frame validation sequence obtained during training. It depicts the three-frame reconstruction of the training validation sequence. Given a relatively low-quality I-frame at t=0 (e.g., a corrupted I-frame), the video model can output progressively higher quality reconstructions of frames 800, 802, and 804. Illustrations 810, 812, and 814 are provided to show magnified versions including a foreground region containing a person. Available for further viewing. Figure 8 The results of ARS are seen in the image, which uses a buffer of two frames. For example, the media encoder 200 and / or the media decoder 300 may select only pixels exhibiting relatively large amounts of motion (e.g., foreground regions containing people) from the most recent reference frame.

[0111] Now we discuss minimizing the propagation of temporal errors. Practical constraints on time and computational resources may limit decoder training to ≤7 (seven) frames. Therefore, one of the fundamental problems specific to NVC is the accumulation of temporal errors over large test-time GOP sizes (which can include hundreds of frames) that are practically untrainable on NVC. The most common approach to addressing this problem is to use “cascaded” loss (Chan et al., 2021; Sheng et al., 2022). Utilizing cascaded loss, the training system can include a weighted objective where the importance of distortion increases with depth in the training sequence. This technique serves a dual purpose: penalizing quality degradation and preventing disproportionate evaluation of earlier frames (e.g., implicitly evaluating earlier frames across all future frames). For example, a model could use:

[0112]

[0113] Where R is the rate (e.g., bit rate), D is the distortion (such as mean square error, PSNR, etc.), and T is the time step. It is a constant that monotonically increases with the time step during the training of the GOP.

[0114] However, while the modifications achieved success in terms of latency degradation, fundamental problems remain: the model struggles to infer the manual design rules inherent in conventional decoders for maintaining quality across long sequences of frames or images. In fact, even the best approaches will still use I-frames every 32 frames, which is uncommon in conventional LD ​​methods. For example, inserting I-frames every 32 frames into the bitstream can significantly reduce decoding efficiency. I-frames tend to be much more bandwidth-intensive than P-frames because they can include all the information within a video data frame, rather than the incremental information between the current frame and another. In conventional decoders such as VVC decoders, a relatively large number of P-frames (e.g., 500-600 frames sequentially) can be transmitted without any intermediate I-frames when there are no scene changes. Therefore, NVC, which transmits I-frames every 32 frames during normal operation, is completely unable to compete with conventional codecs in terms of bandwidth efficiency.

[0115] Therefore, it may be desirable to provide an NVC that does not require relatively frequent transmission of I-frames to maintain quality. According to the techniques disclosed herein, the system can train a P-frame model such that the P-frame model learns to continuously improve image quality and simulate the removal of temporal errors over a larger GOP size. For example, simply penalizing temporal errors during training may be insufficient; training should enable the NVC to learn to self-correct those temporal errors that the NVC would otherwise tend to produce. For example, the media encoder 200 and / or media decoder 300 can be trained to self-correct temporal errors. Self-supervised techniques have been discussed in several documents (He et al., 2022b; Grill et al., 2020).

[0116] In some examples, model-agnostic training techniques (which may be referred to herein as Forced Quality Enhancement (FQE)) can be used to train the media encoder 200 and / or the media decoder 300. FQE can be operated by pre-training a P-frame model using an I-frame that has undergone some form of corruption. This corruption may be inherent in the generated I-frame, imposed after the I-frame is generated, or both. For example, a corrupted I-frame may be an I-frame generated with a PSNR that is lower (e.g., significantly lower) than the PSNR associated with the Lagrange multipliers used in the P-frame model, and / or an I-frame that may have undergone random masking to introduce corruption (or further corruption).

[0117] For example, corrupted I-frames can be used to train media encoder 200 and / or media decoder 300 to accumulate improved reconstructions across subsequent training frames, thereby reducing or minimizing the target subject's own error under the constraint of the target subject itself. While many techniques (including randomized masks) may be effective when subjecting I-frames to corruption, to optimally simulate NVC artifacts, frames configured to have a significantly lower PSNR than that implied by the Lagrange multipliers of the P-frame model (if the I-frame model will be included in end-to-end training) can be utilized. For example, if the P-frame model is configured to attempt to maintain a PSNR of 34 dB at 1000 kbps, an I-frame can be configured to have a PSNR of 32 dB at 500 kbps. As discussed above, Lagrange multipliers can be used to balance rate and distortion. For example, a corrupted I-frame can be configured to achieve 0.25 times the bit rate achieved when the I-frame model is included in end-to-end training. For example, corrupted I-frames (such as I-frames configured to have a PSNR below (or below or equal to) a threshold) can be used to train the P-frame model of the media encoder 200 and / or media decoder 300. This threshold can be based on a Lagrange multiplier associated with the P-frame model. In some examples, the threshold can be, for example, as per... Figure 8 The described 3-frame training sequence.

[0118] Furthermore, the system can train the media encoder 200 and / or the media decoder 300 to learn one of the manually designed biases of a conventional decoder: P-frames oscillate at the bit rate (Bossen et al., 2022). For example, the decoder can initiate a chain of high-bit-rate followed by low-bit-rate P-frame pairs.

[0119] Figure 9 This is a graphical representation of example peak signal-to-noise ratio (PSNR) and bits per pixel across 125 evaluation frames, with and without forced quality enhancement (FQE). Figure 9 The examples depict a comparison of applying FQE (900 and 910) versus not applying FQE (902 and 912) for a so-called infinite GOP spanning the verification sequence. As shown, using FQE yields reliably higher PSNR and oscillating bit-per-pixel speeds compared to not using FQE. Robustness to poor reference frames and missing data is discussed further later in this disclosure.

[0120] The techniques used to evaluate neural decoders are now discussed. In neural image and video compression, a typical evaluation compares rate-distortion performance with conventional decoders such as HEVC and VVC. In this typical evaluation, two commonly used distortion metrics exist: RGB-PSNR and MS-SSIM. The latter is used as a surrogate metric for optimizing decoders against perceptual loss (Zhang et al., 2018; Blau and Michaeli, 2018; Mentzer et al., 2020). Given the flexibility of NVC to optimize for any differentiable objective, this is a significant indication of its superior performance.

[0121] This disclosure focuses more on the former: the “classic” rate-distortion performance defined in RGB-PSNR. This is a metric in which many NVCs described in the paper outperform HEVC, and more recently, VVC (Li et al., 2022a). However, this metric can be disadvantageous for conventional baselines. Therefore, future evaluations for conventional decoders can be better accomplished based on weighted YUV-PSNR, an objective function that guides the development of video standards organizations.

[0122] Now we will discuss evaluating decoders in RGB. The conventional techniques used to evaluate decoders are not designed for RGB. Instead, these techniques are used for typically subsampled YUV... 420 Operations are performed in color spaces. (Subsampled color spaces improve upon complexity by leveraging human perception.) To evaluate the decoder in RGB, it is common practice to perform operations to YUV before evaluating RGB-PSNR. 444 The transformation is performed, the result is compressed, and then the inverse RGB transformation is executed.

[0123] The lack of chroma subsampling and transform operations makes precise comparisons difficult, but this may not be the primary issue. In fact, like MS-SSIM, conventional decoders are not optimized for RGB-PSNR. That is, decades of development have fixed the de facto YUV-PSNR loss, where the channel-wise weighting is approximately 6:1:1 to 8:1:1. In contrast, RGB-PSNR is approximately equivalent to YUV-PSNR 1:1:1 (Bossen et al., 2020). This observation can have significant implications for rate-distortion performance. Specifically, the Y channel has the highest entropy and is therefore likely the most difficult or costly to compress. Consequently, conventional decoders have long suffered from a trade-off of approximately 1:10 in BD rate between Y and UV (Bossen, 2019; Francois et al., 2022): to achieve a 1% BD rate in Y (relative to some reference), the decoder may sacrifice 10% in UV.

[0124] Table 1. BD rate in YUV on I-frame models trained for RGB-PSNR.

[0125]

[0126] To better understand the implications of this observation, two points can be considered. First, if NVC outperforms a conventional baseline in the RGB color space, will this result also be achieved in the YUV color space? Second, to what extent can RGB performance be transferred to YUV—e.g., is there a similar Y:UV BD rate tradeoff? For the first question, the publicly available code from Li et al. (2022a) can be evaluated, which may be the only NVC to report performance outperformance against VTM. To obtain YUV results, the decoder can be evaluated across the JVET CTC. Temporary RGB-to-YUV conversions and vice versa can be employed. For the second question, I-frame models (e.g., those of Media Encoder 200 and / or Media Decoder 300) can be retrained across three YUV-PSNR targets of 1:1:1, 4:1:1, and 8:1:1. The resulting models can be evaluated on the commonly used Kodak dataset (Franzen, 1999) using the RGB-trained model as a reference.

[0127] Table 2. Performance of Li et al. (2022a) relative to VTM-16.0 LDP

[0128]

[0129] Table 2 shows that Li et al. (2022a) outperformed UV in 1:1:1 (or RGB). Similarly, a Y:UV tradeoff of approximately 1:10 can be observed in Table 2. While the latter may not generalize to the results of Li et al. (2022a) or other NVCs, these results demonstrate that methods optimized for weighted YUV targets are most likely disadvantageous in RGB-PSNR. It is also noteworthy that this difference is broadly consistent with the results proposed by Pourreza et al. (2023), who observed weaker comparative performance when using similar distortion targets in YUV.

[0130] While perceptual quality may ultimately be what matters (6:1:1 PSNR-YUV itself is a perceptual choice), RGB comparisons achieve almost the same results as MS-SSIM: demonstrating the flexibility of training for any differentiable objective. NVC can achieve superior visual quality without a win in YUV, but using RGB-PSNR in a "classic" rate-distortion evaluation does not seem to be a fair comparison of decoders.

[0131] The test results are now discussed. The media encoder 200 and media decoder 300 disclosed herein were trained on the Vimeo-90k dataset (Xue et al., 2019). Training followed the multi-stage training scheme proposed by Sheng et al. (2022). For FQE, a frozen I-frame model was used, with the goal of achieving approximately 0.25 times the bit rate as if it were included in the training. For the final training step after FQE, the I-frame model was unfrozen, and the decoders (e.g., media encoder 200 and / or media decoder 300) were trained end-to-end. For evaluation purposes, only one I-frame was used, and all subsequent frames were P-frames.

[0132] Conventional decoders are omitted from the PSNR-RGB comparison; they may be overridden, and instead, JVET CTC is used in YUV. 420 Individual comparative tests were conducted. Many technologies have proposed dedicated networks to process YUV data. 420 Input (Egilmoz et al., 2021; Pourreza et al., 2023; Ho et al., 2022). However, special processing may not be beneficial. Surprisingly, employing the simplest technique of repeatedly upsampling the UVs before applying the RGB transformation seems advantageous. This transformation normalizes the marginal distribution across channels, thus making network training easier (which will be discussed later in this paper). The YUV model was trained using a 6:1:1 YUV-PSNR distortion target.

[0133] Table 3. Ablation results from different model proposals. BD rate was calculated using UVG.

[0134]

[0135] Table 3 above shows the ablation results. FQW presents a good indication of the single largest improvement, and FQW is widely applicable to any LD NVC.

[0136] To compare the media encoder 200 and / or media decoder 300 with other NVCs, video compression was evaluated in RGB against common benchmarks MCL-JCV and UVG (Mercat et al., 2020; Wang et al., 2016). For this evaluation, the media encoder 200 and / or media decoder 300 were compared with (Hu et al., 2021), ELF-VC (Rippel et al., 2021), DCVC (Li et al., 2021), C2F (Hu et al., 2022), VCT (Mentzer et al., 2022), SSF (Agustsson et al., 2020), and Li et al. (Li et al., 2022a). It can be noted that Li et al. only evaluated the first 96 frames of each sequence, which may make the comparison slightly unreliable.

[0137] Figure 10 This is a graphical representation of example RGB rate distortion curves for the UVG and MCL-JCV datasets. Figure 10 The test results are depicted, and performance superiority over each prior method is demonstrated. Curves labeled 1000 and 1002 represent test results for the media encoder 200 and / or media decoder 300 employing the techniques of this disclosure.

[0138] As discussed above, the media encoder 200 and / or media decoder 300 are coupled with YUV... 420 The media encoder 200 and / or media decoder 300 were compared with conventional standards in the RDO. For this comparison, HEVC and VVC test models (HM, VTM) were used, and the evaluation was performed using JVET CTC. The test results are available in Tables 5 and 6. Note the challenges posed by Class E, which consists primarily of static content. It is clear that the media encoder 200 and / or media decoder 300 struggle to compete with the complex skip patterns of conventional methods chosen during RDO (e.g., Wien (2015)). Note that Li et al. (2022a) also faced this problem when evaluating using the public codebase of Li et al. (2022a) (see Table 2). Nevertheless, according to the techniques of this disclosure, the media encoder 200 and / or media decoder 300 are the first reported decoders to outperform VTM LDP on most test sequences—even outperforming LDP on Class C and Class D.

[0139] Table 4. Performance relative to HM-16.24 LDP

[0140]

[0141] Table 5. Performance relative to VTM-16.0 LDP

[0142]

[0143] Decoding speed was also evaluated. A 1080p UVG sequence was used. Media encoder 200 and / or media decoder 300 were run on an NVIDIA RTX 5000 and compared with results obtained from the paper (if available). Achieving competitive BD rate and decoding speed performance clearly remains a challenge for NVC. For example, while ELF-VC is an outstanding approach in terms of FPS, it suffers from significant BD drawbacks.

[0144] Table 6. Estimated FPS and BD rate differences across the UVG dataset. The BD rate is calculated relative to the decoder of this disclosure.

[0145]

[0146] As described in this article, NVCs (such as the Media Encoder 200 and Media Decoder 300) can be used in YUV... 420 It competes with the most advanced conventional methods.

[0147] The media encoder 200 and media decoder 300 can be constructed from a symmetric architecture of 5 residual blocks with 192 channels. For example, the supernetwork can be constructed from 2 residual blocks, both with 256 channels. A kernel of size 3 can be used anywhere, and upsampling can be achieved using transposed convolutions.

[0148] Compared to the usual four downsampling iterations, the first-order latent variable... It can be downsampled a total of 3 times. For example, this choice, combined with content-adaptive quantization bin size, can present a good balance between utilizing spatial redundancy and preserving high-frequency information.

[0149] To demonstrate the robustness learned from FEQ, the impact of poor reference frames can be evaluated. In extreme cases, this may include the impact of missing reference frames. For example, media encoder 200 and / or media decoder 300 can be compared with Li et al. (2022a). For each decoder, the first 32 frames of the UVG sequence HoneyBee can be evaluated at comparable bit rates in both cases with and without missing I-frames (e.g., assuming it is a zero tensor). Such an evaluation can produce results in the form of PSNR difference between decoded sequences with and without I-frames.

[0150] Figure 11This is a graphical representation of an example RGB-PSNR difference between verification sequences with and without I-frames. (Example...) Figure 11 As depicted, the media encoder 200 and / or media decoder 300 (represented by line 1100) recover to within approximately 0.1 dB relatively quickly without inserting new I-frames. In contrast, the verification sequence of Li et al. (2022a) (represented by line 1102) is more adversely affected.

[0151] Figure 12 This is a graphical example illustrating kernel density estimation of channels across the Kodak dataset in RGB and YUV formats. Figure 12 In the example, line 1200 represents B, line 1202 represents G, and line 1204 represents R. Additionally, line 1206 represents U, line 1208 represents V, and line 1210 represents Y. It should be noted that RGB contains more redundancy in terms of statistical differences between channels in YUV compared to RGB. Given that the differences manifest as affine transformations, it is surprising that the RGB-transformed YUV input is preferred in the tests. Most strangely, directly operating on the YUV input (using unlearned upsampling for UV in 420) is one of the worst performing options. It should be noted that previous work in the YUV space used separate network “headers”: one for Y and one for UV (Pourreza et al., 2023; Egilmoz et al., 2021). However, more broadly, YUV… 420 It is designed to save computation in conventional decoders. Using a separate head might achieve the opposite effect. It should be further noted that the RGB conversion allows the use of pre-trained optical flow models, which simplifies training.

[0152] Figure 13 This is a graphical representation of an example RGB rate distortion curve from the Kodak dataset. Figure 13 In the text, dots such as dot 1300 represent data for media encoder 200 and / or media decoder 300 used to implement one or more technologies of this disclosure.

[0153] Figure 14 This is a block diagram illustrating an example system for training a P-frame model according to one or more aspects of this disclosure. Training system 1400 may include one or more processors 1412 coupled to one or more memories 1410. The one or more memories 1410 may store training input 1402, threshold 1440, media encoder 200, and / or media decoder 300. In some examples, training system 1400 may be... Figure 1The training system 1400 is part of the source device 102 and / or destination device 116. In some examples, the training system 1400 may be a separate system from the source device 102 and / or destination device 116. For example, the training system 1400 may be used to train the media encoder 200 and / or media decoder 300 before the media encoder 200 and / or media decoder are deployed to the source device 102 and / or destination device 116, respectively.

[0154] Media encoder 200 may include P-frame model 1420, and media decoder 300 may include P-frame model 1530. In some examples, P-frame model 1420 and P-frame model 1430 include the same P-frame model.

[0155] Training input 1402 may include a corrupted I-frame 1404. The corrupted I-frame 1404 may include an I-frame having a PSNR that satisfies a threshold 1440 (e.g., below or equal to the threshold 1440) and / or includes one or more regions with insertion errors. In some examples, the threshold 1440 may be based on Lagrange multipliers associated with P-frame models 1420 and / or 1430. In some examples, the corrupted I-frame 1404 may include intentionally inserted errors, such as one or more regions with occluded data. In other examples, the corrupted I-frame 1404 may include naturally occurring insertion errors.

[0156] In some examples, one or more processors 1412 may be configured to generate corrupted I-frames 1404. For example, one or more processors 1412 may be configured to generate I-frames that meet threshold 1440 and / or insert errors into one or more regions of the I-frame.

[0157] Training input 1402 may also include P-frame 1406. For example, P-frame 1406 may include multiple P-frames (e.g., up to 7 P-frames) that can be used to train P-frame models 1420 and / or 1430. P-frame 1406 is not necessarily a corrupted P-frame.

[0158] One or more processors 1412 may be configured to train P-frame models 1420 and / or 1430 using corrupted I-frames 1404 and P-frames 1406.

[0159] Figure 15 This is a flowchart illustrating an example technique for encoding media data according to one or more aspects of this disclosure. Media encoder 200 may determine the super-prior variables (1500) of the NVC based on the media data. For example, media encoder 200 may execute a super-prior encoder neural network 404 for the NVC (…). Figure 3A) Determine the prior variables based on the input image 400. The media encoder 200 may apply DCT to the prior variables to generate DCT coefficients (1502). For example, the media encoder 200 may apply DCT procedure 405 to the prior variables to generate DCT coefficients. The media encoder 200 may encode the DCT coefficients (1504). For example, as part of encoding the media data, the media encoder 200 may quantize the DCT coefficients via quantization procedure 406 and entropy encode the DCT coefficients via entropy encoding 408.

[0160] In some examples, as part of encoding the DCT coefficients, the media encoder 200 may quantize the DCT coefficients through channel-wise parametric mixing to generate quantized DCT coefficients. In some examples, as part of encoding the DCT coefficients, the media encoder 200 may entropy encode the quantized DCT coefficients. In some examples, channel-wise parametric mixing includes channel-wise 4×4 parametric mixing.

[0161] In some examples, the prior variables include those of the NVC I-frame model. In some examples, the prior variables include those of the NVC P-frame model. In some examples, the P-frame model includes at least one of a motion estimation model or a conditional decoder model. In some examples, the P-frame model is trained using a corrupted I-frame having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold, or b) one or more regions with insertion errors.

[0162] In some examples, the prior variables are based on latent variables. In some examples, the media encoder 200 may determine the latent variables, and as part of determining the latent variables, the media encoder 200 may apply context modeling to the media data using a subsampling board with non-uniform intervals. In some examples, the media encoder 200 may stack shifted copies of an NxN mask as part of using a subsampling board with non-uniform intervals. In some examples, the media data includes video data.

[0163] Figure 16 This is a flowchart illustrating an example technique for decoding media data according to one or more aspects of this disclosure. Media decoder 300 can determine the DCT coefficients (1600) of the prior variables of the NVC. For example, media decoder 300 can obtain the coefficients from a bitstream (such as bitstream 1410 ( Figure 3 A) Obtain the entropy-encoded DCT coefficients and perform entropy decoding on the entropy-encoded DCT coefficients (similar to...). Figure 3The entropy decoding process 412 of A is used to determine the DCT coefficients. The media decoder 300 may apply an inverse DCT to the DCT coefficients to generate a hyperprior variable (1602). For example, the media decoder 300 may apply a process similar to IDCT 423 to the entropy-decoded DCT coefficients. Figure 3 The inverse DCT of A) is used to determine the hyperprior variables. The media decoder 300 can decode the media data based on the hyperprior variables (1604). For example, the media decoder 300 can apply a hyperprior decoder neural network, such as a hyperprior decoder neural network 424 ( Figure 3 A), as part of decoding media data based on prior variables.

[0164] In some examples, the DCT coefficients include a channel-wise parametric blending of the quantized DCT coefficients generated by the neural media encoder. In some examples, the channel-wise parametric blending includes a channel-wise 4×4 parametric blending. In some examples, the media decoder 300 may perform entropy decoding on the DCT coefficients as part of determining the DCT coefficients.

[0165] In some examples, the prior variables include those of the I-frame model of the neural video decoder. In some examples, the prior variables include those of the P-frame model of the neural video decoder. In some examples, the P-frame model includes at least one of a motion estimation model or a conditional decoder model. In some examples, the P-frame model is trained using a corrupted I-frame having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold, or b) one or more regions with insertion errors.

[0166] In some examples, the super-prior variables are based on latent variables. In some examples, as part of decoding the media data, the media decoder 300 may determine the latent variables and apply context modeling to the latent variables using a subsampling board with uneven spacing. In some examples, as part of using a subsampling board with uneven spacing, the media decoder 300 may stack shifted copies of an NxN mask.

[0167] In some examples, the media decoder 300 may divide latent variable data into multiple groups. The latent variable data may include latent variables. In such examples, the media decoder 300 may use a first group of the multiple groups to decode a first percentage of the latent variable data. The media decoder 300 may use a second group of the multiple groups to decode a second percentage of the latent variable data. The media decoder 300 may use a third group of the multiple groups to decode a third percentage of the latent variable data. In some examples, the first and second percentages are lower than the third percentage. In some examples, the first percentage is 25%, the second percentage is 25%, and the third percentage is 50%.

[0168] In some examples, media data includes video data.

[0169] Figure 17 This is a flowchart illustrating an example technique for training a P-frame model according to one or more aspects of this disclosure. One or more processors 1412 may acquire a corrupted I-frame having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold, or b) one or more regions (1700) with insertion errors. For example, one or more processors 1412 may retrieve a corrupted I-frame 1404 from one or more memories 1410. The corrupted I-frame 1404 may have a PSNR that meets a threshold 1440 and / or one or more regions with insertion errors.

[0170] One or more processors 1412 may use a corrupted I-frame to train a P-frame model (1702). For example, one or more processors 1412 may use a corrupted I-frame 1404 as input to train a P-frame model 1420 and / or a P-frame model 1430.

[0171] In some examples, one or more processors 1412 may generate a corrupted I-frame 1404. In some examples, the corrupted I-frame 1404 has a PSNR that meets a threshold, wherein, in order to meet the threshold, the PSNR is below the threshold or below or equal to the threshold. In some examples, the threshold is based on a Lagrange multiplier associated with a P-frame model. In some examples, the corrupted I-frame is pre-configured to meet the threshold. In some examples, training the P-frame model includes training the P-frame model for seven frames or less. In some examples, training the P-frame model includes training the P-frame model for three frames.

[0172] Figure 18 This is a flowchart illustrating an example technique for decoding media data using a trained P-frame model, according to one or more aspects of this disclosure. A media encoder 200 or a media decoder 300 may apply a pre-trained P-frame model to the media data, the pre-trained P-frame model being trained using corrupted I-frames having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold, or b) one or more regions (1800) with insertion errors. For example, the media encoder 200 may include a P-frame model 1420, which may be pre-trained, for example, by one or more processors 1412 using a corrupted I-frame 1404. The corrupted I-frame 1404 may have a PSNR that meets a threshold and / or one or more regions with insertion errors.

[0173] Media encoder 200 or media decoder 300 may decode media data based on a pre-trained P-frame model applied to the media data (1802). For example, media encoder 200 may use a pre-trained P-frame model to encode media data, and media decoder 300 may use a pre-trained P-frame model to decode media data.

[0174] In some examples, the corrupted I-frame has a PSNR that meets a threshold, and in order to meet the threshold, the PSNR is below the threshold or below or equal to the threshold. In some examples, the media data includes video data. In some examples, decoding includes encoding. In some examples, decoding includes decoding.

[0175] The following is a list of references that may have been mentioned above.

[0176] Agustsson, E. and Theis, L. Universally quantized neural compression. Advances in neural information processing systems, 33:12367-12376, 2020.

[0177] Agustsson, E., Minnen, D., Johnston, N., Balle, J., Hwang, SJ, andToderici, G. Scale-space flow for end-to-end optimized video compression. InProceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition, pp. 8503-8512, 2020.

[0178] Ballé, J., Laparra, V., and Simoncelli, EP Density modeling ofimages using a generalized normalization transformation. arXiv preprintarXiv:1511.06281, 2015.

[0179] Ballé, J., Minnen, D., Singh, S., Hwang, SJ, and Johnston, .Variational image compression with a scale hyperprior. arXiv preprint arXiv:1802.01436, 2018.

[0180] Bjontegaard, G. Calculation of average PSNR differences between RD-curves. VCEG-M33, 2001.

[0181] Blau, Y. and Michaeli, T. The perception-distortion tradeoff. InProceedings of the IEEE conference on computer vision and patternrecognition, pp. 6228-6237, 2018.

[0182] Bossen, F. On reporting combined yuv bd rates. Joint Video ExpertsTeam (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC29, JVET-N0341, 2019.

[0183] Bossen, F., Li, X., Sharman, K., Seregin, V., and Sühring, K. VTM andHM common test conditions and software reference configurations for SDR 4:2:010-bit video. Joint Video Experts Team (JVET) of ITU-T SG and ISO / IECJTC 1 / SC29, JVET-Y2010, 2022.

[0184] Bross, B., Wang, Y.-K., Ye, Y., Liu, S., Chen, J., Sullivan, G. J.,and Ohm, J. R. Overview of the versatile video coding (VVC) standard and itsapplications. IEEE Transactions on Circuits and Systems for Video Technology,31(10):3736-3764, 2021. doi: 10.1109 / TCSVT.2021.3101953.

[0185] Chan, K. C., Wang, X., Yu, K., Dong, C., and Loy, C. C. Basicvsr: Thesearch for essential components in video super-resolution and beyond. InProceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition, pp. 4947-4956, 2021.

[0186] Chen, H., He, B., Wang, H., Ren, Y., Lim, S. N., and Shrivastava, A.NERV: Neural representations for videos. Advances in Neural InformationProcessing Systems, 34: 21557-21568, 2021.

[0187] Cheng, Z., Sun, H., Takeuchi, M., and Katto, J. Learned imagecompression with discretized gaussian mixture likelihoods and attentionmodules. In Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition, pp. 7939-7948, 2020. Djelouah, A., Campos, J., Schaub-Meyer, S., and Schroers, C. Neural inter-frame compression for video coding.In Proceedings of the IEEE / CVF International Conference on Computer Vision,pp. 6421-6429, 2019.

[0188] Duan, Q., Li, X., Yin, Q., Feng, L., Zhao, J., Teng, Y., Duan, X.,Zhao, Y., Gao, M., Wang, J., et al. A study on the generalized normalizationtransformation activation function in deep learning based image compression.In Proceedings of Sixth International Congress on Information andCommunication Technology, pp. 351-359. Springer, 2022.

[0189] Egilmez, H. E., Singh, A. K., Coban, M., Karczewicz, M., Zhu, Y.,Yang, Y., Said, A., and Cohen, T. S. Transform network architectures for deeplearning based end-to-end image / video coding in subsampled color spaces. IEEEOpen Journal of Signal Processing, 2:441-452, 2021. doi: 10.1109 / OJSP.2021.3092257.

[0190] Francois, E., Chen, Y., and Salmon-Legagneur, C. Adjusting luman / chroma BD-rate balance in ECM. Joint Video Experts Team (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC29, JVET-AC0138, 2022.

[0191] Franzen, R. Kodak lossless true color image suite. source: http: / / rOk.us / graphics / kodak, 4(2), 1999.

[0192] Granziol, D., Wan, X., Albanie, S., and Roberts, S. Iterativeaveraging in the quest for best test error. arXiv preprint arXiv:2003.01247,2020.

[0193] Granziol, D., Zohren, S., and Roberts, S. Learning rates as afunction of batch size: A random matrix theory approach to neural networktraining. J. Mach. Learn. Res, 23:1-65, 2022.

[0194] Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P.,Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M.,et al. Bootstrap your own latent-a new approach to self-supervised learning.Advances in neural information processing systems, 33:21271-21284, 2020.

[0195] Guo, Z., Zhang, Z., Feng, R., and Chen, Z. Causal contextualprediction for learned image compression. IEEE Transactions on Circuits andSystems for Video Technology, 32(4):2329-2341, 2021a.

[0196] Guo, Z., Zhang, Z., Feng, R., and Chen, Z. Soft then hard: Rethinkingthe quantization in neural image compression. In International Conference onMachine Learning, pp. 3920-3929. PMLR, 2021b.

[0197] Habibian, A., Rozendaal, T. v., Tomczak, J. M., and Cohen, T. S.Video compression with rate-distortion autoencoders. In Proceedings of theIEEE / CVF International Conference on Computer Vision, pp. 7033-7042, 2019.He, D., Zheng, Y., Sun, B., Wang, Y., and Qin, H. Checkerboard context modelfor efficient learned image compression. In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, pp. .

[0198] He, D., Yang, Z., Peng, W., Ma, R., Qin, H., and Wang, Y. Elic:Efficient learned image compression with unevenly grouped space-channelcontextual adaptive coding. In Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition, pp. 5718-5727, 2022a.

[0199] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R.Masked autoencoders are scalable vision learners. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009,2022b.

[0200] Ho, Y.-H., Lin, C.-H., Chen, P.-Y., Chen, M.-J., Chang, C.-P., Peng,W.-H., and Hang, H.-M. Learned video compression for YUV 4:2:0 content usingflow-based conditional interframe coding. In 2022 IEEE InternationalSymposium on Circuits and Systems (ISCAS), pp. 829-833. IEEE, 2022.

[0201] Hu, Z., Chen, Z., Xu, D., Lu, G., Ouyang, W., and Gu, S. Improvingdeep video compression by resolution-adaptive flow coding. In EuropeanConference on Computer Vision, pp. 193-209. Springer, 2020.

[0202] Hu, Z., Lu, G., and Xu, D. Fvc: A new framework towards deep videocompression in feature space. In Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition, pp. 1502-1511, 2021.

[0203] Hu, Z., Lu, G., Guo, J., Liu, S., Jiang, W., and Xu, D. Coarse-to-fine deep video coding with hyperprior-guided mode prediction. In Proceedingsof the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp.5921-5930, 2022.

[0204] Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deepnetwork training by reducing internal covariate shift. In Internationalconference on machine learning, pp. 448-456. PMLR, 2015.

[0205] Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson,A. G. Averaging weights leads to wider optima and better generalization.arXiv preprint arXiv:1803.05407, 2018.

[0206] Koyuncu, A. B., Gao, H., Boev, A., Gaikov, G., Alshina, E., andSteinbach, E. Contextformer: A transformer with spatio-channel attention forcontext modeling in learned image compression. In European Conference onComputer Vision, pp. 447-463. Springer, 2022.

[0207] Ladune, T. and Philippe, P. Aivc: Artificial intelligence based videocodec. arXiv preprint arXiv:2202.04365, 2022. Ladune, T., Philippe, P.,Hamidouche, W., Zhang, L., and Déforges, O. Conditional coding for flexiblelearned video compression. arXiv preprint arXiv:2104.07930, 2021.

[0208] Lee, J., Cho, S., and Beack, S.-K. Context-adaptive entropy model forend-to-end optimized image compression. arXiv preprint arXiv:1809.10452,2018.

[0209] Lee, J. C., Rho, D., Ko, J. H., and Park, E. FFNeRV: Flow-guidedframe-wise neural representations for videos. arXiv preprint arXiv:2212.12294, 2022.

[0210] Li, J., Li, B., and Lu, Y. Deep contextual video compression.Advances in Neural Information Processing Systems, 34: 18114-18125, 2021.

[0211] Li, J., Li, B., and Lu, Y. Hybrid spatial-temporal entropy modellingfor neural video compression. In Proceedings of the 30th ACM InternationalConference on Multimedia, pp. 1503-1511, 2022a.

[0212] Li, Y., Wei, C., and Ma, T. Towards explaining the regularizationeffect of initial large learning rate in training neural networks. Advancesin Neural Information Processing Systems, 32, 2019.

[0213] Li, Z., Wang, M., Pi, H., Xu, K., Mei, J., and Liu, Y. E-NeRV:Expedite neural video representation with disentangled spatial-temporalcontext. In European Conference on Computer Vision, pp. 267-284. Springer,2022b.

[0214] Lin, J., Liu, D., Li, H., and Wu, F. M-LVC: Multiple framesprediction for learned video compression. In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, pp. 3546-3554, 2020.

[0215] Lu, G., Ouyang, W., Xu, D., Zhang, X., Cai, C., and Gao, Z. Dvc: Anend-to-end deep video compression framework. In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, pp. 11006-11015, 2019.

[0216] Mentzer, F., Toderici, G. D., Tschannen, M., and Agustsson, E. High-fidelity generative image compression. Advances in Neural InformationProcessing Systems, 33: 11913-11924, 2020.

[0217] Mentzer, F., Toderici, G., Minnen, D., Hwang, S.-J., Caelles, S.,Lucic, M., and Agustsson, E. Vct: A video compression transformer. arXivpreprint arXiv:2206.07307, 2022.

[0218] Mercat, A., Viitanen, M., and Vanne, J. UVG dataset: fps sequences for video codec analysis and development. In Proceedings of the11th ACM Multimedia Systems Conference, pp. 297-302, 2020.

[0219] Minnen, D. and Singh, S. Channel-wise autoregressive entropy modelsfor learned image compression. In 2020 IEEE International Conference on ImageProcessing (ICIP), pp. 3339-3343. IEEE, 2020. Minnen, D., Ballé, J., andToderici, G. D. Joint autoregressive and hierarchical priors for learnedimage compression. vances in neural information processing systems,31, 2018.

[0220] Nesterov, Y. Introductory lectures on convex optimization: A basiccourse, volume 87. Springer Science & Business Media, 2003.

[0221] Papyan, V. The full spectrum of deepnet hessians at scale: Dynamicswith SGD training and sample size. arXiv preprint arXiv:1811.07062, 2018.

[0222] Pessoa, J., Aidos, H., Tomás, P., and Figueiredo, M. A. End-to-endlearning of video compression using spatiotemporal autoencoders. In 2020 IEEEWorkshop on Signal Processing Systems (SiPS), pp. 1-6. IEEE, 2020.

[0223] Pourreza, R. and Cohen, T. Extending neural P-frame codecs for B-frame coding. In Proceedings of the IEEE / CVF International Conference onComputer Vision, pp. 6680-6689, 2021.

[0224] Pourreza, R., Le, H., Said, A., Sautiere, G., and Wiggers, A.Boosting neural video codecs by exploiting hierarchical redundancy. InProceedings of the IEEE / CVF Winter Conference on Applications of ComputerVision, pp. 5355-5364, 2023.

[0225] Ranjan, A. and Black, M. J. Optical flow estimation using a spatialpyramid network. In Proceedings of the IEEE conference on computer vision andpattern recognition, pp. 4161-4170, 2017.

[0226] Rippel, O., Anderson, A. G., Tatwawadi, K., Nair, S., Lytle, C., andBourdev, L. Elf-vc: Efficient learned flexible-rate video coding. InProceedings of the IEEE / CVF International Conference on Computer Vision, pp.14479-14488, 2021.

[0227] Santurkar, S., Tsipras, D., Ilyas, A., and Madry, A. How does batchnormalization help optimization? Advances in neural information processingsystems, 31, 2018.

[0228] Sheng, X., Li, J., Li, B., Li, L., Liu, D., and Lu, Y. Temporalcontext mining for learned video compression. IEEE Transactions onMultimedia, 2022.

[0229] Ström, J., Andersson, K., Sjöberg, R., Segall, A., Bossen, F.,Sullivan, G., Ohm, J.-R., and Tourapis, A. Working practices using objectivemetrics for evaluation of video coding efficiency experiments (draft 4).Joint Video Experts Team (JVET) of ITU-TSG 16 WP 3 and ISO / IEC JTC 1 / SC29,JVET-T2016, 2020.

[0230] Sullivan, G. J. and Wiegand, T. Rate-distortion optimization forvideo compression. IEEE signal processing magazine, .Sullivan, G. J., Ohm, J.-R., Han, W.-J., and Wiegand, T. Overview of the highefficiency video coding (HEVC) standard. IEEE Transactions on circuits andsystems for video technology, 22(12):1649-1668, 2012.

[0231] Vahdat, A. and Kautz, J. Nvae: A deep hierarchical variationalautoencoder. Advances in Neural Information Processing Systems, 33:19667-19679, 2020.

[0232] Wallace, G. K. The JPEG still picture compression standard.Communications of the ACM, 34(4):30-44, 1991.

[0233] Wang, H., Gan, W., Hu, S., Lin, J. Y., Jin, L., Song, L., Wang, P.,Katsavounidis, I., Aaron, A., and Kuo, C.-C. J. MCLJCV: a JND-based H. 264 / AVC video quality assessment dataset. In 2016 IEEE international conferenceon image processing (ICIP), pp. 1509-1513. IEEE, 2016.

[0234] Wang, Z., Simoncelli, E. P., and Bovik, A. C. Multiscale structuralsimilarity for image quality assessment. In The Thrity-Seventh AsilomarConference on Signals, Systems & Computers, 2003, volume 2, pp. 1398-1402.Ieee, 2003.

[0235] Wien, M. High efficiency video coding. Coding Tools andspecification, 24, 2015.

[0236] Wu, C.-Y., Singhal, N., and Krahenbuhl, P. Video compression throughimage interpolation. In Proceedings of the European conference on computervision , pp. 416-431, 2018.

[0237] Xue, T., Chen, B., Wu, J., Wei, D., and Freeman, WT Videoenhancement with task-oriented flow. International Journal of ComputerVision, 127(8):1106-1125, 2019.

[0238] Yang, R., Mentzer, F., Gool, LV, and Timofte, R. Learning for video compression with hierarchical quality and recurrent enhancement. InProceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition, pp. 6628-6637, 2020.

[0239] Zhang, R., Isola, P., Efros, AA, Shechtman, E., and Wang, O. Theunreasonable effectiveness of deep features as a perceptual metric. InProceedings of the IEEE conference on computer vision and patternrecognition, pp. 586-595, 2018.

[0240] Zhu, Y., Yang, Y., and Cohen, T. Transformer-based transform coding. In International Conference on Learning Representations, 2021.

[0241] Other aspects of this disclosure are described below.

[0242] Aspect 1A. A method for decoding media data, the method comprising: using a model to model a priori variables using a discrete cosine transform (DCT) to generate DCT coefficients; and decoding the media data based on the DCT coefficients.

[0243] Aspect 2A. The method according to aspect 1A, wherein decoding the media data comprises: quantizing and entropy decoding the media data using channel-by-channel 4x4 parameter mixing of the DCT coefficients.

[0244] Aspect 3A. The method according to aspect 1A or aspect 2A, the method further comprising: decoding the frequency information.

[0245] Aspect 4A. The method according to any one of Aspects 1A to 3A, wherein the model includes an I-frame model.

[0246] Aspect 5A. The method according to any one of Aspects 1A to 3A, wherein the model includes a P-frame model.

[0247] Aspect 6A. The method according to any one of Aspects 1A to 5A, the method further comprising: applying amortized batch normalization to the media data.

[0248] Aspect 7A. The method according to any one of Aspects 1A to 6A, the method further comprising: applying context modeling to the media data using a subsampling chessboard with uneven intervals.

[0249] Aspect 8A. The method according to any one of Aspects 1A to 7A, the method further comprising: quantizing the DCT coefficients using a learned quantization chamber size.

[0250] Aspect 9A. The method according to any one of Aspects 1A to 8A, wherein the media data includes video data.

[0251] Aspect 10A. The method according to any one of aspects 1A to 9A, wherein decoding includes encoding.

[0252] Aspect 11A. The method according to any one of aspects 1A to 10A, wherein decoding includes decoding.

[0253] Aspect 12A. An apparatus for decoding media data, the apparatus comprising: a memory configured to store media data; and one or more processors implemented in a circuit and coupled to the memory, the one or more processors being configured to perform a method according to any one of Aspects 1A to 11A.

[0254] Aspect 13A. The device according to aspect 12A, the device further includes a camera configured to capture the media data.

[0255] Aspect 14A. The device according to aspect 12A or aspect 3A, the device further includes a display configured to display the media data.

[0256] Aspect 15A. An apparatus for decoding media data, the apparatus comprising one or more components for performing the method according to any one of aspects 1A to 11A.

[0257] Aspect 16A. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of Aspects 1A to 11A.

[0258] Aspect 1B. A method for training a P-frame model for a neural media decoder, the method comprising: acquiring a corrupted I-frame; and using the corrupted I-frame to train the P-frame model.

[0259] Aspect 2B. According to the method of aspect 1B, the corrupted I-frame includes I-frames pre-trained to have a lower peak signal-to-noise ratio (PSNR) than the Lagrange multiplier.

[0260] Aspect 3B. The method according to aspect 1B or aspect 2B, wherein training the P-frame model comprises: training the P-frame model to a number of frames less than or equal to seven.

[0261] Aspect 4B. According to the method described in aspect 3B, training the P-frame model includes: training the P-frame model for three frames.

[0262] Aspect 5B. The method according to any one of Aspects 1B to 4B, wherein training the P-frame model comprises: applying forced quality enhancement (FQE).

[0263] Aspect 6B. A method for decoding media data, the method comprising: applying a pre-trained P-frame model to the media data, the pre-trained P-frame model being trained using corrupted I-frames; and decoding the media data based on applying the pre-trained P-frame model to the media data.

[0264] Aspect 7B. The method according to aspect 6B, wherein the media data includes video data.

[0265] Aspect 8B. The method according to aspect 6B or aspect 7B, wherein decoding includes encoding.

[0266] Aspect 9B. The method according to any one of aspects 6B to 8B, wherein decoding includes decoding.

[0267] Aspect 10B. An apparatus for decoding media data, the apparatus comprising: a memory configured to store media data; and one or more processors implemented in a circuit and coupled to the memory, the one or more processors being configured to perform a method according to any one of aspects 1B to 9B.

[0268] Aspect 11B. The device according to aspect 10B, the device further comprising a camera configured to capture the media data.

[0269] Aspect 12B. The device according to aspect 10B or aspect 11B, the device further includes a display configured to display the media data.

[0270] Aspect 13B. An apparatus for decoding media data, the apparatus comprising one or more components for performing the method according to any one of aspects 1B to 9B.

[0271] Aspect 14B. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of aspects 1B to 9B.

[0272] Aspect 1C. A method for encoding media data, the method comprising: determining a priori variables for a neural video decoder based on the media data; applying a discrete cosine transform (DCT) to the priori variables to generate DCT coefficients; and encoding the DCT coefficients.

[0273] Aspect 2C. The method according to aspect 1C, wherein encoding the DCT coefficients comprises: quantizing the channel-wise parameter mixing of the DCT coefficients to generate quantized DCT coefficients.

[0274] Aspect 3C. The method according to aspect 2C, wherein encoding the DCT coefficients further includes: entropy encoding the quantized DCT coefficients.

[0275] Aspect 4C. The method according to aspect 2C or aspect 3C, wherein the channel-by-channel parameter mixing includes channel-by-channel 4x4 parameter mixing.

[0276] Aspect 5C. The method according to any one of Aspects 1C to 4C, wherein the prior variables include the prior variables of the I-frame model of the neural video decoder.

[0277] Aspect 6C. The method according to any one of Aspects 1C to 4C, wherein the prior variables include the prior variables of the P-frame model of the neural video decoder.

[0278] Aspect 7C. The method according to aspect 6C, wherein the P-frame model includes at least one of a motion estimation model or a conditional decoder model.

[0279] Aspect 8C. The method according to aspect 6C or aspect 7C, wherein the P-frame model is trained using a corrupted I-frame having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold, or b) one or more regions with insertion errors.

[0280] Aspect 9C. The method according to any one of Aspects 1C to 8C, wherein the super-prior variable is based on a latent variable, and wherein the method further comprises: determining the latent variable, wherein determining the latent variable comprises: applying context modeling to the media data using a subsampling chessboard with non-uniform intervals.

[0281] Aspect 10C. The method according to aspect 9C, wherein the subsampling chessboard using the non-uniform interval comprises: stacked shifted copies of an NxN mask.

[0282] Aspect 11C. The method according to any one of aspects 1C to 10C, wherein the media data includes video data.

[0283] Aspect 12C. A method for decoding media data, the method comprising: determining DCT coefficients of a priori variables of a neural video decoder; applying an inverse DCT to the DCT coefficients to generate the priori variables; and decoding the media data based on the priori variables.

[0284] Aspect 13C. The method according to aspect 12C, wherein the DCT coefficients comprise a channel-wise parameter mixing of quantized DCT coefficients generated by a neural media encoder.

[0285] Aspect 14C. The method according to aspect 13C, wherein the channel-by-channel parameter mixing includes channel-by-channel 4x4 parameter mixing.

[0286] Aspect 15C. The method according to any one of aspects 12C to 14C, wherein determining the DCT coefficients comprises: entropy decoding the DCT coefficients.

[0287] Aspect 16C. The method according to any one of Aspects 12C to 15C, wherein the prior variables include the prior variables of the I-frame model of the neural video decoder.

[0288] Aspect 17C. The method according to any one of Aspects 12C to 15C, wherein the prior variables include the prior variables of the P-frame model of the neural video decoder.

[0289] Aspect 18C. The method according to aspect 11C, wherein the P-frame model includes at least one of a motion estimation model or a conditional decoder model.

[0290] Aspect 19C. The method according to aspect 17C or aspect 18C, wherein the P-frame model is trained using a corrupted I-frame having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold, or b) one or more regions with insertion errors.

[0291] Aspect 20C. The method according to any one of Aspects 12C to 19C, wherein the super-prior variable is based on a latent variable, and wherein decoding the media data comprises: determining the latent variable; and applying context modeling to the latent variable using a subsampling chessboard with uneven intervals.

[0292] Aspect 21C. The method according to aspect 20C, wherein the subsampling chessboard using the non-uniform interval comprises: stacked shifted copies of an NxN mask.

[0293] Aspect 22C. The method according to aspect 20C or aspect 21C, the method further comprising: dividing latent variable data into multiple groups, the latent variable data including the latent variables; decoding a first percentage of the latent variable data using a first group of the multiple groups; decoding a second percentage of the latent variable data using a second group of the multiple groups; and decoding a third percentage of the latent variable data using a third group of the multiple groups.

[0294] Aspect 23C. The method according to aspect 22C, wherein the first percentage and the second percentage are lower than the third percentage.

[0295] Aspect 24C. The method according to aspect 23C, wherein the first percentage is 25%, the second percentage is 25%, and the third percentage is 50%.

[0296] Aspect 25C. The method according to any one of aspects 12C to 24C, wherein the media data includes video data.

[0297] Aspect 26C. An apparatus for encoding media data, the apparatus comprising: one or more memories configured to store the media data; and one or more processors implemented in a circuit and coupled to the one or more memories, the one or more processors being configured to: determine a priori variables for a neural video decoder based on the media data; apply a discrete cosine transform (DCT) to the priori variables to generate DCT coefficients; and encode the DCT coefficients.

[0298] Aspect 27C. The apparatus according to aspect 26C, wherein, as part of encoding the DCT coefficients, the one or more processors are configured to quantize the channel-wise parameter mixing of the DCT coefficients to generate quantized DCT coefficients.

[0299] Aspect 28C. The apparatus according to aspect 27C, wherein, as part of encoding the DCT coefficients, the one or more processors are configured to entropy encode the quantized DCT coefficients.

[0300] Aspect 29C. The apparatus according to aspect 27C or aspect 28C, wherein the channel-by-channel parameter mixing includes channel-by-channel 4x4 parameter mixing.

[0301] Aspect 30C. The apparatus according to any one of aspects 26C to 29C, wherein the prior variables include the prior variables of the I-frame model of the neural video decoder.

[0302] Aspect 31C. The apparatus according to any one of Aspects 26C to 30C, wherein the prior variables include the prior variables of the P-frame model of the neural video decoder.

[0303] Aspect 32C. The apparatus according to aspect 31C, wherein the P-frame model includes at least one of a motion estimation model or a conditional decoder model.

[0304] Aspect 33C. The device according to aspect 31C or aspect 32C, wherein the P-frame model is trained using a corrupted I-frame having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold, or b) one or more regions with insertion errors.

[0305] Aspect 34C. The device according to any one of Aspects 26C to 33C, wherein the super-prior variable is based on a latent variable, wherein the one or more processors are further configured to determine the latent variable, and wherein, as part of determining the latent variable, the one or more processors are configured to apply context modeling to the media data using a subsampling chessboard with non-uniform intervals.

[0306] Aspect 35C. The device according to aspect 34C, wherein, as part of a subsampling chessboard using the non-uniform interval, the one or more processors are configured as shifted copies of a stacked NxN mask.

[0307] Aspect 36C. The device according to any one of aspects 26C to 35C, wherein the media data includes video data.

[0308] Aspect 37C. The device according to any one of aspects 26C to 36C, the device further comprising a camera configured to capture the media data.

[0309] Aspect 38C. An apparatus for decoding media data, the apparatus comprising: one or more memories configured to store the media data; and one or more processors implemented in circuitry and coupled to the one or more memories, the one or more processors being configured to: determine DCT coefficients of a hyperprior variable of a neural video decoder; apply an inverse DCT to the DCT coefficients to generate the hyperprior variable; and decode the media data based on the hyperprior variable.

[0310] Aspect 39C. The device according to aspect 38C, wherein the DCT coefficients comprise a channel-wise parameter mix of quantized DCT coefficients generated by a neural media encoder.

[0311] Aspect 40C. The apparatus according to aspect 39C, wherein the channel-by-channel parameter mixing includes channel-by-channel 4x4 parameter mixing.

[0312] Aspect 41C. The apparatus according to any one of aspects 38C to 40C, wherein, as part of determining the DCT coefficients, the one or more processors are configured to perform entropy decoding on the DCT coefficients.

[0313] Aspect 42C. The apparatus according to any one of aspects 38C to 41C, wherein the prior variables include the prior variables of the I-frame model of the neural video decoder.

[0314] Aspect 43C. The apparatus according to any one of aspects 38C to 41C, wherein the prior variables include the prior variables of the P-frame model of the neural video decoder.

[0315] Aspect 44C. The apparatus according to aspect 43C, wherein the P-frame model includes at least one of a motion estimation model or a conditional decoder model.

[0316] Aspect 45C. The device according to aspect 43C or aspect 44C, wherein the P-frame model is trained using a corrupted I-frame having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold, or b) one or more regions with insertion errors.

[0317] Aspect 46C. The device according to any one of aspects 38C to 45C, wherein the super-prior variable is based on a latent variable, and wherein, as part of decoding the media data, the one or more processors are configured to: determine the latent variable; and apply context modeling to the latent variable using a subsampling chessboard with uneven intervals.

[0318] Aspect 47C. The device according to aspect 46C, wherein, as part of a subsampling chessboard using the non-uniform interval, the one or more processors are configured as shifted copies of a stacked NxN mask.

[0319] Aspect 48C. The apparatus according to aspect 46C or aspect 47C, wherein the one or more processors are further configured to: divide latent variable data into multiple groups, the latent variable data including the latent variables; decode a first percentage of the latent variable data using a first group of the multiple groups; decode a second percentage of the latent variable data using a second group of the multiple groups; and decode a third percentage of the latent variable data using a third group of the multiple groups.

[0320] Aspect 49C. The device according to aspect 48C, wherein the first percentage and the second percentage are lower than the third percentage.

[0321] Aspect 50C. The device according to aspect 49C, wherein the first percentage is 25%, the second percentage is 25%, and the third percentage is 50%.

[0322] Aspect 51C. The device according to any one of aspects 38C to 50C, wherein the media data includes video data.

[0323] Aspect 52C. The device according to any one of aspects 38C to 51C, the device further comprising a display configured to display the media data.

[0324] Aspect 1D. A method for training a P-frame model for a neural media decoder, the method comprising: acquiring a corrupted I-frame, the corrupted I-frame having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold, or b) having one or more regions with insertion errors; and using the corrupted I-frame to train the P-frame model.

[0325] Aspect 2D. According to the method of aspect 1D, the method further includes: generating the corrupted I-frame.

[0326] Aspect 3D. The method according to aspect 1D or 2D, wherein the corrupted I-frame has the PSNR that satisfies the threshold, and wherein in order to satisfy the threshold, the PSNR is lower than the threshold or lower than or equal to the threshold.

[0327] Aspect 4D. The method according to any one of Aspects 1D to 3D, wherein the threshold is based on a Lagrange multiplier associated with the P-frame model.

[0328] Aspect 5D. The method according to any one of Aspects 1D to 4D, wherein the corrupted I-frame is pre-configured to satisfy the threshold.

[0329] Aspect 6D. The method according to any one of Aspects 1D to 5D, wherein training the P-frame model comprises: training the P-frame model to a number of frames less than or equal to seven.

[0330] Aspect 7D. According to the method described in aspect 6, training the P-frame model includes: training the P-frame model for three frames.

[0331] Aspect 8D. A method for decoding media data, the method comprising: applying a pre-trained P-frame model to the media data, the pre-trained P-frame model being trained using corrupted I-frames, the corrupted I-frames having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold, or b) one or more regions with insertion errors; and decoding the media data based on applying the pre-trained P-frame model to the media data.

[0332] Aspect 9D. The method according to aspect 8D, wherein the corrupted I-frame has a PSNR that satisfies the threshold, and wherein, in order to satisfy the threshold, the PSNR is lower than the threshold or lower than or equal to the threshold.

[0333] Aspect 10D. The method according to aspect 8D or 9D, wherein the media data includes video data.

[0334] Aspect 11D. The method according to any one of aspects 8D to 10D, wherein decoding includes encoding.

[0335] Aspect 12D. The method according to any one of Aspects 8D to 10D, wherein decoding includes decoding.

[0336] Aspect 13D. An apparatus comprising: one or more memories configured to store media data and a P-frame model; and one or more processors implemented in a circuit and coupled to the one or more memories, the one or more processors being configured to: acquire a corrupted I-frame having at least one of the following: a) a peak signal-to-noise ratio (PSNR) below a threshold, or b) one or more regions with insertion errors; and use the corrupted I-frame to train the P-frame model.

[0337] Aspect 14D. The device according to aspect 13D, wherein the one or more processors are further configured to generate the corrupted I-frame.

[0338] Aspect 15D. The device according to aspect 13D or 14D, wherein the corrupted I-frame has the PSNR that satisfies the threshold, and wherein in order to satisfy the threshold, the PSNR is lower than the threshold or lower than or equal to the threshold.

[0339] Aspect 16D. The device according to any one of Aspects 13D to 15D, wherein the threshold is based on a Lagrange multiplier associated with the P-frame model.

[0340] Aspect 17D. The device according to any one of Aspects 13D to 16D, wherein the corrupted I-frame is pre-configured to satisfy the threshold.

[0341] Aspect 18D. The device according to any one of aspects 13D to 17D, wherein the threshold comprises a Lagrange multiplier.

[0342] Aspect 19D. The device according to any one of aspects 13D to 18D, wherein the one or more processors are configured to train the P-frame model to less than or equal to seven frames.

[0343] Aspect 20D. The device according to aspect 19D, wherein one or more processors are configured to train the P-frame model up to three frames.

[0344] Aspect 21D. The device according to any one of aspects 13D to 20D, the device further comprising a camera configured to capture the media data.

[0345] Aspect 22D. The device according to any one of aspects 13D to 20D, the device further comprising a display configured to display the media data.

[0346] Aspect 23D. An apparatus for decoding media data, the apparatus comprising: one or more memories configured to store media data; and one or more processors implemented in a circuit and coupled to the one or more memories, the one or more processors being configured to: apply a pre-trained P-frame model to the media data, the pre-trained P-frame model being trained using corrupted I-frames, the corrupted I-frames having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold, or b) one or more regions with insertion errors; and decode the media data based on applying the pre-trained P-frame model to the media data.

[0347] Aspect 24D. The device according to aspect 23D, wherein the corrupted I-frame has the PSNR that satisfies the threshold, and wherein, in order to satisfy the threshold, the PSNR is lower than the threshold or lower than or equal to the threshold.

[0348] Aspect 25D. The device according to aspect 23D or 24D, wherein the media data includes video data.

[0349] Aspect 26D. The device according to any one of aspects 23D to 25D, wherein, as part of decoding the media data, the one or more processors are configured to encode the media data.

[0350] Aspect 27D. The device according to any one of aspects 23D to 25D, wherein, as part of decoding the media data, the one or more processors are configured to decode the media data.

[0351] Aspect 28D. The device according to any one of aspects 23D to 26D, the device further comprising a camera configured to capture the media data.

[0352] Aspect 29D. The device according to any one of aspects 23D to 25D or 27D, the device further comprising a display configured to display the media data.

[0353] It should be recognized that, based on the examples, certain actions or events of any technique described herein may be performed in a different sequence, and may be added, combined, or omitted entirely (e.g., not all actions or events described are necessary for implementing the technique). Furthermore, in some examples, actions or events may be performed concurrently (e.g., through multithreading, interrupt handling, or multiple processors) rather than sequentially.

[0354] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on a computer-readable medium or transmitted via a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium (which corresponds to a tangible medium such as a data storage medium) or a communication medium, including, for example, any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol. In this way, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. Computer program products may include computer-readable media.

[0355] By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies (such as infrared, radio, and microwave), then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (such as infrared, radio, and microwave) are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but instead refer to non-transient tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs utilize lasers to optically reproduce data. The combinations described above should also be included within the scope of computer-readable media.

[0356] Instructions can be executed by one or more processors, such as one or more DSPs, general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuits. Therefore, the terms "processor" and "processing circuit" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Moreover, these techniques can be fully implemented in one or more circuit or logic elements.

[0357] The techniques disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or IC sets (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but implementation by different hardware units is not necessarily required. Rather, as described above, various units can be combined in a codec hardware unit, or various units can be provided by a collection of interoperable hardware units (including one or more processors as described above) combined with appropriate software and / or firmware.

[0358] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A method for training a P-frame model for a neural media decoder, the method comprising: Acquire a corrupted I-frame, the corrupted I-frame having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold, or b) one or more regions with insertion errors; and The corrupted I-frame is used to train the P-frame model.

2. The method according to claim 1, further comprising: Generate the corrupted I-frame.

3. The method of claim 1, wherein the damaged I-frame has a PSNR that satisfies the threshold, and wherein, in order to satisfy the threshold, the PSNR is lower than the threshold or lower than or equal to the threshold.

4. The method of claim 1, wherein the threshold is based on a Lagrange multiplier associated with the P-frame model.

5. The method of claim 1, wherein the corrupted I-frame is pre-configured to satisfy the threshold.

6. The method of claim 1, wherein training the P-frame model comprises: The P-frame model is trained to have seven frames or less.

7. The method of claim 6, wherein training the P-frame model comprises: The P-frame model was trained for three frames.

8. A method for decoding media data, the method comprising: A pre-trained P-frame model is applied to the media data. This pre-trained P-frame model is trained using corrupted I-frames, which have at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold, or b) one or more regions with insertion errors; and The media data is decoded by applying the pre-trained P-frame model.

9. The method of claim 8, wherein the corrupted I-frame has a PSNR that satisfies the threshold, and wherein, in order to satisfy the threshold, the PSNR is lower than the threshold or lower than or equal to the threshold.

10. The method of claim 8, wherein the media data includes video data.

11. The method of claim 8, wherein decoding includes encoding.

12. The method of claim 8, wherein decoding includes decoding.

13. An apparatus, said apparatus comprising: One or more memories, the one or more memories being configured to store media data and P-frame models; and One or more processors, implemented in a circuit and coupled to one or more memories, are configured to: Acquire a corrupted I-frame, the corrupted I-frame having at least one of the following: a) a peak signal-to-noise ratio (PSNR) below a threshold, or b) one or more regions with insertion errors; and The corrupted I-frame is used to train the P-frame model.

14. The device of claim 13, wherein the one or more processors are further configured to generate the corrupted I-frame.

15. The device of claim 13, wherein the damaged I-frame has a PSNR that satisfies the threshold, and wherein, in order to satisfy the threshold, the PSNR is lower than the threshold or lower than or equal to the threshold.

16. The device of claim 13, wherein the threshold is based on a Lagrange multiplier associated with the P-frame model.

17. The device of claim 13, wherein the corrupted I-frame is pre-configured to satisfy the threshold.

18. The device of claim 13, wherein the threshold comprises a Lagrange multiplier.

19. The device of claim 13, wherein the one or more processors are configured to train the P-frame model to a maximum of seven frames or less.

20. The device of claim 19, wherein the one or more processors are configured to train the P-frame model up to three frames.

21. The device of claim 13, further comprising a camera configured to capture the media data.

22. The device of claim 13, further comprising a display configured to display the media data.

23. An apparatus for decoding media data, the apparatus comprising: One or more memories configured to store media data; and One or more processors, implemented in a circuit and coupled to one or more memories, are configured to: A pre-trained P-frame model is applied to the media data. This pre-trained P-frame model is trained using corrupted I-frames, which have at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold, or b) one or more regions with insertion errors; and The media data is decoded by applying the pre-trained P-frame model.

24. The device of claim 23, wherein the corrupted I-frame has a PSNR that satisfies the threshold, and wherein, in order to satisfy the threshold, the PSNR is lower than the threshold or lower than or equal to the threshold.

25. The device of claim 23, wherein the media data includes video data.

26. The apparatus of claim 23, wherein, as part of decoding the media data, the one or more processors are configured to encode the media data.

27. The apparatus of claim 23, wherein, as part of decoding the media data, the one or more processors are configured to decode the media data.

28. The device of claim 23, further comprising a camera configured to capture the media data.

29. The device of claim 23, further comprising a display configured to display the media data.

Images