Discrete cosine hyper-prior in neural image coding

Abstract

Example techniques and devices for coding media data are described. An example device for encoding media data includes one or more memories configured to store media data and one or more processors implemented in circuitry and coupled to the one or more memories. The one or more processors are configured to determine a hyper-prior variable of a neural video coder based on the media data. The one or more processors are configured to apply a discrete cosine transform (DCT) to the hyper-prior variable to generate DCT coefficients and encode the DCT coefficients.

Description

[0001] This application claims priority to U.S. Patent Application No. 18 / 410,556, filed January 11, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to media encoding and decoding, including the encoding and decoding of image and video data. Background Technology

[0003] Digital media capabilities can be incorporated into a wide variety of devices, including digital televisions, digital live broadcast systems, wireless broadcasting systems, personal digital assistants (PDAs), laptops or desktop computers, tablets, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite wireless phones (so-called "smartphones"), video conferencing equipment, video streaming devices, and so on. Digital video devices implement video decoding technologies, such as those defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4 (Part 10, Advanced Video Decoding (AVC)), ITU-T H.265 / High Efficiency Video Decoding (HEVC), ITU-T H.266 / Variety Video Decoding (VVC) and extensions to such standards, as well as proprietary video codecs / formats such as AOMedia Video1 (AV1) developed by the Open Media Alliance. By implementing such video decoding technologies, video devices can more efficiently send, receive, encode, decode, and / or store digital video information. Summary of the Invention

[0004] Generally, this disclosure describes techniques for media compression, including techniques for video and / or image encoding and decoding. Neural network-based media (e.g., image and / or video) compression methods can outperform current standards and offer several additional advantages. A neural network-based video decoder (which may be a media decoder) may be referred to as a neural video decoder (NVC).

[0005] This disclosure describes techniques for improving decoding using neural media decoders (such as NVCs). Some example NVCs utilize nonparametric methods for univariate density estimation and cross-channel amortization learning, rather than using over-prior variables. However, output variables may retain much information from over-prior variables. Therefore, it may be desirable to utilize over-prior variables when modeling output variables.

[0006] This disclosure also describes improved training techniques. Training NVC can lead to the accumulation of temporal errors over large test-time group of pictures (GOP) sizes. Therefore, some NVCs attempt to correct this accumulation of temporal errors by implementing a weighted objective, where the importance of distortion increases with the depth of the training sequence. While such techniques can delay degradation, they still struggle to maintain quality across long image sequences. Therefore, it may be desirable to train the NVC such that it can accumulate improved reconstructions across subsequent training frames.

[0007] While the decoding capabilities of NVC have improved, work remains to be done to make NVC competitive with conventional or hand-designed methods, such as codecs implementing VVC. It should be noted that the terms codec and decoder may be used interchangeably in this disclosure. This disclosure discusses several techniques for both neural image and video compression (e.g., media compression). These techniques are often simple, model-general, and readily incorporable into other techniques. This disclosure also identifies that current evaluation protocols significantly disadvantage conventional, hand-designed decoders and provides a basis for comparison of test conditions to encourage future use of video standards organizations. Decoders based on the techniques of this disclosure, built on a conditional decoding framework, can outperform the disclosed methods (significantly outperforming many methods). Neural decoders utilizing such techniques can outperform VVC for the following expected metric: weighted YUV-PSNR.

[0008] In one example, a method for encoding media data includes: determining a priori variables for a neural video decoder based on the media data; applying a discrete cosine transform (DCT) to the priori variables to generate DCT coefficients; and encoding the DCT coefficients.

[0009] In another example, a method for decoding media data includes: determining the DCT coefficients of a super-prior variable of a neural video decoder; applying an inverse DCT to the DCT coefficients to generate the super-prior variable; and decoding the media data based on the super-prior variable.

[0010] In another example, an apparatus for encoding media data includes one or more memories configured to store the media data; and one or more processors implemented in a circuit and coupled to the one or more memories, the one or more processors being configured to: determine a priori variables for a neural video decoder based on the media data; apply a discrete cosine transform (DCT) to the priori variables to generate DCT coefficients; and encode the DCT coefficients.

[0011] In another example, an apparatus for decoding media data includes one or more memories configured to store the media data; and one or more processors implemented in a circuit and coupled to the one or more memories, the one or more processors being configured to: determine DCT coefficients of a hyperprior variable of a neural video decoder; apply an inverse DCT to the DCT coefficients to generate the hyperprior variable; and decode the media data based on the hyperprior variable.

[0012] Details of one or more examples are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the description, drawings, and claims. Attached Figure Description

[0013] Figure 1 This is a block diagram illustrating an example media encoding and decoding system that can perform the techniques of this disclosure.

[0014] Figure 2 This is a conceptual diagram illustrating the differences between conventional media decoding and neural-based media decoding.

[0015] Figure 3 This is a block diagram of an example neural image decoder that uses a variational autoencoder and a super-prior network for entropy decoding according to one or more aspects of this disclosure.

[0016] Figure 4 This is a conceptual diagram illustrating an example of adaptive reference selection.

[0017] Figure 5 This is a conceptual diagram illustrating an example causal inference structure for an uneven chessboard context module.

[0018] Figure 6 is an example of y, and after DCT transformation A conceptual diagram of an example channel.

[0019] Figure 7A This is a block diagram illustrating an example of a conditional P-frame model.

[0020] Figure 7B It is a block diagram of an example architecture for conditional analysis and synthetic transformation based on one or more aspects of this disclosure.

[0021] Figure 8 This is a conceptual diagram illustrating an example of a three-frame verification sequence obtained during training.

[0022] Figure 9 This is a graph illustrating example peak signal-to-noise ratio (PSNR) and bits per pixel across 125 evaluation frames with and without forced quality enhancement (FQE).

[0023] Figure 10 This is a graph illustrating example RGB rate distortion curves for the UVG and MCL-JCV datasets.

[0024] Figure 11 This is a graph illustrating the example RGB-PSNR difference between verification sequences with and without I-frames.

[0025] Figure 12 This is a graph illustrating an example of kernel density estimation across channels in the Kodak dataset in RGB and YUV formats.

[0026] Figure 13 This is a graph illustrating an example RGB rate distortion curve from the Kodak dataset.

[0027] Figure 14 This is a block diagram illustrating an example system for training a P-frame model according to one or more aspects of this disclosure.

[0028] Figure 15 This is a flowchart illustrating example techniques for encoding media data according to one or more aspects of this disclosure.

[0029] Figure 16 This is a flowchart illustrating an example technique for decoding media data according to one or more aspects of this disclosure.

[0030] Figure 17 This is a flowchart illustrating an example technique for training a P-frame model according to one or more aspects of this disclosure.

[0031] Figure 18 This is a flowchart illustrating an example technique for decoding media data using a trained P-frame model, according to one or more aspects of this disclosure. Detailed Implementation

[0032] This disclosure describes techniques for encoding and decoding media data (e.g., images or videos) such as video data using neural network-based media decoding techniques. Specifically, this disclosure describes techniques for utilizing prior variables when modeling output variables, for example, by applying a discrete cosine transform (DCT) to the prior variables. This disclosure also describes techniques for training neural media decoders (such as NVC), and the use of such trained neural media decoders to avoid or reduce the accumulation of temporal errors, for example, by using corrupted I-frames. A corrupted I-frame can be an I-frame with a peak signal-to-noise ratio (PSNR) that meets a threshold and / or an I-frame with one or more regions of insertion errors. Such techniques, individually or together, can improve the quality of video decoding, such as PSNR, thereby providing a better viewing experience for viewers decoding video data.

[0033] Figure 1 This is a block diagram illustrating an example media encoding and decoding system 100 capable of performing the techniques of this disclosure. In the context of this disclosure, media may include any digital file to be compressed, including video data and / or images. The example techniques of this disclosure generally relate to decoding (encoding and / or decoding) video data and / or image data. Although Figure 1 Examples will be described with reference to media encoding and decoding, but the techniques of this application are equally applicable to the encoding and decoding of any type of data file using neural-based compression techniques.

[0034] like Figure 1 As shown, in this example, system 100 includes source device 102, which provides encoded media data to be decoded and displayed by destination device 116. Specifically, source device 102 provides the media data to destination device 116 via computer-readable medium 110. Source device 102 and destination device 116 can include any of a wide variety of devices, including desktop computers, laptop computers, mobile devices, tablet computers, set-top boxes, mobile phones such as smartphones, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, broadcast receiver devices, etc. In some cases, source device 102 and destination device 116 may be configured for wireless communication and are therefore referred to as wireless communication devices.

[0035] exist Figure 1 In the example, source device 102 includes a media source 104, memory 106, media encoder 200, and output interface 108. Destination device 116 includes an input interface 122, media decoder 300, memory 120, and display device 118. According to this disclosure, the media encoder 200 of source device 102 and the media decoder 300 of destination device 116 can be configured to apply techniques for media decoding in a neural-based media compression system. Thus, source device 102 represents an example of a media encoding device, while destination device 116 represents an example of a media decoding device. In other examples, the source device and destination device may include other components or arrangements. For example, source device 102 may receive media data from an external media source, such as an external camera. Similarly, destination device 116 may interface with an external display device, rather than including an integrated display device.

[0036] like Figure 1The system 100 shown is merely an example. Generally, any digital media encoding and / or decoding device can perform techniques for media decoding in a neural-based media compression system. Source device 102 and destination device 116 are merely examples of such decoding devices, where source device 102 generates decoded media data for transmission to destination device 116. This disclosure refers to a “decoding” device as a device that performs the decoding (e.g., encoding and / or decoding) of data. Thus, media encoder 200 and media decoder 300 represent examples of decoding devices (specifically, media encoder and media decoder, respectively). In some examples, media encoder 200 and media decoder 300 may be examples of NVCs. In some examples, source device 102 and destination device 116 may operate in a substantially symmetrical manner, such that each of source device 102 and destination device 116 includes both a media encoding component and a decoding component. Therefore, system 100 may support one-way or two-way media transmission between source device 102 and destination device 116, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0037] Generally, media source 104 represents the source of media data (i.e., raw, unencoded media data) and provides a series of consecutive pictures (also referred to as "frames") of media data to media encoder 200, which encodes the data of the pictures. Media source 104 of source device 102 may include video capture devices, such as cameras, video archives containing previously captured raw video, and / or video feed interfaces for receiving video from video content providers. Alternatively, media source 104 may generate computer graphics-based data as source media, or a combination of live media, archived media, and computer-generated media. In each case, media encoder 200 encodes the captured, pre-captured, or computer-generated media data. Media encoder 200 may rearrange the pictures from the receiving order (sometimes referred to as "display order") to a decoding order for decoding. Media encoder 200 may generate a bitstream comprising the encoded media data. Then, the source device 102 can output the encoded media data to the computer-readable medium 110 via the output interface 108 for reception and / or retrieval by, for example, the input interface 122 of the destination device 116.

[0038] The memory 106 of source device 102 and the memory 120 of destination device 116 represent general-purpose memory. In some examples, memories 106 and 120 may store raw media data, such as raw media from media source 104 and raw decoded media data from media decoder 300. Additionally or alternatively, memories 106 and 120 may store software instructions executable by, for example, media encoder 200 and media decoder 300. Although memories 106 and 120 are shown separately from media encoder 200 and media decoder 300 in this example, it should be understood that media encoder 200 and media decoder 300 may also include internal memory for functionally similar or equivalent purposes. Furthermore, memories 106 and 120 may store, for example, encoded media data output from media encoder 200 and input to media decoder 300. In some examples, portions of memories 106 and 120 may be allocated as one or more buffers, for example, to store raw decoded and / or encoded media data.

[0039] Computer-readable medium 110 may represent any type of medium or device capable of transmitting encoded media data from source device 102 to destination device 116. In one example, computer-readable medium 110 represents a communication medium enabling source device 102 to transmit encoded media data directly to destination device 116 in real time, for example, via a radio frequency network or a computer-based network. According to a communication standard such as a wireless communication protocol, output interface 108 may modulate the transmitted signal including the encoded media data, and input interface 122 may demodulate the received transmitted signal. The communication medium may include any wireless or wired communication medium, such as radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network such as a local area network, a wide area network, or a global network (such as the Internet). The communication medium may include a router, switch, base station, or any other equipment that may be useful for facilitating communication from source device 102 to destination device 116.

[0040] In some examples, source device 102 can output encoded data from output interface 108 to storage device 112. Similarly, destination device 116 can access encoded data from storage device 112 via input interface 122. Storage device 112 may include any of a variety of distributed or locally accessed data storage media, such as hard disk drives, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded media data.

[0041] In some examples, source device 102 may output encoded media data to file server 114 or another intermediate storage device that may store the encoded media data generated by source device 102. Destination device 116 may access the stored media data from file server 114 via streaming or downloading.

[0042] File server 114 can be any type of server device capable of storing encoded media data and sending encoded media data to destination device 116. File server 114 may represent a web server (e.g., for a website), a server configured to provide file transfer protocol services (such as File Transfer Protocol (FTP) or FLUTE-based file delivery protocol), a Content Delivery Network (CDN) device, a Hypertext Transfer Protocol (HTTP) server, a Multimedia Broadcast Multicast Service (MBMS) or Enhanced MBMS (eMBMS) server, and / or a Network Attached Storage (NAS) device. File server 114 may additionally or alternatively implement one or more HTTP streaming protocols, such as HTTP-based Dynamic Adaptive Streaming (DASH), HTTP Live Streaming (HLS), Real-Time Streaming Protocol (RTSP), HTTP Dynamic Streaming, etc.

[0043] Destination device 116 can access encoded media data from file server 114 via any standard data connection, including an internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem, etc.), or a combination of both, suitable for accessing encoded media data stored on file server 114. Input interface 122 can be configured to operate according to any or more of the various protocols discussed above for retrieving or receiving media data from file server 114, or other such protocols for retrieving media data.

[0044] Output interface 108 and input interface 122 can represent a wireless transmitter / receiver, a modem, a wired networking component (e.g., an Ethernet card), a wireless communication component operating according to any of the various IEEE 802.11 standards, or other physical components. In examples where output interface 108 and input interface 122 include wireless components, output interface 108 and input interface 122 can be configured to transmit data (such as encoded media data) according to cellular communication standards (such as 4G, 4G-LTE (Long Term Evolution), Advanced LTE, 5G, etc.). In some examples where output interface 108 includes a wireless transmitter, output interface 108 and input interface 122 can be configured to operate according to other wireless standards (such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee)).™ ),Bluetooth ™ The source device 102 and / or destination device 116 may include corresponding system-on-chip (SoC) devices. For example, source device 102 may include an SoC device for performing functions belonging to media encoder 200 and / or output interface 108, and destination device 116 may include an SoC device for performing functions belonging to media decoder 300 and / or input interface 122.

[0045] The technology disclosed herein can be applied to media decoding to support any multimedia application in a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, Internet streaming media transmission (such as HTTP-based Dynamic Adaptive Streaming (DASH)), digital media encoded onto data storage media, decoding of digital media stored on data storage media, or other applications.

[0046] The input interface 122 of the destination device 116 receives an encoded media bitstream from the computer-readable medium 110 (e.g., communication media, storage device 112, file server 114, etc.). The encoded media bitstream may include signaling information defined by the media encoder 200, which is also used by the media decoder 300. The display device 118 displays a decoded image of the decoded media data to the user. The display device 118 may represent any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0047] Despite Figure 1 Not shown, but in some examples, both the media encoder 200 and the media decoder 300 may be integrated with the audio encoder and / or audio decoder, and may include appropriate MUX-DEMUX units or other hardware and / or software to process multiplexed streams that include both audio and media in a common data stream.

[0048] Media encoder 200 and media decoder 300 may each be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the technology is partially implemented in software, the device may store instructions for the software in a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the technology of this disclosure. Each of media encoder 200 and media decoder 300 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined encoder / decoder (decoder) in the respective device. Devices including media encoder 200 and / or media decoder 300 may include integrated circuits, microprocessors, and / or wireless communication devices (such as cellular phones).

[0049] Neural network-based image and video (e.g., media) compression methods can be competitive with current standards and offer several additional advantages.

[0050] This disclosure describes techniques for improving decoding using neural media decoders (such as NVCs). Most NVCs utilize nonparametric methods for univariate density estimation and cross-channel amortization learning, rather than using prior variables. However, output variables may retain much information from prior variables. Therefore, it may be desirable to utilize prior variables when modeling output variables, such as by applying DCT to the prior variables.

[0051] This disclosure also describes improved training techniques for NVC. Training NVC can lead to the accumulation of temporal errors over large test-time group of pictures (GOP) sizes. Therefore, some NVCs attempt to correct this accumulation by implementing a weighted objective, where the importance of distortion increases with the depth of the training sequence. While such techniques can delay degradation, they still struggle to maintain quality across long sequences. Therefore, it may be desirable to train the NVC such that it can accumulate improved reconstructions across subsequent training frames. Techniques disclosed in this disclosure include using corrupted I-frames to train P-frame models, which can reduce or improve the NVC's resistance to the accumulation of temporal errors.

[0052] Figure 2 This is a conceptual diagram illustrating an example of the differences between conventional media decoding and neural-based media decoding. Because conventional (e.g., hand-designed) and neural-based decoders use very different forms of compression, they employ entropy decoding in significantly different ways. A summary of the main differences is as follows: Figure 2As shown, it can be observed that even when two methods use the same decoding method (such as arithmetic decoding), all other aspects are different. This implies that it is beneficial to develop new technologies to optimize the performance of neural-based decoders.

[0053] like Figure 2 As shown, the design phases for both conventional decoding and neural-based decoding technologies include a data organization process 201 and a statistical data modeling process 202. For conventional media decoding technologies, such as H.265, the data organization process 201 includes defining decoding elements (e.g., syntax elements and decoding modes). For neural-based decoding, the data organization process 201 includes training and developing an image / video (e.g., media) autoencoder network. For the statistical data modeling process 202, conventional decoding technologies include context-defined decoding, such as context-adaptive binary arithmetic decoding for entropy decoding. For neural-based decoding, the statistical data modeling process 202 includes developing and training a hyper-prior neural network.

[0054] Both conventional decoding and neural-based decoding techniques include an implementation phase comprising a probability determination process 204 and an information-to-bit conversion process 206. Both the probability determination process 204 and the information-to-bit conversion process 206 are based on the data to be decoded. For conventional decoding, the probability determination process 204 (e.g., a bit rate assignment process) includes adaptive context probability estimation. For neural-based decoding, the probability determination process 204 includes the determination of probability distribution parameters, as will be described in more detail below. Both conventional decoding and neural-based decoding techniques use an arithmetic and / or entropy decoder (e.g., an arithmetic or entropy encoder or decoder) for the information-to-bit conversion process 206. The arithmetic decoder may be a Huffman, range, or Asymmetric Digital System (ANS) decoder.

[0055] The data stream management phase involves managing the compressed data 208. In some example conventional decoding systems, the compressed data resides in a single bit stream. Some example neural-based decoding systems use multiple streams for concurrent decoding.

[0056] An important class of methods proposed for neural-based image and video compression adapts variational autoencoder architectures, where latent variables are quantized and entropy-decoded using a fixed learning distribution. In some examples, this architecture has been extended by adding a smaller neural network called a “super-prior,” which uses the latent variables generated by the autoencoder to define the probability distribution used for entropy decoding of those autoencoder latent variables. This super-prior network also uses quantization and entropy decoding, but with predefined (e.g., learned) PDF parameters.

[0057] Figure 3A is a block diagram of an example neural image decoder using a variational autoencoder and a super-prior network for entropy decoding according to one or more aspects of this disclosure. The process is as follows: Figure 3 Figure A illustrates a neural network used for image analysis and synthesis that implements a variational autoencoder, with the entropy decoding of its variables defined by a super-prior decoder network (used at both the transmitter and receiver). Since "encoder" and "decoder" (e.g., media encoder 200 and media decoder 300) are already used to denote the autoencoder portion, in neural network literature, the systems traditionally referred to by those names are typically called "transmitter" and "receiver," respectively. The input image 400 and output image 428 may comprise frames of picture or video data.

[0058] exist Figure 3 In A, media encoder 200 may include an image analysis neural network 402, a hyperprior encoder neural network 404, a DCT process 405, a quantization process 406, an entropy encoding process 408, a quantization process 416, and an entropy encoding process 418. Media encoder 200 may also include an entropy decoding process 412, a hyperprior decoder neural network 424, an inverse DCT (IDCT) process 423, an entropy decoding process 422, and an image synthesis neural network 426. Commonly, these components or functions may resemble the reconstruction loop or decoding loop in a hybrid video decoder. Although not shown, media decoder 300 may include a structure similar to entropy decoding process 412, hyperprior decoder neural network 424, IDCT process 423, entropy decoding process 422, and image synthesis neural network 426 to reconstruct the output image. For example, media encoder 200 may output bitstream 1 410 and bitstream 2 420. The media decoder 300 can obtain bitstream 1 410 and bitstream 2 420 and, similar to the media encoder 200, generates an output image, such as output image 428, by using a structure similar to that of the entropy decoding process 412, the super-prior decoder neural network 424, the IDCT process 423, the entropy decoding process 422, and the image synthesis neural network 426.

[0059] Image analysis neural network 402 is a neural network configured to encode and / or compress input image 400. The compressed data stream created by image analysis neural network 402 is then processed by super-prior encoder neural network 404. The output of super-prior encoder neural network 404, which may include super-prior variables, then undergoes DCT process 405. DCT process 405 may include applying DCT to the output of super-prior encoder neural network 404 to generate DCT coefficients for the super-prior variables. The output of DCT process 405 (e.g., DCT coefficients) is quantized by quantization process 406 and entropy decoded by entropy encoding process 408 to create bit stream 1 410. Super-prior encoder neural network 404 is part of a process configured to determine parameters (e.g., standard deviation, variance, mean, etc.) of the PDF of the data stream created by image analysis neural network 402. The parameters of the PDF can then be used for entropy encoding of the data stream from image analysis neural network 402.

[0060] The entropy encoding process 408 itself uses learned parameters 414 learned from multiple images to be encoded. Learned parameters 414 may include learned motion parameters. For a P-frame model, such learned parameters 414 may be learned using corrupted I-frames, as discussed further later in this disclosure. The entropy decoding process 412 uses the same PDF parameters to entropy decode bitstream 1 410. The entropy-decoded bitstream may correspond to quantized DCT coefficients. This entropy-decoded bitstream is input to the IDCT process 423, where an inverse DCT is applied to the entropy-decoded bitstream. The output of the IDCT process 423 is processed by a super-prior decoder neural network 424 to generate PDF parameters for a specific image (e.g., input image 400) to be decoded. The PDF parameters may include variance information associated with the super-prior, such as standard deviation, mean, etc.

[0061] The PDF parameters generated by the super-prior decoder neural network 424 are then used by the entropy encoding process 418 to encode the data stream produced by the image analysis neural network 402 after quantization in the quantization process 416. The entropy encoding process 418 creates a bit stream 2420. Bit stream 2420 is then entropy decoded by the entropy decoding process 422 using the same PDF parameters from the super-prior decoder neural network 424. After entropy decoding, the image synthesis neural network 426 processes the decoded data to produce an output image 428.

[0062] By applying the DCT process 405 to the output of the super-prior encoder neural network 404 (e.g., super-prior variables), the media encoder 200 can further compress the super-prior information compared to existing NVCs. The media encoder 200 and the media decoder 300 can use the corresponding IDCT process 423 to produce a quantized version of the output of the super-prior encoder neural network 404.

[0063] The image analysis neural network 402, the super-prior encoder neural network 404, the super-prior decoder neural network 424, and / or the image synthesis neural network 426, or parts thereof, may be similar to... Figure 7A and Figure 7B The example architecture described in the document is used for implementation.

[0064] In some examples, an NVC (such as a media encoder 200) can encode media data (e.g., an input image 400). The media encoder 200 can determine the hyperprior variables of the NVC (e.g., a media decoder 200) based on the media data. For example, the hyperprior variables could be the output of a hyperprior encoder neural network 404. The media encoder 200 can apply a DCT (e.g., DCT 405) to the hyperprior variables to generate DCT coefficients. The media encoder 200 can encode the DCT coefficients, for example, by applying quantization 406 and entropy coding 408.

[0065] In some examples, an NVC (such as media decoder 300) can decode media data (e.g., encoded media data from bitstream 1 410). Media decoder 300 can determine the DCT coefficients of the hyperprior variables of the NVC (e.g., media encoder 200). For example, media decoder 300 can apply entropy decoding 412 to bitstream 1 410 to determine the DCT coefficients. Media decoder 300 can apply inverse DCT 423 to the DCT coefficients to generate hyperprior variables. Media decoder 300 can decode media data based on hyperprior variables. For example, media decoder 300 can utilize a hyperprior decoder neural network 424, entropy decoding 422, and an image synthesis neural network 426 (or a similar structure) to process the hyperprior variables to generate an output image 428.

[0066] The media encoder 200 determines the prior variables of the NVC (e.g., media decoder 200) based on media data. For example, the prior variables could be the output of a prior encoder neural network 404. The media encoder 200 can apply a DCT (e.g., DCT 405) to the prior variables to generate DCT coefficients. The media encoder 200 can encode the DCT coefficients, for example, by applying quantization 406 and entropy coding 408.

[0067] Conventional video decoders (such as those described in the VVC and HEVC standards (Bross et al., 2021; Sullivan et al., 2012)) are based on predictive decoding frameworks and have long been popular for lossy video compression. After over 30 years of development by the standards community, such decoders have demonstrated improved efficiency in storing and transmitting data compared to previous decoders, and have likely supported significant economic growth. However, despite the success of such conventional approaches (which may be limited to handcrafted linear submodules), improvements have slowed, and the gains have diminished.

[0068] In the search for improved performance, interest has grown in the emerging field of neural video decoders (NVCs) (Lu et al., 2019; Agustsson et al., 2020; Hu et al., 2021; Li et al., 2021; Rippel et al., 2021; Li et al., 2022a; Mentzer et al., 2022). Media encoder 200 and media decoder 300 ( Figure 1 This can be an example of NVC. In the case of NVC, the technology can utilize differentiable neural operations to replace (or completely redesign) handcrafted submodules, while training the decoder end-to-end. However, despite offering flexibility, NVC has so far struggled to produce competitive decoding results at real-world decoding speeds. Nevertheless, progress has been relatively rapid, and many of the unresolved issues inherent in these NVC approaches remain unanswered.

[0069] This disclosure describes techniques related to the use of decoded information. Unlike the predictive decoding paradigm of conventional methods, there is no consensus among NVC practitioners on a unified approach to using decoded information. Example methods can be broadly categorized into several frameworks that define the use of past information: predictive decoding (where the decoder decodes the residual between the motion-compensated frame and the target frame) (Lu et al., 2019; Agustsson et al., 2020; Hu et al., 2021); conditional decoding (where the decoder adjusts the target frame based on motion-compensated information) (Li et al., 2022a; 2021; Ladune et al., 2021; Ladune and Philippe, 2022); and techniques where the decoder does not use explicit motion compensation (such as 3D autoencoders) (Habibian et al., 2019; Pessoa et al., 2020).

[0070] This disclosure also describes techniques related to minimizing or reducing the accumulation of temporal errors. Unlike conventional methods, it is uncommon for NVCs to perform test-time decoding using rate distortion optimization (RDO) (Sullivan and Wiegand, 1998). Conventional decoders can allocate bits across groups of pictures (GOPs) via a search process to maintain frame quality and minimize bit rate. Instead of this allocation, most NVCs will experience quality degradation across large GOP sizes, which the decoder cannot train on due to time and memory constraints. While there are some interim solutions to this problem (Chan et al., 2021; Sheng et al., 2022), standard methods still insert high-bit still images (e.g., I-frames) at a higher frequency than the low-latency mode of conventional decoders. This can be a significant disadvantage when compared to conventional decoders. For example, Sheng et al. (2022) pointed out that in HEVC, moving the GOP from 12 (common to NVC) to 32 improved the Bjontegaard-delta (BD) rate (Bjontegaard, 2001) by an average of 23.8%.

[0071] According to the techniques disclosed herein, the conditional decoding framework can operate in an LDP (Low-Latency P-Frame) configuration (e.g., using predictions from a single reference). This technique is supported by information theory results that the residual (Shannon) entropy is greater than or equal to the conditional entropy (Ladune et al., 2021):

[0072] ,

[0073] in and These are the current frame and the motion-compensated frame, respectively. Unlike conventional decoding techniques, neural networks freely explore complex nonlocal pixel dependencies and learn arbitrarily complex conditional relationships through a nonlinear estimator. Therefore, NVC can utilize conditional decoding (Li et al., 2022a).

[0074] According to the technology disclosed herein, the media encoder 200 or media decoder 300 can apply conditional decoding paradigms to motion estimation. The media encoder 200 or media decoder 300 can also apply buffering concepts from conventional decoders to model long-range dependencies in motion estimation, thereby mitigating visual challenges such as occlusion.

[0075] According to the techniques disclosed herein, the media encoder 200 or media decoder 300 can mitigate or avoid the propagation of timing errors inherent in low-latency NVCs in other ways. For example, as with conventional decoders, such a media encoder 200 or media decoder 300 can achieve superior results (when compared with other NVCs) over an “unlimited” period of time across standard verification tasks, such as one I-frame followed by all P-frames.

[0076] This disclosure discusses variations of the probabilistic model of autoencoders, which may otherwise be prevalent in image and video decoders.

[0077] This disclosure discusses the analysis of common evaluation protocols discussed in the literature and provides a basis for encouraging future comparisons with conventional methods that have weighted YUV-PSNR metrics.

[0078] According to the techniques disclosed herein, the resulting NVC decoder (e.g., media encoder 200 or media decoder 300) can achieve improved results when compared with other NVC decoders. Such an NVC decoder may be able to outperform the VVC test model (VTM) in YUV-PSNR using the VTM Common Test Condition (CTC) (Bossen et al., 2022).

[0079] Now let's discuss related work. Some examples of existing work on neural image compression are based on mean-scale hyperprior (Ball'e et al., 2018). These methods are broadly considered as variational autoencoders (VAEs), most often assuming a two-level hierarchical structure of latent variables, such that the rate distortion target is given by the following equation.

[0080]

[0081] in and These are the first and second level latent variables quantified, respectively, and It's a reconstruction. Here. and These are the so-called analyzer, super-analyzer, synthesizer, and super-synthesizer functions. Rate estimation. With distortion function The balance between them is achieved through Lagrange multipliers accomplish.

[0082] Extending this formula, the method considers improvements in rate estimation using autoregressive models (Minnen et al., 2018; He et al., 2021; 2022a; Minnen and Singh, 2020; Koyuncu et al., 2022; Lee et al., 2018), more complex transformations (Cheng et al., 2020; Zhu et al., 2021), and improved gradient estimators using non-differentiable quantization operators (Agustsson and Theis, 2020; Guo et al., 2021b). Such advances have yielded significant results, with many techniques outperforming VVC in RGB-PSNR and MS-SSIM (Wang et al., 2003).

[0083] With the success of image compression, many have applied these techniques to video. Most research here focuses on low-latency (LD) configurations: using a time-preceding reference frame to decode the current frame (e.g., a P-frame). Unlike random access decoders, LD codecs can decode video data in temporally consecutive frame order (F0, F1, F2…). It should be noted that there is a distinction between LDP (Low-Latency P-frames) and LDB (Low-Latency B-frames): for a given pixel in the current frame, B can use any number of previous frames; P can use only one. As an alternative to LD, random access (RA) configurations use both past and future frames for free decoding.

[0084] Causal inference structures for RA, best suited for applications without latency constraints (e.g., streaming services), typically reduce BD rates by about 30%. While several decoders extend NVC to RA using frame interpolation and bidirectional motion estimation (Pourreza and Cohen, 2021; Wu et al., 2018; Djelouah et al., 2019; Yang et al., 2020), research on these techniques remains relatively limited compared to LD NVC.

[0085] In LD, most work employs the conventional approach of predictive decoding (Rippel et al., 2021; Agustsson et al., 2020; Hu et al., 2021; 2020; Lin et al., 2020). In such cases, compressed motion information can be used to derive motion-compensated frames. Methods for motion estimation include enhancing optical flow prediction with Gaussian blur (Agustsson et al., 2020), learning offset maps for deformable convolution (Hu et al., 2021), etc. Given a motion-compensated frame, a decoder using such techniques decodes the sparse difference between the motion-compensated frame and the target frame. Recently, several proposals have extended the predictive decoding framework to conditional decoding (Li et al., 2022a; 2021; Ladune et al., 2021; Ladune and Philippe, 2022). In contrast to decoding residuals, decoders differ significantly in their use of motion, directly decoding the target frame conditional on the motion-compensated decoded information. In practice, this regulation addresses the cascading of conditional information into both analytical and synthetic transformations. As discussed, this result can leverage many advantages of neural formulas.

[0086] Beyond the inductive bias of explicit motion estimation, several researchers have considered 3D autoencoders (Habibian et al., 2019; Pessoa et al., 2020). As a natural extension of image compression, such techniques expand the input dimension to account for convolutions in both space and time. Recently, Mentzer et al. (2022) proposed an enhanced image decoder to include a transformer-based entropy model conditioned on latent variables from previous decoding. Assuming independence between frame reconstructions, this approach avoids quality degradation while remaining competitive with predictive decoding techniques (e.g., computationally competitive).

[0087] Finally, neural representation techniques, which are not suitable for low-latency applications, have emerged as potential alternatives with relatively fast decoding speeds (Lee et al., 2022; Chen et al., 2021; Li et al., 2022b). While such decoding speeds are relatively fast, current results are not competitive with "state-of-the-art" conventional decoders, and the cost of transmitting network weights across sequences is only effective when observing the entire sequence, which is not necessarily the case.

[0088] Figure 4 This is a conceptual diagram illustrating an example of adaptive reference selection. Figure 4In the example, one or more hard masks are used to select regions from the previous reconstruction (see panels 450 and 460) to serve as the basis for prediction, such as for predicting frame 470. The hard masks in panels 450 and 460 are represented as black areas covering portions of the panels. Such hard masks can be used to isolate hard samples representing specific types, such as faces.

[0089] The compression backbone of the media encoder 200 and / or media decoder 300 is now described. According to the techniques of this disclosure, the NVC (such as the media encoder 200 and / or media decoder 300) may use three autoencoder models: one for I-frames and two for P-frames. The P-frame model may include (conditional) motion estimation and conditional decoder modules. The compression backbone of each model may be based on a mean-scale super-prior. However, since the performance of such methods may not be comparable to other conventional decoders, the media encoder 200 and / or media decoder 300 may apply extensions to the model to achieve competitive performance.

[0090] Now, contextual modeling is described. Full decomposition constraints beyond those of earlier entropy modeling remain important for producing competitive results. Unlike the earliest attempts (Minnen et al., 2018), such extensions should not significantly impact time complexity. To avoid sacrificing time complexity, some have considered group-based autoregressive models, such as chessboard models and channel-by-channel models (He et al., 2021; Minnen and Singh, 2020). He et al. (2022a) proposed a non-uniform channel-by-channel grouping strategy.

[0091] According to the technology disclosed herein, the media encoder 200 or media decoder 300 may employ a non-uniform spatial subsampling chessboard. For example, the media encoder 200 and / or media decoder 300 may divide the latent variables into three groups: the first two groups each decode 25% of the data; and the last group decodes the remaining 50%. The media encoder 200 and / or media decoder 300 may construct the chessboard by stacking shifted copies of a 4x4 mask. In this manner, the autoregressive approach alternately accesses the spatial locations between channels, generating a complete spatial context at each autoregressive step. Figure 5 The diagram depicts this type of causal structure.

[0092] Figure 5 This is a conceptual diagram illustrating an example causal inference structure for an uneven chessboard context module. Figure 5 In the example, the causal inference structure proceeds from left to right. Figure 5 In this model, there are three pixel representations. A shaded pixel, such as one marked 500, represents a pixel to be inferred. A shaded pixel, such as one marked 502, represents a decoded pixel. Transparent pixels, such as one marked 504, represent unusable pixels.

[0093] Now we will discuss applying DCT to the prior. Although powerful density estimation techniques are used for... It is very common, but beyond priori The same flexibility is not typically offered. Following Ballé et al. (2018), most NVCs utilize their nonparametric methods for univariate density estimation. Because they “have no prior beliefs about the super-prior,” these methods attempt to amortize across channels. This learning avoids the expression of spatial and / or frequency information.

[0094] Figure 6 is an example of y, and after DCT transformation A conceptual diagram of an example channel. y 600 could be... Figure 3 Example of the output of image analysis neural network 402 of A. Figure 6 The z not shown can be a priori variable, for example, the output of the priori encoder neural network 404. If the priori variable z (e.g., the output of the priori encoder neural network 404) is not subjected to the DCT process 405, but is directly fed into the quantization process 406 (all... Figure 3 A), then 602 can be an example of the output of quantization process 406. After DCT transformation 604 can be Figure 3 Example of the output of DCT procedure 405 of A (e.g., a super-prior variable transformed by DCT).

[0095] Figure 6 The visual inspection revealed 602 retains much of the structure of the input y = 600. In this case, there exists a strong basis for forming a priori beliefs: The statistics of 602 are no different from those of natural images. To take advantage of this similarity, the media encoder 200 and / or media decoder 300 can use discrete cosine transform (DCT) to model the hyperprior. For example, the media encoder 200 and / or media decoder 300 can use channel-wise DCT coefficients based on... Parametric mixing models of blocks (e.g., after DCT transformation) (604), the media encoder 200 or media decoder 300 can quantize and entropy decode the DCT coefficients. Compared to earlier methods, the media encoder 200 and / or media decoder 300 allow for the description of frequency information without relying on the underlying representation. This technique improves performance while minimizing inference complexity, as discussed further below.

[0096] In some examples, the media encoder 200 and media decoder 300 can be symmetric encoders and decoders and can include multiple residual blocks, such as five or six residual blocks, while omitting popular concepts such as non-local concern blocks. The media encoder 200 and / or media decoder 300 can follow the approach of Li et al. (2022a) and utilize multi-granularity quantization. While their work focuses on the quantization interval size learned in multi-rate models, their work improves rate distortion performance in single-rate models. Therefore, in addition to predicting the mean and variance... and In addition, the media encoder 200 and / or media decoder 300 can also predict This can be used to provide adaptive scaling of the content of variables before and after quantization. For example, a media encoder 200 and a media decoder 300 can be built from a symmetric architecture of 5 residual blocks with 192 channels. For example, a supernetwork can be built from 2 residual blocks, each with 256 channels. A kernel size of 3 can be used anywhere, and upsampling can be accomplished using transposed convolutions.

[0097] For example, compared to a typical 4th quantization, such as by quantizing 416, the first-level latent variable y can be downsampled a total of 3 times. For example, this choice, combined with content-adaptive quantization interval size, can present a good balance between utilizing spatial redundancy and preserving high-frequency information.

[0098] The application of such a compressed backbone to P-frame decoding, as discussed above, will now be discussed. As mentioned above, this disclosure discusses the use of decoded information and efforts to limit quality degradation. P-frames of the media encoder 200 and / or media decoder 300 can be processed roughly as follows. A pre-trained optical flow model (Ranjan and Black, 2017) estimates the motion vectors between the current frame and a reference frame. This motion vector can be used with a decoder buffer. Compress and rebuild The following section further discusses the extension of conditional decoding to motion vector compression and the use of decoder buffers. Media encoder 200 and / or media decoder 300 can warp reference features The warped reference features are used as conditional inputs to the conditional autoencoder. This returns a reconstructed version of the target frame. and characteristics .

[0099] Figure 7A This is a block diagram illustrating an example of a conditional P-frame model. Figure 7A In the example, the box labeled 700 represents the conditional encoder, the boxes labeled 702A and 702B represent the synthesis and analysis transforms, and the box labeled 704 represents the probability estimation and entropy decoding.

[0100] The conditional P-frame model 730 (which may be some or all of the examples of the image analysis neural network 402) may include a motion decoder 710 and a conditional decoder 720. The motion decoder 710 may include an autoencoder 730 and is capable of decoding motion associated with the input video sequence. For example, the motion decoder 710 may be based on a motion signal m t and input video sequence x t To encode motion. Input video sequence x t May include Figure 3 The input image is 400. The autoencoder 740 can generate a quantized version of the input video sequence. .

[0101] Box 706 allows the Gubel-Softmax function to be applied from the decoder buffer. The sample. For example, in low-latency decoding, from the previously decoded frame decoder buffer. A reference frame is selected for prediction. In low-latency P, typically only one reference block / pixel / frame can be selected for prediction. To allow the NVC to select one reference block from many reference blocks, the NVC generates a sampling probability for each pixel. For example, if the NVC decodes a frame at time t=4, then the reference buffer (e.g., the decoder buffer) has decoded frames at times t=1, 2, 3. The NVC can output the selection probability for each pixel in each of those potential reference frames from the reference buffer. For example, for any pixel in any color channel, P can be equal to [0.2, 0.3, 0.5], e.g., a 50% probability NVC should include t=3, a 30% probability NVC should include t=2, etc. However, because the NVC is a low-latency NVC, the NVC can select only one of three. Therefore, the NVC can one-hot probabilities transform the P vector (e.g., one bit is high (1) and all other bits are low (0)). To achieve this, the NVC can apply the Gumbel-Softmax function. Therefore, the NVC determines the maximum value of the probability vector in a differentiable manner. In the example of P=[0.2, 0.3, 0.5], the output of box 706 (applying the Gubel-Softmax function) will be gumbel_softmax(p) = [0, 0, 1]. In this case, the NVC can choose a reference frame t=3 for that specific location in the final reference frame.

[0102] The same architecture can be used for both P-frame and I-frame conditional autoencoders (e.g., media encoder 200 and media decoder 300). This technique will be discussed further later in this disclosure. The conditional encoder (e.g., conditional decoder 720) may include small residual blocks and can generate a spatial pyramid of features for cascading to both the analysis and synthesis transforms. Three features (or "context") can be extracted: Highest resolution feature The output can be combined with the output of the conditional synthesis transform via U-Net to produce the final output. In some examples, the autoencoder (such as motion decoder 710) according to the techniques of this disclosure may not include U-Net. Such conditional architectures may include a simplification of Li et al. (2022a), which discloses W-Net and allows downsampled motion vectors to warp each feature scale independently. In the method of Li et al. (2022a), the warped features are updated using a top-down “feature fusion” network, from renew And from renew ,in It is a scale Warp features at the location. In some examples, the media encoder 200 and / or media decoder 300 may not utilize or avoid this update of warp features using a top-down feature fusion network.

[0103] Figure 7B This is a block diagram of an example architecture for conditional analysis and synthetic transformations according to one or more aspects of this disclosure. For example, conditional analysis transformation 750 (which may be...) Figure 7A The conditional analysis transform 702A example can obtain inputs, such as input video sequence x. t. And generate an output (e.g., latent variables), which can be fed into a hyperprior encoder neural network 404 ( Figure 3 The conditional analysis transform 750 can also send its output to an entropy decoder 756 for entropy decoding. In this case, the entropy decoder 756 can be entropy-coded 418 (…). Figure 3 Examples of ). It could be Figure 7A The conditional encoder 752, an example of probability estimation and entropy decoding 704, obtains a conditional input (e.g., a motion vector) and outputs the conditionally decoded data to the entropy decoder 756. The conditional analysis transform 754 obtains the output of the entropy decoder 756 and outputs it to, for example, a U-Net 758. The U-Net 758 generates an output, such as a quantized version of the input video sequence. .

[0104] Now we discuss conditional motion estimation. While there may be temporal redundancy across frames, there are many types of motion, which in turn adds temporal redundancy across motion vectors. This observation has been successfully utilized in previous work—methods for predicting motion vectors before decoding sparse updates have been solved (Rippel et al., 2021; Pourreza et al., 2023). These techniques enhance motion prediction without transmitting additional bits.

[0105] Extending the theoretical advantages of conditional decoding, this concept can be applied to motion estimation. It can be noted that Ladune and Philippe (2022) apply a technique that conditions only the reference frame, rather than the reference motion vector. Here, the media encoder 200 and / or media decoder 300 can use a previously transmitted reference frame to derive the motion vector as a conditional input used by the motion autoencoder. That is, given a reference frame buffer... The media encoder 200 and / or media decoder 300 can use optical flow to re-estimate the vector. It can be noted that the motion vectors are reconstructed. With recalculation of motion vectors used for adjustment The differences between them are also noted. It can also be observed that using previously reconstructed data (e.g., the output of a conditional autoencoder) can make the evaluation of the entire large GOP size unstable. Furthermore, since training sequences of length that periodically fill multi-frame buffers during normal operation may be impractical, the media encoder 200 and / or media decoder 300 can use an I-frame model to initialize a pseudo-buffer for decoded P-frames. This method allows the media encoder 200 and / or media decoder 300 to be trained on relatively short sequences.

[0106] Now we discuss adaptive reference selection. Conventional decoders select reference blocks from the buffer of decoded frames for motion prediction (e.g., Wien (2015)). Such decoders select the best block (and its resulting motion prediction) from the buffer that will minimize the error between the motion-compensated frame and the current frame. This technique allows the decoder to model long-range dependencies while overcoming visual challenges such as occlusion.

[0107] Therefore, NVCs (e.g., media encoders 200 or media decoders 300) can use such buffers to model long-range dependencies. This technique can be termed Adaptive Reference Selection (ARS). That is, given a buffer for a reference frame and its associated motion vector, the media encoder 200 and / or media decoder 300 can return a pixel-wise tensor of sampling probabilities along with the reconstructed motion vector, along with a warped skip pattern. Using the Gumbel-Softmax operator, the media encoder 200 and / or media decoder 300 can sample from the reference buffer using this probability tensor before warping the result using the decoded motion vector. Finally, the media encoder 200 and / or media decoder 300 can use the skip pattern as a cheap (e.g., computationally inexpensive) mechanism to decode identity transformations between the current frame and any reference frame in the buffer. For example, such motion can be defined as…

[0108]

[0109] in It is motion prediction. It is a reference frame sampled using the Gumbel-Softmax operation. It is a jump mode and It is a bilinear warping function.

[0110] Figure 8 This is a conceptual diagram illustrating an example of a three-frame validation sequence obtained during training. It depicts the three-frame reconstruction of the training and validation sequence. Given a relatively low-quality I-frame at t = 0 (e.g., a corrupted I-frame), the video model can output progressively improved quality reconstructions of frames 800, 802, and 804. Embedded elements 810, 812, and 814 are provided to show magnified versions containing foreground regions with people. The results of ARS can be buffered using two frames... Figure 8 As seen in the example, the media encoder 200 and / or media decoder 300 may select only pixels exhibiting a relatively large amount of motion (e.g., containing foreground regions of people) from the most recent reference frame.

[0111] Now we discuss minimizing the propagation of temporal errors. Practical constraints of time and computational resources limit decoder training to ≤7 (seven) frames. Therefore, one of the fundamental problems specific to NVC is the accumulation of temporal errors over large test-time GOP sizes (which can include hundreds of frames), a size on which NVC is practically untrainable. The most popular approach to addressing this problem is to use “cascaded” loss (Chan et al., 2021; Sheng et al., 2022). For cascaded loss, the training system can include a weighted objective where the importance of distortion increases with the depth of the training sequence. This technique serves a dual purpose: penalizing quality degradation and preventing disproportionate evaluation of early frames (e.g., early frames are implicitly evaluated in all future frames). For example, a model could use:

[0112]

[0113] Where R is the rate (e.g., bit rate), D is the distortion (such as mean square error, PSNR, etc.), and T is the time step. It is a constant that monotonically increases with the time step as the GOP is trained.

[0114] However, while the modified target achieved success in latency degradation, the fundamental problem remains: the model struggles to infer the hand-designed rules inherent in conventional decoders to maintain quality across long frames or picture sequences. In fact, even the best such methods still use one I-frame every 32 frames, which is unusual for conventional LD ​​methods. For example, inserting an I-frame into the bitstream every 32 frames significantly reduces decoding efficiency. I-frames tend to be much more expensive than P-frames in terms of bandwidth because an I-frame can include all the information within a video data frame, rather than the incremental information between the current frame and another. In conventional decoders (such as VVC decoders), a relatively large number of P-frames (e.g., 500 to 600 sequentially) can be transmitted without any intervening I-frames when there is no scene change. Therefore, in terms of bandwidth efficiency, NVC is simply not competitive with conventional codecs in transmitting one I-frame every 32 frames during normal operation.

[0115] Therefore, it may be desirable to provide an NVC that does not require relatively frequent transmission of I-frames to maintain quality. According to the techniques disclosed herein, the system can be trained with a P-frame model that learns to permanently improve image quality and mimics the removal of temporal errors across the entire large GOP size. For example, simply penalizing temporal errors during training may not be sufficient; training should lead the NVC to learn to self-correct those temporal errors that the NVC otherwise tends to create. For example, the media encoder 200 and / or media decoder 300 can be trained to self-correct temporal errors. Several documents discuss self-supervised techniques (He et al., 2022b; Grill et al., 2020).

[0116] In some examples, a model-general training technique (which may be referred to herein as Forced Quality Enhancement (FQE)) can be used to train the media encoder 200 and / or the media decoder 300. FQE can be operated by pre-training a P-frame model using I-frames that have undergone some corruption. This corruption can be inherent to the generated I-frames, applied after the I-frames are generated, or both. For example, a corrupted I-frame could be an I-frame generated with a PSNR that is lower (e.g., significantly lower) than the PSNR associated with the Lagrange multipliers used in the P-frame model and / or an I-frame that can be subjected to random occlusion to introduce corruption (or further corruption).

[0117] For example, the media encoder 200 and / or media decoder 300 can use corrupted I-frames for training to accumulate improved reconstructions across subsequent training frames, thereby reducing or minimizing the impact of object-to-object errors on the object itself. While various techniques, including random occlusion, can effectively corrupt I-frames, to best simulate NVC artifacts, it would mean utilizing I-frames configured with significantly lower PSNRs than the P-frame model (if the I-frame model will be included in end-to-end training). For example, if the P-frame model is configured to attempt to maintain a 34 dB PSNR at 1000 kbps, the I-frame could be configured to have a 32 dB PSNR at 500 kbps. As discussed above, Lagrange multipliers can be used to balance rate and distortion. For example, if the I-frame model is included in end-to-end training, corrupted I-frames could be configured for 0.25 times the achieved bit rate. For example, the P-frame model of media encoder 200 and / or media decoder 300 can be trained using corrupted I-frames, such as I-frames configured for PSNR below (or below or equal to) a threshold. This threshold may be based on a Lagrange multiplier associated with the P-frame model. In some examples, the threshold may be, for example, as per [the relevant context]. Figure 8 The described 3-frame training sequence can be used.

[0118] Additionally, the system can train the media encoder 200 and / or the media decoder 300 to learn one of the hand-designed biases of conventional decoders: P-frames oscillate at bit rates (Bossen et al., 2022). For example, the decoder can initiate a chain of P-frame pairs with high bit rates followed by low bit rates.

[0119] Figure 9 This is a graph illustrating example peak signal-to-noise ratio (PSNR) and bits per pixel across 125 evaluation frames with and without forced quality enhancement (FQE). Figure 9The examples depict the results of applying FQE (900 and 910) to a so-called infinite GOP across the verification sequence compared to not applying FQE (902 and 912). As shown, using FQE results in a more reliable PSNR and fewer bits per pixel for oscillations compared to not using FQE. The learned robustness to bad reference frames and missing data will be discussed further later in this disclosure.

[0120] The techniques used to evaluate neural decoders are now discussed. In neural image and video compression, a common evaluation compares rate-distortion performance to conventional decoders such as HEVC and VVC. In this typical evaluation, two commonly used distortion metrics exist: RGB-PSNR and MS-SSIM. The latter acts as a proxy for optimizing the decoder with regard to perceptual loss (Zhang et al., 2018; Blau and Michaeli, 2018; Mentzer et al., 2020). Given the flexibility of NVC to optimize for any differentiable objective, this is an important proof of NVC's strength.

[0121] This disclosure focuses more on the former: the “classic” rate-distortion performance as defined in RGB-PSNR. This is a metric in which many NVC performances described in the paper outperform HEVC, and more recently in VVC (Li et al., 2022a). However, this metric can harm conventional baselines. Therefore, future evaluations for conventional decoders may be better done based on weighted YUV-PSNR (the objective function guiding the development of video standards organizations).

[0122] Now we will discuss evaluating decoders in RGB. The conventional techniques used to evaluate decoders are not specifically designed for RGB. Instead, these techniques are used in YUV—often subsampling. 420 Operations are performed in color spaces. (Subsampled color spaces improve complexity by leveraging human perception.) To evaluate decoders in RGB, a common practice is to perform operations down to YUV. 444 The transformation and compression results are obtained, and an inverse transformation of RGB is performed before evaluating RGB-PSNR.

[0123] The lack of chroma subsampling and transform operations makes accurate comparisons difficult, but this may not be the primary issue. In fact, conventional decoders, such as MS-SSIM, are not optimized for RGB-PSNR. That is, decades of development have addressed the de facto loss in YUV-PSNR, with channel-wise weights ranging from approximately 6:1:1 to 8:1:1. In contrast, RGB-PSNR is roughly equivalent to YUV-PSNR 1:1:1 (Ström et al., 2020). This observation can have significant implications for rate distortion performance. In particular, the Y channel has the highest entropy and is therefore the most difficult or costly to compress. Consequently, conventional decoders have long suffered from a tradeoff of approximately 1:10 in BD rate between Y and UV (Bossen, 2019; Francois et al., 2022): to achieve a 1% BD rate in Y (relative to some references), the decoder may sacrifice 10% in UV.

[0124] Table 1. BD rate in YUV on I-frame models trained for RGB-PSNR.

[0125]

[0126] To better understand the implications of this observation, two points can be considered. First, when the NVC outperforms the conventional baseline in the RGB color space, will this also be the result in the YUV color space? Second, to what extent is RGB performance transferable to YUV—for example, are there similar Y:UV BD rate tradeoffs? For the first question, the public code of Li et al. (2022a) can be evaluated, which may be the only NVC to report excellent performance of the VTM. To obtain YUV results, the decoder can be evaluated across the JVET CTC. Temporary RGB-to-YUV conversions and vice versa can be adopted. For the second question, (e.g., media encoder 200 and / or media decoder 300) I-frame models can be retrained across three YUV-PSNR targets: 1:1:1, 4:1:1, and 8:1:1. The resulting model can be evaluated on the popular Kodak dataset (Franzen, 1999) using the RGB-trained model as a reference.

[0127] Table 2. VTM-16.0 LDP with superior performance compared to Li et al. (2022a).

[0128]

[0129] Table 2 shows that the excellent performance of Li et al. (2022a) at 1:1:1 (or RGB) comes from UV. A similar Y:UV tradeoff of approximately 1:10 can be observed in Table 2. While the latter may not generalize to the results of Li et al. (2022a) or other NVCs, these results demonstrate that methods optimized for weighted YUV targets are most likely at a disadvantage in RGB-PSNR. It can also be noted that this difference is broadly consistent with the results presented by Pourreza et al. (2023), which observed weaker comparative performance when using similar distortion targets in YUV.

[0130] While perceptual quality may ultimately be what matters (where 6:1:1 PSNR-YUV is itself a perceptual choice), RGB comparisons can achieve roughly the same effect as MS-SSIM: demonstrating that the flexibility of NVC to train for any differentiable target can achieve excellent visual quality without winning over YUV, but using RGB-PSNR does not seem to be a fair comparison of decoders when evaluating “classic” rate distortion.

[0131] The test results are now discussed. The media encoder 200 and media decoder 300 disclosed herein were trained on the Vimeo-90k dataset (Xue et al., 2019). Training followed the multi-stage training formula proposed by Sheng et al. (2022). For FQE, a frozen I-frame model objective was used, approximately 0.25 times the bit rate implemented in the training case. For the final training step after FQE, the I-frame model was unfrozen and the decoders (e.g., media encoder 200 and / or media decoder 300) were trained end-to-end. For evaluation purposes, only one I-frame was used, where all subsequent frames were P-frames.

[0132] The conventional decoder is omitted from the PSNR-RGB comparison, and its performance may be surpassed by using YUV as an alternative. 420 The JVET CTC in the test was compared separately. Multiple techniques were proposed to be compared with YUV. 420 Specialized networks that work together with inputs (Egilmez et al., 2021; Pourreza et al., 2023; Ho et al., 2022). However, special processing may not offer any benefit. Surprisingly, employing the simplest technique of repeatedly upsampling the UVs before applying RGB transformation appears advantageous. Such transformation normalizes the marginal distribution across channels, making network training easier (this will be discussed later in this paper). The YUV model is trained using a 6:1:1 YUV-PSNR distortion target.

[0133] Table 3. Ablation results of different model proposals. BD rate was calculated using UVG.

[0134]

[0135] Table 3 above shows the ablation results. FQW easily exhibits the largest single improvement, and FQW is widely applicable to any LD NVC.

[0136] To compare the media encoder 200 and / or media decoder 300 with other NVCs, video compression was evaluated in RGB on popular benchmarks MCL-JCV and UVG (Mercat et al., 2020; Wang et al., 2016). For this evaluation, the media encoder 200 and / or media decoder 300 were compared with (Hu et al., 2021), ELF-VC (Rippel et al., 2021), DCVC (Li et al., 2021), C2F (Hu et al., 2022), VCT (Mentzer et al., 2022), SSF (Agustsson et al., 2020), and Li et al. (Li et al., 2022a). It can be noted that Li et al. only evaluated the first 96 frames of each sequence, which may make the comparison slightly unreliable.

[0137] Figure 10 This is a graph illustrating example RGB rate distortion curves for the UVG and MCL-JCV datasets. Figure 10 The test results are depicted and demonstrate the superior performance of each of the existing methods. The curves labeled 1000 and 1002 represent the test results of the media encoder 200 and / or media decoder 300 employing the techniques of this disclosure.

[0138] As discussed above, the media encoder 200 and / or media decoder 300 are related to YUV... 420 The media encoder 200 and / or media decoder 300 were compared against conventional standards. For this comparison, HEVC and VVC test models (HM, VTM) were used and the JVET CTC was employed for evaluation. The test results are available in Tables 5 and 6. Note the challenges posed by category E, which consists mostly of static content. It is clear that the media encoder 200 and / or media decoder 300 struggle to compete with the complex jump patterns of conventional methods chosen during RDO (e.g., Wien (2015)). Note that Li et al. (2022a) also encountered this problem when evaluating using the public codebase of Li et al. (2022a) (see Table 2). Nevertheless, according to the techniques disclosed herein, the media encoder 200 and / or media decoder 300 are the first reported examples of decoders that outperform VTM LDP on most test sequences—and even surpass LDP performance on categories C and D.

[0139] Table 4. Performance is superior to HM-16.24 LDP.

[0140]

[0141] Table 5. Performance is superior to VTM-16.0 LDP.

[0142]

[0143] Decoding speed was also evaluated. A 1080p UVG sequence was used. Media encoder 200 and / or media decoder 300 ran on an NVIDIA RTX 5000 and results were compared to those obtained from available papers. Achieving competitive BD rate and decoding speed performance for NVC clearly remains a challenge. For example, while ELF-VC is an excellent approach in terms of FPS, it suffers from significant drawbacks in terms of BD.

[0144] Table 6. Differences in FPS and BD rate estimates across the UVG dataset. The BD rate is calculated with respect to the decoder of this disclosure.

[0145]

[0146] As described in this article, NVCs (such as the Media Encoder 200 and Media Decoder 300) can be used with YUV. 420 It is competitive with the most advanced conventional methods in China.

[0147] The media encoder 200 and media decoder 300 can be built using a symmetric architecture of five residual blocks with 192 channels each. For example, the supernetwork can be built using two residual blocks, each with 256 channels. A kernel size of three can be used anywhere, and upsampling can be achieved using transposed convolutions.

[0148] Compared to a typical 4th order, the first-order latent variables It can be downsampled a total of 3 times. For example, this option, combined with the content-adaptive quantization interval size, can present a good balance between utilizing spatial redundancy and preserving high-frequency information.

[0149] To demonstrate the robustness of learning from the FEQ, the impact of poor reference frames can be evaluated. In extreme cases, this can include the impact of missing reference frames. For example, the media encoder 200 and / or media decoder 300 can be compared with Li et al. (2022a). For each decoder, the preceding 32 frames of the UVG sequence HoneyBee can be evaluated at comparable bit rates with and without missing I-frames (e.g., assumed to be zero tensors). Such evaluations can produce results in the form of PSNR differences between decoded sequences with and without I-frames.

[0150] Figure 11This is a graph illustrating example RGB-PSNR differences between verification sequences with and without I-frames. (Example...) Figure 11 As depicted, the media encoder 200 and / or media decoder 300 (represented by line 1100) recover relatively quickly to within about 0.1 dB without inserting new I-frames. In contrast, the verification sequence of Li et al. (2022a) (represented by line 1102) is more adversely affected.

[0151] Figure 12 This is a graph illustrating an example of kernel density estimation across channels of the Kodak dataset in RGB and YUV formats. Figure 12 In the example, line 1200 represents B, line 1202 represents G, and line 1204 represents R. Furthermore, line 1206 represents U, line 1208 represents V, and line 1210 represents Y. Note the statistical differences between the channels in YUV compared to RGB, containing a great deal of redundancy. Given these differences, the affine transformation is resolved, and surprisingly, the YUV input transformed by RGB performs better in the tests. Most strangely, directly operating on the YUV input (using unlearned upsampling of UV in 420) is one of the lowest performing options. Note that previous work in the YUV space used separate network “headers”: one for Y and one for UV (Pourreza et al., 2023; Egilmez et al., 2021). However, more broadly, YUV… 420 It is designed to save computation in conventional decoders. Using a separate head might achieve the opposite effect. Furthermore, it's worth noting that the RGB conversion allows the use of pre-trained optical flow models, which simplifies training.

[0152] Figure 13 This is a graph illustrating example RGB rate distortion curves from the Kodak dataset. Figure 13 In the text, dots (such as dot 1300) represent data for media encoder 200 and / or media decoder 300 used to implement one or more technologies of this disclosure.

[0153] Figure 14 This is a block diagram illustrating an example system for training a P-frame model according to one or more aspects of this disclosure. Training system 1400 may include one or more processors 1412 coupled to one or more memories 1410. The one or more memories 1410 may store training input 1402, threshold 1440, media encoder 200, and / or media decoder 300. In some examples, training system 1400 may be... Figure 1The training system 1400 is part of the source device 102 and / or destination device 116. In some examples, the training system 1400 may be a separate system from the source device 102 and / or destination device 116. For example, the training system 1400 may be used to train the media encoder 200 and / or media decoder 300 before deploying the media encoder 200 and / or media decoder to the source device 102 and / or destination device 116, respectively.

[0154] Media encoder 200 may include P-frame model 1420 and media decoder 300 may include P-frame model 1530. In some examples, P-frame model 1420 and P-frame model 1430 include the same P-frame model.

[0155] Training input 1402 may include a corrupted I-frame 1404. The corrupted I-frame 1404 may include an I-frame with a PSNR that satisfies a threshold 1440 (e.g., below or equal to the threshold 1440) and / or include one or more insertion error regions. In some examples, the threshold 1440 may be based on Lagrange multipliers associated with P-frame models 1420 and / or 1430. In some examples, the corrupted I-frame 1404 may include intentionally inserted errors, such as one or more regions of blocked data. In other examples, the corrupted I-frame 1404 may include naturally occurring insertion errors.

[0156] In some examples, one or more processors 1412 may be configured to generate corrupted I-frames 1404. For example, one or more processors 1412 may be configured to generate I-frames that meet threshold 1440 and / or insert errors into one or more regions of the I-frame.

[0157] Training input 1402 may also include P-frame 1406. For example, P-frame 1406 may include multiple P-frames (e.g., up to seven P-frames) that can be used to train P-frame models 1420 and / or 1430. P-frame 1406 does not need to be a corrupted P-frame.

[0158] One or more processors 1412 may be configured to train P-frame models 1420 and / or 1430 using corrupted I-frames 1404 and P-frames 1406.

[0159] Figure 15 This is a flowchart illustrating an example technique for encoding media data according to one or more aspects of this disclosure. Media encoder 200 may determine the super-prior variables (1500) of the NVC based on the media data. For example, media encoder 200 may execute a super-prior encoder neural network 404 for the NVC (…). Figure 3A) Determine the prior variables based on the input image 400. The media encoder 200 may apply DCT to the prior variables to generate DCT coefficients (1502). For example, the media encoder 200 may apply DCT procedure 405 to the prior variables to generate DCT coefficients. The media encoder 200 may encode the DCT coefficients (1504). For example, as part of encoding media data, the media encoder 200 may quantize the DCT coefficients via quantization procedure 406 and entropy encode the DCT coefficients via entropy encoding 408.

[0160] In some examples, as part of encoding DCT coefficients, the media encoder 200 can quantize the channel-wise parametric mixing of the DCT coefficients to generate quantized DCT coefficients. In some examples, as part of encoding DCT coefficients, the media encoder 200 can entropy-encode the quantized DCT coefficients. In some examples, channel-wise parametric mixing includes channel-wise 4x4 parametric mixing.

[0161] In some examples, the prior variables include those of the NVC I-frame model. In some examples, the prior variables include those of the NVC P-frame model. In some examples, the P-frame model includes at least one of a motion estimation model or a conditional decoder model. In some examples, the P-frame model is trained using corrupted I-frames that have at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold, or b) one or more insertion error regions.

[0162] In some examples, the super-prior variables are based on latent variables. In some examples, the media encoder 200 may determine the latent variables, and as part of determining the latent variables, the media encoder 200 may use a non-uniformly spaced subsampling board to apply context modeling to the media data. In some examples, as part of using a non-uniformly spaced subsampling board, the media encoder 200 may stack shifted copies of an NxN mask. In some examples, the media data includes video data.

[0163] Figure 16 This is a flowchart illustrating an example technique for decoding media data according to one or more aspects of this disclosure. Media decoder 300 can determine the DCT coefficients (1600) of the prior variables of the NVC. For example, media decoder 300 can obtain the coefficients from a bitstream (such as bitstream 1410 ( Figure 3 A) Obtain the entropy-encoded DCT coefficients and perform entropy decoding on the entropy-encoded DCT coefficients (similar to...). Figure 3 The entropy decoding process 412 of A determines the DCT coefficients. The media decoder 300 can apply the inverse DCT to the DCT coefficients to generate super-prior variables (1602). For example, the media decoder 300 can perform a similar IDCT process 423 ( Figure 3The inverse DCT of A) is applied to the entropy-decoded DCT coefficients to determine the hyperprior variables. The media decoder 300 can decode media data based on the hyperprior variables (1604). For example, as part of decoding media data based on hyperprior variables, the media decoder 300 can apply a hyperprior decoder neural network, such as hyperprior decoder neural network 424 (…). Figure 3 A).

[0164] In some examples, the DCT coefficients include a channel-wise parametric blending of the quantized DCT coefficients generated by the neural media encoder. In some examples, the channel-wise parametric blending includes a channel-wise 4x4 parametric blending. In some examples, the media decoder 300 entropy-decodes the DCT coefficients as part of determining the DCT coefficients.

[0165] In some examples, the prior variables include those of the I-frame model of the neural video decoder. In some examples, the prior variables include those of the P-frame model of the neural video decoder. In some examples, the P-frame model includes at least one of a motion estimation model or a conditional decoder model. In some examples, the P-frame model is trained using corrupted I-frames that have at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold, or b) one or more insertion error regions.

[0166] In some examples, the super-prior variables are based on latent variables. In some examples, as part of decoding the media data, the media decoder 300 may determine the latent variables and use a non-uniformly spaced subsampling board to apply context modeling to the latent variables. In some examples, as part of using a non-uniformly spaced subsampling board, the media decoder 300 may stack shifted copies of an NxN mask.

[0167] In some examples, the media decoder 300 may divide the latent variable data into multiple groups. The latent variable data may include latent variables. In such examples, the media decoder 300 may use a first group of the multiple groups to decode a first percentage of the latent variable data. The media decoder 300 may use a second group of the multiple groups to decode a second percentage of the latent variable data. The media decoder 300 may use a third group of the multiple groups to decode a third percentage of the latent variable data. In some examples, the first and second percentages are lower than the third percentage. In some examples, the first percentage is 25%, the second percentage is 25%, and the third percentage is 50%.

[0168] In some examples, media data includes video data.

[0169] Figure 17This is a flowchart illustrating an example technique for training a P-frame model according to one or more aspects of this disclosure. One or more processors 1412 may acquire a corrupted I-frame having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold or b) one or more insertion error regions (1700). For example, one or more processors 1412 may retrieve a corrupted I-frame 1404 from one or more memories 1410. The corrupted I-frame 1404 may have a PSNR that meets a threshold 1440 and / or one or more insertion error regions.

[0170] One or more processors 1412 may use a corrupted I-frame to train a P-frame model (1702). For example, one or more processors 1412 may use a corrupted I-frame 1404 as input to train a P-frame model 1420 and / or a P-frame model 1430.

[0171] In some examples, one or more processors 1412 may generate a corrupted I-frame 1404. In some examples, the corrupted I-frame 1404 has a PSNR that meets a threshold, and wherein, in order to meet the threshold, the PSNR is below the threshold or below or equal to the threshold. In some examples, the threshold is based on a Lagrange multiplier associated with a P-frame model. In some examples, the corrupted I-frame is pre-configured to meet the threshold. In some examples, training the P-frame model includes training the P-frame model for seven or fewer frames. In some examples, training the P-frame model includes training the P-frame model for three frames.

[0172] Figure 18 This is a flowchart illustrating an example technique for decoding media data using a trained P-frame model, according to one or more aspects of this disclosure. A media encoder 200 or a media decoder 300 may apply a pre-trained P-frame model to the media data, the pre-trained P-frame model being trained using corrupted I-frames having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold or b) one or more insertion error regions (1800). For example, the media encoder 200 may include a P-frame model 1420, which may be pre-trained, for example, by one or more processors 1412 using a corrupted I-frame 1404. The corrupted I-frame 1404 may have a PSNR that meets a threshold and / or one or more insertion error regions.

[0173] Media encoder 200 or media decoder 300 may decode media data based on applying a pre-trained P-frame model to the media data (1802). For example, media encoder 200 may encode media data and media decoder 300 may use a pre-trained P-frame model to decode media data.

[0174] In some examples, the corrupted I-frame has a PSNR that meets a threshold, and in order to meet the threshold, the PSNR is below the threshold or below or equal to the threshold. In some examples, the media data includes video data. In some examples, decoding includes encoding. In some examples, decoding includes decoding.

[0175] The following is a list of references that may have been cited above.

[0176] Agustsson, E. and Theis, L. Universally quantized neural compression. Advances in neural information processing systems, 33:12367-12376, 2020.

[0177] Agustsson, E., Minnen, D., Johnston, N., Balle, J., Hwang, SJ, andToderici, G. Scale-space flow for end-to-end optimized video compression. InProceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition, pp. 8503-8512, 2020.

[0178] Ballé, J., Laparra, V., and Simoncelli, EP Density modeling ofimages using a generalized normalization transformation. arXiv preprintarXiv:1511.06281, 2015.

[0179] Ballé, J., Minnen, D., Singh, S., Hwang, SJ, and Johnston, .Variational image compression with a scale hyperprior. arXiv preprint arXiv:1802.01436, 2018.

[0180] Bjontegaard, G. Calculation of average PSNR differences between RD-curves. VCEG-M33, 2001.

[0181] Blau, Y. and Michaeli, T. The perception-distortion tradeoff. InProceedings of the IEEE conference on computer vision and patternrecognition, pp. 6228-6237, 2018.

[0182] Bossen, F. On reporting combined yuv bd rates. Joint Video ExpertsTeam (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC29, JVET-N0341, 2019.

[0183] Bossen, F., Li, X., Sharman, K., Seregin, V., and Sühring, K. VTM andHM common test conditions and software reference configurations for SDR 4:2:010-bit video. Joint Video Experts Team (JVET) of ITU-T SG and ISO / IECJTC 1 / SC29, JVET-Y2010, 2022.

[0184] Bross, B., Wang, Y.-K., Ye, Y., Liu, S., Chen, J., Sullivan, G. J.,and Ohm, J. R. Overview of the versatile video coding (VVC) standard and itsapplications. IEEE Transactions on Circuits and Systems for Video Technology,31(10):3736-3764, 2021. doi: 10.1109 / TCSVT.2021.3101953.

[0185] Chan, K. C., Wang, X., Yu, K., Dong, C., and Loy, C. C. Basicvsr: Thesearch for essential components in video super-resolution and beyond. InProceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition, pp. 4947-4956, 2021.

[0186] Chen, H., He, B., Wang, H., Ren, Y., Lim, S. N., and Shrivastava, A.NERV: Neural representations for videos. Advances in Neural InformationProcessing Systems, 34: 21557-21568, 2021.

[0187] Cheng, Z., Sun, H., Takeuchi, M., and Katto, J. Learned imagecompression with discretized gaussian mixture likelihoods and attentionmodules. In Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition, pp. 7939-7948, 2020. Djelouah, A., Campos, J., Schaub-Meyer, S., and Schroers, C. Neural inter-frame compression for video coding.In Proceedings of the IEEE / CVF International Conference on Computer Vision,pp. 6421-6429, 2019.

[0188] Duan, Q., Li, X., Yin, Q., Feng, L., Zhao, J., Teng, Y., Duan, X.,Zhao, Y., Gao, M., Wang, J., et al. A study on the generalized normalizationtransformation activation function in deep learning based image compression.In Proceedings of Sixth International Congress on Information andCommunication Technology, pp. 351-359. Springer, 2022.

[0189] Egilmez, H. E., Singh, A. K., Coban, M., Karczewicz, M., Zhu, Y.,Yang, Y., Said, A., and Cohen, T. S. Transform network architectures for deeplearning based end-to-end image / video coding in subsampled color spaces. IEEEOpen Journal of Signal Processing, 2:441–452, 2021. doi: 10.1109 / OJSP.2021.3092257.

[0190] Francois, E., Chen, Y., and Salmon-Legagneur, C. Adjusting luman / chroma BD-rate balance in ECM. Joint Video Experts Team (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC29, JVET-AC0138, 2022.

[0191] Franzen, R. Kodak lossless true color image suite. source: http: / / rOk.us / graphics / kodak, 4(2), 1999.

[0192] Granziol, D., Wan, X., Albanie, S., and Roberts, S. Iterativeaveraging in the quest for best test error. arXiv preprint arXiv:2003.01247,2020.

[0193] Granziol, D., Zohren, S., and Roberts, S. Learning rates as afunction of batch size: A random matrix theory approach to neural networktraining. J. Mach. Learn. Res, 23:1-65, 2022.

[0194] Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P.,Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M.,et al. Bootstrap your own latent-a new approach to self-supervised learning.Advances in neural information processing systems, 33:21271-21284, 2020.

[0195] Guo, Z., Zhang, Z., Feng, R., and Chen, Z. Causal contextualprediction for learned image compression. IEEE Transactions on Circuits andSystems for Video Technology, 32(4):2329-2341, 2021a.

[0196] Guo, Z., Zhang, Z., Feng, R., and Chen, Z. Soft then hard: Rethinkingthe quantization in neural image compression. In International Conference onMachine Learning, pp. 3920-3929. PMLR, 2021b.

[0197] Habibian, A., Rozendaal, T. v., Tomczak, J. M., and Cohen, T. S.Video compression with rate-distortion autoencoders. In Proceedings of theIEEE / CVF International Conference on Computer Vision, pp. 7033-7042, 2019.He, D., Zheng, Y., Sun, B., Wang, Y., and Qin, H. Checkerboard context modelfor efficient learned image compression. In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, pp. .

[0198] He, D., Yang, Z., Peng, W., Ma, R., Qin, H., and Wang, Y. Elic:Efficient learned image compression with unevenly grouped space-channelcontextual adaptive coding. In Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition, pp. 5718-5727, 2022a.

[0199] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R.Masked autoencoders are scalable vision learners. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009,2022b.

[0200] Ho, Y.-H., Lin, C.-H., Chen, P.-Y., Chen, M.-J., Chang, C.-P., Peng,W.-H., and Hang, H.-M. Learned video compression for YUV 4:2:0 content usingflow-based conditional interframe coding. In 2022 IEEE InternationalSymposium on Circuits and Systems (ISCAS), pp. 829-833. IEEE, 2022.

[0201] Hu, Z., Chen, Z., Xu, D., Lu, G., Ouyang, W., and Gu, S. Improvingdeep video compression by resolution-adaptive flow coding. In EuropeanConference on Computer Vision, pp. 193-209. Springer, 2020.

[0202] Hu, Z., Lu, G., and Xu, D. Fvc: A new framework towards deep videocompression in feature space. In Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition, pp. 1502-1511, 2021.

[0203] Hu, Z., Lu, G., Guo, J., Liu, S., Jiang, W., and Xu, D. Coarse-to-fine deep video coding with hyperprior-guided mode prediction. In Proceedingsof the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp.5921-5930, 2022.

[0204] Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deepnetwork training by reducing internal covariate shift. In Internationalconference on machine learning, pp. 448-456. PMLR, 2015.

[0205] Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson,A. G. Averaging weights leads to wider optima and better generalization.arXiv preprint arXiv:1803.05407, 2018.

[0206] Koyuncu, A. B., Gao, H., Boev, A., Gaikov, G., Alshina, E., andSteinbach, E. Contextformer: A transformer with spatio-channel attention forcontext modeling in learned image compression. In European Conference onComputer Vision, pp. 447-463. Springer, 2022.

[0207] Ladune, T. and Philippe, P. Aivc: Artificial intelligence based videocodec. arXiv preprint arXiv:2202.04365, 2022. Ladune, T., Philippe, P.,Hamidouche, W., Zhang, L., and Déforges, O. Conditional coding for flexiblelearned video compression. arXiv preprint arXiv:2104.07930, 2021.

[0208] Lee, J., Cho, S., and Beack, S.-K. Context-adaptive entropy model forend-to-end optimized image compression. arXiv preprint arXiv:1809.10452,2018.

[0209] Lee, J. C., Rho, D., Ko, J. H., and Park, E. FFNeRV: Flow-guidedframe-wise neural representations for videos. arXiv preprint arXiv:2212.12294, 2022.

[0210] Li, J., Li, B., and Lu, Y. Deep contextual video compression.Advances in Neural Information Processing Systems, 34: 18114-18125, 2021.

[0211] Li, J., Li, B., and Lu, Y. Hybrid spatial-temporal entropy modellingfor neural video compression. In Proceedings of the 30th ACM InternationalConference on Multimedia, pp. 1503-1511, 2022a.

[0212] Li, Y., Wei, C., and Ma, T. Towards explaining the regularizationeffect of initial large learning rate in training neural networks. Advancesin Neural Information Processing Systems, 32, 2019.

[0213] Li, Z., Wang, M., Pi, H., Xu, K., Mei, J., and Liu, Y. E-NeRV:Expedite neural video representation with disentangled spatial-temporalcontext. In European Conference on Computer Vision, pp. 267-284. Springer,2022b.

[0214] Lin, J., Liu, D., Li, H., and Wu, F. M-LVC: Multiple framesprediction for learned video compression. In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, pp. 3546-3554, 2020.

[0215] Lu, G., Ouyang, W., Xu, D., Zhang, X., Cai, C., and Gao, Z. Dvc: Anend-to-end deep video compression framework. In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, pp. 11006-11015, 2019.

[0216] Mentzer, F., Toderici, G. D., Tschannen, M., and Agustsson, E. High-fidelity generative image compression. Advances in Neural InformationProcessing Systems, 33: 11913-11924, 2020.

[0217] Mentzer, F., Toderici, G., Minnen, D., Hwang, S.-J., Caelles, S.,Lucic, M., and Agustsson, E. Vct: A video compression transformer. arXivpreprint arXiv:2206.07307, 2022.

[0218] Mercat, A., Viitanen, M., and Vanne, J. UVG dataset: fps sequences for video codec analysis and development. In Proceedings of the11th ACM Multimedia Systems Conference, pp. 297-302, 2020.

[0219] Minnen, D. and Singh, S. Channel-wise autoregressive entropy modelsfor learned image compression. In 2020 IEEE International Conference on ImageProcessing (ICIP), pp. 3339-3343. IEEE, 2020. Minnen, D., Ballé, J., andToderici, G. D. Joint autoregressive and hierarchical priors for learnedimage compression. Ad-vances in neural information processing systems, 31,2018.

[0220] Nesterov, Y. Introductory lectures on convex optimization: A basiccourse, volume 87. Springer Science & Business Media, 2003.

[0221] Papyan, V. The full spectrum of deepnet hessians at scale: Dynamicswith SGD training and sample size. arXiv preprint arXiv:1811.07062, 2018.

[0222] Pessoa, J., Aidos, H., Tomás, P., and Figueiredo, M. A. End-to-endlearning of video compression using spatiotemporal autoencoders. In 2020 IEEEWorkshop on Signal Processing Systems (SiPS), pp. 1-6. IEEE, 2020.

[0223] Pourreza, R. and Cohen, T. Extending neural P-frame codecs for B-frame coding. In Proceedings of the IEEE / CVF International Conference onComputer Vision, pp. 6680-6689, 2021.

[0224] Pourreza, R., Le, H., Said, A., Sautiere, G., and Wiggers, A.Boosting neural video codecs by exploiting hierarchical redundancy. InProceedings of the IEEE / CVF Winter Conference on Applications of ComputerVision, pp. 5355-5364, 2023.

[0225] Ranjan, A. and Black, M. J. Optical flow estimation using a spatialpyramid network. In Proceedings of the IEEE conference on computer vision andpattern recognition, pp. 4161-4170, 2017.

[0226] Rippel, O., Anderson, A. G., Tatwawadi, K., Nair, S., Lytle, C., andBourdev, L. Elf-vc: Efficient learned flexible-rate video coding. InProceedings of the IEEE / CVF International Conference on Computer Vision, pp.14479-14488, 2021.

[0227] Santurkar, S., Tsipras, D., Ilyas, A., and Madry, A. How does batchnormalization help optimization? Advances in neural information processingsystems, 31, 2018.

[0228] Sheng, X., Li, J., Li, B., Li, L., Liu, D., and Lu, Y. Temporalcontext mining for learned video compression. IEEE Transactions onMultimedia, 2022.

[0229] Ström, J., Andersson, K., Sjöberg, R., Segall, A., Bossen, F.,Sullivan, G., Ohm, J.-R., and Tourapis, A. Working practices using objectivemetrics for evaluation of video coding efficiency experiments (draft 4).Joint Video Experts Team (JVET) of ITU-TSG 16 WP 3 and ISO / IEC JTC 1 / SC29,JVET-T2016, 2020.

[0230] Sullivan, G. J. and Wiegand, T. Rate-distortion optimization forvideo compression. IEEE signal processing magazine, .Sullivan, G. J., Ohm, J.-R., Han, W.-J., and Wiegand, T. Overview of the highefficiency video coding (HEVC) standard. IEEE Transactions on circuits andsystems for video technology, 22(12):1649-1668, 2012.

[0231] Vahdat, A. and Kautz, J. Nvae: A deep hierarchical variationalautoencoder. Advances in Neural Information Processing Systems, 33:19667-19679, 2020.

[0232] Wallace, G. K. The JPEG still picture compression standard.Communications of the ACM, 34(4):30-44, 1991.

[0233] Wang, H., Gan, W., Hu, S., Lin, J. Y., Jin, L., Song, L., Wang, P.,Katsavounidis, I., Aaron, A., and Kuo, C.-C. J. MCLJCV: a JND-based H. 264 / AVC video quality assessment dataset. In 2016 IEEE international conferenceon image processing (ICIP), pp. 1509-1513. IEEE, 2016.

[0234] Wang, Z., Simoncelli, E. P., and Bovik, A. C. Multiscale structuralsimilarity for image quality assessment. In The Thrity-Seventh AsilomarConference on Signals, Systems & Computers, 2003, volume 2, pp. 1398-1402.Ieee, 2003.

[0235] Wien, M. High efficiency video coding. Coding Tools andspecification, 24, 2015.

[0236] Wu, C.-Y., Singhal, N., and Krahenbuhl, P. Video compression throughimage interpolation. In Proceedings of the European conference on computervision (ECCV), pp. 416-431, 2018.

[0237] Xue, T., Chen, B., Wu, J., Wei, D., and Freeman, WT Videoenhancement with task-oriented flow. International Journal of ComputerVision, 127(8):1106-1125, 2019.

[0238] Yang, R., Mentzer, F., Gool, LV, and Timofte, R. Learning for video compression with hierarchical quality and recurrent enhancement. InProceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition, pp. 6628-6637, 2020.

[0239] Zhang, R., Isola, P., Efros, AA, Shechtman, E., and Wang, O. Theunreasonable effectiveness of deep features as a perceptual metric. InProceedings of the IEEE conference on computer vision and patternrecognition, pp. 586-595, 2018.

[0240] Zhu, Y., Yang, Y., and Cohen, T. Transformer-based transform coding. In International Conference on Learning Representations, 2021.

[0241] Other aspects of this disclosure are described below.

[0242] Aspect 1A. A method for decoding media data, the method comprising: using a model to model a priori variables using a discrete cosine transform (DCT) to generate DCT coefficients; and decoding the media data based on the DCT coefficients.

[0243] Aspect 2A. The method according to aspect 1A, wherein decoding the media data includes quantizing and entropy decoding the media data using a channel-by-channel 4x4 parameter mixing of the DCT coefficients.

[0244] Aspect 3A. The method according to aspect 1A or aspect 2A, the method further comprising: decoding frequency information.

[0245] Aspect 4A. The method according to any one of Aspects 1A to 3A, wherein the model includes an I-frame model.

[0246] Aspect 5A. The method according to any one of Aspects 1A to 3A, wherein the model includes a P-frame model.

[0247] Aspect 6A. The method according to any one of Aspects 1A to 5A, the method further comprising: applying amortization batch normalization to the media data.

[0248] Aspect 7A. The method according to any one of Aspects 1A to 6A, the method further comprising: applying context modeling to the media data using a subsampling chessboard with uneven intervals.

[0249] Aspect 8A. The method according to any one of Aspects 1A to 7A, the method further comprising: quantizing the DCT coefficients using a learned quantization interval size.

[0250] Aspect 9A. The method according to any one of Aspects 1A to 8A, wherein the media data includes video data.

[0251] Aspect 10A. The method according to any one of aspects 1A to 9A, wherein decoding includes encoding.

[0252] Aspect 11A. The method according to any one of aspects 1A to 10A, wherein decoding includes decoding.

[0253] Aspect 12A. An apparatus for decoding media data, the apparatus comprising: a memory configured to store media data; and one or more processors implemented in a circuit and coupled to the memory, the one or more processors being configured to perform a method according to any one of Aspects 1A to 11A.

[0254] Aspect 13A. The device according to aspect 12A, the device further comprising: a camera configured to capture the media data.

[0255] Aspect 14A. The device according to aspect 12A or aspect 3A, the device further comprising: a display configured to display the media data.

[0256] Aspect 15A. An apparatus for decoding media data, the apparatus comprising one or more components for performing the method according to any one of aspects 1A to 11A.

[0257] Aspect 16A. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of Aspects 1A to 11A.

[0258] Aspect 1B. A method for training a P-frame model of a neural media decoder, the method comprising: acquiring a corrupted I-frame; and using the corrupted I-frame to train the P-frame model.

[0259] Aspect 2B. According to the method of aspect 1B, the corrupted I-frame includes I-frames pre-trained for a peak signal-to-noise ratio (PSNR) lower than that of a Lagrange multiplier.

[0260] Aspect 3B. The method according to aspect 1B or aspect 2B, wherein training the P-frame model includes training the P-frame model for seven or fewer frames.

[0261] Aspect 4B. The method according to aspect 3B, wherein training the P-frame model includes training the P-frame model for three frames.

[0262] Aspect 5B. The method according to any one of Aspects 1B to 4B, wherein training the P-frame model includes applying Forced Quality Enhancement (FQE).

[0263] Aspect 6B. A method for decoding media data, the method comprising: applying a pre-trained P-frame model to the media data, the pre-trained P-frame model being trained using corrupted I-frames; and decoding the media data based on applying the pre-trained P-frame model to the media data.

[0264] Aspect 7B. The method described in aspect 6B, wherein the media data includes video data.

[0265] Aspect 8B. The method according to aspect 6B or aspect 7B, wherein decoding includes encoding.

[0266] Aspect 9B. The method according to any one of aspects 6B to 8B, wherein decoding includes decoding.

[0267] Aspect 10B. An apparatus for decoding media data, the apparatus comprising: a memory configured to store media data; and one or more processors implemented in a circuit and coupled to the memory, the one or more processors being configured to perform a method according to any one of aspects 1B to 9B.

[0268] Aspect 11B. The device according to aspect 10B, the device further comprising: a camera configured to capture the media data.

[0269] Aspect 12B. The apparatus according to aspect 10B or aspect 11B, further comprising: a display configured to display the media data.

[0270] Aspect 13B. An apparatus for decoding media data, the apparatus comprising one or more components for performing the method according to any one of aspects 1B to 9B.

[0271] Aspect 14B. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of aspects 1B to 9B.

[0272] Aspect 1C. A method for encoding media data, the method comprising: determining a priori variables for a neural video decoder based on the media data; applying a discrete cosine transform (DCT) to the priori variables to generate DCT coefficients; and encoding the DCT coefficients.

[0273] Aspect 2C. The method according to aspect 1C, wherein encoding the DCT coefficients includes channel-wise parameter mixing of quantizing the DCT coefficients to generate quantized DCT coefficients.

[0274] Aspect 3C. The method according to aspect 2C, wherein encoding the DCT coefficients further includes entropy encoding of the quantized DCT coefficients.

[0275] Aspect 4C. The method according to aspect 2C or aspect 3C, wherein the channel-by-channel parameter mixing includes channel-by-channel 4x4 parameter mixing.

[0276] Aspect 5C. The method according to any one of Aspects 1C to 4C, wherein the prior variables include the prior variables of the I-frame model of the neural video decoder.

[0277] Aspect 6C. The method according to any one of Aspects 1C to 4C, wherein the prior variables include the prior variables of the P-frame model of the neural video decoder.

[0278] Aspect 7C. The method according to aspect 6C, wherein the P-frame model includes at least one of a motion estimation model or a conditional decoder model.

[0279] Aspect 8C. The method according to aspect 6C or aspect 7C, wherein the P-frame model is trained using a corrupted I-frame, the corrupted I-frame having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold or b) one or more insertion error regions.

[0280] Aspect 9C. The method according to any one of Aspects 1C to 8C, wherein the super-prior variable is based on a latent variable, and wherein the method further comprises: determining the latent variable, wherein determining the latent variable includes applying context modeling to the media data using a subsampling chessboard with non-uniform intervals.

[0281] Aspect 10C. The method according to aspect 9C, wherein the subsampling chessboard using the non-uniform interval comprises shifted copies of stacked NxN masks.

[0282] Aspect 11C. The method according to any one of aspects 1C to 10C, wherein the media data includes video data.

[0283] Aspect 12C. A method for decoding media data, the method comprising: determining DCT coefficients of a super-prior variable of a neural video decoder; applying an inverse DCT to the DCT coefficients to generate the super-prior variable; and decoding the media data based on the super-prior variable.

[0284] Aspect 13C. The method according to aspect 12C, wherein the DCT coefficients comprise a channel-wise parameter mixing of quantized DCT coefficients generated by a neural media encoder.

[0285] Aspect 14C. The method according to aspect 13C, wherein the channel-by-channel parameter mixing includes channel-by-channel 4x4 parameter mixing.

[0286] Aspect 15C. The method according to any one of aspects 12C to 14C, wherein determining the DCT coefficients includes entropy decoding of the DCT coefficients.

[0287] Aspect 16C. The method according to any one of Aspects 12C to 15C, wherein the prior variables include the prior variables of the I-frame model of the neural video decoder.

[0288] Aspect 17C. The method according to any one of Aspects 12C to 15C, wherein the prior variables include the prior variables of the P-frame model of the neural video decoder.

[0289] Aspect 18C. The method according to aspect 11C, wherein the P-frame model includes at least one of a motion estimation model or a conditional decoder model.

[0290] Aspect 19C. The method according to aspect 17C or aspect 18C, wherein the P-frame model is trained using a corrupted I-frame, the corrupted I-frame having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold or b) one or more insertion error regions.

[0291] Aspect 20C. The method according to any one of Aspects 12C to 19C, wherein the super-prior variable is based on a latent variable, and wherein decoding the media data comprises: determining the latent variable; and applying context modeling to the latent variable using a subsampling chessboard with uneven intervals.

[0292] Aspect 21C. The method according to aspect 20C, wherein the subsampling chessboard using the non-uniform interval comprises shifted copies of stacked NxN masks.

[0293] Aspect 22C. The method according to aspect 20C or aspect 21C, the method further comprising: dividing latent variable data into multiple groups, the latent variable data including the latent variables; using a first group of the multiple groups to decode a first percentage of the latent variable data; using a second group of the multiple groups to decode a second percentage of the latent variable data; and using a third group of the multiple groups to decode a third percentage of the latent variable data.

[0294] Aspect 23C. The method according to aspect 22C, wherein the first percentage and the second percentage are lower than the third percentage.

[0295] Aspect 24C. The method according to aspect 23C, wherein the first percentage is 25%, the second percentage is 25%, and the third percentage is 50%.

[0296] Aspect 25C. The method according to any one of aspects 12C to 24C, wherein the media data includes video data.

[0297] Aspect 26C. An apparatus for encoding media data, the apparatus comprising: one or more memories configured to store the media data; and one or more processors implemented in circuitry and coupled to the one or more memories, the one or more processors being configured to: determine a priori variables for a neural video decoder based on the media data; apply a discrete cosine transform (DCT) to the priori variables to generate DCT coefficients; and encode the DCT coefficients.

[0298] Aspect 27C. The apparatus according to aspect 26C, wherein, as part of encoding the DCT coefficients, the one or more processors are configured to quantize the channel-wise parameter mixing of the DCT coefficients to generate quantized DCT coefficients.

[0299] Aspect 28C. The apparatus according to aspect 27C, wherein, as part of encoding the DCT coefficients, the one or more processors are configured to entropy encode the quantized DCT coefficients.

[0300] Aspect 29C. The apparatus according to aspect 27C or aspect 28C, wherein the channel-by-channel parameter mixing includes channel-by-channel 4x4 parameter mixing.

[0301] Aspect 30C. The apparatus according to any one of aspects 26C to 29C, wherein the prior variables include the prior variables of the I-frame model of the neural video decoder.

[0302] Aspect 31C. The apparatus according to any one of Aspects 26C to 30C, wherein the prior variables include the prior variables of the P-frame model of the neural video decoder.

[0303] Aspect 32C. The apparatus according to aspect 31C, wherein the P-frame model includes at least one of a motion estimation model or a conditional decoder model.

[0304] Aspect 33C. The device according to aspect 31C or aspect 32C, wherein the P-frame model is trained using corrupted I-frames, the corrupted I-frames having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold or b) one or more insertion error regions.

[0305] Aspect 34C. The device according to any one of Aspects 26C to 33C, wherein the super-prior variable is based on a latent variable, wherein the one or more processors are further configured to: determine the latent variable, and wherein, as part of determining the latent variable, the one or more processors are configured to apply context modeling to the media data using a subsampling chessboard with non-uniform intervals.

[0306] Aspect 35C. The device according to aspect 34C, wherein, as part of a subsampling chessboard using the non-uniform interval, the one or more processors are configured as shifted copies of a stacked NxN mask.

[0307] Aspect 36C. The device according to any one of aspects 26C to 35C, wherein the media data includes video data.

[0308] Aspect 37C. The device according to any one of aspects 26C to 36C, the device further comprising: a camera configured to capture the media data.

[0309] Aspect 38C. An apparatus for decoding media data, the apparatus comprising: one or more memories configured to store the media data; and one or more processors implemented in circuitry and coupled to the one or more memories, the one or more processors being configured to: determine DCT coefficients of a hyperprior variable of a neural video decoder; apply an inverse DCT to the DCT coefficients to generate the hyperprior variable; and decode the media data based on the hyperprior variable.

[0310] Aspect 39C. The device according to aspect 38C, wherein the DCT coefficients comprise a channel-wise parameter mix of quantized DCT coefficients generated by a neural media encoder.

[0311] Aspect 40C. The apparatus according to aspect 39C, wherein the channel-by-channel parameter mixing includes channel-by-channel 4x4 parameter mixing.

[0312] Aspect 41C. The apparatus according to any one of aspects 38C to 40C, wherein, as part of determining the DCT coefficients, the one or more processors are configured to entropy decode the DCT coefficients.

[0313] Aspect 42C. The apparatus according to any one of aspects 38C to 41C, wherein the prior variables include the prior variables of the I-frame model of the neural video decoder.

[0314] Aspect 43C. The apparatus according to any one of aspects 38C to 41C, wherein the prior variables include the prior variables of the P-frame model of the neural video decoder.

[0315] Aspect 44C. The apparatus according to aspect 43C, wherein the P-frame model includes at least one of a motion estimation model or a conditional decoder model.

[0316] Aspect 45C. The device according to aspect 43C or aspect 44C, wherein the P-frame model is trained using corrupted I-frames, the corrupted I-frames having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold or b) one or more insertion error regions.

[0317] Aspect 46C. The device according to any one of aspects 38C to 45C, wherein the super-prior variable is based on a latent variable, and wherein, as part of decoding the media data, the one or more processors are configured to: determine the latent variable; and apply context modeling to the latent variable using a subsampling chessboard with uneven intervals.

[0318] Aspect 47C. The device according to aspect 46C, wherein, as part of a subsampling chessboard using the non-uniform interval, the one or more processors are configured as shifted copies of a stacked NxN mask.

[0319] Aspect 48C. The apparatus according to aspect 46C or aspect 47C, wherein the one or more processors are further configured to: divide latent variable data into multiple groups, the latent variable data including the latent variables; decode a first percentage of the latent variable data using a first group of the multiple groups; decode a second percentage of the latent variable data using a second group of the multiple groups; and decode a third percentage of the latent variable data using a third group of the multiple groups.

[0320] Aspect 49C. The device according to aspect 48C, wherein the first percentage and the second percentage are lower than the third percentage.

[0321] Aspect 50C. The device according to aspect 49C, wherein the first percentage is 25%, the second percentage is 25%, and the third percentage is 50%.

[0322] Aspect 51C. The device according to any one of aspects 38C to 50C, wherein the media data includes video data.

[0323] Aspect 52C. The device according to any one of aspects 38C to 51C, the device further comprising: a display configured to display the media data.

[0324] Aspect 1D. A method for training a P-frame model of a neural media decoder, the method comprising: acquiring a corrupted I-frame, the corrupted I-frame having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold or b) one or more insertion error regions; and using the corrupted I-frame to train the P-frame model.

[0325] Aspect 2D. According to the method of aspect 1D, the method further includes: generating the corrupted I-frame.

[0326] Aspect 3D. The method according to aspect 1D or 2D, wherein the corrupted I-frame has the PSNR that satisfies the threshold and wherein, in order to satisfy the threshold, the PSNR is lower than the threshold or lower than or equal to the threshold.

[0327] Aspect 4D. The method according to any one of Aspects 1D to 3D, wherein the threshold is based on a Lagrange multiplier associated with the P-frame model.

[0328] Aspect 5D. The method according to any one of Aspects 1D to 4D, wherein the corrupted I-frame is pre-configured to satisfy the threshold.

[0329] Aspect 6D. The method according to any one of Aspects 1D to 5D, wherein training the P-frame model comprises training the P-frame model for seven or fewer frames.

[0330] Aspect 7D. According to the method of aspect 6, training the P-frame model includes training the P-frame model for three frames.

[0331] Aspect 8D. A method for decoding media data, the method comprising: applying a pre-trained P-frame model to the media data, the pre-trained P-frame model being trained using corrupted I-frames, the corrupted I-frames having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold or b) one or more insertion error regions; and decoding the media data based on applying the pre-trained P-frame model to the media data.

[0332] Aspect 9D. The method according to aspect 8D, wherein the corrupted I-frame has a PSNR that satisfies the threshold and wherein, in order to satisfy the threshold, the PSNR is lower than the threshold or lower than or equal to the threshold.

[0333] Aspect 10D. The method described according to aspect 8D or 9D, wherein the media data includes video data.

[0334] Aspect 11D. The method according to any one of aspects 8D to 10D, wherein decoding includes encoding.

[0335] Aspect 12D. The method according to any one of Aspects 8D to 10D, wherein decoding includes decoding.

[0336] Aspect 13D. An apparatus comprising: one or more memories configured to store media data and a P-frame model; and one or more processors implemented in a circuit and coupled to the one or more memories, the one or more processors being configured to: acquire a corrupted I-frame having at least one of: a) a peak signal-to-noise ratio (PSNR) below a threshold or b) one or more insertion error regions; and use the corrupted I-frame to train the P-frame model.

[0337] Aspect 14D. The device according to aspect 13D, wherein the one or more processors are further configured to: generate the corrupted I-frame.

[0338] Aspect 15D. The device according to aspect 13D or 14D, wherein the corrupted I-frame has the PSNR that satisfies the threshold and wherein, in order to satisfy the threshold, the PSNR is lower than the threshold or is lower than or equal to the threshold.

[0339] Aspect 16D. The device according to any one of Aspects 13D to 15D, wherein the threshold is based on a Lagrange multiplier associated with the P-frame model.

[0340] Aspect 17D. The device according to any one of Aspects 13D to 16D, wherein the corrupted I-frame is pre-configured to satisfy the threshold.

[0341] Aspect 18D. The device according to any one of aspects 13D to 17D, wherein the threshold comprises a Lagrange multiplier.

[0342] Aspect 19D. The device according to any one of aspects 13D to 18D, wherein the one or more processors are configured to train the P-frame model for fewer than or equal to seven frames.

[0343] Aspect 20D. The device according to aspect 19D, wherein one or more processors are configured to train the P-frame model for three frames.

[0344] Aspect 21D. The device according to any one of aspects 13D to 20D, the device further comprising: a camera configured to capture the media data.

[0345] Aspect 22D. The device according to any one of aspects 13D to 20D, the device further comprising: a display configured to display the media data.

[0346] Aspect 23D. An apparatus for decoding media data, the apparatus comprising: one or more memories configured to store media data; and one or more processors implemented in a circuit and coupled to the one or more memories, the one or more processors being configured to: apply a pre-trained P-frame model to the media data, the pre-trained P-frame model being trained using corrupted I-frames having at least one of: a) a peak signal-to-noise ratio (PSNR) that meets a threshold or b) one or more insertion error regions; and decode the media data based on applying the pre-trained P-frame model to the media data.

[0347] Aspect 24D. The device according to aspect 23D, wherein the corrupted I-frame has a PSNR that satisfies the threshold and wherein, in order to satisfy the threshold, the PSNR is lower than the threshold or lower than or equal to the threshold.

[0348] Aspect 25D. The device according to aspect 23D or 24D, wherein the media data includes video data.

[0349] Aspect 26D. The device according to any one of aspects 23D to 25D, wherein the one or more processors are configured to encode the media data as part of decoding the media data.

[0350] Aspect 27D. The device according to any one of aspects 23D to 25D, wherein the one or more processors are configured to decode the media data as part of decoding the media data.

[0351] Aspect 28D. The device according to any one of aspects 23D to 26D, the device further comprising: a camera configured to capture the media data.

[0352] Aspect 29D. The device according to any one of aspects 23D to 25D or 27D, the device further comprising: a display configured to display the media data.

[0353] It should be recognized that, based on the examples, certain actions or events of any technique described herein may be performed in a different sequence, and may be added, combined, or omitted entirely (e.g., not all actions or events described are necessary for implementing the technique). Furthermore, in some examples, actions or events may be performed concurrently (e.g., through multithreading, interrupt handling, or multiple processors) rather than sequentially.

[0354] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on a computer-readable medium or transmitted via a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium (which corresponds to a tangible medium such as a data storage medium) or a communication medium, including, for example, any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol. In this way, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. Computer program products may include computer-readable media.

[0355] By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies (such as infrared, radio, and microwave), then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (such as infrared, radio, and microwave) are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but instead refer to non-transient tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs utilize lasers to optically reproduce data. The combinations described above should also be included within the scope of computer-readable media.

[0356] Instructions can be executed by one or more processors, such as one or more DSPs, general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuits. Therefore, the terms "processor" and "processing circuit" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Moreover, these techniques can be fully implemented in one or more circuit or logic elements.

[0357] The techniques disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or IC sets (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but implementation by different hardware units is not necessarily required. Rather, as described above, various units can be combined in a codec hardware unit, or various units can be provided by a collection of interoperable hardware units (including one or more processors as described above) combined with appropriate software and / or firmware.

[0358] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A method for encoding media data, the method comprising: Based on the media data, determine the prior variables of the neural video decoder; The Discrete Cosine Transform (DCT) is applied to the prior variables to generate DCT coefficients; as well as The DCT coefficients are encoded.

2. The method of claim 1, wherein encoding the DCT coefficients comprises quantizing the DCT coefficients by channel-wise parameter mixing to generate quantized DCT coefficients.

3. The method according to claim 2, wherein encoding the DCT coefficients further includes entropy encoding of the quantized DCT coefficients.

4. The method of claim 2, wherein the channel-by-channel parameter mixing comprises channel-by-channel 4x4 parameter mixing.

5. The method according to claim 1, wherein the prior variables include the prior variables of the I-frame model of the neural video decoder.

6. The method according to claim 1, wherein the prior variables include the prior variables of the P-frame model of the neural video decoder.

7. The method of claim 6, wherein the P-frame model comprises at least one of a motion estimation model or a conditional decoder model.

8. The method of claim 6, wherein the P-frame model is trained using corrupted I-frames, the corrupted I-frames having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold or b) one or more insertion error regions.

9. The method of claim 1, wherein the prior variable is based on a latent variable, and wherein the method further comprises: The latent variables are determined, wherein determining the latent variables includes applying context modeling to the media data using a subsampling chessboard with uneven intervals.

10. The method of claim 9, wherein the subsampling chessboard using the non-uniform interval comprises shifted copies of stacked NxN masks.

11. The method of claim 1, wherein the media data includes video data.

12. A method for decoding media data, the method comprising: Determine the DCT coefficients of the prior variables of the neural video decoder; The inverse DCT is applied to the DCT coefficients to generate the hyperprior variable; as well as The media data is decoded based on the aforementioned prior variables.

13. The method of claim 12, wherein the DCT coefficients comprise a channel-wise parameter blend of quantized DCT coefficients generated by a neural media encoder.

14. The method of claim 13, wherein the channel-by-channel parameter mixing comprises channel-by-channel 4x4 parameter mixing.

15. The method of claim 12, wherein determining the DCT coefficients comprises entropy decoding of the DCT coefficients.

16. The method of claim 12, wherein the prior variables include the prior variables of the I-frame model of the neural video decoder.

17. The method of claim 12, wherein the prior variables include the prior variables of the P-frame model of the neural video decoder.

18. The method of claim 17, wherein the P-frame model comprises at least one of a motion estimation model or a conditional decoder model.

19. The method of claim 17, wherein the P-frame model is trained using corrupted I-frames, the corrupted I-frames having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold or b) one or more insertion error regions.

20. The method of claim 12, wherein the prior variable is based on a latent variable, and wherein decoding the media data comprises: Identify the latent variables; as well as A subsampling chessboard with uneven spacing is used to apply context modeling to the latent variables.

21. The method of claim 20, wherein the subsampling chessboard using the non-uniform interval comprises shifted copies of stacked NxN masks.

22. The method according to claim 20, further comprising: The latent variable data is divided into multiple groups, and the latent variable data includes the latent variables; The first group of the plurality of groups is used to decode the first percentage of the latent variable data; The second group of the plurality of groups is used to decode the second percentage of the latent variable data; as well as The third group of the plurality of groups is used to decode the third percentage of the latent variable data.

23. The method of claim 22, wherein the first percentage and the second percentage are lower than the third percentage.

24. The method of claim 23, wherein the first percentage is 25%, the second percentage is 25%, and the third percentage is 50%.

25. The method of claim 12, wherein the media data includes video data.

26. An apparatus for encoding media data, the apparatus comprising: One or more memories configured to store the media data; and One or more processors, implemented in a circuit and coupled to one or more memories, are configured to: Based on the media data, determine the prior variables of the neural video decoder; The Discrete Cosine Transform (DCT) is applied to the prior variables to generate DCT coefficients; and The DCT coefficients are encoded.

27. The apparatus of claim 26, wherein, as part of encoding the DCT coefficients, the one or more processors are configured to quantize the channel-wise parameter mixing of the DCT coefficients to generate quantized DCT coefficients.

28. The apparatus of claim 27, wherein, as part of encoding the DCT coefficients, the one or more processors are configured to entropy encode the quantized DCT coefficients.

29. The device of claim 27, wherein the channel-by-channel parameter mixing comprises channel-by-channel 4x4 parameter mixing.

30. The device of claim 26, wherein the prior variables include the prior variables of the I-frame model of the neural video decoder.

31. The device of claim 26, wherein the prior variables include the prior variables of the P-frame model of the neural video decoder.

32. The device of claim 31, wherein the P-frame model comprises at least one of a motion estimation model or a conditional decoder model.

33. The device of claim 31, wherein the P-frame model is trained using corrupted I-frames, the corrupted I-frames having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold or b) one or more insertion error regions.

34. The device of claim 26, wherein the prior variable is based on a latent variable, wherein the one or more processors are further configured to: determine the latent variable, and wherein, as part of determining the latent variable, the one or more processors are configured to apply context modeling to the media data using a subsampling chessboard with non-uniform intervals.

35. The device of claim 34, wherein the one or more processors are configured as shifted copies of a stacked NxN mask as part of a subsampling chessboard using the non-uniform interval.

36. The device of claim 26, wherein the media data includes video data.

37. The apparatus of claim 26, further comprising: A camera configured to capture the media data.

38. An apparatus for decoding media data, the apparatus comprising: One or more memories configured to store the media data; and One or more processors, implemented in a circuit and coupled to one or more memories, are configured to: Determine the DCT coefficients of the prior variables of the neural video decoder; Apply the inverse DCT to the DCT coefficients to generate the hyperprior variable; and The media data is decoded based on the aforementioned prior variables.

39. The device of claim 38, wherein the DCT coefficients comprise a quantized channel-wise parameter blend of DCT coefficients generated by a neural media encoder.

40. The device of claim 39, wherein the channel-by-channel parameter mixing comprises channel-by-channel 4x4 parameter mixing.

41. The device of claim 38, wherein, as part of determining the DCT coefficients, the one or more processors are configured to entropy decode the DCT coefficients.

42. The device of claim 38, wherein the prior variables include the prior variables of the I-frame model of the neural video decoder.

43. The device of claim 38, wherein the prior variables include the prior variables of the P-frame model of the neural video decoder.

44. The device of claim 43, wherein the P-frame model comprises at least one of a motion estimation model or a conditional decoder model.

45. The device of claim 44, wherein the P-frame model is trained using corrupted I-frames, the corrupted I-frames having at least one of the following: a) a peak signal-to-noise ratio (PSNR) that meets a threshold or b) one or more insertion error regions.

46. ​​The device of claim 38, wherein the prior variables are based on latent variables, and wherein, as part of decoding the media data, the one or more processors are configured to: Determine the latent variables; and A subsampling chessboard with uneven spacing is used to apply context modeling to the latent variables.

47. The device of claim 46, wherein the one or more processors are configured as shifted copies of a stacked NxN mask as part of a subsampling chessboard using the non-uniform interval.

48. The device of claim 46, wherein the one or more processors are further configured to: The latent variable data is divided into multiple groups, and the latent variable data includes the latent variables; The first group of the plurality of groups is used to decode the first percentage of the latent variable data; The second group of the plurality of groups is used to decode the second percentage of the latent variable data; as well as The third group of the plurality of groups is used to decode the third percentage of the latent variable data.

49. The device of claim 48, wherein the first percentage and the second percentage are lower than the third percentage.

50. The device of claim 48, wherein the first percentage is 25%, the second percentage is 25%, and the third percentage is 50%.

51. The device of claim 38, wherein the media data includes video data.

52. The apparatus of claim 38, further comprising: A display configured to display the media data.

Images