Entropy coding for neural-based media compression

Abstract

This disclosure describes entropy coding techniques for media data coded using neural-based techniques. A media coder is configured to determine probability distribution function parameters for data elements of a data stream coded by a neural-based media compression technique, the probability distribution function parameters being a logarithmic function of a standard deviation of the probability distribution function of the data stream, determine code vectors based on the probability distribution function parameters, and entropy code the data elements using the code vectors.

Description

[Technical Field]

[0001] This application claims priority to U.S. Patent Application No. 17 / 650,728, filed February 11, 2022, the entire contents of which are incorporated herein by reference.

[0002] TECHNICAL FIELD This disclosure relates to media encoding and decoding, including encoding and decoding of image and video data. [Background technology]

[0003] Digital media capabilities may be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video gaming devices, video game consoles, cellular or satellite wireless telephones, so-called "smartphones," video teleconferencing devices, video streaming devices, and the like. Digital video devices implement video coding techniques such as those described in standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265 / High Efficiency Video Coding (HEVC), ITU-T H.266 / Versatile Video Coding (VVC), and extensions to such standards, as well as proprietary video codecs / formats such as AOMedia Video 1 (AV1) developed by the Alliance for Open Media. By implementing such video coding techniques, video devices can more efficiently transmit, receive, encode, decode, and / or store digital video information. Summary of the Invention

[0004] Generally, this disclosure describes techniques for media compression, including techniques for video and / or image encoding and decoding. Media (e.g., image and / or video) compression methods based on neural networks are competitive with current standards and can offer several additional advantages. Neural-based coding methods are generally designed and tested using high-precision floating-point arithmetic. However, as the technology moves into practical deployment, neural network weights and activation functions are typically quantized and represented with low-precision integers to improve speed and power consumption.

[0005] This disclosure addresses problems that arise when neural network variables related to entropy coding are quantized. Neural network variables are important for the design of neural-based video / image compression schemes because such variables define compression efficiency. Furthermore, the very specific characteristics of entropy coding variables are not considered by common tools for optimizing quantization in neural networks. Tests have shown that the worst-case quantization effects can occur precisely in some of the most common use cases, and that the losses caused by the worst-case quantization effects cannot be recovered by retraining the neural network.

[0006] This disclosure describes techniques for optimizing the definition of trained entropy coding variables, so that the information most important for effective entropy coding is best preserved when represented as low-precision integers. Tests have also shown how the techniques described herein can be used to minimize the amount of memory required for entropy coding. This disclosure describes general approaches for entropy coding design, as well as specific solutions and implementations for commonly used Gaussian distributions. While the techniques of this disclosure may generally be applied to any neural-based compression technique, the examples described below focus on techniques for images and video.

[0007] In one example, a method includes determining probability distribution function parameters for data elements of a data stream coded by a neural-based media compression technique, the probability distribution function parameters being a logarithmic function of the standard deviation of the probability distribution function of the data stream; determining a code vector based on the probability distribution function parameters; and entropy coding the data elements using the code vector.

[0008] In another example, a device includes a memory and one or more processors in communication with the memory, wherein the one or more processors are configured to determine probability distribution function parameters for data elements of a data stream coded by a neural-based media compression technique, where the probability distribution function parameters are logarithmic functions of standard deviations of the probability distribution function of the data stream, determine code vectors based on the probability distribution function parameters, and entropy code the data elements using the code vectors.

[0009] In another example, a computer-readable storage medium is encoded with instructions that, when executed, cause a programmable processor to determine probability distribution function parameters for data elements of a data stream coded using a neural-based media compression technique, where the probability distribution function parameters are logarithmic functions of the standard deviation of the probability distribution function of the data stream; determine code vectors based on the probability distribution function parameters; and entropy code the data elements using the code vectors.

[0010] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0011] [Figure 1]FIG. 1 is a block diagram illustrating an example media encoding and decoding system that may implement the techniques of this disclosure. [Figure 2] Illustrates the differences between traditional media coding and neural-based media coding. [Figure 3] An example of a Gaussian continuous probability distribution function is shown below. [Figure 4] FIG. 1 is a block diagram illustrating an exemplary neural-based image codec. [Figure 5] 1 illustrates an exemplary implementation of neural-based entropy coding. [Figure 6] 1 illustrates an exemplary implementation of neural-based entropy coding with quantized probability distribution function parameters. [Figure 7] 10 illustrates an exemplary measure of redundancy for quantized probability distribution function parameters for different code vectors. [Figure 8] 1 shows an example of non-uniform allocation of code vectors. [Figure 9] An example of the distribution of Gaussian standard deviations by bit rate is shown. [Figure 10] An example of the entropy of a quantized random variable is shown below. [Figure 11] 10 shows an exemplary average relative redundancy from using uniform quantization of the Gaussian standard deviation. [Figure 12] 1 shows an exemplary entropy of a quantized random variable on a logarithmic scale. [Figure 13] 10 shows another example entropy of a quantized random variable on a logarithmic scale. [Figure 14] 1 shows a block diagram of another exemplary neural-based codec. [Figure 15] 1 shows an example of average relative redundancy using uniform quantization of probability distribution function parameters. [Figure 16] 10 shows exemplary code for converting quantized probability distribution function parameters into code vectors. [Figure 17]1 shows an example of relative redundancy using uniform quantization of probability distribution function parameters. [Figure 18] 10 shows another exemplary code for converting quantized probability distribution function parameters into code vectors. [Figure 19] 1 is a flowchart illustrating an example method of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] This disclosure describes techniques for encoding and decoding media data (e.g., images or videos) using neural network-based media coding techniques. In particular, this disclosure describes techniques for entropy coding of media data using optimized probability distribution function (PDF) parameterization. The techniques of this disclosure include definitions and implementations of quantized modified PDF parameters that are non-uniform (e.g., logarithmic) functions of PDF parameters (e.g., standard deviations or any parameters commonly used to define statistical distributions) used to train neural networks. By using the PDF parameters of this disclosure, entropy coding of media data using neural-based coding techniques can be performed more efficiently, using less memory, and the performance of neural-based coding techniques is not significantly degraded when neural network weights, activations, and outputs are represented by low-precision integers.

[0013] 1 is a block diagram illustrating an example media encoding and decoding system 100 that may implement the techniques of this disclosure. In the context of this disclosure, media may include any digital file to be compressed, including video data and / or images. The example techniques of this disclosure are generally directed to coding (encoding and / or decoding) video data and / or images. While the example of FIG. 1 is described with reference to media encoding and decoding, the techniques of the present application are equally applicable to encoding and decoding any type of data file using neural-based compression techniques.

[0014] 1, in this example, system 100 includes a source device 102 that provides encoded media data to be decoded and displayed by a destination device 116. Specifically, source device 102 provides the media data to destination device 116 via a computer-readable medium 110. Source device 102 and destination device 116 may comprise any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, mobile devices, tablet computers, set-top boxes, telephone handsets such as smartphones, televisions, cameras, display devices, digital media players, video gaming consoles, video streaming devices, broadcast receiver devices, etc. In some cases, source device 102 and destination device 116 may be capable of wireless communication and thus may be referred to as wireless communication devices.

[0015] In the example of FIG. 1 , source device 102 includes media source 104, memory 106, media encoder 200, and output interface 108. Destination device 116 includes input interface 122, media decoder 300, memory 120, and display device 118. According to this disclosure, media encoder 200 of source device 102 and media decoder 300 of destination device 116 may be configured to apply techniques for entropy coding and neural-based media compression systems. Thus, source device 102 represents an example of a media encoding device, while destination device 116 represents an example of a media decoding device. In other examples, source device and destination device may include other components or configurations. For example, source device 102 may receive media data from an external media source, such as an external camera. Similarly, destination device 116 may interface with an external display device rather than including an integrated display device.

[0016] The system 100 as shown in FIG. 1 is merely an example. In general, any digital media encoding and / or decoding device may implement techniques for entropy coding a neural-based media compression system. The source device 102 and the destination device 116 are merely examples of coding devices, such that the source device 102 generates coded media data for transmission to the destination device 116. This disclosure refers to a "coding" device as a device that performs coding (encoding and / or decoding) of data. Thus, the media encoder 200 and the media decoder 300 represent examples of coding devices, specifically, media encoders and media decoders, respectively. In some examples, the source device 102 and the destination device 116 may operate in a substantially symmetrical manner, such that each of the source device 102 and the destination device 116 includes media encoding and decoding components. Thus, system 100 may support one-way or two-way media transmission between source device 102 and destination device 116, for example, video streaming, video playback, video broadcasting, or video telephony.

[0017] Generally, the media source 104 represents a source of media data (i.e., raw, unencoded media data) and provides a continuous series of pictures (also called “frames”) of the media data to the media encoder 200, which encodes the data for the pictures. The media source 104 of the source device 102 may include a video capture device, such as a video camera, a video archive containing previously captured raw video, and / or a video feed interface for receiving video from a video content provider. As a further alternative, the media source 104 may generate computer-graphics-based data as source media, or a combination of live media, archived media, and computer-generated media. In each case, the media encoder 200 encodes the captured media data, pre-captured media data, or computer-generated media data. The media encoder 200 may reorder the pictures from the order in which they were received (sometimes referred to as “display order”) to a coding order for coding. The media encoder 200 may generate a bitstream including the encoded media data. Source device 102 may then output the encoded media data via output interface 108 to computer-readable medium 110 for receipt and / or retrieval by input interface 122 of destination device 116, for example.

[0018] The memory 106 of the source device 102 and the memory 120 of the destination device 116 represent general-purpose memory. In some examples, the memories 106, 120 may store raw media data, e.g., raw media from the media source 104 and raw decoded media data from the media decoder 300. Additionally or alternatively, the memories 106, 120 may store software instructions executable by, for example, the media encoder 200 and the media decoder 300, respectively. Although the memories 106 and 120 are shown separate from the media encoder 200 and the media decoder 300 in this example, it should be understood that the media encoder 200 and the media decoder 300 may also include internal memory for functionally similar or equivalent purposes. Furthermore, the memories 106, 120 may store encoded media data, for example, output from the media encoder 200 and input to the media decoder 300. In some examples, portions of memory 106, 120 may be allocated as one or more buffers, for example, for storing raw decoded and / or encoded media data.

[0019] The computer-readable medium 110 may represent any type of medium or device capable of transferring coded media data from the source device 102 to the destination device 116. In one example, the computer-readable medium 110 represents a communication medium for enabling the source device 102 to transmit coded media data directly to the destination device 116 in real time, for example, via a radio frequency network or a computer-based network. The output interface 108 may modulate a transmission signal containing the coded media data, and the input interface 122 may demodulate a received transmission signal in accordance with a communication standard such as a wireless communication protocol. The communication medium may comprise any wireless or wired communication medium, such as the radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful for facilitating communication from the source device 102 to the destination device 116.

[0020] In some examples, source device 102 may output the encoded data from output interface 108 to storage device 112. Similarly, destination device 116 may access the encoded data from storage device 112 via input interface 122. Storage device 112 may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded media data.

[0021] In some examples, source device 102 may output encoded media data to file server 114 or another intermediate storage device, which may store the encoded media data generated by source device 102. Destination device 116 may access the stored media data from file server 114 via streaming or download.

[0022] File server 114 may be any type of server device capable of storing encoded media data and transmitting the encoded media data to destination device 116. File server 114 may represent a web server (e.g., for a website), a server configured to provide file transfer protocol services (such as File Transfer Protocol (FTP) or File Delivery over Unidirectional Transport (FLUTE) protocol), a content delivery network (CDN) device, a hypertext transfer protocol (HTTP) server, a Multimedia Broadcast Multicast Service (MBMS) or Enhanced MBMS (eMBMS) server, and / or a network attached storage (NAS) device. The file server 114 may additionally or alternatively implement one or more HTTP streaming protocols, such as Dynamic Adaptive Streaming over HTTP (DASH), HTTP Live Streaming (HLS), Real Time Streaming Protocol (RTSP), HTTP Dynamic Streaming, etc.

[0023] Destination device 116 may access the encoded media data from file server 114 through any standard data connection, including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem, etc.), or a combination of both, suitable for accessing encoded media data stored on file server 114. Input interface 122 may be configured to operate according to any one or more of the various protocols discussed above for retrieving or receiving media data from file server 114, or other such protocols for retrieving media data.

[0024] Output interface 108 and input interface 122 may represent a wireless transmitter / receiver, a modem, a wired network component (e.g., an Ethernet card), a wireless communication component operating according to any of the various IEEE 802.11 standards, or other physical components. In examples in which output interface 108 and input interface 122 comprise wireless components, output interface 108 and input interface 122 may be configured to transfer data, such as coded media data, according to a cellular communication standard such as 4G, 4G-LTE (Long-Term Evolution), LTE-Advanced, 5G, etc. In some examples in which output interface 108 comprises a wireless transmitter, output interface 108 and input interface 122 may be configured to transfer data, such as coded media data, according to other wireless standards such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee™), the Bluetooth™ standard, etc. In some examples, the source device 102 and / or the destination device 116 may include respective system-on-chip (SoC) devices. For example, the source device 102 may include an SoC device for performing the functions attributed to the media encoder 200 and / or the output interface 108, and the destination device 116 may include an SoC device for performing the functions attributed to the media decoder 300 and / or the input interface 122.

[0025] The techniques of this disclosure may be applied to media coding in support of any of a variety of multimedia applications, such as over-the-air television broadcast, cable television transmission, satellite television transmission, Internet streaming media transmission such as Dynamic Adaptive Streaming over HTTP (DASH), digital media encoded on a data storage medium, decoding of digital media stored on a data storage medium, or other applications.

[0026] The input interface 122 of the destination device 116 receives the encoded media bitstream from the computer-readable medium 110 (e.g., a communications medium, a storage device 112, a file server 114, etc.). The encoded media bitstream may include signaling information defined by the media encoder 200 and also used by the media decoder 300. The display device 118 displays decoded pictures of the decoded media data to a user. The display device 118 may represent any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

[0027] Although not shown in FIG. 1, in some examples, the media encoder 200 and the media decoder 300 may each be integrated with an audio encoder and / or audio decoder, and may include appropriate MUX-DEMUX units or other hardware and / or software to handle multiplexed streams that include both audio and media in a common data stream.

[0028] The media encoder 200 and the media decoder 300 may each be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the techniques are implemented partially in software, a device may store instructions for the software on a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Each of the media encoder 200 and the media decoder 300 may be included in one or more encoders or decoders, any of which may be integrated as part of a combined encoder / decoder (encoder / decoder, CODEC) in the respective device. A device including the media encoder 200 and / or the media decoder 300 may include an integrated circuit, a microprocessor, and / or a wireless communication device such as a cellular telephone.

[0029] Image and video (e.g., media) compression methods based on neural networks are competitive with current standards and can offer several additional advantages. In some examples, neural-based coding methods are commonly designed and tested using high-precision floating-point arithmetic. However, the use of high-precision floating-point arithmetic can impact deployment due to high processing time and power consumption. As the technology moves into practical deployment, neural network weights and activation functions are typically quantized and represented as low-precision integers to improve speed and power consumption.

[0030] That is, in practical deployments, neural network variables (e.g., weights and activation functions) may be quantized. However, quantization of neural network variables can be problematic.

[0031] This disclosure describes exemplary techniques that can address problems that arise when neural network variables related to entropy coding are quantized. Neural network variables related to entropy coding are important for the design of neural-based video / image compression schemes because such variables define compression efficiency. Furthermore, the very specific characteristics of entropy coding variables are not considered by common tools for optimizing quantization in neural networks. Tests have shown that, in fact, the worst-case quantization effects can occur precisely in some of the most common use cases, and the losses caused by the worst-case quantization effects cannot be recovered by retraining the neural network.

[0032] This disclosure describes techniques for further optimizing the definition of entropy coding variables, so that the information most important for effective entropy coding is best preserved. Tests have also shown how the techniques described herein can be used to minimize the amount of memory required for entropy coding. This disclosure describes a general approach for entropy coding design, as well as specific solutions and implementations for commonly used Gaussian distributions.

[0033] As described in more detail below, the media encoder 200 and the media decoder 300 may be configured to determine probability distribution function parameters for data elements of a data stream coded by a neural-based media compression technique, the probability distribution function parameters being derived from a logarithm of the standard deviation of the probability distribution function of the data stream, determine code vectors based on the probability distribution function parameters, and entropy code the data elements using the code vectors.

[0034] Entropy coding is a fundamental part of media compression systems. The entropy coding process optimizes the conversion between media information and the compressed data bitstream, with the goal of obtaining the most compact representation possible. Unlike other elements of media compression, entropy coding is a lossless process, meaning it perfectly preserves the information.

[0035] Several techniques have been developed to implement efficient entropy coding in image and video compression standards. More recently, new compression methods based on deep learning and neural networks have been shown to approach the performance of traditional methods while offering several other practical advantages.

[0036] Because traditional codecs and neural-based codecs use very different forms of compression, traditional codecs and neural-based codecs use entropy coding in significantly different ways. An overview of the main differences is shown in Figure 2, which shows that even when the two approaches use the same coding method, such as arithmetic coding, all other aspects are different. This means that it is beneficial to develop new techniques for optimizing performance in neural-based codecs.

[0037] As shown in FIG. 2, the design phase for both conventional and neural-based coding techniques includes a data organization process 200 and a statistical data modeling process 202. For conventional media coding techniques such as H.265, the data organization process 200 includes the definition of coding elements (e.g., syntax elements and coding modes). For neural-based coding processes, the data organization process 200 includes the training and development of an image / video (e.g., media) autoencoder network. For the statistical data modeling process 202, conventional coding techniques include coding of context definitions for entropy coding processes such as context-adaptive binary arithmetic coding. For neural-based coding processes, the statistical modeling process 202 includes the development and training of a hyperprior neural network.

[0038] Both conventional coding and neural-based coding techniques include implementation stages that include a probability determination process 204 and an information-to-bits conversion process 206. Both the probability determination process 204 and the information-to-bits conversion process 206 are based on the data to be coded. In the case of conventional coding, the probability determination process 204 (e.g., a bit rate allocation process) includes adaptive context probability estimation. In the case of neural-based coding, the probability determination process 204 includes determination of probability distribution parameters, as described in more detail below. Both conventional coding and neural-based coding techniques use arithmetic and / or entropy coders (e.g., or arithmetic or entropy encoders or decoders) for the information-to-bits conversion process 206. The arithmetic coder may be Huffman coding, range coding, or asymmetric numeral system (ANS) coding.

[0039] The data stream management stage is the management of compressed data 208. In some exemplary conventional coding systems, the compressed data is in a single bitstream. Some exemplary neural-based coding systems use multiple streams for simultaneous coding.

[0040] The main entropy coding principle, derived from basic information theory principles, is to determine the optimal number of bits, N, allocated to code a data symbol. b but,

[0041]

number

[0042]

number

[0043]

number

[0044] Symbol probabilities can vary widely and frequently, depending on the particular portion of the media content. Therefore, in traditional media compression, one of the most important tasks is the efficient estimation of these data symbol probabilities and how to accurately update them as they change during coding. Because this probability estimation is performed sequentially, it can be very difficult to parallelize entropy coding in traditional video coding methods.

[0045] Compression methods proposed for neural-based codecs are designed to operate under the assumption that the data being compressed has a certain type of probability density function (PDF), and only the parameters defining this function are allowed to vary. As an example, a PDF may be chosen to have all variables have independent zero-mean Gaussian distributions, each with a different standard deviation σ. This may not simply be an empirical assumption, as the network "learns" to make it a reality. PDFs are typically Gaussian-distributed, as shown in the example in Figure 3, but this is not necessarily the case. Figure 3 shows an example of a Gaussian continuous PDF 300 commonly used in neural-based codecs, along with the quantized values ​​and their respective probabilities (p0, p1, ..., p8). Multiple values ​​corresponding to rare underflow and overflow cases are typically coded suboptimally in a manner that does not require probability estimation.

[0046] Generally, σ is a scaling factor for the "template" distribution. For example, a Gaussian or Laplace with zero mean and unit variance is the template, and there is a scaling parameter to "stretch" or "shrink" the template. Generally, the standard deviation is used for its convenience, but it can be any factor. This disclosure defines techniques to change from parameters or scaling factors that are convenient for humans (e.g., traditionally used by statisticians) to ones that are optimized for entropy coding using low-precision parameter representations.

[0047] In some examples, a separate neural network (e.g., separate from the neural network that performs media compression) is trained to determine the parameters of the distribution. Details of this training are described below. Prior to entropy coding, these variables are discretized using a fixed quantization scheme, and the probabilities of the discrete values ​​(p0, p1, ..., p8 in the example of Figure 3) are derived from the assumed distribution and later used to determine the number of symbol bits according to equation (1).

[0048] Unlike traditional coding methods, which typically require estimating probabilities for all possible values ​​of each data symbol, neural-based methods typically rely only on the parameters of the PDF (e.g., the standard deviation σ), which are calculated in parallel using a neural network, making it much easier to implement joint entropy coding without compression loss. Because entropy coding is entirely defined by the values ​​of those PDF parameters, compression performance can degrade significantly when the PDF parameters are expressed with low integer precision. This disclosure describes new techniques that minimize those compression losses.

[0049] One important class of methods proposed for neural-based image and video compression adapts a variational autoencoder architecture, in which latent variables are quantized and entropy coded using a fixed training distribution. In some examples, this architecture has been extended by adding a smaller neural network, called a hyperprior, that uses the variables generated from the autoencoder to define the probability distribution used to entropy code those autoencoder variables. This hyperprior network also uses quantization and entropy coding, but with predefined (e.g., learned) PDF parameters. Figure 4 is a diagram of a neural image codec that uses a variational autoencoder and a hyperprior network for entropy coding.

[0050] The process is illustrated in Figure 4, where a neural network for image analysis and synthesis implements a variational autoencoder, whose entropy coding of the variables is defined by a hyperprior decoder network (used at both the sender and receiver). Because the terms encoder and decoder (e.g., media encoder 200 and media decoder 300) are used to refer to the autoencoder portion, it is common in the neural network literature to refer to systems traditionally referred to by those names as the sender and receiver, respectively.

[0051] 4, media encoder 200 may include an image analysis neural network 402, a hyperprior encoder neural network 404, a quantization process 406, an entropy encoding process 408, a quantization process 416, and an entropy encoding process 418. Media encoder 200 may also include an entropy decoding process 412 (which may be similar to a reconstruction loop in a hybrid video coder), a hyperprior decoder neural network 424, an entropy decoding process 422, and an image synthesis neural network 426. Although not shown, media decoder 300 may include structures similar to entropy decoding process 412, hyperprior decoder neural network 424, entropy decoding process 422, and image synthesis neural network 426.

[0052] Image analysis neural network 402 is a neural network configured to encode and / or compress input image 400. The compressed data stream produced by image analysis neural network 402 is then processed by hyperprior encoder neural network 404. The output of hyperprior encoder neural network 404 is then quantized by quantization process 406 and entropy coded by entropy coding process 508 to produce bitstream 1 410. Hyperprior encoder neural network 404 is part of a process configured to determine parameters (e.g., standard deviation) of a PDF for the data stream created by image analysis neural network 402. The PDF parameters can then be used to entropy code the data stream from image analysis neural network 402.

[0053] The entropy encoding process 408 itself uses PDF parameters (e.g., learned parameters 414) learned from multiple images being encoded. The entropy decoding process 412 uses the same PDF parameters to entropy decode bitstream 1 410. This entropy decoded bitstream 412 is then processed by a hyperprior decoder neural network 424 to generate PDF parameters for the particular image being decoded (e.g., input image 400).

[0054] The PDF parameters generated by the hyperprior decoder neural network 404 are then used by an entropy encoding process 418 to encode the data stream generated by the image analysis neural network 402 after quantization by the quantization process 416. The entropy encoding process 418 creates bitstream 2 420. Bitstream 2 420 is then entropy decoded by an entropy decoding process 422 using the same PDF parameters from the hyperprior decoder neural network 424. After entropy decoding, an image synthesis neural network 426 processes the decoded data to generate an output image 428.

[0055] Figure 5 provides a more detailed illustration of the coding process for a single data element, denoted by x. In particular, Figure 5 shows an implementation of entropy coding using arithmetic coding, where x is the data to be coded and a vector σ of PDF parameters known by the sender and receiver (e.g., media encoder 200 and media decoder 3) includes parameters that define the PDF of x. More specifically, the vector σ is the standard deviation (e.g., of a normal Gaussian distribution) or, more generally, the parameters determined by a hyperprior network to determine the PDF to use for entropy coding.

[0056] In the context of this disclosure, data element x may include an input to a neural network (e.g., an input to image synthesis neural network 426 of FIG. 4), an output of a neural network (e.g., an output of image analysis neural network 402 of FIG. 4), a syntax element, or other coded data of an encoded media bitstream. Some example types of data that may be encoded to generate data element x in neural-based image / video coding (or more generally, a data element to be entropy coded using the techniques of this disclosure) may include RGB or YUV pixel values ​​in an image or video frame, residuals (pixel values ​​minus predictions from motion) of pixel values ​​in motion-compensated video, motion vectors for block-based motion compensation, and / or dense optical flow data for warping-based motion compensation. In other examples, data element x may include data representing weights, activation functions, or other data used by neural-based media compression techniques.

[0057] No data is coded during the training phase, rather the bit rate calculation process 500 calculates the number of bits N used by the arithmetic coding process. b The code vector calculation process 502 uses the data element x and the PDF parameters σ to estimate the PDF. The vector σ represents the parameters defining the PDF of the data element x that are known by the sender and receiver (e.g., media encoder 200 and media decoder 300). The PDF parameters σ may be calculated by a hyperprior decoder network or may be read from a table of predefined parameters that are learned. As an example, the code vector calculation process 502 may calculate the code vector c(σ) of a particular data element x from its corresponding PDF parameters σ.

[0058] The quantization process (Q x->q) 504 quantizes the data element x to create a quantized data element q. An arithmetic encoder 506 then arithmetically codes the quantized data element q using a code vector c(σ) to create a bit stream B. An arithmetic decoder 508 recovers the quantized data element q by arithmetically decoding it using the same code vector c(σ).

[0059]

number

[0060]

number

[0061]

number

[0062] Note that the techniques of this disclosure are applicable to all compression methods that involve neural networks generating parameters for the probability distributions used for entropy coding, and also to tables of entropy coding parameters stored using low-precision integers. The following sections frequently use examples in which σ represents the standard deviation of a Gaussian PDF, but the general techniques of this disclosure are applicable to all cases in which the PDF is defined by the parameter σ.

[0063] A PDF may be defined by two or more parameters, but not all parameters are directly used for entropy coding. For example, a PDF may be defined by a standard deviation σ and a mean value μ, but only the standard deviation determines the entropy coding characteristics. Since having one parameter is most common in practice, it is the assumption used in the next section, and the techniques described below can be easily generalized.

[0064] As shown in Figure 5, a block called Code Vector Calculation Process 502 defines how the PDF is used to create a code vector c(σ), which is the information needed to create the bitstream B through an arithmetic coding process. For example, a code vector can be a vector with a cumulative distribution function (CDF) corresponding to a probability mass function (PMF) defined by the quantized symbols (p0, p1, ..., p8 in Figure 3). In another example, if a technique called binarization is used, the code vector c(σ) contains the probabilities of binary decisions for coding the same data. In practice, the CDF and probabilities are scaled so that the elements of the vector c are represented as integers.

[0065] The system shown in Figure 5 illustrates how entropy coding is used when neural-based codecs are initially designed, trained, and tested, using highly accurate floating-point arithmetic and powerful processors. However, in practical implementations, neural network values ​​are typically represented as low-precision integers to allow for faster computation and lower power usage. This is shown in the modified system of Figure 6, where the parameter σ is expressed as a low-precision version of

[0066]

number

[0067] Figure 6 shows the PDF parameter σ quantized to a low-precision integer.

[0068]

number

[0069]

number

[0070]

number

[0071]

number

[0072]

number

[0073]

number

[0074] The quantization process (Q x->q ) 604 quantizes the data element x to produce a quantized data element q. Arithmetic encoder 606 then generates a code vector

[0075]

number

[0076]

number

[0077]

number

[0078]

number

[0079]

number

[0080] Note that in Figure 6, the quantization of x is different from the quantization of the PDF parameter σ. Floating-point parameters may work well for very complex neural networks, but are less viable for widespread use on smaller devices (e.g., mobile devices). Power consumption on such mobile devices would be very high. Therefore, the PDF parameter σ is quantized to an integer for more practical applications. The PDF parameter

[0081]

number

[0082] Another practical modification is motivated by the fact that recalculating the code vector c(σ) for each value of x is generally computationally expensive. For example, if x has a Gaussian distribution with standard deviation σ (e.g., the typical case), then the PMF of the quantized values ​​is defined by:

[0083]

number

[0084]

number

[0085] The entropy of this PMF in bits is:

[0086]

number

[0087] Note that in addition to computing the probabilities, an additional task can be performed to create a vector c with the correct integer cumulative sum or binary probabilities, at additional computational cost.

[0088]

number

[0089]

number

[0090]

number

[0091] For example, consider the following scenario:

[0092]

number

[0093]

number

[0094]

number

[0095] The loss incurred by having a limited number of code vectors can be measured using the code redundancy function, which is the average number of additional bits required to encode x using code vector c(r) instead of the optimal code vector c(σ).

[0096] The code redundancy function is defined by the Kullback-Leibler divergence measure as follows:

[0097]

number

[0098] Because media codecs operate over a wide range of bit rates, entropy coding is typically not designed to minimize the average redundancy, which is generally not known a priori. Instead, entropy coding is designed by bounding the relative redundancy, as follows:

[0099]

number

[0100] Designing an entropy coding method with strictly bounded relative redundancy guarantees that all means are also bounded. In other words, this approach may not minimize the average compression loss, but it is guaranteed to be small if the relative loss is always small.

[0101] FIG. 7 shows an example of a redundancy measure for a Gaussian distribution when the code vector is calculated using the following formula:

[0102]

number

[0103] Figure 7 shows an example of the absolute (plot 700, right Y-axis) and relative (plot 702, left Y-axis) measures of redundancy associated with quantized PDF parameters and the use of an integer number of code vectors. This loss is coding efficiency. Essentially, more bits than necessary are used. Quantizing the sigma value introduces uncertainty regarding the actual value of the floating-point sigma.

[0104] Assume that code vectors c(1.5), c(2.5), and c(3.5) are used to entropy code values ​​with PDF parameter σ in the intervals [1,2), [2,3), and [3,4], respectively. We can see that the redundancy is zero when σ = r and positive otherwise. Note that in the example of Figure 7, the maximum redundancy in the interval [1,2) is larger than in the other intervals.

[0105] Tests have shown that the maximum redundancy in an interval depends on the width of the interval and the values ​​within the interval, which means that better results can be obtained with a non-uniform allocation of code vectors.

[0106] Figure 8 shows an example of a non-uniform allocation of code vectors c(σ) designed to obtain the same maximum relative redundancy in all intervals. The relative redundancy graph in Figure 8 can be obtained when code vectors c(1.24), c(1.93), and c(3.14) are used to entropy code values ​​with PDF parameter σ in the intervals [1, 1.52), [1.52, 2.41), and [2.41, 4), respectively. It can be seen that for these values, the maximum relative redundancy is the same in the three intervals.

[0107] It should be noted that the technique described above with reference to Figure 8 relates to code vector assignment, not to the quantization of the PDF parameter σ. For example, if σ is quantized directly to an integer value, the fractional interval threshold shown in Figure 8 cannot be applied.

[0108] The same code vector c(r * (σ a,σ b )) defines the range of σ values ​​to be entropy coded. a ,σ b ), a media coder (e.g., media encoder 200 and media decoder 300) can minimize the maximum relative redundancy within this interval if the following formula is used:

[0109]

number

[0110] In the above code vector design criteria, in Fig. 8, the interval [σ a ,σ b ) redundancy in σ=r * (σ a ,σ b ) and has an approximately parabolic shape with a zero value when σ ...

[0111]

number

[0112] This average is shown as dashed line 800 in the example of Figure 8. Figure 8 shows the case where several intervals have the same maximum redundancy according to equation (9), and these intervals also have approximately the same average redundancy.

[0113] One fact to consider in identifying problems with the implementation shown in FIG. 6 is that the codec designer may choose the type of PDF and the parameters used to define it. However, since the parameter values ​​are defined by network training, it is the actual application of the design that defines the relative frequency of the parameter values. For example, FIG. 9 shows a plot 900 of the relative frequency of Gaussian standard deviation values ​​from a neural-based image codec. As shown in FIG. 9, for low bit rates (most used in practice), the σ values ​​peak at the minimum allowable value σ=0.1, and almost all are smaller than σ=10. On the other hand, for larger bit rates, larger σ values ​​are more common. This disclosure assumes that the standard deviation is within the range [0.1, 1000] to cover a wide range of playback qualities.

[0114] The PDF parameter σ defines the bit rate used by the codec, which is measured by the entropy function defined in Equation (4). Figure 10 shows a plot 1000 that illustrates how the entropy of a quantized value varies with the parameter σ within the range of standard deviations desired to be supported. More specifically, Figure 10 shows the entropy (in bits) of a quantized random variable with a Gaussian distribution as a function of the standard deviation σ.

[0115] A media coder (e.g., media encoder 200 or media decoder 300) quantizes σ from the range [0.1, 1000] to 256 values ​​(that fit into a single byte) and uses equation (9) to obtain

[0116]

number

[0117] With respect to Figures 10 and 11, the following issues can be observed: 1. The entropy function has very high derivatives at small values ​​of σ (the most used range), which complicates learning convergence during training. 2. Entropy is highly nonlinear near the origin, making it difficult to reproduce using quantized neural networks. 3. The important difference for small σ values ​​is

[0118]

number

[0119] In short, using standard deviation as a value for defining entropy coding may work reasonably well in high precision floating point implementations, but may not be beneficial for quantized neural networks used in practical implementations.

[0120] Similar issues arise for other types of probability distributions, since whenever a codec designer chooses PDF parameters, they have well-known intuitive meanings. High-precision neural networks can "learn" how to optimally use those parameters through extensive training, but quantization networks may not. In practical implementations, the assignment of code vectors is often driven by the quantization parameters.

[0121]

number

[0122] The practical problem presented above depends on how the PDF parameter σ shapes the entropy function and on the low-precision integer representation (quantization Q σ The entropy of a quantized value is caused by its conversion to a quantized value (shown in FIG. 6 by a block with σ). It can be difficult to identify the source of these problems from a direct analysis of how equation (4) defines the entropy of a quantized value because it is an infinite sum of terms with no clear intuitive interpretation. More insight can be gained by considering a simpler approximation. For a Gaussian distribution and large values ​​of σ, the entropy can be approximated by:

[0123]

number

[0124]

number

[0125] Figure 12 shows the entropy graph 1200 of Figure 10 when a logarithmic scale is used for the parameter σ, along with the approximation defined above. In both, rapid asymptotic convergence can be observed, with very good agreement observed in the appropriate range of σ, and in the intermediate range where both approximations are nearly identical.

[0126] From Figure 12, it is easier to see that the logarithm of σ provides a more direct relationship to entropy and may therefore be a better parameter for defining the PDF. One remaining aspect to consider is that, as shown in Figure 9, small σ values ​​are expected to be most common, and such small values ​​of σ correspond to very small entropy.

[0127] As shown in Figure 13, using a logarithmic scale for both entropy and standard deviation shows that it can be beneficial to also consider how the entropy function changes at low values ​​of σ. There may be cases where the media encoder 200 or media decoder 300 can encounter a large number of values ​​with very low entropy, so that even a very small absolute error can result in large relative redundancy and large compression loss. Figure 13 shows a plot 1300 illustrating entropy and its approximation from equations (11) and (12), graphed on a logarithmic scale.

[0128] Even though quantization errors are unavoidable, the techniques of this disclosure are based on the fact that it is possible to reduce compression loss by permuting variables to be quantized, taking into account how quantization errors affect relative redundancy.

[0129] One exemplary technique is shown in Figure 14, where a new variable called v is defined to parameterize the PDF. In general, the PDF parameter v may be determined using a nonlinear function (e.g., a logarithm) of the PDF parameter σ. The transformation function between σ and v is defined as follows:

[0130]

number

[0131] Figure 14 shows a modification of the actual codec of Figure 6 to allow for new design variables optimized for entropy coding and a table for more efficient allocation of code vectors. As shown in Figure 14, this function allows the codec designer to continue using the PDF parameter σ in training, and this function is assumed to be differentiable almost everywhere to maintain automatic gradient backpropagation.

[0132] For ease of explanation, the following conventions are adopted:

[0133] [Number]

[0134] Equation (14) specifies that σ ranges from the minimum value (σ min ) to the maximum value (σ max ). The new parameter v ranges from 0 to 1. The conversion function T v→σ when the v value is 0 is equal to σ min . The conversion function T v→σ when the v value is 1 is equal to σ max . The quantization of the new parameter v (e.g., Q v→n ) generates integers in the range from 0 to N - 1. Furthermore, when v0 < v1, T v→σ (v0) < T v→σ (v1) is defined.

[0135] In FIG. 14, the data is not encoded during the training phase. Since the coding phase is configured to use the new PDF parameter v, the training phase includes a conversion (T v→σ ) process 1400 that converts the PDF parameter v back to the PDF parameter σ (e.g., the standard deviation of the PDF). The bit rate calculation process 1401 estimates the number of bits N b used by the arithmetic coding process using the data element x and the PDF parameter v.

[0136] In the example of FIG. 14, the quantization process (Q v→n ) 1402 quantizes the PDF parameter v to create a quantized PDF parameter n. Next, the code selection process 1412 converts the quantized PDF parameter n to a code vector index i. The code vector index i may be an input to the code array process 1414. That is, the code vector index i may be an index to a lookup table of code vectors. Next, the code array process 1414 obtains the code vector c(r * i) is determined. * i ) is used to arithmetically code the data element x.

[0137] The quantization process (Q x->q ) 1404 quantizes the data element x to create a quantized data element q. Then, an arithmetic encoder 1406 quantizes the code vector c(r * i ) to arithmetically code the quantized data elements q to produce a bitstream B. The arithmetic decoder 1408 uses the same code vector c(r * i ) to recover the quantized data element q.

[0138]

number

[0139]

number

[0140]

number

[0141] Generally, in one example of the present disclosure, the media encoder 200 and the media decoder 300 may be configured to determine a probability distribution function parameter (e.g., v) for a data element (x) of a data stream coded by a neural-based media compression technique. The probability distribution function parameter (v) may be determined from a logarithmic function (e.g., the natural logarithm (ln)) of the standard deviation (σ) of the probability distribution function of the data stream. The media encoder 200 and the media decoder 300 may determine a code vector (c(r * i )), and * i )) can be used to entropy code a data element (x).

[0142] 14, in one example, the media encoder 200 and the media decoder 300 may be configured to quantize the probability distribution function parameter v before determining the code vector. In one example, to determine the code vector, the media encoder 200 and the media decoder 300 may be configured to determine a code vector index i based on the probability distribution function parameter v, and to determine the code vector from a table of code vectors based on the code vector index. More information regarding the table of code vectors is provided below.

[0143] From the encoder side, the media encoder 200 may be configured to generate the data element x using a neural-based compression technique, and may quantize the data element to create a quantized data element. In this example, entropy coding the data element using the code vector includes entropy encoding the quantized data element using the code vector. Further, in this example, the media encoder 200 may be configured to process the image or video picture using an image analysis neural network to generate the data element, e.g., as shown in FIG. 4.

[0144] From the decoder side, the media decoder 300 can decode the encoded data elements using the code vectors to create quantized data elements. The media decoder 300 can further dequantize the quantized data elements to create reconstructed data elements. The media decoder 300 may then process the reconstructed data elements using an image synthesis neural network to reconstruct an image or video picture.

[0145] The next aspect to be considered is that the performance of entropy coding is measured by the relative redundancy measure defined in equations (7) and (10). Below we give an example of a practical application of this approach.

[0146] Logarithmic transformation Most commonly, neural-based codecs are configured to operate using the assumption x~N(0,σ), i.e., the variable being coded has a zero-mean Gaussian (or normal) distribution. Some exemplary hyperprior distribution networks use some learned PDF, but more recent work has shown that there are improvements when the learned PDF is replaced with a Gaussian PDF and learned parameters.

[0147] Using this fact and the observation in FIGS. 12 and 13 that the logarithm of the standard deviation provides a more accurate parameterization of the PDF, a variable transformation function can be defined using:

[0148]

number

[0149]

number

[0150] The function ln() is the natural logarithm. When this new parameter v is quantized using N=256, the corresponding C=256 code vector is optimized according to equation (9), resulting in the average relative redundancy shown in Figure 15. Figure 15 shows a plot 1500 illustrating the average relative redundancy (loss in compression) caused by using uniform quantization of PDF parameter v with N=256 and C=256 code vectors. Figure 15 shows that PDF parameter v is a better parameter for defining a Gaussian PDF because the relative redundancy is always less than 1%.

[0151] The transformation equations (15) and (16) shown above use only exponential and logarithmic functions, which are continuous and have derivatives that are easy to calculate during design and training.

[0152] Table for selecting code vectors One additional aspect to consider is that because code vectors are constantly accessed during entropy coding, it may be preferable to store them in expensive, high-speed memory to maintain high data throughput. Thus, reducing the number of code vectors allows for savings in memory requirements and cost.

[0153] Savings in memory requirements and cost can be achieved by using code selection tables shown in Figure 16 after quantization of the PDF parameters v. These are simple tables with N elements, which allow for non-uniform allocation of code vectors, similar to the example in Figure 8, but instead of using floating-point values ​​that define transitions based on the PDF parameters σ, the tables are applied directly to the integer values ​​n obtained after quantization of the PDF parameters v.

[0154] The advantage is that this process is an integer-to-integer conversion that can be performed very efficiently with a simple table lookup implementation. Figure 16 shows an example of such a table 1600 that defines the number of code vectors to be used for entropy coding for each of 256 possible values ​​of n. In the example of Figure 16, the total table memory is 256 bytes. The media encoder 200 and media decoder 300 can use the table 1600 of Figure 16 used to convert PDF parameters v quantized by value n, where N=256, into code vector index i. Table 16 is designed to better guarantee a maximum redundancy of less than 1%, as shown in Figure 17, thus allowing for a reduction in the number of code vectors from 256 to 43. Figure 17 shows a plot 1700 illustrating the relative redundancy obtained according to the standard deviation.

[0155] Another way to limit memory usage is to reduce the range of standard deviation values. For example, a neural base can be implemented using the following PDF parameters:

[0156]

number

[0157]

number

[0158] FIG. 18 shows a code vector selection table 1800 designed for the above transformation to reduce the number of code vectors to 25.

[0159] The techniques of the present disclosure are implemented and tested using arithmetic coding and can be incorporated into neural-based codecs executed on mobile devices. In one example, the arithmetic coding technique can be implemented using 32-bit operations, and probability values can be scaled to 16 bits.

[0160] Table 1 shows some experimental results used to confirm the performance of the techniques of the present disclosure. The hyperprior distribution data was obtained using floating-point operations so that only the quantization of the entropy coding parameters affected the results.

[0161] As shown in the first row of the results in Table 1, the quantization PDF parameter σ is used to parameterize the Gaussian PDF, and approximately 0.5 megabytes of memory is required to store 256 code vectors. The resulting redundancy is as expected from FIG. 11 and approaches 1000%. Better results are possible, but more memory is required for the code vectors.

[0162] The following rows of Table 1 show that much better results are obtained when σ is replaced with the quantization PDF parameter v (e.g., the parameter obtained using the logarithmic function of σ) as defined in Equation (16). Using N = C = 256 actually removes redundancy, but the amount of memory for the code vectors is relatively large.

[0163] The results show that using smaller values of N = C helps reduce the amount of memory required, but the relative redundancy increases non-uniformly. On the other hand, using the above table lookup technique where C < N = 256 (e.g., less than 256 code vectors), about half of the memory is used with more consistent relative redundancy.

[0164] [Table 1]

[0165] As shown above, in the case of image and video coding using neural networks, choosing parameters for entropy coding that are most intuitive and convenient for the designer may work when using high-precision floating-point implementations, but may cause serious degradation when the neural network weights and activations are quantized. Indeed, analysis of commonly used Gaussian distributions shows that the worst degradation in compression efficiency (highest relative redundancy) occurs precisely in the most common cases obtained during compression.

[0166] The proposed technique of this disclosure addresses the problem by redefining the parameters used for entropy coding, maintaining the training process with a single change of variable, but modifying the quantization to equalize the increased redundancy that results from quantization. Furthermore, it is shown that using the new parameterization, it is possible to reduce the amount of memory used for entropy coding by using a simple table lookup for code vector selection.

[0167] Figure 19 is a flowchart illustrating an example method of the present disclosure. The technique of Figure 19 may be performed by one or more components of the media encoder 200 and / or the media decoder 300, including the components shown in the example of Figure 14. In the example of Figure 19, the media encoder 200 and / or the media decoder 300 may be configured to determine probability distribution function parameters for data elements of a data stream coded by a neural-based media compression technique, where the probability distribution function parameters are logarithmic functions of the standard deviation of the probability distribution function of the data stream (1900), determine code vectors based on the probability distribution function parameters (1902), and arithmetically code the data elements using the code vectors (1904).

[0168] Other exemplary aspects of the present disclosure are described below.

[0169] Aspect 1 - A method of coding media data, the method including: determining probability distribution function parameters for data elements of a data stream coded by a neural-based media compression technique, the probability distribution function parameters being a logarithmic function of the standard deviation of the probability distribution function of the data stream; determining a code vector based on the probability distribution function parameters; and entropy coding the data elements using the code vector.

[0170] Aspect 2 - The method of aspect 1, further comprising quantizing probability distribution function parameters before determining the code vector.

[0171] Aspect 3 - The method of any of aspects 1 to 2, wherein determining the code vector includes determining a code vector index based on a probability distribution function parameter; and determining the code vector from a table of code vectors based on the code vector index.

[0172] Aspect 4—The probability distribution function parameter is v, the standard deviation of the probability distribution function is σ, and the minimum standard deviation is σ min and the maximum standard deviation is σ max and v is

[0173]

number

[0174] Aspect 5 - The method of any of aspects 1 to 4, further comprising: generating the data elements using a neural-based compression technique; and quantizing the data elements to create quantized data elements, wherein entropy coding the data elements using the code vector comprises entropy encoding the quantized data elements using the code vector.

[0175] Aspect 6 - The method of aspect 5, wherein generating the data elements using neural-based compression techniques comprises processing the images or video pictures using an image analysis neural network to generate the data elements.

[0176] Aspect 7 - The method of any of aspects 1 to 4, wherein entropy coding the data elements using the code vectors includes entropy decoding the data elements encoded using the code vectors to create quantized data elements, and the method further includes dequantizing the quantized data elements to create reconstructed data elements.

[0177] Aspect 8 - The method of aspect 7, further comprising processing the reconstructed data elements using an image synthesis neural network to reconstruct an image or video picture.

[0178] Aspect 9 - An apparatus configured to code media data, the apparatus comprising: a memory; and one or more processors in communication with the memory, the one or more processors configured to: determine probability distribution function parameters for data elements of a data stream coded by a neural-based media compression technique, the probability distribution function parameters being logarithmic functions of standard deviations of the probability distribution function of the data stream; determine code vectors based on the probability distribution function parameters; and entropy code the data elements using the code vectors.

[0179]

[0023] Aspect 10 - The apparatus of aspect 9, wherein the one or more processors are further configured to quantize the probability distribution function parameters before determining the code vector.

[0180] Aspect 11 - The apparatus of any of aspects 9 to 10, wherein, to determine the code vector, the one or more processors are further configured to: determine a code vector index based on the probability distribution function parameters; and determine the code vector from a table of code vectors based on the code vector index.

[0181] Aspect 12 - The probability distribution function parameter is v, the standard deviation of the probability distribution function is σ, and the minimum standard deviation is σ min and the maximum standard deviation is σ max and v is

[0182]

number

[0183] Aspect 13 - The apparatus of any of aspects 9 to 11, wherein the one or more processors are further configured to generate the data elements using a neural-based compression technique and quantize the data elements to create quantized data elements, and wherein entropy coding the data elements using the code vector includes entropy encoding the quantized data elements using the code vector.

[0184] Aspect 14 - The apparatus of Aspect 13, wherein to generate the data elements using neural-based compression techniques, the one or more processors are further configured to process the images or video pictures using an image analysis neural network to generate the data elements.

[0185] Aspect 15 - The device of aspect 14, further comprising a camera configured to capture images or video pictures.

[0186] Aspect 16 - The apparatus of any of aspects 9 to 12, wherein entropy coding the data elements using the code vectors includes entropy decoding the data elements encoded using the code vectors to create quantized data elements, and the one or more processors are further configured to dequantize the quantized data elements to create reconstructed data elements.

[0187] Aspect 17 - The apparatus of aspect 16, wherein the one or more processors are further configured to process the reconstructed data elements using an image synthesis neural network to reconstruct an image or video picture.

[0188] Example 18 - The device of example 17, further comprising a display configured to display an image or video picture.

[0189] Aspect 19 - A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to determine probability distribution function parameters for data elements of a data stream coded using a neural-based media compression technique, where the probability distribution function parameters are logarithmic functions of the standard deviation of the probability distribution function of the data stream; determine code vectors based on the probability distribution function parameters; and entropy code the data elements using the code vectors.

[0190] Aspect 20 - The probability distribution function parameter is v, the standard deviation of the probability distribution function is σ, and the minimum standard deviation is σ min and the maximum standard deviation is σ max and v is

[0191]

number

[0192] It should be recognized that, depending on the example, certain acts or events of any of the techniques described herein may be performed in a different order, added, combined, or omitted entirely (e.g., not all described acts or events may be required to practice the techniques). Moreover, in certain examples, acts or events may be performed in parallel rather than sequentially, for example, through multithreading, interrupt processing, or multiple processors.

[0193] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which correspond to tangible media such as data storage media, or communication media, including any medium that facilitates transfer of a computer program from one place to another, for example, according to a communication protocol. As such, computer-readable media may generally correspond to (1) tangible computer-readable storage media that is non-transitory, or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0194] By way of example, and not limitation, such computer-readable storage media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included within the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but instead cover non-transitory tangible storage media. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks typically reproduce data magnetically, while discs reproduce data optically using a laser. Combinations of the above are also intended to be included within the scope of computer-readable media.

[0195] The instructions may be executed by one or more processors, such as one or more DSPs, general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuitry. Accordingly, the terms "processor" and "processing circuitry" as used herein may refer to any of the above structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or may be incorporated into a combined codec. The techniques may also be implemented entirely in one or more circuits or logic elements.

[0196] The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described in this disclosure to highlight functional aspects of devices configured to perform the disclosed techniques, but they do not necessarily require realization by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit or may be provided by a collection of interoperable hardware units, including one or more processors as described above, in conjunction with suitable software and / or firmware.

[0197] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. 1. A method for encoding media data, the method comprising: generating data elements of the data stream using a neural-based compression technique, the neural-based compression technique comprising processing an input image or video picture with an image analysis neural network to generate the data stream; determining a probability distribution function parameter for the data element, the probability distribution function parameter being a logarithmic function of a standard deviation of a probability distribution function of the data stream, the probability distribution function parameter being determined by processing the data element using a hypervisor encoder neural network; quantizing the probability distribution function parameters; determining a code vector based on the quantized probability distribution function parameters; quantizing the data elements to generate quantized data elements; entropy encoding the quantized data elements using the code vector; entropy coding the quantized probability distribution function parameters; A method comprising:

2. Determining the code vector comprises: determining a code vector index based on the probability distribution function parameters; and determining the code vector from a table of code vectors based on the code vector index.

3. The probability distribution function parameter is v, the standard deviation of the probability distribution function is σ, and the minimum standard deviation is σ ｍｉｎ and the maximum standard deviation is σ ｍａｘ and v is [Equation 1] The method of claim 1, wherein:

4. A method for decoding media data, the method comprising: determining probability distribution function parameters of data elements of a data stream encoded by a neural-based media compression technique, the probability distribution function parameters being a logarithmic function of the standard deviation of the probability distribution function of the data stream, the data stream encoded by the neural-based media compression technique having been previously encoded by: i) processing input images or video pictures with an image analysis neural network to generate a data stream; ii) processing the data stream with a hyperprior encoder neural network to determine probability distribution function parameters for the data stream; iii) quantizing the data stream; and iv) entropy coding the compressed data stream based on the probability distribution function parameters; determining a code vector based on the probability distribution function parameters; entropy decoding the data elements using the code vectors to generate quantized data elements; dequantizing the quantized data elements to generate reconstructed data elements; processing the reconstructed data elements using an image synthesis neural network to reconstruct an image or video picture; A method comprising:

5. The probability distribution function parameter is v, the standard deviation of the probability distribution function is σ, the minimum standard deviation is σ min , and the maximum standard deviation is σ max , and v is [Equation 2] The method of claim 4, wherein:

6. 1. An apparatus configured to encode media data, the apparatus comprising: Memory and one or more processors in communication with the memory; wherein the one or more processors: generating data elements of the data stream using a neural-based compression technique, the neural-based compression technique comprising processing an input image or video picture with an image analysis neural network to generate the data stream; determining a probability distribution function parameter for the data element, the probability distribution function parameter being a logarithmic function of a standard deviation of a probability distribution function of the data stream, the probability distribution function parameter being determined by processing the data element using a hypervisor encoder neural network; quantizing the probability distribution function parameters; determining a code vector based on the quantized probability distribution function parameters; quantizing the data elements to generate quantized data elements; entropy encoding the quantized data elements using the code vector; entropy coding the quantized probability distribution function parameters; The apparatus is configured to:

7. To determine the code vector, the one or more processors: determining a code vector index based on the probability distribution function parameters; determining the code vector from a table of code vectors based on the code vector index; The apparatus of claim 6 , further configured to:

8. The probability distribution function parameter is v, the standard deviation of the probability distribution function is σ, and the minimum standard deviation is σ ｍｉｎ and the maximum standard deviation is σ ｍａｘ and v is [Equation 3] 7. The device of claim 6, wherein:

9. The device of claim 6 , further comprising a camera configured to capture the image or video picture.

10. An apparatus configured to decode media data, the apparatus comprising: Memory and one or more processors in communication with the memory; wherein the one or more processors: determining probability distribution function parameters of data elements of a data stream encoded by a neural-based media compression technique, the probability distribution function parameters being a logarithmic function of the standard deviation of the probability distribution function of the data stream, the data stream encoded by the neural-based media compression technique having been previously encoded by: i) processing input images or video pictures with an image analysis neural network to generate a data stream; ii) processing the data stream with a hyperprior encoder neural network to determine probability distribution function parameters for the data stream; iii) quantizing the data stream; and iv) entropy coding the compressed data stream based on the probability distribution function parameters; determining a code vector based on the probability distribution function parameters; entropy decoding the data elements using the code vectors to generate quantized data elements; dequantizing the quantized data elements to generate reconstructed data elements; processing the reconstructed data elements using an image synthesis neural network to reconstruct an image or video picture; The apparatus is configured to:

11. The probability distribution function parameter is v, the standard deviation of the probability distribution function is σ, the minimum standard deviation is σ min , and the maximum standard deviation is σ max , and v is [Equation 4] 11. The device of claim 10, wherein:

12. The device of claim 10 , further comprising a display configured to display the image or video picture.

13. 6. A computer readable storage medium storing instructions that, when executed, cause one or more processors to perform the method of any one of claims 1 to 5.

Images