Neural network media compression using quantized entropy coding distribution parameters

Abstract

This disclosure describes entropy coding techniques for media data coded using neural-based techniques. A media coder is configured to determine probability distribution function parameters for data elements of a data stream coded by a neural-based media compression technique, the probability distribution function parameters being functions of a standard deviation of a probability distribution function of the data stream, determine code vectors based on the probability distribution function parameters, and entropy code the data elements using the code vectors.

Description

[Technical field]

[0001]

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 267,857, filed February 11, 2022, and U.S. Patent Application No. 17 / 814,426, filed July 22, 2022, both of which are incorporated by reference in their entireties herein.

[0002] This disclosure relates to media encoding and decoding, including encoding and decoding of image and video data. [Background technology]

[0003]

[0003] Digital media capabilities may be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video gaming devices, video game consoles, cellular or satellite wireless telephones, so-called "smartphones," video teleconferencing devices, video streaming devices, and the like. Digital video devices implement video coding techniques such as those described in standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265 / High Efficiency Video Coding (HEVC), ITU-T H.266 / Versatile Video Coding (VVC), and extensions to such standards, as well as proprietary video codecs / formats such as AOMedia Video 1 (AOMedia Video 1, AV1) developed by the Alliance for Open Media. Video devices may implement such video coding techniques to more efficiently transmit, receive, encode, decode, and / or store digital video information. Summary of the Invention

[0004]

[0004] Generally, this disclosure describes techniques for media compression, including techniques for video and / or image encoding and decoding. Media (e.g., image and / or video) compression methods based on neural networks can be competitive with current standards and provide some additional advantages. Neural-based coding methods are generally designed and tested using high-precision floating-point arithmetic. However, as the technology moves into practical deployment, neural network weights and activation functions are typically quantized and represented in low-precision integers to improve speed and power consumption.

[0005]

[0005] The present disclosure addresses problems that arise when neural network variables related to entropy coding are quantized. Neural network variables are important for the design of neural-based video / image compression schemes, since such variables define the compression efficiency. Furthermore, the very specific properties of entropy coding variables are not taken into account by common tools for optimizing quantization in neural networks. Tests have shown that the worst quantization effects can occur precisely in some of the most common use cases, and the losses caused by the worst quantization effects cannot be recovered by retraining the neural network.

[0006]

[0006] This disclosure describes techniques for optimizing the definition of trained entropy coding variables, so that the most important information for effective entropy coding is best preserved when represented in low precision integers. Tests have also shown how the techniques described herein can be used to minimize the amount of memory required for entropy coding. This disclosure describes a general approach for entropy coding design, as well as specific solutions and implementations for commonly used Gaussian distributions. Although the techniques of this disclosure can be generally applied to any neural-based compression technique, the examples described below focus on techniques for images and videos.

[0007] In one example, a method includes determining probability distribution function parameters for data elements of a data stream coded by a neural-based media compression technique, the probability distribution function parameters being based on a distribution of the data stream optimized for quantization, determining a code vector based on the probability distribution function parameters, and entropy coding the data elements using the code vector. The function can be determined empirically (e.g., logarithmically) or by using an algorithm that evaluates coding redundancy or a solution of an ordinary differential equation.

[0008]

[0008] In another example, a device includes a memory and one or more processors in communication with the memory, the one or more processors configured to determine probability distribution function parameters for data elements of a data stream coded by a neural-based media compression technique, the probability distribution function parameters being based on a distribution of the data stream optimized for quantization, determine code vectors based on the probability distribution function parameters, and entropy code the data elements using the code vectors. The function can be determined empirically (e.g., logarithmically) or by using an algorithm that evaluates coding redundancy or a solution of an ordinary differential equation.

[0009]

[0009] In another example, a computer-readable storage medium is encoded with instructions that, when executed, cause a programmable processor to determine probability distribution function parameters for data elements of a data stream coded by a neural-based media compression technique, the probability distribution function parameters being based on a distribution of the data stream optimized for quantization, determine a code vector based on the probability distribution function parameters, and entropy code the data elements using the code vector. The function can be determined empirically (e.g., logarithmically) or by using an algorithm that evaluates coding redundancy or a solution of an ordinary differential equation.

[0010]

[0010] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0011] [Figure 1]

[0011] FIG. 1 is a block diagram illustrating an example media encoding and decoding system that may implement the techniques of this disclosure. [Diagram 2]

[0012] 1 illustrates the difference between traditional media coding and neural-based media coding. [Diagram 3]

[0013] 1 shows an example of a Gaussian continuous probability distribution function. [Figure 4]

[0014] FIG. 1 is a block diagram illustrating an example neural-based image codec. [Diagram 5]

[0015] 1 illustrates an exemplary implementation of neural-based entropy coding. [Figure 6]

[0016] 1 illustrates an example implementation of neural-based entropy coding with quantized probability distribution function parameters. [Figure 7]

[0017] 1 illustrates an exemplary measure of redundancy for quantized probability distribution function parameters for different codevectors. [Figure 8]

[0018] 1 shows an example of non-uniform allocation of code vectors. [Figure 9]

[0019] FIG. 13 is a diagram illustrating an example of distribution of Gaussian standard deviation according to bit rate. [Figure 10]

[0020] FIG. 2 illustrates an example of the entropy of a quantized random variable. [Figure 11]

[0021] 13 illustrates an exemplary average relative redundancy from using uniform quantization of a Gaussian standard deviation. [Figure 12]

[0022] 1 illustrates an exemplary entropy of a quantized random variable on a logarithmic scale. [Figure 13]

[0023] 13 illustrates another example entropy of a quantized random variable on a logarithmic scale. [Figure 14]

[0024] 1 shows a block diagram of another exemplary neural-based codec. [Figure 15]

[0025] 1 shows an example of average relative redundancy using uniform quantization of probability distribution function parameters. [Figure 16]

[0026] 1 shows an exemplary code for converting quantized probability distribution function parameters into a code vector. [Figure 17]

[0027] 1 shows an example of relative redundancy using uniform quantization of probability distribution function parameters. [Figure 18]

[0028] 13 shows another exemplary code for converting quantized probability distribution function parameters into a code vector. [Figure 19]

[0029] 1 illustrates an example relative redundancy according to one embodiment of the present disclosure. [Figure 20]

[0030] 1 shows an example of probability distribution function parameter division. [Figure 21]

[0031] 1 shows a block diagram of another exemplary neural-based codec. [Figure 22]

[0032] 4 illustrates an example transformation function for probability distribution function parameters. [Diagram 23]

[0033] 4 illustrates an example output of a transformation function of the present disclosure. [Figure 24]

[0034] 13 illustrates an example relative redundancy measure based on different numbers of code vectors using an example transform function of the present disclosure. [Diagram 25]

[0035] 13 illustrates an example relative redundancy measure based on different numbers of code vectors using another example transform function of the present disclosure. [Figure 26]

[0036] 13 illustrates an example relative redundancy measure based on different numbers of code vectors using another example transform function of the present disclosure. [Figure 27]

[0037] FIG. 4 illustrates the derivative of an example transformation function of the present disclosure. [Figure 28]

[0038] FIG. 4 illustrates the relative error of an example conversion function of the present disclosure. [Figure 29]

[0039] 13 illustrates an example relative redundancy measure based on different numbers of code vectors using another example transform function of the present disclosure. [Diagram 30]

[0040] 1 is a flow chart illustrating an example method of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012]

[0041] This disclosure describes techniques for encoding and decoding media data (e.g., images or videos) using neural network-based media coding techniques. In particular, this disclosure describes techniques for entropy coding of media data using optimized probability distribution function (PDF) parameterization. In particular, this disclosure describes techniques for neural network-based media coding using PDF parameterization optimized for quantization. Exemplary techniques of this disclosure include definitions and implementations of quantized modified PDF parameters that are non-uniform (e.g., logarithmic) functions of PDF parameters (e.g., standard deviations or any parameters commonly used to define statistical distributions) used to train the neural network. By using the PDF parameters of this disclosure, entropy coding of media data using neural-based coding techniques may be performed more efficiently, using less memory, and without significant performance degradation of the neural-based coding techniques when neural network weights, activations, and outputs are represented by low-precision integers.

[0013]

[0042] FIG. 1 is a block diagram illustrating an example media encoding and decoding system 100 that may perform techniques of this disclosure. In the context of this disclosure, media may include any digital file that is compressed, including video data and / or images. The example techniques of this disclosure are generally directed to coding (encoding and / or decoding) video data and / or images. Although the example of FIG. 1 is described with reference to media encoding and decoding, the techniques of the present application are equally applicable to encoding and decoding any type of data file using neural-based compression techniques.

[0014]

[0043] 1, in this example, system 100 includes a source device 102 that provides encoded media data to be decoded and displayed by a destination device 116. Specifically, source device 102 provides the media data to destination device 116 via a computer-readable medium 110. Source device 102 and destination device 116 may comprise any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, mobile devices, tablet computers, set-top boxes, telephone handsets such as smartphones, televisions, cameras, display devices, digital media players, video gaming consoles, video streaming devices, broadcast receiver devices, and the like. In some cases, source device 102 and destination device 116 may be capable of wireless communication and thus may be referred to as wireless communication devices.

[0015]

[0044] In the example of FIG. 1, the source device 102 includes a media source 104, a memory 106, a media encoder 200, and an output interface 108. The destination device 116 includes an input interface 122, a media decoder 300, a memory 120, and a display device 118. According to this disclosure, the media encoder 200 of the source device 102 and the media decoder 300 of the destination device 116 may be configured to apply techniques for entropy coding and neural-based media compression systems. Thus, the source device 102 represents an example of a media encoding device, while the destination device 116 represents an example of a media decoding device. In other examples, the source device and the destination device may include other components or configurations. For example, the source device 102 may receive media data from an external media source, such as an external camera. Similarly, the destination device 116 may interface with an external display device rather than including an integrated display device.

[0016]

[0045] The system 100 as shown in FIG. 1 is merely an example. In general, any digital media encoding and / or decoding device may implement the techniques for entropy coding neural-based media compression systems. The source device 102 and the destination device 116 are merely examples of coding devices, such that the source device 102 generates coded media data for transmission to the destination device 116. This disclosure refers to a "coding" device as a device that performs coding (encoding and / or decoding) of data. Thus, the media encoder 200 and the media decoder 300 represent examples of coding devices, specifically, media encoders and media decoders, respectively. In some examples, the source device 102 and the destination device 116 may operate in a substantially symmetrical manner, such that each of the source device 102 and the destination device 116 includes media encoding and decoding components. Thus, system 100 may support one-way or two-way media transmission between source device 102 and destination device 116, for example, video streaming, video playback, video broadcasting, or video telephony.

[0017]

[0046] In general, the media source 104 represents a source of media data (i.e., raw, unencoded media data) and provides a continuous series of pictures (also called "frames") of the media data to the media encoder 200, which encodes the data for the pictures. The media source 104 of the source device 102 may include a video capture device, such as a video camera, a video archive containing previously captured raw video, and / or a video feed interface for receiving video from a video content provider. As a further alternative, the media source 104 may generate computer graphics-based data as source media, or a combination of live media, archived media, and computer-generated media. In each case, the media encoder 200 encodes the captured media data, pre-captured media data, or computer-generated media data. The media encoder 200 may reorder the pictures from the order in which they are received (sometimes referred to as "display order") to a coding order for coding. The media encoder 200 may generate a bitstream including the encoded media data. The source device 102 may then output the encoded media data via the output interface 108 to a computer-readable medium 110 for receipt and / or retrieval by, for example, an input interface 122 of the destination device 116 .

[0018]

[0047] The memory 106 of the source device 102 and the memory 120 of the destination device 116 represent general purpose memories. In some examples, the memories 106, 120 may store raw media data, e.g., raw media from the media source 104 and raw decoded media data from the media decoder 300. Additionally or alternatively, the memories 106, 120 may store software instructions executable by, e.g., the media encoder 200 and the media decoder 300, respectively. Although the memories 106 and 120 are shown in this example separately from the media encoder 200 and the media decoder 300, it should be understood that the media encoder 200 and the media decoder 300 may also include internal memory for functionally similar or equivalent purposes. Additionally, the memories 106, 120 may store encoded media data, e.g., output from the media encoder 200 and input to the media decoder 300. In some examples, portions of the memory 106, 120 may be allocated as one or more buffers, for example, for storing raw decoded and / or encoded media data.

[0019]

[0048] The computer-readable medium 110 may represent any type of medium or device capable of transferring encoded media data from the source device 102 to the destination device 116. In one example, the computer-readable medium 110 represents a communication medium for enabling the source device 102 to transmit encoded media data directly to the destination device 116 in real time, for example, via a radio frequency network or a computer-based network. The output interface 108 may modulate a transmission signal including the encoded media data, and the input interface 122 may demodulate a received transmission signal according to a communication standard, such as a wireless communication protocol. The communication medium may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful for facilitating communication from the source device 102 to the destination device 116.

[0020]

[0049] In some examples, source device 102 may output the encoded data from output interface 108 to storage device 112. Similarly, destination device 116 may access the encoded data from storage device 112 via input interface 122. Storage device 112 may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded media data.

[0021]

[0050] In some examples, source device 102 may output the encoded media data to a file server 114 or another intermediate storage device, which may store the encoded media data generated by source device 102. Destination device 116 may access the stored media data from file server 114 via streaming or download.

[0022]

[0051] File server 114 may be any type of server device capable of storing encoded media data and transmitting the encoded media data to destination device 116. File server 114 may represent a web server (e.g., for a website), a server configured to provide file transfer protocol services (such as File Transfer Protocol (FTP) or File Delivery over Unidirectional Transport (FLUTE) protocol), a content delivery network (CDN) device, a hypertext transfer protocol (HTTP) server, a Multimedia Broadcast Multicast Service (MBMS) or Enhanced MBMS (eMBMS) server, and / or a network attached storage (NAS) device. The file server 114 may additionally or alternatively implement one or more HTTP streaming protocols, such as Dynamic Adaptive Streaming over HTTP (DASH), HTTP Live Streaming (HLS), Real Time Streaming Protocol (RTSP), HTTP Dynamic Streaming, etc.

[0023]

[0052] Destination device 116 may access the encoded media data from file server 114 through any standard data connection, including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem, etc.), or a combination of both that is suitable for accessing the encoded media data stored on file server 114. Input interface 122 may be configured to operate according to any one or more of the various protocols discussed above for retrieving or receiving media data from file server 114, or other such protocols for retrieving media data.

[0024]

[0053] Output interface 108 and input interface 122 may represent wireless transmitters / receivers, modems, wired networking components (e.g., Ethernet cards), wireless communication components operating according to any of the various IEEE 802.11 standards, or other physical components. In examples in which output interface 108 and input interface 122 comprise wireless components, output interface 108 and input interface 122 may be configured to transfer data, such as coded media data, according to a cellular communication standard, such as 4G, 4G-LTE (Long-Term Evolution), LTE-Advanced, 5G, etc. In some examples in which output interface 108 comprises a wireless transmitter, output interface 108 and input interface 122 may be configured to transfer data, such as coded media data, according to other wireless standards, such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee™), the Bluetooth™ standard, etc. In some examples, the source device 102 and / or the destination device 116 may include respective system-on-a-chip (SoC) devices. For example, the source device 102 may include a SoC device for performing the functions attributed to the media encoder 200 and / or the output interface 108, and the destination device 116 may include a SoC device for performing the functions attributed to the media decoder 300 and / or the input interface 122.

[0025]

[0054] The techniques of this disclosure may be applied to media coding to support any of a variety of multimedia applications, such as over-the-air television broadcasts, cable television transmissions, satellite television transmissions, Internet streaming media transmissions such as Dynamic Adaptive Streaming over HTTP (DASH), digital media encoded on a data storage medium, decoding of digital media stored on a data storage medium, or other applications.

[0026]

[0055] The input interface 122 of the destination device 116 receives the encoded media bitstream from the computer-readable medium 110 (e.g., a communications medium, a storage device 112, a file server 114, etc.). The encoded media bitstream may include signaling information defined by the media encoder 200 and also used by the media decoder 300. The display device 118 displays decoded pictures of the decoded media data to a user. The display device 118 may represent any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

[0027]

[0056] Although not shown in FIG. 1, in some examples, the media encoder 200 and the media decoder 300 may each be integrated with an audio encoder and / or an audio decoder and may include appropriate MUX-DEMUX units or other hardware and / or software to handle multiplexed streams that include both audio and media in a common data stream.

[0028]

[0057] The media encoder 200 and the media decoder 300 may each be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the techniques are implemented partially in software, a device may store instructions for the software on a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Each of the media encoder 200 and the media decoder 300 may be included in one or more encoders or decoders, any of which may be integrated as part of a combined encoder / decoder (encoder / decoder, CODEC) in the respective device. A device including the media encoder 200 and / or the media decoder 300 may include an integrated circuit, a microprocessor, and / or a wireless communication device such as a cellular phone.

[0029]

[0058] Image and video (e.g., media) compression methods based on neural networks are competitive with current standards and can offer some additional advantages. In some examples, neural-based coding methods are typically designed and tested using high-precision floating-point arithmetic. However, the use of high-precision floating-point arithmetic can impact deployment due to high processing time and power consumption. As the technology moves into practical deployment, neural network weights and activation functions are typically quantized and represented in low-precision integers to improve speed and power consumption.

[0030]

[0059] That is, in practical deployments, neural network variables (e.g., weights and activation functions) may be quantized. However, quantization of neural network variables may be problematic.

[0031]

[0060] This disclosure describes an exemplary technique that may address the problems that arise when neural network variables related to entropy coding are quantized. Neural network variables related to entropy coding are important for the design of neural-based video / image compression schemes, since such variables define the compression efficiency. Furthermore, the very specific properties of entropy coding variables are not taken into account by common tools for optimizing quantization in neural networks. Tests have shown that, in fact, the worst quantization effects can occur precisely in some of the most common use cases, and the losses caused by the worst quantization effects cannot be recovered by retraining the neural network.

[0032]

[0061] This disclosure describes techniques for further optimizing the definition of the entropy coding variables, so that the most important information for effective entropy coding is best preserved. Tests have also shown how the techniques described herein can be used to minimize the amount of memory required for entropy coding. This disclosure describes a general approach for entropy coding design, as well as specific solutions and implementations for the commonly used Gaussian distribution.

[0033]

[0062] As described in more detail below, the media encoder 200 and the media decoder 300 may be configured to determine probability distribution function parameters for data elements of a data stream coded by a neural-based media compression technique, the probability distribution function parameters being based on a distribution of the data stream optimized for quantization, determine code vectors based on the probability distribution function parameters, and entropy code the data elements using the code vectors. This function may be determined empirically (e.g., logarithmically) or by using an algorithm that evaluates coding redundancy or the solution of ordinary differential equations.

[0034]

[0063] Entropy coding is a fundamental part of media compression systems. The entropy coding process is responsible for optimizing the conversion between the media information and the compressed data bitstream, with the goal of obtaining the most compact representation possible. Unlike other elements of media compression, entropy coding is a lossless process, i.e. it preserves the information perfectly.

[0035]

[0064] Several techniques have been developed to implement efficient entropy coding in image and video compression standards. More recently, new compression methods based on deep learning and neural networks have been shown to approach the performance of traditional methods while offering several other practical advantages.

[0036]

[0065] Because traditional and neural-based codecs use very different forms of compression, traditional and neural-based codecs use entropy coding in significantly different ways. An overview of the main differences is shown in Figure 2, where it can be seen that even when the two approaches use the same coding method, such as arithmetic coding, all other aspects are different. This means that it is beneficial to develop new techniques for optimizing performance in neural-based codecs.

[0037]

[0066] As shown in FIG. 2, the design phase of both conventional and neural-based coding techniques includes a data organization process 201 and a statistical data modeling process 202. For conventional media coding techniques such as H.265, the data organization process 201 includes the definition of coding elements (e.g., syntax elements and coding modes). For neural-based coding processes, the data organization process 201 includes the training and development of an image / video (e.g., media) autoencoder network. For the statistical data modeling process 202, conventional coding techniques include coding of context definitions for entropy coding processes such as context-adaptive binary arithmetic coding. For neural-based coding processes, the statistical data modeling process 202 includes the development and training of a hyperprior neural network.

[0038]

[0067] Both the conventional coding technique and the neural-based coding technique include implementation stages including a probability determination process 204 and an information-to-bits conversion process 206. Both the probability determination process 204 and the information-to-bits conversion process 206 are based on the data to be coded. In the case of conventional coding, the probability determination process 204 (e.g., a bit rate allocation process) includes adaptive context probability estimation. In the case of neural-based coding, the probability determination process 204 includes the determination of probability distribution parameters, as described in more detail below. Both the conventional coding technique and the neural-based coding technique use an arithmetic and / or entropy coder (e.g., or an arithmetic or entropy encoder or decoder) for the information-to-bits conversion process 206. The arithmetic coder may be Huffman coding, range coding or asymmetric numeral system (ANS) coding.

[0039]

[0068] The data stream management stage is the management of compressed data 208. In some exemplary conventional coding systems, the compressed data is in a single bitstream. Some exemplary neural-based coding systems use multiple streams for simultaneous coding.

[0040]

[0069] The main entropy coding principle, derived from basic information theory principles, is to determine the optimal number of bits, N, allocated to code a data symbol. b but,

[0041]

number

[0042] During the ceremony,

[0043]

number

[0044] is shared by the encoder and the decoder.

[0045]

number

[0046] is the estimated probability of the value of the symbol being encoded with knowledge of , (e.g., media encoder 200 and media decoder 300). This optimal number for the average number of bits is a fraction, but in practice can be achieved efficiently using arithmetic coding.

[0047]

[0070] The symbol probabilities may vary widely and frequently depending on the particular portion of the media content. Therefore, in traditional media compression, one of the most important tasks is the efficient estimation of these data symbol probabilities and how to accurately update the data symbol probabilities as they change during coding. Because this probability estimation is done sequentially, it may be difficult to parallelize entropy coding in traditional video coding methods.

[0048]

[0071] Compression methods proposed for neural-based codecs are designed to operate on the assumption that the data to be compressed has a certain type of probability density function (PDF), and only the parameters that define this function can vary. As an example, the PDF can be chosen to have all variables with independent zero-mean Gaussian distributions, each with a different standard deviation σ. This may not simply be an empirical assumption, as the network "learns" to make it a reality. The PDF is usually a Gaussian function, as shown in the example of Figure 3, but this is not necessarily the case. Figure 3 shows an example of a Gaussian continuous PDF 302, the quantized values, and their respective probabilities (p0, p1, ..., p8) commonly used in neural-based codecs. Multiple values ​​corresponding to rare underflow and overflow cases are typically coded suboptimally in a manner that does not require probability estimation.

[0049]

[0072] In general, the standard deviation parameter σ is a scaling factor for the "template" distribution. For example, a Gaussian or Laplace with zero mean and unit variance is the template, and there is a scaling parameter to "stretch" or "shrink" the template. In general, the standard deviation is used for its convenience, but it can be any factor. This disclosure defines techniques to change from parameters or scale factors that are convenient for humans (e.g., traditionally used by statisticians) to ones that are optimized for entropy coding using low-precision parameter representations.

[0050]

[0073] In some examples, a separate neural network (e.g., separate from the neural network that performs the media compression) is trained to determine the parameters of the distribution. Details of this training are described below. Prior to entropy coding, these variables are discretized using a fixed quantization scheme, and the probabilities of the discrete values ​​(p0, p1, ..., p8 in the example of Figure 3) are derived from the assumed distribution and are later used to determine the number of symbol bits according to equation (1).

[0051]

[0074] Unlike traditional coding methods that typically require estimating probabilities for all possible values ​​of each data symbol, neural-based methods may rely only on the parameters of the PDF (e.g., standard deviation σ) that are calculated in parallel using a neural network, making it much easier to implement joint entropy coding without compression loss. Because entropy coding is defined by the values ​​of those PDF parameters, compression performance can degrade significantly when the PDF parameters are expressed with low integer precision. This disclosure describes new techniques that minimize those compression losses.

[0052]

[0075] One important class of methods proposed for neural-based image and video compression adapts a variational autoencoder architecture, where latent variables are quantized and entropy coded using a fixed learning distribution. In some examples, this architecture is extended by adding a smaller neural network, called a hyperprior, that uses the variables generated from the autoencoder to define the probability distributions used to entropy code those autoencoder variables. This hyperprior network also uses quantization and entropy coding, but with predefined (e.g., learned) PDF parameters. Figure 4 is a diagram of a neural image codec that uses a variational autoencoder and a hyperprior network for entropy coding.

[0053]

[0076] The process is illustrated in Figure 4, where a neural network for image analysis and synthesis implements a variational autoencoder, whose entropy coding of the variables is defined by a hyperprior decoder network (used at both the sender and receiver). Because the terms encoder and decoder (e.g., media encoder 200 and media decoder 300) are used to denote the autoencoder portion, it is common in the neural network literature to refer to systems traditionally referred to by those names as the sender and receiver, respectively.

[0054]

[0077] In Figure 4, the media encoder 200 may include an image analysis neural network 402, a hyperprior encoder neural network 404, a quantization process 406, an entropy encoding process 408, a quantization process 416, and an entropy encoding process 418. The media encoder 200 may also include an entropy decoding process 412, a hyperprior decoder neural network 424, an entropy decoding process 422, and an image synthesis neural network 426 (which may be similar to the reconstruction loop in a hybrid video coder). Although not shown, the media decoder 300 may include structures similar to the entropy decoding process 412, the hyperprior decoder neural network 424, the entropy decoding process 422, and the image synthesis neural network 426.

[0055]

[0078] Image analysis neural network 402 is a neural network configured to encode and / or compress input image 400. The compressed data stream generated by image analysis neural network 402 is then processed by hyperprior encoder neural network 404. The output of hyperprior encoder neural network 404 is then quantized by quantization process 406 and entropy coded by entropy coding process 408 to generate bitstream 1 410. Hyperprior encoder neural network 404 is part of a process configured to determine parameters (e.g., standard deviation) of a PDF for the data stream created by image analysis neural network 402. The parameters of the PDF can then be used to entropy code the data stream from image analysis neural network 402.

[0056]

[0079] The entropy encoding process 408 itself uses PDF parameters (e.g., learned parameters 414) that have been learned from multiple images being encoded. The entropy decoding process 412 uses the same PDF parameters to entropy decode bitstream 1 410. This entropy decoded bitstream is then processed by a hyperprior decoder neural network 424 to generate PDF parameters for the particular image being decoded (e.g., input image 400).

[0057]

[0080] The PDF parameters generated by the hyperprior decoder neural network 424 are then used by an entropy encoding process 418 to encode the data stream generated by the image analysis neural network 402 after quantization by the quantization process 416. The entropy encoding process 418 creates bitstream 2 420. Bitstream 2 420 is then entropy decoded by an entropy decoding process 422 using the same PDF parameters from the hyperprior decoder neural network 424. After entropy decoding, an image synthesis neural network 426 processes the decoded data to generate an output image 428.

[0058]

[0081] Figure 5 provides a more detailed view of the coding process for a single data element, denoted by x. In particular, Figure 5 illustrates an implementation of entropy coding using arithmetic coding, where x is the data to be coded, and a vector σ of PDF parameters known by the sender and receiver (e.g., media encoder 200 and media decoder 3) includes parameters that define the PDF of x. More specifically, the vector σ is the standard deviation (e.g., normal Gaussian distribution) or, more generally, the parameters determined by a hyperprior network to determine the PDF to use for entropy coding.

[0059]

[0082] In the context of this disclosure, data element x may include an input to a neural network (e.g., an input to the image synthesis neural network 426 of FIG. 4), an output of a neural network (e.g., an output of the image analysis neural network 402 of FIG. 4), a syntax element, or other coded data of an encoded media bitstream. Some example types of data that may be encoded to generate data element x in neural-based image / video coding (or more generally, a data element to be entropy coded using the techniques of this disclosure) may include RGB or YUV pixel values ​​in an image or video frame, residuals (pixel values ​​minus predictions from motion) of pixel values ​​in motion-compensated video, motion vectors for block-based motion compensation, and / or dense optical flow data for warping-based motion compensation. In other examples, data element x may include data representing weights, activation functions, or other data used by neural-based media compression techniques.

[0060]

[0083] Data is not coded during the training phase, rather the bit rate calculation process 500 calculates the number of bits N used by the arithmetic coding process. b The code vector calculation process 502 uses the data element x and the PDF parameters σ to estimate the PDF c(σ). The vector σ represents the parameters that define the PDF of the data element x that are known by the sender and receiver (e.g., the media encoder 200 and the media decoder 300). The PDF parameters σ may be calculated by a hyperprior decoder network or may be read from a table of predefined parameters that are learned. As an example, the code vector calculation process 502 may calculate the code vector c(σ) of a particular data element x from its corresponding PDF parameters σ.

[0061]

[0084] The quantization process (Q x->q) 504 quantizes the data element x to produce a quantized data element q. An arithmetic encoder 506 then arithmetically codes the quantized data element q using a code vector c(σ) to produce a bit stream B. An arithmetic decoder 508 recovers the quantized data element q by arithmetically decoding it using the same code vector c(σ). The inverse quantization process

[0062]

number

[0063] 510 performs inverse quantization of the quantized data element q to obtain a decoded data element

[0064]

number

[0065] Decode the data element.

[0066]

number

[0067] can then be decoded by neural-based decoding techniques to reconstruct the original media (e.g., pictures or video).

[0068]

[0085] Note that the techniques of this disclosure are applicable to all compression methods that involve neural networks that generate the parameters of the probability distributions used for entropy coding, and also to tables of entropy coding parameters stored using low precision integers. The following sections frequently use examples where σ represents the standard deviation of a Gaussian PDF, but the general techniques of this disclosure are applicable to all cases where the PDF is defined by the parameter σ.

[0069]

[0086] A PDF may be defined by more than one parameter, but not all parameters are directly used for entropy coding. For example, a PDF may be defined by a standard deviation σ and a mean value μ, but only the standard deviation determines the entropy coding characteristics. Since having one parameter is most common in practice, it is the assumption used in the next section, and the techniques described below can be easily generalized.

[0070]

[0087] As shown in Figure 5, a block called Code Vector Calculation Process 502 defines how the PDF is used to create a code vector c(σ), which is the information used to create the bit stream B by the arithmetic coding process. For example, a code vector can be a vector with a cumulative distribution function (CDF) corresponding to the probability mass function (PMF) defined by the quantized symbols (p0, p1, ..., p8 in Figure 3). In another example, if a technique called binarization is used, the code vector c(σ) contains the probability of a binary decision for coding the same data. In practice, the CDF and the probability are scaled so that the elements of the vector c are represented as integers.

[0071]

[0088] The system shown in Figure 5 illustrates how entropy coding is used when neural-based codecs are initially designed, trained, and tested, using highly accurate floating-point arithmetic and powerful processors. However, in practical implementations, neural network values ​​are typically represented as low-precision integers to allow faster computation and smaller power usage. This is shown in the modified system of Figure 6, where the parameter σ is expressed as a low-precision version.

[0072]

number

[0073] , represented as a single 8-bit integer (byte), for example.

[0074]

[0089] Figure 6 shows how the PDF parameter σ is quantized to a low-precision integer.

[0075]

number

[0076] 6 shows a practical implementation of the system of FIG. 5 in which the vectors required for arithmetic coding are pre-computed and stored in an array with C elements. As with the example of FIG. 5, data is not coded during the training phase, rather the bit rate calculation process 600 calculates the number of bits N used by the arithmetic coding process. b We use the data element x and the PDF parameter σ to estimate σ. In the example of Figure 6, the quantization process

[0077]

number

[0078] 602 quantizes the PDF parameter σ and calculates the quantized PDF parameter

[0079]

number

[0080] Then, the code arrangement process 603 creates the quantized PDF parameters

[0081]

number

[0082] From the code vector

[0083]

number

[0084] That is, the quantized PDF parameters

[0085]

number

[0086] is used to determine one of the C elements (eg, code vectors) available to be determined by the code alignment process 603.

[0087]

[0090] The quantization process (Q x->q ) 604 quantizes the data element x to produce a quantized data element q. An arithmetic encoder 606 then generates a code vector

[0088]

number

[0089] quantized data element q is arithmetically coded using the same code vector to produce a bit stream B. The arithmetic decoder 608 uses the same code vector

[0090]

number

[0091] The quantized data element q is restored by arithmetic decoding using

[0092]

number

[0093] 610 performs inverse quantization of the quantized data element q to obtain a decoded data element

[0094]

number

[0095] Decode the data element.

[0096]

number

[0097] can then be decoded by neural-based decoding techniques to reconstruct the original media (e.g., pictures or video).

[0098]

[0091] Note that in Fig. 6, the quantization of x is different from the quantization of the PDF parameter σ. Floating-point parameters may work well for complex neural networks, but are less feasible for widespread use for smaller devices (e.g., mobile devices). Power consumption in such mobile devices may be too high for practical implementation. Therefore, the PDF parameter σ is quantized to an integer for lower power usage and more practical applications. The PDF parameter

[0099]

number

[0100] In one example, x is a 1-byte integer (e.g., 256 possible values). Such quantization results in greater power efficiency.

[0101] Another practical modification is motivated by the fact that recalculating the code vector c(σ) for each value of x is generally computationally expensive. For example, if x has a Gaussian distribution with standard deviation σ (e.g., the typical case), then the PMF of the quantized values ​​is defined by:

[0102]

number

[0103] where the error function (erf) is:

[0104]

number

[0105] The entropy (H) of this PMF in bits is:

[0106]

number

[0107] It is.

[0108]

[0094] Note that in addition to computing the probabilities, an additional task can be performed to create a vector c with the correct integer cumulative sum or binary probabilities, in addition to the computational cost.

[0109]

number

[0110] Code vectors for different quantization values ​​of

[0111]

number

[0112] This can be avoided in a practical implementation by pre-computing the vectors and storing them in an array (e.g., in memory). As shown in the diagram of FIG. 6, the sender and receiver (e.g., the media encoder 200 and the media decoder 300) can use the parameters

[0113]

number

[0114] and have the same sequence of code vectors used for arithmetic encoding and decoding.

[0115] For example, given the following definition:

[0116]

number

[0117] Where:

[0118]

number

[0119] (floor function) is the largest integer less than or equal to σ, and x is a precomputed code vector

[0120]

number

[0121] It can be entropy coded.

[0122]

[0096] The loss incurred by having a limited number of code vectors can be measured using the code redundancy function, which is the average number of additional bits required to encode x using code vector c(r) instead of the optimal code vector c(σ).

[0123]

[0097] The code redundancy function is defined by the Kullback-Leibler divergence measure as follows:

[0124]

number

[0125]

[0098] Because media codecs operate over a wide range of bit rates, entropy coding is typically not designed to minimize the average redundancy. In general, the average value is not known a priori. Instead, entropy coding is designed by bounding the relative redundancy, as follows:

[0126]

number

[0127]

[0099] Designing an entropy coding method with strictly bounded relative redundancy ensures that all averages are also bounded. In other words, in this approach, the average compression loss may not be the smallest, but it is guaranteed to be small when the relative loss is always small.

[0128] FIG. 7 shows an example of a redundancy measure for a Gaussian distribution when the code vector is calculated using the following formula:

[0129]

number

[0130]

[0101] Figure 7 shows an example of absolute (plot 700, right y-axis) and relative (plot 702, left y-axis) measures of redundancy associated with quantized PDF parameters and the use of an integer number of codevectors. This loss is coding efficiency. In essence, more bits than necessary are used. Quantizing the sigma value introduces uncertainty as to the actual value of the floating-point sigma.

[0131]

[0102] It can be assumed that code vectors c(1.5), c(2.5), and c(3.5) are used to entropy code values ​​with PDF parameter σ in intervals [1,2), [2,3), and [3,4), respectively. It can be seen that the redundancy is zero when σ=r, and positive otherwise. Note that in the example of Figure 7, the maximum redundancy in the interval [1,2) is larger than in the other intervals.

[0132]

[0103] Tests have shown that the maximum redundancy in an interval depends on the width of the interval and the values ​​within the interval, which means that better results may be obtained with a non-uniform allocation of code vectors.

[0133]

[0104] Figure 8 shows an example of a non-uniform allocation of code vectors c(σ) designed to obtain the same maximum relative redundancy in all intervals. The relative redundancy graph in Figure 8 can be obtained when code vectors c(1.24), c(1.93), and c(3.14) are used to entropy code values ​​with PDF parameter σ in the intervals [1, 1.52), [1.52, 2.41), and [2.41, 4), respectively. It can be seen that for these values, the maximum relative redundancy is the same in the three intervals.

[0134]

[0105] It should be noted that the technique described above with reference to Figure 8 is related to code vector assignment and not to the quantization of the PDF parameter σ. For example, if σ is directly quantized to an integer value, the fractional interval thresholds shown in Figure 8 cannot be applied.

[0135]

[0106] The same code vector c(r * (σ a ,σ b )) defines the range of σ values ​​to be entropy coded. a ,σ b), a media coder (eg, the media encoder 200 and the media decoder 300) can minimize the maximum relative redundancy within this interval if the following formula is used: r * (σ a ,σ b )={r:L(σ a │r)=L(σ b │r)} (9)

[0136] In the above code vector design criteria, in FIG. a ,σ b ) redundancy in σ=r * (σ a ,σ b ) we can see that it has an approximately parabolic shape with zero value when σ = σ ...

[0137]

number

[0138]

[0108] This average is shown as dashed line 800 in the example of Figure 8. Figure 8 shows the case where several intervals have the same maximum redundancy according to equation (9), and these intervals also have approximately the same average redundancy.

[0139]

[0109] One fact to be considered in identifying problems with the implementation shown in Fig. 6 is that the codec designer may choose the type of PDF and the parameters used for its definition. However, since the parameter values ​​are defined by the network training, it is the practical application of the design that defines the relative frequency of the parameter values. For example, Fig. 9 shows a plot 900 of the relative frequency of Gaussian standard deviation values ​​from a neural-based image codec. As shown in Fig. 9, for low bit rates (most used in practice), the σ values ​​peak at the minimum allowable value σ=0.1, and are almost all smaller than σ=10. Meanwhile, for larger bit rates, larger σ values ​​are more common. This disclosure assumes that the standard deviation is in the range [0.1, 1000] to cover a wide range of playback qualities.

[0140]

[0110] The PDF parameter σ defines the bit rate used by the codec, which is measured by the entropy function defined in equation (4). Figure 10 shows a plot 1000 that illustrates how the entropy of a quantized value varies with the parameter σ within the range of standard deviations desired to be supported. More specifically, Figure 10 shows the entropy (in bits) of a quantized random variable with a Gaussian distribution as a function of the standard deviation σ.

[0141]

[0111] A media coder (e.g., the media encoder 200 or the media decoder 300) quantizes σ from the range [0.1, 1000] to 256 values ​​(that fit into a single byte) and uses equation (9) to obtain

[0142]

number

[0143] If the media coder is configured to create one code vector for each of the 256 intervals defined by C=256, then the media coder obtains the average relative redundancy shown in Figure 11. Figure 11 shows a plot 1100 illustrating the average relative redundancy (compression loss) introduced by using uniform quantization of code vectors with Gaussian standard deviation σ and C=256.

[0144]

[0112] With respect to Figures 10 and 11, the following issues can be observed. 1. The entropy function has very high derivatives at small values ​​of σ (the range most used), complicating learning convergence during training. 2. Entropy is highly nonlinear near the origin, making it difficult to reproduce using quantized neural networks. 3. The important difference for small σ values ​​is

[0145]

number

[0146] 11. With low precision, they cannot be separated, resulting in a very high relative redundancy, shown near the origin in FIG. 4. The range of σ corresponds to the values ​​actually needed, but most of that range is typically rarely used, as shown in Figure 9. In the example shown, the range of small values ​​of σ is the most used and is also where the quantization problem is worst.

[0147]

[0113] In summary, using standard deviation as a value for defining entropy coding may work reasonably well in high precision floating-point implementations, but may not be beneficial for quantized neural networks used in practical implementations.

[0148]

[0114] Whenever a codec designer chooses PDF parameters, they have well-known intuitive meanings, so similar issues arise for other types of probability distributions. High-precision neural networks can "learn" how to optimally use those parameters through extensive training, but quantization networks may not. In practical implementations, the assignment of code vectors is governed by the quantization parameters

[0149]

number

[0150] Since the quantization is done using , the use of non-uniform quantization (eg, as shown in the example of FIG. 8) cannot directly solve these problems.

[0151] The practical problem presented above depends on how the PDF parameter σ shapes the entropy function and on the low-precision integer representation (quantization Q σ The cause of these problems can be difficult to identify from a direct analysis of how equation (4) defines the entropy of a quantized value, since it is an infinite sum of terms with no clear intuitive interpretation. More insight can be gained by considering a simpler approximation. For a Gaussian distribution and large values ​​of σ, the entropy can be approximated by:

[0152]

number

[0153] For small values ​​of σ,

[0154]

number

[0155]

[0116] Figure 12 shows the entropy graph 1200 of Figure 10 when a logarithmic scale is used for the parameter σ, together with the approximation defined above. In both, rapid asymptotic convergence can be observed, with very good agreement being observed in the appropriate range of σ, and in the intermediate range where both approximations are nearly identical.

[0156]

[0117] From Figure 12, it can be more easily recognized that the logarithm of σ provides a more direct relationship to entropy and may therefore be a better parameter for defining the PDF. One remaining aspect to be considered is that, as shown in Figure 9, small σ values ​​are expected to be most common, and such small values ​​of σ correspond to very small entropy.

[0157]

[0118] Using a logarithmic scale for both entropy and standard deviation, as shown in Figure 13, shows that it may be beneficial to also consider how the entropy function changes at low values ​​of σ. There may be cases where the media encoder 200 or the media decoder 300 may encounter a very large number of values ​​with very low entropy, so that even a very small absolute error may result in a large relative redundancy and a large compression loss. Figure 13 shows a plot 1300 showing the entropy and its approximation from equations (11) and (12), graphed on a logarithmic scale.

[0158]

[0119] Even though quantization errors are unavoidable, the techniques disclosed herein are based on the fact that it is possible to reduce compression loss by permuting variables to be quantized, taking into account how quantization errors affect relative redundancy.

[0159]

[0120] One exemplary technique is shown in Figure 14, where a new variable called v is defined to parameterize the PDF. In general, the PDF parameter v may be determined using a nonlinear function (e.g., logarithm) of the PDF parameter σ. The conversion function between σ and v is defined as follows: σ=Tv→σ (v) (13)

[0160]

[0121] Figure 14 shows a modification of the actual codec of FIG. 6 that enables new design variables optimized for entropy coding and tables for more efficient allocation of code vectors. As shown in FIG. 14, this function allows the codec designer to continue using the PDF parameter σ in training, and this function is assumed to be differentiable almost everywhere in order to maintain automatic gradient backpropagation.

[0161]

[0122] For simplicity of explanation, the following rules are adopted.

[0162]

Equation

[0163]

[0123] Equation (14) specifies that σ ranges from the minimum value (σ min ) to the maximum value (σ max ). The new parameter v ranges from 0 to 1. The conversion function T v→σ for v value 0 is equal to σ min . The conversion function T v→σ for v value 1 is equal to σ max . Quantization of the new parameter v (e.g., Q v→n (v)) generates integers in the range of 0 to N - 1. Further, when v0 < v1, it is defined that T v→σ (v0) < T v→σ (v1).

[0164]

[0124] In FIG. 14, the data is not encoded during the training phase. Since the coding phase is configured to use the new PDF parameter v, the training phase performs a conversion (T v→σ) process 1400. The bit rate calculation process 1401 uses the data elements x and the PDF parameters σ to calculate the number of bits N used by the arithmetic coding process. b Estimate.

[0165] In the example of FIG. 14, the quantization process (Q v→n ) 1402 quantizes the PDF parameters v to produce quantized PDF parameters n. A code selection process 1412 then converts the quantized PDF parameters n to a code vector index i. The code vector index i may be an input to a code ordering process 1414. That is, the code vector index i may be an index into a lookup table of code vectors. The code ordering process 1414 then quantizes the code vector c(r * i ) is determined. * i ) is used to arithmetically code the data element x.

[0166]

[0126] The quantization process (Q x->q ) 1404 quantizes the data element x to produce a quantized data element q. Then, an arithmetic encoder 1406 quantizes the code vector c(r * i ) to arithmetically code the quantized data elements q to produce a bitstream B. The arithmetic decoder 1408 uses the same code vector c(r * i ) to recover the quantized data element q.

[0167]

number

[0168] 1410 performs inverse quantization of the quantized data element q to obtain a decoded data element

[0169]

number

[0170] Decode the data element.

[0171]

number

[0172] can then be decoded by neural-based decoding techniques to reconstruct the original media (e.g., pictures or video).

[0173]

[0127] Generally, in one example of the present disclosure, the media encoder 200 and the media decoder 300 may be configured to determine a probability distribution function parameter (e.g., v) for a data element (x) of a data stream coded by a neural-based media compression technique. The probability distribution function parameter (v) may be determined from a logarithmic function (e.g., natural logarithm (ln)) of the standard deviation (σ) of the probability distribution function of the data stream. The media encoder 200 and the media decoder 300 may determine a code vector (c(r * i )) and * i )) can be used to arithmetically code a data element (x).

[0174] 14, in one example, the media encoder 200 and the media decoder 300 may be configured to quantize the probability distribution function parameter v before determining the code vector. In one example, to determine the code vector, the media encoder 200 and the media decoder 300 may be configured to determine a code vector index i based on the probability distribution function parameter v, and determine the code vector from a table of code vectors based on the code vector index. More information regarding the table of code vectors is provided below.

[0175] From the encoder side, the media encoder 200 may be configured to generate the data element x using a neural-based compression technique, and may quantize the data element to create a quantized data element. In this example, arithmetically coding the data element using the code vector includes arithmetically encoding the quantized data element using the code vector. Furthermore, in this example, the media encoder 200 may be configured to process the image or video picture using an image analysis neural network to generate the data element, e.g., as shown in FIG. 4.

[0176]

[0130] From the decoder side, the media decoder 300 may decode the encoded data elements using the code vectors to create quantized data elements. The media decoder 300 may further dequantize the quantized data elements to create reconstructed data elements. The media decoder 300 may then process the reconstructed data elements using an image synthesis neural network to reconstruct an image or video picture.

[0177]

[0131] The next aspect to be considered is that the performance of the entropy coding is measured by the relative redundancy measure defined in equations (7) and (10). In the following, we give an example of a practical application of this approach.

[0178]

[0132] Logarithmic transformation

[0133] Most commonly, neural-based codecs are configured to operate using the assumption x̂N(0,σ), i.e., the variables to be coded have a zero-mean Gaussian (or normal) distribution. Some exemplary hyperprior distribution networks use some learned PDFs, but more recent work has shown that there are improvements when the learned PDFs are replaced with Gaussian PDFs and learned parameters.

[0179]

[0134] Using this fact and the observation in Figures 12 and 13 that the logarithm of the standard deviation provides a more accurate parameterization of the PDF, a variable transformation function can be defined using: σ=T v→σ )=exp(v ln(σ max )+(1-v)ln(σ min )),(15) And vice versa,

[0180]

number

[0181]

[0135] The function ln() is the natural logarithm. When this new parameter v is quantized using N=256, the corresponding C=256 code vector is optimized according to equation (9), resulting in the average relative redundancy shown in Figure 15. Figure 15 shows a plot 1500 illustrating the average relative redundancy (loss in compression) caused by using uniform quantization of PDF parameter v with code vectors of N=256 and C=256. Figure 15 shows that PDF parameter v is a better parameter for defining a Gaussian PDF, since the relative redundancy is always less than 1%.

[0182]

[0136] The transformation equations (15) and (16) shown above use only exponential and logarithmic functions, which are continuous and have derivatives that are easy to calculate during design and training.

[0183] Table for selecting code vectors

[0138] One additional aspect to consider is that since the code vectors are constantly accessed during entropy coding, it may be preferable to store them in more expensive, faster memory in order to maintain high data throughput. Thus, reducing the number of code vectors allows savings in memory requirements and costs.

[0184]

[0139] Savings in memory requirements and costs can be achieved using code selection tables 1600 shown in Figure 16 after quantization of the PDF parameters v. These are simple tables with N elements, which allow for a non-uniform allocation of code vectors, similar to the example in Figure 8, but instead of using floating-point values ​​that define the transitions based on the PDF parameters σ, the tables are applied directly to the integer values ​​n obtained after quantization of the PDF parameters v.

[0185]

[0140] The advantage is that this process is an integer-to-integer conversion that can be done very efficiently with a simple table lookup implementation. Figure 16 shows an example of such a table 1600 that defines, for each of the 256 possible values ​​of n, the number of code vectors to be used for entropy coding. In the example of Figure 16, the total table memory is 256 bytes. The media encoder 200 and the media decoder 300 can use the table 1600 of Figure 16 that is used to convert the PDF parameters v quantized by the value n, where N=256, to code vector indexes i. The table 16 is designed to better guarantee a maximum redundancy of less than 1%, as shown in Figure 17, thus allowing a reduction in the number of code vectors from 256 to 43. Figure 17 shows a plot 1700 showing the relative redundancy obtained according to the standard deviation.

[0186]

[0141] Another way to limit memory usage is to reduce the range of standard deviation values. For example, the neural base may be implemented using the following PDF parameters: σ min =0.112,σ max =334.5⇒ln(σ max / σ min )=8, (17) This corresponds to the following transformation:

[0187]

number

[0188] Here, the multiplication by 32 can be implemented via bit shifting.

[0189] FIG. 18 shows a code vector selection table 1800 designed for the above transformation that reduces the number of code vectors to 25.

[0190]

[0143] The modified entropy coding scheme of Fig. 14 and the PDF parameter transformation of equation (16) are effective in reducing the degradation caused by the quantization of the PDF parameters. In fact, the above-mentioned technique allows moving from the unacceptable redundancy values ​​shown in Fig. 11 to the useful redundancy values ​​shown in Fig. 15.

[0191]

[0144] However, it can also be observed that the redundancy values ​​in FIG. 15 are larger for small values ​​of σ (the most common and important values ​​of σ) and may be unnecessarily small for large, less important values ​​of σ.

[0192]

[0145] If it is desired to use a number of code vectors C equal to the number of quantization intervals N, then one obtains the redundancy curve 1900 shown in Figure 19. Figure 19 shows the relative redundancy introduced by quantization of the PDF parameter v defined by equation (16), dividing the interval [0,1] into N equal-sized intervals (uniform quantization), and using C = N code vectors. The redundancy curves 1900 in Figure 19 are approximately parallel (as will be explained in more detail below), which also means that small values ​​of σ have the worst case relative redundancy, which rises rapidly as C = N is reduced.

[0193]

[0146] Although the code vector selection method described above can alleviate this problem, it may have the following limitations. 1) The relative redundancy for small values ​​of σ corresponds to the worst case, so that the number of quantization intervals N is relatively large. 2) Memory reduction can be achieved by having the number of code vectors, C, smaller than N, which can be addressed using the addition of a table lookup stage, as described above. 3) The optimal tables and code vectors depend on both N and C and can be recalculated when these parameters change.

[0194]

[0147] The problem with the variable transformation defined by equation (16) is caused by the fact that the choice of parameters is motivated by, and provides, simplicity and convenience. This disclosure shows that, using the theory presented above, a transformation function can be defined that provides even better control of relative redundancy.

[0195]

[0148] This disclosure defines another PDF parameter u to represent a new parameter that defines an entropy coding PDF. In general, the PDF parameter u is a parameterization of a distribution (e.g., a distribution of code vectors) that is optimized for quantization. Or, more generally, the parameter u is optimized to provide a lower relative redundancy (e.g., loss) compared to other parameters. For simplicity of notation, this disclosure defines the parameter transformation process using the following two functions: σ=T u→σ (u)=T λ→σ (T u→λ (u)),(19) however, T λ→σ (λ)=exp(λ ln(σ max / σ min )+ln(σ min )),(20) This is similar to the transformation (15) above.

[0196]

[0149] Equations (19) and (20) are assumed to satisfy the following rules:

[0197]

[0150]

number

[0198]

[0151] Equation (21) is a formula for determining whether σ is a minimum value (σ min ) to the maximum value (σ max ) The new parameter u is in the range 0 to 1. The transformation function T with a λ value of 0 λ→σ is σ min The transformation function T with a lambda value of 1 is λ→σ is σ max The transformation function T with u value 0 is equal to u→λ ) is equal to 0. The transformation function T with a u value of 1 u→λ ) is equal to 1. The new parameter u (e.g., Q u→n Quantization of (u) produces integers in the range 0 to N-1.

[0199]

[0152] The example in FIG. 8 uses the code vector optimization of equation (9) to optimize the interval [σ a ,σ b We show that it is possible to design a non-uniform partition of σ σ ≠ σ σ ≠ σ σ , which obtains the same maximum relative redundancy in all subintervals. As noted above, this approach may not be practical when the parameter σ is quantized, but the techniques of this disclosure exploit the fact that a non-uniform partition can be adopted when the parameter u is quantized.

[0200] FIG. 20 shows the PDF parameter interval [σ min ,σ max 20 shows a graph representation 2000 of the process of splitting [. This can be done using the following algorithm:

[0201] Algorithm I 1.N and [σ min ,σ max ], choose an initial maximum redundancy ε and t0=σ 最小 Initialize 2. For n=1,2,...,N: a.

[0202]

number

[0203] So that

[0204]

number

[0205] Request. b.

[0206]

number

[0207] So that t n Request. 3.

[0208]

number

[0209] If so, stop. 4.t N <σ max If so, increase ε; otherwise, decrease ε using a method for one-dimensional search (such as bisection). 5. Proceed to step 2.

[0210] As shown in the lower part of FIG.

[0211]

number

[0212] is a function T u→σ N values ​​for (u) can be provided, and intermediate values ​​can be calculated via interpolation. Because the N quantization intervals are selected to have the same maximum relative redundancy, the media encoder 200 and the media decoder 300 may be configured to use the same number of code vectors, i.e., C=N, eliminating the need for a code selection table and allowing for the simplified codec implementation shown in FIG.

[0213] In Fig. 21, data is not coded during the training phase. Since the coding phase is configured to use the new PDF parameters u, the training phase is implemented by a transformation (T u→σ ) process 2100 (e.g., the standard deviation of the PDF). A bit rate calculation process 2101 uses the data elements x and the PDF parameters σ to calculate the number of bits N used by the arithmetic coding process. b Estimate.

[0214] In the example of FIG. 21, the quantization process (Q u→n ) 2102 quantizes the PDF parameters u to create quantized PDF parameters n. Then, a code array process 2112 converts the quantized PDF parameters n into a code array c(r * n )(code vector c(r * n ) into a code vector c(r * n ) is used to arithmetically code the data element x.

[0215]

[0158] The quantization process (Q x->q ) 2104 quantizes the data element x to produce a quantized data element q. Then, an arithmetic encoder 2106 quantizes the code vector c(r * n ) to arithmetically code the quantized data elements q to produce a bitstream B. The arithmetic decoder 2108 uses the same code vector c(r * i ) to recover the quantized data element q.

[0216]

number

[0217] 2110 performs inverse quantization of the quantized data element q to obtain a decoded data element

[0218]

number

[0219] Decode the data element.

[0220]

number

[0221] can then be decoded by neural-based decoding techniques to reconstruct the original media (e.g., pictures or video).

[0222]

[0159] Generally, in one example of the present disclosure, the media encoder 200 and the media decoder 300 may be configured to determine a probability distribution function parameter (e.g., u) for a data element (x) of a data stream coded by a neural-based media compression technique. The probability distribution function parameter (u) may be based on a distribution optimized for quantization. The distribution function parameter u may be determined according to an algorithm that measures coding redundancy or may be defined by solving an ordinary differential equation. The media encoder 200 and the media decoder 300 may determine a code vector (c(r * n ), and may be further configured to determine a code vector (c(r * n )) can be used to arithmetically code a data element (x).

[0223]

[0160] As shown in Figure 21, in one example, the media encoder 200 and the media decoder 300 may be configured to quantize the probability distribution function parameter u(n) before determining the code vector. In one example, to determine the code vector, the media encoder 200 and the media decoder 300 calculate the code vector (c(r * n ))

[0224] From the encoder side, the media encoder 200 may be configured to generate the data element x using a neural-based compression technique, and may quantize the data element to create a quantized data element. In this example, arithmetically coding the data element using the code vector includes arithmetically encoding the quantized data element using the code vector. Furthermore, in this example, the media encoder 200 may be configured to process the image or video picture using an image analysis neural network to generate the data element, e.g., as shown in FIG. 4.

[0225]

[0162] From the decoder side, the media decoder 300 may decode the encoded data elements using the code vectors to create quantized data elements. The media decoder 300 may further dequantize the quantized data elements to create reconstructed data elements. The media decoder 300 may then process the reconstructed data elements using an image synthesis neural network to reconstruct an image or video picture.

[0226] One possible caveat of the above technique is that the transformation function T u→σ The fact that (u) theoretically depends significantly on the number of quantization intervals N means that the hyper-prior neural network needs to be trained specifically for that value.

[0227] However, the function T calculated for different values ​​of N u→σ (u) produced the same graph 2200 shown in FIG. 22. In fact, as shown in FIG. 23, the function calculated for some values ​​of N is

[0228]

number

[0229] The difference between them is remarkably small. Figure 22 shows the transformation function T u→σ (u) and T u→λ Graph 2300 in FIG. 23 shows the function T in FIG. 22 calculated using algorithm I above. u→λ We show the difference between the value of (u) and different values ​​of the algorithm parameter N.

[0230] This means that for all practical values ​​of N there is a single transformation function T u→σ (u), which means that the value of N does not need to be defined when training the hyper-prior neural network.

[0231]

[0166] Function T for N→∞ u→σ The asymptotic value of (u) can be calculated by considering that in the limit, the curvature of the relative redundancy defined by the parameter u should be a constant value, i.e.

[0232]

number

[0233] where k is a constant. This can be seen intuitively in the bottom part of Figure 22, and means that in the limit, all redundancy curves have the same shape.

[0234]

[0167] Probability defined by parameter σ

[0235]

number

[0236] It can be shown that for any infinite sequence of , we have:

[0237]

number

[0238] For example, if we use the distribution of the quantized normal variables corresponding to equation (2) and define

[0239]

number

[0240] It can be shown that equation (23) corresponds to

[0241]

number

[0242] and equation (21) corresponds to the ordinary differential equation (ODE):

[0243]

number

[0244] Or equivalently, the following equation may hold:

[0245]

number

[0246]

[0169] These differential equations can be solved by simple methods such as Runge-Kutta, and the transformation function T u→σ For an accurate calculation of (u), generate a discrete set of points that can be used with an interpolation method such as cubic splines.

[0247]

[0170] The solution arrays from Algorithm I and the ODE were represented as cubic splines defined using equations (26) or (27), and the boundary derivatives were used to measure the relative redundancy when the interval [0,1] was uniformly divided into N=C intervals (a different value of N than used in Algorithm I). The results are shown in graph 2400 of Figure 24, where it can be observed that the resulting average redundancy is indeed substantially constant. Figure 24 shows the relative redundancy resulting from dividing the interval [0,1] into N equal-sized intervals (uniform quantization) and using C=N code vectors, using the PDF parameter u and the transformation function of Figure 22.

[0248]

[0171] Furthermore, Figure 24 also shows that doubling the interval size results in a fourfold increase in the average redundancy value, which is further evidence that over a wide range of scales, the redundancy curves in Figure 20 are approximately quadratic and governed by the curvature defined by equation (25).

[0249] As another example, if the neural network and entropy coding are designed assuming a Laplace distribution, instead of normal (Gaussian), with a PDF such that

[0250]

number

[0251] Equation (24) can then be replaced by:

[0252]

number

[0253] Then, the corresponding optimal conversion functions are designed using equations (25-27).

[0254]

[0173] As mentioned above, we have assumed that the codec designer always wants to have the same maximum relative redundancy to obtain the same overall average. u→σ The calculation of (u) can be generalized to produce a non-uniform distribution of the relative redundancy.

[0255]

[0174] This can be done by replacing the constant ε in Algorithm I with a function ε(σ) that defines how the redundancy should vary with the PDF parameter σ. Similarly, the constant k in differential equations (26) and (27) can be replaced with a function k(σ). Figures 25 and 26 show examples of results obtained using these modifications, where two types of functions shape the redundancy tolerance.

[0256]

[0175] Figure 25 shows a first set 2500 of examples of average relative redundancy resulting from uniform quantization of the PDF parameter u with varying tolerance for redundancy. Figure 26 shows a second set 2600 of examples of average relative redundancy resulting from uniform quantization of the PDF parameter u with varying tolerance for redundancy.

[0257]

[0176] The simplified codec of FIG. 21 can be used to train the neural network using a parameter transformation function T u→σ We only need to calculate (u). However, since gradients are often beneficial for neural network optimization, it may be better to use an approximation that is continuous and has a continuous first derivative.

[0258]

[0177] As follows, T u→σ There are many mathematical methods defined for function approximation that can be used to approximate (u). 1.Polynomial 2. Rational polynomial 3. Piecewise polynomial 4. Splines One aspect to consider when choosing an approximation is the function T u→σThe fact is that (u) is in the limit defined by a differential equation, so the approximation error should be measured by calculating the relative difference between the derivative of the approximation and the exact value defined by equation (26) or (27). For example, in FIG. 22, the derivative 2u

[0259]

number

[0260]

[0178] Another example is the following third order polynomial approximation:

[0261]

number

[0262] Minimize the maximum relative error of the derivative shown in Figure 28. Figure 28 shows an example graph 2800 of the relative error of the PDF transfer function approximation using the third order polynomial of Equation (30). Using this polynomial for parameter transformation results in the average redundancy shown in graph 2900 of Figure 29. Figure 29 shows the relative redundancy introduced by quantizing the PDF parameter u, dividing the interval [0,1] into N equal sized intervals (uniform quantization), and using C=N code vectors and the approximation of Equation (30). One can observe the correspondence between the peaks in Figure 28 mapped to the corresponding peaks in Figure 29.

[0263]

[0179] It can also be observed that the relatively large relative differential errors in Figure 28 cause similar deviations to the maximum redundancy in Figure 29, but they are smaller than the variations caused by halving N and C, meaning that functions as simple as the polynomials in equation (30) can, in fact, be used in practical applications.

[0264]

[0180] Other approximations include:

[0265]

number

[0266]

[0181] The techniques of this disclosure may be implemented and tested using arithmetic coding and incorporated into a neural-based codec running on a mobile device. In one example, the arithmetic coding technique may be implemented using 32-bit arithmetic and the probability values ​​may be scaled to 16 bits.

[0267]

[0182] Table 1 below shows some experimental results used to verify the performance of the techniques of this disclosure. The hyperprior distribution data was obtained using floating-point arithmetic so that only the quantization of the entropy coding parameters affected the results.

[0268]

[0183] As shown in the first row of the results in Table 1, a quantized PDF parameter σ is used to parameterize a Gaussian PDF, and approximately 0.5 MB of memory is required to store the 256 code vectors, and the resulting redundancy approaches 1000%, as expected from Figure 11. Better results are possible, but would require more memory for the code vectors.

[0269]

[0184] The following rows of Table 1 show that much better results are obtained if we replace σ with the quantized PDF parameters v (e.g., parameters obtained using the logarithm function of σ as defined in equation (16)). Using N=C=256 does indeed remove the redundancy, but the amount of memory for the code vectors is relatively large.

[0270]

[0185] The results show that using smaller values of N = C helps reduce the amount of memory required, but the relative redundancy increases non-uniformly. On the other hand, using the above-mentioned table lookup technique where C < N = 256 (e.g., less than 256 code vectors), about half of the memory is used with more consistent relative redundancy.

[0271]

[0186] The last three rows of Table I show that using the PDF parameter u and the conversion function of FIG. 22 also reduces the memory requirements and provides the most consistent results, but do not require table lookup.

[0272]

[0187]

Table 1

[0273]

[0188] As shown above, in the case of image and video coding using neural networks, selecting the most intuitive and convenient parameters for entropy coding for the designer can function when using a high-precision floating-point implementation form, but may cause serious degradation when the weights and activations of the neural network are quantized. In fact, the analysis of the generally used Gaussian distribution shows that the worst degradation of compression efficiency (the highest relative redundancy) occurs exactly in the most common cases obtained during compression.

[0274]

[0189] The proposed technique of the present disclosure redefines the parameters used for entropy coding and maintains the training process with a single change of variables, but addresses the problem by modifying quantization to equalize the redundancy increase resulting from quantization. Further, it is shown that using the new parameterization, it is possible to reduce the amount of memory used for entropy coding using a simple table lookup for code vector selection or other types of parameter conversion functions that do not use table lookup.

[0275]

[0190] Figure 30 is a flow chart illustrating an example method of the present disclosure. The technique of Figure 30 may be performed by one or more components of the media encoder 200 and / or the media decoder 300, including the components shown in the example of Figure 14. In the example of Figure 30, the media encoder 200 and / or the media decoder 300 may be configured to determine probability distribution function parameters for data elements of a data stream coded by a neural-based media compression technique, the probability distribution function parameters being based on a distribution of the data stream optimized for quantization (3000), determine a code vector based on the probability distribution function parameters (3002), and entropy code the data elements using the code vector (3004). This function may be determined empirically (e.g., logarithmically) or by using an algorithm that evaluates coding redundancy or a solution of an ordinary differential equation.

[0276]

[0191] Other exemplary aspects of the present disclosure are described below.

[0277]

[0192] Aspect 1A - A method for coding media data, the method including determining probability distribution function parameters for data elements of a data stream coded by a neural-based media compression technique, the probability distribution function parameters being a logarithmic function of a standard deviation of the probability distribution function of the data stream, determining a code vector based on the probability distribution function parameters, and entropy coding the data elements using the code vector.

[0278]

[0193] Aspect 2A - The method of aspect 1A, further comprising quantizing probability distribution function parameters prior to determining the code vector.

[0279] Aspect 3A - The probability distribution function parameter is u, the standard deviation of the probability distribution function is σ, and the minimum standard deviation is σmin and the maximum standard deviation is σ max and the relationship between u and σ is σ=T u→σ (u)=T λ→σ (T u→λ (u)), where T λ→σ (λ)=exp(λ ln(σ max / σ min )+ln(σ min )) where the function T is defined according to an algorithm that measures coding redundancy or by solving an ordinary differential equation.

[0280]

[0195] Aspect 4A - A method described in any of aspects 1A to 3A, further comprising generating data elements using a neural-based compression technique and quantizing the data elements to create quantized data elements, wherein entropy coding the data elements using the code vector comprises entropy encoding the quantized data elements using the code vector.

[0281]

[0196] Aspect 5A - The method of aspect 4A, wherein generating data elements using neural-based compression techniques includes processing images or video pictures using an image analysis neural network to generate the data elements.

[0282]

[0197] Aspect 6A - A method as described in any of aspects 1A to 3A, wherein entropy coding the data elements using the code vector includes entropy decoding the data elements encoded using the code vector to create quantized data elements, and the method further includes dequantizing the quantized data elements to create reconstructed data elements.

[0283]

[0198] Aspect 7A - The method of aspect 6A, further comprising processing the reconstructed data elements using an image synthesis neural network to reconstruct an image or video picture.

[0284]

[0199] Aspect 8A - An apparatus configured to code media data, the apparatus comprising a memory and one or more processors in communication with the memory, the one or more processors being configured to determine probability distribution function parameters for data elements of a data stream coded by a neural based media compression technique, the probability distribution function parameters being a logarithmic function of a standard deviation of the probability distribution function of the data stream, determine a code vector based on the probability distribution function parameters, and entropy code the data elements using the code vector.

[0285]

[0200] Aspect 9A-The apparatus of aspect 8A, wherein the one or more processors are further configured to quantize the probability distribution function parameters prior to determining the code vector.

[0286] Aspect 10A - The probability distribution function parameter is u, the standard deviation of the probability distribution function is σ, and the minimum standard deviation is σ min and the maximum standard deviation is σ max and the relationship between u and σ is σ=T u→σ (u)=T λ→σ (T u→λ (u)), where T λ→σ (λ)=exp(λ ln(σ max / σ min )+ln(σ min )) where the function T is defined according to an algorithm that measures coding redundancy or by solving an ordinary differential equation.

[0287]

[0202] Aspect 11A - An apparatus described in any of aspects 8A to 10A, wherein the one or more processors are further configured to generate the data elements using a neural-based compression technique and quantize the data elements to create quantized data elements, and entropy coding the data elements using the code vector includes entropy encoding the quantized data elements using the code vector.

[0288]

[0203] Aspect 12A - The apparatus described in Aspect 11A, wherein to generate data elements using neural-based compression techniques, the one or more processors are further configured to process images or video pictures using an image analysis neural network to generate the data elements.

[0289]

[0204] Aspect 13A - The apparatus of aspect 12A, further comprising a camera configured to capture an image or video picture.

[0290]

[0205] Aspect 14A - An apparatus described in any of aspects 8A to 10A, wherein entropy coding the data elements using the code vector includes entropy decoding the data elements encoded using the code vector to create quantized data elements, and the one or more processors are further configured to dequantize the quantized data elements to create reconstructed data elements.

[0291]

[0206] Aspect 15A-The apparatus described in aspect 14A, wherein the one or more processors are further configured to process the reconstructed data elements using an image synthesis neural network to reconstruct an image or video picture.

[0292]

[0207] Aspect 16A - The apparatus of aspect 15A, further comprising a display configured to display an image or video picture.

[0293]

[0208] Aspect 17A - A non-transitory computer-readable storage medium storing instructions that, when executed, are configured to cause one or more processors to determine probability distribution function parameters for data elements of a data stream coded using a neural-based media compression technique, the probability distribution function parameters being a logarithmic function of the standard deviation of the probability distribution function of the data stream, determine a code vector based on the probability distribution function parameters, and entropy code the data elements using the code vector.

[0294] Aspect 18A - The probability distribution function parameter is u, the standard deviation of the probability distribution function is σ, and the minimum standard deviation is σ min and the maximum standard deviation is σ max and the relationship between u and σ is σ=T u→σ (u) = Tλ → σ(T u→λ (u)), where T λ→σ (λ)=exp(λ ln(σ max / σ min )+ln(σ min 20. The non-transitory computer-readable storage medium of claim 17, wherein the function T is defined according to an algorithm that measures coding redundancy or by solving an ordinary differential equation.

[0295]

[0210] Aspect 1B - A method for coding media data, the method including determining probability distribution function parameters for data elements of a data stream coded by a neural-based media compression technique, the probability distribution function parameters being parameters of a distribution of the data stream optimized for quantization, determining a code vector based on the probability distribution function parameters, and entropy coding the data elements using the code vector.

[0296]

[0211] Aspect 2B - The method of aspect 1B, further comprising quantizing probability distribution function parameters prior to determining the code vector.

[0297] Aspect 3B - The probability distribution function parameter is u, the standard deviation of the probability distribution function is σ, and the minimum standard deviation is σ min and the maximum standard deviation is σ max and the relationship between u and σ is σ=T u→σ (u) = Tλ → σ(T u→λ (u)), where T λ→σ (λ)=exp(λ ln(σ max / σ min )+ln(σ min )) where the function T is defined according to an algorithm that measures coding redundancy or by solving an ordinary differential equation.

[0298]

[0213] Aspect 4B - The method of aspect 1B, wherein the probability distribution function parameter is a function of the standard deviation of the probability distribution function of the data stream.

[0299]

[0214] Aspect 5B - Generating data elements using neural-based compression techniques; The method of embodiment 1B, further comprising: quantizing the data elements to create quantized data elements, and wherein entropy coding the data elements using the code vector comprises entropy encoding the quantized data elements using the code vector.

[0300]

[0215] Aspect 6B - The method of aspect 5B, wherein generating data elements using neural-based compression techniques includes processing images or video pictures using an image analysis neural network to generate the data elements.

[0301]

[0216] Aspect 7B - The method of aspect 6B, further comprising capturing an image or video picture using a camera.

[0302]

[0217] Aspect 8B - The method described in aspect 1B, wherein entropy coding the data elements using the code vector includes entropy decoding the data elements encoded using the code vector to create quantized data elements, and the method further includes dequantizing the quantized data elements to create reconstructed data elements.

[0303] Aspect 9B

[0219] The method of aspect 8B, further comprising processing the reconstructed data elements using an image synthesis neural network to reconstruct an image or video picture.

[0304]

[0220] Aspect 10B - The method of aspect 9B, further comprising displaying the image or video picture.

[0305]

[0221] Aspect 11B - An apparatus configured to code media data, the apparatus comprising a memory and one or more processors in communication with the memory, the one or more processors configured to determine probability distribution function parameters for data elements of a data stream coded by a neural based media compression technique, the probability distribution function parameters being parameters of a distribution of the data stream optimized for quantization, determine a code vector based on the probability distribution function parameters, and entropy code the data elements using the code vector.

[0306]

[0222] Aspect 12B - The apparatus of aspect 11B, wherein the one or more processors are further configured to quantize the probability distribution function parameters before determining the code vector.

[0307] Aspect 13B - The probability distribution function parameter is u, the standard deviation of the probability distribution function is σ, and the minimum standard deviation is σ min and the maximum standard deviation is σ max and the relationship between u and σ is σ=Tu→σ (u)=T λ→σ (T u→λ (u)), where T λ→σ (λ)=exp(λ ln(σ max / σ min )+ln(σ min )) where the function T is defined according to an algorithm that measures coding redundancy or by solving an ordinary differential equation.

[0308]

[0224] Aspect 14B - The apparatus of aspect 11B, wherein the probability distribution function parameter is a function of the standard deviation of the probability distribution function of the data stream.

[0309]

[0225] Aspect 15B - The apparatus described in aspect 11B, wherein the one or more processors are further configured to generate the data elements using a neural-based compression technique and quantize the data elements to create quantized data elements, and entropy coding the data elements using the code vector includes entropy encoding the quantized data elements using the code vector.

[0310]

[0226] Aspect 16B - The apparatus described in aspect 15B, wherein to generate data elements using neural-based compression techniques, the one or more processors are further configured to process images or video pictures using an image analysis neural network to generate the data elements.

[0311]

[0227] Aspect 17B - The apparatus of aspect 16B, further comprising a camera configured to capture an image or video picture.

[0312]

[0228] Aspect 18B - The apparatus described in aspect 11B, wherein entropy coding the data elements using the code vector includes entropy decoding the data elements encoded using the code vector to create quantized data elements, and the one or more processors are further configured to dequantize the quantized data elements to create reconstructed data elements.

[0313]

[0229] Aspect 19B - The apparatus of aspect 18B, wherein the one or more processors are further configured to process the reconstructed data elements using an image synthesis neural network to reconstruct an image or video picture.

[0314]

[0230] Aspect 20B - The apparatus of aspect 19B, further comprising a display configured to display an image or video picture.

[0315]

[0231] Aspect 1C - A method for coding media data, the method including determining probability distribution function parameters for data elements of a data stream coded by a neural-based media compression technique, the probability distribution function parameters being parameters of a distribution of the data stream optimized for quantization, determining a code vector based on the probability distribution function parameters, and entropy coding the data elements using the code vector.

[0316]

[0232] Aspect 2C - The method of aspect 1C, further comprising quantizing probability distribution function parameters prior to determining the code vector.

[0317] Aspect 3C - The probability distribution function parameter is u, the standard deviation of the probability distribution function is σ, and the minimum standard deviation is σ min and the maximum standard deviation is σ max and the relationship between u and σ is σ=T u→σ (u)=T λ→σ (T u→λ(u)), where T λ→σ (λ)=exp(λ ln(σ max / σ min )+ln(σ min )) where the function T is defined according to an algorithm that measures coding redundancy or by solving an ordinary differential equation.

[0318]

[0234] Aspect 4C - The method of any of Aspects 1C to 2C, wherein the probability distribution function parameter is a function of the standard deviation of the probability distribution function of the data stream.

[0319]

[0235] Embodiment 5C - Generating data elements using neural-based compression techniques; The method of any of aspects 1C to 4C, further comprising: quantizing the data elements to create quantized data elements, and wherein entropy coding the data elements using the code vector comprises entropy encoding the quantized data elements using the code vector.

[0320]

[0236] Aspect 6C - The method of aspect 5C, wherein generating data elements using neural-based compression techniques includes processing images or video pictures using an image analysis neural network to generate the data elements.

[0321]

[0237] Aspect 7C - The method of aspect 6C, further comprising capturing an image or video picture using a camera.

[0322]

[0238] Aspect 8C - A method as described in any of aspects 1C to 4C, wherein entropy coding the data elements using the code vector includes entropy decoding the data elements encoded using the code vector to create quantized data elements, and the method further includes dequantizing the quantized data elements to create reconstructed data elements.

[0323] Aspect 9C

[0240] The method of aspect 8C, further comprising processing the reconstructed data elements using an image synthesis neural network to reconstruct an image or video picture.

[0324]

[0241] Aspect 10C - The method of aspect 9C, further comprising displaying the image or video picture.

[0325]

[0242] Aspect 11C - An apparatus configured to code media data, the apparatus comprising a memory and one or more processors in communication with the memory, the one or more processors being configured to determine probability distribution function parameters for data elements of a data stream coded by a neural based media compression technique, the probability distribution function parameters being parameters of a distribution of the data stream optimized for quantization, determine a code vector based on the probability distribution function parameters, and entropy code the data elements using the code vector.

[0326]

[0243] Aspect 12C - The apparatus of aspect 11C, wherein the one or more processors are further configured to quantize the probability distribution function parameters before determining the code vector.

[0327] Aspect 13C - The probability distribution function parameter is u, the standard deviation of the probability distribution function is σ, and the minimum standard deviation is σ min and the maximum standard deviation is σ max and the relationship between u and σ is σ=T u→σ (u)=T λ→σ (T u→λ (u)), where T λ→σ (λ)=exp(λ ln(σ max / σ min )+ln(σ min)) where the function T is defined according to an algorithm that measures coding redundancy or by solving an ordinary differential equation.

[0328]

[0245] Aspect 14C - The apparatus of any of aspects 11C to 12C, wherein the probability distribution function parameter is a function of the standard deviation of the probability distribution function of the data stream.

[0329]

[0246] Aspect 15C - An apparatus described in any of aspects 11C to 14C, wherein the one or more processors are further configured to generate the data elements using a neural-based compression technique and quantize the data elements to create quantized data elements, and entropy coding the data elements using the code vector includes entropy encoding the quantized data elements using the code vector.

[0330]

[0247] Aspect 16C - The apparatus described in aspect 15C, wherein to generate data elements using neural-based compression techniques, the one or more processors are further configured to process images or video pictures using an image analysis neural network to generate the data elements.

[0331]

[0248] Aspect 17C - The apparatus of aspect 16C, further comprising a camera configured to capture an image or video picture.

[0332]

[0249] Aspect 18C - An apparatus described in any of aspects 11C to 14C, wherein entropy coding the data elements using the code vector includes entropy decoding the data elements encoded using the code vector to create quantized data elements, and the one or more processors are further configured to dequantize the quantized data elements to create reconstructed data elements.

[0333]

[0250] Example 19C - The apparatus described in example 18C, wherein the one or more processors are further configured to process the reconstructed data elements using an image synthesis neural network to reconstruct an image or video picture.

[0334]

[0251] Aspect 20C - The apparatus of aspect 19C, further comprising a display configured to display an image or video picture.

[0335]

[0252] It should be recognized that, depending on the example, certain acts or events of any of the techniques described herein may be performed in a different order, or may be added, combined, or omitted entirely (e.g., not all acts or events described may be required to practice the techniques). Moreover, in certain examples, acts or events may be performed in parallel rather than sequentially, for example, through multi-threaded processing, interrupt processing, or multiple processors.

[0336]

[0253] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. The computer-readable medium may include a computer-readable storage medium, which corresponds to a tangible medium, such as a data storage medium, or a communication medium, which includes any medium that facilitates transfer of a computer program from one place to another, for example, according to a communication protocol. As such, a computer-readable medium may generally correspond to (1) a tangible computer-readable storage medium that is non-transitory, or (2) a communication medium, such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0337]

[0254] By way of example and not limitation, such computer-readable storage media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included within the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but instead cover non-transitory tangible storage media. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks typically reproduce data magnetically, while discs reproduce data optically using a laser. Combinations of the above are also intended to be included within the scope of computer readable media.

[0338]

[0255] The instructions may be executed by one or more processors, such as one or more DSPs, general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuits. Thus, the terms "processor" and "processing circuitry" as used herein may refer to any of the above structures or any other structures suitable for implementing the techniques described herein. In addition, in some aspects, the functions described herein may be provided in dedicated hardware and / or software modules configured for encoding and decoding, or may be incorporated into a composite codec. Also, the techniques may be fully implemented in one or more circuits or logic elements.

[0339]

[0256] The techniques of the present disclosure may be implemented in a wide variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or sets of ICs (e.g., chipsets). Various components, modules, or units are described in the present disclosure to highlight functional aspects of devices configured to perform the disclosed techniques, but they do not necessarily require realization by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit or may be provided by a collection of interoperable hardware units, including one or more processors as described above, in conjunction with suitable software and / or firmware.

[0340]

[0257] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. 1. A method for encoding media data, the method comprising: generating data elements of the data stream using a neural-based compression technique comprising processing input images or video pictures with an image analysis neural network to create a data stream; determining probability distribution function parameters for the data elements, the probability distribution function parameters being based on a parameterization of a distribution of code vectors for the data stream optimized for quantization, the probability distribution function parameters being determined by processing the data elements using a hypervisor encoder neural network; quantizing the probability distribution function parameters; determining a code vector based on the quantized probability distribution function parameters; quantizing the data elements to create quantized data elements; entropy encoding the quantized data elements using the code vector; entropy coding the quantized probability distribution function parameters; A method comprising:

2. The probability distribution function parameter is u, the standard deviation of the probability distribution function is σ, and the minimum standard deviation is σ ｍｉｎ and the maximum standard deviation is σ ｍａｘ and the relationship between u and σ is σ=T ｕ→σ (u) = T λ→σ (T ｕ→λ (u)), where T λ→σ (λ)=exp(λ ln(σ ｍａｘ / σ ｍｉｎ ) + ln(σ ｍｉｎ 2. The method of claim 1, wherein the function T is defined according to an algorithm that measures coding redundancy or by solving an ordinary differential equation.

3. The method of claim 1 , wherein the probability distribution function parameter is a function of the standard deviation of the probability distribution function of the data stream.

4. The method of claim 1 , further comprising capturing the image or video picture using a camera.

5. A method for decoding media data, the method comprising: determining probability distribution function parameters for data elements of a data stream encoded by a neural-based media compression technique, the probability distribution function parameters being based on a parameterization of a distribution of code vectors for the data stream optimized for quantization, the data stream encoded by the neural-based media compression technique having been previously encoded by: i) processing input images or video pictures with an image analysis neural network to create the data stream; ii) processing the data stream with a hyperprior encoder neural network to determine probability distribution function parameters for the data stream; iii) quantizing the data stream; and iv) entropy coding the compressed data stream based on the probability distribution function parameters; determining a code vector based on the probability distribution function parameters; entropy decoding the data elements using the code vectors to create quantized data elements; dequantizing the quantized data elements to produce reconstructed data elements; processing the reconstructed data elements using an image synthesis neural network to reconstruct an image or video picture; A method comprising:

6. The method of claim 5 , further comprising displaying the image or the video picture.

7. 1. An apparatus configured to encode media data, the apparatus comprising: Memory and one or more processors in communication with the memory; wherein the one or more processors: generating data elements of the data stream using a neural-based compression technique comprising processing input images or video pictures with an image analysis neural network to create the data stream; determining probability distribution function parameters for the data elements, the probability distribution function parameters based on a parameterization of a distribution of code vectors for the data stream optimized for quantization, the probability distribution function parameters being determined by processing the data elements using a hypervisor encoder neural network; quantizing the probability distribution function parameters; determining a code vector based on the quantized probability distribution function parameters; quantizing the data elements to create quantized data elements; entropy encoding the quantized data elements using the code vector; an apparatus configured to entropy encode the quantized probability distribution.

8. The probability distribution function parameter is u, the standard deviation of the probability distribution function is σ, and the minimum standard deviation is σ ｍｉｎ and the maximum standard deviation is σ ｍａｘ and the relationship between u and σ is σ=T ｕ→σ (u) = T λ→σ (T ｕ→λ (u)), where T λ→σ (λ)=exp(λ ln(σ ｍａｘ / σ ｍｉｎ ) + ln(σ ｍｉｎ 8. The apparatus of claim 7, wherein the function T is defined according to an algorithm that measures coding redundancy or by solving an ordinary differential equation.

9. The apparatus of claim 7 , wherein the probability distribution function parameter is a function of a standard deviation of the probability distribution function of the data stream.

10. The device of claim 7 , further comprising a camera configured to capture the image or video picture.

11. An apparatus configured to decode media data, the apparatus comprising: Memory and one or more processors in communication with the memory; wherein the one or more processors: determining probability distribution function parameters for data elements of a data stream encoded by a neural-based media compression technique, the probability distribution function parameters based on a parameterization of a distribution of code vectors for the data stream optimized for quantization, the data stream encoded by the neural-based media compression technique having been previously encoded by: i) processing input images or video pictures with an image analysis neural network to create the data stream; ii) processing the data stream with a hyperprior encoder neural network to determine probability distribution function parameters for the data stream; iii) quantizing the data stream; and iv) entropy coding the compressed data stream based on the probability distribution function parameters. determining a code vector based on the probability distribution function parameters; entropy decoding the data elements using the code vectors to create quantized data elements; dequantizing the quantized data elements to produce reconstructed data elements; and processing the reconstructed data elements using an image synthesis neural network to reconstruct an image or video picture.

12. The device of claim 11 , further comprising a display configured to display the image or video picture.

Images