Reinforcement learning-based rate control for end-to-end neural network-based video compression

Abstract

An end-to-end neural network-based rate control method based on reinforcement learning implements an embodiment of a video codec. In one embodiment, the codec environment is based on an asymmetric gain variational autoencoder (AG-VAE) architecture. A reinforcement learning (RL) agent is implemented through a deep convolutional neural network. In one embodiment, the RL agent communicates the selection of a gain vector to the AG-VAE codec and receives reward data from the AG-VAE environment. Rate control is optimized over a period of frames, such as a group of pictures (GOP).

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Patent Application No. 63 / 409,271, filed September 23, 2022, which is incorporated herein by reference in its entirety.

[0002] FIELD OF THE INVENTION At least one of the present embodiments generally relates to a method or apparatus for compressing images and videos using neural network-based tools. [Background technology]

[0003] Compression of video content using novel Artificial Neural Network (ANN)-based tools is an area of ​​research being conducted by the Joint Video Exploration Team (JVET) between ISO / MPEG and ITU to replace some modules of the current standard H.266 / VVC, as well as, over the long term, replacing the entire structure with end-to-end autoencoder methods. In the context of end-to-end ANN-based video compression, encoding video content at the highest possible quality within bit-budget constraints, at the sequence or subsequence level, is one goal of such research. Summary of the Invention

[0004] At least one of the present embodiments generally relates to a method or apparatus in the context of image and video compression using novel artificial neural network (ANN)-based tools. In particular, one objective of the described embodiments is to encode video content at the highest quality possible within bit-budget constraints at a sequence or sub-sequence level in the context of end-to-end ANN-based video compression.

[0005] According to a first aspect, there is provided a method, comprising: encoding a portion of a video using a determined number of bits; and determining a number of bits to allocate for the portion of the encoded video based on a number of frames, wherein the determining comprises using a gain vector from a reinforcement learning agent that uses the determined latent ability from the encoding.

[0006] According to a second aspect, there is provided a method comprising the steps of: analyzing video data for a lambda value, determining an index of a vector having a corresponding lambda closest to a target lambda value, where lambda defines a rate-distortion operating point, interpolating values ​​of vectors between the determined index vector and vectors with consecutive indices using the target lambda value and the lambda values ​​of vectors between the determined index vector and vectors with consecutive indices, and decoding the video data using the interpolated vectors.

[0007] According to another aspect, an apparatus is provided, the apparatus comprising a processor, which may be configured to implement the general aspect by performing any of the described methods.

[0008] According to another general aspect of at least one embodiment, there is provided a device comprising an apparatus according to any of the decoding embodiments and at least one of: (i) an antenna configured to receive a signal including the video block; (ii) a band limiter configured to limit the received signal to a band of frequencies including the video block; or (iii) a display configured to display an output representing the video block.

[0009] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided that includes data content generated according to any of the described encoding embodiments or variations.

[0010] According to another general aspect of at least one embodiment, there is provided a signal including video data generated according to any of the described encoding embodiments or variations.

[0011] According to another general aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any of the described encoding embodiments or variations.

[0012] According to another general aspect of at least one embodiment, there is provided a computer program product including instructions that, when executed by a computer, cause the computer to perform any of the described decoding embodiments or variations.

[0013] These and other aspects, features, and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings.

[0014] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided that includes data content that includes instructions for performing either an encoding method or a decoding method. [Brief explanation of the drawings]

[0015] [Figure 1] A random access structure with a GOP of 8 frames is shown. [Figure 2] A reinforcement learning framework is presented. [Figure 3] We present a basic autoencoder for image compression. [Figure 4] The basic AG-VAE architecture is shown. [Figure 5] We present a basic end-to-end NN-based video compression framework. [Figure 6]1 shows a gain video compression autoencoder. [Figure 7] We present the proposed RL-based rate-distortion optimization scheme. [Figure 8] 1 illustrates a system with an alternative video compression model. [Figure 9] We present AG-VAE in addition to a hyper-prior based autoencoder architecture. [Figure 10] 1 illustrates one embodiment of a method for encoding video using the described embodiments. [Figure 11] 1 illustrates one embodiment of a method for decoding video using the described embodiments. [Figure 12] 1 illustrates one embodiment of an apparatus for encoding or decoding using the described embodiments. [Figure 13] Indicates a standard general-purpose video compression method. [Figure 14] Describes a standard general-purpose video decompression method. [Figure 15] 1 illustrates a processor-based system for encoding / decoding under the general aspects described. DETAILED DESCRIPTION OF THE INVENTION

[0016] In traditional video compression, a certain temporal structure allows the encoder to select reference frames from previously decoded pictures to optimize the encoding of each frame, but also to manage the bit budget per frame. Certain key frames are used as references to predict other frames. In that case, it is important to ensure that they are coded with the correct quality.

[0017] A typical structure used in the broadcast ecosystem is termed the random access structure, which consists of periodic Groups of Pictures (GOPs) consisting of a repeating minimal temporal time-frame structure.

[0018] Figure 1 shows such a structure for an eight-frame GOP. The first frame is an intraframe, or I-frame, meaning it is independent of other frames being decoded. It can then be used as a random access point from which the decoder can begin decoding the sequence. In broadcast, they are typically separated by one second of video, allowing a TV viewer to switch channels, begin decoding the selected new channel, and not wait too long for the video to begin displaying. However, because these frames are not predicted using previously decoded content, they typically require more bits to transmit. Between I-frames, other frames are predicted using previously decoded frames. Notice that in the structure of Figure 2, the coding order differs from the display order. This allows the encoder to predict frames using previously reconstructed pictures from the past and future. These frames are therefore called B-frames, after bidirectional prediction. B0 of each GOP is the first frame coded and is predicted using the last key frame (I or B0) from the previous GOP; for example, frame 8 in display order is predicted from frame 0. The next frame in coding order can be predicted using past and future frames, as depicted by the arrows: Frame B1 can use frame B0 of type I, frame B2 can be predicted from frames I, B0, and B1, etc.

[0019] To manage the bit budget between different types of frames, a typical Quantization Parameter (QP) offset is shown in Figure 1, which corresponds to the default QP structure used in the HEVC / H.265 reference software. In other words, if the nominal QP selected for a sequence is 30, I-frames are coded at QP-27, B0-frames are coded at 31, etc.

[0020] As can be seen, the QP assigned per frame is independent of the video content in the input signal, i.e., texture, motion, etc., and depends only on the pre-fixed temporal structure. However, this is suboptimal, since a better trade-off can be found when accounting for how efficient the compression is for sub-portions of sequences and images. For example, when a still sequence is coded, it is preferable to allocate a larger bit budget to the first I-frame, since subsequent frames are expected to require very few bits after being temporally predicted.

[0021] Traditional video compression standards predict to reduce redundancy and decorrelate signals using transforms and entropy coding, but can reach low bit rates by degrading the video based on signal fidelity or visual quality. Compression performance is then evaluated by looking at the bitstream size, i.e., the required number of bits needed to store or transmit the video at a given reconstructed quality after decoding. Codecs are then characterized by the quality of the decoded content as a function of bit rate.

[0022] To adapt to the so-called rate-distortion trade-off, conventional encoders can adapt a quantization parameter, QP, which drives the quantization process of the transmitted data and, in turn, the distortion introduced into the reconstructed content.

[0023] Rate-Distortion Optimization (RDO) refers to an algorithm used by an encoder to minimize transmitted bits for a given target reconstruction quality. Traditional encoders divide an image into non-overlapping blocks of different shapes and sizes. Then, intra- or inter-prediction and transform are applied to reduce redundancy with previously coded content. Finally, the transformed prediction residual is quantized and entropy coded. In this process, only the quantization part is lossy. However, optimization for bitrate reduction can be performed by the encoder at all levels, including block size selection, prediction, transform, and quantization. To this end, the optimization process aims to minimize the Lagrangian criterion. J=R+λD, where R represents the rate, i.e., the number of bits required to represent a picture or group of pictures, and D represents the distortion or quality of the picture reconstructed at the decoder. Lambda is a parameter that defines at which rate-distortion tradeoff or operating point the codec is used.

[0024] It should be noted that D can be measured using different quality metrics, such as MSE (Mean Squared Errors), SSIM (Structural Similarity), etc. However, in conventional video codecs, D is restricted to be tallyable across blocks. In fact, when an encoder decides whether to directly encode a block or to divide it into smaller blocks, the encoder needs to calculate and compare the rate-distortion tradeoff in both cases, which requires summing up the costs of the smaller blocks. This restriction often forces conventional encoders to use MSE as the basic criterion.

[0025] Rate control consists of maximizing the quality of reconstructed video under bitrate constraints. Because frames are coded sequentially, the encoder needs to plan the expected bitrate for the next frame, since quantization parameters can be selected at the frame or block level. In most applications, the encoder does not have the time to perform multiple passes over a sequence or subsequence to make optimal decisions based on content. Therefore, rate control algorithms attempt to estimate the cost of transmitting motion and residual information for future frames when adapting the quantization of the current frame based on already reconstructed frames in the past and present. However, most existing deployed rate control methods rely heavily on empirically developed methods for adaptation. More recent machine learning-oriented methods, such as those in [1], use machine learning mechanisms to address QP variations, but this approach remains limited in its action on QP and is not easily adaptable depending on the video compression use case.

[0026] Reinforcement learning for rate-distortion optimization in the context of traditional video compression Google's DeepMind published a recent study [2] in which they propose using an adaptation of their reinforcement learning algorithm, MuZero, to optimize the bit budget over a group of frames by adjusting the quantization parameters of the VP9 standard [3].

[0027] The VP9 reference software includes a two-pass encoding strategy. The first pass extracts relevant information, known as first-pass statistics, about the texture to be encoded. These statistics help the second and final passes make informed encoding decisions that consider the entire frame, not just the current block to be encoded. In contrast to the final pass, where blocks can have variable sizes from 4x4 to 64x64, the image is divided into 16x16 non-overlapping blocks to extract this information more quickly.

[0028] The first pass statistics of the frame at the current time step, along with additional information such as compression statistics of frames processed so far, the percentage of the bit budget already used so far, etc., are used by the MuZero reinforcement learning agent to determine its action, i.e., determine the QP offset for the current frame to achieve the optimal rate-distortion tradeoff.

[0029] The algorithms these reinforcement learning agents use to learn to take optimal actions, such as MuZero or PPO [4], work based on a feedback loop. The agent determines an action based on the state of the environment. The selected action is applied to the environment, which transitions the environment to a new state. The environment then returns the new state and a reward metric to the agent. Thus, during training, the agent receives a reward for each action it takes, indicating how good or bad that action was. The agent then adjusts its internal parameters to maximize the cumulative reward it receives.

[0030] End-to-end deep learning-oriented video compression In recent years, novel image and video compression methods based on artificial neural networks have been developed. In contrast to traditional methods that apply predefined prediction modes and transformations, ANN-based methods rely on parameters that are learned on large datasets during training by iteratively minimizing a loss function. In the case of compression, the loss function describes both an estimate of the bit rate of the encoded bitstream and the performance of the decoded content, such as the aforementioned Lagrangian criterion. J=R+λD

[0031] Figure 3 shows a basic example autoencoder pipeline.

[0032] The inputs X to the encoder part of the network can consist of: 1. An image or frame from a video; 2. Part of the image, 3. A tensor representing a group of images, or 4. A tensor representing a portion (crop) of a group of images.

[0033] In either case, the input may have one or more components, for example monochrome, RGB, or YCbCr components. 1. An input tensor X is fed into an encoder network, which is typically a sequence of convolutional layers with activation functions: convolutional or spatial-depth 1 A large stride in the operations can be used to reduce the spatial resolution while increasing the number of channels. The encoder network can be viewed as a learned nonlinear transform. 2. The output of the encoder network is a "feature map" or "latent" Y, which is quantized

[0034]

number

[0035]

number

[0036]

number

[0037]

number

[0038] Note that more sophisticated architectures exist, for example adding a "hyper-autoencoder" (hyper-prior) to the network to jointly learn the latent distribution properties of the encoder outputs. The embodiments proposed here are not limited to the use of autoencoders. Any end-to-end differentiable codec can be considered.

[0039] As above, in the following

[0040]

number

[0041] For a long time, state-of-the-art models were trained for each Lagrangian parameter λ, making it necessary to have several pre-trained models to evaluate performance over a range of bitrates. Each trained set corresponds to millions of parameters, which makes the use of such codecs impossible in real-world applications. For example, H.265 / HEVC has a granularity of 51 QP. Adapting to a specific bitrate with such granularity results in a decoder with 51 pre-trained models in memory, which can be switched on-the-fly for rate control.

[0042] To address the memory requirement and avoid the need to switch models, the authors of [5] proposed the Asymmetric Gained Variational Auto-Encoder (AG-VAE). In what follows, we consider a potential tensor Y with dimensions C×H×W, where H and W denote the height and width of the tensor, often fractions of the resolution of the input X, and C denotes the number of channels. Before quantization, this potential tensor is multiplied element-wise by a gain vector of shape C×1×1, i.e., the element of the ith channel in Y is multiplied by G. e At the decoder, the entropy decoded (ED) tensor is also multiplied by the composition function g s before being fed to the inverse gain vector G d Multiplied by

[0043]

number

[0044] Instead of needing a complete set of parameters trained for each operating point, i.e., each lambda, this model is e and its corresponding G d At training time, these vectors are stored in the a and g s can be chosen randomly so that is trained for all selected lambdas, but each vector is optimized for only one lambda. At inference time, i.e., when the codec is actually in use, an appropriate gain vector is selected from a list, or can even be interpolated to refine the target bitrate.

[0045] In the above description, for simplicity, all transmitted information was factored as tensors, as if the input always consisted only of image X. However, for video compression, different tensors may be calculated and transmitted depending on the type of frame considered. In the following, an exemplary video model is described that specifies different models for intra (I) and predicted (P) pictures.

[0046] Figure 5 shows a video compression framework based on a state-of-the-art published model. It processes I-frames and P-frames with two separate architectures. I-frames are processed similarly to image compression, i.e., with the same process as in the model described above. P-frames can be further compressed using already reconstructed information, i.e., previously decoded frames. A warper is used to map a reference picture onto the current picture to be coded to form a predictor. This warper is based on a motion model

[0047]

number

[0048] Then, as in conventional compression, a residual is formed as the difference between the predictor and the source signal. This residual is compressed and transmitted in the same way as an I-frame, i.e., using a dedicated autoencoder as shown in Figure 5.

[0049] An exemplary gain version of the above video compression pipeline is shown in Figure 6, where each autoencoder includes a gain unit on the encoder side and on the decoder side. For a particular lambda for an I-frame, the model considers a pair (G e,I , G d,I), where I marks the vector corresponding to the I frame. However, P frames are now represented by the quadruplet (G e,P,F , G d,P,F , G e,P,R , G d,P,R ), where F and R represent the motion flow and the residual, respectively. The method uses the associated gain (G e,B,F , G d,B,F , G e,B,R , G d,B,R Note that this can be trivially extended to bi-prediction by having

[0050] The embodiments described herein aim to optimize the compression of the bitstream by efficiently spending the bit budget on groups of pictures, or sub-sequences of the input video.

[0051] Reference encoders typically organize the processing of a video sequence in groups of pictures (GOPs), where pictures can be coded depending on previously reconstructed ones according to a predefined temporal structure, which is typically used to empirically assign a quantization parameter (QP) to each frame depending on its position in the GOP and the interdependencies between frames.

[0052] In addition to this, expert encoders develop rate control methods that extract features of the video content to devise QP strategies that meet bit-budget targets. They typically use engineering methods that can remain suboptimal. The work by DeepMind described above addresses the problem by using reinforcement learning (RL)-based mechanisms in the context of traditional video coding, relying on information computed from fast past-encoder estimation.

[0053] The method proposed here also leverages the strengths of RL algorithms when optimizing compression under bit-budget constraints. However, it can run on a novel end-to-end NN-based video compression system. It provides a means to adapt the bitrate from the encoder and a syntax and mechanism for conveying information to the decoder.

[0054] In the proposed embodiment, we propose to use reinforcement learning to perform compression optimization over a sequence of frames under a bit budget constraint. We propose to utilize some compressive autoencoder structure to make the RL algorithm dependent on the relevant data.

[0055] In particular, some variants of the proposed solution take advantage of the AG-VAE architecture described in Section 1, where the analytical transformation g in the encoder a The gain is applied after the potential tensor Y=g a This means that (X) is always the same for any lambda the encoder is operating on. This is because the rate control mechanism involves multiplication by a lambda-dependent gain vector and quantization, which occurs after Y has already been computed. The potential tensor Y therefore provides meaningful information to the RL agent about the compressibility of the input. In contrast, DeepMind's work relies on a first encoding pass that considers limited encoding choices to estimate the bit cost of a frame for a given encoder setting.

[0056] Major general-purpose methods The main concept of the described embodiments is to optimize the rate-distortion tradeoff for a GoP (a subsection of a video) by selecting the amount of bit allocation for each frame. This can be done under the constraint of a target bitrate or a limit bitrate for that GoP. For each frame, an RL agent decides how much bit budget to allocate. This corresponds to selecting a tradeoff point on the rate-distortion curve. Among other possibilities, one such strategy for varying the rate-distortion tradeoff for a neural network-based compression model is to use the AG-VAE architecture described in Section 1.4. Below is a proposed method that uses AG-VAE as a means to optimize the rate-distortion tradeoff. Note that this system can work with any other rate control method.

[0057] Frames from the GoP are fed into the system one at a time. For any given frame, the RL agent receives as input the coded potential of the current frame, among other information such as the bit budget used so far. The agent then outputs a decision on how many bits to allocate to the current frame. Using this decision, a rate control mechanism encodes the frame at a selected bit rate. In this case, the rate control mechanism selects gain and inverse gain vectors for the frame's target bit rate.

[0058] Internally, the agent uses a policy network (in this case, a deep convolutional neural network) as shown in the bottom of Figure 7 to determine frame bit allocation. The agent receives a reward based on how well it optimizes the rate-distortion tradeoff for a GoP. For example, this reward may be calculated based on the total distortion over a sequence of frames with a penalty for exceeding the bit budget. In the example of Figure 7, distortion is measured using PSNR, and bitrate is expressed as bpp (bits per pixel). This reward is used to calculate the gradient of the policy network needed to improve the agent's decision-making. The RL training algorithm iteratively modifies the parameters of the policy network to maximize the reward received for each GoP.

[0059] Different state and action spaces The input to an RL agent is the state of the environment. The set of all possible states the environment can be in is called the state space. A well-designed state space for an RL system should consist of all the information the agent needs to determine its actions.

[0060] Similar to the state space, the set of all possible actions that an RL agent can take in an environment is called the action space.

[0061] Our proposed method, which uses RL for rate-distortion optimization at the GoP level, can have various formulations of the state space and action space based on the specifications of the video compression model. For example, the system shown in Figure 7 has as its state space the coded potential of the frame plus the bit budget used so far in the GoP. Some systems use the frame directly instead of the coded potential, as in Figure 8, which also has an indication of what type of frame is being processed as another additional state-space feature. Another possible state-space representation is to have a concatenated tensor of all possible gain vectors, coded potential multiplications.

[0062] Similarly, the action space can be formulated in several different ways. One possibility is to have a discrete action space where the agent simply chooses between predetermined bitrate points. Another possibility is to have a continuous action space where the agent chooses from a range of bitrate points. This action space is applicable to systems with rate control mechanisms that support continuous rate control, such as AG-VAE.

[0063] The system variations described in the following sections use the particular state and action spaces mentioned above for purposes of illustration, however the system is not limited to using that particular example and is compatible with a variety of other state and action spaces.

[0064] A simple variant when the video is processed in still image mode In one embodiment, we consider the case where the video is processed in all intra mode, i.e., where each picture is coded independently. The above discussion is directly applicable.

[0065] Adaptation of the proposed method in addition to video architectures As mentioned in the previous section, most video compression models treat different types of frames (e.g., I-frames and P-frames) differently. Our proposed method is also compatible with such video compression models, as shown in Figure 8.

[0066] Here, the RL agent can take as an additional input the type of frame it is processing, and its action space is discrete, i.e., if it is an I-frame, it has a payoff vector (G e,I , G d,I ) and if it is a P frame, the gain vector (G e,P,F , G d,P,F , G e,P,R , G d,P,R ) where F represents the motion flow and R represents the residual. Another possibility is to have a continuous action space.

[0067] Adaptation of the proposed method in addition to the hyper-ahead architecture The above description of AG-VAE uses the most basic autoencoder chain and did not detail the entropy bottleneck and the entropy coder and decoder (EC / ED in the diagram above). More advanced methods use the latent tensor

[0068]

number

[0069]

number

[0070]

number

[0071]

number

[0072] where each operating point lambda is a quadruple {G e , G e h , G d , G d h}. It therefore does not change the indexing and information to be sent to the decoder, which contains a list of gains of the same size as the encoder and can interpolate the values ​​of the two vectors in the same way.

[0073] Proposed normative syntax In traditional video compression, the quantization parameter (QP) is generally transmitted per frame or per slice of a frame. Note that it is possible to transmit a QP offset at the block level to spatially adapt the target quality. One symbol coding for a value in the range 0 to 51 is transmitted per frame, and the QP offset mechanism can even reduce the cost of its components.

[0074] No interpolation between gain vectors For the AG-VAE architecture without gain interpolation, the vector G e and G dThe pair of n vectors can be selected from a list of n vectors on which the model was trained. This requires that the bitstream contains the index of the selected pair from the n vectors, i.e., a relatively small positive integer, e.g., less than 256, and thus coded in 8 bits. This syntax element can be coded per frame or per tile of a frame if the frame is divided and processed tile-by-tile with the corresponding autoencoder.

[0075] Interpolation between two pre-trained vectors If the number of predefined gain vectors corresponding to lambda points is small, it is possible to interpolate between the vectors using the target lambda value. To interpolate between two of the existing pre-trained vectors available at both the encoder and decoder, the decoder needs the information necessary to reconstruct the appropriate gain vector.

[0076] In a first embodiment, we assume that the decoder has a mechanism for analyzing the lambda transmitted with the bitstream. Each predefined vector is associated with a lambda value for which it was trained. The decoder then determines the index of the vector whose corresponding lambda is lower than and closest to the target lambda. The target lambda and the lambda values ​​of both surrounding vectors can then be used to interpolate the values ​​of vectors between the latter vector and vectors corresponding to successive index numbers.

[0077] The lambda value can then be transmitted directly in the bitstream. Since lambda is generally a real number, it needs to be truncated to the nearest value with a pre-fixed precision. Since codecs usually avoid sharing floating-point numbers between the encoder and decoder, the transmitted lambda should be represented as a fixed-point value.

[0078] In a second variant, it is proposed to transmit two variables to specify the target vector. - the index of the pre-trained vector whose lambda is closest to and less than the target lambda - the interpolation ratio between two consecutive pre-trained vectors considered for the calculation. This interpolation ratio has a value between 0 and 1 and can be easily expressed as an unsigned integer depending on the selected interpolation precision.

[0079] The syntax element duo may be more efficient in terms of the number of bits required for transmission as well as slightly less complex interpolation operations at the decoder, since the interpolation ratios are transmitted directly and do not need to be calculated.

[0080] 10 illustrates one embodiment of a method 1000 for encoding video data. The method begins at start block 1001 and proceeds to block 1010 for encoding a portion of the video using a determined number of bits. Control proceeds from block 1010 to block 1020, which determines a number of bits to allocate for the portion of the encoded video based on a number of frames, where determining includes using a gain vector from a reinforcement learning agent that uses the determined latent ability from encoding.

[0081] 11 illustrates one embodiment of a method 1100 for decoding video data. The method begins at start block 1101 and proceeds to block 1110, which analyzes the video data for lambda values. Control proceeds from block 1110 to block 1120, which determines an index of a vector having a corresponding lambda closest to a target lambda value, where lambda defines a rate-distortion operating point. Control proceeds from block 1120 to block 1130, which interpolates values ​​of vectors between the determined index vector and vectors with consecutive indices using the target lambda value and the lambda values ​​of vectors between the determined index vector and vectors with consecutive indices. Control proceeds from block 1130 to block 1140, which decodes the video data using the interpolated vectors.

[0082] 12 shows one embodiment of an apparatus 1200 for compressing, encoding, or decoding video using the methods described above. The apparatus includes a processor 1210, which may be interconnected through at least one port to a memory 1220. Both the processor 1210 and the memory 1220 may also have one or more additional interconnections to external connections.

[0083] Additionally, processor 1210 is configured to insert or receive information in a bitstream and compress, encode, or decode using the methods described above.

[0084] The embodiments described herein include various aspects, including tools, features, embodiments, models, techniques, and the like. Many of these aspects are described with specificity and, often in a manner that may sound limiting, at least to indicate their individual characteristics. However, this is for purposes of clarity of description and does not limit the applicability or scope of the aspects. In fact, all of the different aspects can be combined and substituted to provide further aspects. Furthermore, these aspects can also be similarly combined and substituted with aspects described in previous applications.

[0085] Aspects described and contemplated in this application can be implemented in many different forms. While Figures 13, 14, and 15 provide some embodiments, other embodiments are contemplated, and the discussion of Figures 13, 14, and 15 does not limit the scope of implementation aspects. At least one of the above aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as a method, an apparatus, a computer-readable storage medium having stored therein instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored therein a bitstream generated according to any of the described methods.

[0086] In this application, the terms "reconstructed" and "decoded" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image," "picture," and "frame" may be used interchangeably. Typically, although not necessarily, the term "reconstructed" is used on the encoder side, and the terms "decoded" or "reconstructed" are used on the decoder side.

[0087] Various methods are described herein, each of which includes one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for the proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined. It should be noted that terms such as "first," "second," and the like may be used in various embodiments to modify elements, components, steps, operations, etc., e.g., "first decode" and "second decode." The use of such terms does not imply any ordering of the modified operations unless specifically required. Thus, in this example, the first decode need not be performed before the second decode, but may occur, for example, before, during, or during an overlapping period with the second decode.

[0088] Various methods and other aspects described herein can be used to modify modules, such as intra-prediction modules, entropy coding modules, and / or decoding modules (160, 360, 145, 330), of video encoder 100 and video decoder 200 as shown in Figures 13 and 14. Furthermore, aspects of the present disclosure are not limited to VVC or HEVC, but can also be applied to, for example, other standards and recommendations, whether existing or developed in the future, and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise indicated or technically excluded, aspects described herein can be used alone or in combination.

[0089] Various numerical values ​​are used in this application. The specific values ​​are for illustrative purposes and the described embodiments are not limited to these specific values.

[0090] 13 shows an encoder 100. Variations of this encoder 100 are contemplated, but for clarity, the following describes the encoder 100 without describing all possible variations.

[0091] Before encoding, the video sequence may undergo encoding pre-processing (101), such as applying a color transformation to the input color picture (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0) or performing a remapping of the input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata can be associated with the pre-processing and added to the bitstream.

[0092] In encoder 100, a picture is encoded by encoder elements as described below. The picture to be encoded is divided (102) into units, e.g., CUs, and processed. Each unit is coded, e.g., using either intra mode or inter mode. When a unit is coded in intra mode, intra prediction (160) is performed. In inter mode, motion estimation (175) and motion compensation (170) are performed. The encoder decides (105) whether to use intra mode or inter mode to code the unit, and indicates the intra or inter decision, e.g., by a prediction mode flag. A prediction residual is calculated (110), e.g., by subtracting the predicted block from the original image block.

[0093] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, as well as motion vectors and other syntax elements, are entropy coded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., the residual is coded directly without applying a transform or quantization process.

[0094] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150), and the prediction residual is decoded. The decoded prediction residual is combined (155) with the predicted block to reconstruct an image block. An in-loop filter (165) is applied to the reconstructed picture, for example, to perform deblocking / SAO (Sample Adaptive Offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0095] Figure 14 shows a block diagram of video decoder 200. In decoder 200, decoder elements decode the bitstream as described below. Video decoder 200 generally performs a decoding pass that is the reverse of the encoding pass described in Figure 13. Encoder 100 also generally performs video decoding as part of encoding the video data.

[0096] In particular, the decoder's input includes a video bitstream, which may be generated by video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other coded information. Picture partition information indicates how the picture is divided. Thus, the decoder may partition the picture according to the decoded picture partition information (235). The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual is combined with a predicted block (255) to reconstruct an image block. The predicted block may be obtained from intra prediction (260) or motion-compensated prediction (i.e., inter prediction) (275) (270). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0097] The decoded picture may further undergo post-decoding processing (285), such as an inverse color transform (e.g., YcbCr 4:2:0 to RGB 4:4:4) or an inverse remapping that performs the inverse of the remapping process performed in the pre-encoding process (101). The post-decoding process may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0098] FIG. 15 shows a block diagram of an example system in which various aspects and embodiments can be implemented. System 1000 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, Internet appliances, and servers. Elements of system 1000, singly or in combination, may be embodied in a single integrated circuit (IC), multiple ICs, and / or separate components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or separate components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more of the aspects described herein.

[0099] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein, for example, to implement various aspects described herein. The processor 1010 may include embedded memory, input / output interfaces, and various other circuitry known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). System 1000 includes storage device(s) 1040, which may include non-volatile and / or volatile memory, including, but not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), flash, magnetic disk drives, and / or optical disk drives. Storage device(s) 1040 may include, by way of non-limiting example, internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0100] System 1000 includes an encoder / decoder module 1030 configured to process data to provide, for example, encoded or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a module that may be included in a device for performing encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Additionally, the encoder / decoder module 1030 may be implemented as a separate element of system 1000 or may be incorporated within the processor 1010 as a combination of hardware and software, as is known to those skilled in the art.

[0101] Program code to be loaded into the processor 1010 or the encoder / decoder 1030 to perform various aspects described herein may be stored in the storage device 1040 and subsequently read into the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items during the execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, expressions, operations, and computational logic.

[0102] In some embodiments, memory internal to the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be either the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be the memory 1020 and / or the storage device 1040, e.g., dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used to store, for example, the television's operating system. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video coding and decoding operations such as MPEG-2 (MPEG stands for Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard being developed by JVET).

[0103] Inputs to the elements of system 1000 can be provided through various input devices as indicated in block 1130. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives RF signals transmitted throughout a broadcast, for example, by a broadcaster; (ii) a component (COMP) input terminal (or set of COMP input terminals); (iii) a Universal Serial Bus (USB) input terminal; and / or (iv) a High Definition Multimedia Interface (HDMI) input terminal. Although not shown in FIG. 15, other embodiments include composite video.

[0104] In various embodiments, the input devices of block 1130 have associated respective input processing elements known in the art. For example, the RF section may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) downconverting the selected signal, (iii) band-limiting again to a narrower frequency band to select a signal frequency band, which in certain embodiments may be referred to as a channel (for example), (iv) demodulating the downconverted, band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired stream of data packets. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, downconverting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or to baseband. In one embodiment of a set-top box, the RF section and its associated input processing elements receive RF signals transmitted over a wired (e.g., cable) medium and perform frequency selection by filtering, downconverting, and re-filtering to a desired frequency band. Various embodiments rearrange the order of the above (and other) elements, omit some of these elements, and / or add other elements that perform similar or different functions. Adding elements can include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0105] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, e.g., Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or within processor 1010, as desired. Similarly, aspects of USB or HDMI interface processing may be implemented, as desired, in a separate interface IC or within processor 1010. The demodulated, error corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 1010 and an encoder / decoder 1030 operating in combination with memory and storage elements, to process the data streams required for presentation to an output device.

[0106] The various elements of system 1000 may be provided within an integrated housing in which the various elements may be interconnected and transmit data between them using suitable connection arrangements, e.g., internal buses known in the art, including the Inter-IC (I2C) bus, wiring, and printed circuit boards.

[0107] The system 1000 includes a communication interface 1050 that enables communication with other devices over a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data over the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented in a wired medium and / or a wireless medium, for example.

[0108] In various embodiments, data is streamed or otherwise provided to system 1000 using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal in such embodiments is received via communication channel 1060 and communication interface 1050 adapted for Wi-Fi communication. Typically, communication channel 1060 in such embodiments is connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. In other embodiments, streamed data is provided to system 1000 using a set-top box that delivers data via an HDMI connection in input block 1130. In yet other embodiments, streamed data is provided to system 1000 using an RF connection in input block 1130. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0109] System 1000 can provide output signals to various output devices, including a display 1100, speakers 1110, and other peripheral devices 1120. Display 1100 in various embodiments includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 1100 can be for a television, a tablet, a laptop, a mobile phone, or another device. Display 1100 can also be integrated with other components (e.g., as in a smartphone) or can be separate (e.g., an external monitor for a laptop). In various example embodiments, other peripheral devices 1120 include one or more of a standalone digital video disc (or digital versatile disc) (both terms referred to as a DVR), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 to provide functionality based on the output of system 1000. For example, a disc player performs the function of playing the output of the system 1000 .

[0110] In various embodiments, control signals are communicated between system 1000 and display 1100, speakers 1110, or other peripheral devices 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that allow control between devices with or without user intervention. Output devices can be communicatively coupled to system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, output devices can be connected to system 1000 via communication interface 1050 using communication channel 1060. Display 1100 and speakers 1110 may be integrated into a single unit with other components of system 1000 in an electronic device such as a television. In various embodiments, display interface 1070 includes a display driver, such as, for example, a timing controller (TCon) chip.

[0111] Alternatively, the display 1100 and speakers 1110 may be separate from one or more of the other components, for example, if the RF portion of the input 1130 is part of a separate set-top box. In various embodiments in which the display 1100 and speakers 1110 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0112] The embodiments may be implemented by computer software implemented by the processor 1010, by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type appropriate to the technology environment and may be implemented using any suitable data storage technology, including, by way of non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 1010 may be of any type appropriate to the technology environment and may include, by way of non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a multi-core architecture-based processor.

[0113] Various implementations involve decoding. As used herein, "decoding" can encompass all or part of the processing performed on a received encoded sequence to generate a final output suitable for display, for example. In various embodiments, such processing includes one or more of the processing typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or alternatively include processes performed by decoders in various implementations described herein.

[0114] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or to the broader decoding process as a whole will be clear based on the context of the specific description and will be well understood by one of ordinary skill in the art.

[0115] Various implementations involve encoding. As can be seen from the above discussion of "decoding," "encoding," as used herein, can encompass all or part of the processing performed on an input video sequence to, for example, generate an encoded bitstream. In various embodiments, such processing includes one or more of the processing typically performed by an encoder, such as partitioning, differential encoding, transforming, quantizing, and entropy coding. In various embodiments, such processes also or alternatively include processes performed by the encoders of the various implementations described herein.

[0116] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or to the broader encoding process as a whole will be clear based on the context of the specific description and will be well understood by one of ordinary skill in the art.

[0117] It should be noted that the syntax elements used herein are descriptive terms and therefore do not exclude the use of other syntax element names.

[0118] Where a figure is presented as a flow diagram, it should be understood that the figure also provides a block diagram of the corresponding apparatus. Similarly, where a figure is presented as a block diagram, it should be understood that the figure also provides a flow diagram of the corresponding method / process.

[0119] Various embodiments may relate to parametric models or rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is usually considered, often due to computational complexity constraints. This can be measured through a rate-distortion optimization (RDO) metric, or through Least Mean Square (LMS), Mean of Absolute Error (MAE), or other such measures. Rate-distortion optimization is usually formulated to minimize a rate-distortion function, which is a weighted sum of rate and distortion. There are various approaches to solving the rate-distortion optimization problem. For example, these techniques may be based on extensive testing of all encoding options, including all considered modes or coding parameter values, but with a thorough evaluation of their coding costs and the associated distortion of the reconstructed signal after coding and decoding. To reduce encoding complexity, more rapid techniques may also be used, especially with calculation of approximate distortion based on a prediction or prediction residual signal rather than the reconstructed signal. A mixture of these two approaches may also be used, such as by using approximate distortion for only some of the considered encoding options and full distortion for others. Other approaches evaluate only a subset of the possible encoding options. More generally, many approaches employ any of a variety of techniques to perform optimization, but the optimization is not necessarily a complete evaluation of both the coding cost and the associated distortion.

[0120] Implementations and aspects described herein may be implemented as, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed only in the context of a single implementation form (e.g., discussed only as a method), the implementation of the discussed feature may also be implemented in other forms (e.g., an apparatus or a program). For example, an apparatus may be implemented in appropriate hardware, software, and firmware. A method may be implemented, for example, in a processor, which refers to a general processing device including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Furthermore, a processor may include, for example, a communication device such as a computer, a mobile phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate communication of information between end users.

[0121] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation," as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with that embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment," or "in one implementation" or "in an implementation" in various places throughout this application, as well as other variations, are not necessarily all referring to the same embodiment.

[0122] Additionally, the application may refer to "determining" various pieces of information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0123] Additionally, the application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0124] Additionally, the application may refer to "receiving" various information. Receiving, like "accessing," is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" typically involves in some manner, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0125] For example, in the case of "A / B," "A and / or B," and "at least one of A and B," it should be understood that the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), or the selection of only the first and third listed alternatives (A and C), or the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A, B, and C). This can be expanded to include as many items as are listed, as would be apparent to one skilled in this and related arts.

[0126] Also, as used herein, the term "signaling" specifically refers to indicating something to a corresponding decoder. For example, in a particular embodiment, an encoder signals a particular one of multiple transforms, coding modes, or flags. Thus, in some embodiments, the same transform, parameter, or mode is used at both the encoder and decoder sides. Thus, for example, an encoder may transmit a particular parameter to a decoder (explicit signaling) so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter as well as other parameters, signaling without transmission (implicit signaling) may be used to simply allow the decoder to know and select the particular parameter. By avoiding transmitting any actual functionality, bit savings are realized in various embodiments. It will be understood that signaling can be achieved in various ways. For example, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder in various embodiments. While the above relates to the verb form of the word "signal," the word "signal" may also be used as a noun herein.

[0127] As will be apparent to one skilled in the art, implementations can generate a variety of signals formatted to carry information that can be stored or transmitted, for example. Information can include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal can be formatted to carry a bitstream of the described embodiments. For example, such a signal can be formatted as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. As is known, the signal can be transmitted over a variety of different wired or wireless links. The signal can be stored on a processor-readable medium.

[0128] The preceding sections describe several embodiments across various claim categories and types. Features of these embodiments may be provided alone or in any combination. Furthermore, embodiments may include one or more of the following features, devices, or aspects across various claim categories and types, alone or in combination. At least one embodiment includes the following features:

[0129] Encoding and decoding video information using neural networks.

[0130] Using a gain vector from a reinforcement learning agent that uses the determined latent ability from the encoding, determine the number of bits to allocate for a portion of the encoded video based on the number of frames.

[0131] Encoding and decoding using asymmetric gain variational autoencoders.

[0132] A reinforcement agent implemented using deep neural networks.

[0133] The encoding and decoding described above further includes data indicative of the potential included in the coded bitstream.

[0134] A bitstream or signal containing one or more of the described syntax elements or variations thereof.

[0135] A bitstream or signal including syntax carrying information produced by any of the described embodiments.

[0136] Creating and / or transmitting and / or receiving and / or decoding according to any of the described embodiments.

[0137] Analyzing video data or a bitstream to determine the operating point of a codec.

[0138] A method, process, apparatus, instruction storage medium, data storage medium, or signal according to any of the described embodiments.

[0139] Inserting syntax elements into the signaling that allow the decoder to decide to decode the information in a manner that corresponds to the manner used by the encoder.

[0140] Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal that includes one or more of the described syntax elements or variations thereof.

[0141] A TV, set-top box, mobile phone, tablet, or other electronic device that performs the conversion method according to any of the described embodiments.

[0142] A TV, set-top box, mobile phone, tablet, or other electronic device that makes the conversion method decision and displays the resulting image (e.g., using a monitor, screen, or other type of display) according to any of the described embodiments.

[0143] A TV, set-top box, mobile phone, tablet, or other electronic device that selects, band-limits, or tunes (e.g., using a tuner) a channel to receive a signal containing an encoded image and performs a conversion method according to any of the described embodiments.

[0144] A TV, set-top box, mobile phone, tablet, or other electronic device that receives a signal containing encoded images over the air and performs a conversion process.

Claims

1. 1. A method comprising: encoding a portion of the video using the determined number of bits; determining the number of bits to allocate for the portion of the encoded video based on a number of frames; Including, The method, wherein the determining includes using a gain vector from a reinforcement learning agent that uses the latent ability determined from the encoding.

2. 1. An apparatus comprising: encoding a portion of the video using the determined number of bits; determining the number of bits to allocate for the portion of the encoded video based on a number of frames; is configured to perform The apparatus, wherein the determining includes using a gain vector from a reinforcement learning agent that uses the latent ability determined from the encoding.

3. 1. A method comprising: analyzing the video data for a lambda value; determining an index of a vector having a corresponding lambda closest to a target lambda value, the lambda defining a rate-distortion operating point; interpolating values ​​of vectors between the determined index vector and vectors with successive indices using the target lambda value and lambda values ​​of the vectors between the determined index vector and vectors with successive indices; and decoding the video data using the interpolated vector.

4. 1. An apparatus comprising: analyzing the video data for a lambda value; determining an index of a vector having a corresponding lambda closest to a target lambda value, the lambda defining a rate-distortion operating point; interpolating values ​​of vectors between the determined index vector and vectors with successive indices using the target lambda value and lambda values ​​of the vectors between the determined index vector and vectors with successive indices; and decoding the video data using the interpolated vectors.

5. The method of claim 1 or 3 or the apparatus of claim 2 or 4, wherein an Asymmetric Gain Variational Autoencoder (AG-VAE) architecture is used.

6. The method of claim 1 or the apparatus of claim 2, wherein the reinforcement learning is implemented by a deep neural network (DNN).

7. The method of claim 1 or the apparatus of claim 2, wherein the number of frames is a group of pictures (GOP).

8. The method of claim 1 or the apparatus of claim 2, wherein the reinforcement learning agent determines a range of bit rate points.

9. 5. The method of claim 1 or 3 or the apparatus of claim 2 or 4, wherein the video being processed is all intra-mode predicted.

10. The method of claim 1 or the apparatus of claim 2, wherein the reinforcement learning agent receives a frame type as input.

11. A device, An apparatus according to claim 4; 1. A device comprising: at least one of: (i) an antenna configured to receive a signal including a video block; (ii) a band limiter configured to limit the received signal to a frequency band including the video block; and (iii) a display configured to display an output representing the video block.

12. A non-transitory computer-readable medium containing data content generated according to the method of claim 1 or by the apparatus of claim 2 for playback using a processor.

13. A signal comprising video data produced according to the method of claim 1 or by the apparatus of claim 2 for playback using a processor.

14. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method of claim 1 or claim 3.

15. A non-transitory computer readable medium comprising data content including instructions for performing the method of any one of claims 1 or 3 and 5-10.

Images