Rate control machine learning model with feedback control for video encoding

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for encoding a video comprising a sequence of video frames. In one aspect, a method includes, for one or more of the video frames: obtaining a feature embedding of the video frame; processing the feature embedding using a rate control machine learning model to generate a respective score for each of a plurality of quantization parameter values; selecting a quantization parameter value using the scores; determining a cumulative amount of data required to represent (i) an encoded representation of the video frame and (ii) encoded representations of each of the previous video frames; determining, based on the cumulative amount of data, that a feedback control criterion for the video frame is satisfied; updating the selected quantization parameter value; and processing the video frame using an encoding model to generate the encoded representation of the video frame.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application serial number 63 / 109,270 filed on November 3, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This specification relates to processing data using machine learning models, and more specifically to using machine learning models for video encoding. Background Art

[0004] A machine learning model receives input and generates an output, such as a predicted output, based on the received input. Some machine learning models are parametric models and generate an output based on the received input and the values ​​of the model's parameters.

[0005] Some machine learning models are deep models that use multiple layers to generate outputs for received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, each of which applies a nonlinear transformation to the received input to generate an output. Summary of the Invention

[0006] This specification generally describes a video compression system implemented as a computer program on one or more computers at one or more locations that uses a rate-controlled machine learning model to perform video encoding.

[0007] Throughout this specification, "data amount" (e.g., the amount of data required to encode a video frame) may be measured in any suitable unit of measure, such as a number of bits. For example, a target amount of data for encoding a video may be specified by a user of a video compression system.

[0008] According to a first aspect, a method for encoding a video comprising a sequence of video frames to generate a corresponding encoded representation for each video frame is provided, performed by one or more data processing devices, the method comprising, for one or more of the video frames: obtaining a feature embedding of the video frame; processing an input comprising the feature embedding of the video frame using a rate control machine learning model to generate a corresponding score for each of a plurality of possible quantization parameter values; selecting a quantization parameter value from the plurality of possible quantization parameter values ​​using the score; determining a cumulative amount of data required to represent: (i) an encoded representation of the video frame generated according to a quantization step size associated with the selected quantization parameter value, and (ii) an encoded representation of each video frame preceding the video frame; determining, based on the cumulative amount of data, that a feedback control criterion for the video frame is satisfied; updating the selected quantization parameter value in response to determining that the feedback control criterion is satisfied; and processing the video frame using a coding model according to the quantization step size associated with the selected quantization parameter value to generate an encoded representation of the video frame.

[0009] In some implementations, the input processed by the rate control machine learning model also includes a target amount of data for representing the encoded video.

[0010] In some implementations, a feedback control criterion for a video frame specifies a target value range, and wherein determining that the feedback control criterion for the video frame is satisfied comprises determining that a cumulative amount of data required to represent: (i) an encoded representation of the video frame generated according to a quantization step size associated with a selected quantization parameter value, and (ii) an encoded representation of each video frame preceding the video frame, is outside the target value range specified by the feedback control criterion.

[0011] In some implementations, updating the selected quantization parameter value in response to determining that the feedback control criterion is satisfied includes determining an update to the selected quantization parameter value based on a difference between: (i) the accumulated data amount, and (ii) a nearest endpoint of a target value range specified by the feedback control criterion for the video frame.

[0012] In some implementations, updating the selected quantization parameter value in response to determining that a feedback control criterion is satisfied includes: determining that the accumulated data amount is greater than an upper endpoint of a target value range specified by the feedback control criterion; and adjusting the selected quantization parameter value to increase a quantization step size used to generate an encoded representation of the video frame.

[0013] In some implementations, updating the selected quantization parameter value in response to determining that a feedback control criterion is satisfied includes: determining that the accumulated data amount is less than a lower endpoint of a target value range specified by the feedback control criterion; and reducing the selected quantization parameter value to reduce a quantization step size used to generate an encoded representation of the video frame.

[0014] In some implementations, updating the selected quantization parameter value in response to determining that a feedback control criterion is satisfied includes: identifying a set of candidate quantization parameter values, wherein the set of candidate quantization parameter values ​​is a proper subset of multiple possible quantization parameter values ​​having the highest scores; and selecting an updated quantization parameter value from the set of candidate quantization parameter values.

[0015] In some implementations, selecting a quantization parameter value from a plurality of quantization parameter values ​​using a score includes: identifying a proper subset of a plurality of possible quantization parameter values ​​having a highest score; determining a probability distribution over the proper subset of the plurality of possible quantization parameter values; and sampling the quantization parameter value according to the probability distribution over the proper subset of the plurality of possible quantization parameter values.

[0016] In some implementations, identifying a proper subset of the plurality of possible quantization parameter values ​​having the highest score includes identifying a predefined number of the plurality of possible quantization parameter values ​​having the highest score.

[0017] In some implementations, the rate control machine learning model has a neural network architecture that includes one or more recurrent neural network layers.

[0018] In some implementations, the recurrent neural network layer includes one or more long short-term memory (LSTM) recurrent neural network layers.

[0019] In some implementations, for each video frame, the input processed by the rate control machine learning model also includes the quantization parameter value used to generate the encoded representation of the previous video frame.

[0020] In some implementations, the method further includes: obtaining a corresponding initial feature embedding for each video frame based on features of the video frame; processing the initial feature embeddings of the video frame using an embedding neural network comprising one or more self-attention neural network layers to generate an updated feature embedding for each video frame; and determining a corresponding feature embedding for each video frame based at least in part on the updated feature embeddings of the video frame generated by the embedding neural network.

[0021] In some implementations, the embedded neural network includes a transformer neural network.

[0022] In some implementations, for each video frame, the initial feature embedding for each video frame is based at least in part on noise energy characteristics of the video frame, motion vector statistics of the video frame, or both.

[0023] In some implementations, for each video frame, a feature embedding for the video frame is based at least in part on one or more of: a frame index of the video frame, a frame type of the video frame, an amount of data required to represent an encoded representation of a previous video frame in the video, and data characterizing a difference between: (i) a cumulative amount of data required to represent an encoded representation of each video frame in the video that precedes the video frame, and (ii) a target amount of data for representing the encoded video.

[0024] According to another aspect, a method for encoding a video comprising a sequence of video frames to generate a corresponding encoded representation of each video frame is provided, performed by one or more data processing devices, the method comprising, for each video frame: obtaining a feature embedding of the video frame; processing an input comprising the feature embedding of the video frame using a rate control machine learning model to generate a corresponding score for each of a plurality of possible quantization parameter values; selecting a quantization parameter value from the plurality of possible quantization parameter values ​​using the score; and processing the video frame using the encoding model based on a quantization step size associated with the selected quantization parameter value to generate an encoded representation of the video frame; wherein the rate control machine learning model has a plurality of model parameters trained on a set of training examples, wherein each training example comprises data defining: (i) a corresponding feature embedding for each training video frame of the training video, and (ii) a corresponding target quantization parameter value for each training video frame.

[0025] In some implementations, for each video frame, the input processed by the rate control machine learning model also includes a target amount of data for representing the encoded video.

[0026] In some implementations, training the rate control machine learning model on the set of training examples includes, for each training example: processing, using the rate control machine learning model, an input comprising a corresponding feature embedding for each training video frame to generate, for each training video frame, a corresponding score for each of a plurality of possible quantization parameter values; and, for each training video frame, determining an update to a current value of a model parameter of the rate control machine learning model based on an error between: (i) the scores for the plurality of possible quantization parameter values ​​generated for the training video frame, and (ii) a target quantization parameter value for the training video frame.

[0027] In some implementations, the error between (i) the scores for the plurality of possible quantization parameter values ​​generated for the training video frame and (ii) the target quantization parameter value for the training video frame comprises a cross-entropy error.

[0028] In some implementations, for each training video frame, the rate control machine learning model generates an output that also includes an estimate of the amount of data required to represent an encoded representation of the training video frame.

[0029] In some implementations, the method further includes, for each training video frame, determining an update to a current value of a model parameter of a rate control machine learning model based on an error between: (i) an estimate of an amount of data required to represent an encoded representation of the video frame, and (ii) an actual amount of data required to represent the encoded representation of the video frame.

[0030] In some implementations, the method further includes determining an update to a current value of a model parameter of the rate control machine learning model based on an error between: (i) an estimated total amount of data required to represent an encoded representation of the training video frame, and (ii) the total amount of data required to represent the encoded representation of the training video frame.

[0031] In some implementations, for one or more of the training examples, a target quantization parameter value for a training video frame of the training example is generated by performing an optimization to determine a quantization parameter value for the training video frame that minimizes an error metric between: (i) the training video frame, and (ii) a reconstruction of the training video frame determined by processing an encoded representation of the training video frame generated using the quantization parameter value.

[0032] In some implementations, the optimization is a constrained optimization subject to the constraint that a total amount of data required to represent the encoded representation of the training video frames generated using the quantization parameter values ​​is less than a target amount of data for representing the encoded representation of the training video frames.

[0033] In some implementations, each training example also includes data defining a target amount of data for representing an encoded representation of a training video frame in the training video.

[0034] In some implementations, training the rate control machine learning model includes: training the rate control machine learning model on a first set of training examples; generating a second set of training examples using the rate control machine learning model, wherein for each training example in the second set of training examples: determining a corresponding target quantization parameter value for each training video frame based on a current value of a model parameter of the rate control machine learning model; and if each training video frame is encoded using the target quantization parameter value for the training video frame, the target amount of data used to represent an encoded representation of the training video frame in the training video is the amount of data required to represent the encoded representation of the training video frame; and training the rate control machine learning model on the second set of training examples.

[0035] According to another aspect, a system is provided that includes: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the methods described herein.

[0036] According to another aspect, one or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the methods described herein are provided.

[0037] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0038] The video compression system described herein uses a rate control machine learning model that enables the video compression system to encode video at a higher compression rate while maintaining comparable video quality compared to a video compression system using a conventional rate control algorithm. Consequently, the video compression system can achieve more efficient resource usage, for example, by reducing the amount of storage required to store the encoded video and by reducing the amount of bandwidth required to transmit the encoded video over a data communication network (e.g., the Internet).

[0039] The video compression system can use a feedback control mechanism to adjust the quantization parameter value selected using a rate control machine learning model. The feedback control mechanism can adjust the quantization parameter value to increase the likelihood that the encoded video generated by the video compression system is represented by an amount of data (e.g., number of bits) that does not significantly deviate from the target amount of data used to represent the encoded video. That is, the feedback control mechanism can increase the likelihood that the video compression system optimally (fully) utilizes the target amount of data to represent the encoded video. Using less than the target amount of data to represent the encoded video will result in unnecessary information loss, thereby hindering accurate decompression of the encoded video. Using more than the target amount of data to represent the encoded video will cause the encoded video to violate memory usage constraints and thus be unusable. Therefore, the feedback control mechanism can achieve more efficient use of computing resources, such as memory resources.

[0040] The rate control machine learning model of the video compression system can learn an effective rate control strategy through imitation learning, for example, by learning to generate quantization parameter values ​​that match target quantization parameter values ​​specified by training examples in a set of training data. Compared with using other learning techniques (for example, reinforcement learning techniques), the use of imitation learning techniques can achieve training of the rate control machine learning model in a shorter duration and using fewer computing resources (for example, memory and computing power). Specifically, the rate control machine learning model can be trained to imitate an optimal or near-optimal rate control strategy (for example, determined by numerical optimization), which is much faster than trying to learn an effective rate control strategy from scratch through reinforcement learning. (The rate control machine learning model can also be trained by techniques other than imitation learning, such as reinforcement learning).

[0041] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a block diagram illustrating an example of encoding a video comprising a sequence of video frames.

[0043] Figure 2 is a block diagram of an example video compression system.

[0044] Figure 3 is a block diagram of an example rate control model.

[0045] Figure 4 is a diagram illustrating a data amount trajectory with respect to a target value range for each video frame.

[0046] Figure 5 is a flowchart of an example process for encoding a video frame.

[0047] Figure 6 is a flowchart of an example process for training a rate-controlled model.

[0048] Figure 7 is a flow chart of an example process for determining a second set of training examples for training a rate-controlled machine learning model.

[0049] Like reference numbers and designations throughout the various drawings represent like elements. DETAILED DESCRIPTION

[0050] Figure 1 An example block diagram 100 is shown illustrating operations performed by a video compression system for encoding a video comprising a sequence of video frames. In one example, the video frames may comprise images of the real world captured using a camera. Figure 2 An example video compression system, such as video compression system 200 , that may perform the operations illustrated in block diagram 100 is described in further detail.

[0051] The video compression system 200 processes a video 102 comprising a sequence of video frames to generate a corresponding encoded representation of each video frame (e.g., represented by an ordered set of numerical values ​​such as a vector or matrix of values), such that the video compression system 200 performs compression of the video 102 to generate a compressed representation 130 of the video.

[0052] The video may be a black and white video, where each video frame is represented by a corresponding grayscale pixel value for each pixel in the video frame (e.g., represented by a numerical value). In another example, the video may be a color video, where each video frame is represented by a corresponding red, green, and blue (RGB) pixel value for each pixel in the video frame (e.g., represented by a numerical vector).

[0053] The video compression system 200 is configured to process a sequence of video frames (e.g., video frames 104, 114, 124) of the video 102 and, for each video frame in the sequence of video frames, process data representing any previous video frames preceding the video frame that have been compressed by the video compression system 200 (e.g., an amount of data required to represent an encoded representation of the previous video frame, etc.) to generate an encoded representation of each video frame (e.g., encoded representations 106, 116, and 126, respectively). Each of the encoded representations 106, 116, 126 has a smaller amount of data (e.g., a smaller number of bytes) than the corresponding video frame 104, 114, 124. In addition, the video compression system 200 can generate a compressed representation 130 by further processing a target amount of data (a "data budget"). The video compression system 200 can generate the compressed representation 130 using the target amount of data as a target amount of data to represent the compressed representation 130, as described below with reference to Figure 2 The target amount of data for the compressed representation 130 representing the video may be measured in bits, for example, and may be specified by a user of the video compression system 200, for example.

[0054] The encoded representations of the video frames can represent compressed versions of the video frames such that each encoded representation can be represented using a smaller amount of data (e.g., fewer bits) than required to represent the corresponding video frame. That is, the video compression system 200 processes the video 102 to generate the compressed representation 130 such that the compressed representation 130 requires a smaller amount of data to represent than the amount of data required to represent the video 102.

[0055] The encoded representation of a video frame may be represented by an ordered set of values, such as a bit sequence, and the encoded representation of the video frame may collectively define a compressed representation 130 of the video.

[0056] The compressed representation 130 (i.e., including the encoded representation of the video frames) may be transmitted (e.g., over a data communications network, such as the Internet, a local area network (LAN), a wide area network (WAN), etc.) and / or stored in a data storage device (e.g., a physical data storage device or a logical data storage area). The transmission may be to a user device (e.g., a computer or mobile device), a computer server (e.g., a video platform server), or a data storage device.

[0057] The compressed representation 130 of the video may be processed to generate a reconstruction (e.g., an approximate reconstruction or an exact reconstruction) of the video represented by the sequence of video frames processed to generate the encoded representation. Specifically, the video decompression system may process the corresponding encoded representation of each video frame to generate a reconstruction of the video frame, and the reconstructions of the video frames may collectively define a reconstruction of the video.

[0058] Figure 2 An example video compression system 200 is shown. Video compression system 200 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0059] The video compression system 200 processes a video frame 204 from a video 202 comprising a sequence of video frames to generate an encoded representation 228 of the video frame 204. The video frame 204 may comprise one or more pixels (e.g., where each pixel is represented by an ordered set of values ​​for a grayscale pixel or a red, green, and blue (RGB) pixel). The encoded representation 228 may be represented by an ordered set of values, such as a bit sequence, for example.

[0060] Video compression system 200 includes embedding network 208, rate control model 300, quantization parameter (QP) selection engine 214, data estimation engine 218, QP update engine 222, and encoding model 226, each of which will be described in more detail below.

[0061] The video compression system 200 may use the embedding neural network 208 to generate a corresponding feature embedding for each video frame in the video, i.e., a representation of the features of the video frame as an ordered set of values ​​(e.g., a vector or matrix of values).

[0062] For example, the video compression system 200 may generate an “initial” feature embedding for each video frame based on features of the video frame, such as statistical features, such as noise energy features of the video frame and / or motion vector statistics of the video frame. This may be performed using a conventionally designed unit (not shown). The unit may not include adaptive parameters. In one example, the unit may partition the video frame into portions and obtain the initial feature embedding as a data set including at least one corresponding statistical value for each portion. The video compression system 200 may then jointly process the initial feature embeddings of the video frames using the embedding neural network 208 to generate a corresponding updated feature embedding for each video frame. The video compression system 200 may then determine a corresponding feature embedding for each video frame based on the updated feature embedding generated for the video frame by the embedding neural network as described below. Note that in one variation, the embedding network 208 may (e.g., sequentially) receive the original video frames of the video 202 and generate corresponding feature embeddings from them, which function as the “updated feature embeddings” described below.

[0063] The embedding network 208 can have any suitable neural network architecture that enables it to perform its described functions, e.g., processing the corresponding initial feature embeddings of the video frames to generate updated feature embeddings of the video frames. Specifically, the embedding neural network can include any suitable number (e.g., 1 layer, 5 layers, or 25 layers) of any suitable type of neural network layers (e.g., fully connected layers, attention layers, convolutional layers, etc.) connected in any suitable configuration (e.g., as a linear sequence of layers). In a particular example, the embedding neural network can include one or more self-attention neural network layers. For example, the embedding neural network can include a transformer neural network, e.g., the encoder portion of a transformer neural network described in Ashish Vaswani et al., “Attention is all you need,” Advances in Neural Information Processing Systems 30 (NIPS), 2017, arXiv:1706.03762v5.

[0064] For each video frame, the video compression system 200 can determine a corresponding feature embedding for the video frame based at least in part on the updated feature embedding for the video frame (i.e., generated by the embedding neural network 208). In addition to the updated feature embedding for the video frame, the corresponding feature embedding for the video frame can include, for example, a concatenation of: a frame index for the video frame (e.g., indicating the position of the video frame in a sequence of video frames in the video), a frame type for the video frame (e.g., represented by a one-hot vector over possible frame types (e.g., keyframe, alternative reference frame, inter-frame)), an amount of data required to encode a representation of a previous video frame (e.g., represented by a number of bits), and / or data representing a difference between: (i) a cumulative amount of data required to encode a representation of each video frame in the video preceding the video frame, and (ii) a target amount of data for representing the encoded video (a "data budget") (e.g., a number representing the number of bits required to encode the video). Figure 1 The compressed representation is the target data volume of 130).

[0065] As part of generating the encoded representation 228 of the video frame 204, the rate control model 300 processes the input 210, which includes at least the feature embedding of the video frame, to generate a corresponding score 212 for each of a plurality of possible quantization parameter values. The input 210 may also include a target data amount (e.g., represented by a number of bits) for representing the encoded video and / or a quantization parameter value used to generate an encoded representation of a previous video frame (e.g., as a concatenation of the corresponding feature embedding of the video frame, the target data amount, and the previous quantization parameter value). The corresponding score for each possible quantization parameter value may be represented, for example, by a numerical value.

[0066] The possible quantization parameter values ​​can be represented by a set of integer values ​​(e.g., integer values ​​in the range [0, 255]). Each quantization parameter value can be associated with a corresponding quantization step size (e.g., by mapping the quantization parameter value to the corresponding quantization step size according to a predefined monotonic mapping). The coding model 226 uses the quantization step size to generate the encoded representation 228. Here, the "quantization step size" is a numerical parameter that varies inversely proportional to the accuracy required to encode the video frame 204 to form the encoded representation 228. That is, it indicates the level of error that can be tolerated between the video frame 204 and the reconstructed video frame that can be obtained from the encoded representation 228. As described in further detail below, a larger quantization step size results in a smaller amount of data for the encoded video frame, while a smaller quantization step size results in a smaller quantization error.

[0067] The rate control model 300 can have any suitable machine learning model architecture (e.g., a neural network architecture, a random forest architecture, or a support vector machine architecture) such that it can perform its described function, i.e., process inputs comprising at least feature embeddings of video frames to generate quantization parameter scores. In a specific example, if the rate control model 300 is a neural network model, the rate control model can include any suitable type of neural network layers (e.g., fully connected layers, attention layers, convolutional layers, etc.) connected in any suitable number (e.g., 1 layer, 5 layers, or 25 layers) and in any suitable configuration. In a specific example, as described below with reference to Figure 3 As described in further detail, the rate control model may include one or more recurrent neural network layers (e.g., long short-term memory (LSTM) neural network layers) and one or more multi-layer perceptron (MLP) neural network layers.

[0068] A quantization parameter (QP) selection engine 214 processes the scores 212 to select a quantization parameter (QP) value 216 for the video frame 204. The QP selection engine 214 can select the QP value 216 based on the scores 212. Some example techniques that the QP selection engine 214 can use to select the QP value 216 for the video frame 204 are described below.

[0069] In one example, the QP selection engine 214 may select the QP value 216 that corresponds to the maximum score among the scores 212 .

[0070] In another example, the scores 212 may define a probability distribution over a set of possible QP values, and the QP selection engine 214 may sample QP values ​​216 from the possible quantization parameter values ​​according to the probability distribution.

[0071] In another example, the QP selection engine 214 may sample a QP value 216 from a subset of possible quantization parameter values ​​based on corresponding scores in the scores 212. In a specific example, the QP selection engine 214 samples the QP value 216 from N (e.g., 5, 15, or 25) quantization parameter values ​​corresponding to the highest scores in the scores 212, where N is a positive integer (e.g., a predefined integer). The QP selection engine 214 may apply a softmax function to the scores of the N quantization parameter values ​​(i.e., those with the highest scores) to generate a probability distribution over the N quantization parameter values, and sample the QP value 216 from the N quantization parameter values ​​based on the probability distribution over the N quantization parameter values.

[0072] If the video compression system 200 samples QP values ​​216 from the full set of possible QP values, the video compression system 200 may occasionally sample a poorly chosen QP value. Selecting this poorly chosen QP value may result in the amount of data used to generate the encoded representation 228 being too large or too small, and may cause the amount of data required to represent the compressed representation of the video 202 to deviate significantly from a target amount of data for representing the compressed representation. Sampling QP values ​​216 from a subset of possible quantization parameter values ​​associated with the maximum score in the scores 212 may help prevent the video compression system from sampling poor QP values ​​and, therefore, from deviating less from the target amount of data for the compressed representation.

[0073] The data estimation engine 218 processes the QP value 216 to determine a cumulative amount of data (e.g., cumulative data amount 220) required to represent (i) an encoded representation of a video frame generated according to the QP value 216, and (ii) an encoded representation of each video frame preceding the video frame. The cumulative data amount 220 can be represented, for example, by a number of bits. For example, the data estimation engine 218 can determine the cumulative data amount 220 as the sum of: (i) the amount of data required to represent the current video frame 204 if the current video frame were encoded using the QP value 216, and (ii) the total amount of data required to represent the encoded representations of the previous video frames.

[0074] The video compression system 200 uses the accumulated data amount 220 to determine whether the feedback control criterion is met. If the video compression system 200 determines that the feedback control criterion is not met, the video compression system uses the coding model 226 to generate an encoded representation 228 of the video frame 204 using the QP value 216 selected using the rate control model 300. If the video compression system 200 determines that the feedback control criterion is met, the video compression system 200 uses the quantization parameter (QP) update engine 222 to determine an updated QP value 224 and uses the updated QP value 224 to generate an encoded representation of the video frame 204.

[0075] If the accumulated data amount 220 is outside the target value range specified by the feedback control criterion for the video frame, the video compression system 200 may determine that the feedback control criterion is satisfied. The target value range may indicate how much of the data budget (i.e., the target data amount) the video compression system 200 should have used to encode the video frames in the video up to and including the video frame 204. Typically, the feedback control criterion specifies a different target value range for each video frame, i.e., depending on the frame index of the video frame. Figure 4 An example illustrating the corresponding target value range for each video frame in the video is shown.

[0076] The target value range specified by the feedback control standard of the video frame may include an upper endpoint and a lower endpoint. If the accumulated data amount is higher than the upper endpoint, the QP update engine 222 may increase the QP value 216 of the video frame, that is, reduce the number of bits used to encode the current video frame. If the accumulated data amount is lower than the lower endpoint, the QP update engine 222 may decrease the QP value 216 of the video frame, that is, increase the number of bits used to encode the current video frame. For example, as shown in FIG. Figure 5 As further described, QP update engine 222 may determine updates to quantization parameter value 216 based on a difference between (i) accumulated data amount 220 and (ii) the nearest endpoint of a target value range specified by a feedback control criterion for the video frame.

[0077] The coding model 226 is configured to process the video frames 204 according to the selected QP value (i.e., the QP value 216 selected by the rate control model 300, or the updated QP value 224 if the feedback control criteria are met) to generate an encoded representation 228 of the video frame 204. The encoded representation 228 is a compressed representation of the video frame 204 (i.e., one that takes up less space in memory). The coding model 226 can be any suitable coding model, such as the libvpx implementation of VP9, ​​Xvid, FFmpeg MPEG-4, or DivX Pro codec. Corresponding algorithms are known for decompressing the encoded representation 228 (e.g., after all encoded representations of the video frames 204 of the video 202 have been transmitted to a receiver via a communication channel) to reconstruct the video frame 204 with an error that depends on the selected QP value so that a video including the reconstructed video frame can be displayed on a screen.

[0078] Typically, as part of generating an encoded representation 228 of a video frame, the encoding model 226 performs quantization, i.e., quantizing the video frame 204 or data derived from the video frame 204 according to a quantization step size specified by the selected QP value. Quantizing a value may refer to mapping the value to the closest one in a set of quantization values. The encoding model 226 may perform quantization, for example, as part of implementing an entropy coding algorithm. A lower quantization step size may result in finer quantization that minimizes information loss, while a higher quantization step size may result in coarser quantization that increases information loss. For example, if the set of quantization values ​​is considered an ordered sequence of increasing values, the quantization step size may be the pairwise spacing of quantization values ​​in the set. In other examples, the pairwise spacing of quantization values ​​may not be constant according to an equation that uses the quantization step size as a parameter (e.g., the spacing between lower quantization values ​​in the set may be smaller than the spacing between higher quantization values ​​in the set). In either case, determining the quantization step size corresponds to selecting a corresponding set of quantization values. The highest and lowest quantization values ​​for each set may be the same (e.g., all sets may have a lowest quantization value of 0 and a highest quantization value of 255).

[0079] Figure 3 An example rate control model 300 having a neural network architecture is shown. The rate control model 300 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0080] The rate control model 300 processes a corresponding input 302 including a corresponding feature embedding of each corresponding video frame in the video at each of a plurality of time steps to generate a corresponding quantization parameter (QP) value for each video frame. The QP value for each video frame can be used to generate a corresponding quantization parameter (QP) value for each video frame by an encoding model (e.g., Figure 2 The rate control model 300 may be associated with a quantization step size for generating an encoded representation of a video frame using the encoding model 226. Optionally, during training, the rate control model 300 may further generate a predicted amount of data ("data amount") for each video frame. The data amount for each video frame may represent, for example, a predicted amount of data required to generate an encoded representation of the video frame based on the quantization step size associated with the QP value of the video frame.

[0081] The rate control model 300 includes a long short-term memory (LSTM) neural network layer 306, a multi-layer perceptron (MLP) neural network layer 308, and a (second) multi-layer perceptron (MLP) neural network layer 312. Figure 3 As shown, a separate branch may be provided for each of a set of possible QP values ​​314, 324, 334 to generate a score for the corresponding QP value, the branch comprising a corresponding LSTM layer 306, a corresponding (first) MLP neural network layer (this layer may be optionally omitted if the amount of data is not calculated for the QP value), and a corresponding (second) MLP neural network layer 312 to generate the corresponding score. However, in a variant, there may be only a single LSTM layer, a single (optional) MLP neural network, and a single (second) MLP neural network layer, wherein the (multiple) MLP neural networks are configured to generate a corresponding output for each of the corresponding possible QP values.

[0082] The rate control model 300 sequentially processes a corresponding input 302 corresponding to each video frame in the video to generate a corresponding QP value for each video frame in the video. For each video frame, the input includes a feature embedding for the video frame, and for each video frame after the first video frame, the input also includes a QP value selected for the previous video frame. As described above, the QP value selected for the previous video frame can be the QP value generated by the rate control model 300 for the previous video frame, or it can be an updated QP value selected by the video compression system if the feedback control criteria are met for the previous video frame.

[0083] Specifically, for each video frame, the LSTM layer 306 processes the input 302 of the video frame to generate an LSTM layer output, and the MLP layer 312 processes the LSTM layer output to generate a score distribution over a set of possible QP values ​​for the video frame. Optionally, during training, the MLP layer 308 can process the LSTM output to generate a predicted amount of data for the video frame (e.g., 310, 320, 330), i.e., use the QP value selected for the video frame by the rate control model to estimate the amount of data required to represent the video frame (e.g., in bits). (The MLP layer 308 generates the predicted amount of data without having access to the QP value ultimately selected for the video frame). Training the rate control model to accurately estimate the amount of data required to represent each video frame (310, 320, 330) provides an auxiliary training signal that enables the LSTM layer 306 to generate a more informative internal representation of the video frame and, therefore, more efficiently select QP values. This will be discussed below with reference to Figure 6 The training rate control model 300 is described in more detail.

[0084] Figure 4 is a graph illustrating the data volume trajectory generated for encoding a video relative to a target value range for each video frame in the video. For example, the data volume trajectory can be generated by a rate control model (e.g. Figure 2 The rate control model 300) is generated.

[0085] The data amount track 402 is illustrated on a graph with the cumulative data amount 404 as the 'y-axis' and the encoding steps 406 as the 'x-axis'. Each data amount track in the data amount track 402 represents a sequence of the cumulative amount of data required to represent the video frames in the video up to the current video frame. Each video frame is encoded with a corresponding encoding step in the encoding steps 406. Figure 4 ,As described below, the data volume trajectory is shown using dashed and solid lines.

[0086] The graph includes a target value range for each video frame. The target value range includes a sequence spanning the upper endpoint 408 of the encoding step 406 and a sequence spanning the lower endpoint 410 of the encoding step 406. Figure 4 , the upper endpoint 408 and the lower endpoint 410 are respectively shown using thick solid lines.

[0087] Each data amount trajectory including at least one cumulative data amount exceeding the upper endpoint of the target range at the corresponding encoding step is shown using a dotted line. Each data amount trajectory including at least one cumulative data amount less than the lower endpoint of the target range at the corresponding encoding step is also shown using a dotted line. Each data amount trajectory of the cumulative data amount between the upper endpoint 408 and the lower endpoint 410 at each corresponding encoding step is shown as a solid line.

[0088] At each encoding step where the data volume trajectory is outside the target value range, the quantization parameter value used to generate the video frame at that encoding step can be adjusted. For example (taking the case where increasing the QP value corresponds to decreasing the quantization step size), if the cumulative data volume of the encoding step is higher than the upper endpoint, the quantization parameter value can be adjusted downward, that is, the number of bits used to encode the current video frame can be reduced. If the cumulative data volume of the encoding step is lower than the lower endpoint, the quantization parameter value of the video frame can be adjusted upward, that is, the number of bits used to encode the current video frame can be increased, as shown in FIG. Figure 5 described in further detail.

[0089] Figure 5 is a flow chart of an example process for encoding a video frame. For convenience, process 500 is described as being performed by a system of one or more computers located in one or more locations. For example, a video compression system appropriately programmed according to this specification, such as Figure 2 The video compression system 200 may perform process 500 .

[0090] The system obtains feature embeddings for video frames (502). The system can generate a corresponding feature embedding for each video frame in the video, for example, by generating a corresponding initial feature embedding for each video frame based on statistics of the video frame (e.g., noise energy statistics and / or motion vector statistics). The system can then jointly process the initial feature embeddings of the video frames using an embedding neural network to generate a corresponding updated feature embedding for each video frame. The system can optionally include additional data in the updated feature embeddings of the video frames to generate a final feature embedding for the video frames, for example, data defining a frame index and / or a frame type of the video frame.

[0091] The system generates a corresponding score for each of a plurality of possible quantization parameter values ​​(504). The system may use a rate control machine learning model (e.g., Figure 2 For example, the rate control model may process feature embeddings of a video frame to generate a corresponding score for each quantization parameter value in a set of possible quantization parameter values.

[0092] The system selects a quantization parameter value for the video frame (506). The system can select the quantization parameter value from the possible quantization parameter values ​​based on their corresponding scores. For example, the system can select the quantization parameter value corresponding to the maximum score, or the system can select the quantization parameter value by sampling from the possible quantization parameter values ​​based on their corresponding scores. In one example, the system can sample the quantization parameter value from a subset of possible quantization parameter values ​​based on their corresponding scores. The subset can include possible quantization parameter values, for example, corresponding to the highest scores (e.g., 5, 15, or 25 possible quantization parameter values).

[0093] The system determines a cumulative amount of data required to represent (i) an encoded representation of the video frame generated based on the QP value and (ii) an encoded representation of each previous video frame (508). The system can determine the cumulative amount of data by determining a respective amount of data required to represent each video frame (e.g., the video frame and each previous video frame) and summing the respective amounts of data. For example, the system can determine the respective amount of data required to generate the encoded representation of each video frame based on respective quantization parameter values ​​for the video frames.

[0094] The system determines whether a feedback control criterion for the video frame is satisfied (510). The feedback control criterion may include a condition that the cumulative data volume is outside a target value range specified by the feedback control criterion. The target value range specified by the feedback control criterion may include an upper endpoint and (optionally) a lower endpoint. The target value range may be determined, for example, by fitting two parameterized logarithmic functions as bounds on the set of training data volume trajectories used to train the rate control model, such that i) most (e.g., at least a predetermined proportion) of the data volume trajectories fall within the bounds, and ii) the bounds are wide in the middle of the encoding step, such as:

[0095] a1log(a2x+a3)+a4x+a5, (1)

[0096] Where a1, a2, a3, a4, and a5 are parameters determined by the fitting process, and x represents the cumulative amount of data at each encoding step in the data volume trajectory. Other functions can also be used.

[0097] For example, if the accumulated data amount is outside the target value range, eg, above the upper end point of the target value range, or below the lower end point of the target value range, the system may determine that the feedback control criterion is satisfied.

[0098] The system updates the selected quantization parameter value based on whether the feedback control criterion is met (512). If the feedback control criterion is not met, the system does not update the quantization parameter value selected by the rate control model. If the feedback control criterion is met, the system may determine an update to the quantization parameter value based on (i) the accumulated data amount and (ii) the nearest endpoint of a target value range specified by the feedback control criterion for the video frame.

[0099] In one example, if a feedback control criterion is met such that the accumulated data amount is greater than an upper endpoint of a target value range, the system can adjust the selected quantization parameter value to increase the quantization step size used to generate an encoded representation of the video frame (e.g., to reduce the amount of data).

[0100] In another example, if the feedback control criterion is satisfied such that the accumulated data amount is less than the lower endpoint of the target value range, the system can adjust the selected quantization parameter value to decrease the quantization step size (e.g., to increase the data amount) used to generate the encoded representation of the video frame.

[0101] In response to determining that the feedback control criterion is satisfied, the system can identify a set of M candidate quantization parameter values, where the set of candidate quantization parameter values is a proper subset of the plurality of possible quantization parameter values corresponding to the highest score. The plurality of possible quantization parameter values corresponding to the highest score can include N highest scores, e.g., 20, 40, or 60. The numbers N and M, where M is less than N, can be, e.g., predefined. The system can determine the update to the quantization parameter value for the video frame by selecting an updated quantization parameter value from the set of candidate quantization parameter values. For example, the system can determine an ordered list of the first N quantization parameter values from smallest to largest, and select the updated quantization parameter value as the one in the ordered list at position j, where j is an integer between 1 and N. N They are indexed. The system can then determine the update as:

[0102]

[0103] where i indexes the original quantization parameter values QP i indexed, j indexes the updated quantization parameter values QP j indexed, t indexes the video frames, B t denotes the accumulated data amount for video frame t, denotes the lower endpoint of the target value range for video frame t, denotes the upper endpoint of the target value range for video frame t, and a denotes a constant (e.g., a positive real number) that adjusts the strength of the feedback control.

[0104] In other words, the algorithm samples the one of the M quantization parameter values with the highest respective score; determines from it a corresponding integer value i, which is one of the values from 1 to M and indicates the position of the sampled quantization parameter value in the ordered list of the M quantization parameter values; obtains from equation (2) a corresponding value of j; and updates the selected quantization parameter value to the one of the first N quantization parameter values with the highest respective score that is at position j in the ordered list of the N quantization parameter values.

[0105] Note that this is not the only way to update the selected quantization parameter value obtained in (506) in (512). For example, an alternative can be to, when the feedback control criterion is satisfied because the accumulated data amount is greater (less) than the higher (lower) endpoint of the target value range, repeatedly modify the selected quantization parameter value to increase (decrease) the quantization step size until the accumulated data amount is within the target value range.

[0106] Using a feedback control mechanism to adjust quantization parameter values ​​can increase the likelihood that encoded video generated by a video compression system is represented by an amount of data (eg, number of bits) that does not significantly deviate from a target amount of data for representing the encoded video.

[0107] The system can process the video frame using the coding model to generate an encoded representation of the video frame (514). The coding model can generate the encoded representation of the video frame based on the quantization step size associated with the selected quantization parameter value (i.e., updated if the determination in step 510 is positive). The coding model can be any suitable coding model, such as libvpx (i.e., an open source implementation of VP9), Xvid, FFmpeg MPEG-4, or DivX Pro codecs.

[0108] Figure 6 is a flow chart of an example process for training a rate control model. For convenience, process 600 is described as being performed by a system of one or more computers located in one or more locations.

[0109] As described below, the system can use imitation learning to train the rate control machine learning model on the first set of training examples. More generally, the system can use any suitable technique, such as imitation learning or reinforcement learning, to train the rate control machine learning model.

[0110] In one example, after training the rate control model on a first set of training examples, the system can use the rate control model to generate a second set of training examples and then train the rate control model on the second set of training examples. Figure 6 As described in further detail in , the system can use a rate control model to generate a second set of training examples from a first set of training examples.

[0111] The system may perform steps (602)-(610) in each of one or more training iterations to train the rate control model. For convenience, the system is described below for a single training iteration.

[0112] The system receives training examples (602). Each training example may include data defining (i) a corresponding feature embedding for each training video frame of a training video, and (ii) a corresponding target quantization parameter value for each training video frame. For example, the system may sample a batch of training examples from a set of training examples.

[0113] For each video frame in each training example, the system generates a corresponding score for each of a plurality of possible quantization parameter values ​​(604). The system can use a rate control model to process at least the feature embeddings of the video frame to generate corresponding scores for the plurality of possible quantization parameter values. For example, the system can use Figure 2 and Figure 3 The rate control model described in generates corresponding scores for multiple possible quantization parameter values ​​from the integers [0, 255].

[0114] Optionally, for each video frame in each training example, the system selects a quantization parameter value for the video frame (606). The system may select the quantization parameter for the video frame based on the corresponding scores of the multiple possible quantization parameter values. For example, the system may select the quantization parameter value corresponding to the maximum score. In another example, the system may sample quantization parameter values ​​from the multiple possible quantization parameter values ​​based on the corresponding scores. In another example, the system may sample quantization parameter values ​​from a subset of possible quantization parameter values. The subset may include N possible quantization parameter values ​​corresponding to the N maximum scores. The system may generate a probability distribution over the N possible quantization parameter values ​​in the subset (e.g., using a softmax function over the N corresponding scores) and sample the quantization parameter values ​​based on the probability distribution.

[0115] The system determines updates to current values ​​of model parameters of the rate control model using supervised learning techniques (608). For each training example, the system can determine the update based on the error between (i) the scores of possible quantization parameter values ​​generated for the training video frame and (ii) the target quantization parameter value for the training video frame for each training video frame in the training video. The system can determine the update by determining (e.g., using backpropagation) the gradient of an objective function for each training example that measures the error and applying the gradient (e.g., a function of the gradient, such as the average of the gradients) to update the current values ​​of the model parameters of the rate control model using an appropriate gradient descent optimization technique, such as RMSprop or Adam. For example, the target quantization parameter value for each training video frame can be represented by a one-hot vector. The error between (i) the scores of possible quantization parameter values ​​generated for the training video frame and (ii) the target quantization parameter value for the training video frame can include, for example, a cross-entropy error.

[0116] In one example, for each training video frame in each training example, the rate control model may generate an output further including an estimate of the amount of data required to represent an encoded representation of the training video frame (e.g., as described with reference to Figure 3 The system may determine an update to a current value of a model parameter of the rate control model, the update further based on an error between (i) an estimate of an amount of data required to represent an encoded representation of the video frame and (ii) an actual amount of data required to represent an encoded representation of the video frame determined using the target quantization parameter value included in the training example. The system may determine the error for each training example as:

[0117]

[0118] Among them L frame represents the error, t indexes the training video frame, T represents the number of training video frames in the training example, and b t represents an estimate of the amount of data for training video frame t, represents the actual data amount of training video frame t.

[0119] In another example, the system can determine an update to the current model parameters of the rate control model that is further based on an error between (i) an estimated total amount of data required to represent an encoded representation of the training video frame and (ii) an estimated total amount of data required to represent an encoded representation of the training video frame (e.g., determined using a target quantization parameter value included in the training data for the training video frame). The system can determine the error for each training example as,

[0120]

[0121] Among them L totalframe represents the error, t indexes the training video frame, T represents the number of training video frames in the training example, and b t represents an estimate of the amount of data for training video frame t, Indicates the total actual data size of the training video.

[0122] In another example, for each training video frame, the system can determine an update to the current model parameters of the rate control model based on the error between (i) the scores of the possible quantization parameter values ​​generated for the training video frame and (ii) the target quantization parameter value for the training video frame ("QP" error), the "frame" error of equation (3), and the "total frame" error of equation (4). The system can determine the update based on a linear combination of the errors as follows,

[0123] L=L QP +β1L frame +β2L totalframe , (5)

[0124] Where L represents the linear combination, L QP represents the “QP” error, L frame Indicates the "frame" error, L totalframe represents the “total frame” error, β1 represents a weighting factor for the frame error (eg, represented by a positive real number), and β2 represents a weighting factor for the total frame error (eg, represented by a positive real number).

[0125] For one or more of the training examples, a target quantization parameter value for a training video frame of the training example is generated by performing an optimization to determine the quantization parameter value for the training video frame. The optimization can minimize an error between (i) the training video frame and (ii) a reconstruction of the training video frame determined by processing an encoded representation of the training video frame generated using the quantization parameter value. The optimization can be a constrained optimization, e.g., constrained by a constraint that a total amount of data required to represent the encoded representation of the training video frame generated using the quantization parameter value is less than a target amount of data for representing the encoded representation of the training video frame. The system can perform the optimization using any appropriate optimization technique, e.g., a black-box optimization technique, e.g., an optimization technique described with reference to Tim Salimans et al., “Evolution strategies as a scalable alternative to reinforcement learning”. arXiv: 1703.03864, September 7, 2017, which is incorporated by reference herein, or Daniel Golovin et al., “Google Vizier: A Service for Black Box Optimization”, Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, August 13-17, 2017, which is incorporated by reference herein. For example, for each training video, a target function to be optimized can be given by:

[0126] Target Function = PSNR - λ max(0, b - b target ), (6) where PSNR represents a peak signal-to-noise ratio (e.g., in the reconstruction), λ represents a constant (e.g., represented by a positive real number) that introduces a penalty term when the amount of data exceeds the target amount of data, b represents the amount of data to represent the encoded representation of the training video frame generated using the quantization parameter value, and b target represents the target amount of data for representing the encoded representation of the training video frame.

[0127] In practice, performing the optimization to determine the quantization parameter value each time a video is compressed is computationally infeasible. In contrast, using a rate control machine learning model that has been trained to generate the quantization parameter value to mimic the results of the optimization performed to determine an optimal or near-optimal quantization parameter value can be significantly faster and consume less computational resources.

[0128] The training examples used to train the rate control machine learning model using steps (602)-(608) can represent different strategies for encoding the training videos in the training examples, which result in different sequences of quantization parameters with similar performance metrics. Training the rate control machine learning model with training examples generated with different strategies will cause learnability problems of the rate control machine learning model, which will result in the target amount of compressed representation of the video being met only in a small fraction of the time. In order to make the training data more consistent and improve the target amount of data performance, as shown below Figure 7 As described, the system can use the trained rate-controlled machine learning model to generate a second set of training examples.

[0129] Figure 7 is a flow chart of an example process for determining a second set of training examples. For convenience, process 700 is described as being performed by a system of one or more computers located in one or more locations.

[0130] The system can train a rate control model on a first set of training examples (702). For example, as described above with reference to Figure 6 As described, the system can train a rate control model on a first set of training examples.

[0131] For each training example in the second set of training examples, the system determines a corresponding target quantization parameter for each training video frame in the training video (704). The system can determine the corresponding target quantization parameter value by processing the training video frames using the rate control machine learning model. For example, for each training example, Figure 5 The system may process each training video frame in the training examples to select a quantization parameter value for the training video frame. The system may determine a target quantization parameter value for each training video frame in the second set of training examples as the selected quantization parameter value for the training video frame.

[0132] For each training example in the second set of training examples, the system determines a corresponding target amount of data for representing an encoded representation of a training video frame in the training video (706). The system can determine the corresponding target amount for representing the training video as, for example, the sum of the corresponding amounts of data required to represent each video frame in the training video if the video frames were encoded using the target quantization parameter value for the video frames (i.e., selected using the rate control model in step 704).

[0133] The system trains the rate control model on the second set of training examples (708). Figure 6 As described, the system can train a rate control model on the second set of training examples.

[0134] Training the rate control machine learning model using the second set of training examples can enable the rate control machine learning model to learn a more consistent encoding strategy. Learning a more consistent encoding strategy can enable the rate control machine learning model to perform more consistently across a variety of videos by meeting the target data size for the compressed representation of the video a higher percentage of the time.

[0135] This specification uses the term "configured" in conjunction with system and computer program components. A system of one or more computers that is configured to perform specific operations or actions means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when operated, causes the system to perform those operations or actions. One or more computer programs that are configured to perform specific operations or actions means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform those operations or actions.

[0136] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-volatile storage medium for execution by a data processing apparatus or for controlling the operation of the data processing apparatus. A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more of them. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a suitable receiver apparatus for execution by the data processing apparatus.

[0137] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, equipment, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0138] A computer program, which can also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or code portions. A computer program can be deployed for execution on one computer or on multiple computers that are located at one site or distributed across multiple sites and

[0139] In this specification, the term“engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components that are installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0140] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0141] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0142] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0143] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0144] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

[0145] Machine learning models can be implemented and deployed using machine learning frameworks such as TensorFlow, Microsoft Cognitive Toolkit, Apache Singa, or Apache MXNet.

[0146] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer with a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by digital data communication (e.g., a communication network) in any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0147] A computing system may include a client and a server. The client and server are typically remote from each other and typically interact via a communication network. The relationship of client and server arises from computer programs running on their respective computers and having a client-server relationship with each other. In some embodiments, the server sends data, such as an HTML page, to a user device, for example, to display data to a user interacting with the device acting as a client and to receive user input from the user. Data generated at the user device, such as results of user interactions, may be received at the server from the device.

[0148] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or the scope that may be claimed, but rather as descriptions of features that are unique to particular embodiments of particular inventions. Certain features described in this specification in the context of independent embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may in some cases be removed from the combination, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0149] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order described or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0150] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the acts recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order or sequential order illustrated, or that all illustrated operations be performed, to achieve desirable results. In some circumstances, multitasking and parallel processing can be advantageous.

Claims

1. A method, performed by one or more data processing devices, for encoding a video comprising a sequence of video frames to generate a corresponding encoded representation of each video frame, the method comprising, for one or more of the video frames: Obtaining feature embedding of the video frame; processing an input comprising a feature embedding of the video frame using a rate control machine learning model to generate a corresponding score for each of a plurality of possible quantization parameter values; selecting a quantization parameter value from the plurality of possible quantization parameter values ​​using the score; determining a cumulative amount of data required to represent (i) an encoded representation of the video frame generated according to a quantization step size associated with the selected quantization parameter value and (ii) an encoded representation of each video frame preceding the video frame; determining, based on the accumulated data amount, whether a feedback control criterion for the video frame is satisfied; in response to determining that the feedback control criterion is satisfied, updating the selected quantization parameter value; as well as The video frame is processed using a coding model according to a quantization step size associated with the selected quantization parameter value to generate an encoded representation of the video frame.

2. The method according to claim 1, wherein The input processed by the rate control machine learning model also includes a target amount of data for representing the encoded video.

3. The method according to any one of claims 1 to 2, wherein The feedback control criterion for the video frame specifies a target value range, and wherein determining that the feedback control criterion for the video frame is satisfied comprises: Determining that the cumulative amount of data required to represent (i) an encoded representation of the video frame generated according to the quantization step size associated with the selected quantization parameter value and (ii) an encoded representation of each video frame preceding the video frame is outside the target value range specified by the feedback control criterion.

4. The method according to claim 3, wherein: Updating the selected quantization parameter value in response to determining that the feedback control criterion is satisfied includes: An update to the selected quantization parameter value is determined based on a difference between (i) the accumulated data amount and (ii) a nearest endpoint of a target value range specified by a feedback control criterion for the video frame.

5. The method according to claim 3, wherein Updating the selected quantization parameter value in response to determining that the feedback control criterion is satisfied includes: determining that the accumulated data amount is greater than an upper endpoint of a target value range specified by the feedback control criterion; and The selected quantization parameter value is adjusted to increase a quantization step size used to generate an encoded representation of the video frame.

6. The method according to claim 3, wherein: Updating the selected quantization parameter value in response to determining that the feedback control criterion is satisfied includes: determining that the accumulated data amount is less than a lower endpoint of the target value range specified by the feedback control criterion; and The selected quantization parameter value is reduced to reduce a quantization step size used to generate an encoded representation of the video frame.

7. The method according to claim 1 or 2, wherein: Updating the selected quantization parameter value in response to determining that the feedback control criterion is satisfied includes: identifying a set of candidate quantization parameter values, wherein the set of candidate quantization parameter values ​​is a proper subset of the plurality of possible quantization parameter values ​​having the highest scores; and An updated quantization parameter value is selected from the set of candidate quantization parameter values.

8. The method according to claim 1 or 2, wherein: Selecting a quantization parameter value from the plurality of quantization parameter values ​​using the score includes: identifying a proper subset of the plurality of possible quantization parameter values ​​having a highest score; determining a probability distribution over the proper subset of the plurality of possible quantization parameter values; and The quantization parameter values ​​are sampled according to the probability distribution over the proper subset of the plurality of possible quantization parameter values.

9. The method according to claim 8, wherein Identifying the proper subset of the plurality of possible quantization parameter values ​​having the highest score comprises: A predetermined number of possible quantization parameter values ​​having highest scores among the plurality of possible quantization parameter values ​​are identified.

10. The method according to claim 1 or 2, wherein: The rate control machine learning model has a neural network architecture including one or more recurrent neural network layers.

11. The method according to claim 10, wherein: The recurrent neural network layer includes one or more long short-term memory (LSTM) recurrent neural network layers.

12. The method according to claim 1 or 2, wherein: For each video frame, the input processed by the rate control machine learning model also includes the quantization parameter value used to generate the encoded representation of the previous video frame.

13. The method according to claim 1 or 2, further comprising: Obtaining the corresponding initial feature embedding of each video frame based on the features of the video frame; processing the initial feature embeddings of the video frames using an embedding neural network comprising one or more self-attention neural network layers to generate updated feature embeddings for each video frame; and A corresponding feature embedding for each video frame is determined based at least in part on the updated feature embeddings for the video frames generated by the embedding neural network.

14. The method according to claim 13, wherein The embedded neural network includes a transformer neural network.

15. The method according to claim 13, wherein For each video frame, the initial feature embedding for each video frame is based at least in part on noise energy characteristics of the video frame, motion vector statistics of the video frame, or both.

16. The method according to claim 1 or 2, wherein: For each video frame, the feature embedding for the video frame is based at least in part on one or more of: a frame index of the video frame; a frame type of the video frame; the amount of data required to represent an encoded representation of the previous video frame in the video; and Data representing a difference between (i) a cumulative amount of data required to represent an encoded representation of each video frame preceding the video frame in the video and (ii) a target amount of data for representing the encoded video.

17. A video encoding system comprising: one or more computers; as well as One or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1-16.

18. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1-16.

Images