Reinforcement learning based on rate control

Abstract

To provide a computing device, method and storage medium for rate control based on reinforcement learning.SOLUTION: The method comprises: determining an encoding state of a video encoder, the encoding state being associated with encoding of a first video unit by the video encoder; determining, by a reinforcement learning model and based on the encoding state of the video encoder, an encoding parameter associated with rate control for the video encoder; and encoding a second video unit different from the first video unit based on the encoding parameter. In this way, it is possible to achieve better quality of experience (QOE) for real-time communication with computation overhead being reduced.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001]

[0001] In real-time communication (RTC), a common requirement is screen sharing with various users. For example, a participant may need to present his or her desktop screen to other participants in a multi-user video conference. In this situation, the technical goal is to provide a better quality of experience (QOE), which is often determined by various factors such as visual quality, drop rate, transmission delay, etc. Rate control plays an important role in achieving that goal by determining the coding parameters of a video encoder to achieve a target bitrate.

[0002]

[0002] Existing rate control methods are primarily designed for videos with natural scenes. However, unlike natural videos, which mostly involve smooth content movement, screen content is usually combined with complex sudden changes and static scenes. Due to this unique motion characteristic, existing rate control methods designed for natural videos cannot work well for screen content. [Background technology]

[0003]

[0003] Implementations of the subject matter described herein provide a solution for rate control based on reinforcement learning. In this solution, an encoding state of a video encoder is determined, the encoding state being associated with encoding of a first video unit by the video encoder. Encoding parameters for rate control of the video encoder are determined by a reinforcement learning model based on the encoding state of the video encoder. A second video unit different from the first video unit is encoded based on the encoding parameters. The reinforcement learning model is configured to receive the encoding states of one or more video units and determine encoding parameters to use with another video unit. The encoding states have limited state dimensions, enabling better QOE for real-time communication with reduced computational overhead.

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. [Brief explanation of the drawings]

[0005]

[0005] The above and other objects, as well as the features and advantages of the subject matter described herein, will become more apparent through a more detailed description of some implementations of the subject matter illustrated in the accompanying drawings. [Figure 1]

[0006] FIG. 1 illustrates a block diagram of a computing device capable of practicing various implementations of the subject matter described herein. [Figure 2]

[0007] FIG. 2 illustrates a block diagram of a reinforcement learning module according to an implementation of the subject matter described herein. [Figure 3]

[0008] FIG. 3 illustrates an example agent for use in a reinforcement learning module according to an implementation of the subject matter described herein. [Figure 4]

[0009] FIG. 4 illustrates a flowchart of a reinforcement learning-based rate control method according to an implementation of the subject matter described herein.

[0010] Throughout the drawings, the same or similar reference numbers represent the same or similar elements. DETAILED DESCRIPTION OF THE INVENTION

[0006]

[0011] The subject matter described herein will be described below with reference to several example implementations. It should be understood that these implementations are discussed solely for the purpose of enabling those skilled in the art to better understand and practice the subject matter described herein, and are not intended to imply any limitation on the scope of the subject matter.

[0007]

[0012] As used herein, the term "including" and variations thereof should be read as open terms meaning "including, but not limited to." The term "based on" should be read as "based at least in part on." The terms "an implementation" and "implementation" should be read as "at least one implementation." The term "another implementation" should be read as "at least one other implementation." The terms "first," "second," etc. may refer to different or the same object. Other definitions, whether explicit or implicit, may be included below.

[0008]

[0013] 1 illustrates a block diagram of a computing device 100 upon which various implementations of the subject matter described herein may be practiced. It will be understood that the computing device 100 illustrated in FIG. 1 is merely illustrative, without suggesting in any way any limitation on the functionality or scope of an implementation of the subject matter described herein. As illustrated in FIG. 1, the computing device 100 includes a general-purpose computing device 100. Components of the computing device 100 may include, but are not limited to, one or more processors or processing units 110, memory 120, storage devices 130, one or more communication units 140, one or more input devices 150, and one or more output devices 160.

[0009]

[0014] In some implementations, computing device 100 may be implemented as any user terminal or server terminal having computing capabilities. A server terminal may be a server, a larger computing device, or the like, provided by a service provider. A user terminal may be any type of mobile, fixed, or portable terminal, including, for example, a mobile phone, a station, a unit, a device, a multimedia computer, a multimedia tablet, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communications system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / video camera, a positioning device, a television receiver, a wireless broadcast receiver, an e-book device, a gaming device, or any combination thereof, including accessories and peripherals for these devices. It is contemplated that computing device 100 may be capable of supporting any type of interface to a user (e.g., "wearable" circuitry, etc.).

[0010]

[0015] The processing unit 110 may be a physical or virtual processor and may execute various processes based on programs stored in the memory 120. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel, increasing the parallel processing capabilities of the computing device 100. The processing unit 110 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0011]

[0016] Computing device 100 typically includes a variety of computer storage media. Such media may be any media accessible by computing device 100, including, but not limited to, volatile and nonvolatile media, or removable and non-removable media. Memory 120 may be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), or flash memory), or any combination thereof. Storage device 130 may be any removable or non-removable medium, including machine-readable media such as memory, flash memory drives, magnetic disks, or other media, that can be used to store information and / or data and that can be accessed within computing device 100.

[0012]

[0017] Computing device 100 may also include additional removable / non-removable, volatile / non-volatile memory media. Although not shown in FIG. 1, a magnetic disk drive for reading from and / or writing to a removable, non-volatile magnetic disk and an optical disk drive for reading from and / or writing to a removable, non-volatile optical disk may be provided. In such cases, each drive may be connected to a bus (not shown) via one or more data media interfaces.

[0013]

[0018] Communications unit 140 communicates with other computing devices over a communications medium. Furthermore, the functionality of the components within computing device 100 may be implemented by a single computing cluster or multiple computing machines that can communicate over a communications connection. Thus, computing device 100 may operate in a networked environment using logical connections with one or more other servers, networked personal computers (PCs), or more general network nodes.

[0014]

[0019] Input device 150 may be one or more of a variety of input devices, such as a mouse, keyboard, tracking ball, or voice input device. Output device 160 may be one or more of a variety of output devices, such as a display, loudspeaker, or printer. Communications unit 140 enables computing device 100 to further communicate with one or more external devices (not shown), such as storage and display devices, one or more devices that allow a user to interact with computing device 100, or any device (e.g., a network card, modem, etc.) that allows computing device 100 to communicate with one or more other computing devices, as needed. Such communication may occur via an input / output (I / O) interface (not shown).

[0015]

[0020] In some implementations, instead of being integrated into a single device, some or all of the components of computing device 100 may be located in a cloud computing architecture. In a cloud computing architecture, components may be provided remotely and may cooperate to achieve the functionality described in the subject matter described herein. In some implementations, cloud computing provides computing, software, data access, and storage services without requiring end users to be aware of the physical location or configuration of the systems or hardware providing these services. In various implementations, cloud computing provides services over a wide area network (e.g., the Internet) using appropriate protocols. For example, a cloud computing provider may offer applications over a wide area network that can be accessed through a web browser or any other computing component. Software or components of the cloud computing architecture and corresponding data may be stored on remote servers. Computing resources in a cloud computing environment may be collocated or distributed across locations in remote data centers. A cloud computing infrastructure may act as a single access point for users but provide services through a shared data center. Thus, a cloud computing architecture may be used to provide the components and functionality described herein from a remote service provider, or they may be provided from traditional servers or installed directly or otherwise on client devices.

[0016]

[0021] Computing device 100 may be used to implement reinforcement learning-based rate control in implementing the subject matter described herein. Memory 120 may include one or more reinforcement learning modules 122 having one or more program instructions. These modules are accessible and executable by processing unit 110 to perform the functions of various implementations described herein. For example, input device 150 may provide a video or series of frames of computing device 100's environment to reinforcement learning module 122 to enable a video conferencing application, while processing unit 110 and / or memory 120 may provide at least a portion of screen content to reinforcement learning module 122 to enable a screen content sharing application. Multimedia content may be encoded by reinforcement learning module 122 to achieve rate control with good QOE.

[0017]

[0022] Referring now to FIG. 2 , a block diagram of a reinforcement learning module 200 according to implementations described herein is shown. The reinforcement learning module 200 may be implemented within the computing device 100, for example, as the reinforcement learning module 122. The reinforcement learning module 200 includes an encoder 204 configured to encode multimedia content from other components of the computing device 100, such as the processing unit 110, the memory 120, the storage 130, the input device 150, and / or the like. For example, the input device 150 may provide one or more frames of video to the reinforcement learning module 200, while the processing unit 110 and / or the memory 120 may provide at least a portion of screen content to the reinforcement learning module 200. For example, the encoder 204 may be a video encoder, particularly a video encoder optimized for screen content from the computing device 100.

[0018]

[0023] Coding parameters related to rate control, such as the quantization parameter (QP) or lambda, control the granularity of compression of video units, e.g., frames, blocks, or macroblocks within a frame. A larger value means that there will be higher quantization, more compression, and lower quality. A lower value means the opposite. Therefore, by implementing rate control to adjust the coding parameters of the encoder, e.g., the quantization parameter or lambda, it is possible to achieve a good QOE. While reference is made herein to the quantization parameter or lambda, it should be noted that the quantization parameter or lambda is provided for illustrative purposes, and any other suitable coding parameter associated with rate control may be adjusted or controlled.

[0019]

[0024] 2, reinforcement learning module 200 can include an agent 202 configured to make decisions that control encoding parameters of encoder 204. In some implementations, agent 202 may employ a reinforcement learning model implemented by a neural network, e.g., a recurrent neural network.

[0020]

[0025] The encoded bitstream is then output to a transmit buffer. The encoder 204 may also include such a transmit buffer (not shown) to implement the bitstream transmission process. After being encoded, the bitstream of the most recently encoded video unit is stored or added to the transmit buffer. During transmission, the bitstream stored in the transmit buffer is transmitted to one or more receivers over one or more channels at a certain bandwidth, and the transmitted bitstream is removed from the buffer upon transmission over that bandwidth. The state of the transmit buffer is in a constant process of changing due to ingress and egress bitstreams entering and leaving the transmit buffer.

[0021]

[0026] At each time step t, the agent 202 determines the encoding state s of the encoder 204. t Observe the coding state s at time step t. t may be determined based on the encoding of at least the video unit at time step t-1. Based on this input information, the agent 202 makes an inference and takes action a t Outputs action a t indicates how finely the encoder 204 should compress the video unit at time step t. t may be an encoding parameter of the encoder 204 for rate control, such as a quantization parameter (QP), or may be mapped to an encoding parameter of the encoder 204. After obtaining the encoding parameters, the encoder 204 can start encoding a video unit (e.g., a screen content frame). The encoding of a video unit at time step t is performed by the encoding state s of the agent 202 at time step t+1. t+1 It should be understood that the reinforcement learning module 200 may be applied to any other suitable multimedia application other than real-time screen content sharing.

[0022]

[0027] By controlling encoding parameters through agent behavior based on the encoder's encoding state rather than traditional manual rules, a reinforcement learning-based solution implementing the subject matter described herein can achieve better visual quality with negligible drop-rate changes. The encoder's encoding state has a limited state space, thus allowing encoding parameter decisions to be made with reduced computational overhead and improved efficiency. In particular, when a sudden scene change occurs in the screen content, a well-trained reinforcement learning model can very quickly update the encoding parameters to achieve better QOE, which is particularly beneficial for screen content sharing in real-time communication. The reinforcement learning-based architecture is not limited to any particular codec and can work with a variety of codecs, such as H.264, HEVC, and AV1.

[0023]

[0028] In some implementations, to help the agent 202 of the reinforcement learning module 200 make accurate and reliable decisions, the encoding state at time step t as input to the agent 202 may include multiple elements to represent the encoding state from various perspectives. For example, a video unit may be a frame, and the encoding state s t may include at least a state representing the outcome of encoding a frame at time step t-1; a state of a transmit buffer at time step t; and a state associated with the network conditions at time step t for transmitting the encoded frame.

[0024]

[0029] For example, the results related to encoding at least the frame at time step t-1 may further include results related to encoding frames prior to time step t-1, such as frames at time step t-2. In one example, the results may include coding parameters (e.g., QP or lambda) of the frame encoded at time step t-1 and the size of the frame encoded at time step t-1. If a frame is dropped, the coding parameters of the frame encoded at time step t-1 may be set to a predetermined value, such as zero. In one example, the frame size at time step t-1 may be expressed by a frame's frame size ratio, which is defined by the ratio of the frame size to an average target frame size. In other words, the frame size at time step t-1 may be normalized by the average target frame size. For example, the frame size may be expressed by the frame's bitstream size, and the average target frame size may represent the average number of target bits in a frame and may be calculated by dividing the target bitrate by the frame rate. The target bit rate represents the target number of bits to be transmitted, and the frame rate represents the frequency or rate at which frames are transmitted. Both the target bit rate and the frame rate can be determined from the video encoder.

[0025]

[0030] In one example, the transmit buffer status may include buffer usage, such as the ratio of occupied space to maximum buffer space; remaining buffer space measured in frames; or a combination thereof. The remaining buffer space measured in frames may be calculated by dividing the remaining buffer space by the average target frame size. This value describes buffer usage from another perspective, taking into account the effect of frame rate.

[0026]

[0031] In one example, the state related to the network conditions includes a target bits per pixel (BPP). This parameter is defined by the number of bits used by a pixel and may be calculated by dividing the target bit rate by the number of pixels in a frame per unit time. The number of pixels in a frame and the target bit rate may be determined, for example, from a video encoder.

[0027]

[0032] In some implementations, the above-mentioned coding states pertain to frames, and the reinforcement learning module 200 makes decisions on a frame-by-frame basis. In other implementations, the reinforcement learning module 200 may be applied or adapted to any other suitable video unit for compression or encoding. For example, the reinforcement learning module may make decisions at the block level, e.g., macroblock (H.264), coding tree unit (HEVC), superblock (AV1), etc. Thus, the coding states s used as input to the agent 202 may be t may include states representing the outcome of encoding at least one block at time step t−1, the state of the transmission buffer at time step t, and states associated with the network conditions at time step t for transmitting the encoded block.

[0028]

[0033] For example, the results of encoding at least one block may include results of encoding one or more neighboring blocks. The neighboring blocks may include blocks spatially to the left, right, above, and / or below the block being processed. The encoding of the spatially neighboring blocks may have been performed at time step t-1 or other previous time steps. The results of encoding the spatially neighboring blocks may be stored in storage, and the results of encoding the spatially neighboring blocks may be retrieved from storage. Additionally or alternatively, the neighboring blocks may include one or more corresponding blocks in a previous frame, also referred to as temporally neighboring blocks. The results of encoding the temporally neighboring blocks may be stored in storage and retrieved therefrom.

[0029]

[0034] In one example, the outcome may include coding parameters, such as QP or lambda, of at least one coded block and the size of at least one coded block. For example, the size of the coded block may be expressed by a block size ratio defined by the ratio of the coded block size to an average target block size. In other words, the block size may be normalized by the average target block size. For example, the block size may be expressed by a bitstream size for coding the block, and the average target block size may represent the average target number of bits in a block and may be calculated by dividing the target bit rate by the number of blocks transmitted per unit time.

[0030]

[0035] In one example, the transmit buffer status may include buffer usage, such as the ratio of occupied space to maximum space in the buffer, the remaining space in the buffer measured in blocks, or a combination thereof, which may be calculated by dividing the remaining space in the buffer by the average target block size.

[0031]

[0036] In one example, the network state-related state includes the target bits per pixel (BPP), which is defined by the number of bits used by a pixel and can be calculated similarly to the frame-related implementation.

[0032]

[0037] The coding states are described in terms of coding parameters such as quantization parameters or lambda. Note that the coding states may also apply to any other suitable coding parameters associated with the rate control used by the encoder.

[0033]

[0038] Referring again to FIG. 2, the action a output by the agent 202 is t can control the encoding quality of the encoder 204. For example, the action a determined by the agent 202 t may be normalized and fall within the range of 0 to 1. In some implementations, the behavior may be mapped to a QP that the encoder can understand. For example, the mapping may be implemented by the following formula:

number

[0039] where QP max and QP min represent the maximum and minimum QP, respectively, and QP currepresents the QP to be used for encoding by the encoder 204. While this mapping function is illustrated as a linear function, it should be understood that any other suitable function can be used instead. A smaller QP value forces the encoder to perform compression in a more delicate manner and obtain higher reconstruction quality. However, the cost is to generate a larger coded bitstream. An excessively large bitstream can easily overshoot the buffer and correspondingly drop frames (e.g., in the case of frame-level rate control). On the other hand, a larger QP value employs coarser coding but results in a smaller coded bitstream.

[0034]

[0040] In some further implementations, the encoding parameters may be implemented as lambdas. t can be mapped to a lambda that the encoder can understand. For example, the mapping may be implemented by the following formula:

[0035]

number

[0041] where lambda max and lambda min represent the maximum and minimum lambda, respectively, and lambda currepresents the lambda to be used by the encoder 204. This mapping function behaves linearly in the logarithmic domain of lambda. In addition to or instead of the above mapping function, any other suitable function may be used for mapping instead. A lower lambda value controls the encoding in a more delicate manner and obtains higher reconstruction quality. However, it may produce a larger encoded bitstream and the buffer may easily be overshooting, while a higher lambda value employs coarser encoding but will produce a smaller encoded bitstream.

[0036]

[0042] Continuing with reference to FIG. 2, in training the reinforcement learning module 200, it is necessary to evaluate how good the actions taken by the agent 202 are. To this end, an encoder 204 generates an action a t After finishing encoding each video unit with t When the agent 202 acquires a certain number of training samples, the agent 202 receives a reward r t The agent 202 can be trained to converge towards a direction that can maximize the accumulated reward. To obtain a better QOE, one or more factors that reflect the QOE can be incorporated into the reward. For example, the reward r t is set to penalize buffer overshoot and to increase if the coding parameters result in higher visual quality. For example, visual quality increases as the quantization parameter, or lambda, decreases.

[0037]

[0043] In one example, the reward r t may be calculated as follows:

[0038]

number

[0044] where a is a constant factor, b is a negative number, and r base is the base reward, and Bandwidth cur represents the bandwidth of the channel transmitting the bit stream at time step t, and Bandwidth max represents the maximum bandwidth, and r final represents the final reward.

[0039]

[0045] Basic remuneration r base is calculated by Equation (3). For example, higher visual quality may result in higher QOE, especially in screen content sharing scenarios. Therefore, to achieve higher visual quality, it is desirable to use a smaller QP or lambda, and as shown in Equation (3), the current quantization parameter QP cur The reward increases as QP decreases. However, a very small QP value may result in a large bitstream size, which can easily lead to buffer overshoot and, consequently, frame dropping for frame-level rate control. Therefore, in the case of buffer overshoot, the reward is set as a negative number (i.e., b). Setting a negative number as the penalty is used to train the agent 202 to avoid buffer overshoot.

[0040]

[0046] r base After calculating the final reward r, for example, as shown in equation (4), final is the scaling r base The scaling factor relates to the ratio of the bandwidth at time step t to the maximum bandwidth. If the bandwidth at time t is large, the reward r tis scaled to a larger value, and the penalty is also larger if buffer overshoot occurs. Under high-bandwidth conditions, the pursuit of better visual quality is more aggressive, while the occurrence of buffer overshoot is more severe. Note that any other suitable function may be used instead to calculate the reward without departing from the spirit of the implementation described herein.

[0041]

[0047] In some implementations, the Proximal Policy Optimization (PPO) algorithm optimizes the reward r t PPO may be employed to train agents 202 based on a policy π. PPO is implemented based on an actor-critic architecture, which includes an actor network of actors and a critic network of critics. The actors act as agents 202. The input to the actor network is a coding state, and the output of the actor network is an action. The actor network is configured to train policies π θ (a t |s t ), where θ represents a policy parameter (e.g., a weight in the actor network), and a t ,s t represent the action and coding state at time step t, respectively. The critic network is set up to evaluate how good the coding state st is and only operates during the training process.

[0042]

[0048] In the PPO algorithm, the policy loss L is calculated as follows: policy may be used to update actors, and the value loss L value may be used to update the critic:

[0043]

number

[0049] Here, the value loss is

[0044]

number

[0045]

number

[0046]

[0050] The encoding states used in the reinforcement learning module 200 enable a lightweight network architecture for the agent 202 and for training the agent 202. For example, a neural network implementing the agent 202 may use the encoding states s tThe neural network may include one or more input fully connected layers configured to extract features from the input. The extracted features may be provided to one or more recurrent neural networks to extract temporal features or correlations from the features. The features may then be correlated with, for example, behavior a t The neural network may be, for example, a gated recurrent unit (GRU) or a long-short term memory (LSTM). Neural networks have a lightweight yet efficient architecture and can meet the requirements of real-time applications, especially those of screen content coding (SCC).

[0047]

[0051] FIG. 3 illustrates an example of a neural network 300 for training an agent 202 according to an implementation of the subject matter described herein. The neural network 300 includes an actor network 302 and a critic network 304. The actor network 302 and the critic network 304 may share a common network module to reduce the parameters to be optimized. In this example, the input passes through two fully connected (FC) layers and is converted into a feature vector. While a leaky rectified linear unit (RELU) is shown in FIG. 3, it should be understood that any other suitable activation function may be used in the network.

[0048]

[0052] Considering that rate control is a time-series problem, two gated recurrent units (GRUs) are introduced to further extract features in combination with historical information. It should be understood that any other suitable recurrent neural network can be used as well. After the GRU, the actor and critic networks each have their own network modules. Both the actor and critic each use an FC layer to reduce the dimensionality of their feature vectors. Finally, both networks use one FC layer to generate their respective outputs, and the actor network uses a sigmoid layer to normalize the behavior range to [0,1]. It should be understood that any other suitable activation function can be used instead of the sigmoid function.

[0049]

[0053] The neural network 300 has a lightweight yet efficient architecture to meet the requirements of real-time applications. For screen content coding (SCC), the reinforcement learning-based solution can achieve better visual quality with negligible drop rate changes compared to traditional rule-based rate control methods. In particular, this method can provide very rapid quality improvement after abrupt scene changes in screen content. The reinforcement learning network-based architecture is not limited to any codec and can work with a variety of different codecs, such as H.264, HEVC, and AV1.

[0050]

[0054] 4 shows a flowchart of a reinforcement learning-based rate control method 400 according to an implementation of the subject matter described herein. Method 400 may be implemented by computing device 100, such as by reinforcement learning module 122 within computing device 100. Method 400 may also be implemented by any other device, cluster of devices, or distributed parallel system similar to computing device 100. For purposes of explanation, method 400 will be described with reference to FIG. 1.

[0051]

[0055] At block 402, the computing device 100 determines an encoding state of a video encoder. The encoding state may be associated with encoding a first video unit by the video encoder. The video encoder may be configured to encode screen content for real-time communication. For example, the video encoder may be the encoder 204 in the reinforcement learning module 200, as shown in FIG. 2. The encoding state associated with encoding the first video unit includes at least a state representing an outcome related to encoding the first video unit; a state of a buffer configured by the video encoder to buffer the encoded video unit before transmission; and a state associated with network conditions for transmitting the encoded video unit. The video unit may include a frame, a block, or a macroblock within a frame. In some implementations, the outcome related to encoding the first video unit includes coding parameters of the encoded first video unit and a size of the encoded first video unit, the buffer state includes buffer usage, and the state associated with network conditions includes a target bits per pixel. In some implementations, the buffer usage includes at least one of: the ratio of occupied space to maximum space in the buffer; and the remaining space in the buffer measured in video units.

[0052]

[0056] At block 404, the computing device 100 determines, using a reinforcement learning model, encoding parameters associated with rate control of the video encoder based on the encoding state of the video encoder. The encoding parameters may be quantization parameters or lambda. In some implementations, the encoding parameters are determined based on behaviors output by an agent based on the encoding state of the video encoder. The agent may include a neural network implementing the reinforcement learning model, and the behaviors output by the agent are mapped to the encoding parameters.

[0053]

[0057] At block 406, the computing device 100 encodes a second video unit different from the first video unit based on the encoding parameters. The first video unit may be a first frame, and the second video unit may be a second frame that follows the first frame. Alternatively, the first video unit may be a first block, and the second video unit may be a nearby second block, such as a spatially nearby block or a temporally nearby block.

[0054]

[0058] In some implementations, the reinforcement learning model is trained by determining a reward for encoding parameters based on the encoding of the second video unit, where the reward is set to penalize buffer overshoot and to increase if the encoding parameters result in higher visual quality.

[0055]

[0059] In some implementations, determining the reward includes: determining the base reward in such a way that if buffer overshoot occurs, the base reward has a negative value, and if buffer overshoot does not occur, the base reward is proportional to the encoding parameter by a negative coefficient; and scaling the base reward by a scaling factor to obtain the reward, the scaling factor being based on a ratio of the bandwidth associated with encoding the second video unit to the maximum bandwidth of the transmission channel. For example, the reward may be calculated based on Equations (3) and (4).

[0056]

[0060] In some implementations, the reinforcement learning model is further trained by determining an action associated with the encoding parameters based on an encoding state of the video encoder; determining an evaluation value of the encoding state for encoding the second video unit; determining a value loss based on the reward and the evaluation value; determining a policy loss based on the action; and updating the reinforcement learning model based on the value loss and the policy loss.

[0057]

[0061] In some implementations, the agent includes a neural network, the neural network including: at least one input fully connected layer configured to extract features from the encoding state; at least one recurrent neural network coupled to receive the extracted features; and at least one output fully connected layer configured to determine a behavior of the agent.

[0058]

[0062] In some implementations, the neural network is trained based on an actor-critic architecture, where the actor is configured to generate an action based on the coding state and the critic is configured to generate an evaluation value regarding the coding state; the actor and critic share a common portion with the neural network, which includes at least one input fully connected layer, and at least one recurrent neural network.

[0059]

[0063] Some exemplary implementations of the subject matter described herein are listed below.

[0060]

[0064] In a first aspect, the presently described subject matter provides a computer-implemented method, the method including: determining an encoding state of a video encoder, the encoding state being associated with encoding a first video unit by the video encoder; determining, with a reinforcement learning model, encoding parameters for rate control of the video encoder based on the encoding state of the video encoder; and encoding a second video unit, different from the first video unit, based on the encoding parameters.

[0061]

[0065] In some implementations, determining the encoding parameters includes: determining an action based on an encoding state of the video encoder using a reinforcement learning model; and mapping the action to the encoding parameters.

[0062]

[0066] In some implementations, the second video unit follows the first video unit, and the encoding state associated with encoding the first video unit includes: a state representing an outcome related to encoding at least the first video unit; a state of a buffer configured to buffer video units encoded by the video encoder before transmission; and a state associated with network conditions for transmitting the encoded video units.

[0063]

[0067] In some implementations, the outcomes related to encoding the first video unit include coding parameters for the encoded first video unit and a size of the encoded first video unit, the buffer state includes buffer usage, and the state associated with network conditions includes a target bits per pixel.

[0064]

[0068] In some implementations, the buffer usage includes at least one of: the ratio of occupied space to maximum space in the buffer; and the remaining space in the buffer measured in video units.

[0065]

[0069] In some implementations, the reinforcement learning model is trained by determining a reward for encoding parameters based on the encoding of the second video unit, the reward being set to penalize buffer overshoot and to increase if the encoding parameters result in higher visual quality.

[0066]

[0070] In some implementations, determining the reward includes: determining the base reward in such a way that if buffer overshoot occurs, the base reward has a negative value, and if buffer overshoot does not occur, the base reward is proportional to the encoding parameter by a negative coefficient; and scaling the base reward by a scaling factor to obtain the reward, the scaling factor being based on a ratio of the bandwidth associated with encoding the second video unit to the maximum bandwidth of the transmission channel.

[0067]

[0071] In some implementations, the reinforcement learning model is further trained by determining an action associated with the encoding parameters based on an encoding state of the video encoder; determining an evaluation value of the encoding state for encoding the second video unit; determining a value loss based on the reward and the evaluation value; determining a policy loss based on the action; and updating the reinforcement learning model based on the value loss and the policy loss.

[0068]

[0072] In some implementations, the reinforcement learning model includes a neural network for the agent, the neural network including: at least one input fully connected layer configured to extract features from the encoding state; at least one recurrent neural network coupled to receive the extracted features; and at least one output fully connected layer configured to determine a behavior of the agent.

[0069]

[0073] In some implementations, the neural network is trained based on an actor-critic architecture, where the actor is configured to generate an action based on the coding state and the critic is configured to generate an evaluation value regarding the coding state; and the actor and critic share a common portion: a neural network including at least one input fully connected layer and at least one recurrent neural network.

[0074] In some implementations, the coding parameters include at least one of a quantization parameter and a lambda parameter.

[0070]

[0075] In some implementations, the video encoder is configured to encode the screen content for real-time communication.

[0071]

[0076] In a second aspect, the presently described subject matter provides an electronic device comprising a processing unit and a memory coupled to the processing unit and storing instructions that, when executed by the processing unit, cause the electronic device to perform any step of the method described above.

[0072]

[0077] In a third aspect, the presently described subject matter provides a computer program product tangibly stored on a computer storage medium and including machine-executable instructions that, when executed by a device, cause the device to perform a method according to the first aspect. The computer storage medium may be a non-transitory computer storage medium.

[0073]

[0078] In a fourth aspect, the presently described subject matter provides a non-transitory computer storage medium having stored thereon machine-executable instructions that, when executed by a device, cause the device to perform a method according to an aspect of the first aspect.

[0074]

[0079] The functionality described herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0075]

[0080] Program code for implementing the methods of the subject matter described herein may be written in any combination of one or more programming languages. The program code may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that, when executed by the processor or controller, the program code causes the functions / acts shown in the flowcharts and / or block diagrams to be performed. The program code may be executed entirely or partially on the machine, or may be executed partially on the machine and partially on a remote machine, or entirely on a remote machine or server as a stand-alone software package.

[0076]

[0081] In the context of the subject matter described herein, a machine-readable medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium includes, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include an electrical connection including one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0077]

[0082] Furthermore, although acts are described in a particular order, this should not be understood as requiring such acts to be performed in the particular order or sequential order shown, or that all of the acts shown be performed to achieve desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above description, these should not be construed as limitations on the scope of the subject matter described herein, but rather as descriptions of features that may be specific to particular implementations. Certain features that are described in the context of separate implementations may also be implemented in combination within a single implementation. Rather, various features described in a single implementation may also be implemented in multiple implementations separately or in any suitable subcombination.

[0078]

[0083] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter specified in the appended claims is not necessarily limited to the specific features and acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

[0079] (Appendix 1) 1. A computer-implemented method comprising: determining an encoding state of a video encoder, the encoding state associated with encoding a first video unit by the video encoder; determining, by a reinforcement learning model, encoding parameters for rate control of the video encoder based on the encoding state of the video encoder; and encoding a second video unit different from the first video unit based on the encoding parameters; A method comprising: (Appendix 2) 10. The method of claim 1, wherein the step of determining the encoding parameters comprises: determining, by the reinforcement learning model, an action based on the encoding state of the video encoder; and mapping said behaviors to said coding parameters; A method comprising: (Appendix 3) 10. The method of claim 1, wherein the coding state associated with encoding the first video unit is: a state representing an outcome regarding encoding at least the first video unit; a buffer state configured to buffer video units encoded by the video encoder prior to transmission; and a state associated with the network conditions for transmitting said coded video units; A method comprising: (Appendix 4) 4. The method of claim 3, wherein the outcomes related to encoding at least the first video unit include the coding parameters of the encoded first video unit and a size of the encoded first video unit, the buffer state includes a usage of the buffer, and the state associated with the network conditions includes a target bits per pixel. (Appendix 5) In the method according to Appendix 4, the buffer is used in the following manner: the ratio of the occupied space to the maximum space of said buffer; and the remaining space in the buffer measured in video units; The method includes at least one of: (Appendix 6) 2. The method of claim 1, wherein the reinforcement learning model is trained by determining a reward for the encoding parameters based on the encoding of the second video unit, the reward being configured to penalize buffer overshoot and to increase if the encoding parameters result in higher visual quality. (Appendix 7) 7. The method of claim 6, wherein the step of determining the reward comprises: determining a base reward in such a way that if a buffer overshoot occurs, the base reward has a negative value, and if the buffer overshoot does not occur, the base reward is proportional to the encoding parameter by a negative coefficient; and scaling the base reward by a scaling factor to obtain the reward, the scaling factor being based on a ratio of a bandwidth associated with encoding the second video unit to a maximum bandwidth of a transmission channel; A method comprising: (Appendix 8) 7. The method according to claim 6, wherein the reinforcement learning model comprises: determining an action associated with the encoding parameters based on the encoding state of the video encoder; determining an evaluation value of the encoding state for encoding the second video unit; determining a value loss based on the compensation and the assessed value; determining a policy loss based on the behavior; and updating the reinforcement learning model based on the value loss and the policy loss; Further training is provided by the method. (Appendix 9) 10. The method of claim 1, wherein the reinforcement learning model includes a neural network of an agent, the neural network comprising: at least one input fully connected layer configured to extract features from the encoding state; at least one recurrent neural network coupled to receive the extracted features; and at least one output fully connected layer configured to determine a behavior of the agent; A method comprising: (Appendix 10) 10. The method of claim 9, wherein the neural network is trained based on an actor-critic architecture, the actor being configured to generate the behavior based on the encoding state, and the critic being configured to generate an evaluation value related to the encoding state; and The method, wherein the actor and the critic share a common portion with the neural network including the at least one input fully connected layer and the at least one recurrent neural network. (Appendix 11) 2. The method of claim 1, wherein the coding parameters include at least one of a quantization parameter and a lambda parameter. (Appendix 12) 2. The method of claim 1, wherein the video encoder is configured to encode screen content in real-time communication. (Appendix 13) processor; and memory storing instructions for execution by said processor; wherein the instructions, when executed by the processor, cause the device to perform operations, the operations being: determining an encoding state of a video encoder, the encoding state associated with encoding a first video unit by the video encoder; determining, by a reinforcement learning model, encoding parameters for rate control of the video encoder based on the encoding state of the video encoder; and encoding a second video unit different from the first video unit based on the encoding parameters; Including, the device. (Appendix 14) 14. The device of claim 13, wherein the encoding state associated with encoding the first video unit is: a state representing an outcome regarding encoding at least the first video unit; a buffer state configured to buffer video units encoded by the video encoder prior to transmission; and a state associated with the network conditions for transmitting said coded video units; Including, the device. (Appendix 15) 15. The device of claim 14, wherein the outcomes related to encoding at least the first video unit include the coding parameters of the encoded first video unit and a size of the encoded first video unit, the buffer state includes a usage of the buffer, and the state associated with the network conditions includes a target bits per pixel. (Appendix 16) 14. The device of claim 13, wherein the reinforcement learning model is trained by determining a reward for the encoding parameters based on encoding of the second video unit, the reward being configured to penalize buffer overshoot and to increase if the encoding parameters result in higher visual quality. (Appendix 17) 17. The device of claim 16, wherein determining the reward comprises: determining a base reward in such a way that if a buffer overshoot occurs, the base reward has a negative value, and if the buffer overshoot does not occur, the base reward is proportional to the encoding parameter by a negative coefficient; and scaling the base reward by a scaling factor to obtain the reward, the scaling factor being based on a ratio of a bandwidth associated with encoding the second video unit to a maximum bandwidth of a transmission channel; Including, the device. (Appendix 18) 16. The device of claim 15, wherein the reinforcement learning model includes a neural network of an agent, the neural network: at least one input fully connected layer configured to extract features from the encoding state; at least one recurrent neural network coupled to receive the extracted features; and at least one output fully connected layer configured to determine a behavior of the agent; Including, the device. (Appendix 19) 14. The device of claim 13, wherein the video encoder is configured to encode screen content in real-time communication. (Appendix 20) A computer program incorporating program instructions, the program instructions being executable by a processor to cause the processor to perform operations, the operations being: determining an encoding state of a video encoder, the encoding state associated with encoding a first video unit by the video encoder; determining, by a reinforcement learning model, encoding parameters for rate control of the video encoder based on the encoding state of the video encoder; and encoding a second video unit different from the first video unit based on the encoding parameters; computer programs, including

Claims

1. A computing device having one or more processing units and a memory configured to perform the following operations: determining an encoding state of a video encoder, the encoding state associated with encoding a first video unit by the video encoder; determining, by a reinforcement learning model, encoding parameters for rate control of the video encoder based on the encoding state of the video encoder; and encoding a second video unit different from the first video unit based on the encoding parameters; and training the reinforcement learning model based on encoding of the second video unit according to a reward for the encoding parameters, the reward being configured to penalize buffer overshoot and to increase if the encoding parameters result in higher visual quality; wherein the reward is based on a base reward that has a negative value when buffer overshoot occurs and that increases as the encoding parameter decreases when the buffer overshoot does not occur, and wherein the reward is obtained by scaling the base reward by a scaling factor.

2. 10. The computing device of claim 1, wherein the step of determining the encoding parameters comprises: determining an action based on the encoding state of the video encoder using the reinforcement learning model; and mapping the behavior to the coding parameters; a computing device,

3. 10. The computing device of claim 1, wherein the encoding state associated with encoding the first video unit is: a state representing an outcome related to encoding at least the first video unit; a state of a buffer configured to buffer video units encoded by the video encoder before transmission; and a state associated with a network condition for transmitting the encoded video unit; 1. A computing device comprising one or more of:

4. 4. The computing device of claim 3, wherein the outcomes related to encoding at least the first video unit include the encoding parameters from encoding the first video unit and a size of the encoded first video unit, the buffer state includes usage of the buffer, and the state associated with the network condition includes a target bits per pixel.

5. 5. The computing device of claim 4, wherein the buffer is used in the following manner: the ratio of the occupied space to the maximum space of the buffer; and the remaining space in the buffer measured in video units; 10. A computing device comprising:

6. 10. The computing device of claim 1, wherein the scaling factor is based on a ratio of a bandwidth associated with encoding the second video unit to a maximum bandwidth of a transmission channel.

7. 2. The computing device of claim 1, wherein the reinforcement learning model comprises: determining an action associated with the encoding parameter based on the encoding state of the video encoder; determining an evaluation value of an encoding state for encoding the second video unit; determining a value loss based on the reward and the assessed value; determining a policy loss based on the behavior; and updating the reinforcement learning model based on the value loss and the policy loss; A computing device that is trained by

8. 10. The computing device of claim 1, wherein the reinforcement learning model includes a neural network of agents, the neural network comprising: at least one input fully connected layer configured to extract features from the encoding state; at least one recurrent neural network coupled to receive the extracted features; and at least one output fully connected layer configured to determine a behavior of the agent; a computing device,

9. 9. The computing device of claim 8, wherein the neural network is trained based on an actor-critic architecture, the actor configured to generate the behavior based on the coding state, and the critic configured to generate an evaluation value related to the coding state; and A computing device, wherein the actor and the critic share a common portion with the neural network including the at least one input fully connected layer and the at least one recurrent neural network.

10. 10. The computing device of claim 1, wherein the video encoder is configured to encode screen content for real-time communication.

11. One or more non-transitory, machine-readable storage media that store encoded data resulting from execution of machine-executable instructions by a computing device implementing a video encoder to perform an encoding process, the encoding process comprising: generating a first portion of an encoded bitstream by encoding a first video unit with the video encoder; determining an encoding state of the video encoder, the encoding state associated with encoding the first video unit by the video encoder; determining encoding parameters for rate control of the video encoder based on the encoding state of the video encoder using a reinforcement learning model, the reinforcement learning model being trained based on a reward for the encoding parameters, the reward being set to penalize buffer overshoot and to increase if the encoding parameters result in higher visual quality, the reward being based on a base reward that has a negative value if buffer overshoot occurs and that increases as the encoding parameters decrease if the buffer overshoot does not occur, and the reward is obtained by scaling the base reward by a scaling factor; and generating a second portion of the coded bitstream by coding a second video unit different from the first video unit based on the coding parameters; a storage medium.

12. 12. The one or more non-transitory, device-readable storage media of claim 11, wherein the encoding state associated with encoding the first video unit comprises: a state representing an outcome related to encoding at least the first video unit; a state of a buffer configured to buffer video units encoded by the video encoder before transmission; and a state associated with a network condition for transmitting the encoded video unit; A storage medium including one or more of:

13. 13. The one or more non-transitory, device-readable storage media of claim 12, wherein the outcomes related to encoding at least the first video unit include the encoding parameters from encoding the first video unit and a size of the encoded first video unit, the buffer state includes buffer usage, and the state associated with the network conditions includes a target bits per pixel.

14. 12. The one or more non-transitory, device-readable storage media of claim 11, wherein the coding parameters include a quantization parameter.

15. 12. The one or more non-transitory, device-readable storage media of claim 11, wherein the encoding parameters include a lambda parameter.

16. 12. The one or more non-transitory, machine-readable storage media of claim 11, wherein the reinforcement learning model includes a neural network of agents, the neural network comprising: at least one input fully connected layer configured to extract features from the encoding state; at least one recurrent neural network coupled to receive the extracted features; and at least one output fully connected layer configured to determine a behavior of the agent; a storage medium.

17. 12. The one or more non-transitory, device-readable storage media of claim 11, wherein the video encoder is configured to encode screen content of a real-time communication.

18. 1. A method comprising: receiving, in a computing device, encoded data resulting from execution of device-executable instructions by a computing device implementing a video encoder to perform an encoding process; and storing the encoded data, the encoding process comprising: generating a first portion of an encoded bitstream by encoding a first video unit with the video encoder; determining an encoding state of the video encoder, the encoding state associated with encoding the first video unit by the video encoder; determining encoding parameters for rate control of the video encoder based on the encoding state of the video encoder using a reinforcement learning model, the reinforcement learning model being trained based on a reward for the encoding parameters, the reward being set to penalize buffer overshoot and to increase if the encoding parameters result in higher visual quality, the reward being based on a base reward that has a negative value if buffer overshoot occurs and that increases as the encoding parameters decrease if the buffer overshoot does not occur, and the reward is obtained by scaling the base reward by a scaling factor; and generating a second portion of the coded bitstream by coding a second video unit different from the first video unit based on the coding parameters; A method comprising:

19. 20. The method of claim 18, wherein the encoding state associated with encoding the first video unit is: a state representing an outcome related to encoding at least the first video unit; a state of a buffer configured to buffer video units encoded by the video encoder before transmission; and a state associated with a network condition for transmitting the encoded video unit; The method includes one or more of the following:

20. 20. The method of claim 18, wherein the reinforcement learning model includes a neural network of agents, the neural network comprising: at least one input fully connected layer configured to extract features from the encoding state; at least one recurrent neural network coupled to receive the extracted features; and at least one output fully connected layer configured to determine a behavior of the agent; A method comprising:

Images