A frame-level code rate adaptive control method based on deep reinforcement learning

Abstract

The application provides a frame-level code rate self-adaptive control method based on deep reinforcement learning, and the steps are as follows: (1) an original video is generated by a content providing end, and the original video is divided into decision units; (2) for each decision unit, a neural network is used to give a quantization parameter value of video coding according to the current decision; then each decision unit is encoded and packaged for transmission; (3) a video sending end sends the video stream of the decision unit in sequence in units of frames, and a receiving end maintains a receiving buffer and a playing buffer locally; (4) the quality of experience is calculated according to the state information fed back, the parameters of the neural network are updated, and the decision of the next quantization parameter is output; (5) the steps (1) to (4) are repeated until the optimal decision strategy is converged. The method directly decides the quantization parameter required for coding, avoids the inaccuracy of the decision based on the code rate, and reduces the abuse of the network bandwidth.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field, and particularly relates to a frame-level code rate self-adaptive control method based on deep reinforcement learning. BACKGROUND

[0002] Video content accounts for a large proportion of the traffic on the Internet today. Current service providers mainly use adaptive streaming transmission technology to transmit video, that is, according to certain algorithms and rules, the video code rate is dynamically switched to improve the Quality of Experience (QoE) of users. With the change of people's lifestyle, the Real Time Video Communications (RTVC) business is developing rapidly, its content includes two categories of natural video shot by ordinary cameras and screen video generated by intelligent electronic devices, and has the characteristics of high content complexity, various types and rapid scene switching. In addition, unlike video on demand, real-time video communication has extremely strict requirements for delay, which makes the adaptive bitrate algorithm (ABR) unable to maintain a large playback buffer at the playback end, and the resistance to network fluctuations is greatly reduced.

[0003] Most of the traditional adaptive bitrate algorithms use fixed control rules to make bitrate decisions, and their different points mainly lie in the reference indicators, which can be generally divided into the following three categories: (1) prediction based on network throughput; (2) prediction based on playback buffer area; (3) combination of the above two methods. These algorithms are heuristic, and need a lot of adjustment when deployed in specific, and have poor generalization. In recent years, the adaptive bitrate algorithm based on reinforcement learning has made a breakthrough, and a large number of excellent works such as Pensieve, Oboe, ARS (Adaptive Real-time Streaming) have emerged. The main idea is to collect some current information as the input of the neural network, which usually includes the encoding parameters and encoding results of the previous encoding unit, the current network condition, the buffer occupancy, etc. For each encoding unit, the neural network selects one in the action space as the bitrate decision after calculation, and the encoder encodes the encoding unit according to the encoding decision and finally transmits the video to the client. The client feeds back the reward or punishment to the reinforcement learning agent according to the QoE signal adopted, and the agent updates the neural network parameters to maximize the cumulative reward. Among them, Pensieve and Oboe are aimed at the on-demand scenario, while ARS is aimed at the real-time video communication scenario.

[0004] However, the existing rate adaptation algorithms have some shortcomings, mainly two aspects, that is, two challenges faced by guaranteeing user QoE: (1) Play stuttering. In the real world, the available network bandwidth of the user can dynamically change in milliseconds, and the rate adaptation algorithm must adjust the decision in time to adapt to the change of network bandwidth, otherwise it will be difficult to recover the play due to the high packet loss rate. And, due to the characteristics of high content complexity, diverse types, rapid scene switching and other characteristics of video content in the RTVC scene, the robustness of the traditional encoding rate control algorithm is poor, so it is easy to produce large fluctuations in the encoding rate. In view of the above two points, in order to avoid play stuttering, the rate adaptation algorithm must simultaneously resist network bandwidth fluctuations and encoding rate fluctuations. (2) Quality fluctuations. In addition to play stuttering, frequent quality fluctuations will cause the "breathing effect" of the picture, that is, the cycle process of gradually changing from good to bad and then gradually changing from bad to good with the switching of video I frames and P frames, which will make the user feel uncomfortable. SUMMARY

[0005] In view of the defects existing in the prior art, the purpose of the present application is to provide a frame-level rate adaptation control method based on deep reinforcement learning to further improve the user experience quality.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0007] A frame-level rate adaptation control method based on deep reinforcement learning, comprising the following steps:

[0008] (1) The content provider generates an original video, and then divides the original video into decision units by using the operation of video slicing;

[0009] (2) For each decision unit, a neural network is used to give the quantization parameter QP value of video encoding according to the current decision; then each decision unit is encoded and packaged for transmission;

[0010] (3) The video sending end sends the video stream of the decision unit in units of frames in turn, and the receiving end maintains a receiving buffer and a playing buffer locally; the frames that are not received completely are temporarily stored in the receiving buffer, and the continuous frames that are received completely and can be decoded are transferred to the playing buffer for playing in turn; the current real-time packet loss rate and the average frame transmission delay are calculated according to the frames that are not received completely in the receiving buffer, and the number of stuttering frames is calculated according to the number of remaining frames in the playing buffer, so as to reflect the fluctuations of network throughput and the fluctuations of encoding stream size;

[0011] (4) According to the state information fed back to the agent during the transmission process, the user experience quality is calculated, the parameters of the neural network are updated, and the decision of the next quantization parameter QP is outputted;

[0012] (5) Repeat steps (1) to (4) until the optimal decision strategy is converged.

[0013] Further, in step (2), the input of the neural network includes five state parameters: the quantization parameter (QP) value of the previous decision unit, the I frame flag, the packet loss rate, the average frame transmission delay, and the number of frozen frames, wherein the I frame flag is used to indicate whether the video I frame is contained in the next decision unit.

[0014] Further, in step (2), the video frame level decision granularity is adopted.

[0015] Compared with the prior art, the method of the present application avoids the inaccuracy of the code rate-based decision by incorporating the encoding code rate control into the target code rate decision problem, changes the code rate decision to directly decide the required quantization parameter (QP) of encoding, reduces the abuse of network bandwidth, and improves the ability to effectively utilize the bandwidth. In addition, in terms of code rate decision granularity, the present application adopts a smaller granularity decision cycle, which can match the network changes in the time dimension and timely respond to network fluctuations while avoiding the difficulty of neural network convergence, thereby improving the response ability to network conditions. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is the adaptive streaming media transmission system architecture in the control method of the present application;

[0017] Figure 2 is the video VMAF score fluctuation using different mode encoding, (a) using fixed bit rate mode encoding, (b) using QP value encoding;

[0018] Figure 3 is the neural network structure diagram in the embodiment of the present application.

[0019] Figure 4 is the average QoE distribution comparison diagram of the present application and ARS on the test set.

[0020] Figure 5 is the average QoE improvement percentage of the present application compared with ARS.

[0021] Figure 6 is the result comparison diagram of the present application and ARS on the QoE evaluation index. DETAILED DESCRIPTION

[0022] The present application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] The overall system framework of the control method of the present application is as follows Figure 1As shown in the figure, the solid line represents the forward propagation process of the video content from generation, encoding, packet transmission, and finally reaching the playback end, and the dashed line represents the backhaul process of the state information collected from each link of the forward propagation process by the rate adaptation control method. The specific workflow is as follows:

[0024] (1) First, the content provider generates the original video data, and then divides the original video into decision units using video slicing and other operations.

[0025] (2) For each decision unit, the present application uses a neural network to give the QP value of video encoding according to the current decision. Then each decision unit is encoded and packetized for transmission.

[0026] (3) The video sending end sends the video stream in units of frames in turn, and the receiving end maintains a receiving buffer and a playback buffer locally. Incomplete frames are temporarily stored in the receiving buffer, and complete and decodable consecutive frames are transferred to the playback buffer for playback in turn. The real-time packet loss rate and average frame transmission delay are calculated according to the incomplete frames in the receiving buffer, and the number of frames that are stuck is calculated according to the remaining number of frames in the playback buffer, so as to reflect the fluctuations of network throughput and the fluctuations of encoding stream size.

[0027] (4) According to the state information fed back to the agent by each link in the above transmission process, the user QoE is calculated, the neural network parameters are updated, and the next QP decision is output.

[0028] (5) Repeat steps (1) to (4) above until the optimal decision strategy is converged.

[0029] The rate adaptation control method of the present embodiment uses the classic deep reinforcement learning A3C algorithm as an example for training and reasoning, and uses multi-agent parallel training, but also supports using other reinforcement learning algorithms such as DQN, DDPG, PPO, etc.

[0030] 1. Action set

[0031] Based on the decision given by the rate adaptation control method, the action space used by the neural network in the existing rate adaptation control method is a set of discrete code rates, but for many RTVC service video contents (such as document demonstration, web browsing, etc. screen video), some continuous still pictures can be encoded at a very small code rate to obtain very high quality, so the existing rate adaptation method will cause a certain abuse of network bandwidth. The present application considers using the QP required for encoding as a decision, because the QP value has a better linear relationship with the perception of the human eye than the average code rate, which can better reflect the quality of user experience. Figure 2The same video is encoded using high, medium and low quality QPs and high, medium and low quality fixed bit rate modes respectively, and the video perceptual fluctuation is shown. The video perception is calculated using the picture full reference quality index fusion VMAF (Video Multimethod Assessment Fusion) developed by Netflix for subjective video quality evaluation. As can be seen from the figure, compared with using fixed bit rate encoding, the fluctuation of video quality directly using QP encoding is smaller in the span of the whole video.

[0032] The action set of the embodiment is set to a group of discrete QP values, the minimum QP value is 20, corresponding to the highest quality that can be encoded; the maximum QP value is 47, corresponding to the lowest quality that can be encoded. Between the minimum and maximum values, equally spaced intervals are divided by 3, a total of 10 levels. Such a setting not only basically covers all the clarity required by users, but also avoids the problem of neural network difficult to converge caused by too large action set. It is worth noting that changing the minimum, maximum QP value and QP interval, the scheme provided by the present application is also applicable.

[0033] In the actual training process, the output of the neural network is the probability of taking each QP value level, which is equal to 1, as shown in the following formula. Then a QP value is randomly determined according to the probability distribution of the current strategy as the decision.

[0034]

[0035] where P π represents the probability of taking action a t under policy π and state s i .

[0036] It is worth noting that in addition to the above scheme using discrete values as the action set, the scheme provided by the present application is also applicable to using continuous values as the action set. At this time, the output value a of the neural network is a floating point number located in the interval [0, 1], and the calculation formula of the next QP decision is as follows:

[0037] QP = QP min + a*(QP max -QP min ) (2)

[0038] where QP min and QP max represent the minimum and maximum QP values of the decision respectively.

[0039] 2, state set

[0040] Five parameters are set as state inputs in this embodiment: the QP of the previous decision unit, the I-frame flag, the packet loss rate, the average frame transmission delay, and the number of frozen frames. The state inputs are set to the state information of the past 6 decision units in this embodiment, which basically covers all the information needed for decision making. The present application also supports state inputs of more or less than 6 decision units.

[0041] The QP of the previous decision unit, i.e., the action taken by the agent before, can reflect the network throughput to some extent and limit the magnitude of the change in QP by the agent.

[0042] Among the input variables of the neural network, the I-frame flag is introduced to indicate whether the next decision unit contains a video I-frame. This parameter reflects the different biases of the agent's strategy for different decision units and improves the algorithm's resistance to fluctuations in video block size. For decision units containing I-frames, the agent should prioritize ensuring that the decision unit can receive the I-frame and then improve the picture quality as much as possible. For decision units containing only P-frames, the agent can use a more aggressive strategy to improve picture quality.

[0043] The packet loss rate and the average frame transmission delay are both reflections of the current network conditions. The packet loss rate is calculated as follows:

[0044]

[0045] where P NACK,i is the number of data packets that need to be retransmitted for the i-th frame in the receiving buffer, P TOTAL,i is the total number of data packets for the i-th frame in the receiving buffer, and n is the number of frames in the current receiving buffer. The average frame transmission delay is calculated as follows:

[0046]

[0047] where t rcv,i is the reception timestamp of the i-th frame in the receiving buffer, and the default value is -1 when the frame is not completely received and is replaced by the current timestamp when calculating. t snd,i is the sending timestamp of the i-th frame in the receiving buffer, and t rvc,i -t snd,i is the transmission delay of the i-th frame. n is the number of frames in the current receiving buffer.

[0048] The number of frozen frames is defined as the number of frames that are frozen in the past decision unit. Video freezing has a large impact on user QoE, and the input of this parameter can enable the agent to learn how to effectively avoid freezing.

[0049] The present application differs from other existing solutions in that it does not use the actual receiving code rate and the playback buffer occupancy as indicators. The reasons are as follows: (1) For the actual receiving code rate, because the present application does not use a fixed code rate for encoding, the code rate fluctuates greatly within the span of a video. And for different video content, the distribution of the code rate is also different. Therefore, when the network throughput is not full, that is, the receiving code rate is equal to the video code rate, the receiving code rate is not suitable for reflecting the network throughput change. (2) For the playback buffer occupancy, it is an indicator commonly used in the on-demand scenario. But in the RTVC scenario, because the video content is generated in real time, in order to ensure the playback delay, the code rate adaptation method cannot actively increase the local video cache unless the user starts a new session or a more serious stall occurs. Therefore, the playback buffer occupancy cannot be used as an indicator to reflect the network throughput fluctuation.

[0050] 3. Decision granularity

[0051] The quality adjustment granularity selected by the existing code rate adaptation method is mostly seconds, that is, 1-4 seconds are selected as a video block. However, the second-level adjustment of the video quality cannot resist the millisecond-level fluctuation of the network throughput, which will increase the video stall. The present application adopts frame-level decision granularity. Under the default video frame rate of 30 frames per second, the present application makes a decision every 5 frames. This setting not only enables the algorithm to respond to network fluctuations in a timely manner, but also avoids the problem of neural network difficulty in converging due to too many decisions, thereby improving the algorithm's ability to resist network throughput fluctuations and encoding code rate fluctuations. It is worth noting that the present application also supports decision granularity greater than or less than 5 frames (such as 1 frame).

[0052] 4. Neural network

[0053] In terms of neural network settings, in order to reduce the computational overhead, the present embodiment refers to the recurrent neural network (RNN) and the deep neural network (DNN) and adopts a lightweight three-layer neural network, such as Figure 3The Actor network structure is designed as follows: the first layer is a GRU recurrent layer; the second layer and the third layer use full connection layers; the first two layers use a linear activation function Leaky ReLU, which effectively solves the gradient disappearance problem and the neuron death problem when the input is negative; and the last layer uses a softmax activation function to convert the output into a probability distribution of the action set. The Critic network is basically similar to the Actor network, except that the Critic network does not use a softmax activation function at the end but also uses a Leaky ReLU activation function. The addition of the full connection layer enables the neural network to more accurately and finely classify the state, and the addition of the recurrent layer compensates for the problem that the full connection layer cannot handle changes over time. It is worth noting that when using continuous values as the action set, the present application uses a linear activation function such as linear on the last layer of the Actor network, so that the Actor output is a one-dimensional floating point number.

[0054] 5. Effect comparison

[0055] The present application is compared with the ARS scheme, the QoE model setting includes video block quality, positive quality switching, negative quality switching and stall penalty, and an average QoE value is calculated for each network throughput trajectory test data. Under the condition of limiting the maximum playback delay to 1 second, the comprehensive average QoE is improved by 11.47%. Figure 4 The average QoE distribution comparison between the present application and the ARS on the test set is shown, Figure 5 The average QoE improvement percentage of the present application compared with the ARS is shown, each network throughput trajectory test data is a statistical sample, which is shown in the form of a cumulative distribution function, and the steeper the slope in the figure, the more samples in the interval.

[0056] From Figure 4 it can be seen that the cumulative distribution curve of the present application is on the right as a whole, that is, there is an overall significant improvement compared with the ARS, which is also confirmed in Figure 5 , on nearly 80% of the test set samples, the present application has different degrees of improvement.

[0057] Figure 6 The result comparison between the present application and the ARS in the QoE evaluation index is shown, wherein the larger the video block quality and the positive quality switching reward, the better the algorithm performance, and the smaller the negative quality switching penalty and the stall penalty, the better the algorithm performance. From the figure, it can be seen that the present application is obviously superior to the ARS in the comprehensive average QoE index, wherein the video block quality, the negative quality switching penalty and the stall penalty are all superior to the ARS, and only the positive quality switching reward is slightly inferior to the ARS.

Claims

1. A frame-level rate adaptation control method based on deep reinforcement learning, characterized in that, The method comprises the following steps: (1) generating an original video by a content provider, and then dividing the original video into decision units by using a video slicing operation; (2) for each decision unit, using a neural network to give a quantization parameter QP value of video coding according to a current decision; then, encoding each decision unit and transmitting the encoded decision unit; wherein, the input of the neural network comprises five state parameters: a quantization parameter QP value of a previous decision unit, an I frame mark, a packet loss rate, an average frame transmission delay and a number of frozen frames, wherein the I frame mark is used to indicate whether the next decision unit contains a video I frame; (3) a video sending end sends video streams of the decision units in a frame unit in sequence, and a receiving end maintains a receiving buffer and a playing buffer locally; frames that are not received completely are temporarily stored in the receiving buffer, and continuous frames that are received completely and can be decoded are transferred to the playing buffer for playing in sequence; a current real-time packet loss rate and an average frame transmission delay are calculated according to the frames that are not received completely in the receiving buffer, and the number of frozen frames is calculated according to the number of remaining frames in the playing buffer, so as to reflect fluctuations of network throughput and fluctuations of coding stream size; (4) according to state information fed back to an agent during a transmission process, calculating a quality of experience, updating parameters of the neural network, and outputting a next quantization parameter QP decision; (5) repeating the steps (1) to (4) until an optimal decision strategy is converged.

2. The frame-level rate adaptation control method based on deep reinforcement learning according to claim 1, wherein, In the step (2), a decision granularity of a video frame level is adopted.

Images