Adaptive code rate video streaming method based on improved PPO and deep reinforcement learning
Through the improved PPO method and LSTM-SA network structure, deep reinforcement learning has solved the problem of low training efficiency and insufficient stability in adaptive bit rate video streaming, achieving more efficient and stable video streaming experience quality optimization.
Patent Information
- Application Number
- CN202411250688.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-06
- Publication Date
- 2024-12-06
AI Technical Summary
The existing deep reinforcement learning methods have low training efficiency, insufficient stability and insufficient generalization capabilities in adaptive bit rate video streams, making it difficult to maintain efficient performance in complex network environments.
The improved Proximal Policy Optimization (PPO) method is used to combine deep reinforcement learning, and the strategy update is optimized to improve training stability and generalization capabilities through the dual cutting mechanism and LSTM-SA network structure.
It significantly improves the training efficiency, stability and generalization capabilities of the adaptive bit rate method, improves the user experience quality, and shows higher dynamic optimization capabilities and adaptability.
Smart Images

Figure CN119094801A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of multimedia communication and network transmission, and in particular to an adaptive bit rate video streaming method based on improved PPO and deep reinforcement learning. Background Art
[0002] With the rapid development of the Internet and mobile communication technology, video streaming has become an indispensable part of network applications. Users can watch video content anytime and anywhere through various devices and enjoy convenient entertainment and information acquisition methods. Sandvine stated in the "2023 Global Internet Phenomenon Report" that in the global Internet traffic, video accounts for 65% of the total traffic, and video traffic will continue to increase in the future, which also puts higher demands on the demand for high-quality video experience. However, the popularity of video streaming also brings many challenges. During the video streaming transmission process, in the case of poor network conditions or large fluctuations, inappropriate bit rate selection may cause problems such as video freeze and image quality degradation, seriously affecting the quality of experience (Quality of Experience, QoE). This situation will not only reduce user satisfaction, but also directly affect the commercial benefits and market competitiveness of content providers. Therefore, in the case of significant differences in network bandwidth fluctuations and device performance, how to ensure that users continue to obtain high-quality video viewing experience has become an urgent problem to be solved.
[0003] Deep reinforcement learning combines deep learning and reinforcement learning to solve the problem of an agent learning the optimal behavior strategy in the process of interacting with the environment. In deep reinforcement learning, the agent optimizes its strategy by constantly observing the state of the environment, performing actions, and receiving rewards, so that it can make the best decision when encountering similar situations in the future. This approach has achieved major breakthroughs in many fields, including robot control, autonomous driving, and network resource management. For example, AlphaGo applied deep reinforcement learning in the game of Go and achieved performance beyond human level, demonstrating the potential and application value of this technology. In addition, deep reinforcement learning has also performed well in multiple network scenarios, such as software-defined networks (SDNs), wireless sensor networks (WSNs), and vehicle-mounted ad hoc networks (VANETs), demonstrating its great potential in improving network performance and resource utilization efficiency.
[0004] In deep reinforcement learning (DRL), reinforcement learning guides the learning and decision-making process of intelligent agents in complex environments by defining decision frameworks and policy update methods. Deep learning is used to deal with problems such as function approximation and feature extraction in reinforcement learning, so as to better realize the learning and decision-making of intelligent agents in complex environments. The adaptive bitrate (ABR) system dynamically adjusts the video bitrate according to network conditions and user needs to provide the best viewing experience and QoE. Traditional ABR methods are usually based on heuristic rules or model-based methods, which often have difficulty handling complex network environments and video content. In contrast, deep reinforcement learning can automatically discover the relationship between complex network features and video content by learning the mapping from environmental states to optimal actions, and optimize the decision of adaptive bitrate in continuous interaction, thereby improving the user's viewing experience and QoE.
[0005] In video streaming, adaptive bitrate (ABR) is an effective way to solve these problems. The ABR method can dynamically adjust the bitrate of the video according to the current network conditions and device performance to ensure that users get the best viewing experience. In practical applications, traditional heuristic ABR methods mainly include buffer-based and rate-based methods based on network throughput. The buffer-based method adjusts the bitrate based on the monitored buffer status to maximize video quality and buffer utilization, but it is not responsive enough to the dynamic network environment, which may lead to QoE degradation or buffer overflow; the rate-based method dynamically adjusts the bitrate by estimating the network bandwidth to ensure the smoothness of video transmission, but ignores the status of the client buffer, resulting in unstable performance.
[0006] In view of the shortcomings of traditional heuristic methods, researchers have proposed some improved methods based on machine learning based on the research of buffer-based and throughput-based adaptive algorithms. Claeys [Claeys M, Latre S, Famaey J, et al. Design and evaluation of a self-learning HTTP adaptive video streaming client [J]. IEEE communications letters, 2014, 18 (4): 716-719.] et al. proposed an HTTP adaptive streaming client based on adaptive Q learning. Unlike traditional heuristic methods, this client can dynamically learn the optimal behavior corresponding to the current network environment. Chiariotti [Chiariotti F, D'Aronco S, Toni L, et al. Online learning adaptation strategy for DASH clients [C] / / Proceedings of the 7th international conference on multimedia systems. 2016: 1-12.] et al. proposed a dynamic adaptive streaming (DASH) client logic based on reinforcement learning, which selects the best representation through Markov decision process (MDP) optimization and ensures fast and accurate learning convergence through parallel learning technology. Lin HD, Shen ZY, Zhou H K, et al. KNN-Q learning algorithm of bitrate adaptation for video streaming over HTTP [C] / / 2020 Information Communication Technologies Conference (ICTC). IEEE, 2020: 302-306. et al. combined the nearest neighbor (KNN) algorithm with the Q learning algorithm and proposed a new KNN-Q learning algorithm for seamless switching of adaptive bitrates for video streams. Mao H, Chen S, Dimmery D, et al. Real-world Video Adaptation with Reinforcement Learning. 2020 [2024-07-16]. DOI: 10.48550 / arXiv.2008.12858. et al. performed adaptive bitrate based on reinforcement learning, used Bayesian optimization to maximize the quality of experience (QoE), and trained a linear strategy to reduce the delay between the video client and the simulation environment.However, the linear method leads to a decrease in algorithm performance. The above algorithms are all based on reinforcement learning (RL), which optimizes QoE performance in a data-driven way, does not rely on pre-designed models or environmental assumptions, and gradually learns the best adaptive bitrate strategy, with significantly better performance than traditional heuristic methods. However, these RL-based ABR algorithms have the problems of dimensionality curse and slow convergence, and it is difficult to maintain efficient performance in the case of large state and action space.
[0007] Recently, with the development of deep learning technology in various fields, some researchers have combined deep learning with reinforcement learning and achieved remarkable results. For example, Google's Deep Mind team used deep reinforcement learning technology to develop AlphaGo, which became the first artificial intelligence program to defeat the human Go world champion, marking a major breakthrough in artificial intelligence in the field of complex strategies. Later, some researchers proposed an ABR method based on deep reinforcement learning (DRL). Pensieve [Mao H, Netravali R, Alizadeh M. Neural adaptive video streaming with pensieve [C] / / Proceedings of the conference of the ACM special interest group on data communication. 2017: 197-210.] first proposed an ABR method using DRL. As the benchmark of the current DRL-based ABR method, it uses the A3C (Asynchronous Advantage Actor-Critic) method to perform adaptive bitrate by training a neural network model. It can learn the optimal bitrate selection strategy by simulating different network environments and user behaviors, and has strong adaptive ability and intelligence. However, there are shortcomings such as unstable training and inability to request the optimal bitrate. D-DASH [Gadaleta M, Chiariotti F, Rossi M, et al. D-DASH: A deep Q-learning framework for DASH video streaming [J]. IEEE Transactions on Cognitive Communications and Networking, 2017, 3 (4): 703-718.] is a method that uses Deep Q-learning to implement ABR. It evaluates different learning architectures, including feedforward and recurrent deep neural networks, and combines advanced strategies for performance evaluation.LASH[Lekharu A, Moulii KY, Sur A, et al. Deep learning based prediction model for adaptive video streaming[C] / / 2020International Conference on COMmunicationSystems&NETworkS(COMSNETS).IEEE,2020:152-159.] Based on the Pensieve method, the network structure was optimized by introducing LSTM, which improved the performance of the method. However, there is a problem of insufficient baseline function learning, which leads to low training efficiency. At present, the DRL method still has a lot of room for improvement. Summary of the invention
[0008] In view of the technical problems of low training efficiency, insufficient stability and insufficient generalization ability of the current deep reinforcement learning (DRL) method, the present invention proposes an adaptive bitrate video streaming method (Improve Proximal Policy Optimization-Deep Reinforcement Learning-AdaptiveBitrate, IPPO-DRL-ABR) based on improved PPO and deep reinforcement learning, which uses the proximal policy optimization (PPO) method with dual clipping mechanism as the training framework to ensure the stability of policy update and reduce the risk of crash; adopts LSTM-SA network structure, integrates the learning ability of long short-term memory network (LSTM) and self-attention mechanism (Self-Attention), captures long-term, short-term and global dependencies in data, and significantly improves the training efficiency, stability and generalization ability of the adaptive bitrate method.
[0009] In order to achieve the above object, the technical solution of the present invention is achieved as follows:
[0010] An adaptive bitrate video streaming method based on improved PPO and deep reinforcement learning includes the following steps:
[0011] S1: Collect the data set and divide it into training set and test set;
[0012] S2: Define the strategy and build the environment, build the IPPO-DRL-ABR model based on the strategy, initialize the model parameters and input the training set into the IPPO-DRL-ABR model;
[0013] S3: Based on the IPPO-DRL-ABR model, the strategy is used to sample multiple times in the environment and record trajectory data;
[0014] S4: Based on trajectory data, the PPO method with double pruning mechanism is used to update the strategy;
[0015] S5: Repeat steps S3 to S4 until a predetermined training period or performance standard is reached to obtain a trained IPPO-DRL-ABR model;
[0016] S6: Use the test set to verify the trained IPPO-DRL-ABR model and output the optimal bit rate.
[0017] In step S1, the data set includes a network trajectory data set and video data. 80% of the samples are randomly selected from the data set as a training set, and the remaining 20% are used as a test set.
[0018] The method for defining the strategy in step S2 is: define the strategy function π:π(s t ,a i )→[0,1], where π(s t ,a i ) indicates that the policy function π receives the environment state information s at each time step t Take action a i The action probability of the strategy function finally outputs the probability distribution of all actions π(s t )={π(s t ,a1),π(s t ,a2),…}; define the value function V = V(s t ), V(s t ) represents the state value;
[0019] The method for constructing the environment described in step S2 is: defining the network status and video playback information: determining the key parameters required in the environment; simulating the video streaming environment: constructing a simulation platform for generating the network status and video playback information; designing the feedback mechanism of the environment: setting the rewards returned by the environment and the new environment status information.
[0020] The construction method of the IPPO-DRL-ABR model described in step S2 is: constructing a data acquisition module, connecting the data acquisition module with an intelligent agent, the intelligent agent includes a policy network and a value network, the policy network and the value network are both connected to the data acquisition module, and the policy network is connected to the environment; the environment receives the action selected by the intelligent agent based on the policy network, returns a reward to the intelligent agent, and updates the state, the environment is connected to the data acquisition module, and the intelligent agent uses an LSTM-SA network and a fully connected network that combine a self-attention mechanism and a long short-term memory network to learn the characteristics of the environment state information and perform feature extraction;
[0021] The process of initializing the model parameters includes initializing the strategy network parameters θ0 and the value network parameters φ0.
[0022] The method of multiple sampling in the environment based on the IPPO-DRL-ABR model using the strategy described in step S3 is: initialize the environment at the beginning of each training round k; use the data acquisition module to interact with the environment at each time step t in each training round k to collect environmental state information s t , use the strategy network and value network to extract features of environmental state information and output action probability distribution and state value According to the action probability distribution Choose one of the actions t , according to action a t Get reward r from the environment t , the data acquisition module obtains new environmental status information s from the environment t+1 , record trajectory data Repeat multiple samplings for multiple time steps until the predetermined number of time steps T is reached, and finally obtain the trajectory data D after multiple samplings for each training round k k θ k is the policy network parameter of the current training round of the policy network, φ k is the value network parameter of the current training round of the value network.
[0023] The environmental status information s t This includes multiple state parameters of a single value type and multiple state parameters of a continuous value type.
[0024] The method for extracting features from environmental state information using a strategy network and a value network is as follows:
[0025] The policy network uses M parallel fully connected layers to extract features from M single value type state parameters and uses N parallel LSTM-SA networks to extract features from N continuous value type state parameters. The obtained M+N feature extraction results are aggregated and then passed through the fully connected layer and activation function to obtain the action probability distribution π θ (s t ); The value network uses M parallel fully connected layers to extract features from M single value type state parameters and uses N parallel LSTM-SA networks to extract features from N continuous value type state parameters. The obtained M+N feature extraction results are aggregated and then passed through multiple fully connected layers to obtain the state value V φ (s t ).
[0026] The calculation process of the LSTM-SA network is as follows: inputting the state parameters of the continuous value type into the LSTM network, passing through the forget gate, input gate, update memory unit, output gate and hidden state update in each time step of the LSTM-SA network, and dynamically updating the cell state and hidden state;
[0027] The calculation process of the forget gate is: t =σ(W f ·[h t-1 ,x t ]+b f );
[0028] Where σ is the sigmoid function, W f is the weight matrix of the forget gate, b f is the bias of the forget gate, h t-1 represents the hidden state at the t-1th time step, x t is the state parameter of the continuous value type input at the tth time step, [h t-1 ,x t ] means to concatenate the hidden state of the previous time step and the continuous value type state parameter of the input of the current time step;
[0029] The calculation process of the input gate is: t =σ(W i ·[h t-1 ,x t ]+b i );
[0030] Among them, i t represents the output of the input gate, w i is the weight matrix of the input gate, b i is the bias of the input gate;
[0031] The process of updating memory cells:
[0032]
[0033] in, Candidate memory cell state, tanh() represents the activation function used to calculate the candidate memory cell state, W C is the weight matrix for calculating the candidate memory cell state, b C is the bias term C for calculating the candidate memory cell state t is the updated memory unit, ⊙ represents element-by-element multiplication;
[0034] The calculation process of the output gate: o t =σ(W o ·[h t-1 ,x t]+b o );
[0035] Among them, t represents the output of the output gate, w o is the weight matrix of the output gate, b o is the bias of the output gate;
[0036] The hidden state update process is: h t =o t ⊙tanh(C t );
[0037] Among them, h t represents the final hidden state at each time step;
[0038] Finally, the state sequence corresponding to the state parameter of the continuous value type is obtained h=h1,h2,…h i ,…h j ,…,h t ,…,h n , n represents the length of the state parameter of the continuous value type input to the LSTM network, h i 、h i Represent the hidden states of the i-th time step and the j-th time step respectively;
[0039] Calculate the attention weight between each element in the hidden state sequence h and other elements. The specific calculation formula is as follows:
[0040]
[0041] Among them, Q(h i ) and K(h j ) represent the hidden state h through query and key functions respectively. i and the hidden state h j The vector representation projected into the query space and key space, d k Denotes the query vector Q(h i ) dimension;
[0042] The attention weight is used to perform weighted summation on each element in the state sequence h to obtain a weighted representation:
[0043]
[0044] Among them, V(h j ) means that the hidden state h is converted to j Vector representation projected into value space;
[0045] Finally, the weighted representation obtained by weighted summation is used as the feature extraction result of the LSTM-SA network.
[0046] The method for updating the strategy using the PPO method with a double pruning mechanism described in step S4 is: based on the collected trajectory data D k Calculate the cumulative discounted reward at each time step Based on the cumulative discounted reward at each time step and the state value at each time step Compute the advantage estimate at each time step By introducing the correction coefficient and using the double pruning of the double advantage estimation to maximize the PPO objective function, the policy network parameters are updated; the value function of each time step is fitted by mean square error regression Update the value function.
[0047] The cumulative discount reward The calculation process is:
[0048]
[0049] Where γ is the discount factor, r t+b is the reward at the t+bth time step;
[0050] The odds estimate
[0051] The method for maximizing the PPO objective function by introducing a correction coefficient and utilizing double pruning of double advantage estimation is:
[0052]
[0053] Among them, arg max means maximizing the objective function, r t (θ k ) is the correction coefficient, clip() is a clipping function, ε is a small hyperparameter, and c represents a real number greater than 1 used to limit the lower limit of clipping;
[0054] Fitting the value function V at each time step through mean squared error regression φk The method is:
[0055]
[0056] arg min means minimizing the objective function.
[0057] The beneficial effects of the present invention are:
[0058] In the DRL system, the present invention adopts an improved proximal policy optimization (IPPO) method for policy update, which significantly improves the performance of the ABR method. In order to solve the problem of gradient selection bias in the PPO method when the advantage function is negative, that is, the larger the absolute value of the gradient, the easier it is to be selected, the present invention introduces a double clipping mechanism for optimization for adaptive bit rate applications. This method effectively reduces the risk of crash during the policy update process and further improves the training stability and optimization performance of the ABR method.
[0059] For ABR application scenarios, the present invention takes into account the diversity of state parameters such as network throughput and buffer, and designs a more suitable deep neural network structure (LSTM-SA) according to the data type, which improves the feature extraction and learning capabilities of different types of data. By optimizing the network training strategy, the value function and strategy function in DRL can be better fitted, thereby achieving better strategy learning. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0061] Figure 1 This is a framework diagram of the adaptive video streaming protocol (Dynamic Adaptive Streaming over HTTP, DASH) of the present invention.
[0062] Figure 2 An overview diagram of the deep reinforcement learning system.
[0063] Figure 3 This is a framework diagram of the deep reinforcement learning method (IPPO-DRL-ABR) of the present invention.
[0064] Figure 4 This is the architecture diagram of the network model (LSTM-SA) of the present invention.
[0065] Figure 5 It is a comparison chart of the QoE experimental results of the present invention.
[0066] Figure 6 It is a comparison chart of the QoE decomposition experimental results of the present invention.
[0067] Figure 7 This is a generalization comparison diagram of the present invention under 0.2-1.5Mbps (poor) network conditions.
[0068] Figure 8 This is a generalization comparison diagram of the present invention under 0.2-5Mbps and above (good) network conditions. DETAILED DESCRIPTION
[0069] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0070] Figure 1 The IPPO-DRL-ABR method uses PPO as the reinforcement learning framework and integrates the deep neural network (DRL) to optimize the ABR method.
[0071] An adaptive bitrate video streaming method based on improved PPO and deep reinforcement learning, such as Figure 3 As shown, the steps include:
[0072] S1: Collect the dataset and divide it into training set and test set.
[0073] To ensure the ABR method under specified real network conditions, the experimental dataset of the present invention is divided into two parts: a network trajectory dataset and a video dataset.
[0074] Network trajectory dataset: Contains two broadband datasets, FCC and HSDPA. These datasets are derived from data generated when users use mobile devices to watch videos in scenarios such as trains, buses, and trams. The FCC dataset is maintained by the Federal Communications Commission of the United States and covers relevant information of various broadband Internet service providers (ISPs) in the United States. It contains more than 1 million network throughput data with a granularity of 5 seconds, from which 1,000 trajectories were selected; the HSDPA dataset is bandwidth tracking data collected in Norway, providing real-world measurement results with a granularity of 1 second within 30 minutes, and selected trajectories with the same specifications as the FCC dataset. At the same time, in order to optimize the model and avoid falling into the situation of always selecting the maximum bit rate, the average throughput in the dataset is limited to the range of 0.2Mbps to 6Mbps.
[0075] Video dataset: As a benchmark video for evaluating the ABR method, the present invention uses the reference video ' / Envivio' in the DASH player. This video uses the H.264 / MPEG-4 codec and is encoded at six different bit rates. The video is divided into 48 video blocks, each of which is about 4 seconds long.
[0076] In adaptive bitrate (APR) video streams, cumulative QoE is usually used to evaluate the prediction performance of the algorithm. The QoE function calculation formula is as follows:
[0077]
[0078] The QoE indicator consists of three parts: For video data with N video blocks, Rn represents the bit rate of the nth block. The first item is the video quality: q(R n ) represents the mapping of the nth bitrate to the user-perceived quality; the second term is the rebuffering time: T n Indicates the selected code rate R n The rebuffering time when downloading video chunks; the third term is video smoothness: |q(R n+1 )-q(R n )| represents the bitrate difference between two video blocks to perceive the change in video quality. Where μ and ρ represent the penalty weight coefficients of rebuffering time and video smoothness, respectively.
[0079] Experimental settings: 80% of the samples are randomly selected from the dataset as the training set, and the remaining 20% are used as the test set. In the QoE function, the penalty weight coefficients μ and ρ for rebuffering time and video smoothness are set to 4.3 and 1 respectively. For the policy (Actor) network, 8 sets of past state information are passed to the neural network for each round of training, that is, the number of time steps T = 8. Among them, the LSTM network contains 128 neurons, and the fully connected layer uses 128 neurons. The value (Critic) network uses the same network structure, and the network learning rate is configured to 10 -4 ,The optimizer selects Adam, and the discount factor γ=0.99.,All the above experiments use the deep learning library PyTorch for training and testing, and the hyperparameters remain unchanged throughout the experiment.
[0080] S2: Define the strategy and build the environment, build the IPPO-DRL-ABR model based on the strategy, initialize the model parameters and input the training set into the IPPO-DRL-ABR model.
[0081] The method of defining a strategy is: define a strategy function π: π(s t ,a i )→[0,1], where π(s t ,ai ) indicates that the policy function π receives the environment state information s at each time step t Take action a i The action probability of the strategy function finally outputs the probability distribution of all actions π(s t )={π(s t ,a1),π(s t ,a2),…}; define the value function V = V(s t ), V(s t ) represents the state value; the strategy of the present invention is implemented through the constructed IPPO-DRL-ABR model.
[0082] The IPPO-DRL-ABR model is a learning method for agents to interact with the environment to obtain the maximum cumulative reward. First, it needs to interact with the video streaming environment to obtain the current network status and video playback information, such as bandwidth, latency, etc. Then, the policy network is used to select the optimal bit rate based on this environmental status information. Building an environment means establishing an environmental model that can simulate the interactive process of video streaming, which is used for the agents in reinforcement learning to interact with it.
[0083] The method to build the environment is:
[0084] Define network status and video playback information: determine the key parameters required in the environment, such as bandwidth, latency, video buffering, etc.
[0085] Simulated video streaming environment: Build a simulation platform for generating network status and video playback information; this platform needs to be able to update the network status in real time and provide feedback to the agent;
[0086] Design the feedback mechanism of the environment: When the agent selects an action (bitrate), the environment returns the corresponding reward and new environment state information. The reward function is usually based on user experience quality (QoE) indicators such as smoothness, rebuffering, etc.
[0087] Integrate the agent and the environment: Connect the agent to the constructed environment so that the agent can choose actions and receive feedback based on the state information provided by the environment, thereby gradually optimizing the strategy.
[0088] The construction method of the IPPO-DRL-ABR model is as follows: construct a data acquisition module to interact with the constructed environment, obtain the current network status and video playback information, and connect the data acquisition module to the intelligent agent (ABR agent). The intelligent agent includes a policy network (Actor) and a value network (critic). The policy (Actor) network is used to update the strategy and output the probability distribution of the action according to the current environment state, and select and execute the action (bitrate) according to the probability distribution of the action. The value (critic) network is used to evaluate the value of the environment state, help the action decision module to better select actions and perform subsequent policy updates. The policy network and the value network are both connected to the data acquisition module, and the policy network is connected to the environment (video streaming media); the environment receives the action selected by the intelligent agent based on the policy network and returns the user QoE of this video experience, that is, the user reward. The intelligent agent updates the strategy according to this reward. The environment is connected to the data acquisition module to facilitate the data acquisition module to obtain the new state of the environment. The intelligent agent uses the LSTM-SA network and the fully connected network that combine the self-attention mechanism (SA) and the long short-term memory network (LSTM) to learn the features of the environment state information for feature extraction.
[0089] The output of the policy network (Actor) is a vector, in which each element represents the probability of selecting the corresponding action (the bit rate selected by the video block). After obtaining the output of the policy network, the agent will select a specific action (bit rate) according to the output probability distribution and apply it to the video playback process. The output of the value network (Critic) is an estimate of the value of the action corresponding to the current environmental state, that is, the expected return of taking a certain action given the current environmental state. This value estimate can help the policy network (Actor) better select actions to maximize long-term cumulative rewards.
[0090] Initialize the policy network parameters θ0 and the value network parameters φ0 before the entire training begins.
[0091] S3: Based on the IPPO-DRL-ABR model, the strategy is used to sample multiple times in the environment and record trajectory data;
[0092] Each time the agent performs an action and receives a new state and reward from the environment, a time step is completed, and each training round usually contains multiple time steps.
[0093] The method of using the strategy to sample multiple times in the environment based on the IPPO-DRL-ABR model is as follows: the environment is initialized at the beginning of each training round k, and when sampling is performed once at each time step t, the data acquisition module is used to interact with the environment to collect environmental state information s t, use the strategy network and value network to extract features of environmental state information and output action probability distribution and state value According to the action probability distribution Choose one of the actions t , according to the selected action a t Get reward r from the environment t , the data acquisition module obtains new environmental status information s from the environment t+1 , record trajectory data Repeat multiple samplings for multiple time steps until the predetermined number of time steps T is reached, and finally obtain the trajectory data D after multiple samplings for each training round k k :
[0094]
[0095] Among them, θ k is the policy network parameter of the current training round of the policy network, φ k is the value network parameter of the current training round of the value network.
[0096] The present invention designs six environmental state parameters as the input of the policy network (Actor) and the value network (critic). By comprehensively analyzing the characteristics of each environmental state parameter, the input parameters are divided into two categories, one is a single value and the other is a continuous value.
[0097] The agent will pass the state of the environment for each block observed in the environment to the deep neural network:
[0098] The current buffer size c i , the number of remaining video blocks n i and l i The bit rate of the previous video block is a state parameter of a single value type; considering the independent characteristics of a single value, the present invention uses a fully connected layer to map each input feature to a feature space. This structure is simple, easy to understand and implement, and usually converges quickly during training.
[0099] Past video block throughput Elapsed video chunk download time and the next video chunk size All of them are state parameters of continuous value type. Considering that these inputs have continuity characteristics, the present invention adopts LSTM network and self-attention mechanism to design the model. This structure can capture the global dependency more effectively while capturing the long-term and short-term dependencies in the data, thereby improving the model's ability to understand and utilize network environment information.
[0100] The method for extracting features from environmental state information using a strategy network and a value network is as follows:
[0101] The policy network uses M parallel fully connected layers to extract features from M single value type state parameters and uses N parallel LSTM-SA networks to extract features from N continuous value type state parameters. The obtained M+N feature extraction results are aggregated and then passed through the fully connected layer and activation function to obtain the action probability distribution π θ (s t );
[0102] The value network uses M parallel fully connected layers to extract features from M single value type state parameters and uses N parallel LSTM-SA networks to extract features from N continuous value type state parameters. The obtained M+N feature extraction results are aggregated and then passed through multiple fully connected layers to obtain the state value V. φ (s t ).
[0103] By observing the input of the policy network (Actor) and the value network (critic), and comprehensively analyzing their respective characteristics, the present invention adopts a more complex deep neural network for continuous values. These input parameters have sequence characteristics and contain historical information about the download of continuous video blocks in the video streaming system. In order to better capture the relationship and dependency between data points, the past network throughput, download time, and the size of the next video block have an impact on the download of the current and future video blocks, and there may be complex long-term and short-term dependencies between them. The present invention is designed to use a combination of LSTM network and self-attention mechanism. LSTM-SA unit such as Figure 4 As shown in Figure 1, the LSTM network is a neural network structure specially designed to process sequence data. It has memory units and gating mechanisms, which can effectively capture and utilize long-term and short-term dependencies in sequence data.
[0104] LSTM networks are more effective in processing long sequence data. They can capture and utilize long-term dependencies in sequences, avoiding the gradient vanishing and gradient exploding problems in conventional RNNs. The core idea of LSTM networks is to introduce a structure called a "memory cell" that controls the input, output, and state update of the memory cell through a gating mechanism. There is a long-term state called a "cell state" inside the memory cell, which is used to maintain information for a long time. At each time step of the LSTM network, the LSTM network receives the input of the current time step and the hidden state of the previous time step, and then updates the state of the memory cell based on the current input and the previous hidden state.
[0105] The calculation process of the LSTM-SA network is as follows: inputting the state parameters of the continuous value type into the LSTM network, passing through the forget gate, input gate, update memory unit, output gate and hidden state update in each time step of the LSTM-SA network, and dynamically updating the cell state and hidden state;
[0106] The calculation process of the forget gate is: t =σ(W f ·[h t-1 ,x t ]+b f )
[0107] Where σ is the sigmoid function, W f is the weight matrix of the forget gate, b f is the bias of the forget gate, h t-1 represents the hidden state at the t-1th time step, x t is the state parameter of the continuous value type input at the tth time step of the LSTM network, [h t-1 ,x t ] means to concatenate the hidden state of the previous time step and the continuous value type state parameter of the input of the current time step;
[0108] The calculation process of the input gate is: t =σ(W i ·[h t-1 ,x t ]+b i );
[0109] Among them, i t represents the output of the input gate, w i is the weight matrix of the input gate, b i is the bias of the input gate;
[0110] The process of updating memory cells:
[0111]
[0112] in, represents the candidate memory cell state, tanh() represents the activation function used to calculate the candidate memory cell state, and w C is the weight matrix for calculating the candidate memory cell state, b C is the bias term for calculating the candidate memory cell state, C t is the updated memory unit, ⊙ represents element-by-element multiplication;
[0113] The calculation process of the output gate: o t =σ(W o ·[h t-1 ,x t]+b o );
[0114] Among them, t represents the output of the output gate, w o is the weight matrix of the output gate, b o is the bias of the output gate;
[0115] The hidden state update process is: h t =o t ⊙tanh(C t );
[0116] Among them, h t represents the final hidden state at each time step;
[0117] Finally, the state sequence corresponding to the state parameter of the continuous value type is obtained h=h1,h2,…h i ,…h j ,…,h t ,…,h n , n represents the length of the state parameter of the continuous value type input to the LSTM network, h i 、h i Represent the hidden states of the i-th time step and the j-th time step respectively;
[0118] There may be complex associations and dependencies between input data, which may not only be limited to long-term and short-term correlations, but may also be global. Although the LSTM network can capture long-term and short-term dependencies, it has certain limitations in processing global dependencies. In order to solve this problem, the present invention introduces a self-attention mechanism. The self-attention mechanism can pay different degrees of attention to information at different positions in the sequence, and has global relevance and flexibility. It allows the model to simultaneously consider the relationship between all positions in the sequence without being restricted by the length of the sequence, focusing on the most relevant part of the sequence, thereby better capturing global associations. The self-attention mechanism is very flexible and can adapt to sequence inputs of different lengths and forms, and performs well in processing both long and short sequences. The self-attention mechanism is a mechanism for extracting key information from sequence data. Its core principle is to calculate the weight of each element by learning the relationship between elements, and then use these weights to weighted sum the elements to obtain a representation of the sequence.
[0119] Calculate the attention weight between each element in the hidden state sequence h and other elements. The specific calculation formula is as follows:
[0120]
[0121] Among them, Q(h i ) and K(h j) respectively represent the hidden state h through query and key functions i and the hidden state h j The vector representation projected into the query space and key space, d k Denotes the query vector Q(h i ) dimension;
[0122] The attention weight is used to perform weighted summation on each element in the state sequence h to obtain a weighted representation:
[0123]
[0124] Among them, V(h j ) indicates that the hidden state h is converted to j Vector representation projected into value space;
[0125] Finally, the weighted representation obtained by weighted summation is used as the feature extraction result of the LSTM-SA network.
[0126] Combining these two information capture mechanisms can significantly improve the performance of sequence data processing models. The LSTM network effectively captures the long-term and short-term dependencies in the sequence through its gating mechanism, but when processing long sequences, it often faces the problem of information transmission attenuation. The self-attention mechanism allows the model to dynamically allocate attention between different positions, rather than just adjacent positions, so as to better capture global associations. Combining the two, the memory mechanism provided by the LSTM network is used to process the long-term and short-term dependencies in the sequence, and the self-attention mechanism is used to flexibly adjust and weight the information at different positions in the sequence, so that the model can better maintain the transmission and effectiveness of information when processing the sequence. This integration strategy can better adapt to sequence data of different lengths and forms, and show stronger performance in various sequence tasks.
[0127] S4: Based on trajectory data, the PPO method with double pruning mechanism is used to update the strategy;
[0128] The method of updating the strategy using the PPO method with a double pruning mechanism is as follows: according to the collected trajectory data D k Calculate the cumulative discounted reward at each time step Based on the cumulative discounted reward at each time step and the state value at each time step Compute the advantage estimate at each time step By introducing the correction coefficient and using the double pruning of the double advantage estimation to maximize the PPO objective function, the policy network parameters are updated; the value function of each time step is fitted by mean square error regression Update the value function.
[0129] The cumulative discount reward The calculation process is:
[0130]
[0131] Among them, γ is the discount factor, which is a key hyperparameter in reinforcement learning and is used to balance the importance of immediate rewards and future rewards. t+b is the reward at the t+bth time step;
[0132] The odds estimate
[0133] The method for maximizing the PPO objective function by introducing a correction coefficient and utilizing double pruning of double advantage estimation is:
[0134]
[0135] Among them, |D k | represents the trajectory in this set, It means to accumulate every time step of all trajectories in the data set, r t (θ k ) is the correction coefficient, also called the strategy ratio, which is used to calculate the relative probability between the current strategy and the old strategy to measure the magnitude of the strategy update. clip() is a clipping function used to limit the strategy ratio r t (θ k ), in the PPO algorithm, the clip function ensures that the policy update does not deviate too much from the old policy, thereby maintaining the stability of the policy update, ε is a small hyperparameter, c represents a real number greater than 1 used to limit the lower limit of pruning, t represents the time step, and T represents the total number of time steps of a trajectory, that is, the total number of steps a trajectory goes through from the beginning to the end.
[0136]
[0137] Fitting the value function at each time step through mean squared error regression The method is:
[0138]
[0139] arg max and arg min represent the parameters that maximize and minimize the objective function, respectively.
[0140] arg max means "the parameter at the maximum value", that is, find the parameter that makes the objective function reach the maximum value. argmax is used to find the policy network parameter θ that maximizes the objective function (i.e., the PPO objective function) k+1 .
[0141] argmin means "the parameter at the minimum value", that is, find the parameter that minimizes the objective function. argmin is used to find the value function parameter φ that minimizes the mean square error k+1 .
[0142] S5: Repeat steps S3 to S4 until the predetermined training cycle or performance standard is reached to obtain the trained IPPO-DRL-ABR model. The local training round is set to 500,000 times. Each training will record the overall average QoE reward of the model on the training data set. After the training is completed, the model with the best overall average QoE will be selected as the final model based on the log record file.
[0143] S6: Use the test set to verify the trained IPPO-DRL-ABR model and output the optimal bit rate.
[0144] experiment:
[0145] In order to verify the superior performance of the proposed method, five methods were selected for comparison with the IPPO-DRL-ABR method proposed in the present invention.
[0146] (1) Buffer-based: Buffer-based method optimizes video transmission by monitoring the client's buffer.
[0147] (2) Rate-based: A method based on network throughput that dynamically adjusts the bit rate of the video stream according to the predicted network throughput.
[0148] (3) robustMPC: A video streaming optimization method based on model predictive control. It combines network status and buffer filling status and uses model predictive control method to optimize video transmission.
[0149] (4) BOLA: A buffer-based adaptive bitrate selection method designed using Lyapunov optimization. It is the default ABR method for dash.js player.
[0150] (5) Pensieve: A video streaming optimization method based on A3C that dynamically adjusts the video bitrate and quality by learning user viewing behavior and network conditions.
[0151] The present invention draws a comparison chart of the average QoE value and CDF distribution of six ABR methods to better display the experimental results, such as Figure 5As shown in the figure, the IPPO-DRL-ABR method designed by the present invention outperforms the other five most representative ABR methods under the same conditions. Compared with these methods, the average QoE is improved by 34.22%. The IPPO-DRL-ABR method not only performs well in overall performance, but also has significantly better generalization ability on multiple network trajectories than other ABR methods.
[0152] The reason for the significant difference is that most existing methods use fixed control strategies and lack the ability to dynamically adapt to fluctuations in network conditions. Although Pensieve is also based on RL (Reinforcement Learning) methods, compared with the IPPO-DRL-ABR method, the design of other optimization methods lacks pertinence and it is difficult to stably handle environmental changes and sample deviations. In contrast, the IPPO-DRL-ABR method comprehensively considers multiple dynamic variables such as buffer status and network conditions, and uses the more efficient PPO optimization method and the learning ability of deep neural networks (LSTM-SA) to perform dynamic optimization in combination with real-time user QoE feedback.
[0153] In the experimental comparison, the IPPO-DRL-ABR method showed significant advantages in average QoE, generalization performance and stability, and has higher dynamic optimization capabilities, making it suitable for video transmission optimization scenarios under various network conditions.
[0154] In the comparative experiments of ABR methods, the focus is often on the overall effect of user experience quality (QoE). In the previous analysis, the overall effect of QoE was compared. After the disassembly analysis of the QoE function and multiple comparative experiments, it is found that analyzing each component of the QoE function (the performance of each factor) separately helps to better understand the performance of the method. Table 1 shows the performance comparison of different methods.
[0155] Table 1. Performance comparison of different methods
[0156]
[0157]
[0158] Figure 6 The performance comparison of the IPPO-DRL-ABR method and other five representative methods in various QoE factors is shown. According to the QoE function, QoE is defined as the bitrate utility minus the rebuffering penalty minus the video smoothness penalty. In short, the bitrate utility is hoped to be as high as possible, while the rebuffering and video smoothness penalties are hoped to be as low as possible.
[0159] from Figure 6It can be seen that the IPPO-DRL-ABR method does not perform well in the rate utility module, but performs well in the rebuffering penalty and video smoothness penalty modules, which are the least penalized of the six methods. Compared with BOLA and Buffer-based, which perform best in rate utility, IPPO-DRL-ABR reduces rebuffering penalties by 38.17% and 22.54%, and video smoothness penalties by 62.63% and 73.03%, respectively. This obvious performance advantage shows the excellent performance of the IPPO-DRL-ABR method in balancing the comprehensive performance of the three key factors to achieve the most optimized QoE. This comprehensive performance optimization helps to avoid over-optimization of a single performance indicator, thereby sacrificing other performance indicators, and thus ensures the stability and robustness of the method.
[0160] Considering the diversity of real-world network environments, in order to verify the generalization ability of the method in different environments (network throughput range), the present invention conducted an in-depth network data set analysis and selected two typical network environment ranges for experimental testing, namely: 0.2-1.5Mbps (poor) and 0.2-5Mbps and above (good). Under these two network conditions, the present invention demonstrated the real-time bit rate selection and corresponding buffer size of the IPPO-DRL-ABR method, and compared it with the robustMPC and Buffer-based methods. Figure 7 , Figure 8 , where true bandwidth refers to the real experimental network trajectory, that is, the data in the network trajectory dataset introduced in the dataset.
[0161] Example 1: The first example analyzes an application scenario where the overall network conditions are poor. Figure 7 It can be observed that the poor network environment and objective network fluctuations have brought great difficulties to the bitrate selection, especially in the robustMPC and Buffer-based methods. Due to the relatively inefficient buffer control level of the two methods, the network experienced large fluctuations at a timestamp of about 50s, and the buffer size also dropped sharply. The bitrate selection dropped from 1.2Mbps to nearly 0.3Mbps. The difference in video quality before and after was too large, which directly affected the results of the QoE function and reduced the user experience. In contrast, using the learning ability of the IPPO-DRL-ABR method, in the same time period, the bitrate selection was almost not negatively affected, and it was still able to stably increase to a higher level and last for more than 100 seconds, significantly improving the user experience quality when watching videos.
[0162] Example 2: Under good network conditions, the IPPO-DRL-ABR method is also significantly better than the robustMPC and Buffer-based methods. Carefully observing the real-time buffer size in the example, it can be clearly found that the IPPO-DRL-ABR method tends to quickly increase the buffer size to a higher level. Therefore, in the initial stage, the video bitrate selection of the IPPO-DRL-ABR method is relatively conservative, showing a steady increase trend. When the buffer level is high enough, the IPPO-DRL-ABR method will significantly increase the bitrate and ensure high-quality delivery for a longer period of time. Figure 8 It can be seen that when the timestamp is about 100s, the buffer size reaches a high level. Although the network conditions drop significantly in the following 10s or so, the method still ensures the subsequent stable high-quality delivery until the end. On the contrary, the robustMPC and Buffer-based methods often ignore the control of the buffer level in the initial stage and quickly increase the bit rate selection level, which may cause drastic fluctuations in the subsequent bit rate selection level and reduce the performance of the method. In the two examples, the present invention comprehensively considers the situations of poor and good network conditions. By observing the comparison chart of the bit rate selection part, it can be found that the IPPO-DRL-ABR method has a more stable bit rate selection compared with the other two methods, indicating that it has better performance in video smoothness. At the same time, because it can more effectively utilize the buffer size and real-time bandwidth conditions, the IPPO-DRL-ABR method can maintain a high bit rate level for a long time, thereby significantly improving the user's viewing experience quality. These experimental results not only verify the superiority of the IPPO-DRL-ABR method in different network environments and its higher dynamic optimization and generalization capabilities, but also highlight its potential and importance in coping with network fluctuations and improving video quality.
[0163] The present invention proposes an adaptive bitrate video streaming method based on improved PPO and deep reinforcement learning to optimize the user experience quality (QoE) of video streaming transmission. By adopting proximal policy optimization (PPO) and using a dual clipping mechanism, the stability of the policy update is ensured and the risk of crash is reduced. In addition, the learning capabilities of LSTM and self-attention are integrated to capture the long-term, short-term and global dependencies in the data in a personalized way, further improving the adaptive bitrate performance. The experimental results show that compared with the existing representative ABR methods, the IPPO-DRL-ABR method shows significant superiority in the overall QoE and the performance of each QoE factor (bitrate utility, rebuffering penalty, video smoothness penalty), has higher dynamic optimization capabilities, and the average QoE is improved by 34.22%. In addition, the generalization ability of the IPPO-DRL-ABR method in different network environments is also significantly better than other methods. Future research will continue to focus on the performance of optimization methods to further improve the user experience of video streaming services.
[0164] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. An adaptive bitrate video streaming method based on improved PPO and deep reinforcement learning, characterized in that: The following steps are involved: S1: Collect the data set and divide it into training set and test set; S2: Define the strategy and build the environment, build the IPPO-DRL-ABR model based on the strategy, initialize the model parameters and input the training set into the IPPO-DRL-ABR model; S3: Based on the IPPO-DRL-ABR model, the strategy is used to sample multiple times in the environment and record trajectory data; S4: Based on trajectory data, the PPO method with double pruning mechanism is used to update the strategy; S5: Repeat steps S3 to S4 until a predetermined training period or performance standard is reached to obtain a trained IPPO-DRL-ABR model; S6: Use the test set to verify the trained IPPO-DRL-ABR model and output the optimal bit rate.
2. The adaptive bitrate video streaming method based on improved PPO and deep reinforcement learning according to claim 1, characterized in that: In step S1, the data set includes a network trajectory data set and video data. 80% of the samples are randomly selected from the data set as a training set, and the remaining 20% are used as a test set.
3. The adaptive bitrate video streaming method based on improved PPO and deep reinforcement learning according to claim 1 or 2, characterized in that: The method for defining the strategy in step S2 is: define the strategy function π:π(s t ,a i )→[0,1], where π(s t ,a i ) indicates that the policy function π receives the environment state information s at each time step t Take action a i The action probability of the strategy function finally outputs the probability distribution of all actions π(s t )={π(s t ,a1),π(s t ,a2),…}; define the value function V = V(s t ), V(s t ) represents the state value; The method for constructing the environment described in step S2 is: defining the network status and video playback information: determining the key parameters required in the environment; simulating the video streaming environment: constructing a simulation platform for generating the network status and video playback information; designing the feedback mechanism of the environment: setting the rewards returned by the environment and the new environment status information.
4. The adaptive bitrate video streaming method based on improved PPO and deep reinforcement learning according to claim 3, characterized in that: The construction method of the IPPO-DRL-ABR model described in step S2 is: construct a data acquisition module, connect the data acquisition module to the agent, the agent includes a policy network and a value network, the policy network and the value network are both connected to the data acquisition module, and the policy network is connected to the environment; the environment receives the action selected by the agent based on the policy network, returns a reward to the agent, and updates the state, the environment is connected to the data acquisition module, and the agent adopts an LSTM-SA network and a fully connected network that combine a self-attention mechanism and a long short-term memory network to learn the characteristics of the environment state information, Perform feature extraction; The process of initializing the model parameters includes initializing the strategy network parameters θ0 and the value network parameters φ0.
5. The adaptive bitrate video streaming method based on improved PPO and deep reinforcement learning according to claim 4, characterized in that: The method of multiple sampling in the environment based on the IPPO-DRL-ABR model using the strategy described in step S3 is: initialize the environment at the beginning of each training round k; use the data acquisition module to interact with the environment at each time step t in each training round k to collect environmental state information s t , use the strategy network and value network to extract features of environmental state information and output action probability distribution and state value According to the action probability distribution Choose one of the actions t , according to action a t Get reward r from the environment t , the data acquisition module obtains new environmental status information s from the environment t+1 , record trajectory data Repeat multiple samplings for multiple time steps until the predetermined number of time steps T is reached, and finally obtain the trajectory data D after multiple samplings for each training round k k θ k is the policy network parameter of the current training round of the policy network, φ k is the value network parameter of the current training round of the value network.
6. The adaptive bitrate video streaming method based on improved PPO and deep reinforcement learning according to claim 5, characterized in that: The environmental status information s t This includes multiple state parameters of a single value type and multiple state parameters of a continuous value type.
7. The adaptive bitrate video streaming method based on improved PPO and deep reinforcement learning according to claim 6, characterized in that: The method for extracting features from environmental state information using a strategy network and a value network is as follows: The policy network uses M parallel fully connected layers to extract features from M single value type state parameters and uses N parallel LSTM-SA networks to extract features from N continuous value type state parameters. The obtained M+N feature extraction results are aggregated and then passed through the fully connected layer and activation function to obtain the action probability distribution π θ (s t ); The value network uses M parallel fully connected layers to extract features from M single value type state parameters and uses N parallel LSTM-SA networks to extract features from N continuous value type state parameters. The obtained M+N feature extraction results are aggregated and then passed through multiple fully connected layers to obtain the state value V. φ (s t ).
8. The adaptive bitrate video streaming method based on improved PPO and deep reinforcement learning according to claim 7, characterized in that: The calculation process of the LSTM-SA network is as follows: inputting the state parameters of the continuous value type into the LSTM network, passing through the forget gate, input gate, update memory unit, output gate and hidden state update in each time step of the LSTM-SA network, and dynamically updating the cell state and hidden state; The calculation process of the forget gate is: t =σ(W f ·[h t-1 ,x t ]+b f ); Where σ is the sigmoid function, W f is the weight matrix of the forget gate, b f is the bias of the forget gate, h t-1 represents the hidden state at the t-1th time step, x t is the state parameter of the continuous value type input at the tth time step, [h t-1 ,x t ] means to concatenate the hidden state of the previous time step and the continuous value type state parameter of the input of the current time step; The calculation process of the input gate is: t =σ(W i ·[h t-1 ,x t ]+b i ); Among them, i t represents the output of the input gate, w i is the weight matrix of the input gate, b i is the bias of the input gate; The process of updating memory cells: in, represents the candidate memory cell state, tanh() represents the activation function used to calculate the candidate memory cell state, W C is the weight matrix for calculating the candidate memory cell state, b C is the bias term for calculating the candidate memory cell state, C t is the updated memory unit, ⊙ represents element-by-element multiplication; The calculation process of the output gate: o t =σ(W o ·[h t-1 ,x t ]+b o ); Among them, t represents the output of the output gate, w o is the weight matrix of the output gate, b o is the bias of the output gate; The hidden state update process is: h t =o t ⊙tanh(C t ); Among them, h t represents the final hidden state at each time step; Finally, the state sequence corresponding to the state parameter of the continuous value type is obtained h=h1,h2,…h i ,…h j ,…,h t ,…,h n , n represents the length of the state parameter of the continuous value type input to the LSTM network, h i 、h i Represent the hidden states of the i-th time step and the j-th time step respectively; Calculate the attention weight between each element in the hidden state sequence h and other elements. The specific calculation formula is as follows: Among them, Q(h i ) and K(h j ) represent the hidden state h through query and key functions respectively. i and the hidden state h j The vector representation projected into the query space and key space, d k Denotes the query vector Q(h i ) dimension; The attention weight is used to perform weighted summation on each element in the state sequence h to obtain a weighted representation: Among them, V(h j ) means that the hidden state h is converted to j Vector representation projected into value space; Finally, the weighted representation obtained by weighted summation is used as the feature extraction result of the LSTM-SA network.
9. The adaptive bitrate video streaming method based on improved PPO and deep reinforcement learning according to claim 5 or 6, characterized in that: The method for updating the strategy using the PPO method with a double pruning mechanism described in step S4 is: based on the collected trajectory data D k Calculate the cumulative discounted reward at each time step Based on the cumulative discounted reward at each time step and the state value at each time step Compute the advantage estimate at each time step By introducing the correction coefficient and using the double pruning of the double advantage estimation to maximize the PPO objective function, the policy network parameters are updated; the value function of each time step is fitted by mean square error regression Update the value function.
10. The adaptive bitrate video streaming method based on improved PPO and deep reinforcement learning according to claim 9, characterized in that: The cumulative discount reward The calculation process is: Where γ is the discount factor, r t+b is the reward at the t+bth time step; The odds estimate The method for maximizing the PPO objective function by introducing a correction coefficient and utilizing double pruning of double advantage estimation is: Among them, arg max means maximizing the objective function, r t (θ k ) is the correction coefficient, clip() is a clipping function, ε is a small hyperparameter, and c represents a real number greater than 1 used to limit the lower limit of clipping; Fitting the value function at each time step through mean squared error regression The method is: arg min means minimizing the objective function.
Citation Information
Cited By
Video streaming media bit rate self-adaption method and system based on expert guidance
CN120583080A
Low-delay video stream code rate adaptation method and system based on offline element reinforcement learning
CN121711511A
