A serverless computing based adaptive video streaming method and system

Abstract

The application relates to a kind of adaptive video streaming method and system based on serverless computing, belong to streaming media transmission technical field.System is realized by fine-grained serverless pipeline Video delivery, use stateless function to strengthen the response to video request event, use a kind of near-end strategy optimization PPO of three-end clipping based on deep reinforcement learning algorithm to solve the bit rate adaptive sequence decision problem in video playing process.In addition, dynamic video block quality factor is included in user experience QoE index, to configure QoE model, for each video block is assigned a priority weight, improves the robustness of video bit rate decision, thereby reduces video streaming delay, improves user viewing experience.

Description

Technical Field

[0001] This invention relates to an adaptive video streaming method and system based on serverless computing, belonging to the field of streaming media transmission technology. Background Technology

[0002] In recent years, the widespread adoption of 5G networks has provided unprecedented opportunities for video content service providers to stream video content to users, greatly promoting the development of video streaming applications such as live streaming and short video services. Although 5G networks effectively increase network bandwidth, the internet can still be overwhelmed by massive amounts of video streaming traffic, inevitably impacting the user's viewing experience (QoE). Therefore, developing an efficient adaptive video streaming technology that considers current network conditions and client buffer limitations is essential for improving the user's viewing experience.

[0003] Currently, most video service providers use the Dynamic Adaptive Streaming (DASH) protocol to transmit video streams via content delivery networks. In this scenario, the cloud server divides the video into equal-length blocks and encodes these blocks at different bitrates. The client player then runs the ABR algorithm to select the appropriate bitrate video block for playback based on available resources (network, buffers, etc.). However, existing research largely focuses on the trade-off between video bitrate and network bandwidth under dynamic network conditions, neglecting the bitrate and download speed of video blocks. This can lead to frequent video block switching, disrupting playback smoothness and severely impacting user experience, as the initial video block bitrate decision may not be the optimal choice for downloading the video block, and such coarse-grained bitrate decisions cannot be reversed once made. Furthermore, existing video streaming schemes often rely on the client to make video block bitrate decisions based on local network resources, lacking a global perspective. Given the large-scale distributed nature of the internet, a single node can only observe a portion of the video streaming system, resulting in poor long-tail performance.

[0004] Edge platforms can provide video streaming services closer to users. Currently, edge computing-based solutions primarily utilize edge caching to provide services locally and reduce video transmission latency. Furthermore, although edge platforms have a wider field of view than end users, these solutions rarely make bitrate adaptive decisions at the edge, inevitably leading to a poor viewing experience for users.

[0005] To provide customized services, cloud service providers have proposed serverless computing paradigms in recent years, typically in the form of Function as a Service (FaaS). In serverless architectures, FaaS directly handles event-triggered service requests to ensure the availability and resilience of interconnected stateless functions. Serverless computing architectures are ideal for video streaming service applications with high data parallelism and intermittent activity. Currently, no publicly available edge-assisted, fine-grained adaptive decision-making schemes for video streaming based on serverless computing architectures have been found. Summary of the Invention

[0006] The purpose of this invention is to address the technical challenge of efficient video streaming services in complex and dynamic network environments. It innovatively proposes a video streaming method and system based on serverless computing within an edge computing architecture. This invention enhances the response to video request events by implementing a fine-grained video streaming pipeline with serverless support, reducing transmission latency and improving the user viewing experience.

[0007] The innovations of this invention include: Firstly, the design of an edge-assisted adaptive video streaming system supported by a serverless architecture. This system delivers video through a fine-grained serverless pipeline, utilizes stateless functions to enhance the response to video request events, and makes video block bitrate decisions at the edge based on available resources (computing power, network). Due to the Markovian characteristics of the video bitrate adaptation process, this invention uses a deep reinforcement learning algorithm based on proximal policy optimization (PPO) with three-end clipping to solve the bitrate adaptation sequence decision problem during video playback. Furthermore, this invention incorporates dynamic video block quality factors into the user experience (QoE) metric to configure the QoE model, assigning a priority weight to each video block to improve the robustness of video bitrate decisions, thereby reducing video stream transmission latency and improving the user viewing experience.

[0008] The objective of this invention is achieved through the following technical solution.

[0009] An adaptive video streaming method based on serverless computing includes the following steps:

[0010] Step 1: Build an adaptive video streaming system based on serverless computing under edge computing, including a video streaming server, an edge terminal, and a video user terminal.

[0011] Among them, the video streaming server is responsible for storing video content and dividing it into fixed-length video blocks encoded at different bit rates, and transmitting the video content to the video user through a complex and ever-changing dynamic network;

[0012] The edge platform implements highly parallel, stateless functions based on a serverless architecture to respond to video content requests from multiple users and interact with geographically distributed servers. By delivering video content from the server closest to the user, video transmission latency is reduced, maximizing user experience quality.

[0013] The client is used to send local player state (buffer, network, etc.) information and video block request information to the edge, triggering a stateless function response in the form of an event to generate the optimal video block bitrate decision.

[0014] Step 2: Optimize the adaptive bitrate decision of video blocks using a bitrate adaptive algorithm based on deep reinforcement learning-based proximal policy optimization.

[0015] Includes the following steps:

[0016] Step 2.1: An end-to-end deep reinforcement learning framework is used to perform bitrate adaptive decision-making during video stream playback.

[0017] Reinforcement learning is a classic strategy for solving sequential decision-making problems in dynamic environments. Deep neural networks can also handle multi-dimensional input conditions and extract features in depth. End-to-end means that the framework directly accepts information from the video streaming service to drive and output actions, without needing to encode hand-crafted features as input or perform iterative reasoning during the decision-making process.

[0018] In the video stream transmission optimization process based on deep reinforcement learning, the video stream state space information includes current and historical video stream information, network information, and action information. The agent selects an action at a specific video stream bitrate based on the current state space information and attempts to maximize the cumulative user experience (QoE) reward. To capture the complex relationship between video stream state, executed actions, and QoE reward, a pseudo-Siamese architecture is employed to learn these intrinsic relationships.

[0019] Therefore, the problem can be transformed into a Markov decision process, and a solution based on deep reinforcement learning can be designed through the interaction between the agent and the environment.

[0020] Step 2.2: State-space representation.

[0021] Existing research typically categorizes video content and network state information into different classes and uses their concatenated one-hot encoded vectors as representations of the state space to enable iterative inference processes in the model. However, these abstract encoded vectors are prone to overlooking important state information and may fail to capture the complex relationship between video information and optimal bitrate decisions.

[0022] To obtain an efficient and suitable feature representation, thereby enabling direct learning of the features required for bitrate decisions from the state space, this invention designs a novel multidimensional feature for encoding current and historical video block state information. In video streaming services, information related to video transmission is represented by different feature dimensions. Therefore, they are represented as discrete three-dimensional tensors, allowing deep neural networks to learn system models based on historical experience.

[0023] Specifically, the state space tensor is designed with nine channels, representing agent network information, video block information, player information, and historical decision information, respectively. Each channel is a sparse binary matrix, and each position represents the corresponding state space information.

[0024] Step 2.3: Action space representation.

[0025] After state-space representation, the decision on video block bitrate is designed as different video resolution actions (e.g., selecting 720P video blocks). For video block playback sequence input types (e.g., bitrate of past video blocks, download time, etc.), convolutional layers are used to extract basic features. Then, all processing results are concatenated into a fully connected layer to learn the intrinsic relationships between complex features. Finally, the output is transformed into a normalization function layer to compute the probability distribution of the actions.

[0026] Specifically, the strategy π is adjusted by modifying the parameters θ of the neural network. θ Optimize it. For example, for the actor tensor, since there are typically at most four consecutive selectable actions per round, it can be designed as four channels, each channel being a 1*N tensor. b A sparse binary matrix, N b This represents the different bitrates of the selected video blocks. Therefore, in a video streaming session, as the user's player downloads video blocks, the agent selects the appropriate action to maximize the long-term accumulated QoE reward.

[0027] Step 2.4: Setting the reward function.

[0028] After designing states and actions, further design is needed for a specific state s. t Next, take an action a t Reward r t Once the video streaming system begins making bitrate decisions, the agent will take different actions based on the current state, forming different state-action sequences in the historical experience buffer pool, thereby maximizing the cumulative reward function.

[0029] Specifically, the Quality of Experience (QoE) metric is used as the reward function. Considering the differences in content contained in different video blocks, a specific weight is assigned to each video block to encode the sensitivity to content quality. The deep reinforcement learning model, by using the QoE reward function, weighs factors such as user-perceived video clarity and buffering time, encouraging the download of video blocks at higher bitrates and minimizing playback buffering and bitrate switching as much as possible.

[0030] Step 2.5: Training the deep reinforcement learning model.

[0031] In the context of multidimensional input feature representation, a core element of policy gradient training is finding a deep learning paradigm with a suitable loss function. Specifically, an actor-commentator framework with online policy training is employed. The agent directly learns a parameterized policy π based on the expected payoff gradient of the relevant policy parameter θ. θ Actor's Critical Paradigm Training Value Function V θ (S t ) and policy π θ (a t |s t ), and update them by repeatedly sampling from the pool of replayed historical experiences.

[0032] Step 3: Use a three-end pruning mechanism to improve the performance of near-end strategy optimization.

[0033] Based on the deep reinforcement learning Actor-Critic model learned in step 2, the policy is parameterized using a proximal policy optimization algorithm. During training, parameters are updated by optimizing gradients to achieve the optimal objective function. The correction coefficients for the Approximate Policy Optimization (PPO) algorithm are defined as follows: For the current strategy π θ And the old strategy π θ′ The probability ratio.

[0034] In near-end policy optimization algorithms, a three-end pruning mechanism is used to weaken the gradients of certain outliers in order to control the policy update magnitude, thereby accelerating algorithm convergence. Specifically, in the training step of the algorithm iteration, the three-end pruning method is used to reduce the gradients of r... t (θ) represents the ratio of the new and old strategies, constrained within a region. Controlling the size of this region limits the update step size. Compared to using KL divergence for constraint, using the Clip function is more convenient. Here, clip(r) t (θ), 1-∈, 1+∈) will be the correction coefficient r t The (θ) constraint is within the range (1-∈, 1+∈), where ∈ is a hyperparameter. The truncation term acts as a constraint when the policy update offset exceeds the predetermined interval.

[0035] To further improve the performance of near-end policy optimization algorithms, advantage function estimation methods can be used. Specifically, the advantage function is constructed using generalized advantage estimation. To reduce variance and prevent significant fluctuations in the algorithm, t represents the time step, and T represents the total number of time steps. When updating the gradient of the objective function, each gradient calculation requires interactive sampling using the latest policy model to obtain the corresponding sample sequence. After updating the model, the experience gained from previously used samples is utilized by adjusting the distribution of the old policy coefficients.

[0036] Beneficial effects

[0037] Compared with the prior art, the present invention has the following advantages:

[0038] 1. This invention proposes an edge-assisted adaptive video streaming system based on a serverless architecture. It is an end-to-end video streaming transmission system covering "user-edge-server". The system implements the video stream bitrate decision function through fine-grained serverless pipelines. It can adaptively adjust the video stream bitrate according to the current dynamic network environment and client caching and other factors, reduce video stream transmission latency, and thus improve the user's viewing experience quality.

[0039] 2. This invention proposes a novel near-end strategy optimization algorithm based on three-end pruning, which optimizes the video stream bitrate decision-making process. During video stream transmission, the process is formalized as a Markov sequence decision problem. Deep reinforcement learning techniques are then used to adaptively select appropriate bitrates for video blocks, generating optimal video block bitrate decisions, thereby achieving better user experience quality. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of an adaptive video streaming method based on serverless computing, according to an embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram of the near-end strategy optimization algorithm based on three-end pruning in an embodiment of the present invention.

[0042] Figure 3 The bitrate of the video block is shown in different network datasets in the embodiments of the present invention.

[0043] Figure 4 This is an evaluation of the latency performance of different bit rate algorithms in the embodiments of the present invention.

[0044] Figure 5 This is an evaluation of the QoE performance of embodiments of the present invention using different bit rate algorithms. Detailed Implementation

[0045] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples.

[0046] Example

[0047] In this embodiment, the adaptive video streaming system adopts a typical three-layer edge computing architecture, consisting of a cloud server, an edge server, and a client. The client is responsible for playing video, collecting client status (player buffer, network throughput, etc.), and sending video requests. The edge server processes video playback requests from multiple clients and interacts with geographically distributed cloud servers. The video streaming service can be delivered from the cloud server closest to the user to reduce transmission latency. The cloud server is responsible for storing the video source files, dividing the video into fixed-duration video blocks with different bitrates, allocating available bandwidth, and pushing the video stream content to the client player.

[0048] like Figure 1 As shown, an adaptive video streaming method based on serverless computing is proposed. When serverless computing infrastructure is applied to video streaming, the serverless platform located on the edge server is responsible for executing the corresponding stateless function instances asynchronously in real time and performing fine-grained video request responses. Once a predefined video event is invoked, the serverless function concurrently performs video stream bitrate decisions based on the occurrence of the corresponding event.

[0049] An adaptive video streaming method based on serverless computing includes the following steps:

[0050] Step 1: Construct an adaptive video streaming transmission system based on serverless computing under edge computing, including a video streaming server, an edge terminal, and a video user terminal.

[0051] First, analyze the spatiotemporal characteristics of the client during video playback and build a video streaming client application.

[0052] On the video streaming client, JavaScript console logs are extracted from a Chrome browser with an integrated dash.js player. iPerf is then used to extract video chunk status information, player buffers, and network conditions from the client. The ABR (Adaptive Bitrate) controller then organizes the status metadata information, groups it, and forwards it to invoke video chunk download request events. The corresponding video request calls are then sent to the edge server via a serverless function using the HTTP protocol.

[0053] Then, a serverless computing-based video stream decision pipeline is built on the edge server to make the optimal video stream bitrate decision based on the current information.

[0054] At the edge server, the stateless function database is the intelligent core of the entire video streaming system, interacting with containers that execute serverless functions. Within the edge server, a throughput monitoring module probes the network channel link between the client and the cloud server to estimate future network throughput. In response to client video request events, the edge server immediately decrypts the request information. Specifically, the edge server uses information from the client request event (e.g., buffer occupancy and video block bitrate) and throughput predictions to determine the optimal video block bitrate using serverless functions. The edge server then passes the video block bitrate decision to the nearest cloud server. Because the edge server has a broader perspective than the client, it can more effectively improve video quality by considering more relevant information, such as client requests and available bandwidth. The serverless video streaming pipeline has an additional stateless streaming ingestion function that coordinates the execution of the video streaming workflow. At each time interval, the serverless function configurator calls stateless functions to process video streaming events and provide feedback on the video streaming pipeline. The edge server instructs callers to execute stateless functions to make fine-grained video stream bitrate decisions.

[0055] Specifically, serverless pipelines provide readily available infrastructure for real-time concurrent processing of fine-grained video streaming services. The number of serverless function instances is dynamically adjusted using the available resources of the edge server, with the constraints being: (1) achieving high throughput for video streaming services through serverless pipelines; and (2) adaptively adjusting the workload of dynamic video delivery. To effectively integrate the resources of edge servers and cloud servers, an edge-assisted serverless pipeline optimization adaptive video streaming system was designed. The primary responsibility of the edge server is to invoke the execution of serverless functions and deliver the execution results. Therefore, the serverless pipeline design follows the single responsibility principle: each stateless function instance independently undertakes one task. Serverless functions are instantiated when invoked and destroyed upon completion. In a video streaming session, this means that each video event has a fine-grained bitrate decision function.

[0056] To provide users with a better video viewing experience, cloud service providers need to decompose the monolithic video streaming workflow into a set of consecutive stateless functions to provide fine-grained video services. For example, a video streaming pipeline is decomposed into interconnected stateless functions, executed by a containerized runtime environment. Therefore, the entire adaptive video streaming system is decoupled into a serverless pipeline composed of stateless functions. When the serverless pipeline composition and video requests change, the stateless function operations are not recompiled. For unstable serverless environments, this invention designs an event-driven, multi-process video streaming system. For example, when a video request arrives, a serverless function is invoked to determine the optimal bitrate for the video block to be downloaded, and the stateless function calls in this process are asynchronously repeated until the video streaming session terminates.

[0057] In a video session, the serverless caller module enables the video streaming system to coordinate the execution of all stateless function calls and maintain the serverless pipeline state, ensuring the performance, availability, and responsiveness of serverless functions. The caller uses a signal-based mechanism to control request access to containers. Specifically, whenever a trigger request arrives at the edge server, a serverless function instance is invoked promptly to respond, and detailed call statistics are logged to update the configuration files required by the video service. Furthermore, this invention aims to reduce overall serverless pipeline latency while minimizing costs (resource usage). Therefore, the execution of the overall pipeline is optimized by iteratively and dynamically optimizing the calls to each operation using up-to-date information on video stream request status and resource availability. Simultaneously, different latency relaxation values ​​are set for each function call. By continuously configuring function operations for the serverless pipeline, system latency is dynamically reduced while satisfying the relaxation values ​​at minimal cost. Through dynamic relaxation target allocation, pipeline latency can be reduced without considering possible conditional paths. Finally, the serverless video pipeline information is packaged and sent to the configurator.

[0058] To meet the latency targets of the serverless video pipeline, the configurator module identifies two key factors: how much latency margin to allocate for each stateless function call, and how to efficiently allocate resources to meet the latency targets. Specifically, this invention employs a latency mechanism to improve the performance of the serverless functionality. The variability of stateless function execution and system resources requires the configurator to dynamically determine the most efficient resource allocation for each function call. The configurator continuously monitors available resources (e.g., memory and CPU) on the serverless backend. Furthermore, the configurator module opens a connection to the stateless function database and searches for available serverless functions to select the operation to invoke. After selecting configuration options, the configurator forwards the function call with the configuration decision profile to the scheduler for execution.

[0059] After the configurator module completes the stateless function call configuration file, it dispatches the function calls to the scheduler module for execution. The scheduler executes the serverless function calls on the container environment specified by the configurator. During serverless function execution, stateless operation calls can be executed asynchronously and concurrently. The scheduler tracks the execution time of the function calls. If the configured delay target is exceeded or the scheduler receives an execution error, the scheduler notifies the serverless caller to recreate the function call. This re-invoked call is then passed to the configurator to reset the relaxation allocation and execute the configuration process. Specifically, delay targets are set for different video request calls to improve pipeline responsiveness, thereby ensuring that function calls run with optimal configuration. When a function call executes successfully, the scheduler provides the function output to the cloud server. After processing the input workload, the function instance automatically terminates and releases the occupied resources.

[0060] On the cloud servers, each server stores all video content and encodes video blocks at different bitrate tiers, generating fixed-length video blocks of varying resolutions. Once the cloud server receives the video bitrate determination results from the edge servers via the session controller, it immediately allocates available bandwidth and determines the optimal streaming media delivery strategy. Finally, the cloud server uses the DASH protocol to deliver the appropriate video blocks to the client player on the allocated bandwidth.

[0061] Step 2: Use a near-end strategy to optimize the bitrate adaptive algorithm of PPO and optimize the bitrate decision process during video stream transmission.

[0062] Adaptive bitrate selection for video streams can be viewed as a deep reinforcement learning task: the agent learns by observing the state of the dynamic environment and generates actions based on the neural network, selecting an appropriate bitrate to maximize the expected long-term QoE. For example... Figure 2 The diagram shows the network structure of the near-end strategy optimization (PPO) algorithm for performing video block bit rate decision-making during video stream transmission, as designed in this invention.

[0063] Reinforcement learning is a classic strategy for solving sequential decision-making problems in dynamic environments. Deep neural networks can also handle multi-dimensional input conditions, enabling in-depth feature extraction. Therefore, the adaptive video stream bitrate decision problem falls into this category and can be solved using a deep reinforcement learning-based solution. Specifically, this invention transforms this problem into a Markov Decision Process (MDP) task and designs a deep reinforcement learning-based solution through the interaction between the agent and the environment. Typically, an MDP process consists of a set of finite states S = {S1, S2, ..., S...} t}, a group of actions And a reward function R: S×A→R. In each state S t Take action under ∈S a tAfter ∈A, the agent will obtain a new state S according to the transition probability policy. t+1 and receive a reward R(S) t+1 |S t ,a t The agent's goal is to accumulate rewards. Maximize, where γ∈(0,1] is the discount factor.

[0064] At each time point T, the input to the deep reinforcement learning model includes state information from the video block, player buffer information, and network information, which are sent to the top and bottom streams of the connected architecture, respectively. Since action and state representations provide different kinds of state information for model training and learning, and parameter sharing within the connected architecture is isolated, the three convolutional layers (ConvNets) can adaptively learn feature representations, which are then fused through fully connected layers to produce the desired action decisions. The agent can select actions based on the probability distribution of the output. To train the DRL model, this invention uses stochastic gradient descent (SGD) to update the model parameters. Specifically, the final model is obtained through a self-playing procedure that selects the current model by replaying the experience pool of K best historical versions to sample different training data from a vast state space.

[0065] The Proximal Policy Optimization (PPO) algorithm includes an agent, a state space, an action space, and a reward function.

[0066] Agent: An agent is an entity in the system responsible for executing deep reinforcement learning algorithms and making sequence decisions. In the adaptive video streaming problem, at each time step, the agent is triggered to select an optimal bitrate for the video chunk to be downloaded.

[0067] State Space: In video streaming systems, video information and motion information often have different characteristics. At time step t, the state used as input to the DRL agent is represented as s. t =(x t ,τ t ,n t ,b t ,c t ,l t ,d t ,h t ), where x t τ is the network throughput of the past k video blocks. t It represents the download time of the past k video blocks, and n represents the time interval for throughput measurement. t It is a vector of m available bitrates for the next video block, b t This is the current buffer level, c t It is the number of remaining video blocks in the video, lt It is the bitrate at which the previous video chunk was downloaded, d t It is the latency of the past k video blocks, h t This refers to the content quality weights of video blocks. To obtain efficient and appropriate feature representations and learn the required bitrate decision directly from the state, this invention designs a novel multidimensional feature representation for encoding current and historical video block and action information. Specifically, this invention represents them as two separate three-dimensional tensors, each designed with nine channels, representing agent network information, video block information, player information, and historical decisions, respectively. Each channel is a 4*9 sparse binary matrix, with each position representing the corresponding information, allowing the neural network to learn and extract features empirically. For the actor tensor, since there are typically at most four consecutive actions per round, it is designed with four channels. Each channel is a 1*n... b A sparse binary matrix, where n b These represent different bitrates for video blocks.

[0068] Action Space: For time-series input types (such as past video chunk download times), this invention uses ConvNets convolutional layers to effectively extract basic features. Then, all intermediate processing results are concatenated into a fully connected (FC) layer to learn the intrinsic relationships between complex features. Finally, the output is transformed into a softmax activation layer to compute the probability distribution of the agent's action selection. Upon observing a state s... t At this point, the agent needs to take an action to determine the download bitrate of the next video chunk. Specifically, the action is designed to select different bitrates, such as 360P, 720P, 1080P, etc., and this can be adjusted by modifying the parameters θ of the deep neural network to optimize the policy π. θ (s t ,a t ) can be optimized. Therefore, when downloading video chunks, the agent can select the bitrate based on the strategy learned by the model, thereby maximizing the long-term accumulated QoE reward.

[0069] Reward function: Through the reward function r t The feedback from the algorithm accelerates the optimization of the objective function in Deep Reinforcement Learning (DRL). Specifically, the reward function for each training round is set as the QoE metric. After each round of decision-making, the agent receives a feedback signal from the reward function. Through the accumulation of the reward function, the goal of maximizing QoE can ultimately be achieved. Specifically, this invention is based on existing QoE metrics and assigns weights to each video block according to its inherent video content. Specifically, this invention assigns a specific weight to each video block. This represents the sensitivity to video content quality. Where R is the video block quality, ω is the weight parameter, and n is the number of video blocks during a video session. The DRL model incentivizes downloading video blocks at higher bit rates using a QoE reward function. The QoE metric used in this invention is... Where q(·) represents the benefit function of video quality, and T i This indicates a penalty for video playback pause. The smoothness penalty represents the frequency of rapid bitrate changes in the video. μ and δ are non-negative weighting parameters, set to 4.3 and 1.0, respectively. In short, QoE increases with increasing video block bitrate but decreases with decreasing pause time and smoothness.

[0070] First, generate a batch conversion of samples. t ,a t ,r t ,s t+1 The quadruple > forms the replay buffer experience pool, where s t a represents the state at time t. t r represents the action taken at time t. t Represents state s t The following measures were taken: t The reward received, s t+1 This represents the next state.

[0071] Then, a small batch of samples is taken from the historical buffer pool, and the actor network and critic network are trained asynchronously to update the global network parameters.

[0072] This embodiment presents a specific processing flow for the near-end strategy optimization PPO algorithm, including the following:

[0073] Input: Maximum time steps T, learning rate γ, experience pool B;

[0074] Output: Policy network parameters θ;

[0075] S1: Randomly initialize the actor network and the critic network;

[0076] S2: Initialize parameter values, including reward r and time slice t, and initialize State: S0;

[0077] Here, State represents the network state, player state, and video block state information in the video streaming system, and S0 represents the initial system state; the reward value r is the feedback obtained by the agent performing actions in the environment, and rewards are accumulated in this way. The goal of the model is to maximize the reward function. Therefore, whether an action brings a reward is used as the standard affecting the reward value; ​

[0078] S3: Receive the initial state State S0;

[0079] S4: Select an action for each video block based on the environment:

[0080] S41:for episode←1to T*episode

[0081] S42: for t∈[1:T]

[0082] Where T represents the time period and episode is the number of running cycles;

[0083] S5: Generate a through the actor network t ; where a t Indicates the video stream bitrate decision;

[0084] S6: Select and execute action a t Receive an immediate reward. t and the next state s t+1

[0085] S7: Empirical samples t ,a t ,r t ,s t+1 >Store into the historical experience replay pool to update the environment;

[0086] S8: Randomly select N experience samples from experience pool B;

[0087] S9: Calculate the dominance function based on the value function of the critic network;

[0088] S10: Update the actor network by maximizing the objective function of PPO;

[0089] S11: Determine if the time slice has ended. If not, execute S12.

[0090] S12: Update network parameters θ;

[0091] S13: Determine whether the action selection is appropriate and whether a request has been completed;

[0092] If the action is chosen appropriately and the request is fulfilled, the reward value r is increased.

[0093] If the action is not chosen correctly, the reward value r will be reduced.

[0094] S14: Determine if the time slice has ended;

[0095] ​If the process has not ended, return to S4; if the process has ended, reset time slice t, output r within one time period, and then return to S4.

[0096] To train the bit rate decision algorithm, the set of past states is passed to a convolutional layer with 128 filters, each with a size of 4 and a stride of 1. The results of these layers are then aggregated with other inputs through a fully connected layer into a hidden layer. The actor network uses the same neural network structure, and its final output is a linear neuron. During training, the discount factor γ is set to 0.99, the learning rates for the actor and critic networks are 0.00001 and 0.0001, respectively, the entropy factor σ is set to decay from 1 to 0.1 over 100 iterations, the experience pool size is 50000, and all hyperparameters are kept constant throughout the training process.

[0097] Step 3: Use a three-end pruning mechanism to improve the performance of the near-end policy optimization algorithm and accelerate the convergence speed of the deep reinforcement learning algorithm.

[0098] In the context of multidimensional feature representation, the core factor in policy gradient training is a deep learning model with a suitable loss function.

[0099] Specifically, this invention employs an actor-critic framework for training a predetermined policy, achieving policy improvement through gradient-based parameter updates. The agent directly learns the parameterized policy π based on the expected reward gradient of the relevant policy parameters θ. θ The actor-critic framework trains the value function V. θ (s t ) and strategy π θ (a t |s t This invention defines a clipping ratio function, and updates the training model parameters by repeatedly sampling from a pool of replayed historical experiences. For the current strategy π θ And the old strategy π θ′ The probability ratio between them. Advantage function. Describes the relationship between two consecutive states s t+1 ,s t According to the current strategy π θ Select action a t The improvement, and the policy loss function is defined as Ensure r t (θ) lies in the interval (1-∈, 1+∈), where ∈ can be set to 0.2. The loss of the value function is defined as... in This represents traditional rewards. A minimal function is used to control the original and truncation items, preventing policy updates from drifting beyond a predetermined range.

[0100] To improve the performance of the PPO algorithm, this invention applies an optimization method based on advantage function estimation and additional entropy reward.

[0101] Specifically, using generalized advantage estimation (GAE) to construct the advantage function can reduce variance and avoid large fluctuations in the algorithm. This is achieved by following the T-step update method to correct the TD error δ. t The advantage function is defined as Where δ t =R t +ωV θ (s t+1 )-V θ (s t ), where ω and η represent weighting parameters, and R t δ represents the reward at time t; T-1 V represents the TD error; θ The value function is used to calculate the reward obtained after performing an action based on the state at time T. However, the above PPO loss function is difficult to converge in large-scale distributed training. This invention analyzes two main reasons for this problem: (1) When π θ (a t |s t >>π θ (a t |s t or advantage function At that time, the policy loss function (2) Unbounded variance is introduced; (3) Due to the uncertainty of bit rate distribution in the video stream, value loss Often very large. To accelerate and stabilize the model training process, this invention designs a three-terminal pruning PPO loss function. It introduces an additional pruning hyperparameter σ1 into the policy loss, which is... The value loss at time introduces two clipping hyperparameters, σ2 and σ3. The three-terminal clipping function of the policy loss function is defined as follows: Where σ1>1+∈ denotes the upper bound of the loss function. Further, the loss function of the value function is defined as...

[0102] During DRL model training, hyperparameters σ2 and σ3 represent the total number of video blocks downloaded by the player and stored on the server, respectively. Therefore, these two hyperparameters do not require manual adjustment but are dynamically calculated based on the segments played during video playback. This invention implements a distributed version of the three-terminal editing PPO algorithm, which can accelerate model training and improve training performance. This strict constraint significantly reduces the variance of the value function and eliminates the influence of policy irrationality.

[0103] Instance verification

[0104] To further verify the beneficial effects of this invention, a prototype system for adaptive video streaming based on serverless computing was constructed, and the embodiments were evaluated and verified. This invention used a Dell workstation, an NVIDIA Jetson Agx Xavier, and a Macbook Pro as the cloud server, edge device, and client, respectively. Based on dash.js, an open-source DASH player, this invention implemented a video streaming player. All experiments used YouTube videos, as it is the reference video type for dash.js. Specifically, an HTTP / 2-based Nginx server was deployed to provide video chunks, and the Linux TC tool was used on the cloud server to control outgoing traffic. Simultaneously, a Docker container was used to deploy the serverless service to the edge device. Through experimental verification of the serverless computing-based adaptive video streaming algorithm and comparison with four existing algorithms, the following results were obtained. Figure 3 , Figure 4 and Figure 5 The verification results.

[0105] First, to test the QoE metrics of this method (EAVS) under different network data, this invention carefully studied the meta-metrics implemented for each video player under different network conditions. Specifically, in Figure 3 The paper demonstrates the performance of the ABR algorithm under different network trajectories. Clearly, the 5G network trajectory offers higher throughput than the 4G network trajectory. Therefore, as... Figure 3 As shown, in the 5G network dataset, the average video block bitrate is 53.8% higher than that of 4G. The proposed algorithm EAVS performs best, improving the video block bitrate by approximately 9.1%-42.7% compared to other benchmark video streaming algorithms. Through fine-grained video block bitrate adaptation decisions, EAVS can consciously adapt to dynamic network conditions, achieving higher video bitrates and shorter latency.

[0106] Subsequently, to further verify the performance of the present invention, video stream transmission performance under different video block lengths was tested. Compared with other methods, such as... Figure 4As shown, this invention achieves the maximum average QoE reward, demonstrating the good performance of the proposed method. Furthermore, based on the average QoE, this method is the only ABR algorithm to ultimately achieve the maximum average QoE reward. This indicates that the proposed method can adapt more quickly to the underlying dynamic network during video transmission to improve user QoE because it enhances the accuracy of video block bitrate decisions by enabling fine-grained serverless services.

[0107] Finally, an ablation experiment was designed to test the effectiveness of this method with and without serverless video streaming pipeline support. Figure 5 This paper demonstrates the comprehensive performance of a serverless video streaming pipeline on a 5G network dataset. By employing an edge-assisted serverless video pipeline, compared to solutions without a serverless pipeline, this method significantly reduces the average response latency of video streaming services, by up to 27.4%-60.2%. This indicates that serverless configurations perform well in video bitrate adaptation, and this invention can select a suitable stateless function to respond to video request decisions for video delivery workloads. Therefore, the method of this invention uses an edge-assisted serverless computing-based video streaming transmission system, which can efficiently transmit video streams, and the effective bitrate decision algorithm ensures the high performance of this invention.

[0108] In summary, the method of this invention is suitable for dynamic and complex video streaming scenarios under edge computing, and can meet the time-varying video streaming requirements. Furthermore, this invention basically satisfies the needs of video transmission tasks with high end-to-end latency requirements.

[0109] The specific examples described above are further explanations of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications and equivalent substitutions made within the principles and spirit of the present invention should be within the scope of protection of the present invention. Therefore, the scope of protection disclosed in the present invention should be determined by the scope of the claims.

Claims

1. An adaptive video streaming method based on serverless computing, characterized in that, Includes the following steps: Step 1: Build an adaptive video streaming system based on serverless computing under edge computing, including a video streaming server, an edge terminal, and a video user terminal; Among them, the video streaming server is responsible for storing video content and dividing it into fixed-length video blocks encoded at different bit rates, and transmitting the video content to the video user through a complex and ever-changing dynamic network; The edge device implements highly parallel stateless functions based on a serverless architecture to respond to video content requests from multiple users and interact with geographically distributed servers to deliver video content from the server closest to the user. The client is used to send local player status information and video block request information to the edge, triggering a stateless function response in the form of an event to generate the optimal video block bitrate decision. Step 2: Optimize the adaptive bitrate decision for video blocks using a bitrate adaptation algorithm based on deep reinforcement learning-based proximal policy optimization, including the following steps: Step 2.1: An end-to-end deep reinforcement learning framework is used to perform bitrate adaptive decision-making during video stream playback; Step 2.2: State-space representation; We design multidimensional features to encode current and historical video block state information. In video streaming services, information related to video transmission is represented by different feature dimensions. We represent them as discrete three-dimensional tensors, allowing deep neural networks to learn system models based on historical experience. The state space tensor is designed with nine channels, representing agent network information, video block information, player information and historical decision information respectively. Each channel is a sparse binary matrix, and each position represents the corresponding state space information. Step 2.3: Action space representation; After state-space representation, the decision of video block bitrate is designed as different video sharpness actions. For video block playback sequence input type, convolutional layers are used to extract basic features. Then, all the processing results are concatenated into a fully connected layer to learn the intrinsic relationships between complex features; Finally, the output is transformed into a normalization function layer to calculate the probability distribution of the action; Step 2.4: Setting the reward function; Design in a certain state Next action award Once the video streaming system begins to make bitrate decisions, the agent will take different actions based on the current state, forming different state-action sequences in the historical experience buffer pool, thereby maximizing the cumulative reward function. The user viewing experience (QoE) metric is used as the reward function, and a specific weight is assigned to each video block to assess the sensitivity of the encoded content quality. Step 2.5: Training the deep reinforcement learning model; Under multidimensional input feature representation, the core factor in policy gradient training is finding a deep learning paradigm with a suitable loss function; An actor critic framework with online policy training is employed, where agents base their decisions on relevant policy parameters. Directly learning parameterized policies based on expected return gradients Actor's Critical Paradigm Training Value Function and policies And update them by repeatedly sampling from the pool of replayed historical experiences; Step 3: Use a three-end pruning mechanism to improve the performance of near-end policy optimization and accelerate the convergence speed of deep reinforcement learning algorithms; Based on the deep reinforcement learning actor critic model learned in step 2, the policy is parameterized using the proximal policy optimization algorithm. During training, the parameters are updated by optimizing the gradient to achieve the optimal objective function. The correction coefficient of the approximate policy optimization (PPO) algorithm is set to... For the current strategy and old strategies The probability ratio; In the near-end policy optimization algorithm, a three-end pruning mechanism is used to weaken the gradients of some outliers, accelerating the algorithm's convergence; In the training step of the algorithm iteration, a three-end pruning method is used to reduce the probability ratio; That is, the ratio of the new and old strategies is limited to a region, and the update step is limited by controlling the size of the region; the Clip function is used for clipping, where... Correction coefficient Constraints in scope Inside, This is a hyperparameter; when the offset of the policy update exceeds the predetermined range, the truncation term will act as a limit. When updating the gradient of the objective function, each gradient calculation uses the latest policy model to perform interactive sampling to obtain the corresponding sample sequence, thereby performing gradient calculation. After the model is updated, the experience gained from the previously used samples will be utilized through the old strategy distribution of the corrected coefficients; In step 3, the TD error is updated by following the T-step update method. Advantage function ,in , , All represent weight parameters. express Momentary rewards; Indicates TD error; This represents the value function, used to calculate the reward that can be obtained after performing an action based on the state at time T; To accelerate and stabilize the model training process, a three-terminal pruning PPO loss function is designed, which introduces an additional pruning hyperparameter into the policy loss. ,for The time-based value loss introduces two clipping hyperparameters. and The three-terminal clipping function of the strategy loss function is defined as follows: ,in The loss function represents the upper bound of the loss function, and the loss function of the value function is defined as follows: During DRL model training, hyperparameters and These represent the total number of video blocks downloaded by the player and stored on the server, respectively. These two hyperparameters do not require manual adjustment; instead, they are dynamically calculated based on the segments played during video playback.

2. The adaptive video streaming method based on serverless computing as described in claim 1, characterized in that, The adaptive video streaming system described in step 1 involves extracting JavaScript console logs from a browser with an integrated dash.js player on the video streaming client, and using the iPerf tool on the client to extract video block status information, player buffer, and network conditions. Then, the ABR (Adaptive Bitrate) controller organizes the status metadata information and groups it for forwarding to invoke video block download request events. The corresponding video request calls are sent to the edge server via serverless functions using the HTTP protocol. Then, a serverless computing-based video stream decision pipeline is built on the edge server to make the optimal video stream bitrate decision based on the current information; On the edge server, the stateless function database is the intelligent core of the entire video streaming system, interacting with containers that execute serverless functions; In the edge server, the throughput monitoring module probes the network channel link between the client and the cloud server to estimate future network throughput; In response to a client's video request event, the edge server immediately decrypts the request information; The edge server uses information from client request events and throughput predictions to determine the optimal video block bitrate using serverless functions; Then, the edge server passes the video block bitrate decision result to the nearest cloud server; The serverless video streaming pipeline has an additional stateless streaming ingestion function that coordinates the execution of the video streaming workflow. At each time interval, the serverless function configurator calls a stateless function to process video streaming events and provide feedback on the video streaming pipeline information. The edge server instructs the caller to execute a stateless function to make fine-grained video stream bitrate decisions; The serverless pipeline design follows the single responsibility principle: each stateless function instance independently undertakes a task; Serverless functions are instantiated when they are called and destroyed when they are completed. Cloud service providers break down monolithic video streaming workflows into a set of continuous stateless functions to provide fine-grained video services. The entire adaptive video streaming system is decoupled into a serverless pipeline composed of stateless functions; When the serverless pipeline composition and video requests change, stateless function operations will not be recompiled; For unstable serverless environments, a multi-process video streaming system that supports event-driven operation; In a video session, the serverless caller module enables the video streaming system to coordinate the execution of all stateless function calls and maintain the state of the serverless pipeline, ensuring the performance, availability, and responsiveness of the serverless functions. The caller uses a signal-based mechanism to control request access to the container. Whenever a request is triggered and reaches the edge server, the serverless function instance is invoked in a timely manner to respond, and detailed call statistics are recorded to update the configuration files required by the video service. The overall pipeline execution is optimized by iteratively and dynamically optimizing each operation call using the latest information on video stream request status and resource availability. At the same time, different latency relaxation values ​​are set for each function call. By continuously configuring functional operations for the serverless pipeline, the system latency is dynamically reduced while satisfying the relaxation values ​​at the lowest cost. Finally, the serverless video pipeline information is packaged and sent to the configurator. To meet the latency targets of the serverless video pipeline, the configurator module determines two key factors: how much latency margin to set for each stateless function call, and how to effectively allocate resources to meet the latency targets. A latency mechanism is used to improve the performance of the serverless functionality. The volatility of stateless function execution and system resources requires the configurator to dynamically determine the most efficient resource allocation for each function call. The configurator continuously monitors available resources on the serverless backend. Furthermore, the configurator module opens a connection to the stateless function database and searches for available serverless functions to select the operation to call. After selecting configuration options, the configurator forwards the function call with the configuration decision profile to the scheduler for execution. After the configurator module completes the stateless function call configuration file, it dispatches the function calls to the scheduler module for execution. The scheduler executes the serverless function calls on the container environment specified by the configurator. During the execution of the serverless function calls, stateless operation calls can be executed asynchronously and concurrently. The scheduler tracks the execution time of the function calls. If the configured delay target is exceeded or the scheduler receives an execution error, the scheduler notifies the serverless caller to recreate the function call. Then, this repeated call is passed to the configurator to reset the relaxation allocation and execute the configuration process. Delay targets are set for different video request calls to improve pipeline responsiveness and ensure that function calls run with optimal configuration. When the function call executes successfully, the scheduler provides the function output to the cloud server. After processing the input workload, the function instance automatically terminates and releases the occupied resources. On cloud servers, each cloud server stores all video content and encodes video blocks at different bitrate steps to generate video blocks of different resolutions with fixed durations. Once the cloud server receives the video bitrate determination result from the edge server through the session controller, it immediately allocates available bandwidth and determines the optimal streaming media push strategy; finally, the cloud server uses the DASH protocol to provide appropriate video chunks to the client player on the allocated bandwidth.

3. The adaptive video streaming method based on serverless computing as described in claim 1, characterized in that, In step 2, the Proximal Policy Optimization (PPO) algorithm includes an agent, a state space, an action space, and a reward function; Agent: An agent is an entity in the system responsible for executing deep reinforcement learning algorithms and making sequence decisions; in the adaptive transmission problem of video streaming, at each time step, the agent is triggered to select an optimal bitrate for the video chunk to be downloaded; State space: in time steps The state used as input to the DRL agent is represented as , It's the past Network throughput per video block It's the past The download time of each video block represents the time interval for throughput measurement. It is the next video segment. A vector of available bit rates, This is the current buffer level. It represents the number of remaining video blocks in the video. It is the bitrate at which the previous video block was downloaded. It's the past The latency of each video block, It is the content quality weight of the video block; Design a multidimensional feature representation to encode current and historical video chunks and action information, representing them as two separate three-dimensional tensors, each with nine channels representing agent network information, video chunk information, player information, and historical decisions, respectively; each channel is a... A sparse binary matrix, where each position represents corresponding information, allows the neural network to learn and extract features based on experience. For the actor tensor, it is designed as four channels; each channel is a sparse binary matrix, These represent different bitrates of video blocks; Action Space: For time-series input types, ConvNets convolutional layers are used to extract basic features. All intermediate processing results are then concatenated into a fully connected layer. Finally, the output is transformed into a softmax activation layer to calculate the probability distribution of the agent's action selection. Upon observing a state... At this point, the agent needs to take an action to determine the download bitrate of the next video chunk. The action is designed to select different bitrates by adjusting the parameters of the deep neural network. strategy Optimization is performed; when downloading video chunks, the agent selects the bitrate based on a strategy learned by the model; Reward function: through rewards The feedback accelerates the optimization of the objective function in Deep Reinforcement Learning (DRL); the reward function for each training round is set as the QoE metric; after each round of decision-making, the agent receives a feedback signal from the reward function, and the goal of maximizing QoE is achieved through the accumulation and summation of the reward function; a specific weight is assigned to each video block. This represents the sensitivity to video content quality, among which... For video block quality, For weight parameters, The number of video blocks during a video session; the DRL model incentivizes downloading video blocks at a higher bit rate by using the QoE reward function; QoE indicator is ,in, The benefit function representing video quality. This indicates a penalty for video playback pause. This represents the smoothness penalty when the video rapidly switches bitrates. and It is a non-negative weight; QoE increases with the increase of video block bit rate, but decreases with the decrease of pause time and smoothness; First, generate a batch conversion of samples. The quaternion forms a replay buffer experience pool. represent The state at any given moment, represent Actions taken at all times Representing state The following measures The rewards received Represents the next state; Then, a small batch of samples is taken from the historical buffer pool, and the actor network and critic network are trained asynchronously to update the global network parameters.

4. The adaptive video streaming method based on serverless computing as described in claim 3, characterized in that, The processing flow of the near-end policy optimization PPO algorithm includes the following steps: Input: Maximum time steps T, learning rate Experience Pool B; Output: Policy network parameters θ; S1: Randomly initialize the actor network and the critic network; S2: Initialize parameter values, including reward r and time slice t, and initialize... Here, State represents the network status, player status, and video block status information in the video streaming system. This represents the initial system state; the reward value r is the feedback obtained by the agent performing actions in the environment, and the reward is accumulated in this way; the goal of the model is to maximize the reward function, and whether an action can bring a reward is used as the standard affecting the reward value. S3: Receive initial state ; S4: Select an action for each video block based on the environment: S41: S42: Where T represents the time period. It is the number of operating cycles; S5: Generated via actor network ;in, Indicates the video stream bitrate decision; S6: Select and execute the action Receive immediate reward and the next state ; S7: Empirical samples Store in the historical experience replay pool to update the environment; S8: Randomly select N experience samples from experience pool B; S9: Calculate the dominance function based on the value function of the critic network; S10: Update the actor network by maximizing the objective function of PPO; S11: Determine if the time slice has ended. If not, execute S12. S12: Update network parameters θ; S13: Determine whether the action selection is appropriate and whether a request has been completed; If the action is chosen appropriately and the request is fulfilled, the reward value r is increased. If the action is not chosen correctly, the reward value r will be reduced. S14: Determine if the time slice has ended; If the process is not yet complete, return to S4; if it is complete, reset the time slice. It outputs r within a time period and then returns S4.

Images