An expert-guided video streaming bitrate adaptation method and system

Abstract

The application relates to the technical field of video streaming, and relates to a video streaming bit rate self-adaptive method and system based on expert guidance. The method comprises the following steps: acquiring a hybrid throughput trajectory data set, training an initial model according to the data set; clustering the data set into subsets, training an expert model on each subset, recording state-action decision pairs of each expert model and storing the state-action decision pairs into an expert experience pool; generating a virtual video session, collecting state-action-reward triplets of each video segment in a transmission process; calculating cumulative rewards and advantage values, combining the state-action-reward triplets, the cumulative rewards, the advantage values and the state-action decision pairs of the expert model to form training tuples; sampling the training tuples to synchronously optimize an Actor network and a Critic network, and acquiring a general ABR model. The application solves the catastrophic forgetting problem of the ABR algorithm through a multi-expert guidance method, and improves the generalization ability and self-adaptive performance of the model.

Description

Technical Field

[0001] This application relates to the field of video streaming technology, and specifically to an expert-guided video streaming bitrate adaptive method and system. Background Technology

[0002] With the rapid development of the internet and multimedia processing technologies, video streaming has become a core carrier of information dissemination and entertainment consumption. Current mainstream video streaming technologies (such as HTTP-based dynamic adaptive streaming media DASH) pre-encode video content into multiple versions (different bitrates and resolutions) of video blocks (typically 2-6 seconds per block), such as... Figure 15 As shown, the client dynamically downloads the adapted version based on real-time network conditions, and combines this with a playback buffer to achieve "downloading while playing," effectively reducing initial loading latency and improving playback continuity. However, the dynamic nature of the network environment (such as bandwidth fluctuations, signal strength changes, and cross-network switching) poses a severe challenge to fixed bitrate transmission schemes: insufficient bandwidth causes stuttering, while ample bandwidth cannot fully utilize high-definition image resources.

[0003] To address the aforementioned problems, the Adaptive Bit Rate Algorithm (ABR algorithm) was developed, such as... Figure 16 As shown, it dynamically selects the optimal video block bitrate version by analyzing parameters such as network bandwidth estimation, buffer status, and historical download data in real time, maximizing the quality of user experience (QoE) while ensuring smooth playback.

[0004] Existing ABR algorithms are mainly divided into two categories:

[0005] Heuristic methods (such as BOLA and MPC) adjust the bitrate based on preset rules. Although they are simple to implement, they rely on human experience and are difficult to adapt to complex and ever-changing network environments, resulting in suboptimal performance.

[0006] Machine learning methods (such as Pensieve and Oboe): These methods learn environmental features through deep neural networks and can be customized to adapt to specific scenarios, and have become the mainstream solution.

[0007] However, existing machine learning ABR algorithms suffer from a fundamental flaw—catastrophic forgetting: when the model training environment changes, the algorithm forgets previously learned key network features (especially low-frequency or early features), leading to significant performance degradation in certain subsets of environments (such as specific congestion patterns or mobile switching scenarios) (see...). Figure 17 The root cause lies in the fact that existing solutions rely on a single general-purpose model to adapt to all environments. Although they have basic generalization capabilities, they cannot effectively retain key historical features during continuous training, which limits the stability and generalization performance of the model in dynamic networks.

[0008] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0009] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0010] This disclosure provides a video streaming bitrate adaptation method and system based on expert guidance. By using multiple expert ABR models trained in different environments to jointly guide a general model, the general model's ability to learn key features of various network environments is enhanced, thereby significantly alleviating or eliminating the catastrophic forgetting problem and improving the model's adaptive performance under diverse network conditions.

[0011] In some embodiments, the method includes:

[0012] Obtain a mixed throughput trajectory dataset, initialize the neural network, and train the initial model on the dataset using deep reinforcement learning;

[0013] The mixed throughput trajectory dataset is clustered into multiple network subsets. The initial model is used as the basis for pre-training expert models. Expert models are trained on each subset, and the state-action decision pairs of each expert model are recorded and stored in the expert experience pool.

[0014] Virtual video sessions are generated based on a mixed throughput trajectory dataset, and state-action-reward triples for each video segment during transmission are collected.

[0015] The cumulative reward and advantage value are calculated based on the state-action-reward triplet, and the state-action decision pairs of the expert models corresponding to the network subset of the virtual video session are extracted from the expert experience pool. The state-action-reward triplet, cumulative reward, advantage value and the state-action decision pairs of the expert models are combined to form the training tuple.

[0016] A batch of training tuples is randomly sampled as training samples to simultaneously optimize the Actor network and Critic network, thereby obtaining a general ABR model.

[0017] Preferably, the clustering of the hybrid throughput trajectory dataset includes:

[0018] Unsupervised clustering is performed based on network features, including average throughput and network volatility indicators.

[0019] Each network subset represents a type of network environment.

[0020] Preferably, the specific methods for recording the state-action decision pairs of each expert model and storing them in the expert experience pool include:

[0021] Train expert models on each subset, test each expert model in its respective network subset, record its decision strategy as a state-action decision pair, i.e., the environment state and the corresponding bit rate selection, and store it in the expert experience pool.

[0022] Preferably, the cumulative reward calculation formula is as follows:

[0023] ,

[0024] in, Indicates the first The cumulative rewards for each segment, Indicates the first Rewards for each segment, Indicates the discount factor. This indicates the total number of segments in a video session;

[0025] The formula for calculating the dominance value is as follows:

[0026] ,

[0027] in, Indicates the dominance value. Indicates input status The output of the Critic network is the expected reward.

[0028] Preferably, the Actor network and the Critic network are jointly optimized using gradient descent;

[0029] The objective function of the Critic network is:

[0030] ,

[0031] in, This represents the cumulative reward in the m-th tuple. This represents the input environment state in the m-th tuple. This indicates that by adjusting the model's parameters Minimize the subsequent objective function. Represents all tuples;

[0032] The objective function of the Actor network is:

[0033] ,

[0034] in, For weight parameters, This represents the maximum advantage value for the current batch. Indicates the state Choose an action in Actor The probability, This represents the action in the m-th tuple. This represents the dominance value in the m-th tuple. This represents the state of the expert model in the m-th tuple. Let m represent the decision made by the expert model in the m-th tuple. This indicates that by adjusting the model's parameters Maximize the subsequent objective function.

[0035] Preferably, the ABR model adopts the Actor model and receives six-dimensional environmental states: the download throughput of the last N video segments, the download time of the last N video segments, the optional size of the next video block, the bitrate decision of the last video block, the current buffer occupancy rate, and the number of remaining video segments in the current streaming session.

[0036] Preferably, the specific structure of the Actor model is as follows:

[0037] The first three input states are processed by separate one-dimensional convolutional layers, each with 128 filters of size 4; the last three input states are processed by fully connected layers, each containing 128 neurons and employing the ReLU activation function; the features extracted from all input layers are concatenated and processed through a shared hidden fully connected layer; the output layer applies the SoftMax activation function to generate a probability distribution with a usable bit rate.

[0038] In some embodiments, the system includes:

[0039] Video server: Stores multi-bitrate version video blocks;

[0040] ABR Server: Deploys the general ABR model trained using the expert-guided video streaming bitrate adaptive method, and responds to client decision requests;

[0041] Client: Collects network status in real time and sends it to the ABR server, and decides to download video blocks based on the returned bitrate;

[0042] Communication module: Used for data transmission between the client and the ABR server and video server.

[0043] In some embodiments, the apparatus includes a processor and a memory storing program instructions, the processor being configured to execute the expert-guided video streaming bitrate adaptive method when the program instructions are executed.

[0044] In some embodiments, the storage medium stores a computer program that, when executed by a processor, implements the expert-guided video streaming bitrate adaptive method.

[0045] The present disclosure provides an expert-guided video streaming bitrate adaptive method and system, which can achieve the following technical effects:

[0046] This invention employs a multi-expert guidance method, utilizing multiple expert ABR models trained in different network environments to jointly guide a general model. This significantly enhances the general model's learning ability for key features in various network environments, effectively mitigating or eliminating the catastrophic forgetting problem. The expert-guided general model significantly improves its adaptive performance under diverse network conditions by incorporating the decision-making experience of multiple expert models. By pre-training the initial model and utilizing the expert experience pool for joint guidance, the training requirements for new environments or viewing preferences are significantly reduced.

[0047] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0048] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein:

[0049] Figure 1 This is a diagram of the overall learning framework of the general ABR model proposed in this invention.

[0050] Figure 2 This is a flowchart of the expert learning phase.

[0051] Figure 3 This is a flowchart of the general model learning phase.

[0052] Figure 4 These are the normalized evaluation results of the general ABR model proposed in this invention in different throughput network subsets.

[0053] Figure 5 The results are the bit rate evaluation results of the general ABR model proposed in this invention in different throughput network subsets.

[0054] Figure 6 These are the stuttering time evaluation results of the general ABR model proposed in this invention in different throughput network subsets.

[0055] Figure 7 This is the normalized evaluation result of the general ABR model proposed in this invention under different network volatility levels.

[0056] Figure 8 This is the bit rate evaluation result of the general ABR model proposed in this invention under different network fluctuation levels.

[0057] Figure 9 This is the evaluation result of the stuttering time of the general ABR model proposed in this invention under different network fluctuation levels.

[0058] Figure 10 This is the evaluation result of the general ABR model proposed in this invention in the application of video-on-demand streaming.

[0059] Figure 11 This is the evaluation result of the general ABR model proposed in this invention in the application of live video streaming.

[0060] Figure 12 This is the evaluation result of the general ABR model proposed in this invention in the application of video recognition video stream.

[0061] Figure 13 This is a practical application framework diagram of the general ABR model proposed in this invention.

[0062] Figure 14 This is a schematic diagram of the device structure according to an embodiment of the present disclosure.

[0063] Figure 15 This is a schematic diagram of the video stream framework.

[0064] Figure 16 This is a diagram illustrating the ABR decision process.

[0065] Figure 17 This is a simplified diagram of the problem of catastrophic forgetting. Detailed Implementation

[0066] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0067] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0068] Unless otherwise stated, the term "multiple" means two or more.

[0069] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0070] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0071] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0072] Example 1

[0073] like Figure 1 As shown, an expert-guided video streaming bitrate adaptive method comprises three stages: an initial learning stage, an expert learning stage, and a general model learning stage.

[0074] Specifically, including:

[0075] Initial learning stage

[0076] S1: Obtain the mixed throughput trajectory dataset, initialize the neural network, and train the initial model on the dataset through deep reinforcement learning;

[0077] Expert learning stage

[0078] S2: Cluster the mixed throughput trajectory dataset into multiple network subsets, use the initial model as the basis for expert model pre-training, train expert models on each subset, record the state-action decision pairs of each expert model and store them in the expert experience pool.

[0079] General model learning phase

[0080] S3: Generate virtual video sessions based on mixed throughput trajectory datasets, and collect state-action-reward triples for each video segment during transmission;

[0081] S4: Calculate the cumulative reward and advantage value based on the state-action-reward triplet, and extract the state-action decision pairs of the expert models of the network subset corresponding to the virtual video session from the expert experience pool. Combine the state-action-reward triplet, cumulative reward, advantage value, and the state-action decision pairs of the expert models to form a training tuple.

[0082] S5: Randomly sample a batch of training tuples as training samples to simultaneously optimize the Actor network and Critic network, and obtain a general ABR model.

[0083] Here, the state is the environment state, the action is the corresponding bit rate selection, and the reward is QoE performance.

[0084] As a refinement of the above embodiments, the purpose of the initial learning phase is to train a basic ABR model in diverse streaming media environments, providing pre-trained initial weights for subsequent phases and accelerating the training process. The neural network contains randomized neuron weights during initialization and receives training data covering all dimensions of streaming media diversity. It is trained using Deep Reinforcement Learning (DRL), where the model makes bitrate decisions based on environmental state inputs (e.g., past throughput measurements, current buffer occupancy, etc.), and the resulting reward (i.e., QoE performance) is used to calculate gradients, thereby updating the model's neuron weights.

[0085] Specifically, it includes:

[0086] S101: Obtain the mixed throughput trajectory dataset. Extract data from the throughput trajectory dataset containing mixed network conditions, denoted as... These data cover a variety of network environments and are used to simulate diverse network bandwidth conditions.

[0087] S102: Initialize the neural network The neuron weights are random values ​​to ensure that the model learns from scratch.

[0088] S103: Conduct deep reinforcement learning training. Using standard deep reinforcement learning methods, in the collected network environment... Chinese neural network Conduct training:

[0089] ,

[0090] Among them, the function This represents a standard deep reinforcement learning training process, such as A3C, PPO, etc.

[0091] S104: After training is complete, obtain and store the initial model. This serves as the starting point for pre-training in subsequent stages (expert learning and general model learning).

[0092] It should be noted that in this embodiment, only network conditions are considered as streaming media environmental factors that lead to catastrophic forgetting.

[0093] As a refinement of the above embodiments, the expert learning phase trains multiple expert models in different types of network environments, extracts their optimal decision-making experience, and stores it in an experience pool to provide guidance for generalization model training.

[0094] like Figure 2As shown, the specific steps of the expert learning phase are as follows:

[0095] S201: Mixed Throughput Trajectory Dataset Perform unsupervised clustering (such as K-means) to divide the network into K subsets, denoted as . :

[0096] ,

[0097] Each subset Representing a specific type of network condition, the above clustering process is based on network features. (For example, average throughput, network fluctuations).

[0098] S202: Train an expert model in each subset. For each network subset... Using the initial model As weights in the pre-trained model, the corresponding expert model is trained through deep reinforcement learning. :

[0099] ;

[0100] S203: Test and record the behavioral strategies of the expert models. After training is complete, for each expert model... In their respective network subsets The test was conducted, and its decision-making strategy was recorded as state-action pairs. That is, the selection of environmental conditions and corresponding bit rates.

[0101] S204: Store expert experience pool Y. Store the state-action pairs of all expert models into the experience pool Y. This experience pool will serve as a source of expert knowledge to guide subsequent general model learning.

[0102] As a refinement of the above embodiments, the general model learning phase utilizes the expert experience pool Y obtained in the second phase, combined with the hybrid throughput trajectory dataset. , the initial model The model is trained to be a general ABR decision model with generalization ability. It is still trained using deep reinforcement learning, but expert knowledge is introduced in addition.

[0103] The training process at this stage can be formally represented as:

[0104] ,

[0105] in, This refers to the pool of expert decision-making experience generated in the second phase. It is a general model ultimately used for adaptive bit rate (ABR) decisions. This indicates the training process of the general model in the third stage.

[0106] Specifically, the initial model It contains two neural networks: the Actor network responsible for ABR decision-making (denoted as...). ) and the Critic network that assists in the training process (denoted as ).

[0107] like Figure 3 As shown, specifically, step S3 includes: for each streaming session h, from the mixed throughput trajectory dataset A network trajectory is randomly selected to perform virtual streaming media transmission, generating a complete video session h. A typical streaming media session consists of a series of video clips (e.g., a 100-second video session contains 50 two-second video clips). Before transmitting each clip, the environmental state (e.g., past throughput measurements, current buffer levels, etc.) is input into the Actor network to obtain action decisions, i.e., the selected bit rate. Then, the corresponding network conditions are simulated based on the throughput trajectory data, and the clip is transmitted at the selected bit rate. After the clip transmission is completed, the reward obtained is calculated based on the QoE function, i.e., the QoE performance. All clips in session h are transmitted sequentially in this manner. The state, action, and reward of each video clip during transmission are recorded to form a triple. .

[0108] Step S4 includes:

[0109] S401: Based on the collected triples The Computing(.) function is used to generate a cumulative reward (a weighted sum of rewards from the current segment to the last segment of the streaming session) and an advantage value for each video segment. :

[0110] ,

[0111] in, Indicates the first The cumulative rewards for each segment, Indicates the first Rewards for each segment, This indicates the total number of segments in a video session. This represents a discount factor (its value is less than 1.0), used to control the exponential decay of future rewards in cumulative reward calculations. The principle is: a smaller discount factor... It will give higher weight to instant rewards, and The larger the value, the greater the impact of future rewards.

[0112] The formula for calculating the dominance value is as follows:

[0113] ,

[0114] in, Indicates the dominance value. Indicates input status The output of the Critic network is the expected reward. The advantage value of a video segment quantifies the effectiveness of the action taken in that step. For example, if... A value greater than 0 indicates that the actual reward from the chosen action is higher than the expected reward, reflecting performance exceeding expectations; conversely, if... A negative result indicates that the performance was below expectations.

[0115] S402: Introduce expert experience and construct training samples. Extract expert state-action pairs corresponding to session h from the experience pool Y. With the data of this session Together they form a tuple and are stored in cache B:

[0116] .

[0117] Step S5 includes:

[0118] Sample mixed-environment batches and update the neural network. After every few streaming sessions, randomly sample multiple tuples from buffer B to form mixed training batches across environmental conditions. Update the Critic network and Actor network;

[0119] ,

[0120] in, This indicates the total number of tuples in the batch; the throughput trajectory of each streaming session is randomly selected, so multiple sessions together cover diverse network conditions, thus the tuples sampled in each batch come from a mixture of different network environments.

[0121] The Critic network aims to adjust the weights of its neurons. To improve the accuracy of reward prediction and ensure that its output (i.e., the predicted value) approximates the actual value (i.e., the cumulative reward) as closely as possible, the objective function of the Critic network is:

[0122] The objective function of the Critic network is:

[0123] ,

[0124] in, This represents the cumulative reward in the m-th tuple. This represents the input environment state in the m-th tuple. This indicates that by adjusting the model's parameters Minimize the subsequent objective function. This represents all tuples.

[0125] The Actor network incorporates expert experience from the m-th tuple. (i.e., state-action pairs), their neuron weights It is adjusted to maximize the following objective function:

[0126] The objective function of the Actor network is:

[0127] ,

[0128] in, For weight parameters, This represents the maximum advantage value for the current batch. Indicates the state Choose an action in Actor The probability, This represents the action in the m-th tuple. This represents the dominance value in the m-th tuple. This represents the state of the expert model in the m-th tuple. Let m represent the decision made by the expert model in the m-th tuple. This indicates that by adjusting the model's parameters Maximize the subsequent objective function.

[0129] The Actor network combines current network video streaming data with expert experience to optimize strategies that more closely resemble expert decisions.

[0130] After multiple rounds of iterative training, a general ABR model that integrates various network conditions and expert knowledge is output. It possesses excellent cross-environment generalization ability.

[0131] The Actor model in the final general model is used as the ABR model for bitrate decision, which takes six environmental states as input: (1) the download throughput of the last N (8 in this implementation) video segments, (2) the download time of the last N (8 in this implementation) video blocks, (3) the optional size of the next video block, (4) the bitrate decision of the last video block, (5) the current buffer occupancy, and (6) the number of remaining video blocks in the current streaming session.

[0132] The specific structure of the ABR model is as follows: The first three input states (download throughput of the last N video segments, download time of the last N video chunks, and the optional size of the next video chunk) are processed by a separate one-dimensional convolutional layer (1DCNN), each with 128 filters of size 4. The remaining three input states (bitrate decision for the last video chunk, current buffer occupancy, and number of remaining video chunks in the current streaming session) are processed through fully connected layers, each containing 128 neurons and employing the ReLU activation function. The features extracted from all input layers are then concatenated and processed through a shared hidden fully connected layer (containing 128 neurons). The output layer applies the SoftMax activation function to generate the probability distribution for the available bitrate version.

[0133] By simply inputting six environmental states of the client, the ABR model can output the probability distribution of available bitrate versions, and then select the bitrate with the highest probability as the bitrate of the next video block, thus realizing the bitrate decision for a video block.

[0134] Example 2

[0135] Extensive evaluation experiments have shown that the model proposed in this invention effectively solves the problem of catastrophic forgetting. In the evaluation experiments, the model proposed in this invention was named ANOLE. The specific evaluation experiment settings and evaluation results are as follows.

[0136] 1. Experiment setup.

[0137] To accurately evaluate the performance of this invention, a video streaming simulator was developed. In this simulator, each test video segment was 192 seconds long and divided into 48 four-second segments. Each segment was encoded using eight different bitrate versions: {0.2, 0.8, 2.2, 5.0, 10.0, 18.0, 32.0, 50.0} Mbps. The video player's buffer capacity was set to 60 seconds. The simulator simulated repeatable network conditions using TCP throughput trajectory data derived from real networks, encompassing various bandwidth fluctuations and interference characteristics. The average throughput of this trajectory was 17.66 Mbps, with a peak throughput of 131.44 Mbps. The simulator ran on a Linux workstation configured with a dual-core Intel Xeon Gold 6226R CPU (2.90 GHz) and six NVIDIA GeForce RTX 3090 GPUs. In the experiment, five state-of-the-art ABR algorithms were evaluated: (1) MPC—a heuristic method with inputs including throughput measurement and buffer occupancy; (2) Pensieve—based on deep reinforcement learning A3C; (3) Merina—based on meta-reinforcement learning; (4) NetLLM—based on large language models; and (5) Genet—based on curriculum learning. The ANOLE proposed in this invention uses the PPO algorithm in the experiment, with the Actor network acting as the ABR model. Six environmental states are used as ANOLE inputs: (1) download throughput of the last eight video segments, (2) download time of the last eight video segments, (3) candidate video size of the next segment, (4) bitrate decision of the previous segment, (5) current buffer occupancy, and (6) number of segments remaining in the current streaming session.

[0138] To quantify QoE (Quality of User Experience) performance, an existing QoE calculation function is used, defined as follows:

[0139] ,

[0140] in, This represents the total number of segments in a single video streaming session (default in this invention). =48), Indicates the first The video bitrate of each segment Indicates downloading the first The duration of stuttering during each segment This represents the mapping function from bitrate to video quality. In this evaluation, a linear QoE setting is used, that is, letting... The coefficients in the function are set to... =1, =30.

[0141] 2. Experiment 1.

[0142] Table 1 compares the QoE performance of six ABR algorithms and their internal QoE metrics, including bitrate, stutter duration, and image quality fluctuation. These performance results represent overall performance under diverse network conditions. To clearly demonstrate the QoE improvement of the ANOLE algorithm, the QoE scores of all algorithms were normalized, with the score relative to ANOLE set to 100%. The results show that ANOLE consistently outperforms other advanced ABR algorithms, with QoE improvements ranging from 10.5% to 20.7%.

[0143] In terms of three internal metrics, ANOLE achieved the lowest stuttering time and quality fluctuation among all algorithms. Although its bitrate was not the highest, ANOLE effectively balanced these three metrics, thus achieving the best overall QoE performance.

[0144] Table 1 Comparison of QoE Indicators

[0145]

[0146] 3. Experiment 2.

[0147] To make a more granular comparison, based on average throughput ( Figures 4-6 ) and network volatility ( Figures 7-9 The performance of the ABR algorithm was evaluated across different network subsets. Two result graphs are shown. Figure 4 QoE was demonstrated. Figure 5 Display bit rate Figure 6 This demonstrates playback stuttering issues. Figures 4-6 Estimating network conditions with different average throughputs, ANOLE significantly improved QoE across all these network subsets. Figure 4 ), especially excelling in low throughput conditions (0~7Mbps), and in terms of bit rate selection ( Figure 5 ANOLE employs a more conservative strategy under low throughput conditions to reduce stuttering events caused by network fluctuations; while under high throughput conditions, it adopts a more aggressive bitrate decision to fully utilize available network resources. Regarding playback stuttering (…), Figure 6 ANOLE significantly reduced the stuttering time in each subset, which is an important reason for its excellent QoE performance.

[0148] Figures 7-9 The performance was evaluated under different levels of network volatility; higher volatility resulted in a larger coefficient of variation (CoV). The trend of the results is consistent with... Figure 17 Similar observations were made in other networks. ANOLE demonstrated significantly better QoE across all network subsets, especially in environments with high network volatility (coefficient of variation greater than 0.5 for CoV). Figure 7Furthermore, in scenarios with low network volatility, ANOLE tends to make more aggressive bitrate decisions; while under conditions of highly volatile networks, it tends to adopt a more conservative strategy.

[0149] In all these scenarios, playback stuttering was significantly reduced. Overall, Figures 4-9 The results collectively demonstrate that ANOLE maintains balanced and robust performance under diverse network conditions, effectively mitigating performance degradation caused by catastrophic forgetting.

[0150] 4. Experiment 3.

[0151] In addition to the online evaluation results mentioned above, the QoE performance of six ABR algorithms under different viewing preferences was also evaluated (see Table 2). This experiment considered five common viewing preferences: Balanced I, Avoiding stuttering, High quality, Balanced II, and Avoiding quality fluctuations, with QoE values ​​of QoE-lin, QoE-re, QoE-hd, QoE-log, and QoE-vol, respectively. Specifically, the QoE calculation formula in the above experimental settings was treated as a representation of the Balanced I preference (i.e., the QoE-lin calculation formula), and then modified to extend it to the other four preference settings. The modification of the QoE formula is as follows:

[0152] QoE-re: In addition to the coefficients Set it to 90, and keep all other parameters consistent with the formula for Balanced Type 1.

[0153] QoE-hd: Higher bitrates generally result in better video quality, therefore a higher utility value should be assigned to higher bitrates. When When <1.2Mbps, ;otherwise, The penalty coefficient is... =1, =68.

[0154] QoE-log: q(.) represents the logarithmic relationship between bitrate and video quality, that is: ,in =0.2Mbps. The penalty coefficient is set to... =1, =5.52.

[0155] QoE-vol: In addition to the coefficient Set it to 3, and keep all other parameters consistent with the formula for Balanced Type 1.

[0156] All learning-based ABR algorithms, except for MPC and ANOLE, were trained consecutively for three rounds. Each round used three different QoEs: QoE-lin, QoE-re, and QoE-hd. After each round of training, the algorithms were tested on all preference QoEs, and the results are shown in Table 2 (where "Genet-re" indicates the Genet method trained in the second round using QoE-re to avoid stuttering preferences; other naming conventions are similar). It can be seen that existing learning-based ABR algorithms also suffer from the forgetting problem under different viewing preferences; that is, after learning a new viewing preference feature, previously learned viewing preferences are forgotten. The ANOLE proposed in this invention effectively solves this problem, outperforming existing ABR algorithms under five different preference conditions and achieving optimal overall performance.

[0157] Table 2. QoE performance of 6 ABR algorithms under different viewing preferences.

[0158]

[0159] 5. Experiment 4

[0160] Furthermore, this invention was also evaluated in three streaming media applications: video-on-demand, live streaming, and video recognition. To quantify QoE in different applications, different QoE calculation formulas were used for each application, specifically:

[0161] (1) On-demand: The video is pre-stored on the server, and the client requests and plays it on demand. The specific QoE is the classic QoE in the above experimental settings.

[0162] (2) Live Streaming: Live events are captured in real-time by camera equipment and uploaded to the server for encoding. Clients must wait for the video to be processed before initiating a request and playback. The following QoE is used:

[0163] ,

[0164] in, , , They represent the first The bitrate, stutter duration, and playback latency of each segment. The total number of segments in the session. =2、 =50、 =0.02, and when When <1.1 seconds, =0.25, otherwise 0.5.

[0165] (3) Video recognition: The video is captured and encoded on the client side (e.g., camera), and then transmitted to the server for content analysis. The specific QoE formula is as follows:

[0166] ,

[0167] ,

[0168] in, Indicates the first The transmission delay of each segment, Indicates video inference accuracy (in terms of bit rate) (positive correlation), coefficient =5.

[0169] The other five comparison algorithms each train a separate model for each application for evaluation, while ANOLE is a general-purpose model evaluated across three applications, with the following results: Figures 10-12 As shown, the ANOLE model proposed in this invention outperforms other ABR algorithms in three different applications, demonstrating that ANOLE can effectively avoid performance degradation caused by catastrophic forgetting even in different video streaming applications. Furthermore, this invention does not require training a separate model for each application, thus exhibiting stronger generalization capabilities.

[0170] Example 3

[0171] An expert-guided video streaming bitrate adaptive system includes:

[0172] Video server: Stores multi-bitrate version video blocks;

[0173] ABR Server: Deploys the general ABR model trained using the expert-guided video streaming bitrate adaptive method, and responds to client decision requests;

[0174] Client: Collects network status in real time and sends it to the ABR server, and decides to download video blocks based on the returned bitrate;

[0175] Communication module: Used for data transmission between the client and the ABR server and video server.

[0176] As a refinement of the above embodiments, the practical application framework of the model (ABR algorithm) proposed in this invention is as follows: Figure 13The video has already been processed, encoded into multiple bitrate versions using an H.264 codec, and each version is cut into equal-sized video blocks and deployed on a video server. The model proposed in this invention is deployed on an ABR server. Both the ABR server and the video server run continuously, continuously listening for ABR decision requests and video download requests from clients, respectively. Requesting to download each video block involves four steps:

[0177] (1) The client measures the environmental state (e.g., past throughput, buffer occupancy, etc., as input to the ABR model) and converts it into an ABR request, which is sent to the ABR server over the network.

[0178] (2) Upon receiving a request, the ABR server runs a neural network model to determine the optimal bit rate version and returns the decision to the client.

[0179] (3) The client initiates a video download request to the video server based on the received bit rate decision.

[0180] (4) Upon receiving the request, the video server sends the corresponding video segment back to the client for playback at the selected bit rate.

[0181] Combination Figure 14 As shown, this disclosure provides an expert-guided video streaming bitrate adaptation device 300, including a processor 304 and a memory 301. Optionally, the device may further include a communication interface 302 and a bus 303. The processor 304, communication interface 302, and memory 301 can communicate with each other via the bus 303. The communication interface 302 can be used for information transmission. The processor 304 can call logical instructions in the memory 301 to execute the expert-guided video streaming bitrate adaptation method of the above embodiment.

[0182] Furthermore, the logic instructions in the aforementioned memory 301 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0183] The memory 301, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 304 executes functional applications and data processing by running the program instructions / modules stored in the memory 301, thereby implementing the expert-guided video streaming bitrate adaptive method in the above embodiments.

[0184] The memory 301 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 301 may include high-speed random access memory and may also include non-volatile memory.

[0185] This disclosure provides a computer-readable storage medium storing computer-executable instructions configured to execute the above-described expert-guided video streaming bitrate adaptive method.

[0186] The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0187] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media capable of storing program code. It can also be a transient storage medium.

[0188] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0189] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0190] The methods and products (including but not limited to devices and equipment) disclosed in the embodiments herein can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0191] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A method for expert-guided video streaming bitrate adaptation, characterized in that, The method comprises the following steps: obtaining a mixed throughput trajectory dataset, initializing a neural network, training an initial model on the dataset through deep reinforcement learning; clustering the mixed throughput trajectory dataset into multiple network subsets, pre-training an expert model based on the initial model, training the expert model on each subset, and recording the state-action decision pairs of each expert model and storing them in an expert experience pool; generating a virtual video session based on the mixed throughput trajectory dataset, collecting state-action-reward triplets of each video segment during transmission; calculating cumulative rewards and advantage values based on the state-action-reward triplets, extracting the state-action decision pairs of the expert model of the network subset corresponding to the virtual video session from the expert experience pool, and combining the state-action-reward triplets, cumulative rewards, advantage values, and state-action decision pairs of the expert model to form training tuples; randomly sampling a batch of training tuples as training samples to simultaneously optimize the Actor network and the Critic network, and obtaining a general ABR model; jointly optimizing the Actor network and the Critic network through gradient descent; the Critic network objective function: ， wherein, denotes the cumulative reward in the mth tuple, denotes the input environment state in the mth tuple, denotes adjusting the parameters of the model minimizes the following objective function, denotes all tuples; the Actor network objective function: ， wherein, is a weight parameter, is the current batch maximum advantage value, denotes the probability of an action being selected by the actor in state , denotes the action in the mth tuple, denotes the advantage value in the mth tuple, denotes the state of the expert model in the mth tuple, denotes the decision of the expert model in the mth tuple, denotes the parameter of the model adjusted to maximize the following objective function; The general ABR model adopts an Actor model, which receives six-dimensional environmental states: the download throughput of the last N video segments, the download time of the last N video segments, the selectable size of the next video block, the bit rate decision of the last video block, the current buffer occupancy rate, and the number of remaining video segments in the current streaming session. The specific structure of the Actor model is as follows: The first three input states are processed by separate one-dimensional convolution layers, each convolution layer has 128 filters with a filter size of 4; the last three input states are processed by fully connected layers, each fully connected layer contains 128 neurons and uses a ReLU activation function; all the extracted features of the input layers are connected and processed through a shared hidden fully connected layer; the output layer applies a SoftMax activation function to generate a probability distribution of available bit rate versions.

2. The expert-guided video streaming bitrate adaptation method according to claim 1, wherein, The clustering of the mixed throughput trajectory dataset comprises: unsupervised clustering based on network features, including average throughput and network volatility indicators; Each network subset represents a type of network environment.

3. The method of claim 2, wherein the method further comprises: The specific way of recording the state-action decision pairs of each expert model and storing them in the expert experience pool comprises: obtaining the expert model trained on each subset, testing each expert model in its respective network subset, recording its decision strategy as a state-action decision pair, i.e., an environmental state and a corresponding bit rate selection, and storing it in the expert experience pool.

4. The expert-guided video streaming bitrate adaptation method according to claim 1, wherein, The cumulative reward calculation formula is as follows: , wherein, represents the cumulative reward of the th segment, represents the reward of the th segment, represents a discount factor, represents the total number of segments of the video session; The advantage value calculation formula is as follows: ， wherein, represents the advantage value, represents the input state the output of the Critic network, i.e. the expected reward.

5. A system for expert-guided video streaming bitrate adaptation, characterized in that, The method comprises the following steps: video server: storing multiple bit rate version video blocks; ABR server: deploying a general ABR model trained by the expert-guided video streaming bit rate adaptation method according to any one of claims 1-4, and responding to client decision requests; client: collecting network states in real time and sending them to the ABR server, and downloading video blocks according to the returned bit rate decisions; Communication module: for data transmission between the client and the ABR server, video server.

6. An apparatus for expert-guided video streaming bitrate adaptation, comprising a processor and a memory having stored program instructions, wherein, The processor is configured to execute, when running the program instructions, the expert-guided video streaming bitrate adaptation method as claimed in any one of claims 1-4.

7. A computer readable storage medium characterized in that, A computer program is stored thereon, which is executed by a processor to implement the expert-guided video streaming bitrate adaptation method as claimed in any one of claims 1-4.

Images