Video streaming media bit rate self-adaption method and system based on expert guidance

Abstract

The invention relates to the technical field of video stream transmission, and relates to a video stream media bit rate self-adaption method and system based on expert guidance. The method comprises the following steps: acquiring a mixed throughput trajectory data set, and training according to the data set to obtain an initial model; the data set is clustered into subsets, an expert model is trained on each subset, and a state-action decision pair of each expert model is recorded and stored in an expert experience pool; generating a virtual video session, and collecting a state-action-reward triple of each video clip in a transmission process; calculating an accumulated reward and an advantage value, and combining the state-action-reward triple, the accumulated reward, the advantage value and the state-action decision pair of the expert model to form a training tuple; and sampling the training tuple to synchronously optimize the Actor network and the Critic network, and obtaining a universal ABR model. According to the method, the problem of disastrous forgetting of the ABR algorithm is solved through a multi-expert guidance method, and the generalization ability and the self-adaptive performance of the model are improved.

Description

Technical Field

[0001] The present application relates to the technical field of video streaming transmission, and in particular to a method and system for self-adapting the bit rate of video streaming media based on expert guidance. Background Art

[0002] With the rapid development of the Internet and multimedia processing technology, video streaming has become the core carrier of information dissemination and entertainment consumption. The current mainstream video streaming transmission technology (such as dynamic adaptive streaming over HTTP DASH) pre-encodes video content into multiple versions (with different bit rates and resolutions) of video blocks (usually 2 to 6 seconds per block), such as Figure 15 As shown, the client dynamically downloads the adapted version based on real-time network conditions, and uses the playback buffer to achieve "download-while-playing," effectively reducing initial loading latency and improving playback continuity. However, the dynamic nature of the network environment (such as bandwidth fluctuations, signal strength changes, and cross-network switching) poses significant challenges to fixed bitrate transmission solutions: insufficient bandwidth causes lag, while ample bandwidth prevents full utilization of high-definition video resources.

[0003] To solve the above problems, adaptive bit rate algorithm (ABR algorithm) came into being, such as Figure 16 As shown in the figure, it dynamically selects the optimal video block bitrate version by real-time analysis of network bandwidth estimation, buffer status, historical download data and other parameters, maximizing the user experience quality (QoE) while ensuring smooth playback.

[0004] Existing ABR algorithms are mainly divided into two categories: Heuristic methods (such as BOLA and MPC): These methods adjust the bitrate based on preset rules. While simple to implement, they rely on manual experience and are difficult to adapt to complex and changing network environments, resulting in suboptimal performance. Machine learning methods (such as Pensieve and Oboe): They learn environmental features through deep neural networks and can be customized to adapt to specific scenarios. They have become the mainstream solution.

[0005] However, existing machine learning ABR algorithms have a fundamental flaw - the catastrophic forgetting problem: when the model training environment switches, the algorithm will forget the key network features previously learned (especially low-frequency or early features), resulting in significant performance degradation in some environment subsets (such as specific congestion patterns or mobile switching scenarios) (see Figure 17 The root cause is that existing solutions rely on a single universal model to adapt to all environments. While this model possesses basic generalization capabilities, it cannot effectively retain key historical features during continuous training, limiting the model's stability and generalization performance in dynamic networks.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0007] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical elements or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0008] The disclosed embodiments provide a method and system for expert-guided video streaming bitrate adaptation. Multiple expert ABR models trained in different environments are used to jointly guide a general model, thereby enhancing the general model's ability to learn key features of various network environments. This significantly alleviates or eliminates the catastrophic forgetting problem and improves the model's adaptive performance under diverse network conditions.

[0009] In some embodiments, the method comprises: Obtain a mixed throughput trajectory dataset, initialize the neural network, and train the initial model using deep reinforcement learning on the dataset. Cluster the mixed throughput trajectory dataset into multiple network subsets. Use the initial model as the pre-training basis for the expert model. Train the expert model on each subset, record the state-action decision pairs of each expert model, and store them in the expert experience pool. Generate virtual video sessions based on the mixed throughput trajectory dataset and collect the state-action-reward triples of each video clip during the transmission process; Calculate the cumulative reward and advantage value based on the state-action-reward triplet, and extract the state-action decision pairs of the expert model of the network subset corresponding to the virtual video session from the expert experience pool. Combine the state-action-reward triplet, cumulative reward, advantage value, and the state-action decision pair of the expert model to form a training tuple; A batch of training tuples are randomly sampled as training samples to simultaneously optimize the Actor network and Critic network to obtain a universal ABR model.

[0010] Preferably, the clustering of the mixed throughput trajectory dataset includes: performing unsupervised clustering based on network characteristics, wherein the network characteristics include average throughput and network volatility indicators; Each network subset represents a type of network environment.

[0011] Preferably, the specific method of recording the state-action decision pairs of each expert model and storing them in the expert experience pool includes: Obtain the expert model trained on each subset, test each expert model in its own network subset, record its decision strategy as a state-action decision pair, i.e., the environment state and the corresponding bit rate selection, and store it in the expert experience pool.

[0012] Preferably, the cumulative reward calculation formula is as follows: , in, Indicates the The cumulative reward of each segment, Indicates the Rewards for a fragment, represents the discount factor, Indicates the total number of segments in a video session; The advantage value calculation formula is as follows: , in, represents the advantage value, Indicates input status The output of the Critic network is the expected reward.

[0013] Preferably, the Actor network and the Critic network are jointly optimized by gradient descent; The objective function of the Critic network is: , in, represents the cumulative reward in the m-th tuple, represents the input environment state in the mth tuple, By adjusting the parameters of the model Minimize the following objective function, Represents all tuples; The Actor network objective function: , in, is the weight parameter, is the maximum advantage value of the current batch, Indicates that the status Next Actor select action The probability of represents the action in the mth tuple, represents the advantage value in the m-th tuple, represents the state of the expert model in the mth tuple, represents the decision of the expert model in the mth tuple, By adjusting the parameters of the model Maximize the following objective function.

[0014] Preferably, the ABR model adopts an Actor model and receives six-dimensional environmental states: download throughput of the last N video segments, download time of the last N video segments, optional size of the video chunk of the next segment, bitrate decision of the last video chunk, current buffer occupancy, and number of remaining video segments in the current streaming session.

[0015] Preferably, the specific structure of the Actor model is as follows: The first three input states are processed by separate one-dimensional convolutional layers, each with 128 filters and a filter size of 4; the last three input states are processed by fully connected layers, each containing 128 neurons and using a ReLU activation function; the features extracted from all input layers are concatenated and processed through a shared hidden fully connected layer; the output layer applies a SoftMax activation function to generate a probability distribution of available bitrate versions.

[0016] In some embodiments, the system comprises: Video server: stores video chunks with multiple bitrate versions; ABR server: deploys the general ABR model trained by the expert-guided video streaming bitrate adaptation method and responds to client decision requests; Client: collects network status in real time and sends it to the ABR server, and decides to download video chunks based on the bitrate returned; Communication module: used for data transmission between the client and the ABR server and video server.

[0017] In some embodiments, the apparatus includes a processor and a memory storing program instructions, and the processor is configured to execute the method for video streaming bit rate adaptation based on expert guidance when running the program instructions.

[0018] In some embodiments, the storage medium stores a computer program thereon, and when the program is executed by a processor, the method for video streaming media bit rate adaptation based on expert guidance is implemented.

[0019] The embodiment of the present disclosure provides a method and system for video streaming media bitrate adaptation based on expert guidance, which can achieve the following technical effects: This invention utilizes a multi-expert guidance approach, leveraging multiple expert ABR models trained in different network environments to jointly guide a universal model. This significantly enhances the universal model's ability to learn key features of various network environments, effectively mitigating or eliminating the problem of catastrophic forgetting. By incorporating the decision-making experience of multiple expert models, the expert-guided universal model significantly improves its adaptive performance under diverse network conditions. By pre-training the initial model and leveraging the expert experience pool for joint guidance, the training requirements for new environments or viewing preferences are significantly reduced.

[0020] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] One or more embodiments are exemplarily described by corresponding drawings. These exemplary descriptions and drawings do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements. The drawings do not constitute a scale limitation. In addition, Figure 1 This is the overall learning framework diagram of the general ABR model proposed in this invention.

[0022] Figure 2 It is a flowchart of the expert learning stage.

[0023] Figure 3 It is a flowchart of the general model learning phase.

[0024] Figure 4 This is the normalized evaluation result of the universal ABR model proposed in this invention in different throughput network subsets.

[0025] Figure 5 This is the bit rate evaluation result of the general ABR model proposed in this invention in different throughput network subsets.

[0026] Figure 6 This is the freeze time evaluation result of the general ABR model proposed in this invention in different throughput network subsets.

[0027] Figure 7 This is the normalized evaluation result of the universal ABR model proposed in this invention under different network fluctuation levels.

[0028] Figure 8 This is the bit rate evaluation result of the universal ABR model proposed in this invention under different network fluctuation levels.

[0029] Figure 9 This is the freeze time evaluation result of the universal ABR model proposed in this invention under different network fluctuation levels.

[0030] Figure 10This is the evaluation result of the general ABR model proposed in this invention in the application of on-demand video streaming.

[0031] Figure 11 This is the evaluation result of the general ABR model proposed in this invention in the live video streaming application.

[0032] Figure 12 This is the evaluation result of the general ABR model proposed in this invention in the application of video recognition and video stream.

[0033] Figure 13 This is a practical application framework diagram of the universal ABR model proposed in the present invention.

[0034] Figure 14 It is a schematic diagram of the device structure of an embodiment of the present disclosure.

[0035] Figure 15 This is a schematic diagram of the video streaming framework.

[0036] Figure 16 This is a schematic diagram of ABR decision making.

[0037] Figure 17 This is a simplified diagram of the catastrophic forgetting problem. DETAILED DESCRIPTION

[0038] In order to be able to understand the features and technical content of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The accompanying drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the sake of convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0039] In the description and claims of the embodiments of the present disclosure, as well as in the accompanying drawings, the terms "first," "second," and the like are used to distinguish similar items and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate to describe the embodiments of the present disclosure herein. In addition, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions.

[0040] Unless otherwise stated, the term "plurality" means two or more.

[0041] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B means: A or B.

[0042] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0043] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.

[0044] Example 1 like Figure 1 As shown in FIG, a video streaming bitrate adaptation method based on expert guidance includes three stages: the initial learning stage, the expert learning stage and the general model learning stage.

[0045] Specifically, they include: Initial learning phase S1: Obtain a mixed throughput trajectory dataset, initialize the neural network, and train the initial model on the dataset through deep reinforcement learning. Expert learning stage S2: Cluster the mixed throughput trajectory dataset into multiple network subsets. Use the initial model as the basis for pre-training the expert model. Train the expert model on each subset, record the state-action decision pairs of each expert model, and store them in the expert experience pool. General model learning phase S3: Generates a virtual video session based on the mixed throughput trajectory dataset and collects the state-action-reward triplet of each video clip during the transmission process; S4: Calculate the cumulative reward and advantage value based on the state-action-reward triplet, and extract the state-action decision pairs of the expert model of the network subset corresponding to the virtual video session from the expert experience pool. Combine the state-action-reward triplet, cumulative reward, advantage value, and state-action decision pair of the expert model to form a training tuple; S5: Randomly sample a batch of training tuples as training samples and simultaneously optimize the Actor network and Critic network to obtain a universal ABR model.

[0046] Among them, the state is the environment state, the action is the corresponding bitrate selection, and the reward is the QoE performance.

[0047] As a refinement of the above embodiment, the goal of the initial learning phase is to train a basic ABR model in a diverse streaming environment, providing pre-trained initial weights for subsequent phases and accelerating training. The neural network is initialized with random neuron weights and trained on data covering all dimensions of streaming diversity. It is trained using deep reinforcement learning (DRL). The model makes bitrate decisions based on environmental state inputs (such as past throughput measurements and current buffer occupancy). The resulting rewards (i.e., QoE performance) are used to calculate gradients to update the model's neuron weights.

[0048] Specifically include: S101: Obtain a mixed throughput trajectory dataset. Extract data from the throughput trajectory dataset containing mixed network conditions, denoted as These data cover a variety of network environments and are used to simulate various network bandwidth conditions.

[0049] S102: Initialize the neural network , whose neuron weights are random values ​​to ensure that the model learns from scratch.

[0050] S103: Conduct deep reinforcement learning training. Using standard deep reinforcement learning methods, Neural Network To train: , Among them, the function Represents a standard deep reinforcement learning training process, such as A3C, PPO, etc.

[0051] S104: After training is completed, the initial model is obtained and stored , serving as a pre-training starting point for subsequent stages (expert learning and general model learning).

[0052] It should be noted that in this embodiment, only the network condition is considered as the streaming media environment factor that causes catastrophic forgetting.

[0053] As a refinement of the above embodiment, the expert learning stage trains multiple expert models in different types of network environments, extracts their optimal decision-making experience, and stores it in the experience pool to provide guidance for generalization model training.

[0054] like Figure 2 As shown in Figure 2, the specific steps of the expert learning phase are as follows: S201: Mixed Throughput Trajectory Dataset Perform unsupervised clustering (such as Kmeans) and divide it into K network subsets, recorded as : , Each subset Represents a specific type of network condition. The above clustering process is based on network characteristics. (e.g. average throughput, network fluctuations).

[0055] S202: Train the expert model in each subset. For each network subset , using the initial model As the pre-training model weight, the corresponding expert model is trained through deep reinforcement learning : ; S203: Test and record the behavior strategy of the expert model. After the training is completed, In their respective network subsets Test it in the form of state-action pairs. , that is, the environmental status and the corresponding bit rate selection.

[0056] S204: Storing the expert experience pool Y. All state-action pairs of the expert models are uniformly stored in the experience pool Y. The experience pool will serve as a source of expert knowledge to guide subsequent general model learning.

[0057] As a refinement of the above embodiment, the general model learning phase uses the expert experience pool Y obtained in the second phase and combines it with the mixed throughput trajectory dataset , the initial model The model is trained into a general ABR decision model with generalization ability. Deep reinforcement learning is still used for training, but expert knowledge is additionally introduced.

[0058] The training process of this stage can be formally expressed as: , in, represents the expert decision experience pool generated in the second stage, is a general model that is ultimately used for adaptive bitrate (ABR) decision making. Represents the training process of the third stage universal model.

[0059] Specifically, the initial model Contains two neural networks: Actor network responsible for ABR decision-making (denoted as ) and the Critic network (denoted as ).

[0060] like Figure 3As shown, specifically, step S3 includes: for each streaming session h, from the mixed throughput trajectory dataset A network trajectory is randomly selected from the network and virtual streaming is performed to generate a complete video session h. A typical streaming session consists of a series of video clips (for example, a 100-second video session contains 50 2-second video clips). Before transmitting each clip, the environment state (such as past throughput measurements, current buffer levels, etc.) is input into the Actor network to obtain the action decision, that is, the selected bit rate. Then, the corresponding network conditions are simulated according to the throughput trajectory data, and the clip is transmitted at the selected bit rate. After the clip transmission is completed, the reward obtained is calculated according to the QoE function, that is, the QoE performance. All clips in session h are transmitted sequentially in this way. The state, action and reward of each video clip during the transmission process are recorded to form a triplet .

[0061] The step S4 comprises: S401: Based on the collected triples , use the computing function Computing(.) to generate the cumulative reward (the weighted sum of the rewards from the current segment to the last segment of the streaming session) and the advantage value for each video segment : , in, Indicates the The cumulative reward of each segment, Indicates the Rewards for a fragment, Indicates the total number of segments in a video session. Represents a discount factor (its value is less than 1.0), which is used to control the exponential decay of future rewards in the calculation of cumulative rewards. The principle is: smaller will give higher weight to immediate rewards, while The larger it is, the greater the impact of future rewards.

[0062] The advantage value calculation formula is as follows: , in, represents the advantage value, Indicates input status The output of the Critic network at this time is the expected reward. The advantage value of the video segment quantifies the effectiveness of the action taken at that step. For example, if >0, it indicates that the actual reward brought by the selected action is higher than the expected reward, reflecting a performance that exceeds expectations; on the contrary, if If it is negative, it means that the performance is lower than expected.

[0063] S402: Introduce expert experience and construct training samples. Extract the expert state-action pair corresponding to session h from the experience pool Y. , and the data of this session Together they form a tuple and store it in buffer B: .

[0064] The step S5 comprises: Sample mixed environment batches and update the neural network. After every several streaming sessions, randomly sample multiple tuples from buffer B to form a mixed training batch across environmental conditions. , update the Critic network and Actor network; , in, represents the total number of tuples in the batch; the throughput trace of each streaming session is randomly selected, so multiple sessions together cover diverse network conditions. Therefore, the sampled tuples in each batch come from a mixture of different network environments.

[0065] The goal of the Critic network is to adjust its neuron weights Improve the accuracy of reward prediction and ensure that its output (i.e., predicted value) is as close as possible to the actual value (i.e., accumulated reward). The objective function of the Critic network is: The objective function of the Critic network is: , in, represents the cumulative reward in the m-th tuple, represents the input environment state in the mth tuple, By adjusting the parameters of the model Minimize the following objective function, Represents all tuples.

[0066] The Actor network, combined with the expert experience in the mth tuple (i.e., state-action pair), whose neuron weights is adjusted to maximize the following objective function: The Actor network objective function: , in, is the weight parameter, is the maximum advantage value of the current batch, Indicates that the status Next Actor select action The probability of represents the action in the mth tuple, represents the advantage value in the m-th tuple, represents the state of the expert model in the mth tuple, represents the decision of the expert model in the mth tuple, By adjusting the parameters of the model Maximize the following objective function.

[0067] The Actor network combines the current network's video streaming data with expert experience to optimize the strategy to make it closer to expert decision-making.

[0068] After multiple rounds of iterative training, a general ABR model that integrates multiple network conditions and expert knowledge is output. , with excellent cross-environment generalization capabilities.

[0069] The Actor model in the final general model is used as an ABR model for bitrate decision making, which takes six environmental states as input: (1) the download throughput of the last N (8 in this implementation) video segments, 2) the download time of the last N (8 in this implementation) video chunks, (3) the optional size of the video chunk of the next segment, (4) the bitrate decision for the last video chunk, (5) the current buffer occupancy, and (6) the number of remaining video chunks in the current streaming session.

[0070] The ABR model has the following structure: the first three input states (download throughput of the last N video segments, download time of the last N video chunks, and the optional size of the next segment's chunks) are processed by separate one-dimensional convolutional layers (1DCNN), each with 128 filters of filter size 4. The remaining three input states (the bitrate decision for the last chunk, the current buffer occupancy, and the number of remaining chunks in the current streaming session) are processed by fully connected layers, each with 128 neurons and a Reluctant Unit (ReLU) activation function. The features extracted from all input layers are then concatenated and processed through a shared hidden fully connected layer (with 128 neurons). The output layer applies a SoftMax activation function to generate a probability distribution over the available bitrates.

[0071] By simply inputting the client's six environmental states, the ABR model can output a probability distribution of available bitrate versions, and then select the bitrate with the highest probability as the bitrate of the next video block, thereby realizing the bitrate decision of a video block.

[0072] Example 2 Extensive evaluation experiments show that the model proposed in this paper effectively solves the problem of catastrophic forgetting. In the evaluation experiments, the model proposed in this paper is named ANOLE. The specific evaluation experiment settings and evaluation results are as follows.

[0073] 1. Experimental setup.

[0074] In order to accurately evaluate the performance of the present invention, a video streaming simulator was developed, in which each test video was 192 seconds long and divided into 48 4-second segments. Each segment was encoded into eight different bitrate versions: {0.2, 0.8, 2.2, 5.0, 10.0, 18.0, 32.0, 50.0} Mbps. The buffer capacity of the video player was set to 60 seconds. The simulator simulated repeatable network conditions using TCP throughput trajectory data, which was derived from real networks and covered various bandwidth fluctuations and interference characteristics. The average throughput of the trajectory was 17.66 Mbps, with a peak of 131.44 Mbps. The simulator ran on a Linux workstation equipped with a dual-core Intel Xeon Gold 6226R CPU (2.90 GHz) and six NVIDIA GeForce RTX 3090 GPUs. In the experiment, five state-of-the-art ABR algorithms were evaluated, namely: (1) MPC - a heuristic-based method with inputs including throughput measurement and buffer occupancy; (2) Pensieve - based on deep reinforcement learning A3C; (3) Merina - based on meta-reinforcement learning; (4) NetLLM - based on large language models; and (5) Genet - based on curriculum learning. The proposed ANOLE uses the PPO algorithm in the experiment, with the Actor network acting as the ABR model. Six environment states are used as ANOLE inputs: (1) the download throughput of the last eight video segments, (2) the download time of the last eight video segments, (3) the candidate video size of the next segment, (4) the bitrate decision of the previous segment, (5) the current buffer occupancy, and (6) the number of segments remaining in the current streaming session.

[0075] In order to quantify the QoE (user experience quality) performance, an existing QoE calculation function is used, which is defined as follows: , in, Indicates the total number of segments in a video streaming session (default in this invention) =48), Indicates the The video bitrate of each clip, Indicates download The length of the freeze when the clip is In this evaluation, the linear QoE setting is adopted, that is, The coefficients in the function are set to =1, =30.

[0076] 2. Experiment 1.

[0077] Table 1 compares the QoE performance of six ABR algorithms along with their internal QoE metrics, including bitrate, duration of freezes, and image quality fluctuation. These performance results are presented as a whole under diverse network conditions. To clearly demonstrate the QoE improvement achieved by the ANOLE algorithm, the QoE scores of all algorithms were normalized relative to ANOLE's score, set to 100%. The results show that ANOLE consistently outperforms other leading ABR algorithms, with QoE improvements ranging from 10.5% to 20.7%.

[0078] ANOLE achieved the lowest freeze duration and quality fluctuation among all algorithms in terms of the three internal metrics. Although its bitrate was not the highest, ANOLE effectively balanced these three metrics to achieve the best overall QoE performance.

[0079] Table 1. Comparison of QoE indicators

[0080] 3. Experiment 2.

[0081] For a more fine-grained comparison, the average throughput ( Figure 4-Figure 6 ) and network volatility ( Figure 7-Figure 9 ) evaluated the performance of the ABR algorithm in different network subsets. Figure 4 Demonstrated QoE, Figure 5 Display bitrate, Figure 6 Displays playback freezes. Figure 4-Figure 6 We evaluated network conditions with different average throughputs and found that ANOLE significantly improved QoE in all these network subsets ( Figure 4 ), especially in low throughput conditions (0~7Mbps), and in bit rate selection ( Figure 5 ), ANOLE adopts a more conservative strategy under low throughput conditions to reduce the occurrence of freezes caused by network fluctuations; while under high throughput conditions, it adopts a more aggressive bitrate decision to make full use of available network resources. Figure 6 ), ANOLE significantly reduced the duration of freezes in each subset, which is also an important reason for its excellent QoE performance.

[0082] Figure 7-Figure 9The performance under different network volatility levels was evaluated. The higher the volatility, the larger the coefficient of variation CoV. The trend of the results is similar to Figure 17 ANOLE shows significantly better QoE in all network subsets, especially in environments with high network volatility (coefficient of variation CoV greater than 0.5) ( Figure 7 Furthermore, in scenarios with low network volatility, ANOLE tends to make more aggressive bitrate decisions, while in scenarios with high network volatility, it tends to adopt a more conservative strategy.

[0083] In all of these scenarios, playback stuttering was significantly reduced. Overall, Figure 4-Figure 9 The results collectively demonstrate that ANOLE maintains balanced and robust performance under diverse network conditions, effectively alleviating the performance degradation caused by catastrophic forgetting.

[0084] 4. Experiment 3.

[0085] In addition to the above-mentioned online evaluation results, the QoE performance of six ABR algorithms under different viewing preferences was also evaluated (see Table 2). This experiment considered five common viewing preferences: balanced type I, avoiding lag, high image quality, balanced type II, and avoiding image quality fluctuations. Their QoEs are QoE-lin, QoE-re, QoE-hd, QoE-log, and QoE-vol, respectively. Specifically, the QoE calculation formula in the above experimental setting is regarded as a representation of the balanced type I preference (i.e., the calculation formula of QoE-lin), and then it is modified to extend to the other four preference settings. The modifications to the QoE formula are as follows: QoE-re: In addition to the coefficient Set it to 90, and the rest of the parameters are consistent with the formula of balanced type 1.

[0086] QoE-hd: Higher bit rates generally result in better video quality, so higher utility values ​​are assigned to higher bit rates. <1.2Mbps, ;otherwise, The penalty coefficient is =1, =68.

[0087] QoE-log: q(.) represents the logarithmic relationship between bitrate and video quality, that is: ,in =0.2Mbps. The penalty coefficient is set to =1, =5.52.

[0088] QoE-vol: In addition to the coefficient Set to 3, and the rest of the parameters are consistent with the formula of balanced type 1.

[0089] All learning-based ABR algorithms, except MPC and ANOLE, were trained for three consecutive rounds, using three QoEs (QoE-lin, QoE-re, and QoE-hd) in each round. After each round of training, they were tested on all preferred QoEs. The results are shown in Table 2 (where "Genet-re" represents the Genet method, trained in the second round using QoE-re with the lag-avoiding preference; other nomenclatures are similar). It can be seen that existing learning-based ABR algorithms also suffer from the forgetfulness problem under different viewing preferences. That is, after learning the characteristics of a new viewing preference, they forget the previously learned viewing preference. The proposed ANOLE effectively solves this problem, outperforming existing ABR algorithms under five different preferences and achieving optimal overall performance.

[0090] Table 2 QoE performance of six ABR algorithms under different viewing preferences

[0091] 5. Experiment 4 In addition, the present invention also conducts evaluations under three streaming media applications: on-demand, live broadcast, and video recognition. In order to quantify the QoE in different applications, different QoE calculation formulas are used in different applications, specifically: (1) On-demand: The video is pre-stored on the server, and the client requests and plays it on demand. The specific QoE is the classic QoE in the above experimental setting.

[0092] (2) Live broadcast: Live events are captured instantly by cameras and uploaded to the server for encoding. The client must wait for the video to be processed before initiating a request to play it. The following QoE is used: , in, 、 、 Respectively represent bitrate, freeze duration, and playback delay of each clip, is the total number of fragments in the session, =2, =50, =0.02, and when <1.1 seconds, =0.25, otherwise 0.5.

[0093] (3) Video recognition: Video is collected and encoded on the client side (e.g., camera), and then transmitted to the server for content analysis. The specific QoE formula is as follows: , , in, Indicates the The transmission delay of the fragments, Indicates the video inference accuracy (with bit rate positive correlation), coefficient =5.

[0094] The other five comparison algorithms train a separate model for each application for evaluation, while ANOLE is a universal model that is evaluated in three applications. The results are as follows: Figure 10-12 The ANOLE model proposed in the present invention outperforms other ABR algorithms in three different applications, indicating that ANOLE can effectively avoid performance degradation caused by catastrophic forgetting even in different video streaming applications. In addition, the present invention does not require a separate model to be trained for each application, and has stronger generalization.

[0095] Example 3 A video streaming bitrate adaptation system based on expert guidance, comprising: Video server: stores video chunks with multiple bitrate versions; ABR server: deploys the general ABR model trained by the expert-guided video streaming bitrate adaptation method and responds to client decision requests; Client: collects network status in real time and sends it to the ABR server, and decides to download video chunks based on the bitrate returned; Communication module: used for data transmission between the client and the ABR server and video server.

[0096] As a refinement of the above embodiment, the practical application framework of the model (ABR algorithm) proposed in the present invention is as follows Figure 13 The video has been processed and encoded into multiple different bitrate versions using the H.264 codec. Each version is cut into equal-sized video chunks and deployed in the video server. The model proposed in this paper is deployed in the ABR server. Both the ABR server and the video server are running continuously, listening for ABR decision requests and video download requests from the client, respectively. There are four steps to request download of each video chunk: (1) The client measures the environment state (e.g., past throughput, buffer occupancy, etc., as input to the ABR model) and converts it into an ABR request, which is sent to the ABR server over the network.

[0097] (2) After receiving the request, the ABR server runs the neural network model to determine the best bitrate version and returns the decision to the client.

[0098] (3) The client initiates a video download request to the video server based on the received bit rate decision.

[0099] (4) After receiving the request, the video server sends the corresponding video clip back to the client for playback at the selected bit rate.

[0100] Combine Figure 14 As shown, an embodiment of the present disclosure provides a device 300 for expert-guided video streaming bitrate adaptation, comprising a processor 304 and a memory 301. Optionally, the device may also include a communication interface 302 and a bus 303. The processor 304, communication interface 302, and memory 301 may communicate with each other via bus 303. Communication interface 302 may be used for information transmission. Processor 304 may invoke logic instructions in memory 301 to execute the expert-guided video streaming bitrate adaptation method of the above embodiment.

[0101] In addition, the logic instructions in the memory 301 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0102] Memory 301, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as the program instructions / modules corresponding to the methods in the embodiments of the present disclosure. Processor 304 executes the program instructions / modules stored in memory 301 to perform functional applications and data processing, thereby implementing the expert-guided video streaming bitrate adaptation method described in the above-mentioned embodiments.

[0103] The memory 301 may include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data generated based on the use of the terminal device. Furthermore, the memory 301 may include high-speed random access memory and non-volatile memory.

[0104] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above-mentioned method for video streaming media bit rate adaptation based on expert guidance.

[0105] The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0106] The technical solutions of the embodiments of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present disclosure. The aforementioned storage medium may be a non-transitory storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, and other media that can store program code, or a transient storage medium.

[0107] The above description and the accompanying drawings sufficiently illustrate the embodiments of the present disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible variations. Unless expressly required, individual components and functions are optional, and the order of operations may vary. Portions and features of some embodiments may be included in or replace portions and features of other embodiments. Moreover, the terms used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and claims, the singular forms "a", "an" and "the" are intended to also include the plural forms unless the context clearly indicates otherwise. Similarly, the term "and / or" as used in this application means any and all possible combinations of one or more of the associated listings. In addition, when used in this application, the term "comprise" and its variations "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, an element defined by the sentence "comprising a..." does not exclude the presence of other identical elements in the process, method or device that includes the element. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments can be referenced to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can be referred to the description of the method part.

[0108] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. The technicians will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0109] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices and equipment) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units may be merely a logical functional division. In actual implementation, other divisions may be used, such as combining or integrating multiple units or components into another system, or omitting or disabling some features. Furthermore, the coupling or direct coupling or communication connection shown or discussed between each other may be through some interface, or indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to implement the embodiments according to actual needs. Furthermore, the functional units in the embodiments disclosed herein may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit.

[0110] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architectures, functions and operations of the systems, methods and computer program products according to the embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of the code, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical functions. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, which can depend on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in an order different from that disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A video streaming media bit rate adaptation method based on expert guidance, characterized in that: include: Obtain a mixed throughput trajectory dataset, initialize the neural network, and train the initial model using deep reinforcement learning on the dataset. Cluster the mixed throughput trajectory dataset into multiple network subsets. Use the initial model as the pre-training basis for the expert model. Train the expert model on each subset, record the state-action decision pairs of each expert model, and store them in the expert experience pool. Generate virtual video sessions based on the mixed throughput trajectory dataset and collect the state-action-reward triples of each video clip during the transmission process; Calculate the cumulative reward and advantage value based on the state-action-reward triplet, and extract the state-action decision pairs of the expert model of the network subset corresponding to the virtual video session from the expert experience pool. Combine the state-action-reward triplet, cumulative reward, advantage value, and the state-action decision pair of the expert model to form a training tuple; A batch of training tuples are randomly sampled as training samples to simultaneously optimize the Actor network and Critic network to obtain a universal ABR model.

2. The method for video streaming media bit rate adaptation based on expert guidance according to claim 1, characterized in that: The clustering of the mixed throughput trajectory dataset includes: performing unsupervised clustering based on network characteristics, wherein the network characteristics include average throughput and network volatility indicators; Each network subset represents a type of network environment.

3. The method for video streaming media bit rate adaptation based on expert guidance according to claim 2, characterized in that: The specific methods for recording the state-action decision pairs of each expert model and storing them in the expert experience pool include: Obtain the expert model trained on each subset, test each expert model in its own network subset, record its decision strategy as a state-action decision pair, i.e., the environment state and the corresponding bit rate selection, and store it in the expert experience pool.

4. The method for video streaming media bit rate adaptation based on expert guidance according to claim 1, characterized in that: The cumulative reward calculation formula is as follows: , in, Indicates the The cumulative reward of each segment, Indicates the Rewards for a fragment, represents the discount factor, Indicates the total number of segments in the video session; The advantage value calculation formula is as follows: ， in, represents the advantage value, Indicates input status The output of the Critic network is the expected reward.

5. The method for video streaming media bit rate adaptation based on expert guidance according to any one of claims 1 to 4, characterized in that: Jointly optimize the Actor network and the Critic network through gradient descent; The objective function of the Critic network is: ， in, represents the cumulative reward in the m-th tuple, represents the input environment state in the mth tuple, By adjusting the parameters of the model Minimize the following objective function, Represents all tuples; The Actor network objective function: ， in, is the weight parameter, is the maximum advantage value of the current batch, Indicates that the status Next Actor select action The probability of represents the action in the mth tuple, represents the advantage value in the m-th tuple, represents the state of the expert model in the mth tuple, represents the decision of the expert model in the mth tuple, By adjusting the parameters of the model Maximize the following objective function.

6. The method for video streaming media bit rate adaptation based on expert guidance according to claim 1, characterized in that: The ABR model adopts the Actor model and receives six-dimensional environment states: the download throughput of the last N video segments, the download time of the last N video segments, the optional size of the next video chunk, the bitrate decision of the last video chunk, the current buffer occupancy, and the number of remaining video segments in the current streaming session.

7. The method for video streaming media bit rate adaptation based on expert guidance according to claim 6, characterized in that: The specific structure of the Actor model is as follows: The first three input states are processed by separate one-dimensional convolutional layers, each with 128 filters and a filter size of 4; the last three input states are processed by fully connected layers, each containing 128 neurons and using a ReLU activation function; the features extracted from all input layers are concatenated and processed through a shared hidden fully connected layer; the output layer applies a SoftMax activation function to generate a probability distribution of available bitrate versions.

8. A video streaming media bit rate adaptive system based on expert guidance, characterized in that: include: Video server: stores video chunks with multiple bitrate versions; An ABR server deploys a general ABR model trained using the expert-guided video streaming bitrate adaptation method described in any one of claims 1 to 7, and responds to client decision requests. Client: collects network status in real time and sends it to the ABR server, and decides to download video chunks based on the bitrate returned; Communication module: used for data transmission between the client and the ABR server and video server.

9. A video streaming media bit rate adaptive device based on expert guidance, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to execute the video streaming media bit rate adaptation method based on expert guidance according to any one of claims 1 to 7 when running the program instructions.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the program is executed by a processor, the method for video streaming media bit rate adaptation based on expert guidance as described in any one of claims 1 to 7 above is implemented.

Images