Low-latency video stream code rate adaptation method and system based on offline meta-reinforcement learning

Abstract

The application provides a low-latency video stream code rate adaptation method and system based on offline meta-reinforcement learning, comprising: constructing an offline expert trajectory dataset; constructing a code rate adaptation strategy network, taking a video stream transmission state as input, and outputting a download code rate selection result and a network throughput correction value; using a meta implicit Q learning algorithm to perform double-layer alternating optimization on the strategy network, wherein the inner layer updates the policy parameters of the strategy network based on the offline expert trajectory dataset, and the outer layer further adjusts the policy network parameters through meta-reinforcement learning to improve the adaptation ability of the policy network to unseen tasks. The application integrates the experience of various expert algorithms through offline reinforcement learning, does not need online trial-and-error exploration, can realize high generalization and self-adaptation ability in various target latency and complex network environments, and finally realizes high quality and smooth playing of video streams in a low-latency live streaming scene.

Description

Technical Field

[0001] This application relates to the field of video communication and network optimization technology, specifically to a low-latency video stream bitrate adaptation method and system based on offline meta-reinforcement learning. Background Technology

[0002] Currently, with the rapid development of video streaming services, live streaming services account for an increasingly large proportion of internet data traffic. Especially on platforms like Douyin, Kuaishou, and Bilibili, low-latency live streaming has become a core requirement for user interaction. Traditional HTTP adaptive streaming media uses dynamic adaptive streaming transmission technology to provide multiple bitrate versions of video to adapt to different network conditions. However, compared to video-on-demand, low-latency live streaming systems are subject to more stringent end-to-end latency constraints (typically less than 2 seconds) and extremely small buffer space limitations (typically less than 1 second). This forces the system to achieve a dynamic balance between network throughput estimation, bitrate decision-making, and playback speed adjustment within a very short time.

[0003] Existing bitrate adaptive algorithms mainly fall into two categories: rule-based heuristic methods and learning-based intelligent algorithms. The former adjusts the bitrate through empirical rules but struggles to maintain stability in dynamic network environments; the latter uses reinforcement learning or neural networks for online decision-making, improving experience quality in some scenarios, but its generalization ability is weak, and it requires extensive online trial and error in low-latency live streaming environments, resulting in high sampling costs. Furthermore, existing algorithms generally assume relatively stable network dynamics, lacking adaptability to changes in target latency and heterogeneous network conditions. Therefore, achieving joint optimization of throughput estimation and bitrate adaptation in low-latency scenarios, and realizing unified generalization under multiple network conditions, is a pressing problem that current technologies need to address.

[0004] A search revealed patent application number 202411093093.7, which discloses a short video bitrate adaptive system based on meta-learning, achieving personalized adaptation for users through a combination of offline training and online learning. However, this solution primarily targets short video pre-loading scenarios, relying on online collection of state-action-reward sequences for model updates, which carries the risk of trial and error and makes it difficult to guarantee playback stability in the initial stage. It does not consider the strict end-to-end latency constraints in low-latency transmission scenarios, lacks a dynamic correction mechanism for network throughput estimation bias, and does not introduce fine-grained value function modeling. Therefore, it cannot solve the technical challenges of achieving online trial-and-error-free, high generalization ability, and accurate bitrate control in complex network environments. Summary of the Invention

[0005] To address one of the shortcomings of existing technologies, the purpose of this application is to provide a low-latency video stream bitrate adaptation method and system based on offline meta-reinforcement learning.

[0006] A first aspect of this application provides a low-latency video stream bitrate adaptation method based on offline meta-reinforcement learning, comprising: Construct an offline expert trajectory dataset; Construct a bitrate adaptation strategy network, taking the video stream transmission status as input, and outputting the download bitrate selection result and the network throughput adjustment value; The policy network is optimized by alternating inner and outer layers using a meta-implicit Q-learning algorithm. The inner layer updates the policy parameters of the policy network under different task scenarios based on the offline expert trajectory dataset using implicit Q-learning. The outer layer performs global updates of the policy network parameters through meta-reinforcement learning.

[0007] Optionally, constructing the offline expert trajectory dataset includes: From low-latency video stream scenarios under different target latency and network transmission conditions, the bitrate decision trajectories of various existing low-latency video stream bitrate adaptation algorithms are collected; Extract decision record points from each bitrate decision trajectory. Each decision record point is represented as a triplet containing video stream transmission status, corresponding bitrate selection action, and decision reward information. The video stream transmission status includes network transport layer information and player status information. The network transport layer information includes network throughput estimate, end-to-end latency, and relative latency. The player status information includes player buffer occupancy rate, buffer change rate, playback rate, segmented stuttering time, requested download bitrate, actual received bitrate, and normalized decision interval. The bitrate selection action refers to the download bitrate of the selected video segment at the corresponding moment; The decision reward is determined based on user experience quality indicators, taking into account the selected bitrate, playback stuttering time, end-to-end transmission latency, playback rate stability, and bitrate switching amplitude between adjacent video segments, and is obtained through weighted calculation using preset weighting coefficients.

[0008] Optionally, a meta-implicit Q-learning algorithm is used to perform alternating inner and outer layer optimization on the policy network. The inner layer, based on the offline expert trajectory dataset, uses implicit Q-learning to update the policy parameters of the policy network under different task scenarios. The outer layer performs global updates to the policy network parameters through meta-reinforcement learning, including: The offline expert trajectory dataset is divided into categories based on the target latency of the video stream playback and network conditions. A number of different subtasks and subtask datasets; Each training cycle consists of two phases: an inner cycle and an outer cycle. In the inner loop phase, for each subtask, training samples are sampled from the subtask dataset, and several steps of gradient updates are performed on the policy network based on the implicit Q-learning method to obtain local policy parameters adapted to the subtask. In the outer loop phase, training samples are also independently sampled from each subtask. The performance feedback of the updated local policy parameters on each task is used to calculate the meta-policy gradient across tasks. Based on the meta-policy gradient, the parameters of the policy network before the start of this inner loop are globally updated.

[0009] Optionally, in the inner loop phase, for each subtask, training samples are sampled from the subtask dataset, and several steps of gradient updates are performed on the policy network based on the implicit Q-learning method to obtain local policy parameters adapted to the subtask, including: Construct state-value function networks and state-action-value function networks; For each subtask, the implicit Q-learning algorithm is used to train the state value function network and the state-action value function network based on the training samples sampled from the subtask. After the two value function networks are trained, the advantage function values ​​of each state-action pair are calculated based on the same sample set. The training loss function of the policy network is constructed based on the calculated advantage function value, and the parameters of the policy network are optimized and updated based on the training loss function.

[0010] Optionally, when training the state-value function network and the state-action value function network, the decision rewards in the training samples are normalized, that is, the reward values ​​are scaled according to the time interval between two adjacent bitrate decisions, so as to reduce the noise impact caused by long-term decision accumulation.

[0011] Optionally, the training loss function is:

[0012] in, Represents the loss function. Represents the policy network, Let be the training samples for the inner loop, and s represent the state. This represents the discrete download bitrate output value. Represents the dominance function. , For state-value state networks, Representation of state-action value function network, and These are the weighting coefficients. This is a correction amount.

[0013] Optionally, in the outer loop phase, training samples are also independently sampled from each subtask, and the performance feedback of the updated local policy parameters on their respective tasks is used to calculate the cross-task meta-policy gradient. Based on this policy gradient, the parameters of the policy network before the start of the current inner loop are globally updated, including: For each subtask, based on the local parameters updated in its inner loop, the training samples of that subtask are resampled, and the gradient of the corresponding training loss function with respect to the policy network parameters is calculated. Aggregate the gradients across all subtasks to form a cross-task meta-gradient; The meta-gradient is used to uniformly update the policy network parameters, state value function network parameters, and state-action value function network parameters before the start of each inner loop, resulting in updated global model parameters, which are then used for the next round of inner and outer loop training.

[0014] Optionally, the method further includes: During the deployment phase, the video server divides the live video stream into multiple video segments and encodes each segment into versions with different bitrates. The player collects the current network transport layer status and player buffer status information in real time, and inputs them into the bitrate adaptation strategy network that has completed alternating optimization training. The strategy network outputs the adjustment values ​​of the target download bitrate and the estimated network throughput, and requests video segments with the corresponding bitrate based on the output results.

[0015] A second aspect of this application provides a low-latency video stream bitrate adaptation system based on offline meta-reinforcement learning, comprising: Network construction module: Constructs a bitrate adaptation strategy network that takes video stream transmission status as input and download bitrate selection and network throughput correction value as output; Data acquisition module: Constructs an offline expert trajectory dataset; Network training module: The policy network is optimized by alternating inner and outer layers using a meta-implicit Q-learning algorithm. The inner layer is based on the offline expert trajectory dataset and uses implicit Q-learning to update the policy parameters of the policy network under different task scenarios. The outer layer uses meta-reinforcement learning to globally update the policy network parameters.

[0016] In a third aspect, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods for low-latency video stream bitrate adaptation based on offline meta-reinforcement learning.

[0017] This application presents a low-latency video stream bitrate adaptation method based on offline meta-reinforcement learning. By constructing a bitrate adaptation policy network that takes network and player state information as input and outputs optimal bitrate selection and throughput adjustment, and using a meta-implicit Q-learning algorithm for parameter training, it can integrate knowledge from multiple expert adaptive bitrate algorithms in an offline environment. This achieves unified adaptive optimization under different target latency and heterogeneous network conditions, ultimately obtaining an optimal decision policy with cross-scene generalization capabilities. This method can complete policy training without online trial-and-error sampling, thus significantly reducing the training cost of reinforcement learning models.

[0018] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0019] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating a low-latency video stream bitrate adaptation method based on offline meta-reinforcement learning according to an exemplary embodiment; Figure 2 This is a schematic diagram illustrating the framework structure of a low-latency video stream bitrate adaptation method based on offline meta-reinforcement learning, according to an exemplary embodiment. Figure 3 This is a flowchart illustrating the deployment of a trained policy network according to an exemplary embodiment; Figure 4 This is a structural diagram of a low-latency video stream bitrate adaptation system based on offline meta-reinforcement learning. Detailed Implementation

[0020] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0021] Terminology Explanation: Meta-implicit Q-learning algorithm is a two-layer offline reinforcement learning method that combines implicit Q-learning with meta-learning. The inner layer uses IQL to optimize local policies on multiple sub-tasks, while the outer layer updates global initialization parameters through a meta-learning mechanism to improve the model's adaptability to unknown environments.

[0022] To facilitate understanding of this invention, a brief description of the system architecture of the video server and client is provided first. In low-latency live streaming scenarios, the server employs an HTTP-based chunked transmission mechanism, dividing the real-time video stream into multiple segments, and further subdividing them into data chunks with a granularity of approximately 33ms, to support rapid response to network changes. Simultaneously, each segment is pre-encoded into multiple bitrate versions, such as {360p@200 Kbps, 480p@600 Kbps, 720p@1000 Kbps}, adapting to adaptive downloading under dynamic network conditions. The client side includes three modules: throughput estimation, bitrate adaptive decision-making, and playback speed control. The playback buffer is strictly constrained by the target end-to-end latency (typically 1–2.5s), and playback speed control is used to maintain latency stability.

[0023] However, existing rate-adaptive algorithms struggle to achieve stable and efficient performance in such low-latency architectures: rule-based methods rely on empirical thresholds and are poorly adaptable to complex network fluctuations; while learning-based methods can improve the quality of experience in specific scenarios, they generally rely on online trial and error, have high sampling costs, weak generalization ability, and often assume stable network states, lacking the ability to jointly model and collaboratively optimize target latency changes and heterogeneous network conditions.

[0024] To address the aforementioned issues, this application provides a low-latency video stream bitrate adaptation method based on offline meta-reinforcement learning, thereby resolving these problems.

[0025] like Figure 1 As shown in one embodiment of this application, a low-latency video stream bitrate adaptation method based on offline meta-reinforcement learning includes the following steps: S100, construct an offline expert trajectory dataset; S200 constructs a bitrate adaptation strategy network, taking the video stream transmission status as input and outputting the download bitrate selection result and network throughput adjustment value; S300, the policy network is optimized by alternating inner and outer layers using a meta-implicit Q-learning algorithm. The inner layer updates the policy parameters of the policy network under different task scenarios based on the offline expert trajectory dataset using implicit Q-learning. The outer layer performs global updates on the policy network parameters through meta-reinforcement learning.

[0026] The S100-S300 implementations provided in this application effectively overcome the problems of high trial-and-error costs, insufficient cross-environment generalization ability, and lack of collaborative optimization between network state and target latency in low-latency video streaming scenarios by constructing an offline expert trajectory dataset containing multi-algorithm decision-making experience, designing a policy network structure with dual outputs of bitrate selection and throughput adjustment values, and adopting a two-layer optimization training mechanism using meta-implicit Q-learning. Since the input state of the policy function includes the end-to-end target latency requirement for video transmission, and the offline training samples are decision trajectories that meet the target latency conditions output by the expert policy, the bitrate adaptation policy network obtained through offline training can effectively reduce video stuttering and quality fluctuations while ensuring end-to-end low-latency stability, thereby maximizing the overall viewing experience quality for users and improving the bandwidth utilization and system robustness of low-latency video streaming transmission.

[0027] To ensure the completeness and task coverage of the offline expert trajectory dataset, in some specific embodiments of this application, S100, constructing the offline expert trajectory dataset may include the following steps: S101: Collect the bitrate decision trajectory of existing low-latency video stream bitrate adaptation expert algorithms (such as DYN, LoLp MAN, LoLp RAN, LoLp, L2A, GCC and other published algorithms) from low-latency video stream scenarios with different target latency and network transmission conditions. S102, obtain the first [item] in each trajectory. Each decision record point is represented as a triple. state ,action ,award .

[0028] Specifically, the bitrate decision trajectory refers to a trajectory containing state-action pairs obtained during a live video playback process, using an expert algorithm to make bitrate decisions. This trajectory can be used for offline reinforcement learning training. The Kth decision record point refers to the network state and the action executed by the expert algorithm at a certain decision moment in the above decision trajectory.

[0029] Among them, video stream transmission status This includes network transport layer information such as network throughput estimates, end-to-end latency, and relative latency obtained by any method, as well as player status information such as player buffer size, buffer change rate, playback speed, segmented stuttering time, requested bitrate, actual bitrate, and normalized decision interval. action Download bitrate for video segments selected by expert algorithms; The decision reward is defined as the Quality of User Experience (QoE) of the currently downloaded video segment, and is defined as follows:

[0030] in, For the selected download bitrate, To account for playback stuttering time, For end-to-end transmission delay, For playback speed, For changes in bitrate between adjacent segments, These are the weighting coefficients.

[0031] The embodiments described above in this application construct a structured dataset containing state-action-reward triples by collecting decision trajectories of multiple algorithms under multi-objective latency and heterogeneous network conditions. This dataset effectively represents the correlation between network dynamics and playback requirements, enabling the trained policy network to adapt to unknown network environments without online trial and error, thereby improving cross-scenario decision stability and user experience consistency.

[0032] To achieve coordinated optimization of bitrate selection and bandwidth estimation, in some specific embodiments of this application, the bitrate adaptation policy network constructed in S200 is a network based on... Neural networks with parameters Its input is the video stream transmission status. The output includes the current video segment download bitrate and network throughput estimate adjustments, defined as:

[0033] in, This represents the discrete download bitrate output value. This represents the adjustment to the current network throughput estimate; the corrected network throughput estimate is... , This is an estimate of the existing network throughput. The corrected prediction will replace the original network throughput estimate in the state to help the policy network output actions.

[0034] Specifically, the structure of the policy network is as follows: 1) Input layer: A 256-dimensional feedforward encoder to extract basic features.

[0035] 2) Second layer: Two cascaded MiniGRU structures are used to capture the temporal dependency of bit rate and bandwidth changes.

[0036] 3) Third layer: A 256-dimensional feedforward layer further transforms the features. Residual blocks are added to enhance feature representation and ensure gradient stability.

[0037] 4) Output layer: One fully connected layer outputs discrete actions, and another fully connected layer with an added tanh activation function to obtain the throughput adjustment value.

[0038] The bitrate adaptation strategy network in the above embodiments of this application can simultaneously output bitrate decision and throughput correction value during the inference process, forming a collaborative optimization mechanism for bitrate selection and bandwidth estimation, effectively eliminating throughput estimation deviation caused by network fluctuations, and ensuring image quality stability and playback smoothness under low latency constraints.

[0039] To maintain higher video quality and smoother playback under low latency constraints, in some specific embodiments of this application, in S300, a meta-implicit Q-learning algorithm is used to perform two-layer alternating optimization on the policy network. The inner layer, based on an offline expert trajectory dataset, uses implicit Q-learning to update the policy parameters of the policy network under different task scenarios. The outer layer uses meta-reinforcement learning to globally update the policy network parameters. For example... Figure 2 As shown, the following steps can be taken: S301, the offline expert trajectory dataset is divided into categories based on the target latency of the video stream playback and network conditions. Different sub-datasets and their corresponding subtasks ; S302, each alternation training session includes two phases: internal and external cycle training. S3021, in the inner loop phase, for each subtask, sample training samples. The policy network is updated with gradients through the training samples to learn the specific update parameters for the current subtask. S3022, In the outer loop phase, training samples are also sampled for each subtask. Then, based on the training samples of all subtasks, the gradient values ​​of the cross-task meta-policy are calculated, and the policy network is further updated.

[0040] In the embodiments described above, the alternating inner and outer loop training enables the policy network to possess both task-specificity and generalization ability: the inner loop optimizes local policies for specific network conditions, while the outer loop extracts commonalities across multiple tasks to update initial parameters. This mechanism allows the model to quickly adapt to unknown network environments after deployment without online learning, maintaining a balance between high image quality and smooth playback under low latency constraints.

[0041] In order to eliminate offline data bias and suppress decision noise, in some specific embodiments of this application, the preferred steps of S3021 are as follows: First, construct an additional state value function. With state-action value function s represents the state. Indicates the bitrate selection action; Then, within each subtask, the implicit Q-learning (IQL) offline reinforcement learning algorithm is used, based on sampled training samples. For the two value function networks above , Training is performed; however, the decision rewards in the training samples need to be normalized to reduce the noise impact of multiple bitrate decisions. The correction method is as follows: ,

[0042] in, This indicates the corrected normalized decision reward. Indicates the reward for the original decision. Indicates the interval between two consecutive decisions. and These are normalization parameters; Finally, based on the sampled training samples Policy network Parameter optimization is performed, and the training loss function is:

[0043] Among them, the dominant function , and These are the weighting coefficients.

[0044] During the inner loop, the initial parameters are For each subtask, the updated parameters are: The embodiments described above employ an IQL dual-value function architecture and reward normalization to eliminate offline data bias and suppress decision noise, ensuring the accuracy of bitrate selection and the stability of network throughput estimation in low-latency scenarios.

[0045] To further obtain the global parameters of the adjustment policy network, in some specific embodiments of this application, the preferred steps of S3022 are as follows: The outer loop training phase is based on training samples from all subtasks. Policy network parameters and two value network parameters A second update will be performed, including: Based on training samples from all subtasks Calculate the meta gradient:

[0046] in, , This represents the training loss function for the policy network and the value network. The parameters of the three networks after the outer loop update are:

[0047] In this embodiment, the outer loop updates network parameters through cross-task meta-gradient aggregation, enabling the policy network to generalize across scenarios and adapt to unknown network environments without online learning, thus ensuring the transmission stability and playback quality of low-latency video streams. Simultaneously, the throughput-aware regularization term introduced in the policy loss function effectively constrains network bandwidth estimation bias, suppresses bitrate decision oscillations, and reduces rebuffering events, achieving coordinated optimization of video quality and playback smoothness under strict latency constraints.

[0048] After completing the above training, the resulting policy network can maximize the viewing experience under low experimental constraints in different network scenarios. In some specific embodiments of this application, the trained policy model is deployed, such as... Figure 3 As shown, it includes the following steps: S11, on the video server side, divides the live video stream into several video segments of fixed playback length in time, and encodes each video segment into multiple versions with different bitrates and caches them; S12, On the video player, the current network status information and player status information are obtained in real time and input into the bitrate adaptation strategy network; the bitrate strategy network is constructed using the method described in the above embodiment; S13, the bitrate adaptation strategy network infers the optimal bitrate selection and current network throughput estimate adjustment value of the video segment to be downloaded based on network status information and player status information, and sends the optimal bitrate version information and segment request information to the video server. S14, the video server sends the video segment with the corresponding bitrate to the player based on the optimal bitrate version information; the player updates the network throughput estimation result, network transport layer status and player status based on the received video segment, realizing the joint optimization of low-latency video stream download bitrate and throughput estimation, thereby achieving high video quality and smooth playback under low latency constraints.

[0049] In the above embodiments of this application, the deployed policy network outputs the optimal bitrate and throughput correction value in real time, realizing the collaborative optimization of bitrate decision-making and network awareness. Under the low latency constraint of 1-2.5 seconds, it significantly improves the stability of video quality and the smoothness of playback. It can adapt to various network environments without online learning, ensuring the maximum quality of user experience.

[0050] Based on the same technical concept, other embodiments of this application provide a low-latency video stream bitrate adaptation system 100 based on offline meta-reinforcement learning, such as... Figure 4 As shown, it includes: Network construction module 110: Constructs a bitrate adaptation strategy network with video stream transmission status as input and download bitrate selection and network throughput correction value as output; Data acquisition module 120: Constructs an offline expert trajectory dataset; Network training module 130: The meta-implicit Q-learning algorithm is used to perform two-layer alternating optimization of the policy network, with inner and outer layers. The inner layer is based on an offline expert trajectory dataset and uses implicit Q-learning to update the policy parameters of the policy network under different task scenarios. The outer layer uses meta-reinforcement learning to globally update the policy network parameters.

[0051] The specific implementation techniques of each module / unit in the above examples of this application can be referred to the steps of the low-latency video stream bitrate adaptation method of offline meta-reinforcement learning in the above embodiments, and will not be repeated here.

[0052] Based on the same technical concept, other embodiments of this application propose a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described herein.

[0053] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0054] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0055] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0056] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0057] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0058] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0059] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0060] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0061] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0062] The terms "comprising" and "having," and any variations thereof, in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or devices.

[0063] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A low-latency video stream bitrate adaptation method based on offline meta-reinforcement learning, characterized in that, include: Construct an offline expert trajectory dataset; Construct a bitrate adaptation strategy network, taking the video stream transmission status as input, and outputting the download bitrate selection result and the network throughput adjustment value; The policy network is optimized using a meta-implicit Q-learning algorithm with alternating inner and outer layers. The inner layer, based on the offline expert trajectory dataset, updates the policy parameters of the policy network under different task scenarios using implicit Q-learning. The outer layer performs a global update of the policy network parameters through meta-reinforcement learning. Specifically: The offline expert trajectory dataset is divided into categories based on the target latency of the video stream playback and network conditions. A number of different subtasks and subtask datasets; Each training cycle consists of two phases: an inner cycle and an outer cycle. In the inner loop phase, for each subtask, training samples are sampled from the subtask dataset, and several steps of gradient updates are performed on the policy network based on the implicit Q-learning method to obtain local policy parameters adapted to that subtask, including: Construct state-value function networks and state-action-value function networks; For each subtask, an implicit Q-learning algorithm is used to train the state value function network and the state-action value function network based on the training samples sampled from the subtask. After the two value function networks are trained, the advantage function values ​​of each state-action pair are calculated based on the same sample set. The training loss function of the policy network is constructed based on the calculated advantage function value, and the parameters of the policy network are optimized and updated based on the training loss function. In the outer loop phase, training samples are also independently sampled from each subtask. The performance feedback of the updated local policy parameters on each task is used to calculate the meta-policy gradient across tasks. Based on the meta-policy gradient, the parameters of the policy network before the start of this inner loop are globally updated. The training loss function is: ； in, Let π represent the loss function, and let π represent the policy network. The parameters represent the policy network. These are training samples for the inner loop. Indicates state, This represents the discrete download bitrate output value. Indicating the dominance function , For state-value state networks, Representation of state-action value function network, and These are the weighting coefficients. This is an estimate of the existing network throughput. This represents the adjustment value for the current network throughput estimate.

2. The low-latency video stream bitrate adaptation method based on offline meta-reinforcement learning according to claim 1, characterized in that, The construction of the offline expert trajectory dataset includes: From low-latency video stream scenarios under different target latency and network transmission conditions, the bitrate decision trajectory of various low-latency video stream bitrate adaptation algorithms is collected; Extract decision record points from each bitrate decision trajectory. Each decision record point is represented as a triplet containing video stream transmission status, corresponding bitrate selection action, and decision reward information. The video stream transmission status includes network transport layer information and player status information. The network transport layer information includes network throughput estimate, end-to-end latency, and relative latency. The player status information includes player buffer occupancy rate, buffer change rate, playback rate, segmented stuttering time, requested download bitrate, actual received bitrate, and normalized decision interval. The bitrate selection action refers to the download bitrate of the selected video segment at the corresponding moment; The decision reward is determined based on user experience quality indicators, taking into account the selected bitrate, playback stuttering time, end-to-end transmission latency, playback rate stability, and bitrate switching amplitude between adjacent video segments, and is obtained through weighted calculation using preset weighting coefficients.

3. The low-latency video stream bitrate adaptation method based on offline meta-reinforcement learning according to claim 1, characterized in that, When training the state-value function network and the state-action value function network, the decision rewards in the training samples are normalized, that is, the reward value is scaled according to the time interval between two adjacent bitrate decisions to reduce the noise impact caused by long-term decision accumulation.

4. The low-latency video stream bitrate adaptation method based on offline meta-reinforcement learning according to claim 1, characterized in that, In the outer loop phase, training samples are independently sampled from each subtask. The performance feedback of the updated local policy parameters on each task is used to calculate the cross-task meta-policy gradient. Based on this policy gradient, the parameters of the policy network before the start of the current inner loop are globally updated, including: For each subtask, based on the local parameters updated in its inner loop, the training samples of that subtask are resampled, and the gradient of the corresponding training loss function with respect to the policy network parameters is calculated. Aggregate the gradients across all subtasks to form a cross-task meta-gradient; The meta-gradient is used to uniformly update the policy network parameters, state value function network parameters, and state-action value function network parameters before the start of each inner loop, resulting in updated global model parameters, which are then used for the next round of inner and outer loop training.

5. The low-latency video stream bitrate adaptation method based on offline meta-reinforcement learning according to claim 1, characterized in that, Also includes: During the deployment phase, the video server divides the live video stream into multiple video segments and encodes each segment into versions with different bitrates. The player collects the current network transport layer status and player buffer status information in real time, and inputs them into the bitrate adaptation strategy network that has completed alternating optimization training. The strategy network outputs the adjustment values ​​of the target download bitrate and the estimated network throughput, and requests video segments with the corresponding bitrate based on the output results.

6. A low-latency video stream bitrate adaptation system based on offline meta-reinforcement learning, used to implement the method of claim 1, characterized in that, include: Network construction module: Constructs a bitrate adaptation strategy network that takes video stream transmission status as input and download bitrate selection and network throughput correction value as output; Data acquisition module: Constructs an offline expert trajectory dataset; Network training module: The policy network is optimized by alternating inner and outer layers using a meta-implicit Q-learning algorithm. The inner layer is based on the offline expert trajectory dataset and uses implicit Q-learning to update the policy parameters of the policy network under different task scenarios. The outer layer uses meta-reinforcement learning to globally update the policy network parameters.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-5.

Images