A cloud and local decoupled adaptive bitrate streaming optimization method and apparatus

Abstract

The application relates to a cloud and local decoupled adaptive code rate streaming optimization method and device, the method constructs a reinforcement learning architecture of a strategy network and a light neural network value network based on a pre-training large language model code rate scheduling model in the cloud, and performs online interactive iterative training by using network real bandwidth time sequence track data, the model is allowed to actively explore and learn in a simulated real environment, so that a better strategy can be obtained; through a knowledge distillation technology, the code rate scheduling time sequence decision capability of the trained large language model strategy network is migrated to a light child model, and the light child model is deployed on a terminal, the terminal device can perform code rate scheduling decisions similar to the large model under extremely low inference delay and memory occupation, which enables the terminal device to quickly output code rate decisions and perform video slice downloading, thereby ensuring the experience of users in a network fluctuation environment.

Description

Technical Field

[0001] This application relates to the field of streaming media optimization technology, and in particular to an adaptive bitrate streaming media optimization method and apparatus that decouples cloud and local operations. Background Technology

[0002] With the continuous development of large language model technology, its powerful sequence modeling capabilities have brought a brand-new solution to adaptive bitrate (ABR) streaming media transmission. However, the application of large models in ABR inherently presents a contradiction between heavy-duty intelligence and lightweight deployment: While large models possess the capability for long-sequence inference, which aligns well with ABR tasks, their sheer number of parameters and excessively long inference latency present significant challenges. Conversely, lightweight ABR tasks face stringent constraints—inference latency must be kept in the millisecond range, and memory usage must be extremely low. Furthermore, most existing algorithms for adaptive bitstream tasks based on large models employ an offline reinforcement learning paradigm. This approach has significant drawbacks: the model relies entirely on static expert datasets for training, failing to actively explore the environment, thus limiting the performance of large models to the ceiling of suboptimal policies. Summary of the Invention

[0003] To achieve the above objectives, the first aspect of this application proposes an adaptive bitrate streaming optimization method that decouples cloud and local operations, the method comprising: A reinforcement learning architecture for constructing a policy network and a value network in the cloud; wherein the policy network is a pre-trained large language model bitrate scheduling model, and the value network is a lightweight neural network; An adaptive bitrate simulation environment is constructed using real network bandwidth time-series trajectory data. Based on a near-end policy optimization algorithm, the policy network and value network are trained online interactively and iteratively in the cloud to update the parameters of the policy network and the value network, thereby obtaining the trained policy network. The state space in the online interactive iterative training includes observable states, which include playback state and network state. The action space includes optional bitrate decisions. The trained policy network is used as the teacher model, and the lightweight model is used as the student model. The large model bitrate scheduling timing decision-making capability corresponding to the trained policy network is transferred to the student model to obtain the distilled lightweight model, which is then deployed on the terminal. Using real-time acquired current playback status and network status data as input data, the input data is input into the distilled lightweight model in the terminal to obtain the output bitrate decision, and video segment download is performed according to the bitrate decision.

[0004] The method provided in this application has at least the following beneficial effects: This method overcomes the limitations of offline reinforcement learning by constructing a reinforcement learning architecture in the cloud, based on a policy network and a lightweight neural network value network using a pre-trained large language model bitrate scheduling model. It then utilizes real-time bandwidth trajectory data for online interactive iterative training. This approach allows the model to actively explore and learn in a simulated real-world environment, thereby acquiring better policies. The cloud training process simulates various complex network fluctuations, enabling the policy network to learn to make more robust and optimized bitrate decisions under different network conditions. Furthermore, through knowledge distillation, the bitrate scheduling timing decision-making capability of the trained large language model policy network is transferred to a lightweight student model and deployed on the terminal. The terminal device can then execute bitrate scheduling decisions similar to those of the large model with extremely low inference latency and memory usage. This allows the terminal device to quickly output bitrate decisions and execute video segment downloads after acquiring real-time playback and network status data, ensuring a smooth viewing experience for users even under fluctuating network conditions.

[0005] Compared to traditional ABR methods that directly deploy heavy models on the terminal or use simple rules, this method maintains high-performance decision-making while meeting the lightweight and real-time requirements of terminal devices, achieving a perfect combination of cloud intelligence and efficient local execution.

[0006] To achieve the above objectives, a second aspect of this application proposes a cloud- and local decoupled adaptive bitrate streaming media optimization apparatus, the apparatus comprising: The model building module is used to construct a reinforcement learning architecture of policy network and value network in the cloud; wherein, the policy network is a pre-trained large language model bitrate scheduling model, and the value network is a lightweight neural network. The network training module is used to construct an adaptive bitrate simulation environment using real network bandwidth time-series trajectory data, and to perform online interactive iterative training on the policy network and value network in the cloud based on the near-end policy optimization algorithm, so as to update the parameters of the policy network and the value network and obtain the trained policy network; wherein, the state space in the online interactive iterative training includes observable states, which include playback state and network state, and the action space includes optional bitrate decisions; The model distillation module is used to take the trained policy network as the teacher model and the lightweight model as the student model, and to transfer the large model bitrate scheduling timing decision-making capability corresponding to the trained policy network to the student model, so as to obtain the distilled lightweight model and deploy it on the terminal. The model inference module is used to input the real-time acquired current playback status and network status data into the distilled lightweight model in the terminal to obtain the output bitrate decision, and to perform video segment download based on the bitrate decision.

[0007] To achieve the above objectives, a third aspect of this application provides an electronic device comprising: at least one processor and a memory for communicatively connecting to the at least one processor; the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform: a cloud-and-local decoupled adaptive bitrate streaming optimization method as described in the first aspect.

[0008] To achieve the above objectives, a fourth aspect of this application also proposes a computer-readable storage medium storing computer-executable instructions for causing a computer to perform: a cloud-and-local decoupled adaptive bitrate streaming media optimization method as described in the first aspect.

[0009] It is understood that the beneficial effects of the second to fourth aspects compared with the related technologies are the same as the beneficial effects of the first aspect compared with the related technologies. Please refer to the relevant description in the first aspect above, which will not be repeated here. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A schematic diagram illustrating an adaptive bitrate streaming optimization method that decouples cloud and local operations, as provided in an embodiment of this application; Figure 2 A schematic diagram of a reinforcement learning architecture provided in an embodiment of this application; Figure 3 A schematic diagram of an adaptive bitrate streaming media optimization device that decouples cloud and local operations, provided as an embodiment of this application; Figure 4 A schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0013] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0014] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0015] Some embodiments of this application are described with reference to... Figure 1 One embodiment of this application provides an adaptive bitrate streaming media optimization method that decouples cloud and local operations, the method comprising: Step S110: Construct a reinforcement learning architecture of policy network and value network in the cloud; wherein, the policy network is a pre-trained large language model code rate scheduling model, and the value network is a lightweight neural network. Step S120: Construct an adaptive bitrate simulation environment using real network bandwidth time-series trajectory data, and perform online interactive iterative training of the policy network and value network in the cloud based on the near-end policy optimization algorithm to update the parameters of the policy network and value network, thereby obtaining the trained policy network; wherein, the state space in the online interactive iterative training includes observable states, which include playback state and network state, and the action space includes optional bitrate decisions. Step S130: Use the trained policy network as the teacher model and the lightweight model as the student model. Transfer the big model bitrate scheduling timing decision-making capability corresponding to the trained policy network to the student model to obtain the distilled lightweight model and deploy it on the terminal. Step S140: Using the real-time acquired current playback status and network status data as input data, the input data is input into the distilled lightweight model in the terminal to obtain the output bitrate decision, and video segment download is performed according to the bitrate decision.

[0016] For ease of understanding, the following explains some key terms in this embodiment: Reinforcement learning architecture is a machine learning paradigm that learns optimal policies through the interaction of an agent with its environment. The agent performs actions in the environment, which provides rewards and new states based on those actions. The agent learns how to make decisions by maximizing the accumulated rewards. In this method, reinforcement learning architecture is used to optimize adaptive bitrate scheduling for streaming media.

[0017] The policy network is a component of a reinforcement learning architecture, responsible for outputting a probability distribution of actions based on the current state or directly outputting an action. In this method, the policy network is specifically implemented as a pre-trained large language model bitrate scheduling model, which leverages the powerful sequence modeling capabilities of the large language model to handle complex bitrate scheduling tasks.

[0018] The value network is another component in reinforcement learning architectures, used to evaluate the value of a given state or state-action pair. In this approach, the value network is implemented as a lightweight neural network designed to provide auxiliary evaluation of the policy network's decisions, thereby guiding the training of the policy network.

[0019] An adaptive bitrate simulation environment is a simulated or real-world environment built for reinforcement learning training. It simulates factors such as network bandwidth changes and playback status during streaming media playback. The agent interacts within this environment to learn how to adjust the video bitrate according to environmental changes. This environment is constructed using real-time network bandwidth trajectory data, ensuring the realism and effectiveness of the training.

[0020] Proximal Policy Optimization (PPO) is a reinforcement learning algorithm that ensures training stability while achieving high sample efficiency by limiting the magnitude of policy updates. This algorithm is used for online interactive iterative training of the policy network and value network in the cloud to progressively optimize model performance.

[0021] Knowledge distillation is a model compression technique that transfers the knowledge of a large, complex teacher model to a small, lightweight student model. In this method, the trained policy network serves as the teacher model, and its bitrate scheduling timing decision-making capabilities are transferred to the lightweight student model, thereby achieving lightweight model deployment while maintaining performance.

[0022] This embodiment provides an adaptive bitrate streaming optimization method that decouples cloud and local operations, aiming to solve the deployment contradictions and limitations of offline training of large models in adaptive bitrate tasks.

[0023] In step S110, the method constructs a reinforcement learning architecture for a policy network and a value network in the cloud. The policy network is a pre-trained large language model bitrate scheduling model, such as NetLLM. The original core architecture of NetLLM includes three core modules: a multimodal encoder, a network output head, and a data-driven low-rank network adaptation module. The main function of the multimodal encoder is to map the heterogeneous network state features into high-dimensional latent vectors that the large language model can recognize. The network output head transforms the latent states output by the large language model into action probability distributions corresponding to different bitrate levels, effectively solving the model illusion problem. The data-driven low-rank network adaptation module injects a trainable low-rank matrix into the frozen backbone network of the large language model, significantly reducing the number of parameters and memory overhead in the online fine-tuning stage.

[0024] A value network can be a multilayer perceptron (MLP) used to evaluate the value of the current state. Value networks are implemented as lightweight neural networks with a small number of parameters and low computational overhead, which can efficiently assist in the training of policy networks.

[0025] In step S120, an adaptive bit rate simulation environment is constructed using real network bandwidth time-series trajectory data, and the policy network and value network are trained online interactively and iteratively in the cloud based on the near-end policy optimization algorithm.

[0026] In this embodiment, the environment is constructed using real network bandwidth time-series trajectory data, ensuring the authenticity and effectiveness of the training. Online interactive iterative training can employ various reinforcement learning algorithms; in this embodiment, the Proximal Policy Optimization (PPO) algorithm is used for training. This algorithm achieves high sample efficiency while ensuring training stability. During training, the state space comprises observable states, such as playback states (e.g., buffer length, number of downloaded segments) and network states (e.g., real-time bandwidth, latency). The action space can include preset fixed bitrate options; for example, the video can be encoded at multiple bitrate levels such as 1Mbps, 2Mbps, and 3Mbps.

[0027] In step S130, the method uses the trained policy network as the teacher model and the lightweight model as the student model, and transfers the big model bitrate scheduling timing decision-making capability corresponding to the trained policy network to the student model to obtain the distilled lightweight model and deploy it on the terminal.

[0028] The student model can be a neural network with a small number of parameters, such as a small recurrent neural network or a Transformer. Knowledge transfer can be achieved through various distillation techniques. For example, response distillation can be used to allow the student model to fit the output probability distribution of the teacher model. Alternatively, feature distillation can be used to allow the student model to learn the feature representations of the intermediate layers of the teacher model. In this way, the complex decision-making capabilities of a large model are compressed and transferred to a lightweight model, enabling the model to run efficiently on resource-constrained devices.

[0029] In step S140, the method uses real-time acquired current playback status and network status data as input data. This input data is fed into a distilled lightweight model in the terminal to obtain the output bitrate decision, and then the method executes video segment downloading based on the bitrate decision. For example, the real-time acquired playback status and network status data may include the current buffer length, video playback progress, network throughput, round-trip latency, etc., which can be collected through sensors or network interfaces on the terminal device. The input data can be directly input into the lightweight model or preprocessed before input, such as through normalization or feature encoding. The output bitrate decision can be a specific bitrate value, such as 2 Mbps. Alternatively, it can be an index of bitrate levels, such as selecting the third bitrate level. Executing video segment downloading based on the bitrate decision means that the terminal player will select the next video segment to download based on the bitrate given by the model. For example, if the model decides a high bitrate, a high-definition video segment is downloaded; if the decision is a low bitrate, a low-definition video segment is downloaded to adapt to the current network conditions.

[0030] The following example will provide a more detailed explanation of the above technical solution: Suppose a user is watching streaming video on a terminal device. The terminal device is located in an area with an unstable network environment and fluctuating bandwidth. To ensure a good viewing experience for the user, a method is needed that can adaptively adjust the video bitrate in real time.

[0031] First, a reinforcement learning architecture was built in the cloud. The policy network was implemented as a pre-trained large language model bitrate scheduling model, possessing powerful temporal decision-making capabilities. The value network, a lightweight neural network, was used to assist in the training of the policy network.

[0032] To simulate a real network environment, an adaptive bitrate simulation environment was constructed using bandwidth time-series trajectory data collected from actual networks. In this environment, the policy network and value network are trained iteratively online based on a near-end policy optimization algorithm. During training, the policy network makes bitrate decisions based on historically observable playback states (e.g., remaining video buffer time, number of downloaded segments) and network states (e.g., real-time network bandwidth, network latency). For example, when network bandwidth is high, the policy network may attempt to choose a higher bitrate; when network bandwidth decreases or the buffer is about to run out, the policy network will choose a lower bitrate. The value network evaluates the potential long-term benefits of these decisions and feeds them back to the policy network to guide its parameter updates. Through multiple rounds of iterative training, the parameters of the policy network and value network are continuously optimized, ultimately resulting in a fully trained policy network capable of making high-quality bitrate scheduling decisions at the large language model level.

[0033] However, directly deploying a large, trained language model to a user's terminal device is impractical due to the massive number of parameters and high inference latency, which fails to meet the lightweight and real-time requirements of terminal devices. Therefore, this method introduces a knowledge distillation step. The trained policy network is used as the teacher model, and a lightweight model (e.g., Transformer) is selected as the student model. Through knowledge distillation, the complex bitrate scheduling timing decision-making capabilities of the teacher model are transferred to the student model. Specifically, the student model learns to fit the bitrate decision-making behavior of the teacher model under various playback and network conditions, thereby significantly reducing the number of parameters and computational complexity while maintaining decision quality. The distilled lightweight model is then deployed to the user's terminal device.

[0034] When a user starts watching a video, the terminal device acquires real-time playback status (e.g., current buffer length of 5 seconds, duration of played video) and network status data (e.g., real-time network bandwidth of 3Mbps, network latency of 50ms). This real-time data is used as input to a distilled, lightweight model deployed on the terminal device. Based on the input data, the lightweight model quickly outputs a bitrate decision. For example, the model might decide that a video segment with a bitrate of 2Mbps should be downloaded. The terminal device, based on the bitrate decision, requests and downloads the corresponding bitrate video segment from the server. Because the model is lightweight, its inference latency is extremely low, enabling decisions to be made within milliseconds, thus ensuring smooth video playback and a good user experience.

[0035] Based on the above examples, in existing technologies, the application of large language models in adaptive code rate (ABR) tasks faces a contradiction between heavy intelligence and lightweight deployment. Specifically, large models have a large number of parameters and excessively long inference latency, failing to meet the stringent constraints of millisecond-level inference latency and extremely low memory usage required by ABR tasks. Furthermore, existing ABR algorithms based on large models mostly employ offline reinforcement learning paradigms, preventing the model from actively exploring the environment and limiting performance to suboptimal strategies.

[0036] This embodiment overcomes the limitations of offline reinforcement learning by constructing a reinforcement learning architecture in the cloud, based on a pre-trained large language model bitrate scheduling model and a lightweight neural network value network. It utilizes real-time bandwidth trajectory data for online interactive iterative training. Unlike offline training methods that rely entirely on static expert datasets, this method allows the model to actively explore and learn in a simulated real-world environment, thereby acquiring better policies. The cloud training process can simulate various complex network fluctuations, enabling the policy network to learn to make more robust and optimized bitrate decisions under different network conditions, rather than simply replicating existing suboptimal expert behavior.

[0037] Furthermore, this embodiment utilizes knowledge distillation technology to transfer the bitrate scheduling timing decision-making capability of the trained large language model policy network to a lightweight student model, which is then deployed on the terminal. The terminal device can then execute bitrate scheduling decisions similar to those of the large model with extremely low inference latency and memory usage. This allows the terminal device to quickly output bitrate decisions and execute video segment downloads after acquiring real-time playback and network status data, thus ensuring a smooth viewing experience for users even in environments with fluctuating network conditions. Compared to traditional ABR methods that directly deploy heavy models on the terminal or employ simple rules, this method maintains high-performance decision-making while meeting the lightweight and real-time requirements of the terminal device, achieving a perfect combination of cloud intelligence and efficient local execution.

[0038] In some embodiments, the reinforcement learning architecture further includes a prophetic reinforcement context module at the input of the value network. The prophetic reinforcement context module is used as privileged information to extract the size information of future video segments at different bitrates from a preset streaming media MPD file. The input to the value network consists of a spliced ​​vector formed by concatenating privileged information with observable states; The input to the policy network includes observable states.

[0039] The Oracle Enhancement Context Module (OAC) aims to provide forward-looking information for the value assessment of reinforcement learning. Its role is to assist the value network in more accurately evaluating the potential value of the current state by pre-acquiring details of possible future technical developments. The module can be a data parser that reads and processes streaming media metadata files, such as MPEG-DASH Media Presentation Description (MPD) files or HLS playlist files, extracting key future information. Alternatively, the module can be a lightweight data preprocessor that extracts structured future information from known streaming media content descriptions according to predefined rules before or during training. Privileged information refers to the size information of future video segments at different bitrates extracted from the pre-defined streaming media MPD file. This information is usually not available in real-time to the terminal during actual deployment, but it can be provided to the value network as a privilege or prior knowledge during the training phase. Privileged information can include the file sizes of N future video segments at different bitrates. For example, for a video stream, the MPD file lists the URL and size of each video segment at each bitrate level. The Oracle Enhancement Context Module can parse this information and organize it into a vector or matrix for use by the value network.

[0040] The input to the value network consists of a concatenated vector formed by combining privileged information and observable states. This input method allows the value network to consider not only historical playback and network states but also predictions of future video segment characteristics when evaluating the value of the current state. The concatenated vector is formed by concatenating the privileged information vector with the observable state. For example, if the observable state contains information such as playback buffer size, download speed, and packet loss rate, while the privileged information contains the volume of future segments at different bitrates, then these two vectors can be directly concatenated to form a longer input vector.

[0041] The input to the policy network includes observable states. When making bitrate decisions, the policy network relies solely on the terminal's real-time observable historical playback state and network state. This design ensures that after deployment on the terminal, the policy network can make decisions based solely on the information available to the terminal, which aligns with the limitations of terminal resources and information access in real-world application scenarios. Observable states can include information such as the current buffer state, recent download throughput, network latency, CPU utilization, and battery level.

[0042] This embodiment optimizes the training process of adaptive bitrate streaming media by introducing a prophetic enhanced context module into the reinforcement learning architecture and incorporating the privileged information extracted by the module into the input of the value network.

[0043] In some embodiments, each iteration of the online interactive iterative training includes: Obtain a pre-defined pool of expert experience; Read the observable state vector from the adaptive bitrate simulation environment; generate the action vector based on the observable state vector according to the policy network; execute the action vector in the adaptive bitrate simulation environment to obtain the observable state vector for the next round; store the observable state vector, action vector, observable state vector for the next round, and corresponding reward into a buffer. Acquire privileged information and input it into the value network based on the privileged information and the observable state vector to obtain the state value estimate output by the value network; calculate the advantage function based on the state value estimate and generate the state value target value based on the advantage function; Extract batch samples from the buffer and expert samples from the expert experience pool; calculate the loss function based on the batch samples and expert samples, and update the policy network based on the loss function; Calculate the predicted loss based on the state value target value, and update the value network based on the predicted loss.

[0044] This embodiment utilizes the agent's own experience gained from interacting with the environment during each round of online interactive iterative training in the cloud, while also incorporating a pre-defined expert experience pool. In each iteration, the agent first obtains the current observable state vector from the adaptive bitrate simulation environment, generates action vectors based on the policy network, and then executes actions in the environment to obtain the next round's observable state vector and corresponding reward, storing this self-exploration experience in a buffer. Simultaneously, privileged information is acquired and input along with the observable state vectors into the value network to obtain a more accurate state value estimate. Based on this state value estimate, an advantage function is calculated, and a state value target value is generated. During the network update phase, batch samples generated from self-exploration are extracted from the buffer, and expert samples are extracted from the expert experience pool. These two types of samples are used together to calculate a loss function, which guides the policy network update, enabling it to learn from both its own exploration and the high-quality decisions of experts. Furthermore, the value network is also updated based on the predicted loss between its state value estimate and the state value target value to ensure the accuracy of its value assessment.

[0045] This training mechanism, which combines self-exploration with expert guidance and enhances value estimation with privileged information, enables the policy network to learn bitrate scheduling strategies that adapt to complex streaming media environments more efficiently and stably, thereby overcoming the problems of slow convergence and unstable performance that may result from relying solely on self-exploration training.

[0046] In some embodiments, the loss function includes: (1); in, Characterize network parameters, Representation strategies The policy pruning loss is used in the near-end policy optimization algorithm. For the predicted loss of the value network, For policy entropy, For behavioral cloning loss, These are the corresponding weighting coefficients; Behavioral cloning loss is correlated with expert samples; The linear annealing decays with each iteration. include: (2); in, These are the initial weighting coefficients. For the current iteration round With weight decay cutoff round The ratio, To obtain and A function that finds the maximum value in the middle.

[0047] loss function By integrating the policy pruning loss of the near-end policy optimization algorithm Predicted loss of value network Strategy Entropy and behavioral cloning loss These multiple objectives form a unified optimization goal. This ensures the stability of policy updates and avoids drastic fluctuations in the policy during training. This drives the value network to accurately estimate state value, providing a reliable benchmark for strategy evaluation; Encourage strategies to explore fully to prevent them from getting trapped in local optima prematurely; and, most importantly... Utilizing expert samples provides a powerful initial guide for the strategy, significantly accelerating the learning process through dynamically adjusted weight coefficients, particularly the weight coefficients of the behavioral cloning loss. The linear annealing decay mechanism with each iteration allows the training process to flexibly adapt to the needs of different stages. In the early stages of training, higher... This ensures the strategy can quickly learn and mimic expert behavior, thus establishing a robust initial strategy. As training progresses and the strategy accumulates experience through its own exploration, As the number of cases decreases, the strategy's reliance on expert experience also decreases. This smooth transition allows the agent to gradually surpass the expert level while maintaining training stability, exploring and discovering better strategies.

[0048] In some embodiments, the lightweight model includes the Transformer model.

[0049] This embodiment uses the Transformer model as a lightweight model. During knowledge distillation, the Transformer model acts as a student model, effectively learning and fitting the complex bitrate scheduling timing decision-making capabilities of the teacher model. Specifically, leveraging its self-attention mechanism, the Transformer model can globally perceive the historically observable playback and network state data sequences, capturing long-distance dependencies between states at different time steps. This capability allows the student model to more accurately understand and reproduce the teacher model's decision logic in different contexts when receiving the teacher model's decision distribution or decision sequence as a supervisory signal, especially when dealing with complex timing scenarios such as network fluctuations and playback stuttering. In this way, the Transformer model can efficiently transfer the deep understanding of real-time network bandwidth timing data and playback states learned by the large language model teacher in the cloud to the terminal. Thus, when the terminal uses real-time acquired current playback and network state data as input, it can output high-quality bitrate decisions and execute video segment downloads accordingly, ensuring smooth streaming playback and a good user experience.

[0050] In some embodiments, the timing decision-making capability of the large model bitrate scheduling corresponding to the trained policy network is transferred to the student model to obtain a distilled lightweight model, which is then deployed on the terminal, including: By using sequence-preserving knowledge distillation, the timing decision-making capability of the large model bit rate scheduling corresponding to the trained policy network is transferred to the student model, resulting in a lightweight model after distillation, which is then deployed on the terminal. Sequence-preserving knowledge distillation includes: Using the observable states corresponding to multiple consecutive decisions as the smallest training sample unit, the student model takes the complete temporal state sequence as input and fits the continuous bitrate decision distribution of the teacher model under the corresponding sequence, preserving the temporal dependency logic of bitrate scheduling.

[0051] Sequence-preserving knowledge distillation is a special type of knowledge distillation technique. Its core lies in ensuring that the student model, while learning the decision-making capabilities of the teacher model, can effectively capture and reproduce the temporal dependencies exhibited by the teacher model when processing sequential data. This differs from traditional distillation methods based on single samples, which may lose crucial contextual information when handling tasks with strong temporal correlations. Sequence-preserving knowledge distillation aims to enable the student model to understand and mimic the teacher model's decision-making evolution over time by designing specific training samples and loss functions. For example, temporal dependencies can be modeled by introducing modules capable of processing sequential information, such as recurrent neural networks (RNNs), long short-term memory networks (LSTMs), gated recurrent units (GRUs), or attention mechanisms (like Transformer structures), into the student model. Another approach is to match not only the teacher model's immediate decision output during distillation but also its intermediate representations or hidden states at different time steps in the sequence, thereby forcing the student model to learn deeper temporal features.

[0052] This embodiment utilizes sequence-preserving knowledge distillation to transfer the temporal decision-making capabilities of the large-scale model's bitrate scheduling to the student model. Specifically, during distillation, individual state-decision pairs are no longer treated as independent training samples; instead, the observable states corresponding to multiple consecutive decisions are treated as the smallest training sample unit. The student model, such as the Transformer model, is designed to receive complete temporal state sequences as input. During training, the student model aims to fit the continuous bitrate decision distribution generated by the teacher model under the corresponding temporal state sequences. This means that the student model must not only learn the teacher model's bitrate selection at each time step but also the probability distribution and decision logic behind its selections, especially the evolution of this logic over time. In this way, the student model can effectively preserve the temporal dependency logic exhibited by the teacher model in bitrate scheduling, i.e., understand and reproduce how the teacher model makes coherent and forward-looking decisions based on historical information. This method enables the lightweight model to perform intelligent and stable bitrate scheduling on the terminal, just like a large model in the cloud, thus overcoming the temporal information loss problem that may occur with traditional distillation methods and ensuring the quality of streaming media playback and user experience on the terminal.

[0053] In some embodiments, the distillation loss is constructed using KL divergence to supervise the student model's fitting of the teacher model's action probability distribution.

[0054] The solution employs KL divergence as a distillation loss, its core being the quantification of the difference between the action probability distribution output by the student model in a specific state and that of the teacher model in the same state. By minimizing this KL divergence, the student model is forced to fit the action probability distribution of the teacher model. This means that the student model not only learns the bitrate decision ultimately chosen by the teacher model, but more importantly, it learns the inherent logic and tendency of the teacher model in weighing and choosing among various optional bitrates. This fitting of the probability distribution allows the student model to capture the soft objective of the teacher model's decision-making—that is, the teacher model's confidence in different actions—rather than simply the final decision of the hard objective. This is crucial for preserving the temporal dependency logic of bitrate scheduling, because bitrate decisions are often not a single deterministic choice, but rather based on predictions of future states and potential rewards, exhibiting certain probabilistic distribution characteristics. In this way, the student model can better understand and mimic the teacher model's adaptive decision-making capabilities in complex networks and playback states, thereby achieving an efficient and smooth streaming experience on the terminal side.

[0055] like Figure 2 To facilitate understanding by those skilled in the art, this application provides the following embodiments: Step S210: Fine-tune the pre-trained NetLLM using PPO (training phase). Obtain the pre-trained large language model bitrate scheduling model NetLLM; Observable states include playback status and network status, such as buffer size, network throughput, download time of the previous video segment, and the percentage of remaining video segments.

[0056] Step S220: Implant OAC to accelerate the convergence speed during large model training; Embedding OAC (Optical Access Control) at the input of the value network, which provides information about future video segment sizes, can accelerate the convergence speed during model training. To eliminate the convergence gap caused by asymmetric architecture without violating the causal constraints of the adaptive bitrate algorithm, we draw inspiration from asymmetric architectures with privileged information.

[0057] In practical dynamic adaptive bitrate streaming systems, the size of subsequent video segments at different bitrate levels is not unpredictable future data; instead, at the initial stage of the session, the MPD file explicitly sends this information to the terminal. This data is used as privileged prior information to accelerate the value function estimation speed of the value network. Specifically, the size sequence of the subsequent N video segments is encoded, fused with the state vector, and then input into the value network: ; in It contains the size information of future video segments at different bitrates.

[0058] The key is that this privileged information is only supplied to the value network during the training phase, regardless of whether it's training or real-world inference. Reinforcement learning theory demonstrates that augmenting the state value baseline with privileged information reduces variance without introducing bias into the policy gradient. This fully preserves the causality of the ABR decision process while ensuring that the deployment of the large language model policy network does not incur additional computational overhead or inference latency.

[0059] By predicting subsequent transmission load in advance, OAC enables the lightweight value network to more accurately assess the safety margin of the current buffer and provides the action network with stable and forward-looking advantage signals, effectively accelerating the convergence speed of the value network and keeping its convergence rhythm synchronized with the action network of the large language model. In the later stages of fine-tuning, the near-end policy optimization loss will naturally dominate the direction of parameter updates, allowing the model to break through the limitations of the initial expert policy and explore more diverse policy solutions.

[0060] Furthermore, if N is too small, the model's predictive gain will be insufficient; if N is too large, it will introduce long-term noise, interfering with the value estimation results. =5, and experiments show that this value can achieve the optimal balance.

[0061] In each round of interaction between the large model and the environment, an additional pool of expert experience is introduced to prevent the model from quickly forgetting the knowledge learned in the pre-training stage during the early stages of fine-tuning.

[0062] Step S230: A progressive fine-tuning strategy from imitation to exploration; Maintain an expert experience pool during training. This is used to store high-quality state-action pairs obtained from pre-training. During the parameter update iteration phase of PPO, behavior cloning loss is introduced. As a regularization term: (4); Here, The behavior cloning loss function represents the loss function, where These are the parameters of the policy network. Indicates the policy network in state Predicting Expert Actions The probability of.

[0063] The loss function is a strong regularization term that acts as a knowledge anchor during training, forcing the model to maintain a lower bound on the confidence level of expert actions when exploring new strategies. Despite leveraging... Imposing certain constraints on the model can effectively prevent catastrophic forgetting, but it also limits the model's ability to explore the original strategy in a breakthrough way. To achieve iterative upgrade from imitation to transcendence, a time-adaptive hybrid loss function is designed, as shown in the above formulas (1) and (2), with a series of coefficients. All are related to the number of training rounds. (Targeting...) The weights are determined using a linear truncation annealing strategy.

[0064] The truncation annealing strategy divides the training process into two core phases: the knowledge consolidation phase and the strategy evolution phase. In the knowledge consolidation phase (typically the first 50% of the training cycle)... Maintain high numerical values. During this stage, the model is in a warm-up state, relying on expert prior knowledge to quickly adapt to the PPO training rhythm and effectively resist the negative feedback impact from initial environmental interactions. Once the model performance stabilizes... Linear decay occurs, and the model gradually transforms from an imitator to an explorer, with parameter updates dominated by the PPO gradient. This model allows for the exploration of strategies not employed by experts but capable of achieving higher Mean-QoE. The decay process continues until... A value of 0 indicates that the model has fully internalized the expert logic and formed an optimization strategy system. At this point, the model is no longer constrained by imitation weights and can fully explore its performance ceiling.

[0065] This course-based weight scheduling scheme successfully achieves iterative optimization of the NetLLM model's capabilities while preserving the fundamental knowledge base of NetLLM pre-training. The method not only solves the initial training crash problem but also ensures that the model continues to achieve performance breakthroughs in subsequent stages.

[0066] At the same time, Initialize it to a large value to speed up the convergence of the value network in the early stages of training, and then gradually decrease the value to minimize its interference with the gradient of the policy network (actor). Built-in anti-stagnation mechanism: When the policy network converges prematurely, the parameters will be increased adaptively to ensure that the policy has sufficient exploration capabilities in the subsequent fine-tuning stage.

[0067] Step S240: Distill the large model trained by PPO using the Transformer strategy; While the performance of large models trained with PPO has been further improved, exceeding the upper limit of expert trajectory sets, it is difficult to deploy on resource-constrained terminal devices due to the extremely high computational and energy costs. To transform the framework from a decision-making module applicable only to the cloud and edge into a universally deployable local controller, this example proposes a policy distillation paradigm.

[0068] To avoid losing temporal context information during distillation, a lightweight Transformer encoder was specifically chosen as the student model, rather than a single-layer multilayer perceptron. This ensures that the student model inherits the structural ability to process historical state sequences. Let... For the frozen teacher strategy, For trainable student policies. In the distillation phase, the teacher model is run in the environment to generate a rich dataset containing heterogeneous network trajectories. The student model is trained to minimize the Kolb-Leibler (KL) divergence between its predicted action probabilities and the soft labels output by the teacher model. The loss objective is as follows: (5); in, Temperature is a hyperparameter used to smooth the probability distribution, enabling the student model to learn not only the optimal bitrate selection scheme but also the potential uncertainty and safety margin evaluated by the large language model. For the knowledge distillation loss, the temperature hyperparameter is fixed in all training experiments. .

[0069] By adopting this order-preserving distillation method, the inference latency and hardware footprint of the model are significantly reduced, while the inference capability of the large teacher model in the adaptive bitrate adjustment task is preserved.

[0070] The following are the pseudocodes for reinforcement learning and distillation: The first step is to fine-tune NetLLM using PPO: Input: The policy of the NetLLM to be trained NetLLM's inherent pool of expert experience ,environment ; Output: Optimal large model bitrate scheduling timing decision ; Initialization: Initialization of policy network parameters as performed by NetLLM ← and value network initialization ; Set hyperparameters: policy network learning rate Value network learning rate Total number of iterations ; initialization ; for k = 1 to

[0071] for step t=1 to 5 do From the environment Read the current state vector Rewards from the previous step ; According to strategy ,by( ) is the sampling action vector ; Perform actions in the environment Instant rewards for observation Next state ; tuple ( Store in the buffer; End Utilizing value networks ( Calculate the dominance function for each step; calculate ; calculate ; calculate ; 15 for iterates through each batch of samples in the buffer. ; 16 From Sampling expert sample ( ); 17. Calculate the loss of the PPO pruning strategy. ; 18. Calculate the loss of value ; 19. Calculate the behavioral cloning loss. ; Total Losses: 20 ; 21. Update the parameters of the policy network: = θ ; 22. Update the parameters of the value network: = ; 23 end Mini-batch training loop; 24-end total iteration loop; 25 returns .

[0072] Next, we use knowledge distillation to allow NetLLM to migrate its optimized policy to a lightweight transformer: 1. Initialization: Student model (lightweight Transformer encoder), clear distillation dataset; Phase 2, Stage 1: Teacher-model interactive sampling to construct the distillation dataset; 3 for sampling rounds i = 1 to N do 4. Reset the environment Get the initial state ; 5 for time step t = 1 to trajectory length T do; 6. Teacher model calculation of temperature-sensitive soft tag distribution: ; 7. Perform the teacher sampling action and observe the next state. ; 8-unit tuple ( Store in dataset D; 9end 10end; Phase 11, Part 2: Supervised Knowledge Distillation Training Using Student Models; 12for Training rounds e = 1 to E do 13 Shuffle the dataset D and split it into several smaller batches; 14 for each small batch b do; 15 student models with the same temperature output soft distribution: ; 16. Calculate the KL divergence distillation loss:

[0073] 17. Gradient update of student parameters; 18end for 19end for 20 Return: Strategies for Lightweight Models After Distillation ; The beneficial effects of the method provided in this embodiment are as follows: (1) To address the problems of poor generalization of traditional algorithms, insufficient long-term modeling of lightweight reinforcement learning, and the expert performance ceiling of offline LLM, this embodiment uses pre-trained NetLLM as the backbone of the policy network in the cloud and adopts the online PPO reinforcement learning framework to continuously interact with the real network. It is freed from the constraints of static expert datasets and allows large models to autonomously try and explore bitrate scheduling strategies that exceed historical experience, thus breaking the performance ceiling brought about by offline training from the root. (2) To address the shortcomings of the asymmetric capabilities and slow convergence of heavy LLMActor and lightweight MLPCritic in online training, this embodiment introduces OAC into the value network. The sizes of the next N video segments contained in the streaming MPD file are used as privileged information input into the value network to enhance the look-ahead evaluation capability, accelerate training convergence, and eliminate gradient oscillations. At the same time, following the CTDE centralized training and distributed execution paradigm, the policy network does not access future segment information during the inference phase, thus not disrupting the causality of bitrate decisions. (3) To address the problem of sparse rewards in the early stages of online training, which can easily lead to catastrophic forgetting and scheduling strategy collapse, this embodiment designs a progressive fine-tuning strategy. The offline expert experience pool is introduced into the loss function as a knowledge anchor, and the loss weights are cloned through linear annealing dynamic decay behavior: in the early stage of training, high weight constraints imitate expert experience to resist the impact of random negative feedback; in the later stage of training, the constraint weights are gradually reduced to open up the model's autonomous exploration space, and the scheduling strategy is continuously optimized while retaining the pre-training temporal knowledge. (4) To address the paradox of heavy-light deployment where the native NetLLM has high memory consumption and excessive inference latency, making it unsuitable for the lightweight hardware constraints of mobile devices, this embodiment builds a sequence-preserving policy distillation framework: using a NetLLM optimized by online PPO as the teacher model and a lightweight Transformer as the student model, the complete temporal action distribution of the teacher is replicated through KL divergence supervision, transferring the optimal scheduling knowledge of the large cloud model to the small model; after distillation, the model memory is compressed by over 99.8%, the inference latency is reduced to the millisecond level, and the ability to model long sequences is preserved.

[0074] Experimental Examples; By default, Llama2-7B is used as the base large language model. Experiments are then conducted using the training trajectory set and the expert experience pool built into the NetLLM source code. Furthermore, the classic datasets fcc and oboe are introduced to evaluate the model's performance in highly complex network scenarios. RobustMPC, Comyco, Pensieve, Karma, the initial NetLLM, and the final fine-tuned model are compared.

[0075] Because different users have different preferences, the relevant metrics include a commonly used linear service quality experience metric, which is defined as follows: (6); in, Representing the The bitrate selected for each video segment, Used to map bitrate to user-perceived image quality. This indicates the buffer duration for the current fragment. and These are the penalty coefficients for buffering loss and image smoothness loss, respectively. To maintain consistency with other algorithms, they are set as follows: , In addition, several metrics, including smoothness, buffer state, stuttering rate, and bitrate, are introduced to evaluate the performance of each algorithm. The formula for calculating smoothness is as follows: (7); The stuttering rate refers to the proportion of stuttering time to the total playback time. Here, bitrate represents the average bitrate selected each time, and buffer represents the buffer size when the bitrate is selected each time.

[0076] In terms of hardware configuration, the experiment was conducted on a Linux server equipped with four NVIDIA RTX A6000 graphics cards.

[0077] This method (hereinafter referred to as "this model") achieves the lowest stuttering rate among all algorithms while maintaining a relatively high bitrate, thus balancing high video quality and low stuttering rate. Although Comyco can provide a higher bitrate, its stuttering rate is also the highest among all algorithms, indicating that this algorithm pays the price of increased stuttering time in pursuit of high video quality. While ensuring high video quality, this model's bitrate switching is smoother than other algorithms, a performance that Comyco cannot achieve. This model not only achieves the highest experience quality score but also maintains the buffer within a safe capacity range, effectively avoiding playback stuttering caused by sudden network fluctuations. Karma and NetLLM maintain excessively large buffer capacities and lack a proactive scheduling strategy to switch to higher bitrate levels, resulting in relatively lower experience quality scores.

[0078] Subsequently, evaluation experiments were conducted on the more challenging Oboe dataset. The Oboe dataset represents a mobile network scenario with significant fluctuations (in contrast to the low-fluctuation fixed network scenario represented by the Federal Communications Commission (FCC) dataset), which presents a more stringent challenge to the adaptive bitrate algorithm. Under this challenging scenario, our model still achieved the optimal balance between bitrate and stuttering rate. Compared to its performance on the FCC dataset, our model proactively switches to a more conservative buffer management strategy, effectively mitigating the risk of playback stuttering caused by bandwidth fluctuations by increasing buffer capacity. Simultaneously, our model maintains a high quality of user experience, and the bitrate switching process is smooth and seamless, fully demonstrating the model's ability to dynamically adjust its strategy in various heterogeneous scenarios. In contrast, the performance of various benchmark algorithms shows a significant decline with changes in the scenario.

[0079] The following is an ablation experiment; To evaluate the impact of privileged information depth on the model's performance, a sensitivity analysis was conducted with the look-ahead interval N ranging from 0 to 10. The experimental configuration corresponding to N=0 served as the baseline control group, representing the model architecture without the OAC module. Experimental results show that as N increases from 0 to 5, the model's average reward significantly improves, and the OAC module greatly accelerates model convergence. However, when N continues to increase to 10, the performance improvement tends to stagnate, and even shows a slight decrease. This phenomenon indicates that while a larger N value provides contextual information over a longer time dimension, it increases the input dimension of the lightweight value network, easily introducing long-term random noise that interferes with the value evaluation process. In conclusion, N=5 was selected as the optimal look-ahead interval for this model, thus achieving a balance between predictive look-ahead capability and training stability.

[0080] Meanwhile, expert guidance plays a crucial role in adaptive code rate tasks based on large language models. Under pure proximal policy optimization (unconstrained) settings, the performance of large language models immediately collapses, with the average reward value dropping sharply and failing to recover even after 1000 iterations. This phenomenon confirms that without knowledge anchors, the high-dimensional weight space of large models is highly susceptible to the randomness and sparse gradients of early-stage reinforcement learning exploration, leading to catastrophic forgetting. While imitation constraints can stabilize the fine-tuning process, they inevitably create performance bottlenecks. This type of static regularization constitutes a performance ceiling, limiting the full potential of large language models to leverage their long-sequence modeling advantages. Ultimately, the model fails to explore a better decision logic, which could have surpassed the given suboptimal expert trajectory.

[0081] In comparison, this model combines the advantages of both approaches. In the early stages of training (the first 50% of the training phase), it offers higher performance. The weights preserve the general prior knowledge inherited by the model based on behavioral abstraction representations. As the expert weights gradually decay, the model transforms from a learner imitating the expert into an explorer optimizing the environment. This annealing mechanism allows the model to more efficiently utilize the advantageous signals output by the value network enhanced by the optimal action set, ultimately outperforming the expert benchmark model with an average reward of 1.09. This performance rebound and breakthrough demonstrates that incremental fine-tuning is crucial for unleashing the autonomous evolutionary capabilities of large language models in complex network control tasks.

[0082] like Figure 3 As shown in one embodiment of this application, an adaptive bitrate streaming media optimization device with cloud and local decoupling is provided. The device includes: The model building module 1001 is used to build a reinforcement learning architecture of policy network and value network in the cloud; wherein, the policy network is a pre-trained large language model bitrate scheduling model, and the value network is a lightweight neural network. The network training module 1002 is used to construct an adaptive bitrate simulation environment using real network bandwidth time-series trajectory data, and to perform online interactive iterative training of the policy network and value network in the cloud based on the near-end policy optimization algorithm, so as to update the parameters of the policy network and value network and obtain the trained policy network. The state space in the online interactive iterative training includes observable states, which include playback state and network state, and the action space includes optional bitrate decisions. The model distillation module 1003 is used to take the trained policy network as the teacher model and the lightweight model as the student model, and transfer the large model bit rate scheduling timing decision-making capability corresponding to the trained policy network to the student model to obtain the distilled lightweight model and deploy it on the terminal. The model inference module 1004 uses real-time acquired current playback status and network status data as input data, inputs the input data into the distilled lightweight model in the terminal, obtains the output bitrate decision, and performs video segment download based on the bitrate decision.

[0083] The device embodiments provided in this application are based on the same inventive concept as the above-described method embodiments. Therefore, the relevant content of this device embodiment is similar to that of the above-described method embodiments, and will not be repeated here.

[0084] like Figure 4 As shown, one embodiment of this application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned cloud-local decoupling adaptive bitrate streaming media optimization method. The electronic device includes: At least one memory; At least one processor; At least one program; The program is stored in memory, and the processor executes at least one program to implement the cloud-local decoupled adaptive bitrate streaming optimization method described above in this application.

[0085] The electronic device can include any smart terminal such as mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0086] The electronic devices according to embodiments of this application will now be described in detail.

[0087] The processor 1600 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1700 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1700 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called and executed by the processor 1600 to execute a cloud-local decoupled adaptive bitrate streaming media optimization method according to an embodiment of this application.

[0088] The input / output interface 1800 is used to implement information input and output. The communication interface 1900 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, Wi-Fi, Bluetooth, etc.). Bus 2000 transmits information between various components of the device (e.g., processor 1600, memory 1700, input / output interface 1800, and communication interface 1900); The processor 1600, memory 1700, input / output interface 1800 and communication interface 1900 are connected to each other within the device via bus 2000.

[0089] This application embodiment also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the above-described cloud-local decoupled adaptive bitrate streaming media optimization method.

[0090] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0091] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of this application, and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0092] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0093] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0094] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0095] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0096] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0097] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0098] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0099] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0100] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0101] The above is a detailed description of the preferred embodiments of this application. However, the embodiments of this application are not limited to the above-described implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the embodiments of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of the embodiments of this application.

Claims

1. A cloud- and local decoupled adaptive bitrate streaming media optimization method, characterized in that, The method includes: A reinforcement learning architecture for constructing a policy network and a value network in the cloud; wherein the policy network is a pre-trained large language model bitrate scheduling model, and the value network is a lightweight neural network; An adaptive bitrate simulation environment is constructed using real network bandwidth time-series trajectory data. Based on a near-end policy optimization algorithm, the policy network and value network are trained online interactively and iteratively in the cloud to update the parameters of the policy network and the value network, thereby obtaining the trained policy network. The state space in the online interactive iterative training includes observable states, which include playback state and network state. The action space includes optional bitrate decisions. The trained policy network is used as the teacher model, and the lightweight model is used as the student model. The large model bitrate scheduling timing decision-making capability corresponding to the trained policy network is transferred to the student model to obtain the distilled lightweight model, which is then deployed on the terminal. Using real-time acquired current playback status and network status data as input data, the input data is input into the distilled lightweight model in the terminal to obtain the output bitrate decision, and video segment download is performed according to the bitrate decision.

2. The adaptive bitrate streaming media optimization method with cloud and local decoupling as described in claim 1, characterized in that, The reinforcement learning architecture also includes a prophetic enhancement context module at the input of the value network. The prophetic enhancement context module is used as privileged information to extract the size information of future video segments at different bitrates from a preset streaming media MPD file. The input to the value network includes a spliced ​​vector formed by concatenating the privileged information with the observable state; The input to the policy network includes the observable state.

3. The adaptive bitrate streaming media optimization method with cloud and local decoupling as described in claim 2, characterized in that, Each iteration of the online interactive iterative training includes: Obtain a pre-defined pool of expert experience; From the adaptive bitrate simulation environment, read the observable state vector; according to the policy network, generate an action vector based on the observable state vector; execute the action vector in the adaptive bitrate simulation environment to obtain the observable state vector for the next round; store the observable state vector, the action vector, the observable state vector for the next round, and the corresponding reward into a buffer. The privileged information is acquired, and the privileged information and the observable state vector are input into the value network to obtain the state value estimate output by the value network; the advantage function is determined based on the state value estimate, and the target state value is generated based on the advantage function; Extract batch samples from the buffer and expert samples from the expert experience pool; calculate a loss function based on the batch samples and the expert samples, and update the policy network based on the loss function; The predicted loss is calculated based on the state value target value, and the value network is updated based on the predicted loss.

4. The adaptive bitrate streaming optimization method with cloud and local decoupling as described in claim 3, characterized in that, The loss function includes: ； in, Characterize network parameters, Representation strategies The policy pruning loss is used in the near-end policy optimization algorithm. For the predicted loss of the value network, For policy entropy, For behavioral cloning loss, These are the corresponding weighting coefficients; The behavioral cloning loss is associated with the expert sample; The linear annealing decays with each iteration, and ; in, These are the initial weighting coefficients. For the current iteration round With weight decay cutoff round The ratio, To obtain and A function that finds the maximum value in the middle.

5. The adaptive bitrate streaming media optimization method with cloud and local decoupling as described in claim 1, characterized in that, The step of transferring the large model bitrate scheduling timing decision-making capability corresponding to the trained policy network to the student model, obtaining a distilled lightweight model, and deploying it on the terminal includes: By using sequence-preserving knowledge distillation, the timing decision-making capability of the large model bitrate scheduling corresponding to the trained policy network is transferred to the student model, resulting in a distilled lightweight model that is then deployed on the terminal. The sequence-preserving knowledge distillation includes: The observable states corresponding to multiple consecutive decisions are used as the smallest training sample unit, and the complete temporal state sequence is used as the input of the student model. The continuous code rate decision distribution of the teacher model under the corresponding complete temporal state sequence is then fitted.

6. The adaptive bitrate streaming media optimization method with cloud and local decoupling as described in claim 1, characterized in that, Distillation loss is constructed using KL divergence to supervise the student model's fit to the action probability distribution of the teacher model.

7. The adaptive bitrate streaming media optimization method with cloud and local decoupling as described in claim 1, characterized in that, The lightweight model includes the Transformer model.

8. A cloud- and local decoupled adaptive bitrate streaming media optimization device, characterized in that, The device includes: The model building module is used to construct a reinforcement learning architecture of policy network and value network in the cloud; wherein, the policy network is a pre-trained large language model bitrate scheduling model, and the value network is a lightweight neural network. The network training module is used to construct an adaptive bitrate simulation environment using real network bandwidth time-series trajectory data, and to perform online interactive iterative training on the policy network and value network in the cloud based on the near-end policy optimization algorithm, so as to update the parameters of the policy network and the value network and obtain the trained policy network; wherein, the state space in the online interactive iterative training includes observable states, which include playback state and network state, and the action space includes optional bitrate decisions; The model distillation module is used to take the trained policy network as the teacher model and the lightweight model as the student model, and to transfer the large model bitrate scheduling timing decision-making capability corresponding to the trained policy network to the student model, so as to obtain the distilled lightweight model and deploy it on the terminal. The model inference module is used to input the real-time acquired current playback status and network status data into the distilled lightweight model in the terminal to obtain the output bitrate decision, and to perform video segment download based on the bitrate decision.

9. An electronic device, characterized in that, include: At least one processor and a memory for communicatively connecting to said at least one processor; The memory stores instructions that can be executed by the at least one processor to cause the at least one processor to perform a cloud-and-local decoupled adaptive bitrate streaming optimization method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the cloud-and-local decoupled adaptive bitrate streaming media optimization method as described in any one of claims 1 to 7.

Images