Short video rate adaptation method based on meta learning

Abstract

The application discloses a short video code rate self-adaptive method based on meta learning, relates to the technical field of streaming media, and comprises the following steps: S1, offline training, a model is established to represent user characteristics and network prediction information; S2, online learning, according to the characteristics of the current user environment, the model parameters are adjusted and optimized. The short video code rate self-adaptive method based on meta learning is adopted, a new SABR framework based on meta learning is successfully realized, the framework can quickly adapt to different user demands, the practicability and the calculation speed of the system are improved, and the framework has industrial application; the offline training and the online learning technology are successfully combined, the generalization and the stability of the model are enhanced; the idea of action masking is introduced in pre-training, the rationality and the reliability of decision are enhanced, the data amount required by meta learning is effectively reduced, the learning efficiency and the accuracy are improved, and the data demand and the training time in the industrial environment are significantly reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of streaming media, and particularly to a short video bit rate self-adaptive method based on meta learning. BACKGROUND

[0002] In short video applications, users can skip the videos they do not like at any time to realize the "swipe while watching" viewing mode. In order to ensure the smooth viewing experience of users, short video platforms usually need to preload videos in the client buffer. Considering different network conditions, these platforms provide videos in multiple representations or bit rates so that users can select a suitable bit rate to ensure the quality of experience (QoE). Generally, the client tends to preload more video blocks to avoid the occurrence of rebuffering. However, if the user performs a swipe operation, the video data that has been downloaded but not watched will no longer help to improve the QoE of the user, which will result in the waste of bandwidth resources. Therefore, in consideration of the user viewing behavior and network conditions, it is necessary to adaptively and strategically preload videos with the goal of reducing the waste of bandwidth resources.

[0003] Content providers usually use short video bit rate adaptation (SABR) algorithms to optimize the QoE of users and reduce the waste of bandwidth. The SABR algorithm needs to determine which video should be downloaded or kept in a paused state, and determine the bit rate that should be preloaded for a specific video. Learning-based SABR algorithms are proposed to solve such problems. For example, DAM uses a deep reinforcement learning PPO framework, introduces user retention rate as a reward function, learns an optimal policy to make decisions, and incorporates domain knowledge into the decision-making process through action masking to improve the transparency of decisions.

[0004] However, methods similar to DAM have the following problems: on the one hand, the learning process of the reinforcement learning strategy is highly dependent on the authenticity and effectiveness of the data set, which directly affects its performance in the actual environment. On the other hand, these methods usually lack sufficient adaptability, i.e., a reinforcement learning model specific to a certain client is difficult to be widely applied to other clients in similar environments. Therefore, even with a rich historical data set, it is difficult to train a generalized model that adapts to different network types. In addition, considering the diversity of user behavior (such as frequent switching of videos) and the rapid changes in network environment (such as switching between Wi-Fi and mobile data), existing algorithms are difficult to adapt to these dynamic changes.

[0005] In practical application in short video system, the existing method has various challenges to overcome. First, these methods lack knowledge transfer and perform poorly in terms of generalization. Current methods are mostly designed for specific tasks and are trained to work independently in specific network environments, making it difficult to handle unknown scenarios. In the real world, the variability of network environments and user behavior makes the model perform poorly when faced with unmet or less met situations. Second, there are also problems in stability. Existing algorithms are very sensitive to changes in input data, especially when faced with network environment instability (such as network delay and bandwidth fluctuation), the performance of the model may drop sharply. This not only affects the user experience, but also may cause discontinuity of streaming playback. SUMMARY

[0006] The purpose of the present application is to provide a short video rate adaptation method based on meta-learning, a short video rate adaptation model combining offline training with online learning and action masking structure, which enhances the generalization and stability of the model, enhances the rationality and reliability of decision-making, effectively reduces the data volume required by meta-learning, and improves learning efficiency and accuracy.

[0007] To achieve the above purpose, the present application provides a short video rate adaptation method based on meta-learning, comprising the following steps:

[0008] S1, offline training, establishing a model to represent user characteristics and network prediction information;

[0009] S2, online learning, adjusting and optimizing the model parameters according to the characteristics of the current user environment.

[0010] Preferably, the offline training comprises the following steps:

[0011] S11, classify the data according to the network environment;

[0012] S12, input the classified data into the agent and process the data;

[0013] S13, encode the data into model parameters.

[0014] Preferably, S12 includes a short video prefetching model, which comprises the following steps:

[0015] Step one, pre-train the agent using full data and add action masking in pre-training;

[0016] Step two, further train the pre-trained strategy on the classified data using the meta-learning Reptile framework.

[0017] Preferably, the action masking introduces a masking vector , With action probability Multiply to calculate the new action probability distribution after masking. .

[0018] Preferably, short video loading follows these rules:

[0019] (1) All videos are downloaded in chronological order. That is, if the video at the (n-1)th second has not been downloaded, the video at the nth second is not allowed to be downloaded.

[0020] (2) When the buffer occupancy rate is less than 30%, the buffering time for each video shall not exceed 4s; when the buffer occupancy rate is greater than 30% but less than 60%, the buffering time for each video shall not exceed 3s; when the buffer occupancy rate is greater than 60%, the buffering time for each video shall not exceed 2s.

[0021] (3) If the currently playing video has a buffer length of less than 1 second, it should be downloaded immediately.

[0022] Preferably, training the meta-learning Reptile framework includes the following steps:

[0023] Step 1: In each task This allows for updates to traditional RL tasks, enabling the agent to make decisions and obtain multiple [resources / information]. sequence,;

[0024] Step 2: Calculate the loss function Perform gradient descent iterations to obtain the updated parameters. ;

[0025] Step 3: Average the updated parameters and use them as the new model parameters.

[0026] Preferably, online learning includes the following steps:

[0027] Step 1: Obtain the status values ​​in a timely manner With action value Stored in an array middle;

[0028] Step 2: Obtain corresponding reward values ​​based on user behavior. ;

[0029] Step 3: Combine with reward value Matching state values Action value composition The sequence is used for unified updates.

[0030] Preferably, the reward value obtained under the premise of algorithm update With state value and action value Match, empty the storage array immediately after updating .

[0031] Therefore, the present application adopts the short video rate adaptation method based on meta learning described above, successfully realizes a new SABR framework based on meta learning, which can quickly adapt to different user needs, improve the practicability and calculation speed of the system, and make it have industrial application; the combination of offline training and online learning technology, offline training uses metadata for training to obtain a general model, online learning is based on the general model, and further trains a specific user model according to the environment of each user, which helps to enhance the generalization and stability of the model; a pre-training method with action masking, which combines manual rules and deep reinforcement learning, uses the prior knowledge of the task domain to further guide the learning process of the model, aiming to enhance the rationality and reliability of decision-making, which effectively reduces the data amount required by meta learning, improves the learning efficiency and accuracy.

[0032] The technical solutions of the present application will be further described in detail below with the help of the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0033] Fig. 1 The algorithm flowchart of the short video rate adaptation method based on meta learning of the present application;

[0034] Fig. 2 The short video pre-fetching model flowchart of the short video rate adaptation method based on meta learning of the present application. DETAILED DESCRIPTION

[0035] The technical solutions of the present application will be further described in detail below with the help of the accompanying drawings and embodiments.

[0036] Unless otherwise defined, the technical terms or scientific terms used in the present application shall be understood as the usual meaning understood by those skilled in the art to which the present application belongs. The terms "first", "second" and the like used in the present application do not represent any order, number or importance, but are only used to distinguish different components. The terms "include" or "contain" and the like mean that the elements or objects before the terms cover the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connect" or "connect" and the like are not limited to physical or mechanical connection, but can include electrical connection, whether direct or indirect. The terms "up", "down", "left", "right" and the like are only used to represent relative positional relationship, when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0037] EMBODIMENT

[0038] Please see Figs. 1-2 This invention provides a short video bitrate adaptive method based on meta-learning, comprising the following steps:

[0039] S1. Offline training involves classifying data according to network conditions, then inputting relevant task data into the agent. Through processing and learning of this data, common features and knowledge are extracted and encoded into the model parameters. In this way, the MPA algorithm can build a highly generalized model whose parameters can represent user characteristics and network prediction information in a more optimized way, thus significantly improving the algorithm's adaptability.

[0040] The offline training process also includes a short video prefetching model, which includes the following steps:

[0041] Step 1: Pre-train the agent using unclassified data and incorporating action masking. This helps the agent avoid many erroneous actions early on, reducing the number of trial and error attempts required for the learning task and leading to faster convergence and more effective learning. By leveraging human experience for model pre-training, rather than performing meta-training from the outset, we mitigate the inherent limitations of meta-learning: stringent data prerequisites, high training costs, and lengthy training times. This pre-training method allows us to more effectively utilize existing knowledge and experience, significantly improving algorithm performance and efficiency.

[0042] Action masking incorporates domain knowledge into policy learning. Specifically, it introduces masking vectors. ,Will With action probability Multiply to calculate the new action probability distribution after masking. Masking vector Designed by us manually, its structure and the rationality of its movements are relevant. Based on the analysis of short video preloading issues, the following rules are followed:

[0043] (1) All videos are downloaded in chronological order. That is, if the video at the (n-1)th second has not been downloaded, the video at the nth second is not allowed to be downloaded.

[0044] (2) When the cache occupancy rate is less than 30%, the buffering time of each video shall not exceed 4s; when the cache occupancy rate is greater than 30% but less than 60%, the buffering time of each video shall not exceed 3s; when the cache occupancy rate is greater than 60%, the buffering time of each video shall not exceed 2s. This can effectively prevent excessive bandwidth waste and reduce the rebuffering time caused by user behavior (swiping to watch the video).

[0045] (3) The currently playing video, once its play buffer length is less than 1 second, should be downloaded immediately, so as to effectively avoid the rebuffering time caused by user behavior (not sliding to watch the video).

[0046] Step two, further train the pre-trained strategy using the meta-learning Reptile framework, which uses pre-trained classified data in this stage.

[0047] The meta-learning Reptile framework training includes the following steps:

[0048] Step one, update the traditional RL task on each task to make the agent make decisions to obtain multiple sequences.

[0049] Step two, calculate the loss function and perform gradient descent iteration to obtain updated parameters .

[0050] Step three, average the updated parameters to obtain new model parameters.

[0051] During the internal loop training process of the MPA-Reptile algorithm, the model parameters are constantly updated, and the initial parameters of each task are the updated parameters obtained after the training of the previous task. This algorithm does not need to explicitly model the relationship between tasks, but learns how to share knowledge between tasks by repeatedly executing a single task, which makes the algorithm have good scalability and generalization performance.

[0052] S2, online learning, the MPA algorithm adjusts and optimizes the model parameters again according to the characteristics of the current user environment. Through continuous learning and updating, the MPA algorithm can gradually adapt to different users' network environments and video needs, thereby obtaining better performance and effect.

[0053] By using offline meta-learning technology, we obtain a meta-model parameter that can reach the optimal average for all tasks, and this model parameter can reach the global optimum of each task through a few steps of gradient descent. In order to make the agent learn and adapt to each user's personalized needs and environment more effectively, the MPA algorithm uses online learning to update the meta-model parameter in a few steps during user use, to fine-tune and optimize the strategy under each user's unique situation.

[0054] Online learning includes the following steps:

[0055] Step one, store the state value and action value obtained in time in an array .

[0056] Step 2: Obtain corresponding reward values ​​based on user behavior. ;

[0057] Step 3: Combine with reward value Matching state values Action value composition The sequence is used for unified updates.

[0058] The reward value obtained as a prerequisite for algorithm update With state value and action value Match, and immediately clear the storage array after updating. .

[0059] Therefore, this invention employs the aforementioned meta-learning-based short video bitrate adaptive method, successfully implementing a novel meta-learning-based SABR framework. This framework can quickly adapt to different user needs, improve the system's practicality and computational speed, and enable its industrial applications. It combines offline training with online learning; offline training utilizes metadata to train a general model, while online learning, based on the general model, further trains a specific user model according to each user's environment, which helps enhance the model's generalization and stability. A pre-training method with action masking is also included. This method integrates hand-crafted rules and deep reinforcement learning, utilizing prior knowledge of the task domain from hand-crafted rules to further guide the model's learning process, aiming to enhance the rationality and reliability of decision-making. This method effectively reduces the amount of data required for meta-learning and improves learning efficiency and accuracy.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A short video bitrate adaptation method based on meta-learning, characterized in that, Comprise the following steps: S1, offline training, model representing user characteristics and network prediction information is established; S2, online learning, according to the characteristics of the current user environment, the model parameters are adjusted and optimized; Offline meta training includes the following steps: S11, the data is classified according to the network environment condition; S12, the classified data is input into the agent, and the data is processed; S13, encode the data into the model parameters; S12 includes a short video pre-fetching model, which comprises the following steps: Step one, pre-training the agent and adding action masking in pre-training; introducing a masking vector in action masking , the action probability is multiplied by the action probability to calculate the new action probability distribution after masking Step two, using the meta learning Reptile framework to further train the pre-trained strategy; Short video loading follows the following rules: (1) all videos are downloaded in time sequence, that is, if the video of the n-1 second has not been downloaded, the video of the n second is not allowed to be downloaded; (2) when the cache occupancy rate is less than 30%, the buffer time of each video should not be greater than 4s; when the cache occupancy rate is greater than 30% and less than 60%, the buffer time of each video should not be greater than 3s; when the cache occupancy rate is greater than 60%, the buffer time of each video should not be greater than 2s; (3) the currently playing video should be downloaded immediately once its playing buffer length is less than 1 second; The meta learning Reptile framework training includes the following steps: Step one, update the traditional RL task on each task Let the agent make decisions to obtain multiple sequences; Step two, compute the loss function , perform gradient descent iteration to get updated parameters ; Step three, average the updated parameters as new model parameters.

2. The method of claim 1, wherein the method is based on meta-learning. Online learning includes the following steps: Step one, obtain the state value in time and the action value are stored in an array ; Step two, obtaining corresponding reward value according to user behavior ; Step three, the state value matching the reward value , action value is composed sequence to make a unified update.

3. The meta-learning based short video bitrate adaptation method of claim 2, wherein: The premise of the algorithm update is the reward value obtained The state value And the action value Match, empty the storage array immediately after updating .

Images