Adaptive bit rate control method and system

Abstract

The present invention belongs to the field of video transmission technology and discloses an adaptive bitrate control method and system. The method comprises: based on decision time points, converting historical player state data into input features for a deep value network to construct an adaptive bitrate agent; utilizing a Monte Carlo tree search algorithm to simulate the player's future playback state at each decision time point, and evaluating the action value representing the quality of each bitrate decision; training an adaptive bitrate agent based on the action value of the bitrate decision, and utilizing the trained adaptive bitrate agent to dynamically adjust the video transmission bitrate. The present invention simulates the future player playback state through MCTS simulation, thereby accurately and dynamically evaluating the long-term benefits of each bitrate decision and avoiding short-sighted decisions.

Description

Technical Field

[0001] The present invention relates to the technical field of video transmission, and in particular to an adaptive bit rate control method and system. Background Art

[0002] The Adaptive Bitrate (ABR) algorithm is a technology that dynamically adjusts the video transmission bitrate, primarily used in streaming services (such as video on demand and live streaming). Its core goal is to automatically select the optimal video bitrate based on real-time network conditions (such as bandwidth, latency, and buffering status) to balance video quality, playback smoothness, and user experience (QoE).

[0003] Currently, existing adaptive bitrate technologies are mainly divided into the following two categories:

[0004] 1) Heuristic method:

[0005] MPC (Model Predictive Control): predicts future bandwidth based on the harmonic mean of historical bandwidth and solves finite-horizon optimization problems;

[0006] BOLA (Lyapunov Optimization): triggers bitrate switching through buffer thresholds;

[0007] 2) Deep reinforcement learning method:

[0008] Pensieve: Based on the A3C framework, it updates network parameters via asynchronous policy gradients.

[0009] However, existing adaptive bitrate (ABR) algorithms for video streaming suffer from a critical flaw: insufficient assessment of the consequences of delay. Specifically, traditional heuristic methods (such as MPC and BOLA) and more recently, machine learning-based approaches make decisions and optimizations based solely on current bandwidth and buffer status. They fail to quantify the long-term impact of high bitrate selection on subsequent buffering status, leading to relatively frequent delays caused by future network fluctuations, significantly degrading the user's playback experience. Summary of the Invention

[0010] The embodiments of the present invention provide an adaptive bit rate control method and system to solve the above-mentioned technical problems in the prior art.

[0011] To provide a basic understanding of some aspects of the disclosed embodiments, the following is a brief summary. This summary is not intended to be an extensive review, identify key or critical elements, or delineate the scope of these embodiments. Its sole purpose is to present some concepts in a simplified form as a prelude to the detailed description that follows.

[0012] According to a first aspect of an embodiment of the present invention, a method and system for adaptive bit rate control are provided.

[0013] In one embodiment, the adaptive bit rate control method includes:

[0014] Based on the decision time point, the player's historical state data is converted into input features of the deep value network to build an adaptive bitrate agent;

[0015] Using the Monte Carlo Tree Search algorithm, the player's future playback state is simulated at each decision point in time, and the action value representing the quality of each bitrate decision is evaluated;

[0016] An adaptive bitrate agent is trained according to the action value of bitrate decision, and the trained adaptive bitrate agent is used to dynamically adjust the video transmission bitrate.

[0017] In one embodiment, converting the player's historical state data into input features of a deep value network based on the decision time point to construct an adaptive bitrate agent includes:

[0018] At each decision point, the player's historical state data is converted into input features of the deep value network. Based on the deep value neural network, an adaptive bitrate agent is constructed.

[0019] Among them, the deep value neural network first uses a one-dimensional convolutional neural network to extract the temporal dependencies in the input information, then merges features from different inputs based on the fully connected layer, and finally outputs the probability distribution of the bitrate decision.

[0020] In one embodiment, the historical status data includes historical bandwidth, buffer information, and block size information, and the bit rate decision includes increasing the bit rate, maintaining the current bit rate, and decreasing the bit rate.

[0021] In one embodiment, the Monte Carlo tree search algorithm is used to simulate the future playback state of the player at each decision time point to evaluate the action value representing the quality of each bitrate decision, including:

[0022] Node selection: Based on the upper confidence interval tree algorithm, search from the root node to the leaf node until the node has no child nodes, then the node is regarded as a leaf node;

[0023] Simulation: Starting from the player status represented by the leaf node, simulation is performed based on the simulated bandwidth data, and the user experience value obtained from the simulation during the predetermined period is recorded;

[0024] Data update: Use the user experience value obtained by simulation to update the average user experience value of all nodes passed by the current search, and update the average value and historical visit count of all passed nodes;

[0025] The node selection, simulation and data update steps are executed cyclically for a preset number of times, and the average of the user experience values ​​of the three nodes directly connected to the root node is used as the action value of the three bitrate decisions.

[0026] In one embodiment, the upper confidence interval tree algorithm is used to determine the next visited node at each node, and the expression of the upper confidence interval tree algorithm is:

[0027]

[0028] Where, Indicates the next node selected, Indicates the current node n The three child nodes of Represents a child node The estimated value of Represents a child node The number of historical visits, Indicates the current node n The number of historical visits, A represents The lower bound of B represents The upper bound of .

[0029] In one embodiment, the expression for the average user experience is:

[0030]

[0031] Where, Indicates the average user experience. Indicates playback rewards, Indicates the penalty for jamming. represents the bitrate conversion penalty, 、 、 are the positive hyperparameters representing playback reward, lag penalty, and bitrate change penalty, respectively.

[0032] In one embodiment, starting from the player state represented by the leaf node, performing simulation based on simulated bandwidth data, and recording the average user experience obtained by the simulation over a predetermined period further includes:

[0033] During the simulation, the non-judge Morton process is used to generate random bandwidth trajectories to simulate the real non-stationary bandwidth trajectories.

[0034] In one embodiment, the expression used to generate the simulated bandwidth characteristic is:

[0035]

[0036] The expression for generating simulation bandwidth based on bandwidth characteristics is:

[0037]

[0038] Where, J t Indicates correspondence t Time has come t The characteristic value of the average bandwidth between +1 moments, Z 1. Z 2. Z 3 represent random variables used to introduce bandwidth randomness, represents a hyperparameter that controls the non-stationarity of the generated bandwidth, represents the hyperparameter that controls the overall magnitude and bandwidth, Indicates the generated t Time has come t +1 Average bandwidth between moments.

[0039] In one embodiment, the update formula of the average value is:

[0040]

[0041] The update formula for the number of historical visits is:

[0042]

[0043] Where, represents the average value before updating, V n represents the updated average value, Represents the number of historical visits before the update, N n Indicates the updated historical visit count. Indicates the average user experience.

[0044] According to a second aspect of the embodiments of the present invention, an adaptive bit rate control system is provided.

[0045] In one embodiment, the adaptive bit rate control system includes:

[0046] Intelligent agent construction module, simulation module and intelligent agent training control module;

[0047] The agent building module is used to convert the player's historical state data into input features of the deep value network based on the decision time point to build an adaptive bitrate agent;

[0048] The simulation module is used to simulate the future playback state of the player at each decision time point using the Monte Carlo tree search algorithm to evaluate the action value representing the quality of each bitrate decision;

[0049] The agent training control module is used to train the adaptive bit rate agent according to the action value of the bit rate decision, and dynamically adjust the video transmission bit rate using the trained adaptive bit rate agent.

[0050] According to a third aspect of an embodiment of the present invention, a computer device is provided.

[0051] In some embodiments, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0052] According to a fourth aspect of embodiments of the present invention, a computer-readable storage medium is provided.

[0053] In one embodiment, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0054] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0055] 1) This paper proposes a joint training framework based on Monte Carlo Tree Search (MCTS) and deep value networks to train ABR agents, primarily to address the latency issues inherent in ABR algorithms. Specifically, this paper uses MCTS simulations to simulate the future player's playback state, thereby accurately and dynamically evaluating the long-term benefits of each bitrate decision and avoiding short-sighted decisions. During the simulation process, to improve the robustness of decision evaluation to bandwidth fluctuations, this paper simulates highly non-stationary bandwidth trajectories to mimic unstable network conditions in real-world scenarios, thereby ensuring that the optimal decision is robust to bandwidth fluctuations.

[0056] 2) Higher QoE: MCTS quantifies the long-term QoE benefits of actions by simulating the download and playback of 10-20 segments in the future. Compared with Pensieve's A3C framework, the average QoE of the trained ABR agents is improved by 18.7%.

[0057] 3) Variance control and training stability: The decision values ​​obtained by MCTS simulation are more accurate, which significantly improves the convergence speed and stability of ABR training.

[0058] 4) Stable performance on multiple devices: The ABR agent trained based on the proposed MCTS framework has good robustness and performs stably on real bandwidth data collected on different devices.

[0059] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0061] Figure 1 is a flowchart of an adaptive bit rate control method according to an exemplary embodiment;

[0062] Figure 2 is a structural block diagram of an adaptive bit rate control system according to an exemplary embodiment;

[0063] Figure 3 is a diagram of an online video on demand system according to an exemplary embodiment;

[0064] Figure 4 is a specific structural diagram of a neural network according to an exemplary embodiment;

[0065] Figure 5 is a schematic diagram of a training framework for an ABR agent based on MCTS according to an exemplary embodiment;

[0066] Figure 6 is a flowchart of a specific tree search of MCTS according to an exemplary embodiment;

[0067] Figure 7 The figure is a schematic diagram showing the structure of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION

[0068] The following description and accompanying drawings sufficiently illustrate the specific embodiments herein to enable those skilled in the art to practice them. Portions and features of some embodiments may be included in or substituted for portions and features of other embodiments. The scope of the embodiments herein includes the entire scope of the claims, including all available equivalents thereof. Herein, the terms "first," "second," and the like are used solely to distinguish one element from another and do not require or imply any actual relationship or order between these elements. In practice, the first element can also be referred to as the second element, and vice versa. Furthermore, the terms "comprise," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a structure, device, or apparatus comprising a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such structure, device, or apparatus. Without further limitation, an element defined by the phrase "comprising a..." does not preclude the presence of other identical elements in the structure, device, or apparatus comprising the element. The various embodiments herein are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Similar or identical parts between the various embodiments can be referenced to each other.

[0069] The terms "longitudinal", "transverse", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like used herein to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are intended only to facilitate the description of this document and simplify the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the present invention. In the description herein, unless otherwise specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, they can be mechanical or electrical connections, or they can be internal connections between two elements, they can be directly connected, or they can be indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms can be understood according to the specific circumstances.

[0070] As used herein, unless otherwise specified, the term "plurality" means two or more.

[0071] In this document, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B means: A or B.

[0072] In this article, the term "and / or" is used to describe the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0073] It should be understood that, although the various steps in the flowchart are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps may be performed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but may be performed at different times. The execution order of these sub-steps or stages is not necessarily to be performed in sequence, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0074] Each module in the device or system of the present application can be implemented in whole or in part by software, hardware, or a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software so that the processor can call and execute the operations corresponding to the above modules.

[0075] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0076] Figure 1 An embodiment of an adaptive bit rate control method of the present invention is shown.

[0077] In this optional embodiment, the adaptive bit rate control method includes:

[0078] Step S101: Based on the decision time point, convert the player's historical state data into input features of the deep value network to build an adaptive bitrate agent;

[0079] Step S102: Using a Monte Carlo tree search algorithm, simulate the future playback state of the player at each decision time point to evaluate the action value representing the quality of each bitrate decision;

[0080] Step S103: train an adaptive bitrate agent according to the action value of the bitrate decision, and use the trained adaptive bitrate agent to dynamically adjust the video transmission bitrate.

[0081] In this optional embodiment, converting the player's historical state data into input features of a deep value network based on the decision time point to construct an adaptive bitrate agent includes:

[0082] At each decision point, the player's historical state data is converted into input features of the deep value network. Based on the deep value neural network, an adaptive bitrate agent is constructed.

[0083] Among them, the deep value neural network first uses a one-dimensional convolutional neural network to extract the temporal dependencies in the input information, then merges features from different inputs based on the fully connected layer, and finally outputs the probability distribution of the bitrate decision.

[0084] In this optional embodiment, the historical status data includes historical bandwidth, buffer information, and block size information, and the bit rate decision includes increasing the bit rate, maintaining the current bit rate, and decreasing the bit rate.

[0085] In this optional embodiment, the Monte Carlo tree search algorithm is used to simulate the future playback state of the player at each decision time point to evaluate the action value representing the quality of each bitrate decision, including:

[0086] Node selection: Based on the upper confidence interval tree algorithm, search from the root node to the leaf node until the node has no child nodes, then the node is regarded as a leaf node;

[0087] Simulation: Starting from the player status represented by the leaf node, simulation is performed based on the simulated bandwidth data, and the user experience value obtained from the simulation during the predetermined period is recorded;

[0088] Data update: Use the user experience value obtained by simulation to update the average user experience value of all nodes passed by the current search, and update the average value and historical visit count of all passed nodes;

[0089] The node selection, simulation and data update steps are executed cyclically for a preset number of times, and the average of the user experience values ​​of the three nodes directly connected to the root node is used as the action value of the three bitrate decisions.

[0090] Figure 2 An embodiment of an adaptive bit rate control system of the present invention is shown.

[0091] In this optional embodiment, the adaptive bit rate control system includes:

[0092] The agent building module 201 is used to convert the player's historical state data into input features of the deep value network based on the decision time point to build an adaptive bitrate agent;

[0093] The simulation module 202 is used to simulate the future playback state of the player at each decision time point using the Monte Carlo tree search algorithm to evaluate the action value representing the quality of each bitrate decision;

[0094] The agent training control module 203 is used to train the adaptive bit rate agent according to the action value of the bit rate decision, and dynamically adjust the video transmission bit rate using the trained adaptive bit rate agent.

[0095] In order to facilitate understanding of the above technical solutions of the present invention, the above technical solutions of the present invention are further explained from the perspective of architecture and principle as follows:

[0096] The present invention fully considers this limitation and directly uses MCTS to simulate future playback states during decision-making to calculate a more stable video bitrate adjustment strategy. The calculated more optimal strategy is used to train the agent, resulting in a more effective ARB agent.

[0097] The video on-demand system used in the present invention is as follows Figure 3 As shown. The video played by the user is first encoded, and video segments with different bit rates are stored in the cloud server. During the user's video on demand, the ABR algorithm dynamically determines the bit rate of the next generation of downloaded video segments based on the player status at each moment, and sends a segment download request to the cloud server. After receiving the request, the cloud server transmits the selected video segment to the user terminal device via the network and stores it in the local buffer waiting for playback. This paper proposes a new ABR algorithm based on machine learning to maximize the user's playback experience, that is, to select a higher bit rate while avoiding frequent freezes.

[0098] The present invention builds an ABR agent based on a deep value neural network to perform bit rate selection and decision making. At each decision time point t (i.e., you need to decide t The input of the neural network is the matrix , the first ( i , j ) elements for t The first i Bitrate version at time point t - j Here we consider the download time of bandwidth conditions at different time points in history to introduce historical bandwidth data. In addition, the input of the neural network also includes t - j Total playback length of downloaded videos at the moment The specific structure of the neural network is as follows. Figure 4 As shown in the figure, it first extracts the temporal dependency in the input information based on a one-dimensional convolutional neural network (1D CNN), then uses a fully connected layer to merge features from different inputs, and finally outputs the probability distribution of the bit rate decision (i.e., increase the bit rate, maintain the current bit rate, or reduce the bit rate). .

[0099] The core of the present invention is the training framework of the ABR agent constructed above based on MCTS. Figure 5As shown, the technical keys are as follows:

[0100] System state capture: At each decision point, the player state (historical bandwidth, buffer, chunk size) is converted into input features of the deep value network using the method described above.

[0101] MCTS simulation: At each decision time point t , taking the current player state as the root node, MCTS simulates the future playback situation to evaluate the action value representing the good or bad of each decision (i.e., increasing the bit rate, maintaining the current bit rate, or reducing the bit rate) .

[0102] Deep value network training: the action value obtained by MCTS simulation As labels, a deep value network representing the ABR agent is trained via mean squared error (MSE).

[0103] Figure 5 The key step in the process shown is to use MCTS simulation to obtain the action value of the bit rate decision , the specific tree search process of MCTS is as follows Figure 6 shown.

[0104] Each search starts from the current t The root node of the time player state Start by repeating the following process to obtain the value of each decision.

[0105] 1. Node selection: Select a code rate decision from the root node one by one and gradually move towards the leaf node. This paper proposes an upper confidence interval tree (Upper Confidence Bound) algorithm to select a code rate decision from the root node one by one and gradually move towards the leaf node. n The next node to be visited is determined as follows:

[0106]

[0107] Where, Indicates the next node selected, Indicates the current node n The three child nodes (corresponding to the state n Execute three types of decisions), Represents a child node The estimated value of (adjusted over time as MCTS simulations progress), Representation node The number of historical visits, The historical visit count of the current node n, A represents The lower bound of (can be adjusted as a hyperparameter), B represents The upper bound of (can be adjusted as a hyperparameter). Repeatedly call the above confidence interval tree algorithm from the root node Search towards the leaf nodes until you reach a node with no children (e.g. Figure 6 The triangle in the middle indicates the leaf node.

[0108] Starting from the player state represented by the searched leaf node, simulation is performed based on the simulated bandwidth data, and the average user QoE obtained by simulation for a fixed K period (which can be used as a hyperparameter adjustment) is recorded ( Figure 6 The user's QoE at any time point t The measurement formula is:

[0109]

[0110] Where, Indicates the average user experience. Indicates the playback reward, which is positively correlated with the bitrate of the last downloaded segment. Indicates the jam penalty, which is proportional to the time the video playback was jammed during the download of the previous segment. Indicates the bitrate change penalty. It is 1 if the bitrate of the previous fragment is different from that of the previous fragment, otherwise it is 0. 、 、 are positive hyperparameters representing playback rewards, lag penalties, and bitrate conversion penalties, respectively. During the simulation, the following non-judge Morton process is used to generate random bandwidth trajectories to simulate the non-stationary bandwidth trajectories in reality:

[0111]

[0112] The expression for generating simulation bandwidth based on bandwidth characteristics is:

[0113]

[0114] Where, J t Indicates correspondence t Time has come t The characteristic value of the average bandwidth between +1 moments, Z 1. Z 2. Z 3 represent random variables used to introduce bandwidth randomness, which obey the standard normal distribution, represents a hyperparameter that controls the non-stationarity of the generated bandwidth, represents the hyperparameter that controls the overall magnitude and bandwidth, Indicates the generated t Time has come t +1 Average bandwidth between moments.

[0115] Update the obtained user QoE statistics to all nodes passed by this search ( Figure 6 The QoE statistics of all nodes on the route indicated by the dotted line indicated by the value update are averaged. n , first update its average value V n ,Right now:

[0116]

[0117] Then update its historical visit count N n ,Right now:

[0118]

[0119] Where, represents the average value before updating, V n represents the updated average value, Represents the number of historical visits before the update, N n Indicates the updated historical visit count. Indicates the average user experience.

[0120] Cycle the above three steps N After that (can be used as a hyperparameter adjustment), finally the three nodes directly connected to the root node ( Figure 6 The nodes shown , , )’s QoE statistical average (i.e. , , ) as the value of the three decisions obtained through MCTS (i.e., increase the bit rate, maintain the current bit rate, or reduce the bit rate) .

[0121] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store static information and dynamic information data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment are implemented.

[0122] Those skilled in the art will understand that Figure 7 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0123] In addition, the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiment when executing the computer program.

[0124] In addition, the present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0125] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware using a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes in the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0126] The present invention is not limited to the structures described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An adaptive bit rate control method, characterized in that: include: Based on the decision time point, the player's historical state data is converted into input features of the deep value network to build an adaptive bitrate agent; Using the Monte Carlo Tree Search algorithm, the player's future playback state is simulated at each decision point in time, and the action value representing the quality of each bitrate decision is evaluated; An adaptive bitrate agent is trained based on the action value of the bitrate decision, and the trained adaptive bitrate agent is used to dynamically adjust the video transmission bitrate. The method of converting the player's historical state data into input features of a deep value network based on the decision time point to construct an adaptive bitrate agent includes: At each decision point, the player's historical state data is converted into input features of the deep value network. Based on the deep value neural network, an adaptive bitrate agent is constructed. The deep value neural network first uses a one-dimensional convolutional neural network to extract the temporal dependencies in the input information, then merges features from different inputs using a fully connected layer, and finally outputs a probability distribution for bitrate decisions; The Monte Carlo tree search algorithm is used to simulate the future playback state of the player at each decision time point, and the action value representing the quality of each bitrate decision is evaluated, including: Node selection: Based on the upper confidence interval tree algorithm, search from the root node to the leaf node until the node has no child nodes, then the node is regarded as a leaf node; Simulation: Starting from the player status represented by the leaf node, simulation is performed based on the simulated bandwidth data, and the user experience value obtained from the simulation during the predetermined period is recorded; Data update: Use the user experience value obtained by simulation to update the average user experience value of all nodes passed by the current search, and update the average value and historical visit count of all passed nodes; The node selection, simulation and data update steps are executed cyclically for a preset number of times, and the average of the user experience values ​​of the three nodes directly connected to the root node is used as the action value of the three bitrate decisions.

2. The adaptive bit rate control method according to claim 1, wherein: The historical status data includes historical bandwidth, buffer information and block size information, and the bit rate decision includes increasing the bit rate, maintaining the current bit rate and reducing the bit rate.

3. The adaptive bit rate control method according to claim 1, wherein: The upper confidence interval tree algorithm is used to determine the next visited node at each node, and the expression of the upper confidence interval tree algorithm is: ； Where, Indicates the next node selected, Indicates the current node n The three child nodes of Represents a child node a The estimated value of Represents a child node The number of historical visits, Indicates the current node n The number of historical visits, A represents The lower bound of B represents The upper bound of .

4. The adaptive bit rate control method according to claim 1, wherein: The expression of the average user experience is: ； Where, Indicates the average user experience. Indicates playback rewards, Indicates the penalty for jamming. represents the bitrate conversion penalty, 、 、 are the positive hyperparameters representing playback reward, lag penalty, and bitrate change penalty, respectively.

5. The adaptive bit rate control method according to claim 1, wherein: The step of starting from the player state represented by the leaf node, performing simulation based on the simulated bandwidth data, and recording the average user experience value obtained by the simulation during a predetermined period further includes: During the simulation, the non-judge Morton process is used to generate random bandwidth trajectories to simulate the real non-stationary bandwidth trajectories.

6. The adaptive bit rate control method according to claim 5, wherein: The expression used to generate the simulation bandwidth characteristic is: ； The expression for generating simulation bandwidth based on bandwidth characteristics is: ； Where, J t Indicates correspondence t Time has come t The characteristic value of the average bandwidth between +1 moments, Z 1. Z 2. Z 3 represent random variables used to introduce bandwidth randomness, represents a hyperparameter that controls the non-stationarity of the generated bandwidth, represents the hyperparameter that controls the overall magnitude and bandwidth, Indicates the generated t Time has come t +1 Average bandwidth between moments.

7. The adaptive bit rate control method according to claim 1, wherein: The updating formula of the average value is: ； The update formula for the number of historical visits is: ； Where, represents the average value before updating, V n represents the updated average value, Represents the number of historical visits before the update, N n Indicates the updated historical visit count. Indicates the average user experience.

8. An adaptive bit rate control system, characterized in that: It includes intelligent agent construction module, simulation module and intelligent agent training control module; The agent building module is used to convert the player's historical state data into input features of the deep value network based on the decision time point to build an adaptive bitrate agent; The simulation module is used to simulate the future playback state of the player at each decision time point using the Monte Carlo tree search algorithm to evaluate the action value representing the quality of each bitrate decision; The agent training control module is used to train the adaptive bitrate agent according to the action value of the bitrate decision, and dynamically adjust the video transmission bitrate using the trained adaptive bitrate agent; The agent building module converts the player's historical state data into input features of the deep value network based on the decision time point to build an adaptive bitrate agent, including: At each decision point, the player's historical state data is converted into input features of the deep value network. Based on the deep value neural network, an adaptive bitrate agent is constructed. The deep value neural network first uses a one-dimensional convolutional neural network to extract the temporal dependencies in the input information, then merges features from different inputs using a fully connected layer, and finally outputs a probability distribution for bitrate decisions; The simulation module uses the Monte Carlo tree search algorithm to simulate the future playback state of the player at each decision time point and evaluate the action value representing the quality of each bitrate decision, including: Node selection: Based on the upper confidence interval tree algorithm, search from the root node to the leaf node until the node has no child nodes, then the node is regarded as a leaf node; Simulation: Starting from the player status represented by the leaf node, simulation is performed based on the simulated bandwidth data, and the user experience value obtained from the simulation during the predetermined period is recorded; Data update: Use the user experience value obtained by simulation to update the average user experience value of all nodes passed by the current search, and update the average value and historical visit count of all passed nodes; The node selection, simulation and data update steps are executed cyclically for a preset number of times, and the average of the user experience values ​​of the three nodes directly connected to the root node is used as the action value of the three bitrate decisions.

Images