Video transmission rate control method under three-dimensional heterogeneous network and related equipment
By using the Harmony algorithm in the LEO low-orbit satellite environment, the video transmission rate is optimized using the multi-agent DDPG model, the problems of video transmission quality fluctuations and high energy consumption are solved, and the user experience quality and fairness are improved.
Patent Information
- Application Number
- CN202411191932.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-28
AI Technical Summary
In the LEO low-orbit satellite Internet environment, the rate control of video on-demand service is difficult to adapt to dynamic changes in the network, resulting in large fluctuations in video transmission quality, high energy consumption and poor user fairness.
Using deep deterministic strategy gradient DDPG algorithm based on reinforcement learning, the Harmony algorithm solution is designed, and distributed training is carried out through multi-agent DDPG models to achieve adaptation to different ABR algorithms and availability of dynamic users, and optimize the video transmission rate to maximize Improve user experience quality, minimize energy consumption and improve user experience quality fairness.
The stability of video transmission quality in the last hop on-demand video transmission scenario of LEO low-orbit satellite is achieved, which reduces energy consumption, improves user experience quality and fairness, and solves the problems of fluctuations in video transmission quality and high energy consumption.
Smart Images

Figure CN119094823A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of satellite Internet and streaming media technology, and in particular to a method and related equipment for controlling video transmission rate in a three-dimensional heterogeneous network. Background Art
[0002] With the rapid development of Internet technology and the increasing user demand, video on demand (VoD) has become an important part of Internet services. Traditional ground networks cannot meet the coverage needs of remote and rural areas around the world, which has prompted people to explore new ways of Internet access. Low Earth Orbit (LEO) satellite Internet has become one of the important solutions for future Internet access due to its wide coverage and low latency. The low earth orbit satellite system consists of multiple satellites. The satellites orbit the earth at a low altitude (usually between 500 and 2000 kilometers), so their signal propagation delay is low, which is suitable for high-bandwidth, low-latency Internet applications. However, the LEO satellite system also faces many challenges, such as frequent satellite switching, multipath interference, and limited spectrum resources.
[0003] The demand for video on demand services requires stable bandwidth and low latency to ensure a high-quality video transmission experience. In the LEO satellite Internet environment, rate control of video on demand services becomes particularly critical because network conditions may fluctuate due to factors such as satellite movement and weather changes. Therefore, a rate control solution that can adapt to dynamic changes in the network is needed to ensure the quality of video on demand services. Summary of the invention
[0004] In view of this, the purpose of this application is to propose a video transmission rate control method and related equipment in a three-dimensional heterogeneous network.
[0005] Based on the above purpose, the present application provides a method for controlling video transmission rate in a three-dimensional heterogeneous network, including:
[0006] Constructing an optimization target for LEO satellite Internet on-demand video transmission rate control; the optimization target is a multi-dimensional global optimization target, including maximizing the overall user experience quality, minimizing energy consumption, and maximizing the fairness of user experience quality;
[0007] Decomposing the optimization objective into distributed sub-optimization objectives;
[0008] Obtaining the status information and cooperation information of all intelligent agents; wherein the status information of each intelligent agent includes the channel coefficient, other target channel interference and corresponding user experience quality information; the cooperation information includes the average user experience quality of all users except itself at the current moment, the maximum user experience quality difference, and the total power consumption, average power consumption and maximum power consumption of other intelligent agents;
[0009] The state information and cooperation information of the agent are input into the DDPG model of multiple agent interactions to obtain the rate decision set made by the satellite for each user.
[0010] In some embodiments, the optimization goal is to maximize the utility U of the jth video block at each time instant. j ;in, α 1 is a hyperparameter; α 2 is a hyperparameter; α 3 is a hyperparameter; is the rate decision for the actual state at time t, QoE i,j is the user experience quality of the j-th video block of user i; Among them, μ 1 is a constant; μ 2 is a constant; μ 3 is a constant; is the video quality of the jth video block of user i; The impact of rebuffering events on user quality of experience; The effect of changes in video quality during playback; P i,j The energy consumed to download video chunk j for user i; QoE i′,j is the user experience quality of the j-th video chunk of user i'.
[0011] In some embodiments, the distributed sub-optimization objectives include Among them, U i,j is the utility of the jth video block of user i; buffer i,j is the buffer length of user i at the jth video block; i,j-1 is the buffer length of user i at the j-1th video block.
[0012] In some embodiments, the method further comprises determining a DDPG model of the interaction of multiple agents by:
[0013] Constructing a rate decision process model to determine the input, joint strategy and output of the DDPG model of the multiple agents interacting;
[0014] Determine the cumulative value function, action value function, parameter update method of the policy network during training, parameter update method of the evaluation network during training, and output of the target network of the DDPG model of the multiple agents interacting;
[0015] Identify the global network;
[0016] In response to determining that the agent has completed training of the training video clip, centralized updating and distributed propagation are performed.
[0017] In some embodiments, the rate decision process model includes:
[0018] Environment and state, including S≡{S Q ,S P ,S r ,S B ,S h}, where S is the set of all state information of all agents in the satellite network during the rate decision process; S Q is the user experience quality value of n clients watching the video on demand at a given time, S Q ≡{Q 1 ,…,Q n}; S P is the power value allocated by the satellite to n users, S P ≡{P 1 ,…,P n}; S r is the bit rate level request sent by n clients; SB is the buffer status notification; and S h is the channel state parameter of n clients; the environment has a real state s, s∈S;
[0019] Actions, including the actions u of each agent a ;u a ∈U, U is the rate decision set r made by the satellite for n clients; at each time step j, the decision action of each agent will refer to the actions of other agents to form a joint action u of multiple agents;
[0020] The transition probability, including the state transition probability function P, satisfies P(s'│s,u):S×U×S→[0,1]; s is the current state, s' is the next state;
[0021] Rewards, including a global reward R shared by all agents, satisfying R(s,u):S×U→R; the global reward includes the average user quality experience value, average energy consumption and user quality experience fairness considerations of all users at a given time; the global reward includes the optimization target;
[0022] Observation, including the observation value z of each agent at each time step, z∈Z, where Z is the set of all agent observation values; the action observation history τ of each agent a ∈T≡(Z×U)*; the personal reward of each agent includes the distributed sub-optimization objective.
[0023] In some embodiments, the cumulative value function includes: in, is the cumulative value function of agent a under strategy π, z a is the observation value of each agent a, j is the index of the video block, γ j is the discount factor of video block j; E is the set of all agent observations, τ a is the initial state; are the parameters of the policy network;
[0024] The action value function includes: in, is the action value function of agent a under strategy π, is the observation value of agent a on video block j, is the action of agent a on video block j; r a is the rate decision made by the satellite for the client corresponding to agent a;
[0025] The method for updating the parameters of the policy network during the training process includes: updating the policy network using gradient descent Parameters in, Strategy Network Parameters; is the parameter of the policy network corresponding to the policy π;
[0026] The parameter updating method of the evaluation network during the training process includes: updating the evaluation network using a gradient descent based on a time difference algorithm Parameters
[0027] The outputs of the target network include: Among them, y j For the target network Output: is the observation value of agent a on video block j, is the action of agent a on video block j; is the observation value of agent a on video block j+1, is the action of agent a on video block j+1.
[0028] In some embodiments, the centralized updating includes: updating the global network according to the training video segment parameters; the updating includes: Θ g′ ←φ(ep)·Θ a +(1-φ(ep))·Θ g ; Among them, Θ g' To update the global network parameter set; Θ g is the global network parameter set; ep is the currently executing episode; φ(ep) is the effect of the episode parameter; k is the hyperparameter that determines the shape of the φ(·) function; ep 0 is the hyperparameter that determines the center position;
[0029] The distributed propagation includes: Θ a′ =η(j,ep)·Θ g′ +(1-η(j,ep))·Θ a ; where Θ a′ is the updated network parameter set of agent a; Θ a is the network parameter set of agent a; Where η(j,ep) is the combined effect of step and episode parameters; Clip function is a truncation function with a value limit of 0 to 1; L is the upper limit of the lower limit Clip function variable, and H is the lower limit of the upper limit Clip function variable.
[0030] The embodiment of the present application also provides a video transmission rate control device in a three-dimensional heterogeneous network, including:
[0031] An optimization target building module is configured to build an optimization target for LEO satellite Internet on-demand video transmission rate control; the optimization target is a multi-dimensional global optimization target, including maximizing the overall user experience quality, minimizing energy consumption and maximizing user experience quality fairness;
[0032] A distributed sub-optimization target building module, configured to decompose the optimization target into distributed sub-optimization targets;
[0033] An information acquisition module is configured to acquire the state information and cooperation information of all agents; wherein the state information of each agent includes the channel coefficient, other target channel interference and corresponding user experience quality information; the cooperation information includes the average user experience quality of all users except itself at the current moment, the maximum user experience quality difference, and the total power consumption, average power consumption and maximum power consumption of other agents;
[0034] The rate decision set acquisition module is used to input the state information and cooperation information of the agent into the DDPG model of multiple agent interactions to obtain the rate decision set made by the satellite for each user.
[0035] An embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the methods described above when executing the program.
[0036] An embodiment of the present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any of the methods described above.
[0037] An embodiment of the present application further provides a computer program product, comprising computer program instructions, which, when executed on a computer, enable the computer to execute any of the methods described above.
[0038] From the above description, it can be seen that the video transmission rate control method and related equipment in a three-dimensional heterogeneous network provided in the embodiment of the present application can, to a certain extent, solve the problems of large fluctuations in video transmission quality, high energy consumption, and poor user fairness when the existing rate control method is applied to the last-hop on-demand video transmission scenario of LEO low-orbit satellites. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0040] Figure 1 A schematic diagram of the flow of a method for controlling a video transmission rate in a three-dimensional heterogeneous network according to an embodiment of the present application;
[0041] Figure 2 A schematic diagram of the execution logic of the LEO satellite Internet video on demand framework and rate allocation algorithm according to an embodiment of the present application;
[0042] Figure 3 A schematic diagram of a flow chart of a DDPG model for determining the interaction of multiple agents according to an embodiment of the present application;
[0043] Figure 4 A schematic diagram of the design details of the Harmony rate allocation algorithm according to an embodiment of the present application;
[0044] Figure 5A schematic diagram of a video transmission rate control device in a three-dimensional heterogeneous network according to an embodiment of the present application;
[0045] Figure 6 A schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0046] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.
[0047] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0048] At present, most rate control schemes are mainly designed for ground fixed networks, such as the congestion control algorithm in the TCP protocol. These schemes have certain limitations in dealing with the dynamic characteristics of LEO satellite Internet. They are manifested as 1) poor adaptability: traditional rate control algorithms cannot respond to satellite switching and link changes in a timely manner, resulting in large fluctuations in video transmission quality; 2) low resource utilization: the spectrum resources of the LEO satellite network cannot be fully utilized, resulting in low bandwidth utilization; 3) high energy consumption: in satellite communications, high energy consumption is an important issue, and existing schemes lack effective mechanisms for energy consumption control; 4) it is difficult to ensure user fairness: for all users accessing video applications within the coverage of the last hop of a satellite, fairness among viewers is crucial to the sustainable and long-term development of multi-user systems.
[0049] In the context of global advocacy of green energy conservation, how to reduce energy consumption while ensuring service quality and user fairness is an important research direction for LEO satellite Internet. However, there are two challenges in implementing this solution design: 1) The uncontrollability and unpredictability of the ABR (Adaptive Bitrate Streaming) algorithm: Typically, VoD involves the client using the ABR algorithm to request the bit rate level and then sending it to the server. For the last-hop satellite mobile edge computing (MEC) server, it is difficult to determine the type of ABR algorithm used by different client applications because these algorithms are usually designed internally by the content platform. Therefore, it is a major challenge to design a rate control scheme that can be effectively executed in an environment with different ABR algorithms. 2) User dynamics: The user group in the satellite coverage area is highly dynamic, and frequent user joining and leaving increases the complexity of algorithm design. This is particularly challenging for NOMA systems because each user may affect the transmission rate of other users.
[0050] Therefore, the existing rate control schemes, when dealing with the last-hop on-demand video transmission scenario of LEO low-orbit satellites, have problems such as large fluctuations in video transmission quality, high energy consumption, and poor user fairness.
[0051] Based on this, the embodiment of the present application provides a last-hop on-demand video transmission scenario for LEO low-orbit satellites. From the perspective of ensuring service quality, reducing energy consumption and ensuring user fairness, an algorithm solution named Harmony is designed and implemented, which is a deep deterministic policy gradient DDPG algorithm rate control solution based on reinforcement learning. Harmony improves the multi-agent DDPG into a distributed training framework, in which multiple single-agent DDPGs run simultaneously, with a dynamically iterative training architecture and a unified input dimension to achieve adaptation to different ABR algorithms and availability for dynamically changing numbers of users. Harmony implements an excellent on-demand video multi-objective rate control solution that meets user experience quality (Quality of Experience, QoE), energy consumption and user fairness. It can solve the problems of large fluctuations in video transmission quality, high energy consumption and poor user fairness when the existing rate control method is applied to the last-hop on-demand video transmission scenario of LEO low-orbit satellites to a certain extent.
[0052] Reference Figure 1As shown, the embodiment of the present application provides a method for controlling the video transmission rate in a stereoscopic heterogeneous network. The stereoscopic heterogeneous network may be a LEO satellite Internet. Among them, stereoscopic heterogeneity can be understood as a state of cooperation between a satellite and the ground. The video transmission rate control method may include:
[0053] S100, constructing an optimization target for LEO satellite Internet on-demand video transmission rate control; the optimization target is a multi-dimensional global optimization target, including maximizing the overall user experience quality, minimizing energy consumption, and maximizing user experience quality fairness;
[0054] Step S200, decomposing the optimization target into distributed sub-optimization targets;
[0055] Step S300, obtaining the state information and cooperation information of all agents; wherein the state information of each agent includes the channel coefficient, other target channel interference and corresponding user experience quality information; the cooperation information includes the average user experience quality of all users except itself at the current moment, the maximum user experience quality difference, and the total power consumption, average power consumption and maximum power consumption of other agents;
[0056] Step S400, inputting the state information and cooperation information of the agent into the DDPG model of multiple agent interactions, and obtaining the rate decision set made by the satellite for each user.
[0057] Among them, in step S100, if Figure 2 As shown, the optimization objective comprehensively considers the user experience quality QoE of video block j of a single user i, the energy consumption index and the user experience quality QoE fairness among different users in the scene.
[0058] In some embodiments, the rate control time point is set at the beginning of each video block transmission. i Three factors are usually considered: the quality of watching video chunks, playback freezes, and bitrate switching. The user experience quality QoE of each video chunk of a single user can include the quality of watching video chunk j, the negative impact of rebuffering events on QoE, and the adverse effects of frequent changes in video quality. The user experience quality QoE of the jth video chunk of user i i,j The quality of the video block being viewed, playback freezes, and bitrate switching may be a weighted sum.
[0059] In some of these embodiments, Among them, μ 1 is a constant; μ 2 is a constant; μ 3 is a constant; is the video quality of the jth video block of user i. Set v m represents the utility of watching at quality level m, which is usually proportional to the bitrate level. In a common approach, the bitrate level m can be directly used as this v m For example, a video can be encoded at six bitrate levels, including {640, 1600, 4000, 8400, 12000, 16400} kbps. m The value of can be bit rate level m. The impact of rebuffering events on user quality of experience. Among them, S m The size of the chunk file viewed at quality level m, which can be understood as size. is the rate of user i at time t. i,j is the status of the buffer of the j-th video block of user i. The effect of changes in video quality during playback. in, is the video quality of the j-1th video block of user i.
[0060] In some embodiments, the satellite allocates a transmission rate to each user based on the energy consumption index, thereby affecting the transmission power and the energy consumed during the actual video transmission process. Since the satellite mainly relies on solar energy, the energy of the satellite is relatively scarce. Assume that for user i downloading block j, the energy consumed is P i,j .
[0061] In some of the embodiments, regarding the fairness of the user experience quality (QoE) between different users in the scenario, at any given time, there should be no obvious difference in the QoE of different users in the same area accessing the satellite Internet to watch videos, so QoE fairness must be ensured. It can be expressed as Among them, U s,v It can be a collection of users who access the satellite and watch the video in the scene. i′,j is the user experience quality of the j-th video chunk of user i'.
[0062] In some embodiments, the optimization goal is to maximize the overall QoE utility through rate control, minimize energy consumption, and achieve QoE fairness as much as possible. The utility of each j-th block can be expressed as (1), where α z , z∈{1,2,3} is a hyperparameter used to balance different indicators. The overall optimization goal is to maximize U at each moment. j Among them, U j is the utility of the jth video block; is the set of users who access the satellite and watch the video at time t. i,j is the user experience quality of the jth video block of user i; P i,j The energy consumed to download video chunk j for user i; QoE i′,j is the user experience quality of the j-th video chunk of user i'.
[0063] In step S200, the global centralized optimization problem is decomposed into distributed sub-optimization problems, which can simplify the optimization problem. In this way, the global optimization problem in each time slot j (i.e., the time corresponding to the start of the transmission of video block j) which is difficult to solve with general mathematical methods can be solved to a certain extent. j Therefore, the rate decision will affect the decision of its users, which is given by Shannon's formula and becomes an NP-hard problem. And the function for making rate decisions is non-convex, so it is difficult to use optimization algorithms. In addition, among the optimization objectives, the user experience quality QoE does not have a clear mathematical expression and can only be obtained through interaction and feedback with the client application.
[0064] In some of the embodiments, the distributed sub-optimization objective may be as shown in formula (2); Among them, U i,j is the utility of the jth video block of user i; α 1 is a hyperparameter; α 2 is a hyperparameter; α 3 is a hyperparameter; α 4 is a hyperparameter; buffer i,j is the buffer length of user i at the jth video block; i,j-1 is the buffer length of user i at the j-1th video block. The first part, i.e., part 1, is derived from the first two terms in equation (1) of the global objective (i.e., the optimization objective).
[0065] In some embodiments, in formula (2), the third term represents the maximum QoE difference between the current user and other users, which can replace the maximum QoE difference term in the user set in formula (1) to achieve the same effect. Minimize each sub-goal utility U i,j The QoE difference within the range ensures that the overall target utility U j The overall QoE difference within is minimized. This subproblem shows transitivity. The mathematical derivation can be as follows: Therefore, when optimizing, we minimize the QoE of each user i.i -QoE i′ |The difference is a stricter optimization problem. Therefore, the optimality of the subproblems can be transformed into the optimality of the global problem.
[0066] In some of the embodiments, in the sub-problem (2), the buffer change term can be used as an additional optimization target in the sub-problem. This is because: the continuous rapid growth of the buffer length will lead to a decrease in the overall utility. The increase in the buffer length of each user may not necessarily improve the user's QoE, but it will definitely reduce power efficiency. This logical relationship is implicit in the first three terms of formula (2). For algorithm sensitivity, this logical relationship can be explicitly designed into the utility of the sub-problem. Assume that there is a threshold buffer thresh , so that when the buffer i,j >buffer thresh When , no rebuffering event occurs, and the QoE remains unchanged, resulting in Consider the buffer zone utility U i,j Impact: Where (buffer i,j -buffer i,j-1 ) is the buffer increment. Therefore, when the buffer i,j >buffer thresh When , the buffer can be considered large enough and further increments have minimal impact on utility. Therefore, the main impact on utility under these conditions is due to power consumption. Therefore, if the buffer is increased simply by increasing the transmission rate, QoE cannot be improved. On the contrary, individual utility U may be reduced due to the increase in transmission power. i,j , thus the total utility U j Have an adverse effect.
[0067] In step S300, in some of the embodiments, in addition to the state information of each agent, the interaction between multiple agents can be represented by additional input, and the joint state of each agent can be simplified to its own observation state. After that, each agent can perform its operations in a distributed manner. In this way, the multi-agent training process can be simplified to a cooperative game involving multiple single agents. It can solve the problem that the joint state s and joint action u increase sharply with the increase of the number of clients n using traditional centralized multi-agent reinforcement learning, which in turn leads to increased computational complexity, reduced learning efficiency and reduced learning effect. In addition, it can also solve the problem that in a dynamic environment, the fluctuation of the number of clients causes the number of agents to change over time, resulting in discontinuous training.
[0068] In some embodiments, for each agent a, the observation content can be updated as z a′≡{QoE -a ,P -a ,z a}, where z a Contains information specific to agent a, including the channel coefficient h a , other target channel interference And the corresponding user QoE information {buffer a ,bitrate a ,Rebuffer_event a ,Bitrate_change a}. The superscript -a may be a joint variable on an agent other than the given agent a. The joint variable may include centralized update and distributed propagation. That is, the state information of each agent includes channel coefficients, other target channel interferences, and corresponding user experience quality information.
[0069] In some embodiments, the difference between the average user experience quality and the maximum user experience quality of all users except the user at the current moment may include: Among them, N(a) is the set of agents except a, N a =|N(a)| represents the number of -a sets. k is the user index other than itself, is the average QoE of all users except itself at the current moment, The maximum user experience quality other than itself at the current moment; QoE k’ is the minimum user experience quality of the user other than itself at the current moment. In this way, the user experience quality of other users can be measured.
[0070] In some of these embodiments, the total power consumption, average power consumption, and maximum power consumption of other agents may be Among them, ∑ k∈N(a) P k is the total power consumption of other agents; is the average power consumption of other agents; is the maximum power consumption of other agents. In this way, the power consumption of other users can be measured. Combined with the average user experience quality and maximum user experience quality difference of all users except itself at the current moment, the status of other users can be measured.
[0071] In some embodiments, N agents provide on-demand transmission services to N users at the same time, forming a cooperative game alliance, and the joint strategy is Players (i.e., users / agents) choose actions based on their own strategies, and their actions influence each other, as shown in Shannon's formula. The goal of the cooperative game is to maximize the overall utility of the alliance, which is also the optimization goal of formula (1). The optimal joint strategy under the Nash equilibrium is expressed as In a Nash equilibrium, no agent has an incentive to unilaterally change its strategy.
[0072] like Figure 3 As shown, the video transmission rate control method under the three-dimensional heterogeneous network of the embodiment of the present application can also include determining the DDPG model of the interaction of multiple agents by the following method:
[0073] Step S510, constructing a rate decision process model to determine the input, joint strategy and output of the DDPG model of the multiple agents interacting;
[0074] Step S520, determining the cumulative value function, the action value function, the parameter update method of the policy network during the training process, the parameter update method of the evaluation network during the training process, and the output of the target network of the DDPG model of the multiple agents interacting;
[0075] Step S530, determining the global network;
[0076] Step S540, in response to determining that the agent has completed the training of the training video clip, performs centralized updating and distributed propagation.
[0077] In some embodiments, in step S510, the rate decision process model may be a partially observed Markov problem (POMDP), representing a fully cooperative multi-agent task, where the decisions made by the agents affect each other. This can describe the complex process of the rate control problem for VoD services in LEO satellite networks, where the state changes over time and at any given moment depends on the joint decision result at the previous moment, and can take into account the unobservability of the internal application state of each user.
[0078] In some embodiments, let the random game be G, defined as a tuple G j =<S,U,P,R,Z,O,r,n,γ> , where n agents are defined as a∈A≡{1,…,n}, selecting continuous actions; γ∈[0,1) is the discount factor. It is worth noting that G j As the start time of video block transmission (i.e., time slot j) changes, the number of agents n will also change dynamically.
[0079] In some embodiments, the environment and state may include S≡{S Q ,S P ,S r ,SB ,S h}, where S is the set of all state information of all agents in the satellite network during the rate decision process. Q is the QoE value of n clients watching the video on demand at a given time, S Q ≡{Q 1 ,…,Q n}. P is the power value allocated by the satellite to n users, S P ≡{P 1 ,…,P n}; S r The bit rate level request sent by n clients; S B is the buffer status notification; and S h are the channel state parameters of n clients; the environment has a real state s, s∈S.
[0080] In some embodiments, the action may include the action u of each agent a Action a is a continuous value. a ∈U, U is the rate decision set r made by the satellite MEC for n clients; at each time step j, the decision action of each agent will refer to the actions of other agents to form a joint action u of multiple agents. u∈U≡U n .
[0081] In some embodiments, the transition probability may include a state transition probability function P, satisfying P(s'│s,u):S×U×S→[0,1]. Where s is the current state, and s' is the next state. S is the set of all state information of all agents in the satellite network during the rate decision process. U is the rate decision set r made by the satellite MEC for n clients.
[0082] In some embodiments, the reward may include a global reward R shared by all agents, satisfying R(s,u):S×U→R. The global reward is given by the above formula (1), including the average user quality experience QoE value of all users at a given time, the average energy consumption, and the user quality experience QoE fairness consideration. That is, the global reward includes the optimization target.
[0083] In some embodiments, the observation may include a partial observation setting, including at each time step, an observation value z of each agent, z∈Z, where Z is the set of observation values of all agents. The action observation history τ of each agent a ∈T≡(Z×U)*. U is the set of rate decisions r made by the satellite MEC for n clients. This trajectory usually depends on the random strategy π a (ua ,τ a ):T×U→[0,1]. Among them, π a (u a ,τ a ) is the random strategy of each agent a, u a The action of each agent, T is the duration of the action observation history of each agent; U is the rate decision set r made by the satellite MEC for n clients. In general, the personal reward of each agent can be given by formula (2), which can include the distributed sub-optimization objective.
[0084] In some embodiments, in step S520, an iterative process of designing a single-agent DDPG for the last hop rate control of a LEO satellite is designed, that is, an iterative process of determining a DDPG model of multiple agents interacting for the last hop rate control of a LEO satellite. For agent a, the policy network and the critic network can be defined as and Denote the target policy network and target critic network as and The parameter set Θ of each agent a Defined as a set of parameters for four networks,
[0085] In some embodiments, the initial state of agent a is set to τ a , the cumulative value function under strategy π can be: is the cumulative value function of agent a under strategy π, z a is the observation value of each agent a, j is the index of the video block, γ j is the discount factor of video block j; E is the set of all agent observations, r j a is the rate of video block j, are the parameters of the policy network.
[0086] In some embodiments, according to the Bellman equation, the action-value function includes: in, is the action value function of agent a under strategy π, is the observation value of agent a on video block j, is the action of agent a on video block j; is the observation value of agent a on video block j and the action of agent a on video block j, and the action value function under strategy π; r a is the rate decision made by the satellite for the client corresponding to agent a; γ is the discount factor, E is the expectation operation, zj+1 is the observed value of the j+1th video block, Ez j+1 is the expected value of the j+1th video block. At the same time, the cumulative value function It can also be written as
[0087] In some embodiments, the method for updating the parameters of the policy network during training includes: updating the policy network using gradient descent Parameters in, Strategy Network The change value of the parameter; is the parameter of the policy network corresponding to the policy π; is the observed value of agent a and the expected value under the cumulative value function, is the gradient operator, This is the derivation of gradient descent.
[0089] In some embodiments, the parameter updating method of the evaluation network during the training process may include: updating the evaluation network using a gradient descent based on a time difference algorithm Parameters The loss function is Among them, y j For the target network The output of the target network includes: Among them, y j For the target network Output of a is the rate decision made by the satellite for the client corresponding to agent a; is the observation value of agent a on video block j, is the action of agent a on video block j. is the action value function of agent a under the policy network parameters and policy π, is the action value function of agent a under the updated policy network parameters and policy π.
[0090] In some embodiments, in step S540, the Harmony framework is as follows: Figure 4As shown in the figure. In actual satellite service scenarios, users randomly join or leave the media service, which will cause the number of agents in the satellite MEC to change over time. It can be understood that a single satellite serves multiple users, one user corresponds to one agent, and when a user joins or leaves, the number of agents in the satellite service will change. Usually, all agents are maintained as a thread or coroutine in the satellite MEC. Since the cooperation information in the input information is in the form of mean and variance that does not change with the number of users, the problem of the change in the number of agents at the input dimension level is solved.
[0091] In some of the embodiments, the problem of updating the model during training can be solved through a distributed dynamic framework, focusing on the timing of the update. Each agent has a complete DDPG structure, and an agent is maintained for each user in the video application. It is assumed that a user watches a video as a training segment, and watching a video block is a step, where the index j of the video block can be a step parameter, describing the interaction between a single agent and the environment until the jth step; ep is a segment parameter, indicating the segment currently being executed. Therefore, a global network is maintained, and at a certain moment, the network parameter set of agent a is expressed as The global network parameter set is denoted as Θ g When agent a completes the training of an entire episode, two steps are performed: centralized update and distributed propagation.
[0092] In practical applications, the video lengths in the video dataset can be 100, 125, 150, 200, and 250 video blocks, respectively, and the length of each video block can be unified to 4 seconds. For the training process, the classic ABR algorithm BOLA can be implemented in the environment.
[0093] In some embodiments, the centralized updating may include: updating the global network according to the segment parameters; the updating includes: Θ g′ ←φ(ep)·Θ a +(1-φ(ep))·Θ g ; Among them, Θ g' is the set of global network parameters updated after the training of the training video clips is completed; Θ g is the global network parameter set before the training of the training video segment is completed; ep is the segment parameter, indicating the segment currently being executed; φ(ep) is the influence of the episode parameter; k is the hyperparameter that determines the shape of the φ(·) function; ep 0 is a hyperparameter that determines the center position. Thus, under this strategy, as the number of episodes increases, Θ a Pair Θ g That is, the later the agent joins, the smaller its influence on the global network.
[0094] In some embodiments, the distributed propagation may include: Θ a′ =η(j,ep)·Θ g′ +(1-η(j,ep))·Θ a ; Where η(j,ep) is the combined effect of step and episode parameters; Clip function is a truncation function with a value limit of 0 to 1; L is the upper limit of the lower limit Clip function variable, and H is the lower limit of the upper limit Clip function variable. Θ a′ is the network parameter set for updating agent a after training the training video clips; Θ a is the network parameter set of agent a before the training of the training video clip is completed. Thus, under this strategy, the larger the step size, the fewer new global network synchronization updates a single network receives, but it is also affected by the episode parameter. When a new user joins, the parameter set of the current global network It will be directly propagated and synchronized to the new agent, and continue training execution. In other words, the later the agent joins, the greater the influence of the global network, while the earlier the agent joins, the less the influence of the global network.
[0095] In some of the embodiments, in step S400, the state information and cooperation information of all the single agents obtained above are input into the DDPG model of multiple agent interactions, so that the rate decision set made by the satellite for each user can be obtained.
[0096] The Harmony video transmission rate control scheme for the last hop of the LEO low-orbit satellite Internet provided in the embodiment of the present application designs a mechanism for real-time monitoring of the LEO satellite Internet network status, dynamically adjusts the video transmission rate to adapt to the dynamic changes of the network, and ensures the stability of the user experience. A prediction algorithm based on reinforcement learning is proposed to predict the future network status by analyzing historical data and real-time network status, and adjust the transmission rate in advance. An energy-aware rate control algorithm is developed to optimize the transmission path and rate, reduce system energy consumption, and achieve the goal of green energy saving while ensuring the quality of service. Combined with satellite orbit information and network usage, transmission resources are intelligently allocated to avoid unnecessary energy waste. A spectrum resource optimization allocation method is proposed to maximize the use of spectrum resources of the LEO satellite network and improve data transmission efficiency. It can optimize the data transmission path, reduce network congestion, and improve bandwidth utilization through multi-path transmission and traffic load balancing technology. The process of multi-user watching video on demand is modeled as a partially observable Markov process (POMDP). A cooperative alliance game framework is designed to describe the interaction properties between multiple users. A dynamic iteratively improved deep deterministic policy gradient multi-agent scheme is designed to adapt to heterogeneous client ABR algorithms. A unified fixed-dimensional algorithm input based on global attributes is designed to adapt to the dynamically changing number of users, thereby ensuring the consistency of distributed reinforcement learning training. A video-on-demand adaptive rate control framework Harmony based on LEO satellite Internet is implemented.
[0097] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.
[0098] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.
[0099] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.
[0100] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.
[0101] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.
[0102] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0103] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a video transmission rate control device in a three-dimensional heterogeneous network.
[0104] refer to Figure 5 The video transmission rate control device 800 in the three-dimensional heterogeneous network includes:
[0105] The optimization target construction module 810 is configured to construct an optimization target for LEO satellite Internet on-demand video transmission rate control; the optimization target is a multi-dimensional global optimization target, including maximizing the overall user experience quality, minimizing energy consumption and maximizing the fairness of user experience quality;
[0106] A distributed sub-optimization target construction module 820 is configured to decompose the optimization target into distributed sub-optimization targets;
[0107] The information acquisition module 830 is configured to acquire the state information and cooperation information of all agents; wherein the state information of each agent includes the channel coefficient, other target channel interference and corresponding user experience quality information; the cooperation information includes the average user experience quality of all users except itself at the current moment, the maximum user experience quality difference, and the total power consumption, average power consumption and maximum power consumption of other agents;
[0108] The rate decision set acquisition module 840 is used to input the state information and cooperation information of the agent into the DDPG model of multiple agent interactions to obtain the rate decision set made by the satellite for each user.
[0109] In some embodiments, the optimization goal is to maximize the utility U of the jth video block at each time instant. j ;in, α 1 is a hyperparameter; α 2 is a hyperparameter; α 3 is a hyperparameter; is the rate decision for the actual state at time t, QoE i,j is the user experience quality of the j-th video block of user i; Among them, μ 1 is a constant; μ 2 is a constant; μ 3 is a constant; is the video quality of the jth video block of user i; The impact of rebuffering events on user quality of experience; The effect of changes in video quality during playback; P i,j The energy consumed to download video chunk j for user i; QoE i′,j is the user experience quality of the j-th video chunk of user i'.
[0110] In some embodiments, the distributed sub-optimization objectives include Among them, U i,j is the utility of the jth video block of user i; buffer i,j is the buffer length of user i at the jth video block; i,j-1 is the buffer length of user i at the j-1th video block.
[0111] In some embodiments, the video transmission rate control device in the stereoscopic heterogeneous network further includes a training module configured to:
[0112] Constructing a rate decision process model to determine the input, joint strategy and output of the DDPG model of the multiple agents interacting;
[0113] Determine the cumulative value function, action value function, parameter update method of the policy network during training, parameter update method of the evaluation network during training, and output of the target network of the DDPG model of the multiple agents interacting;
[0114] Identify the global network;
[0115] In response to determining that the agent has completed training of the training video clip, centralized updating and distributed propagation are performed.
[0116] In some embodiments, the rate decision process model includes:
[0117] Environment and state, including S≡{S Q ,S P ,S r ,S B ,S h}, where S is the set of all state information of all agents in the satellite network during the rate decision process; S Q is the user experience quality value of n clients watching the video on demand at a given time, S Q ≡{Q 1 ,…,Q n}; S P is the power value allocated by the satellite to n users, S P ≡{P 1 ,…,P n}; S r is the bit rate level request sent by n clients; SB is the buffer status notification; and S h is the channel state parameter of n clients; the environment has a real state s, s∈S;
[0118] Actions, including the actions u of each agent a ;u a ∈U, U is the rate decision set r made by the satellite for n clients; at each time step j, the decision action of each agent will refer to the actions of other agents to form a joint action u of multiple agents;
[0119] The transition probability, including the state transition probability function P, satisfies P(s'│s,u):S×U×S→[0,1]; s is the current state, s' is the next state;
[0120] Rewards, including a global reward R shared by all agents, satisfying R(s,u):S×U→R; the global reward includes the average user quality experience value, average energy consumption and user quality experience fairness considerations of all users at a given time; the global reward includes the optimization target;
[0121] Observation, including the observation value z of each agent at each time step, z∈Z, where Z is the set of all agent observation values; the action observation history τ of each agent a ∈T≡(Z×U)*; the personal reward of each agent includes the distributed sub-optimization objective.
[0122] In some embodiments, the cumulative value function includes: in, is the cumulative value function of agent a under strategy π, z a is the observation value of each agent a, j is the index of the video block, γ j is the discount factor of video block j; E is the set of all agent observations, τ a is the initial state; are the parameters of the policy network.
[0123] In some embodiments, the action-value function includes: in, is the action value function of agent a under strategy π, is the observation value of agent a on video block j, is the action of agent a on video block j; r a is the rate decision made by the satellite for the client corresponding to agent a.
[0124] In some embodiments, the training module is configured to update the policy network using gradient descent. Parameters in, Strategy Network Parameters; is the parameter of the policy network corresponding to the policy π.
[0125] In some embodiments, the training module is configured to: update the evaluation network using a gradient descent based on a temporal difference algorithm Parameters
[0126] The outputs of the target network include: Among them, y j For the target network Output: is the observation value of agent a on video block j, is the action of agent a on video block j; is the observation value of agent a on video block j+1, is the action of agent a on video block j+1.
[0127] In some embodiments, the centralized updating includes: updating the global network according to the training video segment parameters; the updating includes: Θ g′ ←φ(ep)·Θ a +(1-φ(ep))·Θ g ; Among them, Θ g' To update the global network parameter set; Θ g is the global network parameter set; ep is the currently executing episode; φ(ep) is the effect of the episode parameter; k is the hyperparameter that determines the shape of the φ(·) function; ep 0 is a hyperparameter that determines the center location.
[0128] In some embodiments, the distributed propagation includes: Θ a′ =η(j,ep)·Θ g′ +(1-η(j,ep))·Θ a ; where Θ a′ is the updated network parameter set of agent a; Θ a is the network parameter set of agent a; Where η(j,ep) is the combined effect of step and episode parameters; Clip function is a truncation function with a value limit of 0 to 1; L is the upper limit of the lower limit Clip function variable, and H is the lower limit of the upper limit Clip function variable.
[0129] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.
[0130] The device of the above embodiment is used to implement the corresponding video transmission rate control method under the stereoscopic heterogeneous network in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.
[0131] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the video transmission rate control method under the three-dimensional heterogeneous network described in any of the above embodiments is implemented.
[0132] Figure 6 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.
[0133] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.
[0134] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.
[0135] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.
[0136] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).
[0137] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).
[0138] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.
[0139] The electronic device of the above embodiment is used to implement the corresponding video transmission rate control method under the stereoscopic heterogeneous network in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.
[0140] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the video transmission rate control method under the stereoscopic heterogeneous network as described in any of the above embodiments.
[0141] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.
[0142] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the video transmission rate control method under the three-dimensional heterogeneous network as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.
[0143] Based on the same inventive concept, corresponding to the video transmission rate control method under the stereoscopic heterogeneous network described in any of the above embodiments, the present disclosure also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer so that the computer and / or the processor executes the video transmission rate control method under the stereoscopic heterogeneous network. Corresponding to the execution subject corresponding to each step in each embodiment of the video transmission rate control method under the stereoscopic heterogeneous network, the processor that executes the corresponding step may belong to the corresponding execution subject.
[0144] The computer program product of the above embodiment is used to enable the computer and / or the processor to execute the video transmission rate control method under the stereoscopic heterogeneous network as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.
[0145] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.
[0146] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.
[0147] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.
[0148] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.
Claims
1. A method for controlling video transmission rate in a three-dimensional heterogeneous network, characterized in that: include: Constructing the optimization target of LEO satellite Internet on-demand video transmission rate control; The optimization goal is a multi-dimensional global optimization goal, including maximizing the overall user experience quality, minimizing energy consumption, and maximizing the fairness of user experience quality; Decomposing the optimization objective into distributed sub-optimization objectives; Obtaining the status information and cooperation information of all intelligent agents; wherein the status information of each intelligent agent includes the channel coefficient, other target channel interference and corresponding user experience quality information; the cooperation information includes the average user experience quality of all users except itself at the current moment, the maximum user experience quality difference, and the total power consumption, average power consumption and maximum power consumption of other intelligent agents; The state information and cooperation information of the agent are input into the DDPG model of multiple agent interactions to obtain the rate decision set made by the satellite for each user.
2. The video transmission rate control method in a three-dimensional heterogeneous network according to claim 1 is characterized in that: The optimization goal is to maximize the utility U of the jth video block at each moment j ;in, α1 is a hyperparameter; α2 is a hyperparameter; α3 is a hyperparameter; is the rate decision for the actual state at time t, QoE i,j is the user experience quality of the j-th video block of user i; Among them, μ1 is a constant; μ2 is a constant; μ3 is a constant; is the video quality of the jth video block of user i; The impact of rebuffering events on user quality of experience; The effect of changes in video quality during playback; P i,j The energy consumed to download video chunk j for user i; QoE i′,j is the user experience quality of the j-th video chunk of user i'.
3. The video transmission rate control method in a three-dimensional heterogeneous network according to claim 2 is characterized in that: The distributed sub-optimization objectives include Among them, U i,j is the utility of the jth video block of user i; buffer i,j is the buffer length of user i at the jth video block; i,j-1 is the buffer length of user i at the j-1th video block.
4. The video transmission rate control method in a three-dimensional heterogeneous network according to claim 1 is characterized in that: The method also includes determining a DDPG model of the interaction of multiple agents by: Constructing a rate decision process model to determine the input, joint strategy and output of the DDPG model of the multiple agents interacting; Determine the cumulative value function, action value function, parameter update method of the policy network during training, parameter update method of the evaluation network during training, and output of the target network of the DDPG model of the multiple agents interacting; Identify the global network; In response to determining that the agent has completed training of the training video clip, centralized updating and distributed propagation are performed.
5. The video transmission rate control method in a three-dimensional heterogeneous network according to claim 4 is characterized in that: The rate decision process model includes: Environment and state, including S≡{S Q ,S P ,S r ,S B ,S h }, where S is the set of all state information of all agents in the satellite network during the rate decision process; S Q is the user experience quality value of n clients watching the video on demand at a given time, S Q ≡{Q1,…,Q n }; S P is the power value allocated by the satellite to n users, S P ≡{P1,…,P n }; S r is the bit rate level request sent by n clients; SB is the buffer status notification; and S h is the channel state parameter of n clients; the environment has a real state s, s∈S; Actions, including the actions u of each agent a ;u a ∈U, U is the rate decision set r made by the satellite for n clients; at each time step j, the decision action of each agent will refer to the actions of other agents to form a joint action u of multiple agents; The transition probability, including the state transition probability function P, satisfies P(s'│s,u):S×U×S→[0,1]; s is the current state, s' is the next state; Rewards, including a global reward R shared by all agents, satisfying R(s,u):S×U→R; the global reward includes the average user quality experience value, average energy consumption and user quality experience fairness considerations of all users at a given time; the global reward includes the optimization target; Observation, including the observation value z of each agent at each time step, z∈Z, where Z is the set of all agent observation values; the action observation history τ of each agent a ∈T≡(Z×U)*; the personal reward of each agent includes the distributed sub-optimization objective.
6. The method for controlling video transmission rate in a three-dimensional heterogeneous network according to claim 4, characterized in that: The cumulative value function includes: in, is the cumulative value function of agent a under strategy π, z a is the observation value of each agent a, j is the index of the video block, γ j is the discount factor of video block j; E is the set of all agent observations, τ a is the initial state; are the parameters of the policy network; The action value function includes: in, is the action value function of agent a under strategy π, is the observation value of agent a on video block j, is the action of agent a on video block j; r a is the rate decision made by the satellite for the client corresponding to agent a; The method for updating the parameters of the policy network during the training process includes: updating the policy network using gradient descent Parameters in, Strategy Network Parameters; is the parameter of the policy network corresponding to the policy π; The parameter updating method of the evaluation network during the training process includes: updating the evaluation network using a gradient descent based on a time difference algorithm Parameters The outputs of the target network include: Among them, y j For the target network Output: is the observation value of agent a on video block j, is the action of agent a on video block j; is the observation value of agent a on video block j+1, is the action of agent a on video block j+1.
7. The video transmission rate control method in a three-dimensional heterogeneous network according to claim 4 is characterized in that: The centralized update includes: the global network is updated according to the training video segment parameters; the update includes: Θ g′ ←φ(ep)·Θ a +(1-φ(ep))·Θ g ; Among them, Θ g' To update the global network parameter set; Θ g is the global network parameter set; ep is the currently executing episode; φ(ep) is the influence of the episode parameter; k is the hyperparameter that determines the shape of the φ(·) function; ep0 is the hyperparameter that determines the center position; The distributed propagation includes: Θ a′ =η(j,ep)·Θ g′ +(1-η(j,ep))·Θ a ; where Θ a′ is the updated network parameter set of agent a; Θ a is the network parameter set of agent a; Where η(j,ep) is the combined effect of step and episode parameters; Clip function is a truncation function with a value limit of 0 to 1; L is the upper limit of the lower limit Clip function variable, and H is the lower limit of the upper limit Clip function variable.
8. A video transmission rate control device in a three-dimensional heterogeneous network, characterized in that: include: An optimization target building module is configured to build an optimization target for LEO satellite Internet on-demand video transmission rate control; The optimization goal is a multi-dimensional global optimization goal, including maximizing the overall user experience quality, minimizing energy consumption, and maximizing the fairness of user experience quality; A distributed sub-optimization target building module, configured to decompose the optimization target into distributed sub-optimization targets; An information acquisition module is configured to acquire the state information and cooperation information of all agents; wherein the state information of each agent includes the channel coefficient, other target channel interference and corresponding user experience quality information; the cooperation information includes the average user experience quality of all users except itself at the current moment, the maximum user experience quality difference, and the total power consumption, average power consumption and maximum power consumption of other agents; The rate decision set acquisition module is used to input the state information and cooperation information of the agent into the DDPG model of multiple agent interactions to obtain the rate decision set made by the satellite for each user.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program. 10 . A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to claim 1 .
Citation Information
Patent Citations
User perception based cross-layer optimization method for wireless video business
CN102638730A
VR content caching method based on mobile edge computing network
CN113141634A
Multi-beam satellite resource allocation method based on decision performance evaluation
CN114389678A
Centralized and active collaborative low earth orbit satellite system switching method
CN115987373A
Joint optimization method for inter-satellite link planning and power distribution between low-orbit satellite planes
CN117692051A
