Video transmission rate control method and related equipment in three-dimensional heterogeneous networks

Abstract

This application provides a method and related equipment for controlling video transmission rate in a three-dimensional heterogeneous network. The method includes: constructing an optimization objective for controlling the on-demand video transmission rate of LEO satellite internet; the optimization objective includes maximizing the overall user experience quality, minimizing energy consumption, and maximizing the fairness of user experience quality; decomposing the optimization objective into distributed sub-optimization objectives; obtaining the state information and cooperation information of all agents; wherein, the state information of each agent includes channel coefficients, interference from other target channels, and corresponding user experience quality information; the cooperation information includes the average user experience quality of all users other than itself at the current moment, the maximum user experience quality difference, and the total power consumption, average power consumption, and maximum power consumption of other agents; inputting the state information and cooperation information of the agents into a DDPG model of interaction among multiple agents to obtain the rate decision set made by the satellite for each user.

Description

Technical Field

[0001] This application relates to the fields of satellite internet and streaming media technology, and in particular to a video transmission rate control method and related equipment in a three-dimensional heterogeneous network. Background Technology

[0002] With the rapid development of internet technology and the increasing demands of users, video-on-demand (VoD) has become an important part of internet services. Traditional terrestrial networks cannot meet the coverage needs of remote and rural areas worldwide, prompting the exploration of new internet access methods. Low Earth Orbit (LEO) satellite internet, due to its wide coverage and low latency, has become one of the important solutions for future internet access. LEO satellite systems consist of multiple satellites orbiting the Earth at relatively low altitudes (typically between 500 and 2000 kilometers), resulting in low signal propagation latency, making them suitable for high-bandwidth, low-latency internet applications. However, LEO satellite systems also face many challenges, such as frequent satellite switching, multipath interference, and limited spectrum resources.

[0003] For video-on-demand services, stable bandwidth and low latency are required to ensure a high-quality video transmission experience. In a LEO satellite internet environment, rate control for video-on-demand services becomes particularly critical because network conditions can fluctuate due to factors such as satellite movement and weather changes. Therefore, a rate control scheme that can adapt to dynamic network changes is needed to ensure the quality of video-on-demand services. Summary of the Invention

[0004] In view of this, the purpose of this application is to propose a video transmission rate control method and related equipment in a three-dimensional heterogeneous network.

[0005] To achieve the above objectives, this application provides a video transmission rate control method for three-dimensional heterogeneous networks, comprising:

[0006] An optimization objective is established for controlling the transmission rate of LEO satellite internet video-on-demand. This optimization objective is a multi-dimensional global optimization objective, including maximizing the overall user experience quality, minimizing energy consumption, and maximizing the fairness of user experience quality.

[0007] The optimization objective is decomposed into distributed sub-optimization objectives;

[0008] Acquire the state information and cooperation information of all agents; wherein, the state information of each agent includes channel coefficients, interference from other target channels, and corresponding user experience quality information; the cooperation information includes the average user experience quality and maximum user experience quality difference of all users other than itself at the current moment, as well as the total power consumption, average power consumption, and maximum power consumption of other agents;

[0009] Input the state and cooperation information of the agents into the DDPG model of multiple agent interactions to obtain the rate decision set made by the satellite for each user.

[0010] In some embodiments, the optimization objective is to maximize the utility U of the j-th video block at each time step. j ;in, α1 is a hyperparameter; α2 is a hyperparameter; α3 is a hyperparameter; For the rate decision at time t based on the real state, QoE i,j The user experience quality for the j-th video block for user i; Where μ1 is a constant; μ2 is a constant; μ3 is a constant; Let the video quality of the j-th video block be the value of user i. To re-buffer the impact of events on user experience; The impact of changes in video quality during playback; P i,j Energy consumed in downloading video block j for user i; QoE i′,j Let i be the user experience quality of the j-th video block for user i'.

[0011] In some embodiments, the distributed sub-optimization objective includes Among them, U i,j The utility of the j-th video block for user i; buffer i,j Let buffer be the buffer length for user i in the j-th video block. i,j-1 Let be the buffer length for user i in the (j-1)th video block.

[0012] In some embodiments, the method further includes determining a DDPG model of interactions among multiple agents by:

[0013] A rate decision process model is constructed to determine the input, joint policy, and output of the DDPG model for the interaction of the multiple agents;

[0014] The cumulative value function, action value function, parameter update method of the policy network during training, parameter update method of the evaluation network during training, and output of the target network of the DDPG model for interaction of the multiple agents are determined.

[0015] Determine the global network;

[0016] In response to determining that the agent has completed training on the training video segments, centralized updates and distributed propagation are performed.

[0017] In some embodiments, the rate decision process model includes:

[0018] Environment and state, including S≡{S Q ,S P ,S r ,S B ,S h}, where S is the set of all state information of all agents in the satellite network during the rate decision-making process; S Q S represents the user experience quality value for n clients watching video-on-demand at a given time. Q ≡{Q1,…,Q n};S P S is the power value allocated by the satellite to n users. P ≡{P1,…,P n};S r Rate level requests sent to n clients; SB is the buffer status notification; and S... h Let be the channel state parameters for n clients; the environment has a real state s, s∈S;

[0019] Actions, including the actions of each agent. a ;u a ∈U, where U is the set of rate decisions r made by the satellite for n clients; at each time step j, each agent's decision action will refer to the actions of other agents to form a joint action u of multiple agents;

[0020] Transition probabilities, including the state transition probability function P, satisfy P(s'│s,u):S×U×S→[0,1]; s is the current state, s' is the next state;

[0021] The reward includes a global reward R shared by all agents, satisfying R(s,u): S×U→R; the global reward includes the average user quality experience value, average energy consumption, and user quality experience fairness considerations for all users at a given time; the global reward includes the optimization objective.

[0022] Observations include each time step, the observation value z of each agent, z∈Z, where Z is the set of all agent observations; and the action observation history τ of each agent. a ∈T≡(Z×U)*; Each agent's individual reward includes the distributed sub-optimization objective.

[0023] In some embodiments, the cumulative value function includes: in, Let z be the cumulative value function of agent a under policy π. a For each agent a, the observation value is given, j is the index of the video block, and γ is the value of the observation value. j Let τ be the discount factor for video block j; E be the set of all agent observations, and τ be the discount factor for video block j. a This is the initial state; These are the parameters of the policy network;

[0024] The action value function includes: in, Let be the action value function of agent a under policy π. Let be the observation value of agent a on video block j. For agent a, the action on video block j; r a The rate decision made by the satellite as the client corresponding to agent a;

[0025] The parameter update method for the policy network during training includes: updating the policy network using gradient descent. parameters in, For policy networks Parameters; These are the parameters of the policy network corresponding to policy π;

[0026] The parameter update method for the evaluation network during training includes: updating the evaluation network using gradient descent based on the time difference algorithm. parameters

[0027] The output of the target network includes: Among them, y j For target network The output; Let be the observation value of agent a on video block j. The action of agent a on video block j; Let be the observation value of agent a for video block j+1. Let's consider the action of agent a on video block j+1.

[0028] In some embodiments, the centralized update includes: the global network updating based on the training video segment parameters; the update includes: Θ g′ ←φ(ep)·Θ a +(1-φ(ep))·Θ g ; Where, Θ g' To update the global network parameter set; Θ g is the global network parameter set; ep is the currently executing segment; φ(ep) is the effect of the episode parameter; k is the hyperparameter that determines the shape of the φ(·) function; ep0 is the hyperparameter that determines the center position;

[0029] The distributed propagation includes: Θ a′ =η(j,ep)·Θ g′ +(1-η(j,ep))·Θ a ; where Θ a′ For the updated network parameter set of agent a; Θ a For agent a, the network parameter set; Where η(j,ep) represents the combined effect of the step and episode parameters; the Clip function is a truncation function with a value limit of 0 to 1; L is the lower limit of the upper limit of the Clip function variable, and H is the upper limit of the lower limit of the Clip function variable.

[0030] This application also provides a video transmission rate control device for a three-dimensional heterogeneous network, including:

[0031] The optimization target construction module is configured to construct optimization targets for controlling the transmission rate of LEO satellite internet video on demand; the optimization targets are multi-dimensional global optimization targets, including maximizing the overall user experience quality, minimizing energy consumption, and maximizing the fairness of user experience quality.

[0032] A distributed sub-optimization objective construction module is configured to decompose the optimization objective into distributed sub-optimization objectives;

[0033] The information acquisition module is configured to acquire the state information and cooperation information of all intelligent agents; wherein, the state information of each intelligent agent includes channel coefficients, interference from other target channels, and corresponding user experience quality information; the cooperation information includes the average user experience quality and maximum user experience quality difference of all users other than itself at the current moment, as well as the total power consumption, average power consumption, and maximum power consumption of other intelligent agents;

[0034] The rate decision set acquisition module is used to input the state information and cooperation information of the agents into the DDPG model of multiple agents interacting, and obtain the rate decision set made by the satellite for each user.

[0035] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any of the preceding claims.

[0036] This application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing a computer to perform any of the methods described above.

[0037] This application also provides a computer program product, including computer program instructions that, when run on a computer, cause the computer to perform the method described in any of the preceding claims.

[0038] As can be seen from the above, the video transmission rate control method and related equipment under the three-dimensional heterogeneous network provided in this application embodiment can, to a certain extent, solve the problems of large fluctuations in video transmission quality, high energy consumption, and poor user fairness when existing rate control methods are applied to the last-hop on-demand video transmission scenario of LEO low-orbit satellites. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating the video transmission rate control method under a three-dimensional heterogeneous network according to an embodiment of this application.

[0041] Figure 2 This is a schematic diagram illustrating the execution logic of the LEO satellite internet video-on-demand framework and rate allocation algorithm in an embodiment of this application.

[0042] Figure 3 This is a flowchart illustrating the DDPG model for determining the interaction of multiple intelligent agents according to an embodiment of this application.

[0043] Figure 4 A schematic diagram illustrating the design details of the Harmony rate allocation algorithm in embodiments of this application;

[0044] Figure 5 This is a schematic diagram of a video transmission rate control device in a three-dimensional heterogeneous network according to an embodiment of this application.

[0045] Figure 6This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0047] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0048] Currently, most rate control schemes are designed for terrestrial fixed networks, such as congestion control algorithms in the TCP protocol. These schemes have certain limitations in handling the dynamic characteristics of LEO satellite internet. These limitations manifest in the following ways: 1) Poor adaptability: Traditional rate control algorithms cannot respond promptly to satellite handovers and link changes, leading to significant fluctuations in video transmission quality; 2) Low resource utilization: They cannot fully utilize the spectrum resources of LEO satellite networks, resulting in low bandwidth utilization; 3) High energy consumption: High energy consumption is a significant issue in satellite communication, and existing schemes lack effective mechanisms for energy consumption control; 4) Difficulty in ensuring user fairness: For all users accessing video applications within the last hop coverage area of ​​a satellite, fairness among viewers is crucial for the sustainable and long-term development of multi-user systems.

[0049] Against the backdrop of global advocacy for green energy conservation, how to reduce energy consumption while ensuring service quality and user fairness is an important direction for LEO satellite internet research. However, implementing this solution faces two challenges: 1) The uncontrollability and unpredictability of ABR (Adaptive Bitrate Streaming) algorithms: Typically, VoD involves clients using ABR algorithms to request bitrate levels and then sending them to the server. For the last-hop satellite mobile edge computing (MEC) server, it is difficult to determine the type of ABR algorithm used by different client applications, as these algorithms are usually designed internally by the content platform. Therefore, designing a rate control scheme that can effectively execute in environments with different ABR algorithms is a significant challenge. 2) User dynamism: The user base within the satellite coverage area is highly dynamic, with frequent user joining and leaving increasing the complexity of algorithm design. This is particularly challenging for NOMA systems, as each user can potentially affect the transmission rate of other users.

[0050] Therefore, existing rate control schemes suffer from problems such as large fluctuations in video transmission quality, high energy consumption, and poor user fairness when dealing with last-hop on-demand video transmission scenarios from LEO low-Earth orbit satellites.

[0051] Based on this, this application provides a last-hop on-demand video transmission scenario for LEO (Low Earth Orbit) satellites. From the perspectives of ensuring service quality, reducing energy consumption, and guaranteeing user fairness, an algorithm scheme called Harmony is designed and implemented. Harmony is a rate control scheme based on the Deep Deterministic Policy Gradient (DDPG) algorithm using reinforcement learning. Harmony improves the multi-agent DDPG into a distributed training framework, where multiple single-agent DDPGs run simultaneously. It features a dynamically iterative training architecture and a unified input dimension to adapt to different ABR (Advanced Background Reduction) algorithms and adapt to dynamically changing user numbers. Harmony achieves an excellent multi-objective rate control solution for on-demand video, satisfying the requirements of Quality of Experience (QoE), energy consumption, and user fairness. It can, to a certain extent, solve the problems of large fluctuations in video transmission quality, high energy consumption, and poor user fairness that exist when existing rate control methods are applied to the last-hop on-demand video transmission scenario for LEO satellites.

[0052] Reference Figure 1As shown, this application provides a video transmission rate control method in a three-dimensional heterogeneous network. The three-dimensional heterogeneous network can be a LEO satellite internet. Here, three-dimensional heterogeneity can be understood as a state of cooperation between satellites and the ground. The video transmission rate control method may include:

[0053] S100, construct the optimization objective for controlling the transmission rate of LEO satellite internet on-demand video; the optimization objective is a multi-dimensional global optimization objective, including maximizing the overall user experience quality, minimizing energy consumption, and maximizing the fairness of user experience quality;

[0054] Step S200: Decompose the optimization objective into distributed sub-optimization objectives;

[0055] Step S300: Obtain the state information and cooperation information of all intelligent agents; wherein, the state information of each intelligent agent includes channel coefficient, interference from other target channels, and corresponding user experience quality information; the cooperation information includes the average user experience quality and maximum user experience quality difference of all users other than itself at the current moment, as well as the total power consumption, average power consumption, and maximum power consumption of other intelligent agents;

[0056] Step S400: Input the state information and cooperation information of the agents into the DDPG model of multiple agent interactions to obtain the rate decision set made by the satellite for each user.

[0057] In step S100, such as Figure 2 As shown, the optimization objective comprehensively considers the user experience quality (QoE) of video block j for a single user i, energy consumption indicators, and the fairness of the user experience quality (QoE) among different users in the scenario.

[0058] In some embodiments, the bitrate control timing is set at the beginning of each video block transmission, for a single user i's QoE. i Typically, three factors are considered: the quality of the video chunk being viewed, playback stuttering, and bitrate switching. The Quality of User Experience (QoE) for each video chunk for a single user can include the quality of the video chunk j being viewed, the negative impact of rebuffering events on QoE, and the adverse effects of frequent changes in video quality. The Quality of User Experience (QoE) for user i's j-th video chunk is... i,j The quality of the video clips, playback stuttering, and bitrate switching can be weighted and summed.

[0059] In some of these embodiments, Where μ1 is a constant; μ2 is a constant; μ3 is a constant; Let be the video quality of the j-th video block for user i. Where, Setting v mThis represents the utility viewed at quality level m, which is typically proportional to the bit rate level. In common approaches, the bit rate level m can be directly used as this value. m The value of . For example, video can be encoded in six bitrate levels, including {640, 1600, 4000, 8400, 12000, 16400} kbps. That is, v m The value can be in the bit rate level m. To re-buffer the impact of events on user quality experience. Among them, S m This refers to the size of a chunk file viewed at quality level m, which can be understood as "size". Let B be the rate of user i at time t. i,j The state of the buffer for the j-th video block of user i. This relates to the impact of changes in video quality during playback. Among them, in, Let be the video quality of the (j-1)th video block for user i.

[0060] In some embodiments, the satellite allocates a transmission rate to each user based on energy consumption metrics, thereby affecting transmission power and the energy consumed during actual video transmission. Since satellites primarily rely on solar energy, their energy is relatively scarce. Let P be the energy consumed by user i downloading block j. i,j .

[0061] In some embodiments, regarding the fairness of the Quality of User Experience (QoE) among different users in a scenario, at any given time, the QoE of different users accessing satellite internet to watch videos within the same area should not show a significant difference; therefore, QoE fairness must be ensured. This can be represented as... Among them, U s,v It can be used to connect to a satellite in a scene and allow a group of users to watch video. QoE i′,j Let i be the user experience quality of the j-th video block for user i'.

[0062] In some embodiments, the optimization objective is to maximize overall QoE utility, minimize energy consumption, and achieve QoE fairness as much as possible through rate control. The utility of each j-th block can be expressed as in equation (1). Where α z z∈{1,2,3} are hyperparameters used to balance different indices. The overall optimization objective is to maximize U at each time step. j Among them, U j Let the utility of the j-th video block be denoted as . Let t be the set of users who access the satellite and watch the video in the scenario at time t. QoE i,j P represents the user experience quality of the j-th video block for user i. i,j Energy consumed in downloading video block j for user i; QoE i′,j Let i be the user experience quality of the j-th video block for user i'.

[0063] In step S200, the global centralized optimization problem is decomposed into distributed sub-optimization problems, which simplifies the optimization problem. This allows for a degree of solution to the global optimization problem in each time slot j (i.e., the time corresponding to the start of video block j transmission), which is difficult to solve using general mathematical methods. It also addresses the issue of each user having a decision variable, rate r, for video block j. j Therefore, rate decisions will influence users' decisions, as given by Shannon's formula, forming an NP-hard problem. Furthermore, the function that makes rate decisions is non-convex, making it difficult to use optimization algorithms. Additionally, the user experience quality (QoE) in the optimization objective does not have a clear mathematical expression and can only be obtained through interaction and feedback with the client application.

[0064] In some embodiments, the distributed sub-optimization objective can be as shown in equation (2); Among them, U i,j Let α1 be the utility of the j-th video block for user i; α2 be a hyperparameter; α3 be a hyperparameter; α4 be a hyperparameter; and buffer be a hyperparameter. i,j Let buffer be the buffer length for user i in the j-th video block. i,j-1 Let be the buffer length of user i in the (j-1)th video block. The first part, also known as part 1, is derived from the first two terms of equation (1) of the global objective (i.e., the optimization objective).

[0065] In some embodiments, the third term in equation (2), representing the maximum QoE difference between the current user and other users, can replace the maximum QoE difference term in the user set in equation (1) to achieve the same effect. Minimize the utility U of each sub-objective. i,j The QoE difference within the range can ensure the overall target utility U j Minimize the overall QoE difference within the subproblem. This subproblem exhibits transitivity. The mathematical derivation can be as follows: Therefore, during optimization, the |QoE of each user i is minimized. i -QoE i′The difference is a more rigorous optimization problem. Therefore, the optimality of the subproblem can be transformed into the optimality of the global problem.

[0066] In some embodiments, the buffer variation term in subproblem (2) can be used as an additional optimization objective in the subproblem. This is because a continuous and rapid increase in buffer length will lead to a decrease in overall utility. Increasing the buffer length for each user may not necessarily improve the user's QoE, but it will certainly reduce power efficiency. This logical relationship is implicit in the first three terms of formula (2). For algorithmic sensitivity, this logical relationship can be explicitly designed into the utility of the subproblem. Assume there exists a threshold buffer. thresh This makes when the buffer i,j buffer thresh At this time, no rebuffering event will occur, QoE remains unchanged, resulting in Consider the utility U of the buffer pair i,j Impact: Among them (buffer) i,j -buffer i,j-1 ) represents the buffer increment. Therefore, when the buffer... i,j buffer thresh At these conditions, the buffer can be considered sufficiently large, and further increments have minimal impact on utility. Therefore, the main impact on utility under these conditions is due to power consumption. Thus, simply increasing the buffer size by increasing the transmission rate will not improve QoE. On the contrary, it may reduce individual utility U due to increased transmission power. i,j Thus affecting the total utility U j It will have adverse effects.

[0067] In step S300, in some embodiments, in addition to the state information of each agent, the interaction between multiple agents can be represented by additional inputs, and the joint state of each agent is simplified to its own observation state. Afterwards, each agent can perform its operations in a distributed manner. This simplifies the multi-agent training process into a cooperative game involving multiple individual agents. It can solve the problems caused by the rapid increase in joint state s and joint action u with the increase of the number of clients n when using traditional centralized multi-agent reinforcement learning, leading to increased computational complexity, reduced learning efficiency, and reduced learning effectiveness. Furthermore, it can also solve the problem of training discontinuity caused by the fluctuation of the number of clients over time in dynamic environments.

[0068] In some embodiments, for each agent a, the observation can be updated to z. a′ ≡{QoE -a ,P-a ,z a}, where z a It contains information specific to agent a, including channel coefficients h. a Other target channel interference and the corresponding user QoE information {buffer a bitrate a Rebuffer_event a Bitrate_change a The superscript -a can be a joint variable on an agent other than the given agent a. Joint variables can include centralized updates and distributed propagation. That is, the state information of each agent includes channel coefficients, interference from other target channels, and corresponding user experience quality information.

[0069] In some embodiments, the difference between the average user experience quality and the maximum user experience quality of all users other than itself at the current moment may include... Where N(a) is the set of agents other than a, N a =|N(a)| represents the number of elements in set -a. k is the user index excluding itself. The average QoE of all users other than itself at the current moment. The highest quality of user experience excluding itself at the current moment; QoE k’ This represents the minimum user experience quality excluding itself at the current moment. This allows us to measure the user experience quality of other users.

[0070] In some embodiments, the total power consumption, average power consumption, and maximum power consumption of the other intelligent agents can be... Where, ∑ k∈N(a) P k This represents the total power consumption of other intelligent agents; This represents the average power consumption of other intelligent agents; This represents the maximum power consumption of other intelligent agents. This allows us to measure the power consumption of other users. Combined with the difference between the average and maximum user experience quality of all users other than ourselves at the current moment, we can measure the state of other users.

[0071] In some embodiments, N agents simultaneously provide on-demand transmission services to N users, forming a cooperative game alliance, with the joint strategy being... Players (i.e., users / agents) choose actions based on their own strategies, and these actions influence each other, as shown by the Shannon formula. The goal of cooperative game theory is to maximize the overall utility of the alliance, which is the optimization objective of equation (1). The optimal joint strategy under Nash equilibrium is expressed as... Under Nash equilibrium, no agent has an incentive to unilaterally change its strategy.

[0072] like Figure 3 As shown, the video transmission rate control method under a three-dimensional heterogeneous network in this application embodiment may further include determining the DDPG model of multiple agent interactions through the following method:

[0073] Step S510: Construct a rate decision process model to determine the input, joint policy, and output of the DDPG model for interaction among the multiple agents;

[0074] Step S520: Determine the cumulative value function, action value function, parameter update method of the policy network during training, parameter update method of the evaluation network during training, and output of the target network for the DDPG model of interaction among the multiple agents.

[0075] Step S530: Determine the global network;

[0076] Step S540: In response to determining that the agent has completed training of the training video segment, perform centralized updates and distributed propagation.

[0077] In some embodiments, in step S510, the rate decision process model can be a partially observed Markov problem (POMDP), representing a fully cooperative multi-agent task where the decisions made by the agents influence each other. This can describe the complex process in the rate control problem for VoD services in LEO satellite networks, where the state changes over time and depends on the joint decision result of the previous time step at any given moment, and can take into account the unobservability of the application state within each user.

[0078] In some embodiments, the random game is set as G, and defined as a tuple G. j =<S,U,P,R,Z,O,r,n,γ> Here, n agents are defined as a∈A≡{1,…,n}, choosing consecutive actions; γ∈[0,1) is the discount factor. It is worth noting that G... j The number of agents n changes dynamically as the start time (i.e., time slot j) of the video block transmission changes.

[0079] In some embodiments, the environment and state may include S≡{S Q ,S P ,S r ,S B ,S h}, where S is the set of all state information of all agents in the satellite network during the rate decision-making process. QS represents the Quality of User Experience (QoE) value for n clients watching video-on-demand at a given time. Q ≡{Q1,…,Q n}. S P S is the power value allocated by the satellite to n users. P ≡{P1,…,P n};S r Bitrate level requests sent for n clients; S B For buffer state notification; and S h Let be the channel state parameters for n clients; the environment has a real state s, s∈S.

[0080] In some embodiments, actions may include actions u for each agent. a Action u a It is a continuous value. a ∈U, where U is the set of rate decisions r made by the satellite MEC for n clients; at each time step j, each agent's decision action will reference the actions of other agents, forming a joint action u of multiple agents. u∈U≡U n .

[0081] In some embodiments, the transition probability may include a state transition probability function P, satisfying P(s'│s,u): S×U×S→[0,1]. Here, s is the current state, and s' is the next state. S is the set of all state information of all agents in the satellite network during the rate decision-making process. U is the set of rate decisions r made by the satellite MEC for n clients.

[0082] In some embodiments, the reward may include a global reward R shared by all agents, satisfying R(s,u): S×U→R. The global reward is given by the aforementioned equation (1), including the average quality of user experience (QoE) value, average energy consumption, and QoE fairness considerations for all users at a given time. That is, the global reward includes the optimization objective.

[0083] In some embodiments, observation may include partial observation settings, including observation values ​​z, z ∈ Z, for each time step, and Z being the set of all agent observation values. The action observation history τ for each agent... a ∈T≡(Z×U)*. U is the set of rate decisions r made by the satellite MEC for n clients. The trajectory typically depends on the stochastic policy π. a (u a ,τ a ):T×U→[0,1]. Where, π a (u a ,τ a For each agent a, a random policy, ua For each agent's action, T represents the duration of the observation history of each agent's action; U represents the rate decision set r made by the satellite MEC for n clients. Typically, the individual reward for each agent can be given by equation (2), which may include the distributed sub-optimization objective.

[0084] In some embodiments, step S520 involves designing an iterative process for the single-agent DDPG to control the last hop rate of LEO satellites, i.e., determining the iterative process for the DDPG model involving multiple agents interacting to control the last hop rate of LEO satellites. For agent a, the policy network and the commentator network can be defined as follows: and The target policy network and the target critic network are represented as follows: and The parameter set for each agent Θ a Defined as the parameter set of four networks.

[0085] In some embodiments, the initial state of agent a is set to τ. a The cumulative value function under strategy π can be: Let z be the cumulative value function of agent a under policy π. a For each agent a, the observation value is given, j is the index of the video block, and γ is the value of the observation value. j Let r be the discount factor for video block j; E be the set of all agent observations; ... set of all agent observations; r be the set of all agent observations; r be the set of all agent observations; r be the set of all agent observations; r be the set of all agent observations; r be the set of all agent observations; r be the set of all agent observations; r be the set of all agent observations; r be the set of all agent observations; r be the set j a Let j be the rate of video block j. These are the parameters of the policy network.

[0086] In some embodiments, the action value function, according to the Bellman equation, includes: in, Let be the action value function of agent a under policy π. Let be the observation value of agent a on video block j. The action of agent a on video block j; Let r be the observation value of agent a on video block j, the action of agent a on video block j, and the action value function under policy π; a The rate decision is made by the satellite as the client corresponding to agent a; γ is the discount factor, E is the expectation operation, and z j+1 For the observation value of the (j+1)th video block, Ez j+1 Let be the expected value of the (j+1)th video block. Simultaneously, the cumulative value function... It can also be written as

[0087] In some embodiments, the method for updating the parameters of the policy network during training includes: updating the policy network using gradient descent. parameters

[0088] in, For policy networks The change value of the parameter; These are the parameters of the policy network corresponding to policy π; Let be the observed values ​​and the expected value under the cumulative value function for agent a. For gradient operators, This is the derivation of gradient descent.

[0089] In some embodiments, the method for updating the parameters of the evaluation network during training may include: updating the evaluation network using gradient descent based on a time difference algorithm. parameters The loss function is Among them, y j For target network The output of the target network includes: Among them, y j For target network The output of r; a The rate decision made by the satellite as the client corresponding to agent a; Let be the observation value of agent a on video block j. Let a be the action of agent a on video block j. Let be the action value function of agent a under the policy network parameters and policy π. Let be the action value function of agent a under the updated policy network parameters and policy π.

[0090] In some embodiments, in step S540, the Harmony framework is as follows: Figure 4 As shown. In real-world satellite service scenarios, users randomly joining or leaving the media service causes the number of agents in the satellite MEC to change over time. This can be understood as a single satellite serving multiple users, with one agent per user; the number of agents changes when a user joins or leaves. Typically, all agents are maintained as a thread or coroutine within the satellite MEC. Since the cooperative information in the input is in the form of mean and variance that does not change with the number of users, the problem of agent number variation at the input dimension level is solved.

[0091] In some embodiments, a distributed dynamic framework can be used to address the model update problem during training, focusing on the timing of updates. Each agent has a complete DDPG structure, maintaining a separate agent for each user in the video application. A user watching a video segment is defined as a training segment, and watching a video block is defined as a step. The index j of the video block can be a step parameter, describing the interaction between a single agent and the environment up to step j; ep is a segment parameter, representing the segment currently being executed. Therefore, a global network is maintained, and at a certain moment, the network parameter set of agent a is represented as... The global network parameter set is represented as Θ g When agent a completes the training of an entire segment, it performs two steps: centralized update and distributed propagation.

[0092] In practical applications, the video dataset can contain 100, 125, 150, 200, and 250 video blocks, with each block having a uniform length of 4 seconds. For the training process, the classic ABR algorithm BOLA can be implemented in the environment.

[0093] In some embodiments, the centralized update may include: updating the global network based on fragment parameters; the update includes: Θ g′ ←φ(ep)·Θ a +(1-φ(ep))·Θ g ; Where, Θ g' To update the global network parameter set after training the video segments; Θ g Let be the global network parameter set before training the video segments is complete; ep is the segment parameter, representing the segment currently being executed; φ(ep) represents the influence of the episode parameters; k is the hyperparameter determining the shape of the φ(·) function; and ep0 is the hyperparameter determining the center position. Thus, under this strategy, as the number of episodes increases, Θ... a For Θ g The impact decreases. That is, the later an agent is added, the smaller its impact on the global network.

[0094] In some embodiments, the distributed propagation may include: Θ a′ =η(j,ep)·Θ g′ +(1-η(j,ep))·Θ a ; Where η(j,ep) represents the combined effect of the step and episode parameters; the Clip function is a cutoff function with values ​​limited to 0-1; L is the lower bound of the upper bound of the Clip function variable, and H is the upper bound of the lower bound of the Clip function variable. Θ a′To update the network parameter set of agent a after training on the training video segments; Θ a Let be the network parameter set of agent 'a' before training on the video segments is complete. Thus, under this strategy, a larger step size results in fewer new global network synchronous updates received by a single network, but this is also affected by the episode parameter. When a new user joins, the current global network parameter set... This information will be directly propagated and synchronized to new agents, allowing them to continue training and execution. In other words, agents that join later are more affected by the global network, while agents that join earlier are less affected.

[0095] In some embodiments, in step S400, the state information and cooperation information of all the individual agents obtained above are input into the DDPG model of multiple agents interacting, so that the rate decision set made by the satellite for each user can be obtained.

[0096] The Harmony video transmission rate control scheme for the last hop of LEO (Low Orbit) satellite internet provided in this application proposes a mechanism for real-time monitoring of the LEO satellite internet network status, dynamically adjusting the video transmission rate to adapt to dynamic network changes and ensure user experience stability. A reinforcement learning-based prediction algorithm is proposed to predict future network conditions and adjust the transmission rate in advance by analyzing historical data and real-time network status. An energy-aware rate control algorithm is developed to optimize transmission paths and rates while ensuring service quality, reducing system energy consumption and achieving green energy saving goals. Transmission resources are intelligently allocated based on satellite orbit information and network usage to avoid unnecessary energy waste. A spectrum resource optimization allocation method is proposed to maximize the utilization of LEO satellite network spectrum resources and improve data transmission efficiency. Multi-path transmission and traffic load balancing technologies can optimize data transmission paths, reduce network congestion, and improve bandwidth utilization. The process of multiple users watching video-on-demand is modeled as a partially observable Markov process (POMDP). A cooperative alliance game framework is designed to describe the interaction attributes between multiple users. A dynamically iteratively improved deep deterministic policy gradient multi-agent scheme is designed to adapt to the ABR algorithm for heterogeneous clients. A unified, fixed-dimensional algorithm input based on global attributes is designed to adapt to dynamically changing user numbers, thereby ensuring the coherence of distributed reinforcement learning training. A video-on-demand adaptive rate control framework, Harmony, based on the LEO satellite internet, is implemented.

[0097] It is understood that before using the technical solutions of the various embodiments in this disclosure, users will be informed of the type, scope of use, and usage scenarios of the personal information involved in an appropriate manner, and user authorization will be obtained.

[0098] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose, based on the prompt message, whether to provide personal information to the software or hardware such as electronic devices, applications, servers, or storage media performing the operations of this disclosed technical solution.

[0099] As an optional but not limited implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0100] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0101] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0102] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0103] Based on the same inventive concept, and corresponding to any of the above embodiments, this application also provides a video transmission rate control device for a three-dimensional heterogeneous network.

[0104] refer to Figure 5 The video transmission rate control device 800 under the three-dimensional heterogeneous network includes:

[0105] The optimization target construction module 810 is configured to construct an optimization target for controlling the transmission rate of LEO satellite internet video on demand; the optimization target is a multi-dimensional global optimization target, including maximizing the overall user experience quality, minimizing energy consumption, and maximizing the fairness of user experience quality.

[0106] The distributed sub-optimization objective construction module 820 is configured to decompose the optimization objective into distributed sub-optimization objectives;

[0107] The information acquisition module 830 is configured to acquire the state information and cooperation information of all intelligent agents; wherein, the state information of each intelligent agent includes channel coefficients, interference from other target channels, and corresponding user experience quality information; the cooperation information includes the average user experience quality and maximum user experience quality difference of all users other than itself at the current moment, as well as the total power consumption, average power consumption, and maximum power consumption of other intelligent agents;

[0108] The rate decision set acquisition module 840 is used to input the state information and cooperation information of the agents into the DDPG model of multiple agents interacting, so as to obtain the rate decision set made by the satellite for each user.

[0109] In some embodiments, the optimization objective is to maximize the utility U of the j-th video block at each time step. j ;in, α1 is a hyperparameter; α2 is a hyperparameter; α3 is a hyperparameter; For the rate decision at time t based on the real state, QoE i,j The user experience quality for the j-th video block for user i; Where μ1 is a constant; μ2 is a constant; μ3 is a constant; Let the video quality of the j-th video block be the value of user i. To re-buffer the impact of events on user experience; The impact of changes in video quality during playback; P i,j Energy consumed in downloading video block j for user i; QoE i′,j Let i be the user experience quality of the j-th video block for user i'.

[0110] In some embodiments, the distributed sub-optimization objective includes Among them, U i,j The utility of the j-th video block for user i; buffer i,j Let buffer be the buffer length for user i in the j-th video block. i,j-1Let be the buffer length for user i in the (j-1)th video block.

[0111] In some embodiments, the video transmission rate control device under the three-dimensional heterogeneous network further includes a training module, configured as follows:

[0112] A rate decision process model is constructed to determine the input, joint policy, and output of the DDPG model for the interaction of the multiple agents;

[0113] The cumulative value function, action value function, parameter update method of the policy network during training, parameter update method of the evaluation network during training, and output of the target network of the DDPG model for interaction of the multiple agents are determined.

[0114] Determine the global network;

[0115] In response to determining that the agent has completed training on the training video segments, centralized updates and distributed propagation are performed.

[0116] In some embodiments, the rate decision process model includes:

[0117] Environment and state, including S≡{S Q ,S P ,S r ,S B ,S h}, where S is the set of all state information of all agents in the satellite network during the rate decision-making process; S Q S represents the user experience quality value for n clients watching video-on-demand at a given time. Q ≡{Q1,…,Q n};S P S is the power value allocated by the satellite to n users. P ≡{P1,…,P n};S r Rate level requests sent to n clients; SB is the buffer status notification; and S... h Let be the channel state parameters for n clients; the environment has a real state s, s∈S;

[0118] Actions, including the actions of each agent. a ;u a ∈U, where U is the set of rate decisions r made by the satellite for n clients; at each time step j, each agent's decision action will refer to the actions of other agents to form a joint action u of multiple agents;

[0119] Transition probabilities, including the state transition probability function P, satisfy P(s'│s,u):S×U×S→[0,1]; s is the current state, s' is the next state;

[0120] The reward includes a global reward R shared by all agents, satisfying R(s,u): S×U→R; the global reward includes the average user quality experience value, average energy consumption, and user quality experience fairness considerations for all users at a given time; the global reward includes the optimization objective.

[0121] Observations include each time step, the observation value z of each agent, z∈Z, where Z is the set of all agent observations; and the action observation history τ of each agent. a ∈T≡(Z×U)*; Each agent's individual reward includes the distributed sub-optimization objective.

[0122] In some embodiments, the cumulative value function includes: in, Let z be the cumulative value function of agent a under policy π. a For each agent a, the observation value is given, j is the index of the video block, and γ is the value of the observation value. j Let τ be the discount factor for video block j; E be the set of all agent observations, and τ be the discount factor for video block j. a This is the initial state; These are the parameters of the policy network.

[0123] In some embodiments, the action value function includes: in, Let be the action value function of agent a under policy π. Let be the observation value of agent a on video block j. For agent a, the action on video block j; r a The rate decision made by the satellite as the client corresponding to agent a.

[0124] In some embodiments, the training module is configured to update the policy network using gradient descent. parameters in, For policy networks Parameters; These are the parameters of the policy network corresponding to policy π.

[0125] In some embodiments, the training module is configured to update the evaluation network using gradient descent based on a time-difference algorithm. parameters

[0126] The output of the target network includes: Among them, y j For target network The output; Let be the observation value of agent a on video block j. The action of agent a on video block j; Let be the observation value of agent a for video block j+1. Let's consider the action of agent a on video block j+1.

[0127] In some embodiments, the centralized update includes: the global network updating based on the training video segment parameters; the update includes: Θ g′ ←φ(ep)·Θ a +(1-φ(ep))·Θ g ; Where, Θ g' To update the global network parameter set; Θ g is the global network parameter set; ep is the currently executing segment; φ(ep) is the effect of the episode parameter; k is the hyperparameter that determines the shape of the φ(·) function; ep0 is the hyperparameter that determines the center position.

[0128] In some embodiments, the distributed propagation includes: Θ a′ =η(j,ep)·Θ g′ +(1-η(j,ep))·Θ a ; where Θ a′ For the updated network parameter set of agent a; Θ a For agent a, the network parameter set; Where η(j,ep) represents the combined effect of the step and episode parameters; the Clip function is a truncation function with a value limit of 0 to 1; L is the lower limit of the upper limit of the Clip function variable, and H is the upper limit of the lower limit of the Clip function variable.

[0129] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0130] The apparatus described above is used to implement the video transmission rate control method under the corresponding three-dimensional heterogeneous network in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0131] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the video transmission rate control method under the three-dimensional heterogeneous network described in any of the above embodiments.

[0132] Figure 6 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0133] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0134] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0135] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0136] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0137] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0138] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0139] The electronic devices described above are used to implement the video transmission rate control method under the corresponding three-dimensional heterogeneous network in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0140] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to execute the video transmission rate control method under a three-dimensional heterogeneous network as described in any of the above embodiments.

[0141] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0142] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the video transmission rate control method under the three-dimensional heterogeneous network as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0143] Based on the same inventive concept, corresponding to the video transmission rate control method under a three-dimensional heterogeneous network described in any of the above embodiments, this disclosure also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer to cause the computer and / or the processor to execute the video transmission rate control method under a three-dimensional heterogeneous network. Corresponding to the execution entity for each step in each embodiment of the video transmission rate control method under a three-dimensional heterogeneous network, the processor executing the corresponding step can belong to the corresponding execution entity.

[0144] The computer program product of the above embodiments is used to enable the computer and / or the processor to execute the video transmission rate control method under the three-dimensional heterogeneous network as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0145] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0146] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0147] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0148] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A method for controlling video transmission rate in a three-dimensional heterogeneous network, characterized in that, include: To construct an optimization objective for controlling the transmission rate of LEO satellite internet video on demand; The optimization objective is a multi-dimensional global optimization objective, including maximizing the overall user experience quality, minimizing energy consumption, and maximizing the fairness of user experience quality. The optimization objective is decomposed into distributed sub-optimization objectives; Acquire the state information and cooperation information of all agents; wherein, the state information of each agent includes channel coefficients, interference from other target channels, and corresponding user experience quality information; the cooperation information includes the average user experience quality and maximum user experience quality difference of all users other than itself at the current moment, as well as the total power consumption, average power consumption, and maximum power consumption of other agents; Input the state and cooperation information of the agents into the DDPG model of multiple agent interactions to obtain the rate decision set made by the satellite for each user; The method also includes determining a DDPG model for interactions among multiple agents by means of the following method: A rate decision process model is constructed to determine the input, joint policy, and output of the DDPG model for the interaction of the multiple agents; The cumulative value function, action value function, parameter update method of the policy network during training, parameter update method of the evaluation network during training, and output of the target network of the DDPG model for interaction of the multiple agents are determined. Determine the global network; In response to determining that the agent has completed training on the training video segments, centralized updates and distributed propagation are performed; The rate decision process model includes: Environment and conditions, including ,in, S It is the set of all state information of all agents in the satellite network during the rate decision-making process; S Q To watch video on demand at a given time n User experience quality score for each client. ; S P For satellite n Power values ​​allocated to each user. ; S r for n The bitrate level requests sent by each client; S B For buffer state notification; and S h for n Channel state parameters for each client; the environment has a real state. s,s∈S ; Actions, including the actions of each agent. u a ; u a ∈U , U For satellite n A set of rate decisions made by each client r At each time step, each agent's decision-making action will reference the actions of other agents, forming a joint action among multiple agents. u ; Transition probabilities, including the state transition probability function P， satisfy P(s'│s,u):S×U×S→[0,1]; s This is the current state. s' The next state; Rewards, including global rewards shared by all agents. R ,satisfy R(s,u):S×U→R The global reward includes the average user quality experience value, average energy consumption, and user quality experience fairness considerations for all users at a given time; the global reward includes the optimization objective. Observations, including observations for each agent at each time step. z, z∈Z, Z The set of observations from all agents; the action observation history of each agent. ; T The duration of the action observation history for each agent; the individual reward for each agent includes the distributed sub-optimization objective; for Z×U The Kling closure is the set of all finite-length sequences consisting of the elements in the set.

2. The video transmission rate control method under a three-dimensional heterogeneous network according to claim 1, characterized in that, The optimization objective is to maximize the first [value] at each time step. j The utility of a video block ;in, ; α 1 is a hyperparameter; α 2 is a hyperparameter; α 3 is a hyperparameter; Let t be the set of users who access the satellite and watch the video in the scenario at time t. Us,v This refers to the set of users in the scene who are connected to the satellite and watching video. For users i The j User experience quality of each video segment; Where μ1 is a constant; μ2 is a constant; μ3 is a constant; For users The video quality of the j-th video block; To re-buffer the impact of events on user experience; The impact of changes in video quality during playback; P i,j For users i Download video chunks j The energy consumed; For users i’ The j User experience quality of each video segment.

3. The video transmission rate control method under a three-dimensional heterogeneous network according to claim 2, characterized in that, The distributed sub-optimization objective includes ;in, For users i The j The utility of each video block; α4 is a hyperparameter; For users i In the j Buffer length for one video block; For users i In the j- The buffer length for one video block.

4. The video transmission rate control method under a three-dimensional heterogeneous network according to claim 2, characterized in that, The parameter update method for the evaluation network during training includes: updating the evaluation network using gradient descent based on the time difference algorithm. parameters ; The output of the target network includes: ,in, y j For target network The output; For agent a, the video block j The observed values, For intelligent agents a For video blocks j The action; For agent a, the video block j+1 The observed values, For intelligent agents a For video blocks j+1 The action; In strategy The action value function.

5. The video transmission rate control method under a three-dimensional heterogeneous network according to claim 2, characterized in that, The centralized update includes: the global network updates based on the parameters of the training video segments; the update includes: ; ;in, Θ g' To update the global network parameter set; Θ g The global network parameter set; ep represents the segment parameters; for The influence of parameters; k To decide ϕ (⋅) Hyperparameters of function shape; Hyperparameters used to determine the center location; episode The parameters are training period parameters, which are based on a single user's complete viewing period. The distributed propagation includes: ;in, For the updated network parameter set of agent a; For agent a, the network parameter set; ;in for step and episode The combined effect of parameters; Clip The function is a truncation function, with values ​​limited to 0~1; L The lower bound is the upper bound of the Clip function variable. H The lower bound of the variable in the Clip function is the upper bound. step The parameter is the time it takes to download a video block while the user is watching the video.

6. A video transmission rate control device for a three-dimensional heterogeneous network, characterized in that, include: The optimization target building module is configured to build optimization targets for LEO satellite internet on-demand video transmission rate control; The optimization objective is a multi-dimensional global optimization objective, including maximizing the overall user experience quality, minimizing energy consumption, and maximizing the fairness of user experience quality. A distributed sub-optimization objective construction module is configured to decompose the optimization objective into distributed sub-optimization objectives; The information acquisition module is configured to acquire the state information and cooperation information of all intelligent agents; wherein, the state information of each intelligent agent includes channel coefficients, interference from other target channels, and corresponding user experience quality information; the cooperation information includes the average user experience quality and maximum user experience quality difference of all users other than itself at the current moment, as well as the total power consumption, average power consumption, and maximum power consumption of other intelligent agents; The rate decision set acquisition module is used to input the state information and cooperation information of the agents into the DDPG model of multiple agents interacting, and obtain the rate decision set made by the satellite for each user; The training module is used to construct a rate decision process model to determine the input, joint policy, and output of the DDPG model for the interaction of the multiple agents; The cumulative value function, action value function, parameter update method of the policy network during training, parameter update method of the evaluation network during training, and output of the target network of the DDPG model for interaction of the multiple agents are determined. Determine the global network; In response to determining that the agent has completed training on the training video segments, centralized updates and distributed propagation are performed; The rate decision process model includes: Environment and conditions, including Where S is the set of all state information of all agents in the satellite network during the rate decision-making process; S Q To watch video on demand at a given time User experience quality score for each client. ; S P For satellite Power values ​​allocated to each user. ; S r for n The bitrate level requests sent by each client; S B For buffer state notification; and S h for n Channel state parameters for each client; the environment has a real state. s,s∈S ; Actions, including the actions of each agent. u a ; u a ∈U , U For satellite n A set of rate decisions made by each client r At each time step, each agent's decision-making action will reference the actions of other agents, forming a joint action among multiple agents. u ; Transition probabilities, including the state transition probability function P， satisfy P(s'│s,u):S×U×S→[0,1]; s This is the current state. s' The next state; Rewards, including global rewards shared by all agents. R ,satisfy R(s,u):S×U→R The global reward includes the average user quality experience value, average energy consumption, and user quality experience fairness considerations for all users at a given time; the global reward includes the optimization objective. Observations, including observations for each agent at each time step. z, z∈Z, Z The set of observations from all agents; the action observation history of each agent. ; T The duration of the action observation history for each agent; the individual reward for each agent includes the distributed sub-optimization objective; Let Z×U be the Kling closure, and let Z be the set of all finite-length sequences consisting of elements in this set.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as claimed in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1 to 5.

Images