Air-space-ground network congestion control method based on deep reinforcement learning

Abstract

The invention discloses an air-space-ground network congestion control method based on deep reinforcement learning. The method comprises the following steps: acquiring data information of a target air-space-ground network; setting an optimization target and carrying out modeling; setting a state space, an action space and a reward function of the decision process; constructing a state prediction module, a reward redistribution module and an optimization strategy module of the decision process; and performing congestion control of the space-air-ground network based on deep reinforcement learning according to the constructed module. According to the invention, through obtaining the data information of the plurality of target air-space-ground networks, based on the setting of the state space, the action space and the reward function, and the realization of state prediction, reward redistribution and optimization strategies, the congestion control of the air-space-ground network based on deep reinforcement learning is realized, the reliability is higher, and the method is more suitable for large-scale popularization and application. The precision is better, and the effect is better.

Description

Technical Field

[0001] The present invention belongs to the field of communications, and specifically relates to an air-ground-space network congestion control method based on deep reinforcement learning. Background Art

[0002] In order to achieve seamless global coverage and meet users' personalized needs for network services, a space-air-ground integrated network (GASN) that combines low-orbit satellite systems, air networks and ground communications has become an emerging network architecture and has attracted widespread attention. The network transport layer adaptive congestion control mechanism attempts to find a balance between improving network resource utilization and avoiding network congestion, becoming a key technology for users to ensure transmission quality in different network scenarios.

[0003] Compared with the traditional ground network link environment, the mobility of low-orbit satellites in the space-ground network system, the unpredictable weather changes, and the diversity of user needs result in the space-ground network having the characteristics of extended data round-trip time, high bit error rate, and high heterogeneity of uplink and downlink links; therefore, traditional congestion control protocol algorithms applied to ground networks are difficult to achieve good performance in this scenario.

[0004] At present, the traditional and commonly used congestion control methods mainly include state-independent schemes and optimization-based schemes. State-independent methods are usually simple in terms of algorithm or protocol design, but such schemes lack careful consideration of real-time status and optimal solutions, so their performance may be very poor in highly time-varying networks; optimization-based schemes must accurately predict the future values ​​of some key parameters (such as link utilization, user usage) as input, and require accurate mathematical models to estimate or characterize network behavior. Therefore, when facing complex and highly dynamic networks, the comprehensive performance of such schemes is relatively poor, and the implementation is difficult. Summary of the invention

[0005] The purpose of the present invention is to provide an air-ground-space network congestion control method based on deep reinforcement learning with high reliability, good accuracy and better effect.

[0006] The air-ground-space network congestion control method based on deep reinforcement learning provided by the present invention comprises the following steps:

[0007] S1. Obtain data information of the target air-space-ground network;

[0008] S2. According to the data information obtained in step S1, set the optimization goal and perform modeling;

[0009] S3. Set the state space, action space and reward function of the decision process;

[0010] S4. Construct the state prediction module, reward redistribution module and optimization strategy module of the decision-making process;

[0011] S5. According to the constructed modules, congestion control of the air-space-ground network is performed based on deep reinforcement learning.

[0012] Step S2 sets the optimization target and performs modeling based on the data information obtained in step S1, which specifically includes the following steps:

[0013] The optimization goals set include throughput, latency, jitter, packet loss rate, and reliability of user applications;

[0014] Throughput refers to the amount of data successfully transmitted per unit time, and is used to reflect the transmission capacity of the network and the data transmission efficiency of the application;

[0015] Latency indicates the total time from sending to receiving data, and is used to reflect the response speed and real-time performance of user applications;

[0016] Jitter refers to the variation in the time interval between arrival of consecutive data packets and is used to reflect the playback quality of streaming media applications;

[0017] The packet loss rate indicates the ratio of lost packets to the total sent packets during data transmission, which is used to reflect the reliability and integrity of user applications.

[0018] Reliability refers to the ability of a network system to provide uninterrupted services within a given period of time, and is used to reflect the continuity and availability of user applications.

[0019] Let F be the set of flows with at least one and at most two of the optimization objectives, T f (t) is the long-term throughput of subflow f started by a single terrestrial user measured at time t, L f (t) is the long-term packet loss rate of subflow f started by a single terrestrial user measured at time t and is defined as

[0020]

[0021] Where L ep is the number of cycles considered; is the throughput measured in the kth cycle; η is the set attenuation coefficient; The packet loss rate measured for the kth period;

[0022] Adopt D f (t) represents the delay of sub-flow f measured at time t, J f (t) represents the jitter of sub-stream f measured at time t, R f(t) represents the reliability of subflow f measured at time t; the single-index utility function at time t Set to

[0023]

[0024] Where SI represents the single indicator factor considered; T represents the throughput indicator; L represents the delay indicator; D represents the jitter indicator; J represents the packet loss rate indicator; R represents the reliability indicator;

[0025] The single-index utility of all flows of F except subflow f at time t and Expressed as

[0026]

[0027] Where f' represents any flow in F except subflow f; is the single-index utility function of f' at time t;

[0028] For simplicity and without loss of generality, two dual-index utility functions are used, expressed as

[0029]

[0030] In the formula It is a utility function that considers both throughput and latency indicators; is the throughput weight; is the delay weight; It is a utility function that considers the dual indicators of reliability and packet loss rate; is the reliability weight; is the packet loss rate weight; U α () is the fairness coefficient function, and α is the fairness parameter;

[0031] The double index utility of all flows of F except subflow f at time t and Expressed as

[0032]

[0033] The state space of the decision-making process is set in step S3, which specifically includes the following steps:

[0034] Incorporate air-space-ground network environment information and network layer information into the state space to dynamically learn potential bottleneck links;

[0035] The state space at time t includes the air-ground integrated network state, the network layer state, and the transport layer state, expressed as in, is the state of subflow f at time t, and is expressed as is the space-ground integrated network state of subflow f at time t, is the network layer state of subflow f at time t, is the transport layer state of sub-flow f at time t;

[0036] and Specifically expressed as

[0037]

[0038] In the formula is the air-to-ground link channel gain at time t; is the channel gain of the space-to-space link at time t; is the channel gain of the space-to-ground link at time t; is the ratio of the used power of the air-ground link to the maximum power at time t; is the ratio of the used power of the space-sky link to the maximum power at time t; is the ratio of the used power of the sky-ground link to the maximum power at time t; It is the ratio of the frequency band used by the air-ground link to the total frequency band resources at time t; It is the ratio of the frequency band used by the space-to-space link to the total frequency band resources at time t; It is the ratio of the frequency band used by the space-ground link to the total frequency band resources at time t; is the number of data packets backlogged in the LEO satellite; is the number of data packets backlogged in the drone; is the number of data packets backlogged at the satellite earth station; is the channel resource utilization rate of the export link of the LEO satellite; is the channel resource utilization rate of the UAV’s export link; is the channel resource utilization rate of the satellite earth station’s export link; sr t f is the sending rate of sub-flow f at time t; is the throughput of subflow f at time t; is the average RTT of subflow f at time t; is the average RTT deviation of subflow f at time t; is the congestion window size of subflow f at time t; is the congestion window change value between time t-1 and time t of subflow f; is the throughput change between time t-1 and time t of subflow f; is the ratio of the minimum RTT of subflow f to the RTT at time t; lr t f is the loss rate of subflow f at time t; is the RTT of subflow f at time t; It is the interval value of the arrival time of the ACK returned between time t-1 and time t of sub-flow f.

[0039] The action space of the decision-making process is set in step S3, which specifically includes the following steps:

[0040] Set the action expression based on the sending rate control, expressed as

[0041]

[0042] In the formula is the action space expression taken by subflow f at time t; Δsr t f is the sending rate change value, and Δsr t f ≥0 corresponds to the sending rate increment, Δsr t f <0 corresponds to a decreasing value of the sending rate; is the repetition factor of the data packet launched on sub-flow f at time t;

[0043] According to the action expression and the current state, the sending rate update action is converted into the congestion window update action, which is expressed as

[0044]

[0045] Where L MTU is the length of the maximum transmission unit of the transport layer;

[0046] Therefore, the global action space a at time t t Expressed as

[0047] The reward function of the decision-making process in step S3 is set, which specifically includes the following steps:

[0048] Set the reward function of subflow f to

[0049]

[0050] Where r t f is the cumulative reward value; is the trade-off coefficient used to weigh the benefits between sub-flow f and other flows; It represents the utility function of subflow f considering single index factor or multiple index factors;

[0051] Divide all state-action sequences of a time into several equal and length L SASshort sequence, and estimate the cumulative reward for each short sequence; for the time from t to t+L SAS -1 The sequence ends at the moment, the corresponding cumulative reward Expressed as The cumulative reward is clipped to ensure that the value range is [-1,1].

[0052] The state prediction module of the decision-making process in step S4 specifically includes the following steps:

[0053] Obtain the air-space-ground integrated network status, network layer status and transport layer status provided by the current environment, and predict through the constructed status prediction module to obtain the latest air-space-ground integrated network status and network layer status;

[0054] It is assumed that the ground user can implement and obtain the network status and network layer status of the air-space-ground integrated system, and transmit the obtained air-space-ground integrated network system status and network layer status together in the free field of the ACK data packet protocol header to the sender of the response data packet;

[0055] In each cycle, each sender obtains several time-stamped air-ground integrated network states and network layer state sequences to predict the latest air-ground integrated network states and network layer states;

[0056] The indicators for predicting the status of the air-ground integrated network and the network layer status include channel gain, the ratio of used power to maximum power, the ratio of used frequency band to total frequency band resources, the amount of data packet backlog, and the utilization rate of the export link of the bottleneck node;

[0057] Five GRU-based neural network models with the same structure are used to predict the channel gain, the ratio of used power to maximum power, the ratio of used frequency band to total frequency band resources, the data packet backlog and the export link utilization of the bottleneck node respectively and separately. At the same time, the time interval of the state is included in the prediction as a feature. The true values ​​of the air-space-ground integrated network state and the network layer state are obtained by LEO satellites, UAVs, satellite ground station base stations or ground users in each time slot.

[0058] The reward redistribution module of the decision-making process in step S4 specifically includes the following steps:

[0059] The congestion control problem with dynamic delay, equilibrium exploration and fairness guarantee is regarded as a sequential Markov decision process SDP with delayed rewards, which is represented by a five-tuple (S, A, R, P, γ), where S is the global state space, A is the joint action space of all individuals, R is the reward function set of all individuals, P is the state transition probability matrix, and γ is the discount factor;

[0060] Based on the second-order Markov reward redistribution framework, a sequential Markov decision process without cumulative rewards is obtained from the sequential Markov decision process M. And M and Have the same state space, action space, state transition probability and optimal strategy;

[0061] set up For a sequential Markov decision process M with delayed rewards The Q-value function of Denotes a sequential Markov decision process M with delayed rewards The redistribution reward of , then the second-order Markov reward redistribution is expressed as

[0062]

[0063] Set the reward-prediction function g f , used to predict the expected cumulative reward of M at the end of a given state-action sequence at time t; the reward-prediction function g f The output is because Included Therefore, we set the difference function Δ f And used to calculate The information carried in f Defined as and The numerical difference between f The state-action sequence of the process is represented as

[0064]

[0065] At the same time, g f Must ensure

[0066] From this we can get: (1) Depend on Calculated and used as Estimates; (2)

[0067]

[0068] G f (Δ 0:T ) and g f (Δ 0:t ), which is converted to in, State-Action Sequence The contribution to the expected reward of the prediction is given by Calculated;

[0069] set up but and There exists

[0070] because and There is an error between the two, so an additional redistribution reward is set and

[0071]

[0072] Finally get

[0073] The calculation process of the reward redistribution module includes the following steps:

[0074] For a state-action sequence Indicates the expected feedback before being executed. It indicates the expected feedback after execution. and The numerical difference between The contribution of GRU is used to predict gf. During the training process, As input, As a label, Redistributed to each cycle, it eventually forms a sequential Markov decision process with delayed rewards that can be solved by the DRL method

[0075]

[0076] The specific steps include:

[0077] Input: state-action sequence pairs Feedback prediction function gf, difference function Δf, 0≤t≤T, f∈F;

[0078] Output: state-action sequence Contribution to the predicted expected reward

[0079] A. Calculation

[0080] B. Calculation

[0081] C. Calculation get

[0082] D. Calculation get

[0083] E. Return calculation results and

[0084] Correct the calculation process of the reward redistribution module, including the following steps:

[0085] Will The expected value of is expressed as the true value, and we get Divide each round into several blocks of length L SAS The cumulative reward of each short sequence will be redistributed to each action; the redistribution correction method is used to ensure Established;

[0086] The specific steps include:

[0087] Input: Feedback prediction function Accumulated Rewards

[0088] Output: Redistribution of rewards

[0089] a. Calculation

[0090] b. Calculate the uncorrected redistributed rewards for where t≠0, and

[0091]

[0092] c. Calculate the average error

[0093] d. Calculate the revised redistribution reward for

[0094] e. Return

[0095] The optimization strategy module for constructing the decision-making process in step S4 specifically includes the following steps:

[0096] The feature extraction network first takes the states of all activity flows in each cycle as input and outputs a feature vector Since most of the L rp rounds, so at most L rp feature vectors; the global state sequence is expressed as Then, The input is fed into another representation network to generate a global feature vector containing global information. Finally, each agent uses the global feature vector to generate actions. A GRU-based neural network is used as the representation network.

[0097] Each agent shares a global feature vector; the actor network parameters The critic network parameters are used to output the action probability of sub-flow f based on the current state. Used to update and feed back to the actor network by predicting future rewards; based on the feedback behavior, the actor network can continuously adjust the probability of future action selection, and repeat the feedback process in each learning iteration to improve the strategy;

[0098] In order to avoid fluctuations in the training process that are greater than the set range and to ensure the stability of the learning process, the following formula is used to update the neural network parameters:

[0099]

[0100] In the formula is the clipping loss function, which is used to limit the amplitude of strategy update; represents the expected value of all rounds; is the ratio between the new and old strategies, and For the new strategy in state Take action The probability of For the old policy in state Take action probability; is the advantage function of the action at time t, which is used to measure the advantage of the action at time t relative to the average action; is an intermediate function, and ε is a set positive number, which is used to clip the ratio in the objective function. To avoid policy updates being too large or too small;

[0101] The generalized advantage estimator is used to calculate Expressed as

[0102]

[0103] Where γ is the discount factor used to reduce the weight of future returns, λ is the parameter that controls the trade-off between bias and variance, T is the total number of periods, is the Bellman residual term and V() is an approximate value function, and V() is usually estimated by a neural network;

[0104] The total loss is calculated as

[0105]

[0106] In the formula is the value function loss, which is used to measure the approximation between the value function and the expected return at time t; H is the entropy function of the strategy that promotes exploration by hindering deterministic strategies at each time t; c 1 is the first weight coefficient; c 2 is the second weight coefficient; R represents the GRU-based neural network;

[0107] During the optimization process, all logarithmic probability sequences are used to output action sequences to ensure fairness; the individual logarithms associated with each previously selected action are modified to a set value to prevent the softmax function from generating selected actions for the logarithms; the gradients of the logarithms of invalid operations are set to zero; the softmax function is used to output actions based on the modified logarithms, expressed as

[0108] a f,j =softmax(L f )

[0109]

[0110] Where a f,j is the jth action of subflow f; L f is the modified logarithmic probability sequence of substream f, and L f = {l f,1 ,...,l f,j ,...,l f,J}, J is the number of actions that can be selected by subflow f; σ is the set negative reward;

[0111] Subflow f must select an action from each of the following two types of action spaces:

[0112]

[0113] In the formula is the sending rate increment of sub-flow f at time t; A negative value indicates that the sending rate of sub-flow f is reduced. A positive value indicates that the sending rate of sub-flow f is increased. 0 indicates that the same sending rate is caused for sub-flow f; N is the maximum number of repetitions in a data packet; Δsr is the degree of change in the sending rate; is the packet repetition factor of subflow f at time t;

[0114] Therefore, the action Converted from the following formula The action space expression for controlling the sending rate is:

[0115]

[0116] Through the reward redistribution mechanism, it is possible to assume that the action feedback in each sequence will be L after the end of the sequence. SAS Return at any time.

[0117] The air-ground-space network congestion control method based on deep reinforcement learning, wherein the training process includes the following steps:

[0118] Input: maximum training round E, training cycle length T, algorithm weight λ, discount factor γ, batch size N batch ;

[0119] Output: Actor network parameters Critic Network Parameters Characterizing Network Parameters

[0120] (1) Initialize the actor network parameters using random parameters Critic Network Parameters and characterize network parameters

[0121] (2) Initialize replay buffer B f ;

[0122] (3) Initialize an Ornstein-Uhlenbeck random process for exploratory actions;

[0123] (4) At each time t, collect the air-space-ground integrated network status, network layer status, and transport layer status of all flows;

[0124] (5) Use the state prediction module to predict the latest chapter and obtain the global state s t ;

[0125] (6) Through the GRU neural network R from the nearest L rp The global state of the segment obtains the feature vector h t ;

[0126] (7) Get an action based on the strategy function π()

[0127] (8) Action-based Generate actions using a random process

[0128] (9) Execute action And observe the current utility function;

[0129] (10) Store the global state s t , eigenvector h t ,action and the corresponding utility function value;

[0130] (11) Calculation

[0131] (12) Calculate the cumulative reward of each sequence based on the utility function value;

[0132] (13) Through the calculation process of the reward redistribution module, the accumulated rewards are redistributed to obtain

[0133] (14) Transfer process Store to buffer B f ;

[0134] (15) Extract N from the replay buffer batch samples and divided into K batches;

[0135] (16) In each small batch, perform the following steps (17) to (20) in sequence;

[0136] (17) Obtain the feature vector h through the representation network R k for

[0137] (18) Determine the action based on the strategy function π() for

[0138] (19) Calculation and

[0139]

[0140] (20) Based on total loss Update actor network parameters Critic Network Parameters and characterize network parameters

[0141]

[0142] (21) Return the final actor network parameters Critic Network Parameters and characterize network parameters

[0143] The air-space-ground-air network congestion control method based on deep reinforcement learning provided by the present invention not only realizes the congestion control of the air-space-ground-air network based on deep reinforcement learning, but also has higher reliability, better accuracy and better effect by acquiring data information of multiple target air-space-ground-air networks, and based on the setting of state space, action space and reward function, as well as the implementation of state prediction, reward redistribution and optimization strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0144] Figure 1 The figure is a schematic diagram of the method flow of the present invention.

[0145] Figure 2 Schematic diagram of a typical end-to-end data transmission scenario of an air-space-ground integrated network according to the method of the present invention.

[0146] Figure 3 Schematic diagram of the overall structural design of the method of the present invention based on a heterogeneous multi-agent framework.

[0147] Figure 4 Schematic diagram of the feature extraction network architecture of the method of the present invention.

[0148] Figure 5 Schematic diagram of the model training convergence process of an embodiment of the method of the present invention.

[0149] Figure 6 A schematic diagram of satisfying service quality requirements during the model training process of an embodiment of the method of the present invention. DETAILED DESCRIPTION

[0150] like Figure 1 The method flow diagram of the method of the present invention is shown as follows: The air-ground-space network congestion control method based on deep reinforcement learning disclosed in the present invention comprises the following steps:

[0151] S1. Obtain data information of the target air-space-ground network;

[0152] like Figure 2The figure shows a typical end-to-end data transmission scenario in an integrated air-ground-space network. A group of ground users can access a remote application server through an integrated air-ground-space network consisting of wireless access points, satellite earth stations, drones as mobile base stations, and low-orbit satellites. In addition, the network architecture is also capable of transmitting the data collected by the sensor nodes to a remote data center, where the sink node is responsible for establishing a connection with the remote storage device to ensure the quality of data transmission. Of course, isolated sensor nodes can also be responsible for the remote end-to-end transmission of their own sensor data. Due to the burstiness of the sensing data upload task, the sink node needs to deploy a congestion control scheme to avoid congestion on the data upload path. On the contrary, for ground users, the data size of the request packet is usually small, while the data size of the response packet is usually large. Therefore, the application server also needs to deploy a congestion control scheme to avoid congestion on the data download path. It is known that in an integrated air-ground-space network, the channel quality changes of the air-space link and the air-ground link are highly dynamic and can be regarded as a potential bottleneck for the performance of the entire network. According to the general principle of congestion control, when congestion occurs in the network environment due to various complex factors (such as a large number of unconfirmed data packets, link quality deterioration, etc.), the transmission rate of the transport layer should be reduced. The purpose of the present invention is to propose a dynamic and adaptive congestion control method for the potential bottleneck link of the above network, so as to solve the problem that the current congestion control method cannot be well applied to highly dynamic and heterogeneous air-ground-space networks;

[0153] S2. According to the data information obtained in step S1, set the optimization target and perform modeling; specifically, the steps include:

[0154] The present invention defines congestion control as a decision-making problem that considers dynamic delays, seeks equilibrium states, and ensures fairness. The dynamic delay characteristics of the problem mean that the feedback reward for each action will not be returned at a fixed time, and the feedback reward may be generated by several consecutive actions. Therefore, the concept of reward redistribution should be adopted to deal with this problem. The balance-seeking feature of this problem means that it is necessary to find a balance between several mutually exclusive QoS performance objectives, such as achieving good throughput at an acceptable RTT for users. The fairness characteristics of this problem require that when a group of end-to-end flows share limited bandwidth, fairness of all sub-flows must be maintained;

[0155] The optimization goals set include throughput, latency, jitter, packet loss rate, and reliability of user applications;

[0156] Throughput refers to the amount of data successfully transmitted per unit time, which is used to reflect the transmission capacity of the network and the data transmission efficiency of the application. For example, for big data applications such as video and file transmission, a higher throughput is required.

[0157] Latency refers to the total time from sending to receiving data, which is used to reflect the response speed and real-time performance of user applications. For example, voice and video call applications with high real-time requirements require low latency.

[0158] Jitter refers to the change in the time interval between the arrival of consecutive data packets. It is used to reflect the playback quality of streaming media applications and affects the playback quality of streaming media applications. For example, streaming media applications require jitter to be as small as possible.

[0159] The packet loss rate indicates the ratio of lost packets to the total sent packets during data transmission, which is used to reflect the reliability and integrity of user applications. It directly affects the reliability and integrity of applications. For example, for file transfer applications with high reliability requirements, the packet loss rate needs to be as low as possible.

[0160] Reliability refers to the ability of a network system to provide uninterrupted services within a given period of time, and is used to reflect the continuity and availability of user applications. For example, for critical business applications, high reliability guarantees are required.

[0161] In fact, most applications today may focus on one or two of the five QoS indicators mentioned above; applications that have high requirements for three or more indicators at the same time are rare; on this basis, the above congestion control problem should be applied to all combinations of two indicators with practical significance, taking the lower bounds of other QoS indicators as constraints, rather than just limiting them to throughput and delay, so as to more flexibly adapt to the QoS requirements of most applications;

[0162] Let F be the set of flows with at least one and at most two of the optimization objectives, T f (t) is the long-term throughput of subflow f started by a single terrestrial user measured at time t, L f (t) is the long-term packet loss rate of subflow f started by a single terrestrial user measured at time t and is defined as

[0163]

[0164] Where L ep is the number of cycles considered; is the throughput measured in the kth cycle; η is the set attenuation coefficient, and its value range is (0,1]; The packet loss rate measured for the kth period;

[0165] Adopt D f (t) represents the delay of sub-flow f measured at time t, J f (t) represents the jitter of sub-stream f measured at time t, R f (t) represents the reliability of subflow f measured at time t; the single-index utility function at time t Set to

[0166]

[0167] Where SI represents the single indicator factor considered; T represents the throughput indicator; L represents the delay indicator; D represents the jitter indicator; J represents the packet loss rate indicator; R represents the reliability indicator;

[0168] The single-index utility of all flows of F except subflow f at time t and Expressed as

[0169]

[0170] Where f' represents any flow in F except subflow f; is the single-index utility function of f' at time t;

[0171] For simplicity and without loss of generality, two dual-index utility functions are used, expressed as

[0172]

[0173] In the formula It is a utility function that considers both throughput and latency indicators; is the throughput weight; is the delay weight; It is a utility function that considers the dual indicators of reliability and packet loss rate; is the reliability weight; is the packet loss rate weight; U α () is the fairness coefficient function, and α is the fairness parameter;

[0174] The double index utility of all flows of F except subflow f at time t and Expressed as

[0175]

[0176] The present invention proposes an improved semi-independent proximal policy optimization (ISIPPO) model method to solve the problem of optimizing the above utility function;

[0177] S3. Set the state space, action space and reward function of the decision process; specifically include the following steps:

[0178] Multiple response data streams for the same ground user may come from different application servers. Therefore, for the congestion control scheme deployed on the sender, the multi-agent model is more adaptable and scalable. In addition, ground users can simultaneously initiate requests with different QoS requirements, thereby generating response streams with different QoS requirements. Therefore, it is necessary to design a special action space and reward function for each data stream with different QoS requirements to ensure its specific QoS performance requirements. At the same time, these flows need to share the same state space because they all face the same bottleneck network environment to compete for resources. In the present invention, a multi-heterogeneous agent DRL mode is adopted to design a customized congestion control scheme to adapt to different QoS requirements.

[0179] By adopting a semi-independent training method, the state and action information of neighboring agents is introduced, which enhances the observation ability and decision-making performance of the agents. A coordination reward item based on the state and action of neighboring agents is added. This coordination reward encourages agents to make more coordinated and consistent decisions, improves the stability of the system, and thus shows better performance in complex multi-agent reinforcement learning tasks.

[0180] The state space of the decision-making process is set, which includes the following steps:

[0181] Incorporate air-space-ground network environment information and network layer information into the state space to dynamically learn potential bottleneck links;

[0182] The state space at time t includes the air-ground integrated network state, the network layer state, and the transport layer state, expressed as in, is the state of subflow f at time t, and is expressed as is the space-ground integrated network state of subflow f at time t, is the network layer state of subflow f at time t, is the transport layer state of sub-flow f at time t;

[0183] and Specifically expressed as

[0184]

[0185] In the formula is the air-to-ground link channel gain at time t; is the channel gain of the space-to-space link at time t; is the channel gain of the space-to-ground link at time t; is the ratio of the used power of the air-ground link to the maximum power at time t; is the ratio of the used power of the space-sky link to the maximum power at time t; is the ratio of the used power of the sky-ground link to the maximum power at time t; It is the ratio of the frequency band used by the air-ground link to the total frequency band resources at time t; It is the ratio of the frequency band used by the space-to-space link to the total frequency band resources at time t; It is the ratio of the frequency band used by the space-ground link to the total frequency band resources at time t; is the number of data packets backlogged in the LEO satellite; is the number of data packets backlogged in the drone; is the number of data packets backlogged at the satellite earth station; is the channel resource utilization rate of the export link of the LEO satellite; is the channel resource utilization rate of the UAV’s export link; is the channel resource utilization rate of the satellite earth station’s export link; sr t f is the sending rate of sub-flow f at time t; is the throughput of subflow f at time t; is the average RTT of subflow f at time t; is the average RTT deviation of subflow f at time t; is the congestion window size of subflow f at time t; is the congestion window change value between time t-1 and time t of subflow f; is the throughput change between time t-1 and time t of subflow f; is the ratio of the minimum RTT of subflow f to the RTT at time t; lr t f is the loss rate of subflow f at time t; is the RTT of subflow f at time t; is the interval value of the arrival time of the ACK returned between time t-1 and time t of sub-flow f;

[0186] The data backlog and egress link utilization of the LEO satellite here specifically refer to the values ​​when it sends data to the satellite earth station or drone relay station; similarly, the satellite earth station or drone relay station also refers to the values ​​when they send data to the wireless access point;

[0187] Setting the action space of the decision-making process includes the following steps:

[0188] Compared with the existing method of controlling congestion by updating the congestion window size, timely adjusting the sending rate will be more direct and effective in handling congestion events; however, in actual systems, the transmission operation of the transport layer is performed at the granularity of data packets; therefore, converting the sending rate limit into the number of packets that determine the congestion window size is an effective method; on this basis, considering different congestion control targets, based on the same sending rate limit, the determined congestion control window size should be different;

[0189] Therefore, the action expression based on the sending rate control is set as

[0190]

[0191] In the formula is the action space expression taken by subflow f at time t; Δsr t f is the sending rate change value, and Δsr t f ≥0 corresponds to the sending rate increment, Δsr t f <0 corresponds to a decreasing value of the sending rate; is the repetition factor of the data packet launched on sub-flow f at time t;

[0192] According to the action expression and the current state, the sending rate update action is converted into the congestion window update action, which is expressed as

[0193]

[0194] Where L MTU is the length of the maximum transmission unit of the transport layer;

[0195] Therefore, the global action space a at time t t Expressed as

[0196] Setting the reward function of the decision-making process includes the following steps:

[0197] The reward function should be designed according to the actual needs of the upper-layer application. Since there are data flows with different congestion control targets, each data flow with a specific congestion control target should have its own unique reward function. In addition, in order to avoid affecting the interests of other flows due to greed, the overall benefit of the system must be considered when designing individual reward functions.

[0198] Therefore, the reward function of subflow f is set to

[0199]

[0200] Where r tf is the cumulative reward value; is the trade-off coefficient used to weigh the benefits between sub-flow f and other flows; Indicates the utility function of the sub-flow considering single indicator factors or multiple indicator factors;

[0201] Divide all state-action sequences of a time into several equal and length L SAS short sequence, and estimate the cumulative reward for each short sequence; for the time from t to t+L SAS -1 The sequence ends at the moment, the corresponding cumulative reward Expressed as The cumulative reward is clipped to ensure that the value range is [-1,1];

[0202] S4. Construct the state prediction module, reward redistribution module and optimization strategy module of the decision-making process; specifically, the following steps are included:

[0203] Each agent receives three types of status information from the environment: the air-ground integrated network status (GASN), the network layer status (NL), and the transport layer status (TL). These status information are first predicted by the GASN-NL status prediction module to obtain the latest GASN and NL status;

[0204] Then, the predicted GASN-NL state and the TL state of all flows are merged as the global state and input into the representation network to extract the feature vector. In the action selection process, each agent adopts the classic actor-critic (AC) architecture to optimize its utility function by receiving the GASN, NL and TL state information related to the flow it is responsible for and the feature vector, and then determines the sending rate of the specific flow it controls;

[0205] In the specific training process, the reward redistribution module redistributes the cumulative rewards provided by the environment in a specific period to each state-action pair, and finally constitutes an experience sample that can be directly learned (i.e., state, action, redistributed reward, next state); the overall framework is shown in the attached figure. Figure 3 As shown;

[0206] Constructing the state prediction module of the decision-making process includes the following steps:

[0207] Obtain the air-space-ground integrated network status, network layer status and transport layer status provided by the current environment, and predict through the constructed status prediction module to obtain the latest air-space-ground integrated network status and network layer status;

[0208] It is assumed that the ground user can obtain the air-space-ground integrated network status and network layer status, and transmit the obtained air-space-ground integrated network status and network layer status together in the idle field of the ACK data packet protocol header to the sender of the response data packet;

[0209] In each cycle, each sender obtains several time-stamped air-ground integrated network states and network layer state sequences to predict the latest air-ground integrated network states and network layer states;

[0210] The indicators for predicting the status of the air-ground integrated network and the network layer status include channel gain, the ratio of used power to maximum power, the ratio of used frequency band to total frequency band resources, the amount of data packet backlog, and the utilization rate of the export link of the bottleneck node;

[0211] Five GRU-based neural network models with the same structure are used to predict the channel gain, the ratio of used power to maximum power, the ratio of used frequency band to total frequency band resources, the data packet backlog and the export link utilization rate of the bottleneck node respectively and separately. At the same time, the time interval of the state is included in the prediction as a feature. The true value of the air-ground integrated network state and the network layer state is obtained by LEO satellites, drones, satellite ground station base stations or ground users in each time slot.

[0212] Constructing the reward redistribution module of the decision-making process includes the following steps:

[0213] The congestion control problem with dynamic delay, equilibrium exploration and fairness guarantee is regarded as a sequential Markov decision process SDP with delayed rewards, which is represented by a five-tuple (S, A, R, P, γ), where S is the global state space, A is the joint action space of all individuals, R is the reward function set of all individuals, P is the state transition probability matrix, and γ is the discount factor;

[0214] Based on the second-order Markov reward redistribution framework, a sequential Markov decision process without cumulative rewards is obtained from the sequential Markov decision process M. And M and Have the same state space, action space, state transition probability and optimal strategy;

[0215] set up For a sequential Markov decision process M with delayed rewards The Q-value function of Denotes a sequential Markov decision process M with delayed rewards The redistribution reward of , then the second-order Markov reward redistribution is expressed as

[0216]

[0217] Set the reward-prediction function gf , used to predict the expected cumulative reward of M at the end of a given state-action sequence at time t; the reward-prediction function g f The output is because Included Therefore, we set the difference function Δ f And used to calculate The information carried in f Defined as and The numerical difference between f The state-action sequence of the process is represented as

[0218]

[0219] At the same time, g f Must ensure

[0220] From this we can get: (1) Depend on Calculated and used as Estimates; (2)

[0221]

[0222] G f (Δ 0:T ) and g f (Δ 0:t ), which is converted to in, State-Action Sequence The contribution to the expected reward of the prediction is given by Calculated;

[0223] set up but and There exists

[0224] because and There is an error between the two, so an additional redistribution reward is set and

[0225]

[0226] Finally get

[0227] The calculation process of the reward redistribution module specifically includes the following steps:

[0228] For a state-action sequence Indicates the expected feedback before being executed. It indicates the expected feedback after execution. and The numerical difference between Contribution of GRU-based network f To make predictions, during the training process, As input, As a label, Redistributed to each cycle, it eventually forms a sequential Markov decision process with delayed rewards that can be solved by the DRL method

[0229]

[0230] The specific steps include:

[0231] Input: state-action sequence pairs Feedback prediction function g f , the difference function Δ f , 0≤t≤T, f∈F;

[0232] Output: state-action sequence Contribution to the predicted expected reward

[0233] A. Calculation

[0234] B. Calculation

[0235] C. Calculation get

[0236] D. Calculation get

[0237] E. Return calculation results and

[0238] At the same time, the calculation process of the reward redistribution module is corrected, including the following steps:

[0239] Will The expected value of is expressed as the true value, and we get Divide each round into several blocks of length L SAS The cumulative reward of each short sequence will be redistributed to each action; the redistribution correction method is used to ensure Established;

[0240] The specific steps include:

[0241] Input: Feedback prediction function Accumulated Rewards

[0242] Output: Redistribution of rewards

[0243] a. Calculation

[0244] b. Calculate the uncorrected redistributed rewards for where t≠0, and

[0245]

[0246] c. Calculate the average error

[0247] d. Calculate the revised redistribution reward for

[0248] e. Return

[0249] Constructing the optimization strategy module of the decision-making process includes the following steps:

[0250] The feature extraction network first takes the states of all activity flows in each cycle as input and outputs a feature vector Since most of the L rp rounds, so at most L rp feature vectors; the global state sequence is expressed as Then, The input is sent to another representation network to generate a global feature vector containing global information; finally, each agent uses the global feature vector to generate actions; a GRU-based neural network is used as the representation network, as follows Figure 4 As shown;

[0251] Each agent shares a global feature vector; the actor network parameters The critic network parameters are used to output the action probability of sub-flow f based on the current state. Used to update and feed back to the actor network by predicting future rewards; based on the feedback behavior, the actor network can continuously adjust the probability of future action selection, and repeat the feedback process in each learning iteration to improve the strategy;

[0252] In order to avoid fluctuations in the training process that are greater than the set range and to ensure the stability of the learning process, the following formula is used to update the neural network parameters:

[0253]

[0254] In the formula is the clipping loss function, which is used to limit the amplitude of strategy update; represents the expected value of all rounds; is the ratio between the new and old strategies, and For the new strategy in state Take action The probability of For the old policy in state Take action probability; is the advantage function of the action at time t, which is used to measure the advantage of the action at time t relative to the average action; is an intermediate function, and ε is a set positive number, which is used to clip the ratio in the objective function. To avoid policy updates being too large or too small;

[0255] The generalized advantage estimator is used to calculate Expressed as

[0256]

[0257] Where γ is the discount factor used to reduce the weight of future returns, λ is the parameter that controls the trade-off between bias and variance, T is the total number of periods, is the Bellman residual term and V() is an approximate value function and is usually estimated by a neural network;

[0258] The total loss is calculated as

[0259]

[0260] In the formula is the value function loss, which is used to measure the approximation between the value function and the expected return at time t; H is the entropy function of the strategy that promotes exploration by hindering deterministic strategies at each time t; c 1 is the first weight coefficient; c 2 is the second weight coefficient; R represents the GRU-based neural network;

[0261] During the optimization process, all logarithmic probability sequences are used to output action sequences to ensure fairness; the individual logarithms associated with each previously selected action are modified to a set value to prevent the softmax function from generating selected actions for the logarithms; the gradients of the logarithms of invalid operations are set to zero; the softmax function is used to output actions based on the modified logarithms, expressed as

[0262] a f,j=softmax(L f )

[0263]

[0264] Where a f,j is the jth action of subflow f; L f is the modified logarithmic probability sequence of substream f, and L f = {l f,1 ,...,l f,j ,...,l f,J}, J is the number of actions that can be selected by subflow f; σ is the set negative reward;

[0265] Subflow f must select an action from each of the following two types of action spaces:

[0266]

[0267] In the formula is the sending rate increment of sub-flow f at time t; A negative value indicates that the sending rate of sub-flow f is reduced. A positive value indicates that the sending rate of sub-flow f is increased. 0 indicates that the same sending rate is caused for sub-flow f; N is the maximum number of repetitions in a data packet; Δsr is the degree of change in the sending rate; is the packet repetition factor of subflow f at time t;

[0268] Therefore, the action Converted from the following formula The action space expression for controlling the sending rate is:

[0269]

[0270] Through the reward redistribution mechanism, it is possible to assume that the action feedback in each sequence will be L after the end of the sequence. SAS Return at all times;

[0271] S5. According to the constructed modules, congestion control of the air-space-ground network is performed based on deep reinforcement learning.

[0272] In specific implementation, in the air-ground-space network congestion control method based on deep reinforcement learning, the training process includes the following steps:

[0273] Input: maximum training round E, training cycle length T, algorithm weight λ, discount factor γ, batch size N batch ;

[0274] Output: Actor network parameters Critic Network Parameters Characterizing Network Parameters

[0275] (1) Initialize the actor network parameters using random parameters Critic Network Parameters and characterize network parameters

[0276] (2) Initialize replay buffer B f ;

[0277] (3) Initialize an Ornstein-Uhlenbeck random process for exploratory actions;

[0278] (4) At each time t, collect the air-space-ground integrated network status, network layer status, and transport layer status of all flows;

[0279] (5) Use the state prediction module to predict the latest chapter and obtain the global state s t ;

[0280] (6) Through the GRU neural network R from the nearest L rp The global state of the segment obtains the feature vector h t ;

[0281] (7) Get an action based on the strategy function π()

[0282] (8) Action-based Generate actions using a random process

[0283] (9) Execute action And observe the current utility function;

[0284] (10) Store the global state s t , eigenvector h t ,action and the corresponding utility function value;

[0285] (11) Calculation

[0286] (12) Calculate the cumulative reward of each sequence based on the utility function value;

[0287] (13) Through the calculation process of the reward redistribution module, the accumulated rewards are redistributed to obtain

[0288] (14) Transfer process Store to buffer B f ;

[0289] (15) Extract N from the replay buffer batch samples and divided into K batches;

[0290] (16) In each small batch, perform the following steps (17) to (20) in sequence;

[0291] (17) Obtain the feature vector h through the representation network R k for

[0292] (18) Determine the action based on the strategy function π() for

[0293] (19) Calculation and

[0294]

[0295] (20) Based on total loss Update actor network parameters Critic Network Parameters and characterize network parameters

[0296]

[0297] (21) Return the final actor network parameters Critic Network Parameters and characterize network parameters

[0298] The method of the present invention is further described below in conjunction with an embodiment:

[0299] Taking the end-to-end data transmission scenario in a typical space-ground integrated network and the downlink network parameters of the StarLink low-orbit satellite Internet constellation as the main reference, the simulation simulates the data transmission scenario between ground users and 22 satellites in the 550km upper orbit plane through 4 drones that communicate at a fixed speed for partial area coverage and satellite earth stations. The specific simulation parameters are detailed in Table 1.

[0300] Table 1 Simulation parameter diagram

[0301] parameter value LEO satellite orbit altitude 550km Number of LEO satellites 22 Satellite flight speed 7.59km / s Satellite operating frequency band Ku Single carrier bandwidth 50-400MHz Antenna gain 30-40dBi Equivalent isotropic radiated power 20-50dBW Free space loss 150-200dB System Noise Temperature 300K Other losses 0-30dB Number of sensing nodes 200 Number of aggregation nodes 4 Drone flight altitude 100m Drone movement speed 30m / s Data segment size 1KB ACK packet size 40B

[0302] In the specific implementation process, the user started a session flow with multiple QoS indicators through the PC, including: dual-indicator session flow (delay and bandwidth, jitter and packet loss rate), single-indicator session flow (high reliability, low delay, high bandwidth). The data center network, as the responder, is responsible for the transmission of a large number of data packets, with long transmission time and data density. Therefore, this example only considers performing congestion control operations on the responder.

[0303] In the present invention, firstly, according to different application requirements (such as video conferencing, file transfer, etc.), a priority is set for each QoS indicator. Ensure that key indicators (such as delay and reliability) are given priority when the network is congested. Then, a monitoring module is deployed on the response end to collect network status information in real time, including delay, bandwidth, jitter, etc. By using a sliding window update mechanism, the current network status is dynamically evaluated. In each time slot, firstly, based on the state information of all sub-streams collected, the trained GRU prediction model is used to predict the latest global link state, and then the observed state of each intelligent agent is obtained. These states are input into the scheme of the present invention for training iteration, and finally reasonable network parameters are obtained, thereby achieving a significant improvement in bandwidth resource utilization on the basis of acceptable transmission delay. See Table 2 for specific training parameters.

[0304] Table 2 Schematic diagram of deep reinforcement learning simulation parameters

[0305] parameter value Discount factor γ 0.99 Learning rate ξ 0.0005 Playback buffer size N 5000 Batch size 32 GAE hyperparameter λ 0.95 Total training rounds 30000 <![CDATA[Weight coefficients c in the reward function 1 and c 2 > 0.8，0.2 The penalty term ε in the reward function 0.001 The number of neurons in the hidden layer 128 Hidden layer activation function ReLU Output layer activation function Identity Number of GRU layers 1 The number of neurons in the GRU hidden layer 256 GRU training rounds 10000

[0306] The present invention uses python+pytorch tools to implement the above solution and obtains Figure 5 and Figure 6 Results shown.

[0307] Figure 5 The convergence of the method of the present invention and the model using only the ISIPPO module is shown. It can be seen that the method of the present invention significantly improves the system performance and stability of the model by adding a reward redistribution strategy and a state prediction module. Specifically, these improvements help the model allocate resources more efficiently and adapt to dynamic network environments, thereby enhancing the overall responsiveness. Figure 5 It can be observed that the model is able to reach convergence faster with this strategy than without the reward redistribution strategy. This accelerated convergence effect indicates that the algorithm is more precise in adjusting the strategy, reducing uncertainty during training. In addition, the fluctuations and spikes in the late stages of the graph are significantly reduced, reflecting the improved stability of the model in the long run. This stability is critical for practical applications because it ensures the reliability and sustained performance of the system under various network conditions.

[0308] In order to further illustrate the beneficial effects of the present invention, the present invention abstracts the above-mentioned air-ground network hybrid path into a single special link, whose delay and packet loss rate are the delay and packet loss rate of the total path, and the bandwidth is the bottleneck link bandwidth. Based on this link, the classic congestion control scheme Cubic algorithm, the optimization-based scheme BBR, and the artificial intelligence-based scheme DRL_CC are obtained as follows: Figure 6The results are shown in the figure. As can be seen from the figure, first, in terms of delay and bandwidth utilization, the method of the present invention can effectively and dynamically adjust its sending rate, significantly reducing the frequency of network congestion. In contrast, the Cubic algorithm performs poorly under high-delay and rapidly changing network conditions, which easily leads to packet loss and retransmission, thereby reducing the overall throughput. Although BBR has improved bandwidth utilization, its sending strategy may cause excessive delay fluctuations when the network load is heavy. Secondly, the method of the present invention can monitor the network status in real time and make adaptive adjustments by introducing a reward redistribution mechanism and a state prediction module. This mechanism enables the method of the present invention to take preventive measures before congestion occurs, thereby effectively reducing the possibility of increased packet loss rate and delay. Although DRL_CC optimizes the congestion control strategy using deep reinforcement learning, its training and reasoning process is complex and has certain limitations in real time. Finally, by comparing the stability of different algorithms under long-term operation, the method of the present invention shows better stability and consistency. This shows that the method of the present invention can not only improve instantaneous performance, but also maintain efficient network resource utilization in long-term use. In summary, the method of the present invention has shown significant advantages in the field of congestion control, has higher adaptability and stability, and is suitable for complex and changeable network environments.

[0309] This paper proposes a new congestion control method based on heterogeneous multi-agent deep reinforcement learning (MHADRL) from a cross-layer perspective, incorporating GASN and network layer (NL) information in addition to transport layer (TL) information. This helps each agent to directly understand the information of potential bottlenecks and take measures to prevent congestion in GASN. At the same time, this paper also proposes a method to predict the latest GASN-NL information from the delayed GASN-NL information.

[0310] The proposed model is used to make congestion control decisions for each active flow, and a gated recurrent unit (GRU)-based model is used to learn the representation of all active flows with variable numbers and different quality of service (QoS) characteristics. At the same time, the GASN-NL and TL information of all active flows are extracted as a fixed-size feature vector using a GRU-based method. This feature vector is concatenated with the personalized information of each target flow as the input of each agent.

[0311] The performance of the present invention in key performance indicators (such as throughput, latency) is significantly improved compared to the baseline. The effectiveness and robustness of the present invention method in handling congestion control of multiple data flows with heterogeneous optimization objectives are demonstrated.

Claims

1. A method for controlling air-ground network congestion based on deep reinforcement learning, comprising the following steps: S1. Obtain data information of the target air-space-ground network; S2. According to the data information obtained in step S1, set the optimization goal and perform modeling; S3. Set the state space, action space and reward function of the decision process; S4. Construct the state prediction module, reward redistribution module and optimization strategy module of the decision-making process; S5. According to the constructed modules, congestion control of the air-space-ground network is performed based on deep reinforcement learning.

2. The air-ground network congestion control method based on deep reinforcement learning according to claim 1 is characterized in that Step S2 sets the optimization target and performs modeling based on the data information obtained in step S1, which specifically includes the following steps: The optimization goals set include throughput, latency, jitter, packet loss rate, and reliability of user applications; Throughput refers to the amount of data successfully transmitted per unit time, and is used to reflect the transmission capacity of the network and the data transmission efficiency of the application; Latency indicates the total time from sending to receiving data, and is used to reflect the response speed and real-time performance of user applications; Jitter refers to the variation in the time interval between arrival of consecutive data packets and is used to reflect the playback quality of streaming media applications; The packet loss rate indicates the ratio of lost packets to the total sent packets during data transmission, which is used to reflect the reliability and integrity of user applications. Reliability refers to the ability of a network system to provide uninterrupted services within a given period of time, and is used to reflect the continuity and availability of user applications. Let F be the set of flows with at least one and at most two of the optimization objectives, T f (t) is the long-term throughput of subflow f started by a single terrestrial user measured at time t, L f (t) is the long-term packet loss rate of subflow f started by a single terrestrial user measured at time t and is defined as Where L ep is the number of cycles considered; is the throughput measured in the kth cycle; η is the set attenuation coefficient; The packet loss rate measured for the kth period; Adopt D f (t) represents the delay of sub-flow f measured at time t, J f (t) represents the jitter of sub-stream f measured at time t, R f (t) represents the reliability of subflow f measured at time t; the single-index utility function at time t Set to Where SI represents the single indicator factor considered; T represents the throughput indicator; L represents the delay indicator; D represents the jitter indicator; J represents the packet loss rate indicator; R represents the reliability indicator; The single-index utility of all flows of F except subflow f at time t and Expressed as Where f' represents any flow in F except subflow f; is the single-index utility function of f' at time t; Two dual-index utility functions are used, expressed as In the formula It is a utility function that considers both throughput and latency indicators; is the throughput weight; is the delay weight; It is a utility function that considers the dual indicators of reliability and packet loss rate; is the reliability weight; is the packet loss rate weight; U α () is the fairness coefficient function, and α is the fairness parameter; The double index utility of all flows of F except subflow f at time t and Expressed as 3. The air-ground-space network congestion control method based on deep reinforcement learning according to claim 2 is characterized in that The state space of the decision-making process is set in step S3, which specifically includes the following steps: Incorporate air-space-ground network environment information and network layer information into the state space to dynamically learn potential bottleneck links; The state space at time t includes the air-ground integrated network state, the network layer state, and the transport layer state, expressed as in, is the state of subflow f at time t, and is expressed as is the space-ground integrated network state of subflow f at time t, is the network layer state of subflow f at time t, is the transport layer state of sub-flow f at time t; and Specifically expressed as In the formula is the air-to-ground link channel gain at time t; is the channel gain of the space-to-space link at time t; is the channel gain of the space-to-ground link at time t; is the ratio of the used power of the air-ground link to the maximum power at time t; is the ratio of the used power of the space-sky link to the maximum power at time t; is the ratio of the used power of the sky-ground link to the maximum power at time t; It is the ratio of the frequency band used by the air-ground link to the total frequency band resources at time t; It is the ratio of the frequency band used by the space-to-space link to the total frequency band resources at time t; It is the ratio of the frequency band used by the space-ground link to the total frequency band resources at time t; is the number of data packets backlogged in the LEO satellite; is the number of data packets backlogged in the drone; is the number of data packets backlogged at the satellite earth station; is the channel resource utilization rate of the export link of the LEO satellite; is the channel resource utilization rate of the UAV’s export link; is the egress link channel resource utilization rate of the satellite earth station; is the sending rate of sub-flow f at time t; is the throughput of subflow f at time t; is the average RTT of subflow f at time t; is the average RTT deviation of subflow f at time t; is the congestion window size of subflow f at time t; is the congestion window change value between time t-1 and time t of subflow f; is the throughput change between time t-1 and time t of subflow f; is the ratio of the minimum RTT of subflow f to the RTT at time t; is the loss rate of subflow f at time t; is the RTT of subflow f at time t; It is the interval value of the arrival time of the ACK returned between time t-1 and time t of sub-flow f.

4. The air-ground network congestion control method based on deep reinforcement learning according to claim 3 is characterized in that The action space of the decision-making process is set in step S3, which specifically includes the following steps: Set the action expression based on the sending rate control, expressed as In the formula is the action space expression taken by subflow f at time t; is the sending rate change value, and Corresponding to the increasing value of the sending rate, Corresponding to the decreasing value of the sending rate; is the repetition factor of the data packet launched on sub-flow f at time t; According to the action expression and the current state, the sending rate update action is converted into the congestion window update action, which is expressed as Where L MTU is the length of the maximum transmission unit of the transport layer; Therefore, the global action space a at time t t Expressed as The reward function of the decision-making process in step S3 is set, which specifically includes the following steps: Set the reward function of subflow f to In the formula is the cumulative reward value; is the trade-off coefficient used to weigh the benefits between sub-flow f and other flows; It represents the utility function of subflow f considering single index factor or multiple index factors; Divide all state-action sequences of a time into several equal and length L SAS short sequence, and estimate the cumulative reward for each short sequence; for the time from t to t+L SAS -1 The sequence ends at the moment, the corresponding cumulative reward Expressed as The cumulative reward is clipped to ensure that the value range is [-1,1].

5. The air-ground network congestion control method based on deep reinforcement learning according to claim 4 is characterized in that The state prediction module of the decision-making process in step S4 specifically includes the following steps: Obtain the air-space-ground integrated network status, network layer status and transport layer status provided by the current environment, and predict through the constructed status prediction module to obtain the latest air-space-ground integrated network status and network layer status; It is assumed that the ground user can obtain the air-space-ground integrated network status and network layer status, and transmit the obtained air-space-ground integrated network status and network layer status together in the idle field of the ACK data packet protocol header to the sender of the response data packet; In each cycle, each sender obtains several time-stamped air-ground integrated network states and network layer state sequences to predict the latest air-ground integrated network states and network layer states; The indicators for predicting the status of the air-ground integrated network and the network layer status include channel gain, the ratio of used power to maximum power, the ratio of used frequency band to total frequency band resources, the amount of data packet backlog, and the utilization rate of the export link of the bottleneck node; Five GRU-based neural network models with the same structure are used to predict the channel gain, the ratio of used power to maximum power, the ratio of used frequency band to total frequency band resources, the amount of data packet backlog, and the egress link utilization of bottleneck nodes respectively and separately. At the same time, the time interval of the state is included in the prediction as a feature. The true values ​​of the air-ground integrated network status and network layer status are obtained by LEO satellites, drones, satellite ground station base stations or ground users in each time slot.

6. The method for controlling air-ground network congestion based on deep reinforcement learning according to claim 5 is characterized in that The reward redistribution module of the decision-making process in step S4 specifically includes the following steps: The congestion control problem with dynamic delay, equilibrium exploration and fairness guarantee is regarded as a sequential Markov decision process SDP with delayed rewards, which is represented by a five-tuple (S, A, R, P, γ), where S is the global state space, A is the joint action space of all individuals, R is the reward function set of all individuals, P is the state transition probability matrix, and γ is the discount factor; Based on the second-order Markov reward redistribution framework, a sequential Markov decision process without cumulative rewards is obtained from the sequential Markov decision process M. And M and Have the same state space, action space, state transition probability and optimal strategy; set up For a sequential Markov decision process M with delayed rewards The Q-value function of Denotes a sequential Markov decision process M with delayed rewards The redistribution reward of , then the second-order Markov reward redistribution is expressed as Set the reward-prediction function g f , used to predict the expected cumulative reward of M at the end of a given state-action sequence at time t; the reward-prediction function g f The output is because Included Therefore, we set the difference function Δ f And used to calculate The information carried in f Defined as and The numerical difference between f The state-action sequence of the process is represented as At the same time, g f Must ensure From this we can get: (1) Depend on Calculated and used as Estimates; (2) G f (Δ 0:T ) and g f (Δ 0:t ), which is converted to in, State-Action Sequence The contribution to the expected reward of the prediction is given by Calculated; set up but and There exists because and There is an error between the two, so an additional redistribution reward is set and Finally get 7. The method for controlling air-ground-space network congestion based on deep reinforcement learning according to claim 6, characterized in that The calculation process of the reward redistribution module includes the following steps: For a state-action sequence Indicates the expected feedback before being executed. It indicates the expected feedback after execution. and The numerical difference between The contribution of GRU-based network is f To make predictions, during the training process, As input, As a label, Redistributed to each cycle, it eventually forms a sequential Markov decision process with delayed rewards that can be solved by the DRL method The specific steps include: Input: state-action sequence pairs Feedback prediction function g f , the difference function Δ f , 0≤t≤T, f∈F; Output: state-action sequence Contribution to the predicted expected reward A. Calculation B. Calculation C. Calculation get D. Calculation get E. Return calculation results and 8. The method for controlling air-ground network congestion based on deep reinforcement learning according to claim 7, characterized in that Correct the calculation process of the reward redistribution module, including the following steps: Will The expected value of is expressed as the true value, and we get Divide each round into several blocks of length L SAS The cumulative reward of each short sequence will be redistributed to each action; the redistribution correction method is used to ensure Established; The specific steps include: Input: Feedback prediction function 0≤t≤T, f∈F, cumulative reward Output: Redistribution of rewards a. Calculation b. Calculate the uncorrected redistributed rewards for where t≠0, and c. Calculate the average error d. Calculate the revised redistribution reward for e. Return 9. The method for controlling air-ground network congestion based on deep reinforcement learning according to claim 8, characterized in that The optimization strategy module for constructing the decision-making process in step S4 specifically includes the following steps: The feature extraction network first takes the states of all activity flows in each cycle as input and outputs a feature vector Since most of the L rp rounds, so at most L rp feature vectors; the global state sequence is expressed as Then, The input is fed into another representation network to generate a global feature vector containing global information. Finally, each agent uses the global feature vector to generate actions. A GRU-based neural network is used as the representation network. Each agent shares a global feature vector; the actor network parameters The critic network parameters are used to output the action probability of sub-flow f based on the current state. Used to update and feed back to the actor network by predicting future rewards; Based on the feedback behavior, the actor network can continuously adjust the probability of future action selection, and repeat the feedback process in each learning iteration to improve the strategy; In order to avoid fluctuations in the training process that are greater than the set range and to ensure the stability of the learning process, the following formula is used to update the neural network parameters: In the formula is the clipping loss function, which is used to limit the amplitude of strategy update; represents the expected value of all rounds; is the ratio between the new and old strategies, and For the new strategy in state Take action The probability of For the old policy in state Take action probability; is the advantage function of the action at time t, which is used to measure the advantage of the action at time t relative to the average action is an intermediate function, and ε is a set positive number; The generalized advantage estimator is used to calculate Expressed as Where γ is the discount factor used to reduce the weight of future returns, λ is the parameter that controls the trade-off between bias and variance, T is the total number of periods, is the Bellman residual term and V() is the approximate value function; The total loss is calculated as In the formula is the value function loss, which is used to measure the approximation between the value function and the expected return at time t; H is the entropy function of the strategy that promotes exploration by hindering deterministic strategies at each time t; c1 is the first weight coefficient; c2 is the second weight coefficient; R represents a neural network based on GRU; During the optimization process, all logarithmic probability sequences are used to output action sequences to ensure fairness; the individual logarithms associated with each previously selected action are modified to a set value to prevent the softmax function from generating selected actions for the logarithms; the gradients of the logarithms of invalid operations are set to zero; the softmax function is used to output actions based on the modified logarithms, expressed as a f,j =softmax(L f ) Where a f,j is the jth action of subflow f; L f is the modified logarithmic probability sequence of substream f, and L f = {l f,1 ,...,l f,j ,...,l f,J }, J is the number of actions that can be selected by subflow f; σ is the set negative reward; Subflow f must select an action from each of the following two types of action spaces: In the formula is the sending rate increment of sub-flow f at time t; A negative value indicates that the sending rate of sub-flow f is reduced. A positive value indicates that the sending rate of sub-flow f is increased. 0 indicates that the same sending rate is caused for sub-flow f; N is the maximum number of repetitions in a data packet; Δsr is the degree of change in the sending rate; is the packet repetition factor of subflow f at time t; Therefore, the action Converted from the following formula The action space expression for controlling the sending rate is: Through the reward redistribution mechanism, it is possible to assume that the action feedback in each sequence will be L after the end of the sequence. SAS Return at any time.

10. The air-ground-space network congestion control method based on deep reinforcement learning according to claim 9, characterized in that The air-ground-space network congestion control method based on deep reinforcement learning, wherein the training process includes the following steps: Input: maximum training round E, training cycle length T, algorithm weight λ, discount factor γ, batch size N batch ; Output: Actor network parameters Critic Network Parameters Characterizing Network Parameters (1) Initialize the actor network parameters using random parameters Critic Network Parameters and characterize network parameters (2) Initialize replay buffer B f ; (3) Initialize an Ornstein-Uhlenbeck random process for exploratory actions; (4) At each time t, collect the air-space-ground integrated network status, network layer status, and transport layer status of all flows; (5) Use the state prediction module to predict the latest chapter and obtain the global state s t ; (6) Through the GRU neural network R from the nearest L rp The global state of the segment obtains the feature vector h t ; (7) Get an action based on the strategy function π() (8) Action-based Generate actions using a random process (9) Execute action And observe the current utility function; (10) Store the global state s t , eigenvector h t ,action and the corresponding utility function value; (11) Calculation (12) Calculate the cumulative reward of each sequence based on the utility function value; (13) Through the calculation process of the reward redistribution module, the accumulated rewards are redistributed to obtain (14) Transfer process Store to buffer B f ; (15) Extract N from the replay buffer batch samples and divided into K batches; (16) In each small batch, perform the following steps (17) to (20) in sequence; (17) Obtain the feature vector h through the representation network R k for (18) Determine the action based on the strategy function π() for (19) Calculation and (20) Based on total loss Update actor network parameters Critic Network Parameters and characterize network parameters (21) Return the final actor network parameters Critic Network Parameters and characterize network parameters

Images