Satellite network congestion control method based on deep reinforcement learning for XQUIC protocol

Abstract

The satellite network congestion control method based on deep reinforcement learning for the XQUIC protocol involves the field of network transmission congestion control. It solves the problems of heterogeneous training environment, difficult method convergence, and low parallel training efficiency caused by the high latency and high dynamics of satellite networks in existing methods. The method of the present invention designs a new sampling interaction mechanism minRTT‑100 sampling mechanism and adaptive action space for the high latency and high dynamic characteristics of satellite networks, which ensures the convergence performance of the congestion control method while balancing the size of the action space coverage, making the method more sensitive. An asynchronous distributed parallel training architecture is also proposed to solve the problems of low sampling efficiency, unbalanced distribution of sampling data, and limited single-node network simulation capabilities in high-latency and high-dynamic satellite network environments through parallel training.

Description

Technical Field

[0001] The present invention relates to the field of network transmission congestion control, and in particular to a satellite network congestion control method based on deep reinforcement learning for the XQUIC (Alibaba Cloud self-developed standardized protocol library) protocol. Background Art

[0002] Satellite network topology changes dynamically, links are interrupted intermittently, and are highly dynamic and have large latency. This emerging business scenario places more complex and diverse demands on network transmission quality. Congestion control is a key factor affecting the quality of data stream transmission, and heuristic mechanism design has gradually failed to adapt to highly dynamic networks. Satellite networks with high latency and high dynamics require more adaptive congestion control. On the other hand, the TCP protocol has problems such as high handshake delay and head-of-line (HOL) blocking. To solve these problems, Google proposed the Quick UDP Internet Connection Protocol (QUIC), which combines the designs of TCP, TLS, and http2, and implements secure and reliable transmission based on UDP, supports 0RTT handshakes and efficient multiplexing, and achieves better transmission performance than TCP, but its congestion control is still based on heuristic mechanisms, so it also cannot meet the needs of satellite networks. There is currently no data-driven congestion control method for satellite network scenarios and the QUIC protocol.

[0003] In recent years, reinforcement learning has shown strong adaptability and generalization in the field of decision control, such as game decision making. Using data-driven methods such as reinforcement learning to study congestion control with strong adaptability and good control performance has become a new idea. However, most of the existing work is designed for special ground network environments, lacking a general data-driven congestion control framework based on real protocols, especially lacking solutions for satellite network environments. RemyCC and Indigo are congestion control methods based on offline learning. They calculate the network utility brought by the action by performing different actions in different states of the network. PCC is a method based on online machine learning. This method evaluates the current increase and decrease of network utility by increasing and decreasing the sending window and sending rate. PCC-Vivace is a variant of PCC. It further combines online convex optimization theory to add a more sophisticated rate control mechanism and achieve faster and better convergence performance. However, these methods do not consider the characteristics of dynamically changing network environments in the design of action space and reward function, and have certain limitations.

[0004] To address the above problems, the present invention proposes a congestion control method based on deep reinforcement learning, and designs an asynchronous sampling and action interaction mechanism, as well as an asynchronous distributed parallel training method, to address the problems of heterogeneous training environment, difficult method convergence, and low efficiency of parallel training caused by the high latency and high dynamics of satellite networks. Summary of the invention

[0005] In order to solve the problems of high latency and high dynamics of satellite networks caused by existing methods, such as heterogeneous training environment, difficult method convergence, low efficiency of parallel training, etc., the present invention provides a satellite network congestion control method based on deep reinforcement learning for the XQUIC protocol.

[0006] A satellite network congestion control method based on deep reinforcement learning for the XQUIC protocol is implemented by the following steps:

[0007] Step 1: Start the xquic flow, and use the identifier CID of the xquic flow to build a state communication channel and an action communication channel;

[0008] Step 2: Obtain satellite network environment status information based on time interval sampling; specifically:

[0009] Determine whether the minimum round-trip delay minRTT of the current link is greater than 100ms. If so, perform time interval sampling according to the sum of 100ms and the minimum round-trip delay minRTT; otherwise, perform time interval sampling according to twice the minimum round-trip delay minRTT;

[0010] Step 3: sending the environmental status information to a status communication channel, and reading the environmental status information from the status communication channel as an input of a function module, wherein the function module outputs a corresponding action according to the environmental status information;

[0011] Step 4: Send the corresponding action output in step 3 to the action communication channel. The executor adjusts the congestion window cwnd according to the direction and size of the corresponding action, samples the change of the network environment state in the next sampling period, and transmits the network environment state information to the functional module.

[0012] Step 5: The functional module calculates the reward value R according to the network environment status information; and introduces the advantage function to guide the strategy update to achieve satellite network congestion control.

[0013] Beneficial effects of the present invention:

[0014] The congestion control method described in the present invention designs a new sampling interaction mechanism minRTT-100 sampling mechanism and an adaptive action space for the high-latency and high-dynamic characteristics of satellite networks, which ensures the convergence performance of the congestion control method while balancing the size of the action space coverage, making the method more sensitive.

[0015] The present invention designs a satellite network simulation scenario, which is composed of some satellites of the Iridium constellation and the Beidou constellation. The inter-satellite distance is calculated through orbital data, and then the propagation delay of the link is calculated, and then the end-to-end round-trip delay is obtained in combination with the end-to-end path. At the same time, in order to ensure the diversity of the training environment, two satellites are first randomly selected as the connection ends, and then the route of each time slot is calculated through the topology simulation program, and then the RTT change sequence is obtained, and the random generation range of bandwidth and packet loss rate is set, which are 5-200Mbps and 0-10%, respectively, to simulate the dynamics of the satellite network. The single-stream performance of the method proposed in the present invention was tested using measureFramework, and an xquic stream transmission task with a duration of 60 seconds was performed on the link. The round-trip transmission delay of the link constantly changes with the change of the relative position of the satellite. The experimental statistical results show that the congestion control method proposed in the present invention achieves higher throughput than the PCC method, and outperforms several other congestion methods in terms of delay control.

[0016] In the method of the present invention, an asynchronous distributed parallel training architecture is also proposed, which solves the problems of low sampling efficiency, unbalanced distribution of sampling data, and limited single-node network simulation capabilities in high-latency and high-dynamic satellite network environments through parallel training. The present invention experimentally analyzes the asynchronous distributed parallel training scheme, and the results show that asynchronous distributed parallel training achieves a higher training throughput than the synchronous scheme, with the former throughput being about 3 to 4 times that of the latter. Statistics of the interactive experiments of the environment during the training process were conducted, and the average single-step model inference time was about 2.1ms, and the average single-step sampling time was 0.5ms, indicating that the time overhead introduced by the interactive execution is very small. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is an architecture diagram of the satellite network congestion control method based on deep reinforcement learning for the XQUIC protocol described in the present invention;

[0018] Figure 2 A swim lane diagram of the satellite network congestion control method based on deep reinforcement learning for the XQUIC protocol according to the present invention;

[0019] Figure 3 Switching throughput variation graph for satellite network link;

[0020] Figure 4 The throughput and round-trip delay comparison chart of different congestion control methods;

[0021] Figure 5 This is a diagram of the asynchronous distributed parallel training architecture.

[0022] Figure 6 This is a graph showing how training throughput changes with the number of training steps;

[0023] Figure 7 This is a statistical diagram of the interaction time in asynchronous distributed parallel training. DETAILED DESCRIPTION

[0024] Specific implementation method 1. Combination Figures 1 to 4 The present embodiment describes a satellite network congestion control method based on deep reinforcement learning for the XQUIC protocol. The method includes two core components: an actuator and a functional module, such as Figure 1 shown.

[0025] The executor is an XQUIC-based congestion module that collects stream status information from the QUIC data stream and executes the congestion control actions issued.

[0026] The functional modules include the openAIGym interface that integrates the training module and the satellite network simulation training environment, the training module and the satellite network simulation training environment established using the open source deep reinforcement learning library, the deployment module that deploys the trained model to the actual network environment, and the visualization module that provides visualization functions. Each module interacts with the actuator through the communication channel of the publish-subscribe mechanism (including the state communication channel and the action communication channel).

[0027] The deep reinforcement learning method used in the training module includes two neural networks: a policy network and a value network. The policy network is a deep neural network that receives the state of the environment as input and outputs the probability distribution of taking a certain action in a given state. The value network is used to estimate the value function, which includes the state value function V π (s t and the state-action value function Q π (s t ,a t ) and calculates the advantage function based on the reward value and the value function. During training, the agent is an entity that performs actions, interacts with the environment, performs actions, and collects experience data. Based on the collected data, the value network calculates the advantage function of the state. The policy network uses this advantage function to calculate the gradient to be updated, and uses proximal policy optimization to ensure that the amplitude of each update is limited to improve the stability of the congestion control method.

[0028] When the xquic flow starts, the executor periodically collects flow status information and sends it to the state communication channel. The functional module subscribes to the flow communication channel, receives the flow status information and passes the corresponding action back to the specified action communication channel. The executor obtains the congestion action from the corresponding action communication channel and executes it.

[0029] like Figure 2 As shown, the method described in this embodiment is implemented by the following steps:

[0030] Step 1: Start the xquic stream and initialize the neural network and hyperparameters (including learning rate, number of training iterations, etc.), and use the CID (stream identifier) ​​of the stream to build the state communication channel and action communication channel.

[0031] Step 2: The executor obtains the satellite network environment status by sampling at intervals. The status information includes the congestion window cwnd, the sending rate pacing_rate, etc., and determines the current minimum round-trip delay (minRTT). If minRTT is less than or equal to 100ms, execute step 21. If the current minRTT is greater than 100ms, execute step 22.

[0032] Step 21: When the minRTT of the link is less than or equal to 100ms, sampling is performed at 2 times the minRTT.

[0033] Step 22: When minRTT is greater than 100ms, sample according to 100ms+minRTT.

[0034] Step 3: The executor sends the collected satellite network environment status to the state communication channel of the stream.

[0035] Step 4: The functional module reads the satellite network environment status from the status communication channel and outputs corresponding actions according to the satellite network environment status.

[0036] In this embodiment, the training module in the functional module can realize three action modes: multiplicative action, discrete action and adaptive action;

[0037] The multiplicative action changes the sending congestion window or sending rate by predicting a parameter α as a multiplier:

[0038] CWND new =CWND old ·a(0.5≤a≤3)

[0039] Where CWND new is the size of the new congestion window, CWND old is the size of the old congestion window;

[0040] The discrete linear actions set seven window change actions:

[0041] A t ={-10,-3,-1,0,1,3,10}

[0042] CWND neW =CWNDD old +A t

[0043] In this embodiment, an adaptive action mode is adopted, and the adaptive action space consists of two parts: action scale and action direction. The action scale introduces the control variable uc of increasing the congestion window, the smoothing factor us of increasing the congestion window, the control variable dc of decreasing the congestion window, and the smoothing factor ds of decreasing the congestion window to calculate the action scale, and their initial values ​​are all set to 5, and the exponential moving average (EMA) formula is introduced to calculate the change of the congestion window;

[0044]

[0045] Where aim is the target value, i.e. the new data point or the latest value that we want to integrate into the average value; value is the current EMA value, i.e. the exponential moving average of the previous moment; rate is the smoothing factor, which is used to determine the impact of the new data point on EMA; the smaller the value, the greater the impact of the new data point on EMA.

[0046] The action direction is related to the sign of the congestion window change, and the values ​​are increase, decrease and unchanged. When the action direction is increase, execute step 41, when the action direction is decrease, execute step 42, and when the action direction is unchanged, execute step 43;

[0047] Step 41: Update uc, us, dc through the exponential moving average (EMA) formula and calculate the size of the congestion window adjustment action cwnd to be performed change ;

[0048] uc=EMA(10,uc,16);

[0049] us=EMA(0.2,us,16);

[0050] dc=EMA(5,dc,4);

[0051]

[0052] CWND new =CWND old +cwnd change ;

[0053] Step 42: Update dc, ds, uc through the exponential moving average (EMA) formula and calculate the size of the congestion window adjustment action cwnd to be performed change ;

[0054] dc = EMA(10, dc, 16);

[0055] ds = EMA(0.2, ds, 16);

[0056] ucEMA(5,uc,4);

[0057] cwnd change = -dc / ds;

[0058] CWND new =CWND old +Cwnd change ;;

[0059] Step 43: Update uc, us, dc, ds through the exponential moving average (EMA) formula and calculate the size of the congestion window adjustment action cwnd to be performed change ;

[0060] uc=EMA(5,uc,4);

[0061] us=EMA(5,us,4);

[0062] dc=EMA(5,dc,4);

[0063] ds = EMA(5, ds, 4);

[0064] cwnd change =0;

[0065] CWND new =CWND old +cwnd change ;

[0066] Step 5: The function module transmits the change direction and size of the congestion window cwnd to the executor, and the executor adjusts cwnd according to the change direction and size of the congestion window cwnd, samples the change of the network environment state in the next sampling period, and transmits the network environment state information to the function module;

[0067] The functional module calculates the reward value R according to the network environment status information;

[0068] R=log(λ)-δlog(d)-β·l

[0069] Among them, λ is the sending rate of the sender; d is the RTT measured by the end; δ is a hyperparameter used to balance the pursuit of throughput and delay control capability; β is used to balance the penalty for packet loss; l is the packet loss change rate

[0070] The calculation formula is as follows:

[0071] l=(lost_pkts-lost_pkts) / lost_pkts′

[0072] Wherein, lost_pkts represents the number of lost packets in the previous sampling period; lost_pkts′ represents the number of lost packets in the current sampling period.

[0073] Step 6: The value network calculates the advantage function based on the reward value R and the value function. Specifically:

[0074] Define the action space as A, the state space as S, and the policy function π(a|s)=P(S=s|A=a) to represent the probability of taking action a in the state space S. Use a neural network instead of the policy function, then the policy function is π(a|θ, where θ is the parameter of the neural network.

[0075] Consider a state transition of a decision trajectory starting from time t. t Select Action A t The state transition starting at time t is:

[0076] S t ,A t ,S t+1 ,A t+1 …

[0077] Let U t is the sum of all reward values ​​R starting at time t, and action a is performed at time t t The state-action value function Q π (s t ,a t )for:

[0078] Q π (s,a t )=E[U t |S t =s t ,A t =a t ]

[0079] Status t The state value of is the expectation of the action value function to the policy function in this state; that is, the state value function V π (s t );

[0080]

[0081] The advantage function is used to measure the superiority of each action relative to the current strategy. The advantage function is:

[0082] A π (s t ,a t )=Q π (s t ,a t )-V π (s t )

[0083] Step 7: The policy network uses the advantage function to update parameters such as action probability distribution, and uses proximal policy optimization to ensure that the magnitude of each update is limited, thereby improving the stability of the congestion control method.

[0084] Combination Figure 3 and Figure 4 To illustrate this implementation mode, a satellite network simulation scenario is designed in this implementation mode, which is composed of some satellites of the Iridium constellation and the Beidou constellation. The inter-satellite distance is calculated through orbital data, and then the propagation delay of the link is calculated, and then the end-to-end round-trip delay is obtained in combination with the end-to-end path. At the same time, in order to ensure the diversity of the training environment, two satellites are first randomly selected as the connection ends, and then the route of each time slot is calculated through the topology simulation program, and then the RTT change sequence is obtained, and the random generation range of bandwidth and packet loss rate is set to 5-200Mbps and 0-10%, respectively, to simulate the dynamics of the satellite network. The single-stream performance of the method proposed in this implementation mode is tested using measureFramework, as shown in the figure. Figure 3 As shown in the figure, an xquic stream transmission task lasting 60 seconds was performed on the link. The link experienced two switches, the first switch of about 15 seconds, switching from a 20Mbps, 113ms link to a 15Mbps, 132ms link; and the second switch of about 35 seconds, switching from a 20Mbps, 132ms link to a 30Mbps, 110ms link. The round-trip transmission delay of the link changes continuously with the change of the relative position of the satellite. The experimental statistical results are shown in Figure 4 As shown, the congestion control method proposed in this embodiment achieves higher throughput than the PCC method, and performs better than several other congestion methods in terms of delay control.

[0085] Specific implementation method 2: Combination Figure 5-Figure 7 This embodiment is described. This embodiment is an application example of the satellite network congestion control method based on deep reinforcement learning for the XQUIC protocol described in Specific Implementation Method 1: the method described in Specific Implementation Method 1 is subjected to asynchronous distributed parallel training to solve the problems faced by deep reinforcement learning for congestion control training in a high-latency and high-dynamic satellite network environment, such as low sampling efficiency due to large sampling intervals, uneven distribution of sampling data as the training environment changes, and limited single-node network simulation capabilities.

[0086] The asynchronous distributed parallel training method described in this embodiment is implemented by a training system consisting of a satellite network simulation training environment and a learning node;

[0087] The satellite network simulation training environment: builds a simulated satellite network based on satellite orbit data, starts multiple xquic flows in parallel in the simulated network, dynamically adjusts end-to-end transmission parameters such as bandwidth, delay, and packet loss rate of each flow according to changes in network topology, and periodically collects the state of the flow and sends it to a publish-subscribe communication channel.

[0088] The learning node includes a gym instance group, an inference node group, a policy network and a value network. Each gym instance in the gym instance group corresponds to an xquic stream in the satellite network simulation training environment. On the one hand, it obtains the state of the stream from the publish-subscribe communication channel and forwards the inference node group. At the same time, it obtains the action to be executed from the inference node group and sends it to the stream through the publish-subscribe communication channel; on the other hand, it calculates the reward of the current action, and combines the stream state, action and reward into trajectory sampling. Then, the trajectory sampling from multiple gym instances is batched, i.e., batch trajectory sampling, and sent to the value network to evaluate the quality of action execution. The policy network performs gradient update according to the evaluation results of the value network, and then synchronously updates the policy network instances of each inference node.

[0089] like Figure 5 As shown, the specific training process is as follows:

[0090] Step A: Start the satellite network simulation training environment, set the link bandwidth, delay, packet loss rate and other parameters according to the satellite orbit data, start multiple end nodes at the same time, and associate the end nodes with the switching nodes in the network.

[0091] Step B: Start multiple xquic flows in parallel on multiple end nodes, and periodically send the flow status, i.e., flow round-trip delay, throughput and other information, to the gym instance through the publish-subscribe communication channel.

[0092] Step C: The gym instance obtains the stream state from the publish-subscribe communication channel and sends it to the inference node group.

[0093] Step D: The gym instance obtains the action to be executed from the inference node group and sends it to the end node of the corresponding stream through the publish-subscribe communication channel, while calculating the reward value of the current action.

[0094] Step E: The gym instance group collects the sampled trajectories of the gym instances, including flow states, actions, and reward values, and sends the collected sampled trajectories to the value network in batches.

[0095] Step F: The value network combines the received batch sample trajectories into a data set, splits it into small batches, and calculates how well the current action is executed to optimize the policy network.

[0096] Step G: Use stochastic gradient descent to update the policy network parameters, such as action probability distribution.

[0097] Step H: Send the updated policy network parameters to each inference node and return to step C.

[0098] Combination Figure 6 and Figure 7 To explain this embodiment, Figure 6 In this embodiment, an experimental analysis of the asynchronous distributed parallel training scheme is carried out, and the results show that the asynchronous distributed parallel training achieves a higher training throughput than the synchronous scheme, and the throughput of the former is about 3 to 4 times that of the latter. Figure 7 In the paper, the interactive experiments of the environment during training were statistically analyzed. The average single-step model inference time was about 2.1ms, and the average single-step sampling time was 0.5ms, indicating that the time overhead introduced by interactive execution is very small.

[0099] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0100] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A satellite network congestion control method based on deep reinforcement learning for the XQUIC protocol, characterized by: The method is implemented by the following steps: Step 1: Start the xquic flow, and use the identifier CID of the xquic flow to build a state communication channel and an action communication channel; Step 2: Obtain satellite network environment status information based on time interval sampling; specifically: Determine whether the minimum round-trip delay minRTT of the current link is greater than 100ms. If so, perform time interval sampling based on the sum of 100ms and the minimum round-trip delay minRTT; Otherwise, the sampling interval is twice the minimum round-trip delay minRTT; Step 3: sending the environmental status information to a status communication channel, and reading the environmental status information from the status communication channel as an input of a function module, wherein the function module outputs a corresponding action according to the environmental status information; The adaptive action space consists of two parts: action scale and action direction; the action scale introduces the control variable uc for increasing the congestion window, the smoothing factor us for increasing the congestion window, the control variable dc for decreasing the congestion window, and the smoothing factor ds for decreasing the congestion window to calculate the action scale, and their initial values ​​are all set to 5, and the change of the congestion window is calculated by the exponential moving average EMA formula; The action direction is related to the sign of the congestion window change, and the values ​​are increase, decrease and unchanged; when the action direction is increase, execute step 31, when the action direction is decrease, execute step 32, when the action direction is unchanged, execute step 33; Step 31: Update the control variable uc for increasing the congestion window through the EMA formula, increase the smoothing factor us of the congestion window, reduce the control variable dc of the congestion window, and calculate the size of the congestion window adjustment action cwnd to be executed change ; uc=EMA(10,uc,16); us=EMA(0.2,us,16); dc=EMA(5,dc,4); CROWD new =SHOW old +cond change ; Where CWND new is the size of the new congestion window, CWND old is the size of the old congestion window; Step 32: Update the control variable dc that reduces the congestion window through the EMA formula, reduce the smoothing factor ds of the congestion window, increase the control variable uc of the congestion window, and calculate the size of the congestion window adjustment action cwnd to be executed change ; dc=EMA ( 10,dc,16 ) ; ds=EMA ( 0.2,ds,16 ) ; uc=EMA ( 5,high,4 ) ; cwnd change =-dc / ds; CROWD new =SHOW old +cond change ; Step 3: Update the control variable uc for increasing the congestion window through the EMA formula, increase the smoothing factor us of the congestion window, reduce the control variable dc for decreasing the congestion window, reduce the smoothing factor ds of the congestion window, and calculate the size cwnd of the congestion window adjustment action to be executed. change ; uc=EMA ( 5,high,4 ) ; us=EMA ( 5,us,4 ) ; dc=EMA ( 5,dc,4 ) ; ds=EMA ( 5,ds,4 ) ; culvert change =0; CROWD new =SHOW old +cond change ; Step 4: Send the corresponding action output in step 3 to the action communication channel. The executor adjusts the congestion window cwnd according to the direction and size of the corresponding action, samples the change of the network environment state in the next sampling period, and transmits the network environment state information to the functional module. Step 5: The functional module calculates the reward value R according to the network environment status information; and introduces the advantage function to guide the strategy update to achieve satellite network congestion control.

2. The satellite network congestion control method based on deep reinforcement learning for the XQUIC protocol according to claim 1, characterized in that: The functional modules include a training module and a satellite network simulation training environment established using an open source deep reinforcement learning library, an openAI Gym interface that integrates the training module and the satellite network simulation training environment, a deployment module that deploys the trained model to the actual satellite network environment, and a visualization module that provides visualization functions; each module interacts with the actuator through a communication channel of a publish-subscribe mechanism.

3. The satellite network congestion control method based on deep reinforcement learning for the XQUIC protocol according to claim 2, characterized in that: The training module includes a strategy network and a value network. The strategy network is used to receive the network information environment state as input and output the probability distribution of taking a certain action under a given state; the value network is used to estimate the state value function or the state-action value function and calculate the advantage function based on the reward value and the value function.

4. The satellite network congestion control method based on deep reinforcement learning for the XQUIC protocol according to claim 3, characterized in that: The reward value R is expressed as follows: R=log(λ)-δlog(d)-β·l In the formula, λ is the sending rate of the sender; d is the RTT measured by the end; δ is a hyperparameter used to balance the pursuit of throughput and delay control capability; β is used to balance the penalty for packet loss; l is the packet loss change rate. The calculation formula is as follows: l=(lost_pkts-lost_pkts') / lost_pkts' Where lost_pkts is the number of lost packets in the previous sampling period, and lost_pkts' is the number of lost packets in the current sampling period.

5. The satellite network congestion control method based on deep reinforcement learning for the XQUIC protocol according to any one of claims 1 to 4, characterized in that: The satellite network congestion control method is trained asynchronously and distributedly in parallel. The specific training method is implemented by a training system consisting of a satellite network simulation training environment and a learning node. The specific steps are as follows: Step A: Start the satellite network simulation training environment, set the link bandwidth, delay, and packet loss rate according to the satellite orbit data, start multiple end nodes at the same time, and associate the end nodes with the switching nodes in the network; Step B: Start multiple xquic streams in parallel on multiple end nodes, and periodically send the stream status to the gym instance through the publish-subscribe communication channel; Step C: The gym instance obtains the stream state from the publish-subscribe communication channel and sends it to the inference node group; Step D: The gym instance obtains the action to be executed from the inference node group and sends it to the end node of the corresponding stream through the publish-subscribe communication channel, while calculating the reward value of the current action; Step E: The gym instance group collects the sampled trajectories of the gym instances, including flow states, actions, and reward values, and sends the collected sampled trajectories to the value network in batches; Step F: The value network combines the received batch sample trajectories into a data set, splits the data set into batches, and calculates the degree of execution of the current action to optimize the policy network; Step G: Update the policy network parameters using stochastic gradient descent, send the updated policy network parameters to each inference node, and return to step C.

6. The satellite network congestion control method based on deep reinforcement learning for the XQUIC protocol according to claim 5, characterized in that: The satellite network simulation training environment builds a simulated satellite network based on satellite orbit data, starts multiple xquic flows in parallel in the simulated network, dynamically adjusts the bandwidth, delay, and packet loss rate of each flow according to changes in network topology, and periodically collects the state of the flow and sends it to the publish-subscribe communication channel; The learning node includes a gym instance group, an inference node group, a policy network and a value network; each gym instance in the gym instance group corresponds to an xquic stream in the satellite network simulation training environment, obtains the state of the stream from the publish-subscribe communication channel and forwards the inference node group, and obtains the action to be executed from the inference node group and sends it to the stream through the publish-subscribe communication channel; And calculate the reward of the current action, combine the flow state, action and reward into trajectory sampling, and then batch sample the trajectories of multiple gym instances and send them to the value network to evaluate the execution of the action; The policy network performs gradient updates based on the evaluation results of the value network, and then synchronously updates the policy network instances of each inference node.

Images