Satellite network congestion control method and control system based on Q-learning for TCP protocol

Abstract

A Q-learning-based satellite network congestion control method and control system for TCP protocol, involving the field of network transmission congestion control, solves the problem that the existing reinforcement learning congestion control method is not suitable for satellite network scenario applications. The method of the present invention is implemented through a control system, which includes an actuator, a transponder and a functional module; the method implements a user-mode interface suitable for reinforcement learning algorithms and a corresponding openAI Gym standard reinforcement learning environment, and provides a satellite network simulation training environment and a dynamic visualization component. At the same time, taking into account the limited computing power on the satellite end and the high latency of the satellite network, a lightweight data-driven TCP congestion control method based on Q-learning and tile encoding is further designed and implemented. This method designs a new discrete action space, which achieves model lightweight while ensuring the convergence of the algorithm.

Description

Technical Field

[0001] The present invention relates to the field of network transmission congestion control, and in particular to a satellite network congestion control method and control system based on Q-learning and oriented to TCP protocol. Background Art

[0002] Compared with traditional terrestrial networks, satellite networks have the characteristics of high latency and dynamic periodic changes. This emerging business scenario also puts forward more complex and diversified requirements for network transmission quality. As a key factor affecting the quality of data stream transmission, the congestion control algorithm has gradually been unable to adapt to the complex and changing network environment based on heuristic algorithm design. Reinforcement learning has shown strong adaptability and generalization capabilities in the field of decision control. Using data-driven algorithms such as reinforcement learning to study congestion control with strong adaptability and good control performance has become a new idea. Data-driven algorithms based on learning can effectively enhance the adaptability of congestion control algorithms and achieve better congestion control performance. However, there are still many problems in applying the current learning-based methods in terrestrial networks to satellite networks.

[0003] The high latency and dynamic periodic changes of satellite networks pose new challenges to the design of learning-based methods. Onboard communication equipment is usually required to be small, light, and low power. The allocation of power resources between modules is strictly restricted, which requires that the complexity of the protocols, algorithms, and strategies involved in the networking process should be as low as possible. Congestion control is a high-frequency control problem, and high algorithm complexity will introduce performance problems. In addition, dynamic satellite networks also increase the heterogeneity of the network environment, resulting in low training efficiency and difficulty in convergence of reinforcement learning algorithms.

[0004] Existing data-driven congestion control methods, such as the machine learning-based congestion control method (RemyCC) and the experience-based congestion control method (Indigo), use non-real TCP protocols to design congestion generation schemes for offline training in ground network environments. The deep reinforcement learning congestion control method (Aurora) and the imitation reinforcement learning congestion control method (Eagle) use Python to implement a gym environment to simulate the behavior of the transmission protocol for training, but are not based on real transmission protocols. The data-driven solution (Owl) method for SDN scenarios is not suitable for satellite network scenarios. The method for data center network environments (Iroko), whose sampling and action mechanisms are implemented at the granularity of global optimization, lacks the ability to control congestion for end-to-end flows. Based on Facebook's QUIC protocol implementation mvfst, a specific implementation of the mvfst-rl congestion control method is designed. None of these existing works have solutions designed for satellite network scenarios, and their designs are mostly for specific algorithms and cannot be expanded for more extensive research.

[0005] In summary, the present invention designs a lightweight TCP congestion control method based on reinforcement learning for satellite networks. This method implements a user-mode interface suitable for reinforcement learning algorithms and the corresponding openAI Gym standard reinforcement learning environment, and provides a satellite network simulation training environment and a dynamic visualization component based on Streamlit. At the same time, taking into account the limited computing power on the satellite side and the high latency of the satellite network, a lightweight data-driven TCP congestion control method based on Q-learning and tile encoding is further designed and implemented. This method designs a new discrete action space, which achieves lightweight model while ensuring the convergence of the algorithm. Summary of the invention

[0006] Aiming at the problem that the existing reinforcement learning congestion control method is not suitable for satellite network scenario application, the present invention provides a Q-learning-based satellite network congestion control method and control system for TCP protocol. It realizes the user state interface suitable for reinforcement learning algorithm and the corresponding openAI Gym standard reinforcement learning environment, and provides satellite network simulation training environment and Streamlit-based dynamic visualization components.

[0007] A Q-learning-based satellite network congestion control method for TCP protocol: The method is implemented by the following steps:

[0008] Step 1: Start the TCP flow. The executor increases the congestion window according to the length of the current confirmation data packet and enters the reinforcement learning state.

[0009] Step 2: The executor obtains the satellite network environment status and transmits the network environment status value to the forwarder through netlink;

[0010] Step 3: The forwarder transmits the received status value to the training module through the message middleware;

[0011] Step 4: The Q-learning algorithm of the training module selects the action with the largest q value in the current state according to the state value and the Q table according to the ∈ greedy principle, determines the change direction of the congestion window according to the action with the largest q value, and transmits the change direction of the congestion window and the window action adjustment size to the forwarder;

[0012] Step 5: The forwarder transmits the corresponding information to the space executor through Netlink. The executor samples the network environment status change in the next sampling period according to the change direction of the congestion window and the window action adjustment size, and transmits the status information to the forwarder through Netlink.

[0013] Step 6: The forwarder transmits the status information to the training module through the message middleware;

[0014] Step 7: The training module calculates a reward value R according to the received state information, and updates the q value corresponding to the previous state and action according to the Q-learning algorithm according to the reward value R;

[0015] Step 8: Determine the minimum round-trip time RTT min Is the update time greater than 10s? If so, enter RTT min In the exploration state, the congestion window size is set to 4 for RTT measurement and the process returns to step 2; otherwise, the process returns to step 2.

[0016] The present invention also provides a satellite network congestion control system based on Q-learning for TCP protocol, which is used in the satellite network congestion control method based on Q-learning for TCP protocol; the control system includes an actuator, a repeater and a functional module;

[0017] The executor is used to collect flow status information from the TCP data flow and execute the congestion control action issued;

[0018] The forwarder is used to carry out message communication between the upper-layer functional module and the lower-layer executor, including forwarding of flow status messages and action decision messages; the forwarder uses netlink technology to communicate with the executor through a publish-subscribe mechanism and interact with the upper-layer functional module;

[0019] The functional modules include various components for reinforcement learning, specifically a training module, a satellite network simulation training environment, a visualization module, an openAI Gym interface, and a deployment module; each module subscribes to a message channel, receives a stream state message, and transmits an action control message to a forwarder.

[0020] Beneficial effects of the present invention:

[0021] 1. Aiming at the characteristics of high latency, high packet loss and periodic dynamic changes of links in satellite networks, the present invention designs and implements a satellite network congestion control method for TCP protocol based on Q-learning. A new discrete action space is proposed, which enables the method to achieve fast convergence and precise action control, and achieves high throughput while maintaining low RTT.

[0022] Second, the present invention was trained and verified in a satellite network simulation scenario, and the satellite network scenario used included 75 low-orbit satellites and 3 high-orbit satellites. The inter-satellite distance was calculated through orbital data, and then the propagation delay of the link was calculated, and then the end-to-end round-trip delay was obtained in combination with the end-to-end path. At the same time, in order to ensure the diversity of the training environment, the random generation range of bandwidth and packet loss rate was set, which were 5-200Mbps and 0-10%, respectively. Then, the performance test environment was built using measureFramework to conduct a single-stream transmission experiment on the present invention. The end-to-end path selected in the experiment underwent two switches. It switched from a path of 20Mbps and 113ms to a path of 15Mbps and 132ms in about 15s, and then switched to a path of 30Mbps and 110ms in about 35s. Its round-trip transmission delay is constantly affected by the change in the relative position of the satellite. The experimental results show that the method of the present invention is sensitive, can respond quickly when the path switches, change the sending window in time, and can fully utilize the bandwidth of the path to make it close to the ideal value, so it performs well in a dynamic satellite network environment.

[0023] 3. The method of the present invention was compared with other congestion control methods in the Pantheon platform. The experiment ran two flows on a simulated link, each flow ran for 60 seconds, and the experiment was repeated 10 times. The results show that in terms of RTT, the present invention is close to BBR and Vegas (about 24ms), and significantly lower than PCC-Vivace, Verus, Cubic, Reno and other methods; in terms of throughput, the present invention achieved good throughput performance (12Mbps), which is comparable to other methods; in terms of packet loss rate, the present invention maintained a very low level (less than 1%); in the fairness experiment of five flows sharing a 100Mbps bottleneck link, the present invention also achieved very high fairness (0.995), which is close to BBR and Vegas.

[0024] In summary, the method of the present invention can perform well in dynamic satellite network scenarios, maintain low latency while also achieving good throughput and having very high fairness. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is the architecture diagram of the satellite network congestion control method based on Q-learning for TCP protocol;

[0026] Figure 2 It is a flow chart of the satellite network congestion control method based on Q-learning for TCP protocol according to the present invention;

[0027] Figure 3 This is a diagram showing the throughput variation in dynamic satellite network link scenarios;

[0028] Figure 4 This is a comparison chart of throughput and round-trip transmission delay of different congestion control algorithms;

[0029] Figure 5 The packet loss rate statistics of different congestion control algorithms are shown below;

[0030] Figure 6 The Jain fairness index statistics for different congestion control algorithms. DETAILED DESCRIPTION

[0031] Specific implementation method 1. Combination Figure 1 The present embodiment is described, a Q-learning based satellite network congestion control system for TCP protocol, which consists of three core components: actuator, forwarder and functional module;

[0032] The executor is responsible for collecting flow status information from the TCP data flow and executing the issued congestion control actions, and is implemented in the form of a Linux kernel module.

[0033] The forwarder is responsible for message communication between the upper-layer functional modules and the lower-layer executors, including forwarding of flow status messages and action control messages. The forwarder uses netlink technology to communicate with the executor through a publish-subscribe mechanism, and interacts with various functional modules in the upper-layer functional modules in the system. It is a user space program.

[0034] The functional modules include various components related to reinforcement learning, such as the openAI Gym interface, a training module built using the Q-learning model in the rllib library, a satellite network simulation training environment, a deployment module that deploys the trained model to the actual network environment, and a visualization module that provides data visualization. These modules subscribe to the message communication pipeline, receive stream state messages, and pass action control messages to the forwarder.

[0035] In this implementation, the interaction between the functional module and the forwarder is implemented through a consistent message middleware to ensure seamless interaction between its various components. When a TCP stream is started using a specified congestion control algorithm, the TCP stream registers a congestion control function in the executor containing the congestion control logic. According to the rules defined by the monitoring interval, the current state information of the stream is collected periodically in this implementation, a unique identifier is assigned to the stream as its tag, and this information is passed to the forwarder through the netlink technology. Subsequently, the forwarder deserializes and serializes the stream information and sends it to a dedicated communication pipe associated with the stream, which is uniquely identified by the stream identifier. The functional modules subscribe to their respective stream communication channels, receive relevant stream state information and pass the corresponding response actions back to the communication pipe of the specified action. In the case of multiple streams at the same time, the forwarder ensures communication isolation by directing the stream information to different communication pipes according to the unique identifier of each stream, ensuring effective communication management in scenarios involving multiple concurrent streams.

[0036] In this implementation, a three-dimensional Q table is designed to store the quality of selecting different actions under different states. The first two dimensions represent the fluctuation of the round-trip delay RTT and the loss fluctuation, respectively, and the third dimension represents the action. The action indicates the direction of change of the congestion window cwnd, that is, increase, decrease and unchanged, and the size of the congestion window cwnd is adjusted according to the result calculated by the exponential moving average algorithm (EMA). Each item in the Q table stores the action value q = Q (s, a) of the corresponding state and action index, and the q value is used to characterize the quality of selecting the action in the current state;

[0037] Q(s,a)←(1-α)·Q(s,a)+α·(r+γ·maxQ(s',α))

[0038] Among them, α∈[0,1) represents the learning rate, which indicates how much new information is learned each time; γ∈[0,1) is the discount factor, which indicates the importance of future rewards; r is the immediate reward obtained after taking action a in state s.

[0039] In order to strike a balance between development and exploration during training, this method adopts an ∈-greedy strategy, selects the action with the largest q value in the current Q table as the output with a probability of ∈, and selects a random action with a probability of 1-∈. Then the selected action is executed, interacts with the satellite network simulation training environment to obtain the reward value of the environment feedback, and transfers to the next state s'. As an off-orbit strategy algorithm, Q-learning uses the q value of the action with the maximum q value corresponding to the state s' in the Q table as the future estimated value to be included in the update equation, calculates and uses the new q value after the current selected action to update the Q table.

[0040] Specific implementation method 2: Combination Figures 2 to 6This embodiment is described. This embodiment is implemented by executing the Q-learning-based satellite network congestion control system for TCP protocol described in the first embodiment. The system has three states when in motion: startup state; reinforcement learning state and minimum RTT exploration state. The specific implementation method is implemented by the following steps:

[0041] Step A: Start the TCP stream and directly enter the startup state;

[0042] Step A1: Increase the congestion window cwnd, and the executor accumulates the window according to the length of the current confirmed data packet;

[0043] cwnd new =cwnd old +acked

[0044] Where cwnd new is the calculated new congestion window size, cwnd old is the old congestion window size, and acked indicates the number of confirmed packets.

[0045] Step A2: determine whether it is the first time that packet loss and retransmission occur. If so, enter the reinforcement learning state and execute step A3; otherwise, execute step A1;

[0046] Step A3: The executor obtains the satellite network environment status by sampling at intervals: RTT fluctuation (delay) and loss fluctuation (loss flu );

[0047] delay=RTT-RTT min

[0048] loss flu =loss last -loss ave

[0049] Among them, RTT min The minimum RTT within 10 seconds, RTT is the real-time RTT; loss ave is the average packet loss rate, loss last is the packet loss rate of the previous sampling interval;

[0050] Step A4: The executor transmits the network environment status value sampled in step A3 to the user space forwarder through netlink;

[0051] Step B: The forwarder transmits the sampled state value to the training module through the message middleware;

[0052] Step C: The training module inputs the sampled state value into the Q-learning algorithm. The Q-learning algorithm selects the action with the largest q value in the current state (i.e., the congestion window cwnd increases, decreases, or remains unchanged) according to the ∈ greedy principle based on the state value and the Q table; the direction of change of the congestion window cwnd is determined according to the action with the largest q value; specifically:

[0053] Step C1, introduce four variables required for calculating the congestion window change later: increase the control variable uc of the congestion window, increase the smoothing factor us of the congestion window, decrease the control variable dc of the congestion window, and decrease the smoothing factor ds of the congestion window; set their initial values ​​to 5, and introduce the exponential moving average algorithm (EMA) formula. If the q value indicates that the congestion window cwnd should be increased, execute step C2; if the congestion window cwnd should be reduced, execute step C3; if the congestion window cwnd should remain unchanged, execute step C4; the EMA formula is as follows:

[0054]

[0055] Where aim is the target value, i.e. the new data point or the latest value that we want to integrate into the average value; value is the current EMA value, i.e. the exponential moving average of the previous moment; rate is the smoothing factor, which is used to determine the impact of the new data point on EMA; the smaller the value, the greater the impact of the new data point on EMA.

[0056] Step C2: Update uc, us, dc through the EMA formula and calculate the congestion window cwnd adjustment action size cwnd to be executed change ;

[0057] uc=EMA(10,uc,16);

[0058] us=EMA(0.2,us,16);

[0059] dc=EMA(5,dc,4);

[0060]

[0061] Step C3: Update dc, ds, uc through the EMA formula and calculate the congestion window cwnd adjustment action size cwnd to be executed change ;

[0062] dc=EMA(10,dc,16);

[0063] ds=EMA(0.2,ds,16);

[0064] uc=EMA(5,uc,4);

[0065] cwnd change = -dc / ds;

[0066] Step C4: Update uc, us, dc, ds through the EMA formula and calculate the congestion window cwnd adjustment action size cwnd to be executed change ;

[0067] uc=EMA(5,uc,4);

[0068] us=EMA(5,us,4);

[0069] dc=EMA(5,dc,4);

[0070] ds=EMA(5,ds,4);

[0071] cwnd change =0;

[0072] Step D: passing the change direction and size of cwnd to the forwarder through the message middleware;

[0073] Step E: The forwarder passes the corresponding information to the kernel space executor through the netlink technology;

[0074] Step F, the kernel space executor adjusts the congestion window cwnd according to the direction and size of the congestion window cwnd; the kernel space executor samples the change of the network environment status in the next sampling period, and transmits the status information to the user space forwarder through netlink;

[0075] Step G, the forwarder transmits the information to the training module through the message middleware; the training module calculates the reward value R based on the received information using the following formula;

[0076] R=η·delivered_rate-β·delay

[0077] Where delivered_rate is the effective delivery rate of the sender; delay = RTT-RTT min , represents the delay fluctuation; η, β are hyperparameters used to balance the pursuit of throughput and delay control capability.

[0078] According to the calculated reward value R, the q value corresponding to the previous state and the congestion window adjustment action is updated according to the q-learning algorithm update formula;

[0079]

[0080] Where Q(s,a) is the q value to be updated after the current state s executes the currently selected action a; is the maximum q value that the next state s' can take according to the current Q table, representing the expectation of future rewards, a' is the corresponding action when the next state takes the maximum q value; α∈[0,1) represents the learning rate, indicating how much new information is learned each time; γ∈[0,1) is the discount factor, indicating the importance of future rewards;

[0081] Step H: When RTT min If there is no update for more than 10 seconds, it will enter RTT min In the exploration state, the congestion window size is set to 4 to measure the RTT. After the minimum RTT is found, this state ends and the process returns to step A3.

[0082] Combination Figure 3 This embodiment is explained. The method of this embodiment is trained and verified in a satellite network simulation scenario. The satellite network scenario used includes 75 low-orbit satellites and 3 high-orbit satellites. The inter-satellite distance is calculated by orbital data, and then the propagation delay of the link is calculated, and then the end-to-end round-trip delay is obtained in combination with the end-to-end path. At the same time, in order to ensure the diversity of the training environment, the random generation range of bandwidth and packet loss rate is set, which is 5-200Mbps and 0-10% respectively. Then the performance test environment is built using the measurement framework measureFramework to conduct a single-stream transmission experiment on the present invention. The end-to-end path selected in the experiment has undergone two switches. It switched from a path of 20Mbps and 113ms to a path of 15Mbps and 132ms in about 15s, and switched to a path of 30Mbps and 110ms in about 35s. Its round-trip transmission delay is constantly affected by the change in the relative position of the satellite. The experimental results show that the method of the present invention is sensitive, can respond quickly when the path switches, change the sending window in time, and can fully utilize the bandwidth of the path to make it close to the ideal value, so it performs well in a dynamic satellite network environment.

[0083] Combination Figures 4 to 6This embodiment is explained, and the method of this embodiment is compared with other congestion control methods in the Pantheon platform. The experiment runs two flows on the simulated link, each flow runs for 60s, and the experiment is repeated 10 times. The results show that in terms of RTT, the method of this embodiment is close to the congestion control method based on bottleneck bandwidth and round-trip propagation time (BBR) and Vegas (about 24ms), and is significantly lower than the performance-oriented congestion control method (PCC-Vivace), Verus, Cubic, Reno and other methods; in terms of throughput, the method of this embodiment achieves good throughput performance (12Mbps), which is comparable to other methods; in terms of packet loss rate, the method of this embodiment maintains a very low level (less than 1%); in the fairness experiment of five flows sharing a 100Mbps bottleneck link, the method of this embodiment also achieves very high fairness (0.995), which is close to BBR and Vegas.

[0084] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0085] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A Q-learning-based satellite network congestion control method for TCP protocol, characterized by: The method is implemented by the following steps: Step 1: Start the TCP flow. The executor increases the congestion window according to the length of the current confirmation data packet and enters the reinforcement learning state. Step 2: The executor obtains the satellite network environment status and transmits the network environment status value to the forwarder through netlink; Step 3: The forwarder transmits the received status value to the training module through the message middleware; Step 4: The Q-learning algorithm of the training module selects the action with the largest q value in the current state according to the state value and the Q table according to the ∈ greedy principle, determines the change direction of the congestion window according to the action with the largest q value, and transmits the change direction of the congestion window and the window action adjustment size to the forwarder; The direction of the congestion window change is determined based on the action with the largest q value. The specific process is as follows: Step 41: Determine whether the congestion window cwnd increases according to the action with the largest q value. If yes, execute step 42; otherwise, execute step 43; Step 42: Update uc, us, dc through the exponential moving average EMA formula and calculate the size of the congestion window adjustment action cwnd to be executed change ; uc=EMA(10,uc,16); us=EMA(0.2,us,16); dc=EMA(5,dc,4); In the formula, uc is the control variable for increasing the congestion window, us is the smoothing factor for increasing the congestion window, and dc is the control variable for reducing the congestion window; Step 43: Determine whether the congestion window cwnd is reduced. If so, execute step 44; otherwise, execute step 45; Step 44: Update dc, ds, uc through the exponential moving average EMA formula and calculate the size of the congestion window adjustment action cwnd to be performed change ; dc=EMA(10,dc,16); ds=EMA(0.2,ds,16); uc=EMA(5,uc,4); cwnd change =-dc / ds; Where ds is the smoothing factor that reduces the congestion window; Step 4 and 5: Update uc, us, dc, ds through the exponential moving average EMA formula and calculate the size of the congestion window adjustment action cwnd to be executed change ; uc=EMA(5,uc,4); us=EMA(5,us,4); dc=EMA(5,dc,4); ds=EMA(5,ds,4); culvert change =0; Step 5: The forwarder transmits the corresponding information to the space executor through Netlink. The executor samples the network environment status change in the next sampling period according to the change direction of the congestion window and the window action adjustment size, and transmits the status information to the forwarder through Netlink. Step 6: The forwarder transmits the status information to the training module through the message middleware; Step 7: The training module calculates a reward value R according to the received state information, and updates the q value corresponding to the previous state and action according to the Q-learning algorithm according to the reward value R; Step 8: Determine the minimum round-trip time RTT min Is the update time greater than 10s? If so, enter RTT min In the exploration state, the congestion window size is set to 4 for RTT measurement. The RTT is explored. min End this state; otherwise, return to step 2.

2. The Q-learning-based satellite network congestion control method for TCP protocol according to claim 1, characterized in that: In step 1, the executor accumulates the congestion window according to the length of the current confirmation data packet, which is expressed as follows: cwnd new =cwnd old +acked Where cwnd new is the calculated new congestion window size, cwnd old is the old congestion window size, acked is the number of acknowledged packets; And determine whether it is the first time that packet loss and retransmission occur. If so, enter the reinforcement learning state and execute step 2; otherwise, execute step 1.

3. The satellite network congestion control method based on Q-learning for TCP protocol according to claim 1, characterized in that: In step 2, the actuator obtains the satellite network environment status by sampling at intervals, including RTT fluctuations and loss fluctuations; delay=RTT-RTT min loss flu =loss last -loss ave Where RTT min is the minimum round-trip delay within 10 seconds, RTT is the real-time round-trip delay; loss ave is the average packet loss rate, loss last is the packet loss rate in the previous sampling interval.

4. The Q-learning-based satellite network congestion control method for TCP protocol according to claim 1, characterized in that: The exponential moving average EMA formula is as follows: Where aim is the target value, value is the current EMA value, and rate is the smoothing factor, which is used to determine the impact of new data points on EMA.

5. The satellite network congestion control method based on Q-learning for TCP protocol according to claim 1, characterized in that: The Q table is designed as a three-dimensional structure to store the quality of different actions selected under different states. The first two dimensions represent the fluctuation of round-trip delay (RTT) and loss, respectively. The third dimension represents the action, which indicates the direction of the change of the congestion window (cwnd). The size of the congestion window (cwnd) is adjusted according to the result of EMA calculation. Each item in the Q table stores the action value q = Q(s, a) of the corresponding state and action index. The q value is used to represent the degree of goodness of choosing the action in the current state. Q(s,a)←(1-α)·Q(s,a)+α·(r+γ·maxQ(s',a)) Where α∈[0,1) represents the learning rate, γ∈[0,1) is the discount factor, and r is the immediate reward obtained after taking action a in state s.

6. The Q-learning-based satellite network congestion control method for TCP protocol according to claim 1, characterized in that: In step 7, the reward value R is calculated and expressed as follows: R=η·delivered_rate-β·delay Where delivered_rate is the effective delivery rate of the sender; delay = RTT-minRTT is the delay fluctuation; η, β are hyperparameters; The Q-learning algorithm update formula is as follows: Where Q(s,a) is the q value to be updated after the current state s executes the currently selected action a; is the maximum q value that the next state s' can take according to the current Q table, indicating the expectation of future rewards, a' is the corresponding action when the next state takes the maximum q value; α∈[0,1) represents the learning rate, indicating how much new information is learned each time; γ∈[0,1) is the discount factor, indicating the importance of future rewards.

7. A satellite network congestion control system based on Q-learning for TCP protocol, characterized by: The control system is used to execute the Q-learning-based satellite network congestion control method for TCP protocol as described in any one of claims 1 to 6; the control system includes an actuator, a repeater and a functional module; The executor is used to collect flow status information from the TCP data flow and execute the congestion control action issued; The forwarder is used to carry out message communication between the upper-layer functional module and the lower-layer executor, including forwarding of flow status messages and action decision messages; the forwarder uses netlink technology to communicate with the executor through a publish-subscribe mechanism and interact with the upper-layer functional module; The functional modules include various components for reinforcement learning, specifically a training module, a satellite network simulation training environment, a visualization module, an openAI Gym interface, and a deployment module; each module subscribes to a message channel, receives stream messages, and delivers action control messages to a forwarder.

8. The Q-learning-based satellite network congestion control system for TCP protocol according to claim 7, characterized in that: The functional module and the forwarder interact with each other through the message middleware. When a TCP flow is started with the specified congestion control algorithm, the TCP flow registers the congestion control function in the executor that includes the congestion control logic, periodically collects the current state information of the flow according to the rules defined by the monitoring interval, assigns a unique identifier to the flow as its tag, and passes this information to the forwarder via netlink; The forwarder performs deserialization and serialization of the stream information and transmits it to a dedicated channel associated with the stream, which is uniquely identified by a stream identifier; The functional modules subscribe to their respective stream communication channels, receive relevant stream status information and pass corresponding response actions back to the designated action channel; in the case of multiple streams at the same time, the forwarder ensures communication isolation by directing the stream information to different channels according to the unique identifier of each stream.

Images