A method and apparatus for congestion control in a low earth orbit satellite network

Abstract

The application provides a low-orbit satellite network congestion control method and device, the method comprises the following steps: in an offline state, obtaining a network data training set to perform reinforcement learning on a target agent. The training set contains time series data of states, actions, next time states and reward values at different times. In the reinforcement learning process, based on the reward value, the target action value function and the estimated action value function, a TD loss function is constructed by introducing an expected quantile regression function. A first loss function and a second loss function are constructed by using the state value function, the target action value function and the estimated action value function. Finally, a policy function is extracted. The parameters of all functions are iteratively updated. In an online state, the network data training set is updated based on the time series data obtained in real time. The updated training set is used to perform reinforcement learning to optimize the parameters. The application can adapt to highly dynamic LEO satellite networks, provide safe and effective strategies, quickly adapt to new environments and maintain high performance.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a method and apparatus for congestion control in low-Earth orbit satellite networks. Background Technology

[0002] With the development of multi-interface devices, the Multipath Transmission Control Protocol (MPTCP) has become increasingly important. MPTCP divides transmitted traffic into more sub-streams, establishes a separate end-to-end session for each sub-stream, and then remotely reassembles the received sub-streams into a single stream. MPTCP is widely used, especially in low Earth orbit (LEO) satellite networks. However, LEO satellite networks are highly dynamic, including routing changes, latency variations, and frequent packet loss rate fluctuations, which leads to a sharp decline in the performance of existing MPTCP congestion control (CC) schemes. Therefore, it is essential to design an MPTCP congestion control scheme for LEO satellite networks that can quickly adapt to new network conditions.

[0003] Several MPTCP congestion control schemes for satellite networks have been proposed in the prior art. These schemes require pre-setting appropriate round-trip time (RTT) and bandwidth thresholds to determine whether congestion events have occurred in the network. Unlike high-orbit satellite networks with relatively fixed latency, LEO satellite networks are very dynamic, with frequent link switching, each of which causes changes in link latency, making it difficult to set appropriate thresholds.

[0004] Besides the preset threshold method, there are also classic MPTCP congestion control algorithms and learning-based MPTCP congestion control algorithms. Classic algorithms are mostly single-path TCP algorithms extended to multi-path networks, making them unable to adapt to constantly changing network scenarios and erroneous responses to random packet loss, thus struggling to achieve consistently high throughput and adaptability. Learning-based algorithms can provide higher performance than classic algorithms; however, these algorithms suffer from transfer problems, meaning their performance degrades significantly in new network environments, failing to adapt to rapidly changing network conditions. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a method and apparatus for congestion control of low-Earth orbit satellite networks, in order to eliminate or improve the problem that existing transmission control protocols cannot adapt to the highly dynamic networks of low-Earth orbit satellites.

[0006] One aspect of the present invention provides a congestion control method for low-Earth orbit satellite networks, the method comprising the following steps:

[0007] In offline mode, a network data training set is acquired. The network data training set is time-series data that records the corresponding state, action, next state, and reward value at each time. The state includes the throughput, congestion window, round-trip time, and average deviation of the round-trip time of the low-Earth orbit satellite network. The action is a numerical parameter and is mapped to the congestion window.

[0008] In the reinforcement learning process, the agent selects actions based on a preset policy function, and calculates the expected cumulative reward using a preset target action value function, an estimated action value function, and a state value function. Based on the reward value, the target action value function, and the estimated action value function, an expected quantile regression function is introduced to construct a TD loss function. A first loss function is constructed based on the mean squared error of the state value function and the target action value function. A second loss function is constructed based on the mean squared error of the estimated action value function and the target action value function. Finally, an optimization function for the policy function is constructed based on the state value function and the target action value function.

[0009] After initializing the parameters of the state value function, the estimated action value function, the target action value function, and the policy function, the optimization function of the TD loss function, the first loss function, the second loss function, and the policy function is minimized by the gradient descent method. The parameters are then updated and iterated to obtain the target agent for low-Earth orbit satellite network congestion control.

[0010] In the online state, the target agent is deployed, and the network data training set is updated based on the time-series data of the target agent's state, action, next state, and reward value obtained in the online state.

[0011] The reinforcement learning process is performed using the updated network data training set, and the parameters of the target agent are fine-tuned and optimized.

[0012] In some embodiments of the present invention, the action is a numerical parameter and is mapped to a congestion window, and the mapping relationship is as follows:

[0013]

[0014] Among them, c t Let a represent the congestion window at time t. t This represents the action at time t.

[0015] In some embodiments of the present invention, the formula for calculating the reward value is:

[0016]

[0017] in, This represents the network throughput of substream i at time t.

[0018] In some embodiments of the present invention, the TD loss function expression is as follows:

[0019]

[0020] Where θ represents the parameters of the TD loss function L(θ), s represents the current state of the network environment, a represents the action taken in the current state, s′ represents the next state, a′ represents the action corresponding to the next state, D represents the experience replay buffer storing the network data training set, and E (s，a，s′，a′)～D This indicates that the desired operation will be performed on the samples in the experience replay buffer. Let represent the expected quantile regression function, γ represent the discount factor, and r(s, a) represent the immediate reward obtained by taking action a in state s. Let Q represent the target action value function. θ (s, a) represents the Q value corresponding to taking action a in state s, i.e., the estimated action value function.

[0021] In some embodiments of the present invention, the first loss function is calculated as follows:

[0022]

[0023] in, This represents the first loss function that approximates the state value function. Represents the first loss function The parameters are: s represents the current state of the network environment, and a represents the action to be taken in the current state. This represents the target action value function. This represents the expected quantile regression function. Let D represent the state value function, D represent the experience replay buffer storing the network data training set, and E( s，a)～D This indicates the desired operation to be performed on the samples in the experience playback buffer.

[0024] In some embodiments of the present invention, the second loss function is calculated as follows:

[0025]

[0026] Among them, L Q (θ) represents the second loss function that approximates the target action value function, and θ represents the second loss function L. Q The parameters of (θ) are: s represents the current state of the network environment, a represents the action taken in the current state, a′ represents the action corresponding to the next state, and Q. θ (s, a) represents the estimated action value function. The state-value function represents the state value under state s′. Let γ represent the expected quantile regression function, γ represent the discount factor, D represent the experience replay buffer storing the training set of the network data, and E represent the expected quantile regression function. (s，a，s′)～D This indicates the desired operation to be performed on the samples in the experience playback buffer.

[0027] In some embodiments of the present invention, the optimization function of the policy function is calculated as follows:

[0028]

[0029] Among them, L π (φ) represents the optimization function of the policy function, π φ Let φ represent the policy function, s represent the current state of the network environment, a represent the action to be taken in the current state, and logπ represent the policy function. φ (a|s) represents the policy function π φ Given state s, the logarithmic probability of choosing action a. This represents the target action value function. Let represent the state-value function under state s, β represent the hyperparameter used to control the degree of influence on the policy function, D represent the experience replay buffer storing the training set of the network data, and E represent the state-value function under state s. (s，a)～D This indicates the desired operation to be performed on the samples in the experience playback buffer.

[0030] In some embodiments of the present invention, the network data training set in the offline state is selected from suboptimal data or expert data; the network data training set is preprocessed by methods including data normalization or data cleaning.

[0031] Another aspect of the present invention provides a low-Earth orbit satellite network congestion control device, the device including a processor and a memory, the memory storing computer instructions, the processor executing the computer instructions stored in the memory, and the device implementing the steps of the above method when the computer instructions are executed by the processor.

[0032] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the above-described method.

[0033] The beneficial effects of the present invention are at least as follows:

[0034] This invention provides a method and apparatus for congestion control in low-Earth orbit satellite networks. In an offline state, a network data training set is acquired. During reinforcement learning, the agent selects actions according to a preset policy function and calculates the target action value function, estimated action value function, and state value function for expected cumulative reward. A TD loss function is constructed based on the reward value, target action value function, and estimated action value function, while simultaneously introducing an expected quantile regression function. A first loss function is constructed based on the mean squared error between the state value function and the target action value function. A second loss function is constructed based on the mean squared error between the estimated action value function and the target action value function. Finally, an optimization function for the policy function is constructed. The parameters of the state value function, estimated action value function, target action value function, and policy function are initialized. The TD loss function, the first loss function, the second loss function, and the optimization function of the policy function are minimized using gradient descent, and the parameters are updated iteratively to obtain the target agent for low-Earth orbit satellite network congestion control. In an online state, the target agent is deployed, and the network data training set is updated based on the time-series data obtained by the target agent in the online state. A reinforcement learning process is executed using the updated network data training set to fine-tune and optimize the parameters of the target agent. This method enables congestion control in low-Earth orbit satellite networks and can adapt to highly dynamic LEO satellite networks. When the network environment changes, especially when latency and packet loss change, it maintains high throughput while reducing flow completion time.

[0035] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0036] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0037] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings:

[0038] Figure 1 This is a flowchart of a low-Earth orbit satellite network congestion control method according to an embodiment of the present invention.

[0039] Figure 2 This is a framework diagram of a low-Earth orbit satellite network congestion control method according to another embodiment of the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0041] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0042] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0043] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0044] Low Earth Orbit (LEO) satellite networks are communication networks composed of satellites operating in orbits relatively close to Earth. LEO satellite networks are characterized by low latency, high bandwidth, global coverage, and flexibility, and are widely used in areas such as internet access, communication services, and the Internet of Things (IoT). Because the satellites in LEO satellite networks are constantly moving and changing positions, a network transmission control protocol capable of rapidly adapting to this highly dynamic nature is required.

[0045] One embodiment of the present invention provides a congestion control method for low-Earth orbit satellite networks, the method flow is as follows: Figure 1 As shown, the process includes the following steps S101-S105:

[0046] Step S101: In offline mode, acquire network data for training. The network data training set consists of time-series data recording the state, action, next state, and reward value at each time step. The state includes the throughput, congestion window, round-trip time, and average deviation of the round-trip time of the low-Earth orbit satellite network. The action is a numerical parameter mapped to the congestion window.

[0047] Step S102: During reinforcement learning, the agent selects actions using a pre-set policy function, and calculates the expected cumulative reward using a pre-set target action value function, an estimated action value function, and a state value function. Based on the reward value, the target action value function, and the estimated action value function, an expected quantile regression function is introduced to construct the TD loss function. Based on the mean squared error of the state value function and the target action value function, a first loss function is constructed; based on the mean squared error of the estimated action value function and the target action value function, a second loss function is constructed. Based on the state value function and the target action value function, an optimization function for the policy function is constructed.

[0048] Step S103: After initializing the parameters of the state value function, estimated action value function, target action value function, and policy function, minimize the TD loss function, first loss function, second loss function, and optimization function of policy function using gradient descent, update and iterate the parameters to obtain the target agent for low-Earth orbit satellite network congestion control.

[0049] Step S104: In the online state, deploy the target agent and update the network data training set based on the time series data of the target agent at each time step, including the corresponding state, action, next time step state, and reward value.

[0050] Step S105: Using the updated network data training set, perform the reinforcement learning process and fine-tune and optimize the parameters of the target agent.

[0051] In step S101, the action is a numerical parameter and is mapped with the congestion window. The mapping relationship is as follows:

[0052]

[0053] Among them, c t Let a represent the congestion window at time t. t This represents the action at time t.

[0054] In step S101, the formula for calculating the reward value is:

[0055]

[0056] in, This represents the network throughput of substream i at time t.

[0057] In step S101, the offline network data training set is selected from suboptimal data or expert data. Methods for preprocessing the network data training set include data normalization or data cleaning.

[0058] In step S102, the policy function defines the action the agent will take given a state. It maps the current state to an action or a probability distribution of actions, determining how the agent chooses its actions to maximize long-term cumulative rewards. Optimizing the policy function is key to improving the agent's performance in the environment.

[0059] The target action-value function, estimated action-value function, and state-value function are all types of value functions. A value function is used to evaluate the long-term cumulative reward that an agent expects to obtain given a state or state-action pair. Value functions include two forms: state-value functions and action-value functions. The state-value function measures the value of a state, while the action-value function measures the value of taking a particular action given a state. By continuously updating the value function, an agent can optimize its decision-making strategy to maximize its ability to obtain long-term rewards.

[0060] In step S102, the expression for the TD loss function is:

[0061]

[0062] Where θ represents the parameters of the TD loss function L(θ), s represents the current state of the network environment, a represents the action taken in the current state, s′ represents the next state, a′ represents the action corresponding to the next state, D represents the experience replay buffer storing the network data training set, and E (s，a，s′，a′)～D This indicates that the samples in the experience replay buffer will be subjected to the desired operation. Let represent the expected quantile regression function, γ represent the discount factor, and r(s, a) represent the immediate reward obtained by taking action a in state s. Q represents the target action value function. θ (s, a) represents the Q value corresponding to taking action a in state s, i.e., the estimated action value function, and r(s, a) represents the reward value obtained by taking action a in state s.

[0063] Specifically, the Temporal Difference (TD) loss function is used in reinforcement learning to evaluate the prediction error of the value function. In reinforcement learning, an agent learns how to make optimal decisions through interactions with its environment. The TD loss function measures the difference between the agent's current estimate and the actual reward received, thus guiding the agent to update its estimate of the value function. For example, in each interaction, the agent observes the current state and selects an action based on it. The agent then observes the reward returned by the environment and the next state. The TD loss function updates the estimate of the current state by comparing it with the estimate of the next state plus the reward. By minimizing the TD loss function, the agent can adjust its value estimates for states and actions, gradually approaching the true values, thereby improving the accuracy and effectiveness of its decisions.

[0064] Specifically, the expected quantile regression function Its function is to calculate the difference between the target value and the predicted value, and to fit the quantile by minimizing the sum of squared residuals between the target value and the predicted value.

[0065] In step S102, the formula for calculating the first loss function is:

[0066]

[0067] in, This represents the first loss function that approximates the state-value function. Represents the first loss function The parameters are: s represents the current state of the network environment, and a represents the action to be taken in the current state. Represents the target action value function. This represents the expected quantile regression function. Let E represent the state-value function, D represent the experience replay buffer storing the network training data, and E represent the state-value function. (s，a)～D This indicates the desired operation to be performed on the samples in the experience replay buffer.

[0068] In step S102, the formula for calculating the second loss function is:

[0069]

[0070] Among them, L Q (θ) represents the second loss function that approximates the target action value function, and θ represents the second loss function L. Q The parameters of (θ) are: s represents the current state of the network environment, a represents the action taken in the current state, a′ represents the action corresponding to the next state, and Q. θ (s, a) represents the estimated action value function. Let represent the expected quantile regression function, and γ represent the discount factor, a parameter used to measure the importance of the current decision to future rewards. Let E represent the state-value function under state s′, D represent the experience replay buffer storing the network training data, and E represent the state-value function under state s′. (s，a，s′)～D This indicates the desired operation to be performed on the samples in the experience replay buffer.

[0071] In step S102, the optimization function of the policy function is calculated as follows:

[0072]

[0073] Among them, L π (φ) represents the optimization function of the policy function, π φ Let φ represent the policy function, s represent the current state of the network environment, a represent the action to be taken in the current state, and logπ represent the policy function parameters. φ (a|s) represents the policy function π φ Given state s, the logarithmic probability of choosing action a. Let β represent the target action-value function, β represent the hyperparameter used to control the degree of influence on the policy function, D represent the experience replay buffer storing the network training data, and E represent the experience replay buffer. (s，a)～D This indicates the desired operation to be performed on the samples in the experience replay buffer.

[0074] Furthermore, in this embodiment, parameters are updated using a training dataset D during the offline learning phase. During the online fine-tuning phase, new data is collected through interaction with the environment and added to the dataset for parameter updates. This iterative optimization process provides prior knowledge to shorten learning time and adjusts the agent in real-time based on network conditions to achieve better results.

[0075] Another embodiment of the present invention provides a congestion control method for low-Earth orbit satellite networks, called IQLCC, to address the problem that existing congestion control schemes cannot quickly adapt to the constantly changing network conditions of LEO satellite networks and suffer performance degradation. This invention uses a dataset to train a reinforcement learning strategy offline, and then rapidly interacts with it during online deployment on the LEO satellite network to improve the strategy, bringing it to a convergent state, achieving sustained high performance, and avoiding lengthy retraining when encountering new network environments. The method framework provided in this embodiment is as follows: Figure 2 As shown, the buffer represents the space for temporarily storing the network dataset, and the sample batch represents a portion of the data set in the network dataset. The specific technical solution is as follows:

[0076] This invention mainly consists of two parts: offline learning and online fine-tuning.

[0077] During the offline phase, IQLCC uses a network dataset D as its training strategy. The dataset D is in the form {(s, a, s′, r)}. j Let}, s, a, s′, and r represent the state, action, next action, and reward value, respectively. The dataset can be expert data or suboptimal data.

[0078] IQLCC uses SARSA-style and Expectile Regression to reconstruct the policy and value function, and employs Advantage Weighted to extract the regression policy. IQLCC aims to determine how the Q-value changes with different actions, avoiding out-of-sample actions and improving the trained policy.

[0079] During the online phase, the offline-trained agent interacts with the environment, adding newly collected data (s, a, s′, r) to dataset D. The agent is then updated using this new dataset. This approach allows the agent to benefit from prior knowledge, shortening the learning time, and to adjust itself in real-time based on the network's status, leading to better results. The agent can adapt to the real-time conditions of the LEO satellite network, preventing significant throughput drops due to environmental changes, particularly substantial fluctuations in packet loss rate.

[0080] IQLCC sets the state to [thr, c, r, m], where thr, c, r, and m represent throughput, congestion window (cwnd), RTT (round-trip time), and mdev_us (mean deviation of RTT), respectively. These data effectively characterize the current network state. To flexibly change the congestion window, IQLCC uses a continuous action space, defining the relationship between cwnd and actions as follows:

[0081]

[0082] Among them, c t Let a represent the congestion window at time t. t This represents the action at time t.

[0083] The formula for calculating the reward value is:

[0084]

[0085] in, This represents the network throughput of sub-stream i at time t, which can simultaneously consider the throughput of multiple paths in MPTCP (Multipath TCP) and the fairness of TCP (Transmission Control Protocol) traffic.

[0086] Current offline RL methods are based on approximate dynamic programming methods, minimizing the TD error according to the following loss:

[0087]

[0088] Where θ represents the parameters of the TD loss function L(θ), s represents the current state of the network environment, a represents the action taken in the current state, s′ represents the next state, and a′ represents the action corresponding to the next state. Q represents the action-value function of the target network. θ (s, a) represents the Q-value corresponding to taking action a in state s, i.e., the estimated action-value function; r(s, a) represents the reward value obtained by taking action a in state s; γ represents the discount factor, a parameter used to measure the importance of the current decision to future rewards; D represents the experience replay buffer storing the training data set; E (s，a，s′)～D This indicates that the samples in the experience replay buffer will be subjected to the desired operation.

[0089] Where the policy function π(s) = argmax a Q θ (s, a), therefore, actions outside the training data distribution will result in an incorrect calculation for a′. This leads to overestimation, and the error will continue to increase, rendering the learning strategy useless.

[0090] The IQLCC algorithm provided in the embodiments of the present invention, combined with Expectile Regression to modify the SARSA-style TD loss function, yields:

[0091]

[0092] in, Let represent the Expecfile Regression function, θ represent the parameters of the TD loss function L(θ), s represent the current state of the network environment, a represent the action taken in the current state, s′ represent the next state, a′ represent the action corresponding to the next state, D represent the experience replay buffer storing the network data training set, and E represent the experience replay buffer. (s，a，s′，a′)～D Let represent the expected operation performed on samples in the experience replay buffer, γ represent the discount factor, and r(s, a) represent the immediate reward obtained by taking action a in state s. Q represents the target action value function. θ (s, a) represents the Q value corresponding to taking action a in state s, i.e., the estimated action value function, and r(s, a) represents the reward value obtained by taking action a in state s.

[0093] Based on this TD loss function, IQLCC avoids out-of-sample actions and overestimation errors, and can estimate the maximum Q value of actions that support the data distribution to prevent the policy from becoming useless.

[0094] The IQLCC algorithm also constructs a separate value function and a Q function. The value function is calculated as follows:

[0095]

[0096] in, This represents the first loss function that approximates the state-value function. Represents the first loss function The parameters are: s represents the current state of the network environment, and a represents the action to be taken in the current state. Represents the target action value function. This represents the expected quantile regression function. Let E represent the state-value function, D represent the experience replay buffer storing the network training data, and E represent the state-value function. (s，a)～D This indicates the desired operation to be performed on the samples in the experience replay buffer.

[0097] The formula for calculating the Q function is as follows:

[0098]

[0099] Among them, L Q (θ) represents the second loss function that approximates the target action value function, and θ represents the second loss function L. Q The parameters of (θ) are: s represents the current state of the network environment, a represents the action taken in the current state, a′ represents the action corresponding to the next state, and Q. θ (s, a) represents the estimated action value function. The state-value function represents the state value under state s′. Let E represent the expected quantile regression function, D represent the experience replay buffer storing the network training data, and E represent the expected quantile function. (s，a，s，)～D This indicates the desired operation to be performed on the samples in the experience replay buffer.

[0100] Although the IQLCC's TD learning process learns an approximation of the optimal Q-function, it does not explicitly represent the corresponding policy. Therefore, a separate policy extraction step is required. IQLCC uses AWR (Advantage Weighted Regression) to extract the policy:

[0101]

[0102] Among them, L π (φ) represents the optimization function of the policy function, π φLet φ represent the policy function, s represent the current state of the network environment, a represent the action to be taken in the current state, and logπ represent the policy function parameters. φ (a|s) represents the policy function π φ Given state s, the logarithmic probability of choosing action a. Let β represent the target action-value function, β represent the hyperparameter used to control the degree of influence on the policy function, D represent the experience replay buffer storing the network training data, and E represent the experience replay buffer. (s，a)～D This indicates the desired operation to be performed on the samples in the experience replay buffer.

[0103] The IQLCC algorithm includes the following steps S201 to S203:

[0104] Step S201: Initialize parameters θ、 φ and dataset B.

[0105] Step S202: In the offline phase, the agent samples {(s, a, s′, r)} from dataset B. j}, update parameters θ、 φ.

[0106] Step S203: In the online fine-tuning phase, at each time step, the agent interacts with the environment, adding (s, a, s′, r) to dataset B, and sampling {(s, a, s′, r)} from dataset B. j}, update parameters θ、 φ.

[0107] IQLCC can deploy the reinforcement learning part in user space and the rest in kernel space. IQLCC can maintain high throughput even when the environment changes.

[0108] In summary, this invention provides a method and apparatus for congestion control in low-Earth orbit (LEO) satellite networks. The method includes: in an offline state, acquiring a network data training set to perform reinforcement learning on a target agent. This training set contains time-series data of states, actions, next-time states, and reward values ​​at different times. During reinforcement learning, a TD loss function is constructed by introducing an expected quantile regression function based on the reward value, the target action value function, and the estimated action value function. A first loss function and a second loss function are constructed using the state value function, the target action value function, and the estimated action value function. Finally, a policy function is extracted. The parameters of all functions are iteratively updated. In an online state, the network data training set is updated based on time-series data acquired from the real-time network. Reinforcement learning is then performed using the updated training set to optimize the parameters. This invention can adapt to highly dynamic LEO satellite networks, maintaining high throughput even when the environment changes, particularly latency and packet loss. It also reduces flow completion time and maintains good fairness.

[0109] Corresponding to the above method, the present invention also provides an apparatus comprising a computer device including a processor and a memory, the memory storing computer instructions, the processor executing the computer instructions stored in the memory, and the apparatus performing the steps of the method as described above when the computer instructions are executed by the processor.

[0110] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0111] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0112] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0113] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0114] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A congestion control method for low-Earth orbit satellite networks, characterized in that, include: In offline mode, a network data training set is acquired. The network data training set is time-series data that records the corresponding state, action, next state, and reward value at each time. The state includes the throughput, congestion window, round-trip time, and average deviation of the round-trip time of the low-Earth orbit satellite network. The action is a numerical parameter and is mapped to the congestion window. In the reinforcement learning process, the agent selects actions by pre-set policy functions, and calculates the expected cumulative reward by pre-set target action value functions, estimated action value functions, and state value functions. Based on the reward value, the target action value function, and the estimated action value function, a TD loss function is constructed by introducing an expected quantile regression function; a first loss function is constructed based on the mean square error of the state value function and the target action value function; a second loss function is constructed based on the mean square error of the estimated action value function and the target action value function; and an optimization function for the policy function is constructed based on the state value function and the target action value function. After initializing the parameters of the state value function, the estimated action value function, the target action value function, and the policy function, the optimization function of the TD loss function, the first loss function, the second loss function, and the policy function is minimized by the gradient descent method. The parameters are then updated and iterated to obtain the target agent for low-Earth orbit satellite network congestion control. In the online state, the target agent is deployed, and the network data training set is updated based on the time-series data of the target agent's state, action, next state, and reward value obtained in the online state. The reinforcement learning process is performed using the updated network data training set, and the parameters of the target agent are fine-tuned and optimized.

2. The low-Earth orbit satellite network congestion control method according to claim 1, characterized in that, The action is a numerical parameter and is mapped to the congestion window. The mapping relationship is as follows: Among them, c t Let a represent the congestion window at time t. t This represents the action at time t.

3. The low-Earth orbit satellite network congestion control method according to claim 1, characterized in that, The formula for calculating the reward value is: in, This represents the network throughput of substream i at time t.

4. The low-Earth orbit satellite network congestion control method according to claim 1, characterized in that, The expression for the TD loss function is: Where θ represents the parameters of the TD loss function L(θ), s represents the current state of the network environment, a represents the action taken in the current state, s′ represents the next state, a′ represents the action corresponding to the next state, D represents the experience replay buffer storing the network data training set, and E (s，a，s′，a′)～D This indicates that the desired operation will be performed on the samples in the experience replay buffer. Let r(s, a) represent the expected quantile regression function, and let r(s, a) represent the immediate reward obtained by taking action a in state s. Let Q represent the target action value function. θ (s, a) represents the Q value corresponding to taking action a in state s, i.e. the estimated action value function, and γ represents the discount factor.

5. The low-Earth orbit satellite network congestion control method according to claim 4, characterized in that, The formula for calculating the first loss function is: in, This represents the first loss function that approximates the state value function. Represents the first loss function The parameters are: s represents the current state of the network environment, and a represents the action to be taken in the current state. This represents the target action value function. The state-value function represents the state value under state s. This represents the expected quantile regression function. Let D represent the state value function, D represent the experience replay buffer storing the network data training set, and E represent the state value function. (s，a)～D This indicates the desired operation to be performed on the samples in the experience playback buffer.

6. The low-Earth orbit satellite network congestion control method according to claim 4, characterized in that, The formula for calculating the second loss function is: Among them, L Q (θ) represents the second loss function that approximates the target action value function, and θ represents the second loss function L. Q The parameters of (θ) are: s represents the current state of the network environment, a represents the action taken in the current state, a′ represents the action corresponding to the next state, and Q. θ (s, a) represents the estimated action value function. The state-value function represents the state value under state s′. Let γ represent the expected quantile regression function, γ represent the discount factor, D represent the experience replay buffer storing the training set of the network data, and E represent the expected quantile regression function. (s，a，s′)～D This indicates the desired operation to be performed on the samples in the experience playback buffer.

7. The low-Earth orbit satellite network congestion control method according to claim 4, characterized in that, The optimization function of the policy function is calculated as follows: Among them, L π (φ) represents the optimization function of the policy function, π φ Let φ represent the policy function, s represent the current state of the network environment, a represent the action to be taken in the current state, and logπ represent the policy function. φ (a|s) represents the policy function π φ Given state s, the logarithmic probability of choosing action a. Let β represent the target action value function, β represent the hyperparameter used to control the degree of influence on the policy function, D represent the experience replay buffer storing the network data training set, and E represent the target action value function. (s，a)～D This indicates the desired operation to be performed on the samples in the experience playback buffer.

8. The low-Earth orbit satellite network congestion control method according to claim 1, characterized in that, The network data training set in the offline state selects suboptimal data or expert data; The network data training set is preprocessed using methods including data normalization or data cleaning.

9. A congestion control device for low-Earth orbit satellite networks, comprising a processor and a memory, characterized in that, The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 8.

Images