Low earth orbit satellite communication network routing method based on reinforcement learning

Abstract

The invention discloses a low-earth-orbit satellite communication network routing method based on reinforcement learning, and relates to the field of low-earth-orbit satellite communication networks in 5G NTN, in particular to the low-earth-orbit satellite communication network routing method based on reinforcement learning. The objective of the invention is to solve the problem that existing reinforcement learning Q-Learning applications are mostly focused on a ground network, and the unique dynamic topology and three-dimensional space routing problems of a satellite network are not solved. The invention provides a lightweight and adaptive low earth orbit satellite communication network routing optimization method based on reinforcement learning. The method is realized through distributed Q-Learning: dynamically adapting to topological change, dynamically adapting to three-dimensional space routing, balancing link load, minimizing end-to-end delay, and reducing on-satellite computing overhead.

Description

Technical Field

[0001] This invention relates to the field of 5GNTN low-Earth orbit satellite communication networks, and more particularly to a routing method for low-Earth orbit satellite communication networks based on reinforcement learning, specifically applicable to dynamic routing decision optimization in low Earth orbit (LEO) satellite constellation networks. Background Technology

[0002] With the continuous development of wireless communication technology, traditional terrestrial mobile communication networks can no longer meet the needs of today's society. Satellite communication networks, as an important supplement to terrestrial mobile communication networks, have attracted much attention. As an important component of 5G NTN, satellite communication networks, particularly low-Earth orbit satellite network routing technology, involve complex decision-making processes aimed at optimizing data transmission efficiency, and have thus received extensive research. Traditional routing methods (such as Dijkstra and OSPF) suffer from drawbacks such as high computational overhead and poor adaptability. In recent years, the idea of ​​combining reinforcement learning with satellite routing has gradually gained attention. On June 18, 2024, the R18 standard was officially frozen, proposing an evolution towards network intelligence and dynamic intelligent routing to address key issues such as on-board payloads, inter-satellite topology, and routing in integrated space-air-ground scenarios. From the on-board transparent forwarding scenarios of non-terrestrial networks in R15-R17 to the base station and core network in R19, the on-board processing capabilities have been continuously enhanced, making the combination of reinforcement learning and satellite routing possible. Existing Q-Learning applications are mostly concentrated in terrestrial networks, failing to address the dynamic topology and three-dimensional spatial routing problems unique to satellite networks. Summary of the Invention

[0003] The purpose of this invention is to address the issue that existing reinforcement learning Q-Learning applications are mostly concentrated in terrestrial networks and have not solved the dynamic topology and three-dimensional spatial routing problems unique to satellite networks. Therefore, this invention proposes a routing method for low-Earth orbit satellite communication networks based on reinforcement learning.

[0004] The specific process of a low-Earth orbit satellite communication network routing method based on reinforcement learning is as follows:

[0005] Step 1: Randomly initialize the Q-tables of all satellites, where the Q-table of the i-th satellite stores the Q-value Q of each state-action pair of the i-th satellite. i (S,a);

[0006] 'a' represents the action; 'S' represents the state space.

[0007] Initialize the total time steps to T;

[0008] Randomly initialize the learning rate α, discount factor γ, and exploration rate ε;

[0009] The satellite is a LEO satellite;

[0010] Step 2: Set time step t = 1;

[0011] Step 3: At time step t, the i-th satellite receives the data packet and obtains the status S of the i-th satellite at time step t. t ;

[0012] An ε-greedy strategy is used to select actions in the action space;

[0013] Step 4: Execute the action 'a' selected in Step 3 and calculate the reward function R. t (s,a) and the state S of the i-th satellite at time step t+1. t+1 Simultaneously update the data transmission path record, recording the satellite nodes that the data packets pass through;

[0014] Determine if the data packet has reached the target satellite;

[0015] If the data packet reaches the target satellite, the data packet is sent directly to the target satellite, ending the data packet transmission; proceed to step five.

[0016] If the target satellite is not reached, the reward function R is used. t (s,a) Update the Q value, and update the Q-table based on the updated Q value; proceed to step five.

[0017] Step 5: Set time step t = t + 1;

[0018] Repeat steps three and four until the total time step is T, and obtain the optimal Q-table.

[0019] Step 6: Deploy the optimal Q-table on the satellite. For any source satellite receiving data packets, obtain the satellite communication network route for the data packets to reach the target satellite.

[0020] The beneficial effects of this invention are as follows:

[0021] This invention proposes a lightweight, adaptive, reinforcement learning-based routing optimization method for low-Earth orbit satellite communication networks. It achieves the following through distributed Q-Learning: dynamic adaptation to topology changes, dynamic adaptation to three-dimensional spatial routing, balancing link load, minimizing end-to-end latency, and reducing on-board computing overhead.

[0022] Q-tables are a common method in reinforcement learning, especially in Q-Learning algorithms. Each cell in a Q-table stores the expected reward (Q-value) for taking a specific action in a given state. By continuously updating the Q-table, the agent can learn the optimal policy—the best action to take in each state to maximize cumulative rewards.

[0023] This invention presents a reinforcement learning Q-Learning algorithm, a routing method based on reinforcement learning, which adaptively adjusts routing decisions according to dynamic changes in the network. Under high load conditions, network congestion becomes more complex and variable. Traditional shortest path-based routing algorithms (such as data rate benchmark algorithms) often rely on fixed path selection criteria and cannot adapt to changes in network conditions in a timely manner. In contrast, this invention's reinforcement learning Q-Learning algorithm, by continuously learning and updating the Q-value, can better cope with network congestion and select optimal paths.

[0024] Furthermore, the reinforcement learning Q-Learning algorithm of this invention utilizes local information for decision-making, with each satellite node interacting only with nearby nodes, reducing the overhead of global communication. Under high load conditions, the overhead of global communication increases significantly, potentially exacerbating network congestion. This invention's reinforcement learning Q-Learning algorithm, by utilizing local information, reduces communication overhead and improves the overall performance of the system. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the indirect access satellite-ground integrated network model in Specific Implementation Method 1.

[0026] Figure 2 This is a schematic diagram of a low-Earth orbit satellite constellation in Specific Implementation Method 1, with a custom xyz axis, x perpendicular to y, and z perpendicular to the xy plane;

[0027] Figure 3 This is a flowchart of the routing method for low-Earth orbit satellite communication networks based on the Q-Learning algorithm of this invention. Detailed Implementation

[0028] Specific Implementation Method 1: The specific process of this implementation method for routing low-Earth orbit satellite communication networks based on reinforcement learning is as follows:

[0029] The research scenario of the satellite communication system of this invention is as follows: Figure 1 As shown, in this scenario, users access a communication network via satellite. Satellites are interconnected through inter-satellite links, and user data is transmitted to its destination via the satellite communication network. The satellite accessed by the user is called the source satellite, and the satellite corresponding to the destination of the data packet is called the destination satellite. The communication processing payload deployed on the satellite serves as the hardware execution carrier of the satellite communication routing method of this invention. Relying on its onboard signal processing and data forwarding capabilities, it completes the calculation and execution of the routing strategy. Simultaneously, the satellite is equipped with a memory with a certain storage capacity to store the queue of user data to be forwarded and satellite network topology information, etc. Figure 1This embodiment describes a satellite network routing optimization method and system based on reinforcement learning, specifically applicable to dynamic routing decision optimization in low Earth orbit (LEO) satellite constellation networks.

[0030] Constructing a space-ground integrated network architecture for users to indirectly access satellites; specifically:

[0031] The satellite is a LEO satellite;

[0032] The LEO satellite, as a payload satellite, also has the capability of an "airborne base station" (Sat-gNodeB, i.e., satellite-side 5G base station function). The communication processing payload deployed on the satellite serves as the hardware execution carrier for the satellite communication routing method of this invention. Relying on its on-board signal processing and data forwarding capabilities, it completes the calculation and execution of routing strategies. Through inter-satellite links within and between orbits, it collaborates with other satellites to expand coverage and optimize transmission paths.

[0033] The ground station interacts with the satellite through the feeder circuit. The ground station is responsible for receiving and transmitting data transmitted by the satellite, uploading user data sent by the satellite cell base station to the satellite, and forwarding downlink data from the satellite to the satellite cell base station, thus realizing two-way communication between the satellite and the ground.

[0034] A satellite cell is a communication area formed by satellite coverage. User equipment within the communication area accesses the satellite cell base station through NR (5G New Radio) technology. Within a satellite cell, communication can be achieved through satellite links, expanding the coverage boundaries of traditional terrestrial cells and allowing remote areas and oceans to be included in the communication network.

[0035] Step 1: Randomly initialize the Q-tables of all satellites, where the Q-table of the i-th satellite stores the Q-value Q of each state-action pair of the i-th satellite. i (S,a);

[0036] 'a' represents the action; 'S' represents the state space.

[0037] Initialize the total time steps to T;

[0038] Randomly initialize the learning rate α, discount factor γ, and exploration rate ε;

[0039] The satellite is a LEO satellite;

[0040] Step 2: Set time step t = 1;

[0041] Step 3: At time step t, the i-th satellite receives the data packet and obtains the status S of the i-th satellite at time step t. t ;

[0042] An ε-greedy strategy is used to select actions in the action space;

[0043] The ε-greedy strategy is to explore with a probability of ε, that is, to randomly select one of the four neighboring satellites and utilize it with a probability of 1-ε, choosing the one with the largest Q value among the four neighboring satellites in the current state of the Q table;

[0044] Step 4: Execute the action 'a' selected in Step 3 and calculate the reward function R. t (s,a) and the state S of the i-th satellite at time step t+1. t+1 Simultaneously update the data transmission path record, recording the satellite nodes that the data packets pass through;

[0045] Determine if the data packet has reached the target satellite;

[0046] If the data packet reaches the target satellite, the data packet is sent directly to the target satellite, ending the data packet transmission; proceed to step five.

[0047] If the target satellite is not reached, the reward function R is used. t (s,a) Update the Q value, and update the Q-table based on the updated Q value; proceed to step five.

[0048] Step 5: Set time step t = t + 1;

[0049] Repeat steps three and four until the total time step is T, and obtain the optimal Q-table.

[0050] Step 6: Deploy the optimal Q-table on the satellite. For any source satellite receiving data packets, obtain the satellite communication network route for the data packets to reach the target satellite.

[0051] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that S is a state space, which is a four-dimensional vector, S = [Source, Neighbors, Queues, Destination];

[0052] Wherein, Source represents the current satellite coordinate information;

[0053] Neighbours represents the coordinate differences between the current satellite and its northern neighbor satellite, the southern neighbor satellite, the western neighbor satellite, and the eastern neighbor satellite (Neighbours represents 4 values); North, South, West, and East are the directions centered on the current satellite, and are custom directions, not strictly due North, Due South, Due West, and Due East.

[0054] Queues represents the number of data packets waiting to be forwarded in the satellite's northward neighbor satellites, the number of data packets waiting to be forwarded in the satellite's southward neighbor satellites, the number of data packets waiting to be forwarded in the satellite's westward neighbor satellites, and the number of data packets waiting to be forwarded in the satellite's eastward neighbor satellites, directly reflecting the satellite's load status.

[0055] Destination indicates the satellite index (known) corresponding to the destination;

[0056] The perceived information is combined into a multi-dimensional vector, which serves as the state representation of the current network environment.

[0057] The learning rate α determines the speed at which the algorithm learns new information, and its value is generally between 0 and 1, set to 0.25;

[0058] The discount factor γ reflects the importance attached to future rewards and is usually close to 1, so we set it to 0.9.

[0059] The exploration rate ε controls the balance between exploration and exploitation. The initial value is set to 0.99, and an exponential decay strategy is adopted during the training process.

[0060] The satellite acquires its own and the surrounding network environment status in real time, specifically including the coordinate information of the current satellite, the coordinate difference between the current satellite and neighboring satellites, the queue status of neighboring satellites, and the satellite index corresponding to the destination of the data packet. Based on the above observation of the environmental status, a state vector is constructed, and the above acquired information is combined into a multi-dimensional vector as a representation of the current network environment status.

[0061] The value of ε gradually decreases as the number of learning steps increases. The formula for the decay strategy of the exploration rate ε is:

[0062] ε=ε min +(ε max -ε min )×e -λ×episode

[0063] in,

[0064] ε min It is the minimum value, which is 0.001.

[0065] ε max It is the maximum value, i.e., the initial value of 0.99;

[0066] λ is the decay rate coefficient;

[0067] episode represents a training round;

[0068] This decay strategy results in a high exploration rate in the early stages of learning, which gradually decreases as learning progresses, while the proportion of utilizing already learned knowledge gradually increases.

[0069] The initial value of ε is set to a relatively high value (0.99), indicating that the agent is more inclined to explore during the initial learning phase. By randomly selecting actions, the agent can extensively explore the state space, discover potential better actions, and avoid getting trapped in local optima too early. This high exploration rate helps the agent quickly understand the environment and collect more information from different states, providing a rich data foundation for subsequent learning. A relatively low minimum value (0.001) is also set to ensure that the agent does not remain in an exploratory state throughout the learning process, but gradually transitions to an exploitation phase. When ε reaches its minimum value, the agent primarily relies on learned experience to select actions with higher Q-values ​​to maximize rewards.

[0070] To achieve a smooth transition from exploration to utilization, an exponential decay strategy was adopted.

[0071] The other steps and parameters are the same as in Specific Implementation Method 1.

[0072] Specific Implementation Method 3: This implementation method differs from Specific Implementation Method 1 or 2 in that the action space in step 2 includes all actions that the current satellite can take;

[0073] All actions are composed of neighboring satellites in the four directions of north, south, west, and east of the current satellite;

[0074] The process for determining the neighboring satellites is as follows:

[0075] Based on the satellite's orbital parameters and the parameters of the communication equipment, the distance and relative angle between adjacent satellites are calculated using the satellite's onboard computer. If the distance between satellite A and satellite B is less than the maximum communication distance of the communication equipment and the relative angle is within the effective coverage of the antenna, then satellite B is added to satellite A's list of neighboring satellites.

[0076] The satellite's orbital parameters include semi-major axis, eccentricity, orbital inclination, right ascension of the ascending node, argument of perigee, and true anomaly. These parameters are acquired through the satellite's orbital measurement equipment or onboard sensors and stored in the onboard computer.

[0077] The parameters of the communication equipment are communication frequency, transmission power, and antenna pattern; these parameters are determined during the manufacturing and debugging phases of the satellite communication equipment and are pre-stored in the parameter database of the onboard computer.

[0078] Other steps and parameters are the same as in specific implementation method one or two.

[0079] Specific Implementation Method Four: This implementation method differs from Specific Implementation Methods One to Three in that, based on the satellite's orbital parameters and the parameters of the communication equipment, the distance and relative angle between adjacent satellites are calculated using the satellite's onboard computer. If the distance between satellite A and satellite B is less than the maximum communication distance of the communication equipment, and the relative angle is within the effective coverage range of the antenna, then satellite B is added to the list of neighboring satellites of satellite A.

[0080] The specific process is as follows:

[0081] 1) The calculation process for the distance between adjacent satellites is as follows:

[0082] The onboard computer carried by the satellite calculates the distance between adjacent satellites based on the acquired satellite orbital parameters using specific orbital mechanics algorithms; for example, by substituting the satellite's orbital parameters into the Kepler orbital equations and combining them with time parameters, the satellite's position coordinates (x, y, z) in space at different times can be accurately calculated.

[0083] For two satellites A and B, the distance d between satellites A and B is given by the Euclidean distance formula. Perform calculations;

[0084] In the formula, (x A ,y A ,z A (x) represents the position coordinates of satellite A. B ,y B ,z B ( ) represents the position coordinates of satellite B;

[0085] 2) The calculation process for the relative angle between adjacent satellites is as follows:

[0086] The calculation of the relative angle is based on the satellite's position coordinates and the antenna's pointing information;

[0087] Assume that the antenna pointing vector of satellite A is The position vector of satellite B relative to satellite A is The relative angle θ between satellite A and satellite B can be expressed by the vector dot product formula. Calculations show that · represents the vector dot product;

[0088] 3) If the distance between satellite A and satellite B is less than the maximum communication distance d of the communication equipment. max And the relative angle is within the effective coverage range of the antenna (i.e., θ satisfies the effective coverage angle range of the antenna [θ]). min ,θ max If satellite B is added to satellite A's list of neighboring satellites, then satellite B will be added to satellite A's list of neighboring satellites.

[0089] Among them, the maximum communication distance d of the communication equipmentmax The calculation is performed using the communication link budget formula based on parameters such as communication frequency, transmit power, and antenna pattern; the specific process is as follows:

[0090]

[0091] Among them, P r For received power, P t For transmission power, G t G r Here, λ represents the transmit and receive antenna gains (related to the antenna pattern), d represents the wavelength (determined by the communication frequency), and L represents the communication distance.

[0092] By setting the minimum acceptable receiving power, the maximum communication distance d can be derived. max ;

[0093] After calculation and judgment, satellite A determines its neighboring satellites to be satellites B, C, D, and E, and stores this information in the neighboring satellite list for use in subsequent action selection.

[0094] The other steps and parameters are the same as those in one of the specific implementation methods one to three.

[0095] Specific Implementation Method Five: This implementation method differs from Specific Implementation Methods One to Four in that the reward function R in step three... t The formula for (s,a) is:

[0096] R t (s,a)=R arrive +w1×R queue +w2×R distance +w3×R penalty

[0097] In the formula,

[0098] R arrive This indicates the arrival reward, which is triggered when the data block successfully reaches the target satellite and is set to 1; otherwise, it is set to 0.

[0099] R queue Indicates queue status rewards to balance network traffic;

[0100] R distance This indicates a distance bonus, encouraging users to choose paths that are closer to the goal;

[0101] R penalty This indicates a comprehensive penalty item to prevent loops and invalid actions;

[0102] w1 represents the queue state reward R. queue The weighting coefficient is set to 20.

[0103] w2 represents the distance bonus R. distance The weighting coefficient is set to 20.

[0104] w3 represents the comprehensive penalty item R. penalty The weighting coefficient is set to 1.

[0105] The reward function R(s,a) takes task achievement as its core objective, balances congestion control, path optimization, communication stability and loop prevention mechanisms, and achieves multi-objective collaborative optimization through weighted summation.

[0106] The other steps and parameters are the same as those in one of the specific implementation methods one to four.

[0107] Specific Implementation Method Six: This implementation method differs from Specific Implementation Methods One to Five in that the arrival of the prize R... arrive The formula is:

[0108] R arrive =δ arrive ×R a

[0109]

[0110] In the formula, δ arrive The arrival determination indicator function takes a value of 1 when action a causes the data packet to reach the target satellite, and a value of 0 otherwise; δ arrive Used to determine whether the current action has successfully delivered the data packet to the target satellite;

[0111] R a To reach the base value for the reward, R a =50, providing sparse, high-value rewards to drive agents to complete core tasks.

[0112] The other steps and parameters are the same as those in one of the specific implementation methods one to five.

[0113] Specific Implementation Method Seven: This implementation method differs from Specific Implementation Methods One through Six in that the queue state reward R... queue The formula is:

[0114]

[0115] In the formula,

[0116] q neighbor This indicates the number of data packets waiting to be forwarded in the selected neighboring satellites, directly reflecting the satellite's load status, and is defined as 0-10;

[0117] q current This indicates the number of data packets currently queued for forwarding in the satellite;

[0118] qmax q represents the maximum number of data packets allowed to be queued within a satellite. max =10;

[0119] This represents the current queue penalty coefficient. By encouraging nodes to choose those with low loads themselves and their neighbors, network traffic can be balanced and congestion controlled.

[0120] The other steps and parameters are the same as those in one of the specific implementation methods one to six.

[0121] Specific Implementation Method Eight: This implementation method differs from Specific Implementation Methods One to Seven in that the distance reward R... distance For path optimization, the formula is:

[0122]

[0123] In the formula, d(s) is the distance from the current state to the target satellite;

[0124] d(s′) represents the distance from the target satellite to the new state after the action is performed;

[0125] d max The maximum inter-satellite link distance for the constellation is set at 3700 km;

[0126] R distance This represents the reduction in slant distance caused by the normalized execution action. Path optimization is achieved by giving positive rewards for getting closer to the target and negative penalties for getting further away from the target.

[0127] The comprehensive penalty item R penalty As a constraint guarantee, the formula is:

[0128] R penalty =δ again ×P a +δ unav ×P u

[0129]

[0130] In the formula, δ again The indicator function for the action selected by the ε-greedy policy.

[0131] δ unav Invalid action indicator function,

[0132] P a For loop penalty, P a =-10;

[0133] P u To penalize invalid actions and thus prevent loops and invalid actions, Pu =-10.

[0134] Neighbor unavailable means that the next-hop neighbor node that the satellite intends to select is unable to establish a communication connection or forward data for various reasons. This includes hardware failures: damage to the neighbor satellite's transceiver antennas or radio frequency modules, preventing signal transmission and reception.

[0135] Satellites need to obtain neighbor availability through real-time status monitoring and periodically measure the link quality with neighbors. If it is below a threshold, it is determined to be unavailable.

[0136] The other steps and parameters are the same as those in any of the specific implementation methods one to seven.

[0137] Specific Implementation Method Nine: This implementation method differs from Specific Implementation Methods One through Eight in that the reward function R is used in the following way. t (s,a) updates the Q value, and updates the Q-table based on the updated Q value;

[0138] The specific process is as follows:

[0139] The Q value is updated based on the time difference (TD), using the following formula:

[0140] Q*(s t ,a t )←Q(s t ,a t )+α[r t +γmax a Q(s t+1 ,a)-Q(s t ,a t )]

[0141] In the formula,

[0142] α is the learning rate, set to 0.25;

[0143] γ is the discount factor, set to 0.9;

[0144] r t The reward function R is the action at time step t. t (s,a), calculated using the reward function described above;

[0145] Q(s t ,a t ) represents state s t Choose a below t Q value;

[0146] Q*(s t ,a t () represents the updated Q value;

[0147] Q(s t+1a) represents state s t+1 Choose a below t Q value;

[0148] max a Q(s t+1 a) is the maximum Q value in the next state;

[0149] Q*(s) of the satellite based on the updated time step t t ,a t Update the Q-table.

[0150] The other steps and parameters are the same as those in one of the specific implementation methods one to eight.

[0151] Specific Implementation Method Ten: This implementation method differs from Specific Implementation Methods One through Nine in that, in step six, the optimal Q-table is deployed on the satellite. For any source satellite receiving data packets, the satellite communication network route of the data packets to the target satellite is obtained; the specific process is as follows:

[0152] Step 61: The source satellite receives data packets and obtains the satellite's status S. t ;

[0153] Step 62: Find the action corresponding to the maximum Q value of the state in the final Q-table, and execute the action to obtain the new state;

[0154] Step 63: Find the action corresponding to the maximum Q value of the new state in Step 62 in the final Q-table, and execute the action to obtain the new state;

[0155] Repeat steps 62 to 63 until the data packet reaches the target satellite, thus obtaining the satellite communication network route of the data packet from the source satellite to the target satellite.

[0156] The other steps and parameters are the same as those in any of the specific implementation methods one to nine.

[0157]

[0158] This invention may have other embodiments. Without departing from the spirit and essence of this invention, those skilled in the art can make various corresponding changes and modifications according to this invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A routing method for low-Earth orbit satellite communication networks based on reinforcement learning, characterized in that: The specific process of the method is as follows: Step 1: Randomly initialize the Q-tables of all satellites, where the Q-table of the i-th satellite stores the Q-value Q of each state-action pair of the i-th satellite. i (S,a); 'a' represents the action; 'S' represents the state space. Initialize the total time steps to T; Randomly initialize the learning rate α, discount factor γ, and exploration rate ε; The satellite is a LEO satellite; Step 2: Set time step t = 1; Step 3: At time step t, the i-th satellite receives the data packet and obtains the status S of the i-th satellite at time step t. t ; An ε-greedy strategy is used to select actions in the action space; Step 4: Execute the action 'a' selected in Step 3 and calculate the reward function R. t (s,a) and the state S of the i-th satellite at time step t+1. t+1 Simultaneously update the data transmission path record, recording the satellite nodes that the data packets pass through; Determine if the data packet has reached the target satellite; If the data packet reaches the target satellite, the data packet is sent directly to the target satellite, ending the data packet transmission; proceed to step five. If the target satellite is not reached, the reward function R is used. t (s,a) Update the Q value, and update the Q-table based on the updated Q value; proceed to step five. Step 5: Set time step t = t + 1; Repeat steps three and four until the total time step is T, and obtain the optimal Q-table. Step 6: Deploy the optimal Q-table on the satellite. For any source satellite receiving data packets, obtain the satellite communication network route for the data packets to reach the target satellite.

2. The routing method for low-Earth orbit satellite communication networks based on reinforcement learning according to claim 1, characterized in that: S is the state space, which is a four-dimensional vector, S = [Source, Neighbours, Queues, Destination]. Wherein, Source represents the current satellite coordinate information; Neighbours represents the coordinate difference between the current satellite and its northern neighbor satellite, the coordinate difference between the current satellite and its southern neighbor satellite, the coordinate difference between the current satellite and its western neighbor satellite, and the coordinate difference between the current satellite and its eastern neighbor satellite. Queues represents the number of data packets waiting to be forwarded in the northward neighboring satellites, the number of data packets waiting to be forwarded in the southward neighboring satellites, the number of data packets waiting to be forwarded in the westward neighboring satellites, and the number of data packets waiting to be forwarded in the eastward neighboring satellites. Destination represents the satellite index corresponding to the destination; The learning rate α is set to 0.25; The discount factor γ is set to 0.9; The initial exploration rate ε is set to 0.99; The formula for the decay strategy of the exploration rate ε is: e=e min +(e max -e min )×e -λ×episode in, ε min It is the minimum value, which is 0.

001. ε max It is the maximum value, i.e., the initial value of 0.99; λ is the decay rate coefficient; episode refers to a training round.

3. The low-Earth orbit satellite communication network routing method based on reinforcement learning according to claim 2, characterized in that: The action space in step two includes all actions that the current satellite can take. All actions are composed of neighboring satellites in the four directions of north, south, west, and east of the current satellite; The process for determining the neighboring satellites is as follows: Based on the satellite's orbital parameters and the parameters of the communication equipment, the distance and relative angle between adjacent satellites are calculated using the satellite's onboard computer. If the distance between satellite A and satellite B is less than the maximum communication distance of the communication equipment and the relative angle is within the effective coverage of the antenna, then satellite B is added to satellite A's list of neighboring satellites. The satellite's orbital parameters include semi-major axis, eccentricity, orbital inclination, right ascension of the ascending node, argument of perigee, and true anomaly. The parameters of the communication equipment are communication frequency, transmission power, and antenna pattern.

4. The low-Earth orbit satellite communication network routing method based on reinforcement learning according to claim 3, characterized in that: Based on the satellite's orbital parameters and the parameters of the communication equipment, the satellite's onboard computer calculates the distance and relative angle between adjacent satellites. If the distance between satellite A and satellite B is less than the maximum communication distance of the communication equipment and the relative angle is within the effective coverage range of the antenna, then satellite B is added to satellite A's list of neighboring satellites. The specific process is as follows: 1) The calculation process for the distance between adjacent satellites is as follows: The satellite's onboard computer calculates the distance between adjacent satellites based on the acquired satellite orbital parameters; For two satellites A and B, the distance d between satellites A and B is given by the Euclidean distance formula. Perform calculations; In the formula, (x A ,y A ,z A (x) represents the position coordinates of satellite A. B ,y B ,z B ( ) represents the position coordinates of satellite B; 2) The calculation process for the relative angle between adjacent satellites is as follows: Assume that the antenna pointing vector of satellite A is The position vector of satellite B relative to satellite A is The relative angle θ between satellite A and satellite B can be expressed by the vector dot product formula. Calculations show that · represents the vector dot product; 3) If the distance between satellite A and satellite B is less than the maximum communication distance d of the communication equipment. max If the relative angle is within the effective coverage area of ​​the antenna, then satellite B will be added to the list of neighboring satellites of satellite A. Among them, the maximum communication distance d of the communication equipment max for: Among them, P r For received power, P t For transmission power, G t G r Here, λ represents the gain of the transmitting and receiving antennas, d represents the wavelength, d represents the communication distance, and L represents the loss. By setting the minimum acceptable receiving power, the maximum communication distance d can be derived. max .

5. A low-Earth orbit satellite communication network routing method based on reinforcement learning according to claim 4, characterized in that: The reward function R in step three t The formula for (s,a) is: R t (s,a)=R arrive +w1×R queue +w2×R distance +w3×R penalty In the formula, R arrive This indicates the arrival reward, which is triggered when the data block successfully reaches the target satellite and is set to 1; otherwise, it is set to 0. R queue Indicates the queue status reward; R distance Indicates distance reward; R penalty This indicates a comprehensive penalty item; w1 represents the queue state reward R. queue The weighting coefficient is set to 20. w2 represents the distance bonus R. distance The weighting coefficient is set to 20. w3 represents the comprehensive penalty item R. penalty The weighting coefficient is set to 1.

6. A routing method for low-Earth orbit satellite communication networks based on reinforcement learning according to claim 5, characterized in that: The arrival award R arrive The formula is: R arrive =d arrive ×R a In the formula, δ arrive The arrival determination indicator function takes a value of 1 when action a causes the data packet to reach the target satellite, and a value of 0 otherwise. R a To reach the base value for the reward.

7. A low-Earth orbit satellite communication network routing method based on reinforcement learning according to claim 6, characterized in that: The queue status reward R queue The formula is: In the formula, q neighbor This indicates the number of data packets waiting to be forwarded in the selected neighboring satellites; q current This indicates the number of data packets currently queued for forwarding in the satellite; q max q represents the maximum number of data packets allowed to be queued within a satellite. max =10; This represents the current queue penalty coefficient.

8. A routing method for low-Earth orbit satellite communication networks based on reinforcement learning according to claim 7, characterized in that: The distance reward R distance For path optimization, the formula is: In the formula, d(s) is the distance from the current state to the target satellite; d(s′) represents the distance from the target satellite to the new state after the action is performed; d max This represents the maximum inter-satellite link distance within the constellation. The comprehensive penalty item R penalty As a constraint guarantee, the formula is: R penalty =d again ×P a +d unav ×P u In the formula, δ again The indicator function for the action selected by the ε-greedy policy. δ unav Invalid action indicator function, P a For loop penalty, P a =-10; P u For invalid actions, P u =-10.

9. A routing method for low-Earth orbit satellite communication networks based on reinforcement learning according to claim 8, characterized in that: The reward function R t (s,a) Updates the Q value, and then updates the Q-table based on the updated Q value; the specific process is as follows: The Q value is updated based on the time difference (TD), using the following formula: Q*(s t ,a t )←Q(s t ,a t )+α[r t +γmax a Q(s t+1 ,a)-Q(s t ,a t )] In the formula, α is the learning rate, set to 0.25; γ is the discount factor, set to 0.9; r t The reward function R is the action at time step t. t (s,a); Q(s t ,a t ) represents state s t Choose a below t Q value; Q*(s t ,a t () represents the updated Q value; Q(s t+1 a) represents state s t+1 Choose a below t Q value; max a Q(s t+1 a) is the maximum Q value in the next state; Q*(s) of the satellite based on the updated time step t t ,a t Update the Q-table.

10. A routing method for low-Earth orbit satellite communication networks based on reinforcement learning according to claim 9, characterized in that: In step six, the optimal Q-table is deployed on the satellite. For any source satellite receiving data packets, the satellite communication network route of the data packets to the target satellite is obtained. The specific process is as follows: Step 61: The source satellite receives data packets and obtains the satellite's status S. t ; Step 62: Find the action corresponding to the maximum Q value of the state in the final Q-table, and execute the action to obtain the new state; Step 63: Find the action corresponding to the maximum Q value of the new state in Step 62 in the final Q-table, and execute the action to obtain the new state; Repeat steps 62 to 63 until the data packet reaches the target satellite, thus obtaining the satellite communication network route of the data packet from the source satellite to the target satellite.

Images