Load balancing intelligent routing method for low-orbit satellite networks based on DQN

Abstract

The present invention relates to the technical field of low-orbit satellite network communications, and more specifically, to a load-balanced intelligent routing method for low-orbit satellite networks based on a distributed query network (DQN). The method adopts a time slicing method to learn the node connection relationship and load conditions in the static topology of each time slot, jointly optimizes multiple objectives such as the distance between target satellites, node congestion, and link available bandwidth, provides an optimized next-hop selection strategy for each node, updates the link connectivity matrix for different time slots, and continues training to update network parameters. The method has better performance in terms of latency and network throughput, reduces the reuse of satellite nodes in the network, alleviates the problem of uneven load in the network, and improves network performance.

Description

Technical field:

[0001] The present invention relates to the field of low-orbit satellite network communication technology. Specifically, it is a DQN-based low-orbit satellite network load balancing intelligent routing method that combines the characteristics of low-orbit satellite networks with the advantages of DQN, comprehensively considers inter-satellite link connectivity and link load conditions, and jointly optimizes multiple objectives to improve network performance and reduce latency. Background technology:

[0002] With the rapid development and widespread adoption of low-Earth Orbit (LEO) satellite networks, their importance in global communication coverage and data transmission has become increasingly prominent. However, due to the unique characteristics of Low Earth Orbit (LEO) satellite networks, such as high-speed operation, limited channel capacity, and high latency, network load balancing and routing scheduling face challenges. Current LEO satellite network routing methods mostly rely on traditional methods such as Dijkstra's shortest path, which lack real-time adaptability and intelligence to dynamic network changes.

[0003] Since traditional satellite routing methods lack real-time adaptability and intelligence to dynamic changes in the network, in order to cope with the challenges brought by the complex network structure and the increasing amount of data in satellite communications in the future, more scholars are focusing on novel satellite routing methods. Low-orbit satellite network routing methods are mainly used to solve the problem of efficient and reliable end-to-end data transmission in LEO satellite networks. In this problem, the routing method plans an optimal path in the satellite network, which requires moving data packets from the source node to the destination node. The measurement of the optimal path can be based on various indicators, such as delay, link bandwidth, link stability, link status, satellite load, number of hops, etc. Due to the highly dynamic changes in the satellite network topology, traditional routing technology has the following disadvantages: (1) It cannot fundamentally avoid link switching and connection switching and the series of switching control and rerouting calculation problems caused by them. (2) The computational overhead is relatively large and difficult to implement on the satellite, and generally requires auxiliary calculations from the ground system. (3) To achieve integration with the ground IP network, a series of intermediate processes such as protocol conversion and data format conversion are required, which will bring additional time and processing overhead, making the system implementation more complex. Therefore, the current research on low-orbit satellite routing protocols mainly focuses on non-connected routing methods.

[0004] In recent years, with the continuous development of machine learning (ML) algorithms and artificial intelligence (AI), the combination of machine learning algorithms and traditional algorithms has produced many good results, bringing new directions for improving algorithm performance. Summary of the invention:

[0005] In view of the shortcomings and deficiencies in the prior art, the present invention proposes a load balancing intelligent routing method for low-orbit satellite networks based on DQN, which can improve network performance and reduce latency.

[0006] The present invention is achieved by the following measures:

[0007] A load-balanced intelligent routing method for low-orbit satellite networks based on DQN is characterized in that, in combination with the satellite routing problem, the routing process is formalized as an MDP, and the state, behavior and reward representation of the agent are first designed, wherein, (1) State: The observed state consists of three parts, including the satellite node where the data packet is currently located, the load status of the adjacent satellite links, and the connectivity and distance between the satellite nodes, which are divided into three state matrices: position relationship matrix, load matrix and distance matrix, and merged into one state; (2) Action: The action is defined as an output interface selected by the data packet. Considering a grid-like satellite network, each node is connected to four adjacent nodes, which have regular directions: east, west, south and north, representing the four output directions respectively. At the same time, the real-time connectivity of the four links is also considered for judgment, and different output values ​​are given; (3) Reward: The definition of reward depends on the set goal. The most important goal to be achieved is to reduce the delay of data transmission and reduce the congestion of each link and node. The objective function is as follows:

[0008] reward=-ω1*d n -ω2*C n +ω3*B n (13),

[0009] Among them, reward means reward, d n Indicates the distance from node n to the destination node, C n represents the congestion degree of node n, C n =q n / Q m ,q n Indicates the queue length of the current node n, Q m Indicates the maximum queue length, B n represents the remaining available bandwidth of the link, ω1, ω2, and ω3 represent weight coefficients, ω1+ω2+ω3=1, and the weight coefficients can be adjusted according to network conditions. The algorithm process is divided into two stages: offline learning and online decision-making. The offline stage of the DQN routing model uses historical traffic data and simulation data for learning: first, the data is preprocessed, then each hop action is selected according to the reward function, and then each hop action and its before and after states are stored. The states include multi-dimensional matrices such as connectivity, node congestion, and link load. Finally, a multi-channel convolutional neural network (CNN) is used for learning.

[0010] The online phase of the present invention first calculates the connection status and distance between each node in the satellite network within a time slot and transmits network topology information to each satellite node, sets the number and size of data packets, initializes the state of each node, and distributes an initial load that obeys a Poisson distribution; then, starting from the source node, a request is sent to the satellite node with which a link is established to obtain the load status of each link and the distance from the node to the destination node, calculates the objective function value based on the link load and distance, and selects the optimal value as the next hop node; thereafter, the above process is continuously repeated until the destination node is reached, and at the same time, when entering the next time slot, the network topology information is updated, the state matrix is ​​updated, the current state action is stored, and learning is continued, experience replay is performed, and network parameters are updated to obtain a more optimal routing decision under this network state.

[0011] The DQN routing model in this invention is described as a Markov decision process (MDP), which is a four-tuple (s, a, s', r). These are all decisions about the agent to maximize the cumulative reward, where s refers to the state space that the agent can observe, a represents the action space related to the decision, s' refers to the next state space reached after action a, and r represents the reward obtained by the agent. With the help of the function approximation ability of deep neural networks, more complex decision-making problems with high-dimensional inputs can be solved. Among them, the action value function (Q function) represents the expected return obtained by taking behavior a according to state s under the guidance of strategy π. Strategy π represents the mapping from state to behavior, which is equivalent to the decision-making strategy of the agent and selects different behaviors according to different states, that is, a = π(s). Under the guidance of strategy π, the Q function of the agent is defined as follows:

[0012] Q(s,a)=E π [G t |S t =s,A t =a] (9),

[0013] Among them G t represents the discount reward, which is defined as follows:

[0014]

[0015] Where γ is the discount factor, that is, the value ratio of the expected reward in the future at the current moment, γ t As the training process iteratively decreases, γ t The smaller it is, the less impact future rewards have on current rewards. The Bellman equation of the Q function is expressed as follows:

[0016]

[0017] Where R(s t ,a t ) indicates that the agent is in st When taking a t The instant reward obtained is expressed as follows by selecting the maximum action value function to solve the optimal behavior strategy:

[0018]

[0019] The DQN algorithm constructs the labels for algorithm training through the reward value of the behavior, and the experience replay and target network effectively solve the problems of data correlation and non-static distribution.

[0020] The model scenario of the present invention considers a small low-orbit satellite constellation and deploys an 8×8 single-layer mesh constellation. Using the time slicing method, the network topology structure within the same time slot is considered to be fixed. The mesh structure assumes that each satellite node has at most two co-orbit links and two adjacent orbit links. On this basis, the connectivity between satellites is updated considering the influence of polar regions and dynamic topology. Here, routing decisions within different time slots are considered, and the time length is set to update the network topology. The data flow from the source satellite node to the destination satellite node through each hop routing selection. During the process, each satellite node can obtain the link connection status of the time slot and the distance to each node.

[0021] Since the communication occurs in a free space environment, inter-satellite communication is mainly affected by free space path loss (FSPL) and additive white Gaussian noise (AWGN). k,i To the next hop v k,i+1 A single data packet, satellite v k,i With v k,i+1 The FSPL between can be expressed as

[0022]

[0023] in, Indicates link e k,i,i+1 length, f is the carrier frequency, c is the speed of light, and satellite v k,i With v k,i+1 The signal-to-noise ratio can be expressed as

[0024]

[0025] Among them, P tr is the transmission power, G tr and G r are the gains of the satellite transmitting and receiving antennas, respectively, and k B is the Boltzmann constant, T e is the equivalent noise temperature, B is the channel bandwidth. Then the satellite v k,i With satellite v k,i+1 The maximum achievable data transfer rate between can be expressed as:

[0026]

[0027] Consider alternative path P k The total end-to-end delay of the pth packet after transmission can be expressed as

[0028]

[0029] in, Indicates that the pth data packet is on link e k,i,i+1 The required propagation delay on is the node v in the path k,i The transmission delay at p size is the size of a single data packet; For data packet p at node v k,i The required queuing delay at Obeys Poisson distribution. If the kth path is selected for data transmission in the routing decision cycle, the total number of data packets forwarded from the source node to the destination node can be expressed as in When selecting path k, the source node v k,s The data transmission rate is , then the packet loss rate calculation of the routing decision cycle can be expressed as

[0030] loss k =1-pac k / n k (5),

[0031] Among them, pac k When path k is selected in the routing decision cycle, it reaches v k,d The number of packets.

[0032] The constraints of the satellite network of the present invention are as follows:

[0033] (1) Bandwidth constraint: To ensure that data can be transmitted normally in the link, the bandwidth required by the task must not exceed the minimum available bandwidth of all links on the path:

[0034] B P ≤min(B ij ,B jk ,…,B pq ) (6),

[0035] Among them, B P Indicates the bandwidth required for the task, and the bandwidth of each sub-link is represented by B ij express;

[0036] (2) Link connectivity constraints:

[0037] Within the same time slot, the network topology can be considered fixed, and the link connectivity can be determined based on the link duration. To ensure that the link remains connected during data transmission, the minimum duration of all links on the path must be no less than the set time slot length:

[0038] min(c ij ,c jk ,…,c pq )≥T(7), where T is the time slot and the bandwidth of each sub-link is c ij express;

[0039] (3) Transmission rate constraints:

[0040] The task transmission rate requirement cannot exceed the set maximum transmission rate:

[0041] R p ≤R m (8), where R p represents the task transmission rate, R m Indicates the set maximum transmission rate.

[0042] The present invention takes into account the high dynamics, long latency, and uneven network load of low-orbit satellite networks, introduces the idea of ​​deep reinforcement learning into the routing scheduling strategy, and proposes a low-orbit satellite network load balancing intelligent routing method based on DQN by combining the characteristics of low-orbit satellite networks and the advantages of DQN. The algorithm adopts the time slicing method to learn the node connection relationship and load conditions in the static topology of each time slot, and jointly optimizes multiple objectives such as the distance between target satellites, node congestion, and link available bandwidth, and provides an optimized next-hop selection strategy for each node. The link connectivity matrix is ​​updated for different time slots, and training is continued to update the network parameters. By comparison with the traditional satellite network routing method, it can be seen that the algorithm proposed by the present invention has better performance in terms of latency and network throughput, reduces the reuse of satellite nodes in the network, alleviates the problem of uneven load in the network, and improves network performance. Description of the drawings:

[0043] Attachment Figure 1 This is a model diagram of a low-orbit satellite network in the present invention.

[0044] Attachment Figure 2 It is a schematic diagram of the DQN structure in the present invention.

[0045] Attachment Figure 3 Schematic diagram of the DQN routing model in the present invention.

[0046] Attachment Figure 4 This is a schematic diagram comparing end-to-end delay performance in the present invention.

[0047] Attachment Figure 5 It is a network throughput performance comparison curve diagram in the present invention.

[0048] Attachment Figure 6 It is a comparison curve diagram of the number of satellites involved in forwarding in the present invention. Specific implementation method:

[0049] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0050] The low-orbit satellite network model of the present invention is shown in the attached figure. Figure 1 As shown, the model scenario currently considers a small low-orbit satellite constellation, deploying an 8×8 single-layer mesh constellation. Due to the highly dynamic nature of low-orbit satellite networks, the network topology changes over time, and the connectivity and distances between satellites are also constantly changing. Therefore, a time-slicing approach is used, assuming that the network topology within a time slot is fixed. The mesh structure assumes that each satellite node has at most two co-orbital links and two adjacent orbital links. Furthermore, inter-satellite connectivity is updated to account for factors such as polar regions and dynamic topology. Routing decisions are considered within different time slots, and a set interval is set to update the network topology. Data flows from the source satellite node through each hop routing selection to the destination satellite node. During this process, each satellite node can obtain the link connectivity status and distance to each node in that time slot.

[0051] Since the communication occurs in a free space environment, inter-satellite communication is mainly affected by free space path loss (FSPL) and additive white Gaussian noise (AWGN). k,i To the next hop v k,i+1 A single data packet, satellite v k,i With v k,i+1 The FSPL between can be expressed as

[0052]

[0053] in, Indicates link e k,i,i+1 length, f is the carrier frequency, and c is the speed of light. Satellite v k,i With v k,i+1 The signal-to-noise ratio can be expressed as

[0054]

[0055] Among them, P tr is the transmission power, G tr and G r are the gains of the satellite transmitting and receiving antennas, respectively, and k B is the Boltzmann constant, T e is the equivalent noise temperature, B is the channel bandwidth. Then the satellite v k,iWith satellite v k,i+1 The maximum data transfer rate that can be achieved between

[0056]

[0057] Consider alternative path P k The total end-to-end delay of the pth packet after transmission can be expressed as

[0058]

[0059] in, Indicates that the pth data packet is on link e k,i,i+1 The required propagation delay on is the node v in the path k,i The transmission delay at p size is the size of a single data packet; For data packet p at node v k,i The required queuing delay at Obeys Poisson distribution. If the kth path is selected for data transmission in the routing decision cycle, the total number of data packets forwarded from the source node to the destination node can be expressed as in When selecting path k, the source node v k,s The data transmission rate is , then the packet loss rate calculation of the routing decision cycle can be expressed as

[0060] loss k =1-pac k / n k (5),

[0061] Among them, pac k When path k is selected in the routing decision cycle, it reaches v k,d The number of packets.

[0062] The common constraints of the satellite network in the present invention are as follows:

[0063] (1) Bandwidth Constraints

[0064] To ensure that data can be transmitted normally on the link, the bandwidth required by the task must not exceed the minimum available bandwidth of all links on the path:

[0065] B P ≤min(B ij ,B jk ,…,B pq ) (6),

[0066] Among them, B P Indicates the bandwidth required for the task, and the bandwidth of each sub-link is represented by B ij express.

[0067] (2) Link connectivity constraints

[0068] Within the same time slot, the network topology can be considered fixed, and the link connectivity can be determined based on the link duration. To ensure that the link remains connected during data transmission, the minimum duration of all links on the path must be no less than the set time slot length:

[0069] min(c ij ,c jk ,…,c pq )≥T (7),

[0070] Where T is the time slot, and the bandwidth of each sub-link is c ij express.

[0071] (3) Transmission rate constraints

[0072] The task transmission rate requirement cannot exceed the set maximum transmission rate:

[0073] R p ≤R m (8),

[0074] Among them, R p represents the task transmission rate, R m Indicates the set maximum transmission rate.

[0075] The DQN satellite routing model in this invention is as follows: The most critical task of the routing strategy is to give a route from the source node v s To the destination node v d A routing path that meets traffic requirements, such as end-to-end latency. The routing problem in satellite networks can generally be described as follows: The network controller observes the current network state, including link congestion information and the satellite network topology. It then determines the next hop for all packets to be sent and sends these packets to other nodes. The network controller then receives updated network information. Based on this information, it selects the next hop for the packets. The action taken by each node is dependent only on the current state of the network and not on historical statistics. The DQN routing model can generally be described as a Markov decision process (MDP), which is a four-tuple (s, a, s', r). These decisions are about the agent making decisions to maximize the cumulative reward, where s refers to the state space the agent can observe, a represents the action space associated with the decision, s' refers to the next state space reached after taking action a, and r represents the reward received by the agent. The function approximation capabilities of deep neural networks can be used to solve more complex decision-making problems with high-dimensional inputs.

[0076] The action-value function (Q-function) represents the expected reward from taking action a based on state s under the guidance of policy π. Policy π represents the mapping from state to action, which is equivalent to the agent's decision-making strategy. It selects different actions based on different states, that is, a = π(s). Under the guidance of policy π, the Q-function is defined as follows:

[0077] Q(s,a)=E π [G t |S t =s,A t =a] (9),

[0078] Among them G t represents the discount reward, which is defined as follows:

[0079]

[0080] Where γ is the discount factor, which is the ratio of the expected future reward to the current value. t As the training process iteratively decreases, γ t The smaller the value, the less impact future rewards have on current rewards. The Bellman equation for the Q function is as follows:

[0081]

[0082] Where R(s t ,a t ) indicates that the agent is in s t When taking a t The immediate reward obtained. The formula for solving the optimal behavior strategy by selecting the maximum action value function is as follows:

[0083]

[0084] The DQN algorithm constructs labels for algorithm training based on the reward values ​​of actions. Its experience replay and target network effectively address the issues of data correlation and non-stationary distribution. The following is a schematic diagram of the DQN algorithm's structure.

[0085] The DQN routing model process based on load balancing is as follows Figure 3 shown.

[0086] In combination with the satellite routing problem, the routing process is formalized as an MDP. It is necessary to first design the state, behavior and reward representation of the agent. The design of state, action and reward is crucial to the success of the deep reinforcement learning method. (1) State: The observed state consists of three parts, including the satellite node where the data packet is currently located, the load status of the adjacent satellite links, and the connectivity and distance between the satellite nodes. It is divided into three state matrices: position relationship matrix, load matrix and distance matrix, and merged into one state. (2) Action: The action is defined as an output interface for data packet selection. Consider a grid-like satellite network, where each node is connected to four adjacent nodes, which have regular directions: east, west, south and north, representing the four output directions respectively. At the same time, the real-time connectivity of the four links is also considered for judgment, and different output values ​​are given. (3) Reward: The definition of reward depends on the goal we set. The most important goal we should achieve is to reduce the delay of data transmission and reduce the congestion of each link and node. The objective function is as follows:

[0087] reward=-ω1*d n -ω2*C n +ω3*B n (13),

[0088] Among them, reward means reward, d n Represents the distance from node n to the destination node. C n represents the congestion degree of node n, C n =q n / Q m ,q n Indicates the queue length of the current node n, Q m Indicates the maximum queue length. B n Indicates the remaining available bandwidth of the link. ω1, ω2, and ω3 are weight coefficients. ω1+ω2+ω3=1. The weight coefficient can be adjusted according to the network conditions.

[0089] The algorithm flow of the present invention is broadly divided into two phases: offline learning and online decision-making. The offline phase of the DQN routing model primarily utilizes historical traffic data and simulated data for learning. Data is first preprocessed, and then each hop action is selected based on a reward function. Each hop action and its preceding and following states are stored. The states include multi-dimensional matrices such as connectivity, node congestion, and link load. Finally, learning is performed using a multi-channel convolutional neural network (CNN). The online phase first calculates the connectivity and distance between nodes in the satellite network within a time slot and transmits network topology information to each satellite node. The number and size of data packets are set, the state of each node is initialized, and an initial load following a Poisson distribution is assigned. The source node then sends requests to satellite nodes with established links to obtain the load of each link and the distance from the node to the destination node. The objective function value is calculated based on the link load and distance, and the optimal value is selected as the next hop node. This process is repeated until the destination node is reached. At the next time slot, the network topology information and state matrix are updated, and information such as the current state and action are stored for continued learning. Experience is replayed and network parameters are updated to obtain the most optimal routing decision for this network state. See Algorithm 1 for the specific algorithm flow.

[0090]

[0091]

[0092] Example:

[0093] Using STK, a single-layer LEO constellation of 64 satellites was generated, with eight orbits, eight satellites per orbit, an orbital altitude of 895.5 km, and an inclination of 86.4°. STK functions were then used to calculate the connection status and distances between satellite nodes, generating link and distance matrices. Because the network's multidimensional information needs to be considered, multiple channels are used to represent the network state as a multidimensional matrix, which serves as the model input. The neural network in the DQN model uses a convolutional neural network to better utilize the matrix's computational power to process multidimensional resources and extract features. The CNN model includes two convolutional layers and two fully connected layers, with 16 and 32 kernels. The simulation parameters are shown in Table 1. The link channel capacity is set to 100 Mb / s, the maximum data rate for each task is 2 Mb / s, the initial load of each node follows a Poisson distribution, and the packet size is set to 512 bits. The source and destination nodes of the tasks are set.

[0094] Table 1 Simulation parameters

[0095]

[0096]

[0097] In the simulation, the performance of this routing method is compared with the traditional Dijkstra shortest path algorithm and the maximum flow algorithm. Figure 4 The performance differences of the three algorithms in terms of total end-to-end delay are shown. It can be seen from the simulation diagram that the proposed algorithm has a lower end-to-end delay. When the number of data packets is small, the delay performance of the three algorithms is close, because the congestion in the network is not serious at this time, and the main factors are propagation and transmission delays. As the number of data packets increases, the congestion of the shortest path algorithm becomes more and more obvious, and the queuing time in the node gradually increases, resulting in a significant increase in the total delay. Although the maximum flow algorithm can select alternative paths to appropriately alleviate congestion, the selection of some routes comes at the cost of increasing the path length, which increases the propagation delay, and the total delay performance is slightly better than the shortest path algorithm. The DQN-based load balancing routing method proposed in the present invention has better performance in terms of end-to-end delay.

[0098] Figure 5 The performance differences of the three algorithms in terms of network throughput are shown. It can be seen from the simulation diagram that the proposed algorithm has a higher network throughput. When the number of data packets is small, the throughput performance of the three algorithms is close, because the network load is low at this time, and there is no congestion when transmitting data packets. As the number of data packets increases, the congestion of the shortest path algorithm becomes more and more obvious due to the limitation of link capacity. It reaches and maintains the maximum value first, and the network throughput is the lowest. The maximum flow algorithm has a higher network throughput because it can find an optimal traffic distribution plan in the network, so that the total traffic in the network reaches the maximum. However, due to its higher latency, its network throughput peak is slightly lower than the algorithm proposed in the present invention. The algorithm continuously searches for nodes with lower congestion according to changes in network load, expands alternative paths, averages the network load to more nodes, reduces the number of single-node reuses, and relieves network pressure. It can be seen that the load balancing routing method based on DQN proposed in the present invention also has good performance in terms of network throughput.

[0099] Figure 6The three algorithms differ in the number of satellites involved in forwarding packets, a metric that primarily reflects their load balancing performance. A greater number of satellites involved in forwarding packets reduces the number of satellite nodes reused, allowing for the discovery of more satellite nodes and routing paths, thus reducing network load balancing. The simulation graphs show that the proposed algorithm utilizes a larger number of satellites for forwarding packets and continuously expands new nodes as the number of packets increases, reducing the reuse of older nodes and alleviating network congestion. Since the shortest path algorithm only considers the shortest path, its forwarding satellites only include nodes on that path, leading to link congestion, a large packet backlog, and high transmission and queuing delays. The maximum flow algorithm, however, is constrained by network traffic. When the current traffic on a link reaches its maximum capacity, the path containing that edge cannot be used as an augmenting path and expand new satellite nodes. As data volume increases, the network traffic near the destination node approaches saturation, and the maximum flow algorithm no longer includes new nodes for forwarding packets. This demonstrates that the DQN-based load balancing routing method proposed in this paper utilizes a larger number of satellites for forwarding packets and offers superior load balancing capabilities.

[0100] The present invention takes into account the high dynamics, long latency, and uneven network load of low-orbit satellite networks, introduces the idea of ​​deep reinforcement learning into the routing scheduling strategy, and proposes a low-orbit satellite network load balancing intelligent routing method based on DQN by combining the characteristics of low-orbit satellite networks and the advantages of DQN. The algorithm adopts the time slicing method to learn the node connection relationship and load conditions in the static topology of each time slot, and jointly optimizes multiple objectives such as the distance between target satellites, node congestion, and link available bandwidth, and provides an optimized next-hop selection strategy for each node. The link connectivity matrix is ​​updated for different time slots, and training is continued to update the network parameters. By comparison with the traditional satellite network routing method, it can be seen that the algorithm proposed by the present invention has better performance in terms of latency and network throughput, reduces the reuse of satellite nodes in the network, alleviates the problem of uneven load in the network, and improves network performance.

Claims

1. A low-orbit satellite network load balancing intelligent routing method based on DQN, characterized in that: Combined with the satellite routing problem, the routing process is formalized as an MDP. The state, behavior and reward representation of the agent are first designed, where: (1) State: The observed state consists of three parts, including the satellite node where the data packet is currently located, the load of the adjacent satellite links, and the connectivity and distance between the satellite nodes. It is divided into three state matrices: position relationship matrix, load matrix and distance matrix, and merged into one state; (2) Action: The action is defined as an output interface selected by the data packet. Considering a grid-like satellite network, each node is connected to four adjacent nodes, which have regular directions: east, west, south and north, representing the four output directions respectively. At the same time, the real-time connectivity of the four links is also considered for judgment, and different output values ​​are given; (3) Reward: The definition of reward depends on the set goal. The most important goal to be achieved is to reduce the delay of data transmission and reduce the congestion of each link and node. The objective function is as follows: reward=-ω1*d n -ω2*C n +ω3*B n (13), Among them, reward means reward, d n Indicates the distance from node n to the destination node, C n represents the congestion degree of node n, C n =q n / Q m ,q n Indicates the queue length of the current node n, Q m Indicates the maximum queue length, B n Indicates the remaining available bandwidth of the link. ω1, ω2, and ω3 are weight coefficients. ω1+ω2+ω3=1. The weight coefficient can be adjusted according to the network conditions. The algorithm process is divided into two stages: offline learning and online decision-making. The offline stage of the DQN routing model uses historical traffic data and simulation data for learning: first, the data is preprocessed, then each hop action is selected based on the reward function, and then each hop action and its previous and subsequent states are stored. The state includes connectivity, node congestion, and a multi-dimensional matrix of link load. Finally, a multi-channel convolutional neural network (CNN) is used for learning. In the online phase, the connection status and distance between nodes in the satellite network are first calculated within the time slot and the network topology information is transmitted to each satellite node. The number and size of data packets are set, the state of each node is initialized, and an initial load following a Poisson distribution is assigned. Then, the source node sends a request to the satellite node with which it has established a link to obtain the load status of each link and the distance from the node to the destination node. The objective function value is calculated based on the link load and distance, and the optimal value is selected as the next hop node. Thereafter, the above process is repeated until the destination node is reached. At the same time, when entering the next time slot, the network topology information is updated, the state matrix is ​​updated, the current state action is stored, and learning continues. Experience is replayed and network parameters are updated to obtain a more optimal routing decision under this network state. The DQN routing model is described as a Markov decision process (MDP), which is a four-tuple (s, a, s', r). These are all decisions about the agent to maximize the cumulative reward, where s refers to the state space observed by the agent, a represents the action space related to the decision, s' refers to the next state space reached after action a, and r represents the reward obtained by the agent. With the help of the function approximation ability of deep neural networks, more complex decision-making problems with high-dimensional inputs can be solved. Among them, the action value function, namely the Q function, represents the expected return obtained by taking behavior a according to state s under the guidance of strategy π. Strategy π represents the mapping from state to behavior, which is equivalent to the decision-making strategy of the agent and selects different behaviors according to different states, that is, a = π(s). Under the guidance of strategy π, the Q function of the agent is defined as follows: Q(s,a)=E π [G t |S t =s,A t =a] (9), Among them G t represents the discount reward, which is defined as follows: Where γ is the discount factor, that is, the value ratio of the expected reward in the future at the current moment, γ t As the training process iteratively decreases, γ t The smaller it is, the less impact future rewards have on current rewards. The Bellman equation of the Q function is expressed as follows: Where R(s t ,a t ) indicates that the agent is in s t When taking a t The instant reward obtained is expressed as follows by selecting the maximum action value function to solve the optimal behavior strategy: The DQN algorithm constructs the labels of algorithm training through the reward value of the behavior, and the experience replay and target network effectively solve the problems of data correlation and non-static distribution; The model scenario considers a small low-orbit satellite constellation and deploys an 8×8 single-layer mesh constellation. Using the time slicing method, it is assumed that the network topology structure within the same time slot is fixed. The mesh structure assumes that each satellite node has at most two co-orbit links and two adjacent orbit links. On this basis, the connectivity between satellites is updated considering the influence of polar regions and dynamic topology. Here, the routing decision in different time slots is considered, and the time length is set to update the network topology. The data flow from the source satellite node to the destination satellite node through each hop routing selection. During the process, each satellite node can obtain the link connection status of the time slot and the distance to each node. Since the communication occurs in a free space environment, the inter-satellite communication is mainly affected by free space path loss (FSPL) and additive white Gaussian noise (AWGN). Considering the satellite node v k,i To the next hop v k,i+1 A single data packet, satellite v k,i With v k,i+1 The FSPL between in, Indicates link e k,i,i+1 length, f is the carrier frequency, c is the speed of light, and satellite v k,i With v k,i+1 The signal-to-noise ratio can be expressed as Among them, P tr is the transmission power, G tr and G r are the gains of the satellite transmitting and receiving antennas, respectively, and k B is the Boltzmann constant, T e is the equivalent noise temperature, B is the channel bandwidth, then the satellite v k,i With satellite v k,i+1 The maximum achievable data transfer rate between can be expressed as: Consider alternative path P k The total end-to-end delay of the pth packet after transmission can be expressed as in, Indicates that the pth data packet is on link e k,i,i+1 The required propagation delay on is the node v in the path k,i The transmission delay at p size is the size of a single data packet; For data packet p at node v k,i The required queuing delay at Following the Poisson distribution, if the kth path is selected for data transmission in the routing decision cycle, the total number of data packets forwarded from the source node to the destination node can be expressed as in When selecting path k, the source node v k,s The data transmission rate is , then the packet loss rate calculation of the routing decision cycle can be expressed as loss k =1-pac k / n k (5), Among them, pac k When path k is selected in the routing decision cycle, it reaches v k,d The number of packets; The constraints of the satellite network are as follows: (1) Bandwidth constraint: To ensure that data can be transmitted normally in the link, the bandwidth required by the task must not exceed the minimum available bandwidth of all links on the path: B P ≤min(B ij ,B jk ,…,B pq ) (6), Among them, B P Indicates the bandwidth required for the task, and the bandwidth of each sub-link is represented by B ij express; (2) Link connectivity constraints: Within the same time slot, the network topology can be considered fixed, and the link connectivity can be determined based on the link duration. To ensure that the link remains connected during data transmission, the minimum duration of all links on the path must be no less than the set time slot length: min(c ij ,c jk ,…,c pq )≥T (7), where T is the time slot and the bandwidth of each sub-link is c ij express; (3) Transmission rate constraints: The task transmission rate requirement cannot exceed the set maximum transmission rate: R p ≤R m (8), where R p represents the task transmission rate, R m Indicates the set maximum transmission rate.

Images