Low earth orbit satellite network load balancing intelligent routing algorithm based on DQN

Abstract

The invention relates to the technical field of low earth orbit satellite network communication, in particular to a low earth orbit satellite network load balancing intelligent routing algorithm based on a DQN, which adopts a time slicing method to learn node connection relation and load condition in each time slot static topology, according to the method, multiple targets such as the distance between target satellites, the node congestion degree and the link available bandwidth are jointly optimized, an optimized next-hop selection strategy is given for each node, a link connectivity matrix is updated for different time slots, training is continued to update network parameters, and the method has better performance in the aspects of time delay and network throughput, and can be applied to the field of wireless communication. The multiplexing of each satellite node in the network is reduced, the problem of uneven load in the network is relieved, and the network performance is improved.

Description

Technical field:

[0001] The present invention relates to the field of low-orbit satellite network communication technology, and specifically to a DQN-based low-orbit satellite network load balancing intelligent routing algorithm that combines the characteristics of the low-orbit satellite network and the advantages of DQN, comprehensively considers inter-satellite link connectivity and link load conditions, performs joint optimization on multiple objectives, and can improve network performance and reduce latency. Background technology:

[0002] With the rapid development and popularization of low-orbit satellite networks, their importance in global communication coverage and data transmission has become increasingly prominent. However, due to the particularity of low-Earth Orbit (LEO) satellite networks, such as high-speed operation, limited channel capacity, and large latency, network load balancing and routing scheduling face challenges. Current low-orbit satellite network routing algorithms mostly use traditional Dijkstra shortest path methods, which lack real-time adaptability and intelligence to dynamic changes in the network.

[0003] Since traditional satellite routing algorithms lack real-time adaptability and intelligence to dynamic changes in the network, in order to cope with the challenges brought by the complex network structure in the future and the increasing amount of data in satellite communications, more scholars are focusing on novel satellite routing algorithms. Low-orbit satellite network routing algorithms are mainly used to solve the problem of efficient and reliable end-to-end data transmission in LEO satellite networks. In this problem, the routing algorithm plans an optimal path in the satellite network, which requires moving data packets from the source node to the destination node. The measurement of the optimal path can be based on various indicators, such as delay, link bandwidth, link stability, link status, satellite load, number of hops, etc. Due to the highly dynamic changes in the satellite network topology, traditional routing technology has the following disadvantages: (1) It cannot fundamentally avoid link switching and connection switching and the series of switching control and rerouting calculation problems caused by them. (2) The computational overhead is relatively large and difficult to implement on the satellite, and generally requires auxiliary calculations from the ground system. (3) To achieve integration with the ground IP network, a series of intermediate processes such as protocol conversion and data format conversion are required, which will bring additional time and processing overhead, making the system implementation more complicated. Therefore, the current research on low-orbit satellite routing protocols mainly focuses on non-connected routing algorithms.

[0004] In recent years, with the continuous development of machine learning (ML) algorithms and artificial intelligence (AI), the combination of machine learning algorithms and traditional algorithms has produced many good results, bringing new directions for improving algorithm performance. Summary of the invention:

[0005] In view of the shortcomings and deficiencies in the prior art, the present invention proposes a DQN-based low-orbit satellite network load balancing intelligent routing algorithm that can improve network performance and reduce latency.

[0006] The present invention is achieved by the following measures:

[0007] A load balancing intelligent routing algorithm for low-orbit satellite networks based on DQN is characterized in that, in combination with the satellite routing problem, the routing process is formalized as an MDP, and the state, behavior and reward representation of the agent are designed first, where: (1) State: The observed state consists of three parts, including the satellite node where the data packet is currently located, the load status of the adjacent satellite links, and the connectivity and distance between the satellite nodes, which are divided into three state matrices: position relationship matrix, load matrix and distance matrix, and merged into one state; (2) Action: The action is defined as an output interface selected by the data packet. Considering a grid-like satellite network, each node is connected to four adjacent nodes, which have regular directions: east, west, south and north, representing four output directions respectively. At the same time, the real-time connectivity of the four links is also considered for judgment, and different output values ​​are given; (3) Reward: The definition of the reward depends on the set goal. The most important goal to be achieved is to reduce the delay of data transmission and reduce the congestion of each link and node. The objective function is as follows:

[0008] reward=-ω1*d n -ω2*C n +ω3*B n (13)

[0009] Among them, reward means reward, d n represents the distance from node n to the destination node, C n represents the congestion degree of node n, C n =q n / Q m ,q n Indicates the queue length of the current node n, Q m Indicates the maximum queue length, B n represents the remaining available bandwidth of the link, ω1, ω2, ω3 represent weight coefficients, ω1+ω2+ω3=1, and the weight coefficient can be adjusted according to the network conditions; the algorithm process is divided into two stages: offline learning and online decision-making. The offline stage of the DQN routing model uses historical traffic data and simulation data for learning: first, the data is preprocessed, and then each hop action is selected according to the reward function, and then each hop action and its before and after states are stored. The states include multi-dimensional matrices such as connectivity, node congestion, and link load. Finally, a multi-channel convolutional neural network (CNN) is used for learning.

[0010] In the online stage of the present invention, firstly, the connection status and distance between each node in the satellite network are calculated within the time slot and the network topology information is transmitted to each satellite node, the number and size of data packets are set, the state of each node is initialized, and the initial load obeying the Poisson distribution is distributed; then, a request is sent from the source node to the satellite node with which the link is established to obtain the load status of each link and the distance from the node to the destination node, the target function value is calculated according to the link load and the distance, and the optimal value is selected as the next hop node; thereafter, the above process is continuously repeated until the destination node is reached, and at the same time, when entering the next time slot, the network topology information is updated, the state matrix is ​​updated, the current state action is stored to continue learning, and experience replay and network parameter update are performed to obtain a better routing decision under this network state.

[0011] The DQN routing model in the present invention is described as a Markov decision process (MDP), which is a four-tuple (s, a, s', r), which are all about the agent's decision to maximize the cumulative reward, where s refers to the state space that the agent can observe, a represents the action space related to the decision, s' refers to the next state space reached after action a, and r represents the reward obtained by the agent. With the help of the function approximation ability of deep neural networks, more complex decision problems with high-dimensional inputs can be solved. Among them, the action value function (Q function) represents the expected return obtained by taking behavior a according to state s under the guidance of strategy π. Strategy π represents the mapping from state to behavior, which is equivalent to the decision-making strategy of the agent, and selects different behaviors according to different states, that is, a = π (s). Under the guidance of strategy π, the Q function of the robot is defined as follows:

[0012] Q(s,a)=E π [G t |S t =s,A t =a] (9),

[0013] Among them G t Represents a discount reward, which is defined as follows:

[0014]

[0015] Where γ is the discount factor, that is, the value ratio of the expected reward in the future at the current moment, γ t As the training process iterates, γ t The smaller it is, the less impact the future rewards have on the rewards at the current moment. The Bellman equation of the Q function is expressed as follows:

[0016]

[0017] In the formula, R(s t ,a t ) indicates that the agent is in st When taking a t The instant reward obtained is expressed as follows by selecting the maximum action value function to solve the optimal behavior strategy:

[0018]

[0019] The DQN algorithm constructs the labels for algorithm training through the reward values ​​of behaviors, and the experience replay and target network therein effectively solve the problems of data correlation and non-static distribution.

[0020] The model scenario of the present invention considers a small low-orbit satellite constellation and deploys an 8×8 single-layer mesh constellation. The time slicing method is used. It is considered that the network topology structure in the same time slot is fixed. The mesh structure believes that each satellite node has at most two co-orbit links and two adjacent orbit links. On this basis, the polar regions and dynamic topology are considered to affect the connectivity between satellites. The routing decisions in different time slots are considered here, and the duration is set to update the network topology. The data flow from the source satellite node to the destination satellite node through each hop routing selection. During the process, each satellite node can obtain the link connection status of the time slot and the distance to each node.

[0021] Since the communication occurs in a free space environment, the inter-satellite communication is mainly affected by free space path loss (FSPL) and additive white Gaussian noise (AWGN). k,i To the next hop v k,i+1 A single data packet, satellite v k,i With v k,i+1 The FSPL between can be expressed as

[0022]

[0023] in, Indicates link e k,i,i+1 length, f is the carrier frequency, c is the speed of light, and the satellite v k,i With v k,i+1 The signal-to-noise ratio can be expressed as

[0024]

[0025] Where P tr is the transmission power, G tr and G r are the gains of the satellite transmitting and receiving antennas, respectively, and k B is the Boltzmann constant, T e is the equivalent noise temperature, and B is the channel bandwidth. Then the satellite v k,i With satellite v k,i+1 The maximum achievable data transfer rate between can be expressed as:

[0026]

[0027] Consider alternative path P k The total end-to-end delay of the pth packet after transmission can be expressed as

[0028]

[0029] in, Indicates that the pth data packet is on link e k,i,i+1 The required propagation delay on

[0030] is the node v in the path k,i The transmission delay at p size is the size of a single packet; For data packet p at node v k,i The required queuing delay at It obeys Poisson distribution. If the kth path is selected for data transmission in the routing decision cycle, the total number of data packets forwarded from the source node to the destination node can be expressed as in When selecting path k, the source node v k,s The data transmission rate is , then the packet loss rate calculation of the routing decision cycle can be expressed as

[0031] loss k =1-pac k / n k (5)

[0032] Among them, pac k When path k is selected in the routing decision cycle, it reaches v k,d The number of packets.

[0033] The constraints of the satellite network of the present invention are as follows:

[0034] (1) Bandwidth constraint: To ensure that data can be transmitted normally in the link, the bandwidth required by the task must not exceed the minimum value of the available bandwidth of all links on the path:

[0035] B P ≤min(B ij ,B jk ,…,B pq ) (6),

[0036] Among them, B P Indicates the bandwidth required for the task, and the bandwidth of each sub-link is represented by B ij express;

[0037] (2) Link connectivity constraints:

[0038] In the same time slot, the network topology can be considered fixed, and the link connection status can be determined based on the link duration. To ensure that the link remains connected during data transmission, the minimum duration of all links on the path is required to be no less than the set time slot length:

[0039] min(c ij ,c jk ,…,c pq )≥T(7), where T is the time slot and the bandwidth of each sub-link is c ij express;

[0040] (3) Transmission rate constraints:

[0041] The task transmission rate requirement cannot exceed the set maximum transmission rate:

[0042] R p ≤R m (8), where R p represents the task transmission rate, R m Indicates the set maximum transmission rate.

[0043] The present invention takes into account the high dynamics, long latency, and uneven network load of low-orbit satellite networks, introduces the idea of ​​deep reinforcement learning in the routing scheduling strategy, and proposes a low-orbit satellite network load balancing intelligent routing algorithm based on DQN by combining the characteristics of low-orbit satellite networks and the advantages of DQN. The algorithm adopts the time slicing method to learn the node connection relationship and load conditions in the static topology of each time slot, and jointly optimizes multiple objectives such as the distance between target satellites, node congestion, and link available bandwidth, and provides an optimized next-hop selection strategy for each node. The link connectivity matrix is ​​updated for different time slots, and training continues to update network parameters. By comparing with the traditional satellite network routing algorithm, it can be seen that the algorithm proposed by the present invention has better performance in latency and network throughput, reduces the reuse of satellite nodes in the network, alleviates the problem of uneven load in the network, and improves network performance. Description of the drawings:

[0044] Attached Figure 1 This is a low-orbit satellite network model diagram of the present invention.

[0045] Attached Figure 2 It is a schematic diagram of the DQN structure in the present invention.

[0046] Attached Figure 3 It is a schematic diagram of the DQN routing model in the present invention.

[0047] Attached Figure 4 It is a schematic diagram of end-to-end delay performance comparison in the present invention.

[0048] Attached Figure 5 It is a network throughput performance comparison curve diagram in the present invention.

[0049] Attached Figure 6 It is a comparison curve chart of the number of satellites involved in forwarding in the present invention. Specific implementation method:

[0050] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0051] The low-orbit satellite network model of the present invention is shown in the attached figure. Figure 1 As shown in the figure, the model scenario currently considers a small low-orbit satellite constellation and deploys an 8×8 single-layer mesh constellation. Due to the high dynamics of the low-orbit satellite network, the network topology will change over time, and the connection relationship and distance between satellites are also constantly changing. Therefore, the time slicing method is adopted, and the network topology structure in the same time slot is considered to be fixed. The mesh structure assumes that each satellite node has at most two co-orbit links and two adjacent orbit links. On this basis, the polar region and dynamic topology are considered to affect the update of the connectivity between satellites. Here, the routing decisions in different time slots are considered, and the duration is set to update the network topology. The data flow from the source satellite node to the destination satellite node through each hop routing selection, during which each satellite node can obtain the link connection status of the time slot and the distance to each node.

[0052] Since the communication takes place in a free space environment, inter-satellite communication is mainly affected by free space path loss (FSPL) and additive white Gaussian noise (AWGN). k,i To the next hop v k,i+1 A single data packet, satellite v k,i With v k,i+1 The FSPL between can be expressed as

[0053]

[0054] in, Indicates link e k,i,i+1 length, f is the carrier frequency, and c is the speed of light. Satellite v k,i With v k,i+1 The signal-to-noise ratio can be expressed as

[0055]

[0056] Where P tr is the transmission power, G tr and G r are the gains of the satellite transmitting and receiving antennas, respectively, and k B is the Boltzmann constant, T eis the equivalent noise temperature, and B is the channel bandwidth. Then the satellite v k,i With satellite v k,i+1 The maximum data transfer rate that can be achieved between

[0057]

[0058] Consider alternative path P k The total end-to-end delay of the pth packet after transmission can be expressed as

[0059]

[0060] in, Indicates that the pth data packet is on link e k,i,i+1 The required propagation delay on is the node v in the path k,i The transmission delay at p size is the size of a single packet; For data packet p at node v k,i The required queuing delay at It obeys Poisson distribution. If the kth path is selected for data transmission in the routing decision cycle, the total number of data packets forwarded from the source node to the destination node can be expressed as in When selecting path k, the source node v k,s The data transmission rate is , then the packet loss rate calculation of the routing decision cycle can be expressed as

[0061] loss k =1-pac k / n k (5)

[0062] Among them, pac k When path k is selected in the routing decision cycle, it reaches v k,d The number of packets.

[0063] The common constraints of the satellite network in the present invention are as follows:

[0064] (1) Bandwidth Constraints

[0065] To ensure that data can be transmitted normally in the link, the bandwidth required by the task must not exceed the minimum value of the available bandwidth of all links on the path:

[0066] B P ≤min(B ij ,B jk ,…,B pq ) (6),

[0067] Among them, B PIndicates the bandwidth required for the task, and the bandwidth of each sub-link is represented by B ij express.

[0068] (2) Link Connectivity Constraints

[0069] In the same time slot, the network topology can be considered fixed, and the link connection status can be determined based on the link duration. To ensure that the link remains connected during data transmission, the minimum duration of all links on the path is required to be no less than the set time slot length:

[0070] min(c ij ,c jk ,…,c pq )≥T(7),

[0071] Where T is the time slot and the bandwidth of each sub-link is c ij express.

[0072] (3) Transmission rate constraints

[0073] The task transmission rate requirement cannot exceed the set maximum transmission rate:

[0074] R p ≤R m (8)

[0075] Among them, R p represents the task transmission rate, R m Indicates the set maximum transmission rate.

[0076] The DQN satellite routing model in the present invention is as follows: The most critical task of the routing strategy is to provide a route from the source node v s To the destination node v d The routing problem in satellite networks can be generally described as follows: the network controller observes the current network status, including link congestion information and satellite network topology. Then, it decides the next hop for all packets to be sent and sends these packets to other nodes. After that, the network controller obtains updated network information. Based on this information, it selects the next hop for the packet. The action taken by each node is only related to the current state of the network, not to historical statistics. The DQN routing model can generally be described as a Markov decision process (MDP), which is a four-tuple (s, a, s', r). These are all about the agent making decisions to maximize the cumulative reward, where s refers to the state space that the agent can observe. a represents the action space related to the decision, s' refers to the next state space reached after action a, and r represents the reward received by the agent. With the function approximation ability of deep neural networks, more complex decision problems with high-dimensional inputs can be solved.

[0077] The action value function (Q function) represents the expected return obtained by taking action a according to state s under the guidance of strategy π. Strategy π represents the mapping from state to behavior, which is equivalent to the decision-making strategy of the intelligent agent, and chooses different behaviors according to different states, that is, a = π(s). Under the guidance of strategy π, the Q function of the robot is defined as follows:

[0078] Q(s,a)=Eπ[G t |S t =s,A t =a](9),

[0079] Among them G t Represents a discount reward, which is defined as follows:

[0080]

[0081] Where γ is the discount factor, which is the value ratio of the expected reward in the future at the current moment. t As the training process iterates, γ t The smaller it is, the less impact future rewards have on current rewards. The Bellman equation of the Q function is expressed as follows:

[0082]

[0083] In the formula, R(s t ,a t ) indicates that the agent is in s t When taking a t The instant reward obtained. The formula for solving the optimal behavior strategy by selecting the maximum action value function is as follows:

[0084]

[0085] The DQN algorithm constructs the labels for algorithm training through the reward value of the behavior, and the experience replay and target network effectively solve the problems of data correlation and non-static distribution. The structural diagram of the DQN algorithm is as follows.

[0086] The DQN routing model process based on load balancing is as follows Figure 3 shown.

[0087] Combined with the satellite routing problem, the routing process is formalized as an MDP. It is necessary to first design the agent's state, behavior and reward representation. The design of state, action and reward is crucial to the success of deep reinforcement learning methods. (1) State: The observed state consists of three parts, including the satellite node where the data packet is currently located, the load of the adjacent satellite links, and the connectivity and distance between the satellite nodes. It is divided into three state matrices: position relationship matrix, load matrix and distance matrix, and merged into one state. (2) Action: The action is defined as an output interface selected by the data packet. Consider a grid-like satellite network, where each node is connected to four adjacent nodes, which have regular directions: east, west, south and north, representing four output directions respectively. At the same time, the real-time connectivity of the four links is also considered for judgment, and different output values ​​are given. (3) Reward: The definition of reward depends on the goal we set. The most important goal we should achieve is to reduce the delay of data transmission and reduce the congestion of each link and node. The objective function is as follows:

[0088] reward=-ω1*d n -ω2*C n +ω3*B n (13)

[0089] Among them, reward means reward, d n Represents the distance from node n to the destination node. n represents the congestion degree of node n, C n =q n / Q m ,q n Indicates the queue length of the current node n, Q m Indicates the maximum queue length. n It represents the remaining available bandwidth of the link. ω1, ω2, and ω3 represent weight coefficients. ω1+ω2+ω3=1. The weight coefficient can be adjusted according to the network conditions.

[0090] The algorithm flow of the present invention is mainly as follows: it is roughly divided into two stages: offline learning and online decision-making. The offline stage of the DQN routing model mainly uses historical traffic data and simulation data for learning. First, the data is preprocessed, and then each jump action is selected according to the reward function, and then each jump action and its previous and subsequent states are stored. The state includes multi-dimensional matrices such as connectivity, node congestion, and link load. Finally, a multi-channel convolutional neural network (CNN) is used for learning. In the online stage, the connection and distance between each node in the satellite network are first calculated within the time slot, and the network topology information is transmitted to each satellite node, the number and size of data packets are set, and the state of each node is initialized and the initial load that obeys the Poisson distribution is allocated. Then, starting from the source node, a request is sent to the satellite node with which the link is established to obtain the load of each link and the distance from the node to the destination node, and the objective function value is calculated according to the link load and distance, and the optimal value is selected as the next hop node. The above process is repeated until the destination node is reached. At the same time, when entering the next time slot, the network topology information is updated, the state matrix is ​​updated, the current state action and other information are stored to continue learning, and experience replay and network parameters are updated to obtain a better routing decision under this network state. See Algorithm 1 for the specific algorithm flow.

[0091]

[0092] Example:

[0093] A single-layer LEO constellation of 64 satellites is generated by STK, with a total of 8 orbits, 8 satellites in each orbit, an orbital altitude of 895.5km, and an inclination of 86.4°. The corresponding functions of STK are used to calculate the connection status and distance between each satellite node, and generate the link matrix and distance matrix. Since the multi-dimensional information of the network needs to be considered, multiple channels can be used to express the network status as a multi-dimensional matrix as the input of the model. The neural network in the DQN model uses a convolutional neural network to better utilize its matrix computing power to process multi-dimensional resources and extract features. Two convolutional layers and two fully connected layers are set in the CNN model, and the number of convolutional kernels is 16 and 32. The simulation parameters are shown in Table 1. The link channel capacity is set to 100Mb / s, the maximum data transmission rate of each task is 2Mb / s, the initial load of each node follows the Poisson distribution, and the data packet size is set to 512bit. Set the task source node and destination node.

[0094] Table 1 Simulation parameters

[0095]

[0096] In the simulation, the performance of this routing algorithm is compared with that of the traditional Dijkstra shortest path algorithm and the maximum flow algorithm. Figure 4The performance difference of the three algorithms in the total end-to-end delay is shown. It can be seen from the simulation diagram that the proposed algorithm has a lower end-to-end delay. When the number of data packets is small, the delay performance of the three algorithms is close, because the congestion in the network is not serious at this time, and the main factors are propagation and transmission delays. As the number of data packets increases, the congestion of the shortest path algorithm becomes more and more obvious, and the queuing time in the node gradually increases, resulting in a significant increase in the total delay. Although the maximum flow algorithm can select alternative paths to appropriately alleviate congestion, the selection of some routes comes at the cost of increasing the path length, which increases the propagation delay, and the total delay performance is slightly better than the shortest path algorithm. The DQN-based load balancing routing algorithm proposed in the present invention has better performance in terms of end-to-end delay.

[0097] Figure 5 The performance difference of the three algorithms in network throughput is shown. It can be seen from the simulation diagram that the proposed algorithm has a higher network throughput. When the number of data packets is small, the throughput performance of the three algorithms is close, because the network load is low at this time, and there is no congestion when transmitting data packets. With the increase in the number of data packets, the congestion of the shortest path algorithm becomes more and more obvious due to the limitation of link capacity, and it reaches and maintains the maximum value first, and the network throughput is the lowest. The maximum flow algorithm can find an optimal traffic distribution scheme in the network, so that the total traffic in the network reaches the maximum, so the network throughput under this algorithm is higher, but due to its high latency, its network throughput peak is slightly lower than the algorithm proposed in the present invention. The algorithm continuously searches for nodes with lower congestion according to changes in network load, expands alternative paths, averages network load to more nodes, reduces the number of single node reuses, and relieves network pressure. It can be seen that the load balancing routing algorithm based on DQN proposed in the present invention also has good performance in terms of network throughput.

[0098] Figure 6The figure shows the difference between the three algorithms in the number of satellites involved in forwarding. This indicator mainly reflects the performance of the algorithm in load balancing. The more satellites involved in forwarding, the fewer times each satellite node is reused, and more satellite nodes and routing paths are found to share the network pressure to achieve load balancing. It can be seen from the simulation diagram that the proposed algorithm has a large number of satellites involved in forwarding, and as the number of data packets increases, new nodes will continue to expand, reducing the number of reuses of old nodes and alleviating the congestion of the network. Since the shortest path algorithm only considers the shortest path, the satellites involved in forwarding only include nodes on the path, resulting in link congestion, a large number of data packets backlog, and high transmission delay and queuing delay. The maximum flow algorithm is limited by network traffic. When the current traffic in the link reaches the maximum capacity, the path containing the edge cannot be used as an augmented path and expand new satellite nodes. As the amount of data continues to increase, the traffic near the destination node in the network is close to saturation, and the maximum flow algorithm no longer has new nodes involved in forwarding. It can be seen that the DQN-based load balancing routing algorithm proposed in the present invention has a large number of satellites involved in forwarding and a stronger load balancing ability.

[0099] The present invention takes into account the high dynamics, long latency, and uneven network load of low-orbit satellite networks, introduces the idea of ​​deep reinforcement learning in the routing scheduling strategy, and proposes a low-orbit satellite network load balancing intelligent routing algorithm based on DQN by combining the characteristics of low-orbit satellite networks and the advantages of DQN. The algorithm adopts the time slicing method to learn the node connection relationship and load conditions in the static topology of each time slot, and jointly optimizes multiple objectives such as the distance between target satellites, node congestion, and link available bandwidth, and provides an optimized next-hop selection strategy for each node. The link connectivity matrix is ​​updated for different time slots, and training continues to update network parameters. By comparing with the traditional satellite network routing algorithm, it can be seen that the algorithm proposed by the present invention has better performance in latency and network throughput, reduces the reuse of satellite nodes in the network, alleviates the problem of uneven load in the network, and improves network performance.

Claims

1. A low-orbit satellite network load balancing intelligent routing algorithm based on DQN, characterized in that: Combined with the satellite routing problem, the routing process is formalized as an MDP. First, the agent's state, behavior and reward representation are designed, where: (1) State: The observed state consists of three parts, including the satellite node where the data packet is currently located, the load of the adjacent satellite links, and the connectivity and distance between the satellite nodes. It is divided into three state matrices: position relationship matrix, load matrix and distance matrix, and merged into one state; (2) Action: The action is defined as an output interface selected by the data packet. Considering a grid-like satellite network, each node is connected to four adjacent nodes, which have regular directions: east, west, south and north, representing four output directions respectively. At the same time, the real-time connectivity of the four links is also considered for judgment, and different output values ​​are given; (3) Reward: The definition of the reward depends on the set goal. The most important goal to be achieved is to reduce the delay of data transmission and reduce the congestion of each link and node. The objective function is as follows: reward=-ω1*d n -ω2*C n +ω3*B n (13), Among them, reward means reward, d n represents the distance from node n to the destination node, C n represents the congestion degree of node n, C n =q n / Q m ,q n Indicates the queue length of the current node n, Q m Indicates the maximum queue length, B n represents the remaining available bandwidth of the link, ω1, ω2, ω3 represent weight coefficients, ω1+ω2+ω3=1, and the weight coefficient can be adjusted according to the network conditions; the algorithm process is divided into two stages: offline learning and online decision-making. The offline stage of the DQN routing model uses historical traffic data and simulation data for learning: first, the data is preprocessed, and then each hop action is selected according to the reward function, and then each hop action and its before and after states are stored. The states include multi-dimensional matrices such as connectivity, node congestion, and link load. Finally, a multi-channel convolutional neural network (CNN) is used for learning.

2. According to claim 1, a DQN-based low-orbit satellite network load balancing intelligent routing algorithm is characterized in that: In the online phase, the connection status and distance between nodes in the satellite network are first calculated within the time slot and the network topology information is transmitted to each satellite node, the number and size of data packets are set, the state of each node is initialized, and an initial load that obeys Poisson distribution is allocated; Then, the source node sends a request to the satellite nodes that establish links with it to obtain the load of each link and the distance from the node to the destination node. The objective function value is calculated based on the link load and distance, and the optimal value is selected as the next hop node. After that, the above process is repeated until the destination node is reached. At the same time, when entering the next time slot, the network topology information is updated, the state matrix is ​​updated, the current state action is stored to continue learning, experience replay and network parameters are updated to obtain better routing decisions under this network state.

3. According to claim 1, a DQN-based low-orbit satellite network load balancing intelligent routing algorithm is characterized in that: The DQN routing model is described as a Markov decision process (MDP), which is a four-tuple (s, a, s', r). These are all decisions made by the agent to maximize the cumulative reward, where s refers to the state space that the agent can observe, a represents the action space related to the decision, s' refers to the next state space reached after action a, and r represents the reward obtained by the agent. With the help of the function approximation ability of deep neural networks, more complex decision problems with high-dimensional inputs can be solved. Among them, the action value function (Q function) represents the expected return obtained by taking behavior a according to state s under the guidance of strategy π. Strategy π represents the mapping from state to behavior, which is equivalent to the decision-making strategy of the agent, and selects different behaviors according to different states, that is, a = π (s). Under the guidance of strategy π, the Q function of the robot is defined as follows: Q(s,a)=E π [G t |S t =s,A t =a] (9), Among them G t Represents a discount reward, which is defined as follows: Where γ is the discount factor, that is, the value ratio of the expected reward in the future at the current moment, γ t As the training process iterates, γ t The smaller it is, the less impact the future rewards have on the rewards at the current moment. The Bellman equation of the Q function is expressed as follows: In the formula, R(s t ,a t ) indicates that the agent is in s t When taking a t The instant reward obtained is expressed as follows by selecting the maximum action value function to solve the optimal behavior strategy: The DQN algorithm constructs the labels for algorithm training through the reward values ​​of behaviors, and the experience replay and target network therein effectively solve the problems of data correlation and non-static distribution.

4. According to claim 1, a DQN-based low-orbit satellite network load balancing intelligent routing algorithm is characterized in that: The model scenario considers a small low-orbit satellite constellation and deploys an 8×8 single-layer mesh constellation. The time slicing method is used. It is assumed that the network topology structure in the same time slot is fixed. The mesh structure assumes that each satellite node has at most two co-orbit links and two adjacent orbit links. On this basis, the polar regions and dynamic topology are considered to affect the connectivity between satellites. Here, the routing decisions in different time slots are considered, and the duration is set to update the network topology. The data flow from the source satellite node to the destination satellite node through each hop routing selection. During the process, each satellite node can obtain the link connection status of the time slot and the distance to each node.

5. According to claim 1, a DQN-based low-orbit satellite network load balancing intelligent routing algorithm is characterized in that: Since the communication takes place in a free space environment, inter-satellite communication is mainly affected by free space path loss (FSPL) and additive white Gaussian noise (AWGN). Considering the k,i To the next hop v k,i +1 A single data packet, satellite v k,i With v k,i+1 The FSPL between can be expressed as in, Indicates link e k,i,i+1 length, f is the carrier frequency, c is the speed of light, and the satellite v k,i With v k,i+1 The signal-to-noise ratio can be expressed as Where P tr is the transmission power, G tr and G r are the gains of the satellite transmitting and receiving antennas, respectively, and k B is the Boltzmann constant, T e is the equivalent noise temperature, and B is the channel bandwidth. Then the satellite v k,i With satellite v k,i+1 The maximum achievable data transfer rate between can be expressed as: Consider alternative path P k The total end-to-end delay of the pth packet after transmission can be expressed as in, Indicates that the pth data packet is on link e k,i,i+1 The required propagation delay on is the node v in the path k,i The transmission delay at p size is the size of a single data packet; For data packet p at node v k,i The required queuing delay at It obeys Poisson distribution. If the kth path is selected for data transmission in the routing decision cycle, the total number of data packets forwarded from the source node to the destination node can be expressed as in When selecting path k, the source node v k,s The data transmission rate is , then the packet loss rate calculation of the routing decision cycle can be expressed as loss k =1-pac k / n k (5), Among them, pac k When path k is selected in the routing decision cycle, it reaches v k,d The number of packets.

6. According to claim 1, a DQN-based low-orbit satellite network load balancing intelligent routing algorithm is characterized in that: The constraints of the satellite network are as follows: (1) Bandwidth constraint: To ensure that data can be transmitted normally in the link, the bandwidth required by the task must not exceed the minimum value of the available bandwidth of all links on the path: B P ≤min(B ij ,B jk ,…,B pq ) (6), Among them, B P Indicates the bandwidth required for the task, and the bandwidth of each sub-link is represented by B ij express; (2) Link connectivity constraints: In the same time slot, the network topology can be considered fixed, and the link connection status can be determined based on the link duration. To ensure that the link remains connected during data transmission, the minimum duration of all links on the path is required to be no less than the set time slot length: min(c ij ,c jk ,…,c pq )≥T (7), where T is the time slot and the bandwidth of each sub-link is c ij express; (3) Transmission rate constraints: The task transmission rate requirement cannot exceed the set maximum transmission rate: R p ≤R m (8), where R p represents the task transmission rate, R m Indicates the set maximum transmission rate.

Images