Multi-satellite collaborative distributed routing method based on multi-agent reinforcement learning

Abstract

The present invention provides a multi-satellite collaborative distributed routing method based on multi-agent reinforcement learning. The method utilizes time slicing technology to construct a static topology model of a satellite network and establishes an objective function that minimizes the end-to-end delay of data packets. A satellite agent network is constructed based on the static topology model of the satellite network, and satellite interaction experience data is obtained. The satellite agent network comprises a multi-satellite collaborative hybrid network and a satellite decision network corresponding to each agent. The satellite agent network is trained based on the satellite interaction experience data to obtain a trained satellite agent network. The satellite decision networks in the trained satellite agent network are deployed to corresponding agents, allowing the agents to make routing decisions based on the deployed satellite decision networks. In this way, through the collaborative work of the satellite decision network and the multi-satellite collaborative hybrid network, efficient distributed routing decisions can be achieved for low-orbit satellite constellations, reducing latency and balancing loads.

Description

Technical Field

[0001] The present invention relates to the technical field of low-orbit satellite routing optimization, and in particular to a multi-satellite collaborative distributed routing method using multi-agent reinforcement learning. Background Art

[0002] In recent years, with the rapid development of large low-orbit satellite constellations, network size and complexity have increased dramatically, accompanied by a continued increase in the variety and volume of services. The uneven temporal and spatial distribution of user traffic often leads to sudden traffic spikes on certain intersatellite links, causing localized congestion and even link outages. This not only increases packet loss but also reduces quality of service (QoS).

[0003] Traditional centralized routing methods cannot dynamically adjust routing plans according to real-time network conditions, and therefore have shortcomings such as insufficient flexibility, difficulty in scalability, and high communication overhead. These limitations make it difficult to meet the needs of different business traffic. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-satellite collaborative distributed routing method based on multi-agent reinforcement learning to solve the problems of dynamic topology changes, continuous growth of user demand and uneven distribution of network traffic in low-orbit satellite networks.

[0005] The present invention provides a multi-satellite collaborative distributed routing method for multi-agent reinforcement learning, comprising:

[0006] Obtain network structure data of the low-orbit satellite constellation and use time slicing technology to construct a static topology model of the satellite network; each satellite in the low-orbit satellite constellation is regarded as an intelligent agent;

[0007] Based on the static topology model of satellite networks, an objective function is established to minimize the end-to-end delay of data packets.

[0008] A satellite agent network is constructed based on a static topology model of the satellite network, and satellite interaction experience data is obtained. The satellite agent network includes a multi-satellite collaborative hybrid network and a satellite decision network corresponding to each agent. The multi-satellite collaborative hybrid network is used to calculate joint rewards based on the objective function, and the satellite decision network is used to output the routing decision of each agent based on its current local observation information.

[0009] The satellite agent network is trained based on satellite interaction experience data to obtain a trained satellite agent network. Each satellite decision network is used to calculate a local value function based on local observation information. The multi-satellite collaborative hybrid network is used to generate a global value function based on global state information and all local value functions. The global value function and joint reward are used to update network parameters.

[0010] The satellite decision networks in the trained satellite agent network are deployed to the corresponding agents respectively, so that the agents make routing decisions based on the deployed satellite decision networks.

[0011] In an optional embodiment, the network structure data includes the number of satellite orbits, the number of satellites on each orbit, the orbit altitude and two rows of orbit data, and the satellite network static topology model includes an undirected graph consisting of a set of satellite nodes and a set of links between satellites in each time slice.

[0012] In an optional embodiment, the step of establishing an objective function for minimizing the end-to-end delay of a data packet based on a static topology model of a satellite network includes:

[0013] Based on the static topology model of satellite network, the inter-satellite link model and data packet delay model are established by using logical address and graph theory methods.

[0014] Based on the inter-satellite link model and data packet delay model, the satellite routing problem is formulated as a multi-objective optimization problem with the goal of minimizing the end-to-end delay of data packets, and the objective function is obtained.

[0015] In an optional embodiment, the objective function is expressed as:

[0016] ;

[0017] ;

[0018] Where K represents the set of data packets, Indicates the number of data packets, D( k ) indicates a data packet k On the path P k The end-to-end delay on , V k Indicates data packet k The set of nodes passed, E k Indicates data packet k The set of edges along the way, V t represents the set of satellite nodes, E t represents the set of links between satellites, H Indicates the maximum lifetime of a data packet, dis i,j Indicates satellite i With satellite j The distance between Indicates satellite i With satellite j The maximum viewing distance between Indicates data packet k exist t Is the satellite in the moment?i In the queue, Indicates data packet k exist t Is the satellite in the moment? g 、 i Transmission between Z t,i Indicates satellite i exist t The neighbor set at that moment, Indicates satellite i The maximum capacity of the queue, Indicates data packet k exist t Is the satellite in the moment? i was discarded.

[0019] In an optional embodiment, the step of obtaining satellite interaction experience data includes:

[0020] Build the satellite agent interaction environment and experience replay pool, and initialize the network parameters of the satellite agent network;

[0021] The satellite interaction experience data generated by each agent through the interaction between the initialized satellite agent network and the satellite agent interaction environment is obtained, and the satellite interaction experience data is stored in the experience replay pool; among them, each agent obtains local observation information from the satellite agent interaction environment and makes routing decisions through the initialized corresponding satellite decision network. The central node in each agent obtains the local observation information and decision actions of each agent, and calculates the joint reward through the initialized multi-satellite collaborative hybrid network.

[0022] In an optional embodiment, the satellite interaction experience data includes multiple data samples in different time slices, and each data sample includes global state information, joint local observation information, joint action and joint reward.

[0023] In an optional embodiment, the satellite interaction experience data is stored in an experience replay pool; and the step of training the satellite agent network based on the satellite interaction experience data to obtain the trained satellite agent network includes:

[0024] Perform non-uniform sampling in the experience replay pool to obtain target sample data;

[0025] According to the target sample data, the global value function is obtained through the satellite agent network;

[0026] According to the joint reward in the global value function and the target sample data, the network parameters of the multi-satellite collaborative hybrid network and the network parameters of each satellite decision network are updated through the back propagation algorithm.

[0027] In an optional embodiment, the sampling probability of each data sample in the satellite interaction experience data isp t as follows:

[0028] ;

[0029] ;

[0030] in, ε Indicates the preset value. ε To prevent the sampling probability from being zero, Indicates status Under the multi-satellite collaborative hybrid network, the error between the predicted value and the true value is s t express t The global state at the moment, a t express t The joint action of the moment, Indicates that the multi-satellite collaborative hybrid network is in state , network parameters The value of Indicates that the multi-satellite cooperative hybrid network is in state ( ), network parameters The value of r t express t Joint rewards of the moment, represents the preset discount factor, and A represents the action space of the satellite.

[0031] In an optional embodiment, the satellite agent network further includes a target hybrid network and a target decision network corresponding to the multi-satellite collaborative hybrid network and the satellite decision network, respectively; at predetermined intervals, network parameters of the target hybrid network are adjusted to be the same as those of the multi-satellite collaborative hybrid network, and network parameters of the target decision network are adjusted to be the same as those of the corresponding satellite decision network; and the steps of updating the network parameters of the multi-satellite collaborative hybrid network and the network parameters of each satellite decision network by a back propagation algorithm based on a joint reward in a global value function and target sample data include:

[0032] According to the joint rewards in the target sample data, the target value function is obtained through the target hybrid network;

[0033] According to the global value function and the target value function, the target loss function is calculated;

[0034] According to the target loss function, the gradient descent method is used to update the network parameters of the multi-satellite collaborative hybrid network and the network parameters of each satellite decision network.

[0035] Target value function is defined as:

[0036] ;

[0037] The objective loss function is defined as:

[0038] ;

[0039] Network parameters of multi-satellite collaborative hybrid network and the network parameters of the satellite decision network The update formula is:

[0040] ;

[0041] ;

[0042] in, r t express t Joint rewards of the moment, represents the preset discount factor, Indicates that the target hybrid network is in state ( ), network parameters The value of s t+1 express t+ The global state at time 1, Expressed as t+ 1 moment is used to maximize the joint action of the local value function, Represents the network parameters of a multi-satellite collaborative hybrid network and satellite i The corresponding network parameters of the satellite decision network The objective loss function under b represents the target sample data sampled from the experience replay pool, Indicates that the multi-satellite collaborative hybrid network is in state , network parameters The value of α t Represents a preset learning rate that decreases over time.

[0043] The multi-satellite collaborative distributed routing method for multi-agent reinforcement learning provided by the present invention can obtain network structure data of a low-orbit satellite constellation and construct a static topology model of the satellite network using time slicing technology. Each satellite in the low-orbit satellite constellation serves as an agent. Based on the static topology model of the satellite network, an objective function for minimizing the end-to-end delay of data packets is established. A satellite agent network is constructed based on the static topology model of the satellite network, and satellite interaction experience data is obtained. The satellite agent network includes a multi-satellite collaborative hybrid network and a satellite decision network corresponding to each agent. The multi-satellite collaborative hybrid network is used to calculate a joint reward based on the objective function, and the satellite decision network is used to output a routing decision for the agent based on the current local observation information of the agent. The satellite agent network is trained based on the satellite interaction experience data to obtain a trained satellite agent network. Each satellite decision network is used to calculate a local value function based on local observation information, and the multi-satellite collaborative hybrid network is used to generate a global value function based on global state information and all local value functions. The global value function and the joint reward are used to update network parameters. The satellite decision networks in the trained satellite agent network are respectively deployed to corresponding agents so that the agents make routing decisions based on the deployed satellite decision networks. In this way, during the training process, the satellite decision network and the multi-satellite collaborative hybrid network work together to enhance the decision-making ability and collaborative efficiency of the satellite intelligent body. The trained satellite decision network can make routing decisions that meet various business needs in a distributed manner based on real-time local observation information and business preferences, thereby solving the problems of dynamic topology changes, continuous growth in user demand, and uneven distribution of network traffic in low-orbit satellite networks, ensuring the end-to-end transmission delay of the business, and achieving effective load balancing between satellite nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 A flowchart of a multi-satellite collaborative distributed routing method for multi-agent reinforcement learning provided by an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of the overall process provided by an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of a satellite routing scenario provided by an embodiment of the present invention;

[0048] Figure 4 A network structure diagram of a multi-agent distributed routing algorithm provided by an embodiment of the present invention;

[0049] Figure 5 A diagram illustrating the working principle of the multi-agent distributed routing algorithm provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0051] In the routing optimization scenario of large-scale low-orbit Earth orbit constellations, satellite routing solutions have gradually shifted to a distributed routing architecture based on virtual topology to address the dynamic changes in satellite network topology, the continuous growth of service traffic, and the shortcomings of centralized routing strategies. This architecture uses time slicing technology to divide continuous time into multiple time slots. Within each time slot, the network topology is considered stable and unchanging. By using logical topology and graph theory methods to construct a static topology model, it analyzes the connectivity relationships between nodes and the optimal routing path. This distributed routing architecture based on virtual topology effectively reduces the frequency of interactions between satellites, reduces computing requirements, and improves network scalability and flexible distributed deployment capabilities.

[0052] With the continuous advancement of artificial intelligence, satellite routing algorithms incorporating deep reinforcement learning (DRL) have demonstrated strong adaptability to dynamically changing network conditions. These novel routing schemes leverage the continuous interaction between satellite agents and their environment to dynamically learn and optimize routing decisions to adapt to the ever-changing satellite network environment, surpassing traditional routing strategies in both flexibility and performance. Given the complexity of satellite routing tasks, multi-agent reinforcement learning techniques provide new insights and tools for designing routing methods for low-orbit satellite networks.

[0053] Based on this, an embodiment of the present invention provides a multi-satellite collaborative distributed routing method based on multi-agent reinforcement learning, which applies multi-agent reinforcement learning technology to traditional satellite routing design, not only ensuring the end-to-end transmission delay of the service, but also achieving effective load balancing between satellite nodes.

[0054] See also Figure 1 The flowchart of a multi-star collaborative distributed routing method for multi-agent reinforcement learning is shown, and the method mainly includes the following steps S110 to S150:

[0055] Step S110: obtaining network structure data of a low-orbit satellite constellation and constructing a static topology model of the satellite network using time slicing technology; wherein each satellite in the low-orbit satellite constellation is regarded as an intelligent agent.

[0056] The network structure data for a Low Earth Orbit Satellite Constellation (LEOSC) can include the number of satellite orbits, the number of satellites in each orbit, orbital altitude, and two-line elements (TLE) data. Time slicing technology is used to construct a static topology model of the satellite network. Time slicing technology discretizes dynamic systems over a period of time to facilitate analysis and modeling. The static topology model of the satellite network can include an undirected graph consisting of the set of satellite nodes and the set of links between satellites in each time slice.

[0057] Step S120: establishing an objective function for minimizing the end-to-end delay of data packets based on the static topology model of the satellite network.

[0058] In some possible embodiments, the above-mentioned step S120 may include: based on the static topology model of the satellite network, using logical addresses and graph theory methods, establishing an inter-satellite link model and a data packet delay model; based on the inter-satellite link model and the data packet delay model, with the goal of minimizing the end-to-end delay of the data packet, expressing the satellite routing problem as a multi-objective optimization problem to obtain an objective function.

[0059] Among them, logical addresses can be used to identify each node in the satellite network, so that each satellite has a unique identifier to facilitate communication and routing. Inter-satellite link models can be divided into two categories: links between satellites in the same orbit and links between satellites in different orbits. For links between satellites in the same orbit, the connection between them is usually relatively stable, and the link is only disconnected when the distance between satellites exceeds the maximum communication range. For links between satellites in different orbits, in addition to considering the communication distance, the relative movement speed between satellites must also be considered. Especially when satellites are located in high latitudes or in the reverse gap of their orbits, the high-speed movement between satellites makes it difficult to establish a stable communication link, and these links are usually in a disconnected state.

[0060] The packet delay model can be composed of three components: transmission delay, queuing delay, and propagation delay. Transmission delay is defined as the time required for a packet to enter the transmission medium (i.e., the channel), which is related to the packet length and transmission rate. Queuing delay is the time a packet waits in the satellite node's buffer for transmission. Propagation delay is the time it takes for a packet to travel from the current satellite to the next hop, and is primarily determined by the inter-satellite distance and signal propagation speed.

[0061] Alternatively, the above objective function can be expressed as:

[0062] ;

[0063] ;

[0064] Where K represents the set of data packets, Indicates the number of data packets, D( k ) indicates a data packet k On the path P k The end-to-end delay on , V k Indicates data packet k The set of nodes passed, E k Indicates data packet k The set of edges along the way, V t represents the set of satellite nodes, E t represents the set of links between satellites, H Indicates the maximum lifetime of a data packet, dis i,j Indicates satellite i With satellite j The distance between Indicates satellite i With satellite j The maximum viewing distance between Indicates data packet k exist t Is the satellite in the moment? i In the queue, Indicates data packet k exist t Is the satellite in the moment? g 、 i Transmission between Z t,i Indicates satellite i exist t The neighbor set at that moment, Indicates satellite i The maximum capacity of the queue, Indicates data packet k exist t Is the satellite in the moment? i was discarded.

[0065] In the above formula P Indicates the target, st Indicates the constraints. Constraint C1 stipulates that the lifetime of the data packet in the network must not exceed its maximum lifetime (Time to Live, referred to as TTL). Constraint C2 means that the routing path selected by the satellite must be in a communication state at the current moment. Constraint C3 requires that the time tThe sum of the number of packets buffered in the satellite's queue and the number of packets received from neighboring satellites at the next moment must not exceed the maximum capacity of the satellite queue. Constraint C4 states that the packet remains in one of the following states throughout the forwarding process: either in the satellite's queue or discarded during routing.

[0066] Step S130: construct a satellite agent network based on the static topology model of the satellite network and obtain satellite interaction experience data; wherein the satellite agent network includes a multi-satellite collaborative hybrid network and a satellite decision network corresponding to each agent, the multi-satellite collaborative hybrid network is used to calculate the joint reward according to the objective function, and the satellite decision network is used to output the routing decision of the agent according to the current local observation information of the agent.

[0067] Building a multi-satellite collaborative hybrid network and Satellite Decision Network ,in, Indicates that the multi-satellite collaborative hybrid network is in state , network parameters The value of s t express t The global state at the moment, a t express t The joint action of the moment, Indicates satellite i The corresponding satellite decision network is in state , network parameters The value of Indicates satellite i exist t The local observation at the moment, the decision action at the previous moment and the one-hot ID encoding of the agent, a t,i Indicates satellite i exist t Each satellite is considered as an intelligent agent, and a satellite decision network is deployed on each intelligent agent. , these agents are connected through a multi-star collaborative hybrid network They coordinate with each other to find the global optimal solution in the routing decision process. Among them, the multi-satellite collaborative hybrid network is deployed on the central node of each intelligent agent.

[0068] Local observation information can include tThe joint reward can be calculated based on the local observation information, decision actions, and objective function, where the decision actions account for the effects of queuing delay and propagation delay.

[0069] In some possible embodiments, the above-mentioned step of obtaining satellite interaction experience data may include: constructing a satellite agent interaction environment and an experience replay pool, and initializing the network parameters of the satellite agent network; obtaining satellite interaction experience data generated by each agent through the interaction between the initialized satellite agent network and the satellite agent interaction environment, and storing the satellite interaction experience data in the experience replay pool; wherein each agent obtains local observation information from the satellite agent interaction environment and makes routing decisions through the initialized corresponding satellite decision network, the central node in each agent obtains the local observation information and decision actions of each agent, and calculates the joint reward through the initialized multi-satellite collaborative hybrid network.

[0070] The satellite interaction experience data mentioned above can include multiple data samples from different time slices, each of which includes global state information, joint local observation information, joint actions, and joint rewards. The satellite interaction experience data in the experience replay pool will subsequently be extracted by the central node for training the multi-satellite collaborative hybrid network and the satellite decision network. Network training can be performed online or offline. During online training, each agent first deploys an initial satellite decision network. The central node calculates the target loss function of the hybrid network and sends it to each agent. Each agent then updates its network parameters based on the target loss function.

[0071] Step S140: Train the satellite agent network based on the satellite interaction experience data to obtain a trained satellite agent network; wherein each satellite decision network is used to calculate a local value function based on local observation information, and the multi-satellite collaborative hybrid network is used to generate a global value function based on global state information and all local value functions. The global value function and the joint reward are used to update the network parameters.

[0072] In some possible embodiments, the satellite interaction experience data is stored in an experience replay pool; the step of training the satellite agent network based on the satellite interaction experience data to obtain the trained satellite agent network may include: performing non-uniform sampling in the experience replay pool to obtain target sample data; obtaining a global value function through the satellite agent network based on the target sample data; and updating the network parameters of the multi-satellite collaborative hybrid network and the network parameters of each satellite decision network through a backpropagation algorithm based on the joint reward in the global value function and the target sample data.

[0073] Optionally, the sampling probability of each data sample in the above satellite interaction experience data is p t It can be as follows:

[0074] ;

[0075] ;

[0076] in, ε Indicates the preset value. ε To prevent the sampling probability from being zero, Indicates status Under the multi-satellite collaborative hybrid network, the error between the predicted value and the true value is s t express t The global state at the moment, a t express t The joint action of the moment, Indicates that the multi-satellite collaborative hybrid network is in state , network parameters The value of Indicates that the multi-satellite cooperative hybrid network is in state ( ), network parameters The value of r t express t Joint rewards of the moment, represents the preset discount factor, and A represents the action space of the satellite.

[0077] In order to reduce the bias propagation caused by bootstrapping and the overestimation of target values caused by maximization operations, target networks with the same network structure are constructed for the multi-satellite collaborative hybrid network and the satellite decision network, that is, the above-mentioned satellite intelligent agent network also includes a target hybrid network and a target decision network corresponding to the multi-satellite collaborative hybrid network and the satellite decision network, respectively; at predetermined intervals, the network parameters of the target hybrid network are adjusted to be the same as the network parameters of the multi-satellite collaborative hybrid network, and the network parameters of the target decision network are adjusted to be the same as the network parameters of the corresponding satellite decision network.

[0078] Based on this, the above-mentioned step of updating the network parameters of the multi-satellite collaborative hybrid network and the network parameters of each satellite decision network through the back propagation algorithm according to the global value function and the joint reward in the target sample data may include: obtaining the target value function through the target hybrid network according to the joint reward in the target sample data; calculating the target loss function according to the global value function and the target value function; and updating the network parameters of the multi-satellite collaborative hybrid network and the network parameters of each satellite decision network using the gradient descent method according to the target loss function.

[0079] Optionally, the above target value function The definition of can be:

[0080] ;

[0081] The definition of the objective loss function can be:

[0082] ;

[0083] Network parameters of multi-satellite collaborative hybrid network and the network parameters of the satellite decision network The update formula can be:

[0084] ;

[0085] ;

[0086] in, r t express t Joint rewards of the moment, represents the preset discount factor, Indicates that the target hybrid network is in state ( ), network parameters The value of s t+1 express t+ The global state at time 1, Expressed as t+ 1 moment is used to maximize the joint action of the local value function, Represents the network parameters of a multi-satellite collaborative hybrid network and satellite i The corresponding network parameters of the satellite decision network The objective loss function under b represents the target sample data sampled from the experience replay pool, Indicates that the multi-satellite collaborative hybrid network is in state , network parameters The value of α tRepresents a preset learning rate that decreases over time.

[0087] Step S150 , deploying the satellite decision networks in the trained satellite agent network to corresponding agents respectively, so that the agents make routing decisions based on the deployed satellite decision networks.

[0088] The trained satellite decision network is deployed on the corresponding satellite agent. The satellite agent can make routing decisions that meet different business needs based on real-time local observation information and business preference information. This embodiment of the present invention adopts a normalized joint reward scheme that takes into account end-to-end distance, queuing delay, and load balancing, allowing the satellite agent to heuristically adjust reward weights based on business needs. Business preference information can be a preference for a specific aspect. For example, if the preference is for minimal propagation delay, the reward weight for propagation delay in routing decisions can be increased. Alternatively, if the preference is for overall delay, the reward weight for overall delay in routing decisions can be increased.

[0089] The multi-satellite collaborative distributed routing method for multi-agent reinforcement learning provided by the embodiments of the present invention can enhance the decision-making ability and collaboration efficiency of satellite agents through the collaborative work of the satellite decision network and the multi-satellite collaborative hybrid network during the training process. The trained satellite decision network can make routing decisions that meet various business needs in a distributed manner based on real-time local observation information and business preferences, thereby solving problems such as dynamic topology changes, continuous growth in user demand, and uneven network traffic distribution in low-orbit satellite networks, ensuring end-to-end transmission delay of services, and achieving effective load balancing between satellite nodes.

[0090] For ease of understanding, the multi-star collaborative distributed routing method for multi-agent reinforcement learning mentioned above is introduced in detail below.

[0091] In response to the dynamic changes in topology, the continuous growth of user demand, and the uneven distribution of network traffic in low-orbit satellite networks, the embodiment of the present invention designs a multi-satellite collaborative distributed routing method based on multi-agent reinforcement learning. Specifically, it includes: according to the periodic motion characteristics of the satellite, the satellite topology is divided into time slots using time slicing technology; combining logical addresses and graph theory methods to establish an inter-satellite link model and a data packet delay model, thereby converting the routing problem into a multi-objective optimization problem; deploying an agent for each satellite, and coordinating routing decisions through a multi-satellite collaborative hybrid network to enhance collaboration between satellite agents; implementing a normalized joint reward scheme that takes into account end-to-end distance, queuing delay, and load balancing, allowing satellite agents to heuristically adjust reward weights according to business needs. By integrating multi-agent reinforcement learning technology into traditional satellite routing design, not only is the end-to-end transmission delay of the business guaranteed, but effective load balancing between satellite nodes is also achieved.

[0092] In large-scale low-orbit constellation routing optimization scenarios, addressing the dynamic changes in satellite network topology, the continuous growth of service traffic, and the shortcomings of centralized routing strategies, the present invention proposes a multi-satellite collaborative distributed routing algorithm based on multi-agent reinforcement learning to solve the problem of end-to-end path selection for satellite routing. Experimental results demonstrate that the algorithm can effectively coordinate routing decision conflicts between satellite agents, ensure end-to-end service transmission latency, and achieve load balancing between satellite nodes. The specific innovations of this solution are as follows:

[0093] A. A multi-agent Transformer-MIX routing algorithm is proposed, which combines the satellite observation attention mechanism to capture the latent information in the satellite sequence data and generates a joint action-value function through a Transformer-based parameter loop mechanism to achieve a more stable training process.

[0094] B. Adopt a centralized training and distributed execution strategy. During the training phase, a multi-satellite collaborative hybrid network is introduced to coordinate conflicts between local and global optima among agents, promoting cooperation among them. During the execution phase, satellites can rely solely on their own observation space to make routing decisions, significantly reducing inter-satellite communication overhead.

[0095] C. A normalized joint reward scheme is designed that comprehensively considers end-to-end distance, queuing delay and load balancing, so that the satellite agent can heuristically adjust the weight according to business needs to meet different business requirements.

[0096] Based on the above ideas and innovations, the distributed routing method proposed in the embodiment of the present invention includes the following steps:

[0097] Step 1: Determine the network structure of the low-orbit satellite constellation, including the number of satellite orbits, the number of satellites on each orbit, the orbital altitude, and two lines of orbital data, and use time slicing technology to build a static topology model of the satellite network.

[0098] Step 2: Using logical addresses and graph theory methods, establish an inter-satellite link model and a data packet delay model to minimize the end-to-end delay of data packets and formulate the satellite routing problem as a multi-objective optimization problem.

[0099] Intersatellite link models can be divided into two categories: links between satellites in the same orbit and links between satellites in different orbits. For links between satellites in the same orbit, the connection is generally stable, with the link only disconnected when the distance between the satellites exceeds the maximum communication range. For links between satellites in different orbits, in addition to the communication distance, the relative speed between the satellites must also be considered. Especially when satellites are located at high latitudes or in orbital gaps, the high-speed movement between satellites makes it difficult to establish a stable communication link, and these links are often disconnected.

[0100] The packet delay model can be composed of three components: transmission delay, queuing delay, and propagation delay. Transmission delay is defined as the time it takes for a packet to enter the transmission medium, which is related to the packet length and transmission rate. Queuing delay is the time a packet waits in the satellite node's buffer for transmission. Propagation delay is the time it takes for a packet to travel from the current satellite to the next hop, which is primarily determined by the inter-satellite distance and signal propagation speed.

[0101] Step 3: Build a multi-satellite collaborative hybrid network and Satellite Decision Network In order to reduce the bias propagation caused by bootstrapping and the overestimation of the target value caused by maximization estimation, target networks are introduced into the multi-satellite cooperative hybrid network and the satellite decision network respectively. 、 The parameters of the original network are updated to its target network at fixed intervals to stabilize the training process and avoid continuous changes in the target update.

[0102] The satellite decision network takes the local observation information of the current satellite agent and the decision action at the previous moment (the decision action refers to which neighboring satellite the data packet is sent to) as input. It uses the observation attention mechanism and the gated recurrent unit (GRU) to generate the current satellite's routing decision and calculate the local value function. The multi-satellite collaborative hybrid network takes the local value functions and global state information of all satellite decision networks as input and outputs a global value function, which is used to evaluate the joint decision-making of all agents.

[0103] Step 4: Construct the satellite agent interaction environment and experience replay pool. The satellite agent generates experience data six-tuples by observing network information and selecting routing decisions according to the ε-greedy strategy. , respectively represent the current (i.e. t moment), global state information, joint local observation information, joint actions and joint rewards, and the next moment (i.e. tThe ε-greedy strategy is an action selection method used in reinforcement learning to balance exploration and exploitation.

[0104] Step 5: Centralized training of routing algorithm. First, randomly extract training samples from the experience replay pool. Then, the satellite decision network o t Calculate the local value function. At the same time, the multi-satellite collaborative hybrid network uses the global state information s t The global value function is generated by combining all local value functions. Then, the back propagation algorithm is executed to update the network parameters using the local value function and the global value function until the algorithm parameters converge.

[0105] Among them, the satellite decision network and the multi-satellite collaborative hybrid network both have a replica target network to solve the problem of overestimation of target values caused by maximization. The central node randomly extracts samples from the experience replay pool to calculate the loss function, which is specifically calculated as the global value function output by the satellite decision network at the current moment The difference between the expected value function at the next moment predicted by the target network can be expressed as:

[0106] ;

[0107] ;

[0108] The central node uses this loss function to update the network parameters through the gradient descent method. In addition, every fixed training cycle, the parameters of the target network are updated. Will be replaced with the parameters of the current network , to stabilize the training process.

[0109] Step 6: Deploy the trained satellite decision network to the satellite agent, which can make routing decisions that meet different business needs based on real-time local observation information and business preference information.

[0110] See also Figure 2 The overall process diagram shown in the figure shows that the multi-satellite collaborative distributed routing method based on multi-agent reinforcement learning mainly includes the following processes: satellite static topology modeling, multi-objective optimization problem formulation, satellite agent network construction, centralized training, agent network parameter update and distributed routing strategy deployment. The centralized training includes initializing the satellite agent environment and collecting satellite interaction experience data. The specific steps are as follows:

[0111] Step 1: Build a static satellite topology model. This includes steps 101 and 102.

[0112] Step 101: When considering the low-orbit satellite routing scenario, Figure 3 As shown, first of all, the period is M The topological state of the low-orbit satellite network (all satellites have the same orbital height and period) is discretized in the time dimension. Specifically, the length of each time slot is , thus obtaining discrete time slices In any time slice , the network topology is considered to be stable and unchanging, then the satellite network can be modeled as an undirected graph ,in V t represents the set of satellite nodes, i.e. ,and E t represents the set of links between satellites and is defined as .

[0113] Step 102: Further optimize the intersatellite link configuration. In each time slice, the intersatellite links adopt a mesh topology, with each satellite establishing communication links with up to four adjacent satellites, including two intra-plane links and two inter-plane links. The satellite's TLE data and other satellite network information (such as the satellite orbit number, the number of satellites in each orbit, and orbital altitude) are used to calculate the spatial coordinates of the satellites in each time slice. These coordinates are used to establish intersatellite links between adjacent satellites. Specifically, if the distance between two satellites meets the communication link establishment conditions and they are not located in high latitudes, then a valid intersatellite communication link exists between the two satellites. Conversely, if these conditions are not met, there is no communication link between the two satellites.

[0114] Step 2: Multi-objective optimization problem representation. This includes steps 201 to 203.

[0115] Step 201: Establish an inter-satellite link model. Inter-satellite link models are mainly divided into two categories. The first category is the link between satellites in the same orbit. These links are usually relatively stable and only disconnect when the distance between satellites exceeds the maximum communication distance. Assume that the satellites in the same orbit i With satellite j The distance between them can be calculated by the following formula:

[0116] ;

[0117] in, is the relative angle between the two satellites. In addition, the establishment of a link between satellites in the same orbit must also meet the condition of not being blocked by the earth:

[0118] ;

[0119] in, is the radius of the Earth, is the orbital radius of the satellite.

[0120] The second type is satellite communication links between different orbits. Since the satellites are located in different orbital planes, the angles between the two satellites and the center of the earth are It can be calculated by the following formula:

[0121] ;

[0122] in, 、 Represents satellites i The longitude and latitude of 、 Indicates satellite j longitude and latitude.

[0123] Therefore, the distance between satellites It can be expressed as:

[0124] .

[0125] The maximum visible distance between two satellites is reached when the line connecting them is tangent to the Earth's surface. If the distance continues to increase, the inter-orbit communication link cannot be established due to the obstruction of the earth. Therefore, the distance between satellites is dis i,j Must be less than the maximum viewing distance:

[0126] .

[0127] Step 202: Establish a data packet delay model, which includes transmission delay, queuing delay and propagation delay. In the satellite network, define the data packet set as K. For each data packet , whose forwarding path from the source satellite to the destination is expressed as , where V k is the set of nodes passed through, E k is the set of edges along the way. k On the path P k The end-to-end delay on can be expressed as:

[0128] .

[0129] Transmission delay It is the time required for a data packet to completely enter the transmission medium, which depends on the packet length pkt and the data transmission rate .in, is the channel bandwidth of the intersatellite link between satellites, is the signal-to-noise ratio between nodes. The data transmission rate can be defined by the following formula:

[0130] ;

[0131] Therefore, the transmission delay can be defined as:

[0132] .

[0133] Queuing delay This is the time a data packet spends buffered in a satellite node waiting to be sent again. Since satellite nodes in the network can establish intersatellite links with up to four adjacent satellites within communication range, once a data packet arrives at a satellite, it often takes some time before it is forwarded to the next-hop satellite. This waiting time is related to the number of data packets in the current satellite node's send queue. Therefore, the queuing delay can be expressed as:

[0134] ;

[0135] in, is the update interval of the satellite routing table, Represented as a satellite i Queue at time t The number of packets in a queue can be described by considering the dynamics of packets arriving and leaving the queue:

[0136] .

[0137] Propagation delay refers to data packets k Via communication link The time required for this is mainly determined by the distance between the communication links dis i,j and the speed of information dissemination c Therefore, the propagation delay can be defined as:

[0138] .

[0139] Step 203: Multi-objective optimization problem representation.

[0140] Based on the above model, a joint optimization problem can be established to minimize the end-to-end delay D( k ), expressed as:

[0141] ;

[0142] ;

[0143] Among them, constraint C1 stipulates that the lifetime of the data packet in the network must not exceed its maximum lifetime. Constraint C2 means that the routing path selected by the satellite must be in a communication state at the current moment. Constraint C3 requires that the time t The sum of the number of packets buffered in the satellite's queue and the number of packets received from neighboring satellites at the next moment must not exceed the maximum capacity of the satellite queue. Constraint C4 states that the packet remains in one of the following states throughout the forwarding process: either in the satellite's queue or discarded during routing.

[0144] Step 3: Build a multi-satellite collaborative hybrid network and a satellite decision network. Each satellite is considered as an intelligent agent, and a satellite decision network is deployed on each intelligent agent. These agents collaborate through a multi-star hybrid network Coordinate with each other to find the global optimal solution in the routing decision process. It includes steps 301 and 302.

[0145] Step 301: The satellite decision network receives local observation information of each satellite agent and the decision action at the previous moment As input. Through the observation attention mechanism and GRU processing, the network outputs the routing decision of the current satellite. The multi-satellite collaborative hybrid network receives the output local value function and global state information of all satellite decision networks. s t , and output the global value function , used to evaluate the joint decision-making effect of all agents.

[0146] Step 302: In order to reduce the bias propagation caused by bootstrapping and the overestimation of the target value caused by maximization operation, target networks with the same network structure are constructed for the multi-satellite cooperative hybrid network and the satellite decision network. and By updating the parameters of the original network to the target network at fixed intervals, the stability of the target during training can be ensured, avoiding the problem of the target constantly changing during the update process.

[0147] Step 4: Build the satellite agent interaction environment and experience replay pool, and initialize the network parameters. Each agent obtains its own local network information from the environment , and makes routing decisions based on local observations. The joint decisions of all agents are then executed together, and the joint reward is calculated. At each time slot, the global state, local observations, joint actions, and joint rewards are stored as experience in the experience replay pool. This data is then extracted by the central node for network training.

[0148] Step 5: The routing algorithm is centrally trained. The central node calculates the loss function by extracting empirical data and uses the gradient descent method to update the network parameters, including steps 501 and 502.

[0149] Step 501: Prioritize experience playback. Define a sampling probability for each data sample. p t , allowing the central node to perform non-uniform sampling in the experience replay pool, giving priority to experiences with higher expected learning value. The sampling probability p t It can be expressed as:

[0150] ;

[0151] in, ε Is a very small number used to prevent the sampling probability from approaching zero and ensure that all samples can be drawn with non-zero probability. Is the current state Under this circumstance, the error between the value predicted by the multi-satellite collaborative hybrid network and the true value is defined as:

[0152] ;

[0153] in, A The satellite's decision space

[0154] Step 502: Update the agent network parameters. The central node randomly selects data based on the sampling probability of the experience sample to calculate the loss function, and optimizes the target loss function and updates the network weights through the gradient descent method. The target loss function is defined as:

[0155] ;

[0156] in, b Represents training data sampled from the experience replay pool. The target value function calculated for the target network of the multi-satellite collaborative hybrid network is defined as:

[0157] ;

[0158] in, is the discount factor for future rewards, ranging from 0 to 1; Expressed as t+ 1 moment is used to maximize the joint action of the local value function. Specifically, for each satellite agent i Local action It can be expressed as:

[0159] .

[0160] Finally, the central node uses the gradient descent method to update the network parameters according to the loss function. The specific update process can be expressed as:

[0161] ;

[0162] ;

[0163] in, α t Indicates a preset learning rate that decreases over time, for example, between 0.01 and 0.005.

[0164] At the same time, after a fixed number of training rounds, the parameters of the target network Will be replaced by the original network parameters , to ensure the stability of training.

[0165] Step 6: Deploy the trained satellite decision network to the satellite agent, which can make routing decisions that meet different business needs based on real-time local observation information and business preference information.

[0166] See also Figure 4 The network structure diagram of the multi-agent distributed routing algorithm shown in Figure 2 is for the proposed satellite decision network ( Figure 4 In (a), each satellite agent i Maintaining action-value function , where the input of the satellite decision network includes local observations , the decision action at the last moment and the agent's one-hot ID encoding The proposed satellite decision network integrates an attention participation unit (SelfAtt) and a gated recurrent unit (GRU), which work in parallel to assist the agent in making routing decisions, and then pass the parallel results through two fully connected layers (MLP) and ε -greedy action sampling strategy obtains the satellite's decision action; among them, the hidden layer state 、 are the input and output of GRU respectively. For the proposed multi-star collaborative hybrid network ( Figure 4 In (c), the multi-satellite collaborative hybrid network uses a Transformer-based parameter recursion mechanism to address the temporal correlation in the training data. This mechanism is used to combine the action-value functions of all satellite agents into a joint value function. By utilizing the Transformer module hybrid network, the temporal dependency and long-range correlation in the data can be effectively captured, thereby dynamically generating the parameters of the joint function. are the weights generated by the hybrid network through the Transformer-based parameter recursive mechanism, TB represents the processing performed by the Transformer module, and Represents the global state processed by the fully connected layer s t The environmental feature vector of Corresponding to the time step t The weight at -1, and by the equation Update the weights, where relu Represents the use of relu activation function to process parameters, Indicates that the absolute value operation is used to process the parameter.

[0167] See also Figure 5 The working principle diagram of the multi-agent distributed routing algorithm shown in the figure shows that during the training phase, each satellite agent maintains a queue to effectively manage data packets. By obtaining local observation information from neighboring satellites and itself, each agent uses the satellite decision network to independently make the next hop decision for the data packet, and then jointly executes the decision actions of all agents to calculate the joint reward. r t . Second, the global state, action, joint reward, and observation are combined as , and at time t Stored in the experience replay pool. Finally, the multi-satellite assistance hybrid network samples these stored experience data to update its parameters, thereby coordinating and optimizing the decision-making process of all satellite agents.

[0168] In summary, the embodiments of the present invention propose a multi-satellite collaborative distributed routing method based on multi-agent reinforcement learning, which aims to solve the end-to-end path selection problem of large-scale low-orbit constellation networks. In this method, the satellite decision network and the multi-satellite collaborative hybrid network work together to enhance the decision-making ability and collaborative efficiency of satellite agents. Each satellite decision network integrates the attention mechanism and the gated recurrent unit, so that the agent can not only effectively process and extract relevant features in the decision data, but also make more reasonable routing choices. The multi-satellite collaborative hybrid network adopts a Transformer-based parameter circulation mechanism to generate nonlinear parameters to fuse the local action value functions of all agents into a global value function, thereby achieving stability in the training process. After training is completed, these optimized satellite decision networks will be deployed to each satellite agent, enabling it to make routing decisions that meet various business needs in a distributed manner based on real-time local observation information and business preferences.

[0169] In all examples shown and described herein, any specific values should be interpreted as merely exemplary and not limiting, and thus other examples of the exemplary embodiments may have different values.

[0170] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and a part of the module, program segment or code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-star collaborative distributed routing method based on multi-agent reinforcement learning, characterized in that: include: Obtaining network structure data of a low-orbit satellite constellation and constructing a static topology model of the satellite network using time slicing technology; wherein each satellite in the low-orbit satellite constellation serves as an intelligent agent; Based on the static topology model of the satellite network, an objective function for minimizing the end-to-end delay of data packets is established; A satellite agent network is constructed based on the static topology model of the satellite network, and satellite interaction experience data is obtained; wherein the satellite agent network includes a multi-satellite collaborative hybrid network and a satellite decision network corresponding to each agent, the multi-satellite collaborative hybrid network is used to calculate a joint reward according to the objective function, and the satellite decision network is used to output a routing decision of the agent according to the current local observation information of the agent; The satellite agent network is trained based on the satellite interaction experience data to obtain a trained satellite agent network; wherein each satellite decision network is used to calculate a local value function based on local observation information, and the multi-satellite collaborative hybrid network is used to generate a global value function based on global state information and all the local value functions, and the global value function and the joint reward are used to update network parameters; Deploying the satellite decision networks in the trained satellite agent network to the corresponding agents, respectively, so that the agents make routing decisions based on the deployed satellite decision networks; The satellite interaction experience data is stored in an experience replay pool; and the steps of training the satellite agent network according to the satellite interaction experience data to obtain a trained satellite agent network include: Performing non-uniform sampling in the experience replay pool to obtain target sample data; According to the target sample data, obtaining the global value function through the satellite agent network; According to the global value function and the joint reward in the target sample data, the network parameters of the multi-satellite collaborative hybrid network and the network parameters of each satellite decision network are updated by a back propagation algorithm; The sampling probability of each data sample in the satellite interaction experience data p t as follows: ； ； in, ε Indicates the preset value. ε To prevent the sampling probability from being zero, Indicates status Under the above conditions, the error between the value predicted by the multi-satellite collaborative hybrid network and the true value is s t express t The global state at the moment, a t express t The joint action of the moment, Indicates that the multi-satellite cooperative hybrid network is in state , network parameters The value of Indicates that the multi-satellite cooperative hybrid network is in state ( ), network parameters The value of r t express t Joint rewards of the moment, represents the preset discount factor, and A represents the action space of the satellite.

2. The method according to claim 1, characterized in that The network structure data includes the number of satellite orbits, the number of satellites on each orbit, the orbit altitude and two rows of orbit data. The satellite network static topology model includes an undirected graph consisting of a set of satellite nodes and a set of links between satellites in each time slice.

3. The method according to claim 1, characterized in that The step of establishing an objective function for minimizing the end-to-end delay of data packets based on the satellite network static topology model comprises: Based on the static topology model of the satellite network, an inter-satellite link model and a data packet delay model are established using logical addresses and graph theory methods; Based on the intersatellite link model and the data packet delay model, with the goal of minimizing the end-to-end delay of data packets, the satellite routing problem is expressed as a multi-objective optimization problem to obtain the objective function.

4. The method according to claim 3, characterized in that The objective function is expressed as: ； ； Where K represents the set of data packets, Indicates the number of data packets, D( k ) indicates a data packet k On the path P k The end-to-end delay on , V k Indicates data packet k The set of nodes passed, E k Indicates data packet k The set of edges along the way, V t represents the set of satellite nodes, E t represents the set of links between satellites, H Indicates the maximum lifetime of the data packet, dist i,j Indicates satellite i With satellite j The distance between Indicates satellite i With satellite j The maximum viewing distance between Indicates data packet k exist t Is the satellite in the moment? i In the queue, Indicates data packet k exist t Is the satellite in the moment? g 、 i Transmission between Z t,i Indicates satellite i exist t The neighbor set at that moment, Indicates satellite i The maximum capacity of the queue, Indicates data packet k exist t Is the satellite in the moment? i was discarded.

5. The method according to claim 1, wherein The steps to obtain satellite interaction experience data include: Constructing a satellite agent interaction environment and an experience replay pool, and initializing network parameters of the satellite agent network; Obtain satellite interaction experience data generated by each of the intelligent agents through the interaction of the initialized satellite intelligent agent network and the satellite intelligent agent interaction environment, and store the satellite interaction experience data in the experience replay pool; wherein, each of the intelligent agents obtains local observation information from the satellite intelligent agent interaction environment and makes routing decisions through the initialized corresponding satellite decision network, the central node in each of the intelligent agents obtains the local observation information and decision actions of each of the intelligent agents, and calculates the joint reward through the initialized multi-satellite collaborative hybrid network.

6. The method according to claim 5, characterized in that The satellite interaction experience data includes multiple data samples in different time slices, and each data sample includes global state information, joint local observation information, joint action and joint reward.

7. The method according to claim 1, characterized in that The satellite agent network further includes a target hybrid network and a target decision network corresponding to the multi-satellite collaborative hybrid network and the satellite decision network, respectively; at predetermined intervals, network parameters of the target hybrid network are adjusted to be the same as those of the multi-satellite collaborative hybrid network, and network parameters of the target decision network are adjusted to be the same as those of the corresponding satellite decision network; and the steps of updating the network parameters of the multi-satellite collaborative hybrid network and the network parameters of each of the satellite decision networks by a back propagation algorithm according to the joint reward in the global value function and the target sample data include: Obtaining a target value function through the target hybrid network according to the joint reward in the target sample data; Calculating a target loss function based on the global value function and the target value function; According to the target loss function, the network parameters of the multi-satellite collaborative hybrid network and the network parameters of each satellite decision network are updated using the gradient descent method.

8. The method according to claim 7, characterized in that The target value function is defined as: ； The objective loss function is defined as: ； The network parameters of the multi-satellite cooperative hybrid network and the network parameters of the satellite decision network The update formula is: ； ； in, r t express t Joint rewards of the moment, represents the preset discount factor, Indicates that the target hybrid network is in state ( ), network parameters The value of s t+1 express t+ The global state at time 1, Expressed as t+ 1 moment is used to maximize the joint action of the local value function, Represents the network parameters of the multi-satellite cooperative hybrid network and satellite i The corresponding network parameters of the satellite decision network The objective loss function under b represents the target sample data sampled from the experience replay pool, Indicates that the multi-satellite cooperative hybrid network is in state , network parameters The value of α t Represents a preset learning rate that decreases over time.

Images