A low-orbit satellite communication network routing method based on federated deep reinforcement learning

Abstract

This application discloses a low-orbit satellite communication network routing method based on federated deep reinforcement learning, which relates to the field of satellite communication technology. The method includes: clustering satellites in a target low-orbit satellite network to obtain multiple clusters, and maintaining each cluster according to the satellite status after clustering; the cluster head in each cluster serves as a low-orbit satellite server, and the cluster members serve as low-orbit satellite clients; building a federated learning framework in each cluster, and based on the federated learning framework, using the data of each low-orbit satellite client to train a routing decision model; when training the local model in the low-orbit satellite client, the routing decision process is constructed as a real-time Markov decision process for reinforcement learning; in the real-time Markov decision process, the current state is determined by the state and action at the previous moment; the local model corresponding to each low-orbit satellite client is used to output a routing decision for routing and forwarding data. This application improves the throughput of the low-orbit satellite network.

Description

Technical Field

[0001] The present application relates to the field of satellite communication technology, and in particular to a low-orbit satellite communication network routing method based on federated deep reinforcement learning. Background Art

[0002] With the decreasing cost of satellite launches, Low Earth Orbit (LEO) satellite networks have become a viable solution for providing global broadband access. Compared to terrestrial networks, LEO satellite networks offer many advantages, including seamless global coverage and freedom from geographical and infrastructure constraints. However, LEO satellite networks also present challenges, such as the rapid motion of LEO satellites relative to the ground, frequent state changes in inter-satellite links, and uneven network traffic distribution. These challenges make traditional terrestrial network routing algorithms difficult to apply in LEO satellite networks. Therefore, designing an efficient routing method is crucial and fundamental to achieving data transmission across large-scale LEO satellite networks.

[0003] Because traditional routing is typically based on fixed routing tables, it struggles to adapt to dynamic network environments and traffic changes. Routing strategies cannot be flexibly adjusted to accommodate the complex and ever-changing LEO satellite network environment, leading to resource waste and performance degradation. In recent years, AI-based routing methods have garnered widespread attention to address these issues. Leveraging the perception and decision-making capabilities of deep reinforcement learning (DRL) algorithms, LEO satellite networks can learn optimal routing decisions through training models. However, most existing research fails to consider the distance offset caused by real-time environmental changes during satellite decision-making when designing routing methods. In practical LEO satellite networks, the online inference time of DRL models typically fluctuates between 0.5ms and 2ms, depending on computing power and model size. In practical routing tasks, the online inference time for action selection is significant. The entire environment cannot be paused for at least 0.5ms to select an action for each packet. In other words, the network environment cannot remain static during the action selection process in DRL-based routing control tasks. Therefore, when applying traditional DRL algorithms to real-world satellite networks, problems such as latency uncertainty and high packet loss rates persist, ultimately limiting the throughput of LEO satellite networks. Summary of the Invention

[0004] The purpose of this application is to provide a low-orbit satellite communication network routing method based on federated deep reinforcement learning, which improves the throughput of the low-orbit satellite network.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] This application provides a low-orbit satellite communication network routing method based on federated deep reinforcement learning, including:

[0007] The satellites in the target low-orbit satellite network are clustered to obtain multiple clusters, and after clustering, each cluster is maintained according to the satellite status of each satellite; the cluster head in each cluster serves as the low-orbit satellite server, and the cluster members serve as the low-orbit satellite clients;

[0008] A federated learning framework is established in each cluster. Based on the federated learning framework, a routing decision model is trained using data from each low-orbit satellite client. When training the local model in the low-orbit satellite client, the routing decision process is constructed as a real-time Markov decision process for reinforcement learning. In the real-time Markov decision process, the current state is determined by the state and action at the previous moment. The local model corresponding to each low-orbit satellite client is used to output a routing decision for forwarding data.

[0009] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0010] This application provides a low-orbit satellite communication network routing method based on federated deep reinforcement learning. It solves the problem of real-time network environment decision-making in a dynamic environment through reinforcement learning of a real-time Markov decision process, reduces the distance offset caused by real-time changes in the environment during decision-making, thereby reducing latency and packet loss rate, and improving the throughput of the low-orbit satellite network. In addition, federated learning is used to improve the convergence speed of model training, thereby reducing the communication overhead of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A flowchart of a low-orbit satellite communication network routing method based on federated deep reinforcement learning provided in one embodiment of the present application;

[0013] Figure 2 A schematic diagram of the low-orbit satellite cluster initialization process provided in one embodiment of the present application;

[0014] Figure 3 A schematic diagram of the federated learning principle provided in one embodiment of the present application;

[0015] Figure 4 A schematic diagram of the reinforcement learning principle provided in one embodiment of the present application. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0017] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0018] This application provides a low-orbit satellite communication network routing method based on federated deep reinforcement learning, such as Figure 1 As shown, a low-orbit satellite communication network routing method based on federated deep reinforcement learning includes:

[0019] Step 101: Cluster the satellites in the target low-orbit satellite network to obtain multiple clusters, and maintain each cluster according to the satellite status of each satellite after clustering; the cluster head in each cluster serves as the low-orbit satellite server, and the cluster members serve as the low-orbit satellite clients.

[0020] Step 102: A federated learning framework is established in each cluster. Based on the federated learning framework, a routing decision model is trained using data from each low-orbit satellite client. When training the local model in the low-orbit satellite client, the routing decision process is constructed as a real-time Markov decision process for reinforcement learning. In the real-time Markov decision process, the current state is determined by the state and action at the previous moment. The local model corresponding to each low-orbit satellite client is used to output a routing decision for forwarding data.

[0021] In this application, the local model is used to output the routing decision for forwarding the output data of the corresponding low-orbit satellite. The global model trained based on the federated learning framework is the routing decision model.

[0022] The model parameters of the global model are initialized at the low-orbit satellite server of each cluster, and the model parameters of the global model are distributed to the low-orbit satellite clients in the cluster. The low-orbit satellite clients use the received model parameters of the global model as the initialization model parameters of the local model; the structures of the global model and the local model are the same.

[0023] This application solves the problem of real-time network environment decision-making in a dynamic environment by adopting a real-time Markov decision process (RTMDP). The self-attention mechanism is introduced in the global model and the local model to help the intelligent agent understand the semantics of the state elements, so as to realize adaptive adjustment of routing strategies and real-time intelligent routing according to the changing environmental conditions. At the same time, a federated learning mechanism (FL) based on satellite clustering is introduced to divide the entire low-orbit satellite constellation system into three parts: ground station, low-orbit satellite server (cluster head) and low-orbit satellite client (cluster member). Each low-orbit satellite client obtains the model sent by the low-orbit satellite server and trains the model locally using its own computing power. Each low-orbit satellite client only uploads the weights of the locally trained model. The model is aggregated on the low-orbit satellite server and sent to each low-orbit satellite client. Combining this with the DRL under the RTMDP framework can further improve the convergence speed of routing model training and reduce network communication overhead.

[0024] Wherein, step 101 specifically includes:

[0025] Step 1011: constructing a logical topology model of the target low-orbit satellite network using a virtual topology method; the logical topology model includes satellite nodes that correspond one-to-one to satellites in the target low-orbit satellite network.

[0026] Step 1012: clustering the satellite nodes in the logical topology model using a clustering algorithm to obtain multiple clusters, and determining the cluster head and cluster members of each cluster.

[0027] The logical topology model specifically includes the modeling of elements such as inter-satellite communication links, network topology structure, and satellite queue load. Through in-depth analysis of the system architecture of the low-orbit satellite constellation system, an accurate low-orbit giant constellation network routing model, namely the logical topology model, is constructed.

[0028] The target low-orbit satellite network of this application considers connecting ground users through a low-orbit satellite constellation, and users are connected to low-orbit satellite coverage and services. When the source user and the target user access satellite services from different low-orbit satellites, an inter-satellite link (ISL) relay is required. The inter-satellite link is divided into intra-orbit inter-satellite link and inter-orbit inter-satellite link. Each forwarding is defined as a hop, and the corresponding data packet is continuously forwarded through each satellite node to reach the target user, thereby establishing an end-to-end service. The source user is defined as the source node, and the target user is defined as the destination node.

[0029] The LEO satellite constellation is specifically a Walker-Delta constellation consisting of M×N satellites, where M represents the number of orbital planes in the constellation and N represents the number of satellites in each orbit. All orbits have the same inclination α and are equally spaced along the equator. The difference in right ascension of the ascending node between adjacent planes is ΔΩ = 2π / M. Satellites are evenly distributed across all planes. The phase difference between adjacent satellites in the same plane is ΔΦ = 2π / N. The phase offset between satellites in adjacent planes is Δf = 2πF / MN, where F is the phase factor.

[0030] Each low-orbit satellite is numbered (i, j), where i represents the orbit number of the satellite (i=1, 2, 3…M) and j represents the satellite number within the orbit (j=1, 2, 3…N). According to the constellation configuration, except for the satellites on both sides of the reverse gap and in the polar regions, each low-orbit satellite establishes inter-satellite links with the four adjacent low-orbit satellites.

[0031] The virtual topology method is used to construct the logical topology model of the target low-orbit satellite network. The partial topology of the first 100 satellites in the fourth group constellation model of Starlink Phase I is constructed through the virtual topology method. The satellite orbit altitude is 550km, the orbit inclination is 53°, the number of orbital planes is 10, the number of satellites in each orbital plane is 10, and the elevation angle is 25°.

[0032] This application adopts the virtual topology method to construct the logical topology model of the target low-orbit satellite network, specifically including: modeling the target low-orbit satellite communication network as a directed graph g t ={V t ,E t ,R t}.

[0033] Among them, the directed graph It is the logical topology model. Defined as a set of satellite nodes in time slot t, v i,t Satellite node i in time slot t, 1≤i≤MN.

[0034] definition is the set of links in time slot t, and a single link can be expressed as

[0035] definition is the maximum achievable rate of the intersatellite link in time slot t, R ij,t is the maximum achievable rate between satellite node i and satellite node j in time slot t.

[0036] To reduce end-to-end latency and improve the average network throughput, the optimization goal is to maximize the average system throughput: in Represents the path set of all end-to-end services corresponding to the routing in the network, l s Indicates the data volume of a single end-to-end service transmission data packet, t max It represents the time it takes for the last data packet in all services to reach the destination node. Therefore, the original problem can be equivalent to minimizing t max .

[0037] Intersatellite links (ISLs) can be divided into intra-orbital and inter-orbital links, depending on whether the low-orbit satellites at both ends of the ISL are in the same orbital plane. Generally, two continuous intra-orbital links can be established, while ISLs require continuous switching based on satellite motion. By default, all ISLs are initialized to a normal state, with the same transmission bandwidth and buffer queue size.

[0038] This application divides the time range of the dynamic network into T time slots of equal length, each time slot has a length of τ, and the total time is Tτ. Assume there are S end-to-end services, and each service is represented as a tuple (s s ,d s ,l s ,δ s ,H s ), which represent the source node, destination node, data size, delay tolerance, and number of hops experienced of the sth service respectively.

[0039] For data packet k, the processing delay T i comp It refers to the time delay from receiving the data packet to satellite node i making the routing decision; queuing delay T i queue It refers to the queuing waiting time required for routing decision at a certain satellite node i, and the transmission delay T i trans It refers to the time required for satellite node i to send a data frame, the propagation delay It refers to the time required for a signal to travel from one low-orbit satellite i to another low-orbit satellite j along the intersatellite link.

[0040] Each low-orbit satellite includes a receive queue and four transmit queues. The receive queue is used to store received data packets. Each of the four transmit queues transmits data with a low-orbit satellite via an inter-satellite link. Data packets in the receive queue are transferred to one of the four transmit queues based on the routing decision output by the trained local model.

[0041] The maximum length of the receiving queue is limited by the storage capacity C of the satellite node. iAssume that the maximum length of each of the four directional transmit queues is m, and each of the four directional transmit queues is associated with each of the four intersatellite links. Packets in the receive queue are transferred to one of the four transmit queues based on the routing policy and then propagated to the next-hop satellite via the corresponding antenna. Each queue follows the first-in, first-out (FIFO) principle.

[0042] The present invention aims to reduce end-to-end delay and improve the average throughput of the network, and sets the optimization goal to maximize the average throughput of the system.

[0043] The clustering algorithm of the present invention uses the comprehensive weight of satellite nodes as an important basis for cluster head election and cluster structure maintenance. The weight calculation mainly considers factors such as connectivity, payload service capability, and priority. In order to reduce the additional overhead caused by clustering, the weight calculation and initialization stages are completed by the ground control terminal.

[0044] In an exemplary embodiment, clustering satellites in a target low-orbit satellite network to obtain multiple clusters specifically includes: clustering satellites in the target low-orbit satellite network to obtain multiple clusters through a ground station; and the ground station is communicatively connected to the target low-orbit satellite network.

[0045] The clustering algorithm consists of two main phases: cluster initialization and maintenance. The initialization phase is responsible for partitioning all node states at the initial moment and designating appropriate nodes (satellite nodes) as cluster heads. The maintenance phase is responsible for maintaining the cluster structure when the cluster network topology changes, ensuring that all nodes remain in the cluster structure.

[0046] The cluster initialization phase is completed by the ground terminal, which calculates the weight of each node based on the cluster information and divides the node status.

[0047] After the initialization phase is completed, the target low-orbit satellite network immediately enters the cluster maintenance phase, and each node completes its own state change through certain information exchanges.

[0048] There are four main types of node state changes: cluster head node state change, cluster member node state change, free node state change, and undetermined node state change.

[0049] The clustering algorithm is used to cluster the satellite nodes in the logical topology model to obtain multiple clusters, and the cluster head and cluster members of each cluster are determined, such as Figure 2 As shown, specifically including:

[0050] The weight of each satellite node is determined according to the connectivity of each satellite node in the logical topology model.

[0051] The calculation formula of the satellite node weight is:

[0052]

[0053] Among them, α i is the average connectivity of satellite node i, and α i is defined as the weight of satellite node i, and T is the number of divided time slices. i (t) represents the number of neighbor nodes of satellite node i in the tth time slice (time slot he).

[0054] in, a ij 1 indicates that satellite node i and satellite node j are connected in the tth time slice, a ij If it is 0, it means that satellite node i and satellite node j are not connected in the tth time slice.

[0055] Determine whether the satellite node at the current moment has a neighbor node. If no neighbor node exists, the satellite node is determined to be in a free state.

[0056] If there are neighbor nodes, determine whether there is a cluster head node among the neighbor nodes.

[0057] If there is a cluster head node, the satellite node is added to the cluster with the highest priority among the clusters corresponding to the existing cluster head node, and the satellite node is determined to be a member of the cluster.

[0058] The formula for priority measurement is:

[0059] Among them, w(i,CH v ) represents the priority metric of satellite node i joining cluster head v; a v represents the weight of cluster head v; n represents the number of nodes in the cluster, and h represents the number of hops from satellite node i to cluster head v. v Indicates the cluster where the cluster head v is located.

[0060] If there is no cluster head node, the weight of the satellite node is compared with that of all satellite nodes in an undetermined state within K hops. If the weight of the satellite node is the largest, the satellite node is determined to be the cluster head (the number of nodes in the cluster is initialized to 1). If the weight of the satellite node is not the largest, the process returns to the step of determining whether the satellite node at the current moment has a neighboring node. If there is no neighboring node, the satellite node is determined to be in a free state, until the node states of all satellite nodes in the logical topology model are determined. The node states include free state, undetermined state, normal state and cluster head. The normal state indicates that the satellite node is a member of the cluster. K is a set value.

[0061] After clustering the satellite nodes in the logical topology model using a clustering algorithm to obtain multiple clusters and determining the cluster head and cluster members of each cluster, the low-orbit satellite communication network routing method based on federated deep reinforcement learning also includes: performing cluster maintenance when the logical topology model changes.

[0062] The cluster maintenance is performed when the logical topology model changes, specifically including the following steps:

[0063] 1) Cluster head state change: When the cluster head has no neighbor nodes, the node state of the satellite node corresponding to the cluster head is converted to the free state.

[0064] 2) Cluster Member Node State Changes: When a satellite node in normal state is about to be disconnected from its cluster head, or when the number of hops exceeds K and the duration exceeds a set time, a leave request is sent to the cluster head and the satellite node is checked to see if it has any neighboring nodes. If so, the satellite node's state transitions to the undetermined state; if not, the satellite node transitions to the free state. When the distance between two satellites exceeds a maximum threshold, a disconnection message is sent via the intersatellite link; therefore, exceeding the maximum threshold is considered a condition for disconnection.

[0065] 3) When a satellite node in a free state retrieves a connection relationship with other satellite nodes, the state of the satellite node is converted to an undetermined state.

[0066] 4) Undetermined node state change: The undetermined state is a temporary state. The node cannot stay in the undetermined state and needs to immediately change to the member state or cluster head state.

[0067] Initialize the global model parameters of the above agent on the low-orbit satellite server and distribute the model to each LE0 client participating in the connection. Each LE0 client trains the model locally.

[0068] Federated learning is a decentralized machine learning approach that aims to train a global model without transmitting raw data from devices (LEO clients) to a central server (LEO server).

[0069] Federated learning includes two architectures, homogeneous architecture and heterogeneous architecture. In homogeneous architecture, the global model and the local model are structurally consistent, while in heterogeneous architecture, they are inconsistent. This application uses the homogeneous architecture to randomly initialize the global model parameters (weights and biases, etc.) of the above-mentioned intelligent agent on the low-orbit satellite server.

[0070] The global model is distributed to each participating low-orbit satellite client through the inter-satellite link within the same cluster, and each low-orbit satellite client trains the model locally.

[0071] After updating the parameters, the LEO satellite client in this application transmits the compressed model to the LEO satellite server. The LEO satellite server aggregates the received models and updates the parameters in the global model. If the updated global model achieves the preset model accuracy in the test task, the routing algorithm training is considered complete. Otherwise, the above steps are repeated until the global model achieves the preset model accuracy in the test task.

[0072] In an exemplary embodiment, step 102 specifically includes: triggering a program to train a routing decision model using data from each low-orbit satellite client at a set interval, so that the routing decision model can achieve the purpose of adaptive adjustment according to changing environmental conditions.

[0073] Each time the routing decision model is trained using data from each low-orbit satellite client, the following steps are specifically performed:

[0074] In each training iteration, the local model in each cluster is traversed in turn.

[0075] Perform local model training on each low-orbit satellite client in the currently traversed cluster.

[0076] The model parameters of the currently trained local model are transmitted to the corresponding low-orbit satellite server.

[0077] The low-orbit satellite service end aggregates the received model parameters of the currently trained local model, and uses the aggregated model parameters to update the model parameters of the global model in each cluster.

[0078] The global model with the updated model parameters is tested for model accuracy. If the preset model accuracy is achieved, training stops and the global model that has achieved the preset model accuracy is used as the routing decision model. Otherwise, the low-orbit satellite server distributes the model parameters of the current global model to each low-orbit satellite client, updates the model parameters of each local model, and returns to the step of traversing the local model in each cluster in turn.

[0079] The aforementioned FedProx algorithm is used to train the routing decision model using data from each LEO satellite client. FedProx is an improved federated learning algorithm designed to address the issues of non-IID data and device heterogeneity in federated learning. During each training iteration, a subset of clients (cluster) is selected to perform local model updates and upload the updated results to the corresponding LEO satellite server for aggregation, ultimately achieving global model convergence.

[0080] FedProx introduces a proximal term based on the FedAvg loss function. The model parameters obtained by the LEO satellite client after local training will not deviate too much from the initial server model parameters. When the local model is trained locally on the LEO satellite client, it is necessary to minimize the following objective function:

[0081]

[0082] Among them, h k (w; w t ) is the objective function value for local training, w represents the model parameters of the local model, w t represents the model parameters of the global model, F k (w) represents the minimized local function of the local model k, and μ represents an adjustable parameter. The entire LEO satellite constellation system consists of three parts: a ground station, a LEO satellite server (cluster head), and multiple LEO satellite client nodes (cluster members). In the federated learning mechanism, each LEO satellite client obtains the model delivered by the LEO satellite server and trains the received model locally.

[0083] After multiple rounds of training, the low-orbit satellite clients in the low-orbit satellite cluster will transmit the compressed training model to the low-orbit satellite server. The low-orbit satellite server will aggregate the received models and return the updated parameters to the relevant low-orbit satellite clients.

[0084] The training method of the local model and the global model in the low-orbit satellite client in the federated learning mechanism is specifically an improved multi-agent deep reinforcement learning routing algorithm based on RTMDP and self-attention designed in this application, which is mainly used to reduce end-to-end latency and improve low-orbit satellite network throughput. The training method specifically includes the following contents:

[0085] The routing performance optimization problem of low-orbit satellite networks is transformed into a real-time Markov decision process to better describe the system uncertainty and randomness, effectively handle complex decision problems, and match the real-time characteristics in real routing scenarios.

[0086] Typically, Markov decision processes are episodic, where the agent selects an action while the environment and the agent remain static until the agent receives the next state from the environment. Unlike these, in RTMDPs, the action and state change simultaneously, with the new state-action pair being determined by the previous state-action pair. Therefore, simply selecting an action is insufficient to determine the next state-action pair.

[0087] Corresponding to the action-value function under the traditional MDP, the action-value function of the real-time Markov decision process is expressed as:

[0088]

[0089] Among them, s t is the state in time slot t, s t+1 is the state in time slot t+1, a t is the action in time slot t, a t =a t-1 , a t is the action in the previous time slot t-1, a t+1 is the action in time slot t+1, r(st,at) is the reward in time slot t, In a given state s t and action a t , the agent moves to state s t+1 The expected action value under the corresponding probability distribution; for the action value function, Indicates that based on the current policy π, given a state-action pair (s t+1 ,a t ) and take action a t+1 The expectation of the posterior local model estimate, Indicates that in the state-action pair (s t ,a t ) Take action a t The corresponding action-value function, Indicates that in the state-action pair (s t+1 ,a t ) Take action a t+1 The corresponding estimated value.

[0090] Among them, special attention should be paid to the non-bold a t The bold a is the action in the previous time slot t-1. t is the action within time slot t.

[0091] According to the above formula, action a t There is no direct effect on the reward or the next state, action a t The only impact a t+1 , r(st +1 ,at +1 ) and s t+2 have an impact.

[0092] Similarly, corresponding to the state value function under the traditional MDP, the state value function of the real-time Markov decision process is expressed as:

[0093]

[0094] in, For s t and a t The state-value function of the state-action pair; To satisfy a t+1 =a t Under the condition, s t+1 and a t The state value of the state-action pair; Indicates that based on the current policy π, given a state-action pair (s t ,a t ), the expectation of the state-value function of the local model under .

[0095] According to the above formula, no matter which action a is chosen, t , s t 、a t 、s t+1 There is always a valid transformation. Therefore, when using a real-time framework, discrete policy learning can be performed using the value function.

[0096] Assuming that changes in the LEO satellite network state occur at the initial stage of each time slice ΔT, based on the characteristics of a real-time Markov decision process, we define the five elements {s, a, μ, p, r} required for a real-time Markov decision process. These elements represent: s represents the current state set, a represents the action set, μ represents the initial state distribution, p represents the state transition probability, and r represents the reward function. Each LEO satellite (specifically, its corresponding local model) is considered an intelligent agent. Currently, there is no parameter sharing between agents, and they are trained independently. The two parameters μ and p, which are part of the five elements of the Markov decision process, are randomly generated through automatic model initialization and are continuously updated during training.

[0097] The structures of the global model and the local model both include a fully connected layer, an attention layer and a Softmax layer.

[0098] Due to the complex environment of low-orbit satellite communication networks, each agent has a large state space and needs to process a large amount of observation information from the environment. To learn the semantics of complex elements, a self-attention mechanism is used. This self-attention mechanism assigns weights to observation elements in the neural network by improving the traditional soft attention mechanism to help the agent understand the semantics of state elements and the actions derived from the synthetic state. The basic principle of the self-attention mechanism can be expressed as follows:

[0099]

[0100] Among them, Q is the query matrix, K is the key matrix, V is the value matrix, d k is the dimension of each column feature in the key matrix K, T is the transpose, and in the self-attention mechanism, the Q query matrix is ​​a transformation of its own input, not an input from the outside.

[0101] In the low-orbit satellite communication network routing scenario, the self-attention layer learns the relevant key matrix of the satellite network state and assigns dynamic weights to it. t Convert it to a one-dimensional vector X. Then, the attention layer calculates the attention score for the input vector, performs Softmax calculation for normalization, obtains the weighted sum, and concatenates the weighted sums together to obtain the final output of the entire neural network.

[0102] Using deep reinforcement learning technology, through collaboration and learning between intelligent agents, feedback and rewards are continuously obtained from the low-orbit giant constellation network environment to optimize the inter-satellite routing strategy to improve the routing performance and throughput of the entire target low-orbit satellite network; each low-orbit satellite contains two deep neural networks: an estimated Q network and a target Q network.

[0103] At the beginning of local model training, the experience replay pool and network parameters are initialized, source and destination nodes are randomly generated in the low-orbit satellite communication network topology, and routing decisions are made from the source nodes. During the decision-making process, the agent makes decisions using a dynamic greedy strategy.

[0104] Before updating the Q network, the agent selects a small batch of transition samples from the experience replay pool and uses these samples to calculate the error loss through the target Q network and the estimated Q network to update the model parameter weights of the local model.

[0105] In this application, each intelligent agent makes decisions based on the status information of adjacent nodes, which conforms to the satellite distributed architecture in actual environments, enhances the adaptability to low-orbit satellite network routing, further improves the model convergence speed of the routing algorithm based on multi-agent reinforcement learning, and reduces network communication overhead.

[0106] To address the challenges brought by statistical distribution differences, the FedProx algorithm adds a correction term to the local training objective function, thereby effectively alleviating the adverse effects of local training update divergence on global convergence.

[0107] Specifically, this correction term can constrain the local training target of the low-orbit satellite client so that it can be close to the global training target. It reduces the negative impact of statistical distribution differences on global convergence without the need for manual intervention to adjust the local duration, and enables the low-orbit satellite server to safely aggregate a dynamic number of local model update results.

[0108] All LEO satellite clients participating in the training download the global model from the LEO satellite server as their respective local models, and use local data to train for multiple epochs. After the training is completed, the model parameters and gradients of the local model are encrypted and uploaded to the LEO satellite server.

[0109] The LEO satellite server aggregates the uploaded LEO satellite client model parameters and uses the FedProx algorithm to update the server global model parameters. The updated parameters are then returned to the relevant LEO satellite clients. The specific algorithm process is as follows:

[0110] Initialize the input low-orbit satellite cluster set as The set of low-orbit satellite clients in the cluster is The set of model parameters within the cluster is

[0111] c1,c2,…c M They are the 1st cluster, the 2nd cluster,…, the Mth cluster respectively.

[0112] d1,d2,...,d N They are the first low-orbit satellite client, the second low-orbit satellite client, ..., the Nth low-orbit satellite client.

[0113] are the model parameters of the first global model, the model parameters of the second global model, …, the model parameters of the Mth global model.

[0114] Traverse each cluster and the client devices in each cluster.

[0115] Calculate model update weights

[0116] where h n (w; w t ) is the minimization objective function, F n (ω) is the minimized local function of the local model n

[0117] It is the difference between the parameters of the global model and the parameters of the client model. μ is an adjustable parameter. Reducing it can reduce the constraint effect.

[0118] Each LEO satellite client device sends updated weights To the low-orbit satellite service terminal.

[0119] The low-orbit satellite server updates the global model, that is, the cluster head averages and aggregates the model parameters of the local model of each node in the cluster, which is the following formula: Where N is the total number of low-orbit satellite clients in the current cluster.

[0120] If the updated global model achieves the preset model accuracy in the test task, the routing algorithm training is considered complete. Otherwise, the above steps are repeated until the global model achieves the preset model accuracy in the test task.

[0121] Due to the instability of intersatellite link connection, the LEO link may fail after a certain session. At this time, the LEO satellite server will hand over the routing task to the upcoming candidate LEO satellite server and ground station, and the new LEO satellite server will re-cluster the client to continue the task. The overall training process is as follows: Figure 3 As shown, the CH-CH link is the link between cluster heads, the CH-CM link is the link between the low-orbit satellite server and the low-orbit satellite client within the cluster, and the CM-CM link is the link between the low-orbit satellite client and the low-orbit satellite client within the cluster.

[0122] In this federated learning mechanism, each LEO satellite client is trained by an independent agent, which makes decisions based on the state information of neighboring nodes in its own cluster. The routing decision process for the entire network is structured as a real-time Markov decision process. The agent acquires actions and makes routing decisions through a fully connected layer, an attention layer, and a softmax layer.

[0123] The dynamic routing decision problem of each satellite is modeled as a real-time Markov decision process, and each satellite is regarded as an intelligent agent. The routing decision process of the entire network is constructed as a real-time Markov decision process. The specific process is as follows:

[0124] Based on the characteristics of the real-time Markov decision process, the five elements {s, a, μ, p, r} that need to be included in the real-time Markov decision process are defined. Their meanings are: s is the current state set, a is the action set, μ is the initial state distribution, p is the state transition probability, and r is the reward function.

[0125] The elements in the real-time Markov decision process include state (environment state), action and reward function.

[0126] The status is represented as:

[0127] in, is the state set of the current satellite i (satellite node i) preparing to route and forward the data packet p in time slot t, φ i,p The source node of the data packet p that is currently being forwarded by satellite i in time slot t, The destination node of the data packet p that the current satellite i prepares to route and forward in time slot t. Both the source node and the destination node are satellite nodes. is the queue blocking rate of the receiving queue of the satellite in the current time slot, is the link connection status between satellite i in the current time slot and its adjacent low-orbit satellite, is the action performed in the last time slot of the current satellite i. Since the RTMDP mechanism is added in this application, the last time slot t of the current satellite node i is pre Actions performed Also added to the status.

[0128] Action a is defined as selecting the next-hop satellite node. Based on the current state, the agent can select its own neighbor node for the next-hop routing. The action is expressed as:

[0129] in, is the action set of the current satellite i preparing the routing forwarding data packet p in time slot t. When , it means the agent chooses v j As the next hop low-orbit satellite for forwarding data packet p, represents the set of neighboring satellites of the current satellite in time slot t, and each link is a bidirectional data packet transmission link, v j is low-orbit satellite j (satellite node j), and the agent is the local model corresponding to the current satellite i;

[0130] The reward function is expressed as:

[0131] The goal of this reward function is to ensure that the data packets are transmitted to their destination nodes while increasing the number of data packets arriving at the destination nodes, minimizing the average delay, and appropriately distributing the satellite load.

[0132] Among them, r t i,p is the reward function after the current satellite i routes and forwards the data packet p to the destination node in time slot t, ΔD p is the difference in propagation delay, specifically defined as the last time slot t pre When , the difference between the propagation delay from satellite i where data packet p is located to destination node d and the propagation delay from satellite j where data packet p is located to destination node d is, D is the propagation time from satellite i where data packet p is located to destination node d, p (v j,t ,v d ) is the propagation time from satellite j to destination node d for data packet p, v d For the destination node d, v j,t At time slot t, at satellite j, In time slot t pre Satellite i, time slot t pre is the previous time slot of time slot t, k1 and k2 are penalty coefficients, O t is the total amount of remaining data packets in the target low-orbit satellite network in time slot t, represents the receiving queue load reward of satellite j in time slot t, Indicates the queue load in time slot t, the queue load in the current time slot The smaller the The bigger.

[0133] like Figure 4 As shown, during the initial training of the agent, RTMDP will be based on the initial state s t Use the greedy strategy to randomly generate an action a t , use a t Make the routing decision for this round. After routing, the environment will return the reward r for the current round. t And the action of the previous moment is a t and the current state s t+1 Splice as the new state at this moment (s t+1 ,a t ), the agent continues to select actions through this state, and obtains a new action a through the fully connected layer, attention layer and Softmax layer t And make a new round of routing decisions. The specific algorithm process is as follows:

[0134] Initialize the algorithm parameters such as the number of training rounds, simulation time, number of satellite nodes, and node queue length. The node queue length includes the sending queue length and the receiving queue length.

[0135] After each round of training, the satellite queue is cleared and data packets are generated and sent to the target low-orbit satellite network.

[0136] At a specific simulation time, each satellite node is traversed to perform agent training.

[0137] Traversing each data packet in the satellite queue, the agent initializes past actions based on the greedy strategy at the initial stage of training implement And get the status s t .

[0138] Splicing t and is a vector to select an action.

[0139] The attention layer passes Get the corresponding weight w; vector After passing through the fully connected layer, it is multiplied by the weight w.

[0140] Output action a t , the next hop node and the current receive queue capacity.

[0141] If the current receiving queue capacity is less than the maximum queue capacity, perform action a t , and receive a reward of r t , the next state s t+1, and the end binary variable done. Otherwise, the data packet is placed at the end of the next node's send queue. When the data packet arrives at the destination node, its own binary variable attributes are automatically changed. The end binary variable is used to indicate whether the data packet has arrived at the corresponding destination node.

[0142] Update past actions

[0143] If the end binary variable done is not equal to 0, the number of data packets arriving at the destination node is increased by 1.

[0144] After the current data packet routing is completed, the agent will store the transition sample (s t ,a t ,r t ,s t+1 ,done) to experience pool D.

[0145] Sampling transition samples (s) from the experience pool D t ,a t ,r t ,s t+1 ,done), and by minimizing the loss function The local model weights are updated via the alternating direction method of multipliers (ADMM) algorithm.

[0146] where N b is the random sampling capacity, s t is the current state, a t is the current action. θ is the parameter of the estimated Q network, and Q() represents the estimated Q network. t The next state s of the target Q network is calculated by t+1 When using action a t+1 The target Q value is obtained.

[0147] y t =r t +γQ(s t+1 ,argmax a Q(s t+1 ,a t+1 |θ);θ').

[0148] Among them, r t Indicates that the agent is in state s t Take action t The parameter of the target Q network is θ', and γ∈[0,1] is expressed as the decay factor.

[0149] Through the above process, training is continued until the algorithm converges, that is, the model accuracy of the global model reaches the preset model accuracy, and finally a stable routing decision algorithm (global model) can be obtained.

[0150] This application introduces RTMDP to the field of low-orbit satellite network routing. The RTMDP mechanism is a continuous, uninterrupted action selection mechanism. The state and actions of the agent can evolve simultaneously to match the real-time characteristics of the actual routing scenario. This solves the problem of distance deviation from the actual situation caused by the static environment when making routing decisions in a dynamic environment, and enhances the effectiveness of the routing control method. To better learn routing strategies, a self-attention mechanism is added to help the agent understand the semantics of state elements. By cutting the input synthetic network state into independent elements and learning to assign different weights to them during training, it has better performance and faster convergence speed than traditional fully connected neural networks.

[0151] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A low-orbit satellite communication network routing method based on federated deep reinforcement learning, characterized in that: The low-orbit satellite communication network routing method based on federated deep reinforcement learning includes: The satellites in the target low-orbit satellite network are clustered to obtain multiple clusters, and after clustering, each cluster is maintained according to the satellite status of each satellite; the cluster head in each cluster serves as the low-orbit satellite server, and the cluster members serve as the low-orbit satellite clients; A federated learning framework is established in each cluster. Based on the federated learning framework, a routing decision model is trained using data from each low-orbit satellite client. When training the local model in the low-orbit satellite client, the routing decision process is constructed as a real-time Markov decision process for reinforcement learning. In the real-time Markov decision process, the current state is determined by the state and action at the previous moment. The local model corresponding to each low-orbit satellite client is used to output a routing decision for forwarding data. The action-value function of the real-time Markov decision process is expressed as: Among them, s t is the state in time slot t, s t+1 is the state in time slot t+1, a t is the action in time slot t, a t =a t-1 , a t is the action in the previous time slot t-1, a t+1 is the action in time slot t+1, r(s t ,a t ) is the reward in time slot t, Indicates that in a given state s t and action a t , the agent moves to state s t+1 The expected action value under the corresponding probability distribution; for the action value function, Indicates that based on the current policy π, given a state-action pair (s t+1 ,a t ) and take action a t+1 The expectation of the posterior local model estimate, Indicates that in the state-action pair (s t ,a t ) Take action a t The corresponding action-value function, Indicates that in the state-action pair (s t+1 ,a t ) Take action a t+1 The corresponding estimated value; among them, action a t Does not affect the reward and the next state, action a t Only affects the next action a t+1 ; In the next time slot, action a t Will affect r(s t+1 ,a t+1 ); The state value function of the real-time Markov decision process is expressed as: in, For s t and a t The state-value function of the state-action pair; To satisfy a t+1 =a t Under the condition, s t+1 and a t The state value of the state-action pair; Indicates that based on the current policy π, given a state-action pair (s t ,a t ), the expectation of the state-value function of the local model under .

2. The low-orbit satellite communication network routing method based on federated deep reinforcement learning according to claim 1 is characterized in that: The satellites in the target low-orbit satellite network are clustered to obtain multiple clusters, including: A virtual topology method is used to construct a logical topology model of a target low-orbit satellite network; the logical topology model includes satellite nodes corresponding one-to-one to satellites in the target low-orbit satellite network; A clustering algorithm is used to cluster the satellite nodes in the logical topology model to obtain multiple clusters, and the cluster head and cluster members of each cluster are determined.

3. The low-orbit satellite communication network routing method based on federated deep reinforcement learning according to claim 2 is characterized in that Clustering the satellite nodes in the logical topology model into multiple clusters using a clustering algorithm, and determining the cluster head and cluster members of each cluster, specifically including: Determining the weight of each satellite node according to the connectivity of each satellite node in the logical topology model; Determine whether the satellite node at the current moment has a neighboring node. If no neighboring node exists, the satellite node is considered to be in a free state. If there are neighbor nodes, determine whether there is a cluster head node among the neighbor nodes; If there is a cluster head node, the satellite node is added to the cluster with the highest priority among the clusters corresponding to the existing cluster head node, and the satellite node is determined to be a member of the cluster; If there is no cluster head node, the weight of the satellite node is compared with that of all satellite nodes in an undetermined state within K hops. If the weight of the satellite node is the largest, the satellite node is determined to be the cluster head. If the weight of the satellite node is not the largest, the process returns to the step of determining whether the satellite node at the current moment has a neighboring node. If there is no neighboring node, the satellite node is determined to be in a free state, until the node states of all satellite nodes in the logical topology model are determined. The node states include free state, undetermined state, normal state and cluster head. The normal state indicates that the satellite node is a member of the cluster. K is a set value.

4. The low-orbit satellite communication network routing method based on federated deep reinforcement learning according to claim 3 is characterized in that: After clustering the satellite nodes in the logical topology model using a clustering algorithm to obtain a plurality of clusters and determining a cluster head and cluster members of each cluster, the low-orbit satellite communication network routing method based on federated deep reinforcement learning further includes: performing cluster maintenance when the logical topology model changes; The cluster maintenance performed after the logical topology model changes specifically includes: When the cluster head has no neighbor nodes, the node state of the satellite node corresponding to the cluster head is converted to the free state; When the number of hops between a satellite node in a normal state and the cluster head in its cluster is greater than K and the duration exceeds the set time, a leave request is sent to the cluster head and a query is performed to determine whether the satellite node has any neighboring nodes. If so, the state of the satellite node is converted to an undetermined state; if not, the state of the satellite node is converted to a free state. When a satellite node in a free state retrieves a connection relationship with other satellite nodes, the state of the satellite node is converted to an undetermined state.

5. The low-orbit satellite communication network routing method based on federated deep reinforcement learning according to claim 1, characterized in that: The method includes: using the data of each low-orbit satellite client to train the routing decision model, specifically including: triggering a program for using the data of each low-orbit satellite client to train the routing decision model at a set time interval; Each time the routing decision model is trained using data from each low-orbit satellite client, the following steps are specifically performed: In each training iteration, the local model in each cluster is traversed in turn; Perform local model training on each low-orbit satellite client in the currently traversed cluster; Transmit the model parameters of the currently trained local model to the corresponding low-orbit satellite server; The low-orbit satellite service end aggregates the model parameters of the received currently trained local model, and uses the aggregated model parameters to update the model parameters of the global model in each cluster; The global model with the updated model parameters is tested for model accuracy. If the preset model accuracy is achieved, training stops and the global model that has achieved the preset model accuracy is used as the routing decision model. Otherwise, the low-orbit satellite server distributes the model parameters of the current global model to each low-orbit satellite client, updates the model parameters of each local model, and returns to the step of traversing the local model in each cluster in turn.

6. The low-orbit satellite communication network routing method based on federated deep reinforcement learning according to claim 1, characterized in that: Each low-orbit satellite includes a receive queue and four transmit queues. The receive queue is used to store received data packets. Each of the four transmit queues transmits data with a low-orbit satellite via an inter-satellite link. Data packets in the receive queue are transferred to one of the four transmit queues based on the routing decision output by the trained local model.

7. The low-orbit satellite communication network routing method based on federated deep reinforcement learning according to claim 6, characterized in that: The elements in the real-time Markov decision process include state, action and reward function; The status is represented as: in, is the state set of the current satellite i preparing to route and forward the data packet p in time slot t, φ i,p The source node of the data packet p that is currently being forwarded by satellite i in time slot t, The destination node of the data packet p that the current satellite i prepares to route and forward in time slot t. Both the source node and the destination node are satellite nodes. is the queue blocking rate of the receiving queue of the satellite in the current time slot, is the link connection status between satellite i in the current time slot and its adjacent low-orbit satellite, The action performed in the previous time slot of the current satellite i; The action is represented as: in, is the action set of the current satellite i preparing the routing forwarding data packet p in time slot t. When , it means the agent chooses v j As the next hop low-orbit satellite for forwarding data packet p, represents the set of neighboring satellites of the current satellite in time slot t, v j is satellite j, and the agent is the local model corresponding to the current satellite i; The reward function is expressed as: in, is the reward function after the current satellite i routes and forwards the data packet p to the destination node in time slot t, ΔD p is the difference between the propagation delay from satellite i where data packet p is located to the destination node d and the propagation delay from satellite j where data packet p is located to the destination node d, D is the propagation time from satellite i where data packet p is located to destination node d, p (v j,t ,v d ) is the propagation time from satellite j to destination node d for data packet p, v d For the destination node d, v j,t At time slot t, at satellite j, In time slot t pre Satellite i, time slot t pre is the previous time slot of time slot t, k1 and k2 are penalty coefficients, O t is the total amount of remaining data packets in the target LEO satellite network in time slot t, represents the receiving queue load reward of satellite j in time slot t, ξ t j represents the queue load in time slot t.

8. The low-orbit satellite communication network routing method based on federated deep reinforcement learning according to claim 1, characterized in that: The satellites in the target low-orbit satellite network are clustered to obtain multiple clusters, including: The satellites in the target low-orbit satellite network are clustered to obtain a plurality of clusters through a ground station; the ground station is communicatively connected with the target low-orbit satellite network.

9. The low-orbit satellite communication network routing method based on federated deep reinforcement learning according to claim 5, characterized in that: The structures of the global model and the local model both include a fully connected layer, an attention layer and a Softmax layer.

Images