Low earth orbit satellite communication network routing method based on federal deep reinforcement learning

By adopting federated deep reinforcement learning and real-time Markov decision-making process in low-orbit satellite networks, combined with self-attention mechanism, the performance problems of traditional routing algorithms in dynamic environments are solved, and the throughput and model of low-orbit satellite networks are improved. Training efficiency.

CN118921102AActive Publication Date: 2024-11-08BEIJING INFORMATION SCI & TECH UNIV

Patent Information

Application Number
CN202411162084.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-08
Estimated Expiration
2044-08-22

AI Technical Summary

Technical Problem

Traditional routing algorithms are difficult to adapt to the dynamic environment and traffic changes of low-orbit satellite networks, resulting in waste of resources and degradation of performance. The existing DRL model has problems of uncertain delay and high packet loss rate in actual applications, which limits low-orbit satellite networks. throughput.

Method used

Using a method based on federated deep reinforcement learning, low-orbit satellite networks are clustered and a federated learning framework is built in each cluster. The routing decision model is trained through real-time Markov decision-making process and self-attention mechanism to reduce decision delay and packet loss rate to improve the convergence speed of model training.

Benefits of technology

It effectively solves the problem of real-time network environment decision-making in dynamic environments, reduces latency and packet loss rates, improves the throughput of low-orbit satellite networks, and improves the efficiency of model training through federated learning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118921102A_ABST
    Figure CN118921102A_ABST
Patent Text Reader

Abstract

The invention discloses a low earth orbit satellite communication network routing method based on federal deep reinforcement learning, and relates to the technical field of satellite communication, and the method comprises the steps: carrying out the clustering of satellites in a target low earth orbit satellite network, obtaining a plurality of clusters, and carrying out the maintenance of each cluster according to the state of the satellites after clustering; a cluster head in each cluster is used as a low-orbit satellite server, and cluster members are used as low-orbit satellite clients; building a federated learning framework in each cluster, and training a routing decision model by adopting data of each low-orbit satellite client based on the federated learning framework; when a local model in a low earth orbit satellite client is trained, a routing decision process is constructed into a real-time Markov decision process for reinforcement learning; the state at the current moment in the real-time Markov decision process is determined by the state and the action at the previous moment; and the local model corresponding to each low-orbit satellite client is used for outputting a routing decision for routing and forwarding the data. According to the invention, the throughput of the low-orbit satellite network is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of satellite communication technology, and in particular to a low-orbit satellite communication network routing method based on federated deep reinforcement learning. Background Art

[0002] With the reduction of satellite launch costs, low earth orbit (LEO) satellite networks have become a feasible solution for providing global broadband access. Compared with ground networks, low earth orbit satellite networks have many advantages such as seamless global coverage and no geographical and infrastructure restrictions. However, low earth orbit satellite networks also have some challenges, such as the high speed of low earth orbit satellites relative to the ground, frequent changes in inter-satellite link status, and uneven distribution of network traffic, which makes it difficult to apply traditional ground network routing algorithms in low earth orbit satellite networks. Therefore, in order to achieve large-scale data transmission in low earth orbit satellite networks, designing an efficient routing method is the key and foundation.

[0003] Since traditional routing is usually based on fixed routing tables, it is difficult to adapt to dynamic network environments and traffic changes, and it is impossible to flexibly adjust routing strategies to adapt to the complex and changeable low-orbit satellite network environment, resulting in resource waste and performance degradation. In recent years, in order to solve the above problems, routing methods based on artificial intelligence have received widespread attention. With the help of the perception and decision-making capabilities of the Deep Reinforcement Learning (DRL) algorithm, low-orbit satellite networks can learn the optimal routing decisions through training models. However, most existing studies do not consider the distance offset caused by real-time changes in the environment when designing routing methods. In actual low-orbit satellite networks, depending on the computing power and model size, the online reasoning time of the DRL model generally fluctuates between 0.5ms and 2ms. In actual routing tasks, the online reasoning time of action selection cannot be ignored. The entire environment cannot be paused for at least 0.5ms to select an action for each data packet. That is, in the action selection process based on the DRL routing control task, the network environment cannot be static. Therefore, when the traditional DRL algorithm is applied to actual satellite networks, there are still problems such as uncertain delay and high packet loss rate, which ultimately limit the throughput of low-orbit satellite networks. Summary of the invention

[0004] The purpose of this application is to provide a low-orbit satellite communication network routing method based on federated deep reinforcement learning, which improves the throughput of the low-orbit satellite network.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] The present application provides a low-orbit satellite communication network routing method based on federated deep reinforcement learning, including:

[0007] The satellites in the target low-orbit satellite network are clustered to obtain multiple clusters, and after clustering, each cluster is maintained according to the satellite status of each satellite; the cluster head in each cluster serves as the low-orbit satellite server, and the cluster members serve as the low-orbit satellite clients;

[0008] A federated learning framework is built in each cluster. Based on the federated learning framework, the data of each low-orbit satellite client is used to train the routing decision model. When training the local model in the low-orbit satellite client, the routing decision process is constructed as a real-time Markov decision process for reinforcement learning. In the real-time Markov decision process, the state at the current moment is determined by the state and action at the previous moment. The local model corresponding to each low-orbit satellite client is used to output the routing decision for forwarding the data.

[0009] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0010] The present application provides a low-orbit satellite communication network routing method based on federated deep reinforcement learning, which solves the problem of real-time network environment decision-making in a dynamic environment through reinforcement learning of a real-time Markov decision process, reduces the distance offset caused by real-time changes in the environment during decision-making, thereby reducing latency and packet loss rate, and improving the throughput of the low-orbit satellite network. In addition, federated learning is used to improve the convergence speed of model training, thereby reducing the communication overhead of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0012] Figure 1 A flowchart of a low-orbit satellite communication network routing method based on federated deep reinforcement learning provided in one embodiment of the present application;

[0013] Figure 2 A schematic diagram of a low-orbit satellite cluster initialization process provided in an embodiment of the present application;

[0014] Figure 3 A schematic diagram of the federated learning principle provided for an embodiment of the present application;

[0015] Figure 4 A schematic diagram of the reinforcement learning principle provided in one embodiment of the present application. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0017] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0018] This application provides a low-orbit satellite communication network routing method based on federated deep reinforcement learning, such as Figure 1 As shown, a low-orbit satellite communication network routing method based on federated deep reinforcement learning includes:

[0019] Step 101: clustering the satellites in the target low-orbit satellite network into multiple clusters, and maintaining each cluster according to the satellite status of each satellite after clustering; the cluster head in each cluster serves as a low-orbit satellite server, and the cluster members serve as low-orbit satellite clients.

[0020] Step 102: A federated learning framework is built in each cluster. Based on the federated learning framework, the data of each low-orbit satellite client is used to train the routing decision model. When training the local model in the low-orbit satellite client, the routing decision process is constructed as a real-time Markov decision process for reinforcement learning. In the real-time Markov decision process, the state at the current moment is determined by the state and action at the previous moment. The local model corresponding to each low-orbit satellite client is used to output a routing decision for forwarding data.

[0021] In this application, the local model is used to output the routing decision for forwarding the output data of the corresponding low-orbit satellite. The global model trained based on the federated learning framework is the routing decision model.

[0022] The model parameters of the global model are initialized at the low-orbit satellite server of each cluster, and the model parameters of the global model are distributed to the low-orbit satellite clients in the cluster. The low-orbit satellite clients use the received model parameters of the global model as the initialization model parameters of the local model; the structures of the global model and the local model are the same.

[0023] The present application solves the problem of real-time network environment decision-making in a dynamic environment by adopting a real-time Markov Decision Process (RTMDP), and introduces a self-attention mechanism in the global model and the local model to help the intelligent agent understand the semantics of the state elements, so as to realize adaptive adjustment of routing strategies and real-time intelligent routing according to the changing environmental conditions; at the same time, a federated learning mechanism (FL) based on satellite clustering is introduced to divide the entire low-orbit satellite constellation system into three parts: a ground station, a low-orbit satellite server (cluster head) and a low-orbit satellite client (cluster member). Each low-orbit satellite client obtains the model sent by the low-orbit satellite server, and trains the model locally through its own computing power. Each low-orbit satellite client only uploads the weight of the local training model, aggregates the model on the low-orbit satellite server and sends it to each low-orbit satellite client. Combining it with DRL under the RTMDP framework can further improve the convergence speed of routing model training and reduce the communication overhead of the network.

[0024] Wherein, step 101 specifically includes:

[0025] Step 1011: construct a logical topology model of the target low-orbit satellite network using a virtual topology method; the logical topology model includes satellite nodes that correspond one-to-one to satellites in the target low-orbit satellite network.

[0026] Step 1012: clustering the satellite nodes in the logical topology model using a clustering algorithm to obtain multiple clusters, and determining the cluster head and cluster members of each cluster.

[0027] The logical topology model specifically includes the modeling of elements such as inter-satellite communication links, network topology structure, and satellite queue load. Through in-depth analysis of the system architecture of the low-orbit satellite constellation system, an accurate low-orbit giant constellation network routing model, namely the logical topology model, is constructed.

[0028] The target low-orbit satellite network of this application considers connecting ground users through a low-orbit satellite constellation, and users are connected to low-orbit satellite coverage and services. When the source user and the target user access satellite services from different low-orbit satellites, an inter-satellite link (ISL) relay is required. The inter-satellite link is divided into intra-orbit inter-satellite link and inter-orbit inter-satellite link. Each forwarding is defined as a hop, and the corresponding data packet is continuously forwarded through each satellite node to reach the target user, thereby establishing an end-to-end service. The source user is defined as the source node, and the target user is defined as the destination node.

[0029] The low-orbit satellite constellation is specifically a Walker-Delta constellation consisting of M×N satellites, where M represents the number of constellation orbital planes and N represents the number of satellites contained in each orbit. All orbits have the same inclination α and are equally spaced along the equator. The difference in right ascension of the ascending node between adjacent planes is ΔΩ=2π / M. The satellites are evenly distributed in each plane. The phase difference between adjacent satellites in the same plane is ΔΦ=2π / N. The phase offset between satellites in adjacent planes is Δf=2πF / MN, where F is the phase factor.

[0030] Each low-orbit satellite is numbered (i, j), where i represents the orbit number of the satellite (i=1, 2, 3…M), and j is the satellite number in the orbit of the satellite (j=1, 2, 3…N). According to the constellation configuration, except for the satellites on both sides of the reverse gap and in the polar regions, each low-orbit satellite establishes inter-satellite links with four adjacent low-orbit satellites.

[0031] The virtual topology method is used to construct the logical topology model of the target low-orbit satellite network. The partial topology of the first 100 satellites in the fourth constellation model of the first phase of Starlink is constructed by the virtual topology method. The satellite orbit altitude is 550km, the orbit inclination is 53°, the number of orbital planes is 10, the number of satellites in each orbital plane is 10, and the elevation angle is 25°.

[0032] This application adopts the virtual topology method to construct the logical topology model of the target low-orbit satellite network, specifically including: modeling the target low-orbit satellite communication network as a directed graph g t = {V t ,E t ,R t}.

[0033] Among them, the directed graph It is the logical topology model. Defined as a set of satellite nodes in time slot t, v i,t Satellite node i in time slot t, 1≤i≤MN.

[0034] definition is the set of links in time slot t, and a single link can be expressed as

[0035] definition is the maximum achievable rate of the intersatellite link in time slot t, R ij,t is the maximum achievable rate between satellite node i and satellite node j in time slot t.

[0036] In order to reduce the end-to-end delay and improve the average throughput of the network, the optimization goal is set to maximize the average system throughput: in Represents the path set corresponding to all end-to-end services in the network, l s It represents the data volume of a single end-to-end service transmission data packet, t max represents the time it takes for the last data packet in all services to reach the destination node. Therefore, the original problem can be equivalent to minimizing t max .

[0037] According to whether the low-orbit satellites at both ends of the intersatellite link are in the same orbital plane, the intersatellite link can be divided into intra-orbit intersatellite link and inter-orbit intersatellite link. In general, two continuous intra-orbit links can be established, and the inter-orbit link needs to be switched continuously according to the movement of the satellite. By default, all intersatellite links are initialized to the normal state, with the same transmission bandwidth and the same size of buffer queue space.

[0038] This application divides the time range of the dynamic network into T time slots of equal length, each time slot has a length of τ, and the total time is Tτ. Assume that there are S end-to-end services, and each service is represented as a tuple (s s ,d s ,l s ,δ s ,H s ), which represent the source node, destination node, data size, delay tolerance, and number of hops experienced of the s-th service respectively.

[0039] For data packet k, the processing delay is T i comp It refers to the time delay from receiving the data packet to satellite node i making a routing decision; queuing delay T i queue It refers to the queuing waiting time required for a certain satellite node i to wait for routing decision, and the transmission delay T i trans It refers to the time required for satellite node i to send a data frame, the propagation delay It refers to the time required for a signal to travel from one low-orbit satellite i to another low-orbit satellite j along the intersatellite link.

[0040] Each low-orbit satellite includes a receiving queue and four sending queues. The receiving queue is used to store received data packets. Each of the four sending queues transmits data with a low-orbit satellite through an inter-satellite link. The data packets in the receiving queue are transferred to one of the four sending queues according to the routing decision output by the trained local model.

[0041] The maximum length of the receiving queue is limited by the storage capacity C of the satellite node. iAssuming that the maximum length of the four directional transmission queues is m, the four directional transmission queues are associated with each of the four intersatellite links. The data packets in the receiving queue will be transferred to one of the four transmission queues according to the routing strategy, and then propagated to the next-hop satellite through the corresponding antenna. Each queue follows the first-in-first-out (FIFO) principle.

[0042] The present invention aims to reduce end-to-end delay and improve the average throughput of the network, and sets the optimization goal to maximize the average throughput of the system.

[0043] The clustering algorithm of the present invention uses the comprehensive weight of satellite nodes as an important basis for cluster head election and cluster structure maintenance. The weight calculation mainly considers factors such as connectivity, load service capability and priority. In order to reduce the additional overhead caused by clustering, the weight calculation and initialization stages are completed by the ground control terminal.

[0044] In an exemplary embodiment, clustering satellites in a target low-orbit satellite network to obtain a plurality of clusters specifically includes: clustering satellites in the target low-orbit satellite network to obtain a plurality of clusters through a ground station; and the ground station is communicatively connected with the target low-orbit satellite network.

[0045] The clustering algorithm is mainly divided into two stages: the cluster initialization stage and the maintenance stage. The main task of the initialization stage is to divide the states of all nodes at the initial moment and designate appropriate nodes (satellite nodes) as cluster heads. The cluster maintenance stage is responsible for maintaining the cluster structure when the cluster network topology changes to ensure that all nodes are in the cluster structure.

[0046] The initialization phase of the cluster is completed by the ground terminal, which calculates the weight of each node based on the cluster information and divides the node status.

[0047] After the initialization phase is completed, the target low-orbit satellite network immediately enters the cluster maintenance phase, and each node completes its own state change through certain information exchanges.

[0048] There are four main types of node state changes, namely cluster head node state changes, cluster member node state changes, free node state changes and undetermined node state changes.

[0049] The satellite nodes in the logical topology model are clustered by using a clustering algorithm to obtain multiple clusters, and the cluster head and cluster members of each cluster are determined, such as Figure 2 As shown, specifically including:

[0050] The weight of each satellite node is determined according to the connectivity of each satellite node in the logical topology model.

[0051] The calculation formula of the satellite node weight is:

[0052]

[0053] Among them, α i is the average connectivity of satellite node i, and α i is defined as the weight of satellite node i, and T is the number of divided time slices. i (t) represents the number of neighbor nodes of satellite node i in the tth time slice (time slot he).

[0054] in, a ij 1 means that satellite node i and satellite node j are connected in the tth time slice, a ij If it is 0, it means that satellite node i and satellite node j are not connected in the tth time slice.

[0055] It is determined whether the satellite node at the current moment has a neighboring node. If there is no neighboring node, the satellite node is determined to be in a free state.

[0056] If there are neighboring nodes, determine whether there is a cluster head node among the neighboring nodes.

[0057] If there is a cluster head node, the satellite node is added to the cluster with the highest priority among the clusters corresponding to the existing cluster head node, and the satellite node is determined as a member of the cluster.

[0058] The formula for priority measurement is:

[0059] Among them, w(i,CH v ) represents the priority metric of satellite node i joining cluster head v; a v represents the weight of cluster head v; n represents the number of nodes in the cluster, and h represents the number of hops from satellite node i to cluster head v. v Indicates the cluster where the cluster head v is located.

[0060] If there is no cluster head node, compare the weight of the satellite node with all satellite nodes in undetermined states within K hops. If the weight of the satellite node is the largest, determine the satellite node as the cluster head (the number of nodes in the cluster is initialized to 1). If the weight of the satellite node is not the largest, return to the step of determining whether there is a neighbor node for the satellite node at the current moment. If there is no neighbor node, determine that the satellite node is in a free state until the node states of all satellite nodes in the logical topology model are determined; the node state includes a free state, an undetermined state, a normal state and a cluster head, and the normal state indicates that the satellite node is a member of the cluster; K is a set value.

[0061] After clustering the satellite nodes in the logical topology model using a clustering algorithm to obtain multiple clusters and determining the cluster head and cluster members of each cluster, the low-orbit satellite communication network routing method based on federated deep reinforcement learning also includes: performing cluster maintenance when the logical topology model changes.

[0062] The cluster maintenance is performed when the logical topology model changes, specifically including the following steps:

[0063] 1) Cluster head state change: When there are no neighbor nodes in the cluster head, the node state of the satellite node corresponding to the cluster head is converted to a free state.

[0064] 2) Cluster member node status change: When a satellite node in a normal state is about to be disconnected from the cluster head in the cluster or the duration of the hop count greater than K exceeds the set time, a cluster leaving request is sent to the cluster head, and the satellite node is retrieved to see if there are neighbor nodes. If there are neighbor nodes, the state of the satellite node is converted to an undetermined state; if there are no neighbor nodes, the state of the satellite node is converted to a free state. When the distance between two satellites exceeds the maximum threshold, the information about the upcoming disconnection will be sent through the intersatellite link; therefore, the distance between the two satellites exceeding the maximum threshold is used as a condition for the upcoming disconnection.

[0065] 3) When a satellite node in a free state retrieves a connection relationship with other satellite nodes, the state of the satellite node is converted to an undetermined state.

[0066] 4) Undetermined node state change: The undetermined state is a temporary state. The node cannot stay in the undetermined state and needs to be immediately transformed into the member state or cluster head.

[0067] Initialize the global model parameters of the above agent on the low-orbit satellite server, and distribute the model to each LE0 client participating in the connection. Each LE0 client trains the model locally.

[0068] Federated learning is a decentralized machine learning approach that aims to train a global model without transmitting raw data from devices (LEO clients) to a central server (LEO servers).

[0069] Federated learning includes two architectures, homogeneous architecture and heterogeneous architecture. The homogeneous architecture means that the global model and the local model are structurally consistent, while the heterogeneous architecture is inconsistent. This application adopts the homogeneous architecture to randomly initialize the global model parameters (weights and biases, etc.) of the above-mentioned intelligent agent on the low-orbit satellite server.

[0070] The global model is distributed to each low-orbit satellite client participating in the connection through the inter-satellite links within the same cluster, and each low-orbit satellite client trains the model locally.

[0071] After updating the parameters, the low-orbit satellite client of this application transmits the compressed model to the low-orbit satellite server. The low-orbit satellite server aggregates the received model and updates the parameters in the global model. If the updated global model reaches the preset model accuracy in the test task, the training of the routing algorithm is determined to be completed. Otherwise, the above steps are repeated until the global model reaches the preset model accuracy in the test task.

[0072] In an exemplary embodiment, step 102 specifically includes: triggering a program to train a routing decision model using data from each low-orbit satellite client at a set time interval, so that the routing decision model can achieve the purpose of adaptive adjustment according to changing environmental conditions.

[0073] Each time the data of each low-orbit satellite client is used to train the routing decision model, it specifically includes:

[0074] In each training iteration, the local model in each cluster is traversed in turn.

[0075] Perform local model training on each low-orbit satellite client in the currently traversed cluster.

[0076] The model parameters of the currently trained local model are transmitted to the corresponding low-orbit satellite server.

[0077] The low-orbit satellite service end aggregates the model parameters of the received currently trained local model, and uses the aggregated model parameters to update the model parameters of the global model in each cluster.

[0078] The model accuracy of the global model with updated model parameters is tested. If the preset model accuracy is achieved, the training stops and the global model with the preset model accuracy is used as the routing decision model. Otherwise, the low-orbit satellite server distributes the model parameters of the current global model to each low-orbit satellite client, updates the model parameters of each local model, and returns to the step of traversing the local model in each cluster in turn.

[0079] The FedProx algorithm is used in each training of the routing decision model using the data of each LEO satellite client; FedProx is an improved federated learning algorithm used to solve the problems of non-independent and identically distributed data and device heterogeneity in federated learning. In each training iteration, a subset of clients (cluster) will be selected to perform local model updates and upload the local update results to the corresponding LEO satellite server for aggregation, ultimately achieving convergence of the global model.

[0080] FedProx introduces the proximal term based on the FedAvg loss function. The model parameters obtained by the LEO satellite client after local training will not deviate too much from the initial server model parameters. When the local model is trained locally on the LEO satellite client, the following objective function needs to be minimized:

[0081]

[0082] Among them, h k (w; w t ) is the objective function value of local training, w represents the model parameters of the local model, and w t represents the model parameters of the global model, F k (w) represents the minimized local function of the local model k, μ represents an adjustable parameter, and the entire LEO satellite constellation system consists of three parts: the ground station, the LEO satellite server (cluster head), and multiple LEO satellite client nodes (cluster members). For the federated learning mechanism, each LEO satellite client obtains the model sent by the LEO satellite server and trains the received model locally.

[0083] After multiple rounds of training, the LOR satellite clients within the LOR satellite cluster will transmit the compressed training model to the LOR satellite server, which will aggregate the received models and return the updated parameters to the relevant LOR satellite clients.

[0084] The training method of the local model and the global model in the low-orbit satellite client in the federated learning mechanism is specifically an improved multi-agent deep reinforcement learning routing algorithm based on RTMDP and self-attention designed in this application, which is mainly used to reduce end-to-end delay and improve low-orbit satellite network throughput. The training method specifically includes the following contents:

[0085] The routing performance optimization problem of low-orbit satellite networks is transformed into a real-time Markov decision process to better describe the system uncertainty and randomness, effectively handle complex decision-making problems, and match the real-time characteristics in real routing scenarios.

[0086] Normally, Markov decision processes are turn-based, where the agent selects an action while the environment and agent remain static until the agent receives the next state from the environment. Unlike the former, the action and state of RTMDP change simultaneously, and the new state-action pair is determined by the old state-action pair. Therefore, simply selecting an action is not enough to determine the next state-action pair.

[0087] Corresponding to the action-value function under the traditional MDP, the action-value function of the real-time Markov decision process is expressed as:

[0088]

[0089] Among them, s t is the state in time slot t, s t+1 is the state in time slot t+1, a t is the action in time slot t, a t =a t-1 , a t is the action in the previous time slot t-1, a t+1 is the action in time slot t+1, r(st,at) is the reward in time slot t, Indicates that in a given state s t and action a t , the agent moves to state s t+1 The expected action value under the corresponding probability distribution; for the action value function, It means that based on the current strategy π, given a state-action pair (s t+1 ,a t ) and take action a t+1 The expectation of the posterior local model estimate, Indicates that in the state-action pair (s t ,a t ) Take action a t The corresponding action-value function is, Indicates that in the state-action pair (s t+1 ,a t ) Take action a t+1 The corresponding estimated value.

[0090] Among them, special attention should be paid to the non-bold a t The bold a is the action in the previous time slot t-1. t is the action within time slot t.

[0091] According to the above expression, action a t has no direct effect on the reward or the next state, action a t The only impact t+1 , r(st +1 ,at +1 ) and s t+2 have an impact.

[0092] Similarly, corresponding to the state value function under the traditional MDP, the state value function of the real-time Markov decision process is expressed as:

[0093]

[0094] in, For t and a t The state-value function of the state-action pair; To satisfy a t+1 =a t Conditions, s t+1 and a t The state value of the state-action pair; It means that based on the current strategy π, given a state-action pair (s t ,a t ), the expectation of the state-value function of the local model under .

[0095] According to the above formula, no matter what action a is chosen, t ,s t 、a t 、s t+1 There is always a valid transition. Therefore, when using a real-time framework, it is possible to use the value function for discrete policy learning.

[0096] Assuming that the changes in the state of the low-orbit satellite network occur at the initial stage of each time slice ΔT, based on the characteristics of the real-time Markov decision process, the five elements {s, a, μ, p, r} that the real-time Markov decision process needs to contain are defined. Their meanings are as follows: s is the current state set, a is the action set, μ is the initial state distribution, p is the state transition probability, and r is the reward function. Each low-orbit satellite (specifically, the local model corresponding to each low-orbit satellite) is regarded as an intelligent agent. Currently, there is no parameter sharing between intelligent agents, and they are trained independently of each other. The two parameters μ and p belong to the five elements of the Markov decision process. They are randomly generated through automatic initialization of the model. They are a type of parameter and will change continuously according to the training process.

[0097] The structures of the global model and the local model both include a fully connected layer, an attention layer and a Softmax layer.

[0098] Due to the complex environment of low-orbit satellite communication networks, each agent has a huge state space and needs to process a large amount of observation information from the environment. In order to learn the semantics of complex elements, a self-attention mechanism is used. This self-attention mechanism assigns weights to observation elements in the neural network by improving the traditional soft attention mechanism to help the agent understand the semantics of state elements and actions from synthetic states. The basic principle of the self-attention mechanism can be expressed as:

[0099]

[0100] Among them, Q is the query matrix, K is the key matrix, V is the value matrix, and d k is the dimension of each column feature in the key matrix K, T is the transpose, and in the self-attention mechanism, the Q query matrix is ​​a transformation of its own input rather than input from the outside.

[0101] In the low-orbit satellite communication network routing scenario, the self-attention layer learns the relevant key matrix of the satellite network state and assigns dynamic weights to it. t Convert to a one-dimensional vector X. Then, the attention layer calculates the attention score for the input vector, and then performs Softmax calculation to normalize it to get the weighted sum. The weighted sums are concatenated together, and the result is used as the final output of the entire neural network.

[0102] Using deep reinforcement learning technology, through collaboration and learning between intelligent agents, feedback and rewards are continuously obtained from the low-orbit giant constellation network environment, and the inter-satellite routing strategy is optimized to improve the routing performance and throughput of the entire target low-orbit satellite network; each low-orbit satellite contains two deep neural networks: an estimated Q network and a target Q network.

[0103] At the beginning of local model training, the experience replay pool and network parameters are initialized, source nodes and destination nodes are randomly generated in the low-orbit satellite communication network topology, and routing decisions are made from the source nodes. In the decision-making process, the agent makes decisions through a dynamic greedy strategy.

[0104] Before updating the Q network, the agent selects a small batch of transition samples from the experience replay pool and uses these samples to calculate the error loss through the target Q network and the estimated Q network to update the model parameter weights of the local model.

[0105] In this application, each intelligent agent makes decisions based on the status information of adjacent nodes, which conforms to the satellite distributed architecture in the actual environment, enhances the adaptability to low-orbit satellite network routing, further improves the model convergence speed of the routing algorithm based on multi-agent reinforcement learning, and reduces network communication overhead.

[0106] To address the challenges brought by statistical distribution differences, the FedProx algorithm adds a correction term to the local training objective function, thereby effectively alleviating the adverse effects of local training update divergence on global convergence.

[0107] Specifically, this correction term can constrain the local training target of the low-orbit satellite client so that it can be close to the global training target. It reduces the negative impact of statistical distribution differences on global convergence without the need for manual intervention to adjust the local duration, and enables the low-orbit satellite server to safely aggregate a dynamic number of local model update results.

[0108] All LEO satellite clients participating in the training download the global model from the LEO satellite server as their respective local models, and use local data to train for multiple epochs. After the training is completed, the model parameters and gradients of the local model are encrypted and uploaded to the LEO satellite server.

[0109] The LEO satellite server aggregates the uploaded LEO satellite client model parameters and uses the FedProx algorithm to update the server global model parameters, and then returns the updated parameters to the relevant LEO satellite clients. The specific algorithm flow is as follows:

[0110] Initialize the input low-orbit satellite cluster set as The set of LEO satellite clients in the cluster is The set of model parameters within the cluster is

[0111] c1,c2,…c M They are the 1st cluster, the 2nd cluster,…, the Mth cluster respectively.

[0112] d1,d2,...,d N They are the first low-orbit satellite client, the second low-orbit satellite client, …, the Nth low-orbit satellite client.

[0113] are the model parameters of the first global model, the model parameters of the second global model,…, the model parameters of the Mth global model respectively.

[0114] Traverse each cluster and the client devices in each cluster.

[0115] Calculate model update weights

[0116] where h n (w; w t ) is the minimization objective function, F n (ω) is the minimized local function of the local model n

[0117] It is the difference between the parameters of the global model and the parameters of the client model. μ is an adjustable parameter. Reducing it can reduce the constraint effect.

[0118] Each LEO satellite client device sends an updated weight To the low-orbit satellite service terminal.

[0119] The low-orbit satellite server updates the global model, that is, the cluster head averages and aggregates the model parameters of the local models of the nodes in each cluster, which is the following formula: Where N is the total number of low-orbit satellite clients in the current cluster.

[0120] If the updated global model reaches the preset model accuracy in the test task, the training of the routing algorithm is determined to be completed. Otherwise, the above steps are repeated until the global model reaches the preset model accuracy in the test task.

[0121] Due to the instability of the intersatellite link connection, the LEO link may fail after a certain session. At this time, the LEO satellite server will hand over the routing task to the upcoming candidate LEO satellite server and ground station, and the new LEO satellite server will re-cluster the client to continue the task. The overall training process is as follows: Figure 3 As shown, the CH-CH link is the link between cluster heads, the CH-CM is the link between the low-orbit satellite server and the low-orbit satellite client within the cluster, and the CM-CM link is the link between the low-orbit satellite client and the low-orbit satellite client within the cluster.

[0122] In the above federated learning mechanism, each low-orbit satellite client is trained by an independent agent, which makes decisions based on the status information of adjacent nodes in its own cluster. The routing decision process of the entire network is constructed as a real-time Markov decision process. The agent obtains actions and makes routing decisions through the fully connected layer, attention layer and Softmax layer.

[0123] The dynamic routing decision problem of each satellite is modeled as a real-time Markov decision process, and each satellite is regarded as an intelligent agent. The routing decision process of the entire network is constructed as a real-time Markov decision process. The specific process is as follows:

[0124] Based on the characteristics of the real-time Markov decision process, the five elements {s, a, μ, p, r} that need to be included in the definition of the real-time Markov decision process are: s is the current state set, a is the action set, μ is the initial state distribution, p is the state transition probability, and r is the reward function.

[0125] The elements in the real-time Markov decision process include state (environment state), action and reward function.

[0126] The status is represented by:

[0127] in, is the state set of the current satellite i (satellite node i) preparing to route and forward the data packet p in time slot t, φ i,p The source node of the data packet p that the current satellite i prepares to route and forward in time slot t, is the destination node of the data packet p that the current satellite i prepares to route and forward in time slot t. Both the source node and the destination node are satellite nodes. is the queue blocking rate of the satellite receiving queue at the current time slot, is the link connection status between satellite i in the current time slot and its adjacent low-orbit satellite, is the action performed in the previous time slot of the current satellite i. Since the RTMDP mechanism is added in this application, the previous time slot t of the current satellite node i is pre Actions performed Also added to the status.

[0128] Action a is defined as selecting the next-hop satellite node. According to the current state, the agent can select its own neighbor node for the next-hop routing. The action is expressed as:

[0129] in, is the action set of the current satellite i to prepare the routing forwarding data packet p in time slot t. When , it means that the agent chooses v j As the next hop low-orbit satellite for forwarding data packet p, represents the set of neighboring satellites of the current satellite in time slot t, and each link is a bidirectional data packet transmission, v j is low-orbit satellite j (satellite node j), and the agent is the local model corresponding to the current satellite i;

[0130] The reward function is expressed as:

[0131] The goal of this reward function is to ensure that the data packets are transmitted to their destination nodes while increasing the number of data packets arriving at the destination node, minimizing the average delay, and appropriately distributing the satellite load.

[0132] Among them, r t i,p is the reward function for the current satellite i after routing and forwarding data packet p to the destination node in time slot t, ΔD p is the difference in propagation delay, specifically defined as the last time slot t pre When , the difference between the propagation delay from satellite i where data packet p is located to destination node d and the propagation delay from satellite j where data packet p is located to destination node d is, D is the propagation time from satellite i where data packet p is located to destination node d, p (v j,t ,v d ) is the propagation time from satellite j to destination node d for data packet p, v d For the destination node d, v j,t At time slot t, at satellite j, In time slot t pre Satellite i, time slot t pre is the previous time slot of time slot t, k1 and k2 are penalty coefficients, O t is the total amount of remaining data packets in the target low-orbit satellite network in time slot t, represents the receiving queue load reward of satellite j in time slot t, Indicates the queue load in time slot t, the queue load in the current time slot The smaller the The bigger.

[0133] like Figure 4 As shown, during the initial training of the agent, RTMDP will be based on the initial state s t Use the greedy strategy to randomly generate an action a t , use a t Make the routing decision for this round. After routing, the environment will return the reward r for the current round. t , and the action at the previous moment, that is, a t With the current state s t+1 Splice as the new state at this moment (s t+1 ,a t ), the agent continues to select actions through this state, and obtains a new action a through the fully connected layer, attention layer and Softmax layer. t And make a new round of routing decisions. The specific algorithm process is as follows:

[0134] Initialize the algorithm's training round number, simulation time, number of satellite nodes, node queue length and other parameters. The node queue length includes the sending queue length and the receiving queue length.

[0135] After each round of training, the satellite queue is cleared and data packets are generated and sent to the target low-orbit satellite network.

[0136] At a specific simulation time, each satellite node is traversed to perform agent training.

[0137] Traversing each data packet in the satellite queue, the agent initializes past actions based on the greedy strategy at the initial stage of training implement And get the state s t .

[0138] Splicing t and For vector to select an action.

[0139] The attention layer passes Get the corresponding weight w; vector After passing through the fully connected layer, it is multiplied by the weight w.

[0140] Output action a t , the next hop node and the current receive queue capacity.

[0141] If the current receiving queue capacity is less than the maximum queue capacity, execute action a t , and receive a reward t , the next state s t+1, and the end binary variable done, otherwise the data packet is placed at the end of the next node's sending queue. When the data packet arrives at the destination node, it will automatically change its own binary variable attributes. The end binary variable is used to indicate whether the data packet has arrived at the corresponding destination node.

[0142] Update past actions

[0143] If the end binary variable done is not equal to 0, the number of data packets arriving at the destination node is increased by 1.

[0144] After the current data packet routing is completed, the agent will store the transition sample (s t ,a t ,r t ,s t+1 , done) to experience pool D.

[0145] Sampling transition samples (s) from the experience pool D t ,a t ,r t ,s t+1 , done), and by minimizing the loss function The local model weights are updated via the alternating direction method of multipliers (ADMM) algorithm.

[0146] Where N b is the random sampling capacity, s t is the current state, a t is the current action. θ is the parameter of the estimated Q network, and Q() represents the estimated Q network. t The next state s of the target Q network is calculated by t+1 When using action a t+1 The target Q value is obtained.

[0147] y t =r t +γQ(s t+1 ,argmax a Q(s t+1 ,a t+1 |θ);θ').

[0148] Among them, r t Indicates that the agent is in state s t Take action t The parameter of the target Q network is θ', and γ∈[0,1] is represented as the decay factor.

[0149] Through the above process, training is continued until the algorithm converges, that is, the model accuracy of the global model reaches the preset model accuracy, and finally a stable routing decision algorithm (global model) can be obtained.

[0150] This application introduces RTMDP into the field of low-orbit satellite network routing. The RTMDP mechanism is a continuous and uninterrupted action selection mechanism. The state and action of the agent can evolve simultaneously to match the real-time characteristics in the actual routing scenario. It solves the problem that the environment must be static when making routing decisions in a dynamic environment, resulting in a distance offset from the actual situation, and enhances the effectiveness of the routing control method. In order to better learn routing strategies, a self-attention mechanism is added to help the agent understand the semantics of state elements. By cutting the input synthetic network state into independent elements and learning to assign different weights to them during training, it has better performance and faster convergence speed than traditional fully connected neural networks.

[0151] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A low-orbit satellite communication network routing method based on federated deep reinforcement learning, It is characterized in that The low-orbit satellite communication network routing method based on federated deep reinforcement learning includes: The satellites in the target low-orbit satellite network are clustered to obtain multiple clusters, and after clustering, each cluster is maintained according to the satellite status of each satellite; the cluster head in each cluster serves as the low-orbit satellite server, and the cluster members serve as the low-orbit satellite clients; A federated learning framework is built in each cluster. Based on the federated learning framework, the data of each low-orbit satellite client is used to train the routing decision model. When training the local model in the low-orbit satellite client, the routing decision process is constructed as a real-time Markov decision process for reinforcement learning. In the real-time Markov decision process, the state at the current moment is determined by the state and action at the previous moment. The local model corresponding to each low-orbit satellite client is used to output the routing decision for forwarding the data.

2. According to claim 1, the low-orbit satellite communication network routing method based on federated deep reinforcement learning, It is characterized in that The satellites in the target low-orbit satellite network are clustered to obtain multiple clusters, including: A virtual topology method is used to construct a logical topology model of a target low-orbit satellite network; the logical topology model includes satellite nodes corresponding one to one to satellites in the target low-orbit satellite network; A clustering algorithm is used to cluster the satellite nodes in the logical topology model to obtain multiple clusters, and a cluster head and cluster members of each cluster are determined.

3. The low-orbit satellite communication network routing method based on federated deep reinforcement learning according to claim 2, It is characterized in that A clustering algorithm is used to cluster the satellite nodes in the logical topology model to obtain multiple clusters, and a cluster head and cluster members of each cluster are determined, specifically including: Determining the weight of each satellite node according to the connectivity of each satellite node in the logical topology model; Determine whether the satellite node at the current moment has a neighbor node. If no neighbor node exists, the satellite node is determined to be in a free state. If there are neighboring nodes, determine whether there is a cluster head node among the neighboring nodes; If there is a cluster head node, the satellite node is added to the cluster with the highest priority in the cluster corresponding to the existing cluster head node, and the satellite node is determined as a member of the cluster; If there is no cluster head node, compare the weight of the satellite node with all satellite nodes in undetermined states within K hops. If the weight of the satellite node is the largest, determine the satellite node as the cluster head. If the weight of the satellite node is not the largest, return to the step of determining whether there is a neighbor node at the satellite node at the current moment. If there is no neighbor node, determine that the satellite node is in a free state, until the node states of all satellite nodes in the logical topology model are determined; the node state includes a free state, an undetermined state, a normal state and a cluster head, and the normal state indicates that the satellite node is a member of the cluster; K is a set value.

4. The low-orbit satellite communication network routing method based on federated deep reinforcement learning according to claim 3, It is characterized in that After clustering the satellite nodes in the logical topology model by using a clustering algorithm to obtain a plurality of clusters, and determining the cluster head and cluster members of each cluster, the low-orbit satellite communication network routing method based on federated deep reinforcement learning further includes: performing cluster maintenance when the logical topology model changes; The cluster maintenance is performed when the logical topology model changes, specifically including: When there is no neighbor node in the cluster head, the node state of the satellite node corresponding to the cluster head is converted to the free state; When the duration of the hop count between the satellite node in the normal state and the cluster head in the cluster is greater than K and exceeds the set time, a cluster leaving request is sent to the cluster head, and the satellite node is retrieved to see whether there is a neighbor node. If there is a neighbor node, the state of the satellite node is converted to an undetermined state; if there is no neighbor node, the state of the satellite node is converted to a free state. When a satellite node in a free state retrieves a connection relationship with other satellite nodes, the state of the satellite node is converted to an undetermined state.

5. The low-orbit satellite communication network routing method based on federated deep reinforcement learning according to claim 1, It is characterized in that The data of each low-orbit satellite client is used to train the routing decision model, specifically including: triggering a program for training the routing decision model with the data of each low-orbit satellite client at a set time interval; Each time the data of each low-orbit satellite client is used to train the routing decision model, it specifically includes: In each training iteration, the local model in each cluster is traversed in turn; Perform local model training on each low-orbit satellite client in the currently traversed cluster; Transmit the model parameters of the currently trained local model to the corresponding low-orbit satellite server; The low-orbit satellite service end aggregates the model parameters of the currently trained local model received, and updates the model parameters of the global model in each cluster with the aggregated model parameters; The model accuracy of the global model with updated model parameters is tested. If the preset model accuracy is achieved, the training stops and the global model with the preset model accuracy is used as the routing decision model. Otherwise, the low-orbit satellite server distributes the model parameters of the current global model to each low-orbit satellite client, updates the model parameters of each local model, and returns to the step of traversing the local model in each cluster in turn.

6. The low-orbit satellite communication network routing method based on federated deep reinforcement learning according to claim 1, It is characterized in that Each low-orbit satellite includes a receiving queue and four sending queues. The receiving queue is used to store received data packets. Each of the four sending queues transmits data with a low-orbit satellite through an inter-satellite link. The data packets in the receiving queue are transferred to one of the four sending queues according to the routing decision output by the trained local model.

7. The low-orbit satellite communication network routing method based on federated deep reinforcement learning according to claim 6, It is characterized in that The elements in the real-time Markov decision process include state, action and reward function; The status is represented by: in, is the state set of the current satellite i preparing to route and forward the data packet p in time slot t, φ i,p The source node of the data packet p that the current satellite i prepares to route and forward in time slot t, is the destination node of the data packet p that the current satellite i prepares to route and forward in time slot t. Both the source node and the destination node are satellite nodes. is the queue blocking rate of the receiving queue of the satellite in the current time slot, is the link connection status between satellite i in the current time slot and its adjacent low-orbit satellite, The action performed in the previous time slot of the current satellite i; The action is represented by: in, is the action set of the current satellite i to prepare the routing forwarding data packet p in time slot t. When , it means that the agent chooses v j As the next hop low-orbit satellite for forwarding data packet p, represents the set of neighboring satellites of the current satellite in time slot t, v j is satellite j, and the agent is the local model corresponding to the current satellite i; The reward function is expressed as: in, is the reward function for the current satellite i after routing and forwarding data packet p to the destination node in time slot t, ΔD p is the difference between the propagation delay from satellite i where data packet p is located to destination node d and the propagation delay from satellite j where data packet p is located to destination node d, D is the propagation time from satellite i where data packet p is located to destination node d, p (v j,t ,v d ) is the propagation time from satellite j to destination node d for data packet p, v d For the destination node d, v j,t At time slot t, at satellite j, In time slot t pre Satellite i, time slot t pre is the previous time slot of time slot t, k 1 and k 2 are penalty coefficients, O t is the total amount of remaining data packets in the target LEO satellite network in time slot t, represents the receiving queue load reward of satellite j in time slot t, Represents the queue load in time slot t.

8. The low-orbit satellite communication network routing method based on federated deep reinforcement learning according to claim 7, It is characterized in that The action-value function of the real-time Markov decision process is expressed as: Among them, s t is the state in time slot t, s t+1 is the state in time slot t+1, a t is the action in time slot t, a t =a t-1 , a t is the action in the previous time slot t-1, a t+1 is the action in time slot t+1, r(st,at) is the reward in time slot t, In a given state s t and action a t , the agent moves to state s t+1 The expected action value under the corresponding probability distribution; for the action value function, It means that based on the current strategy π, given a state-action pair (s t+1 ,a t ) and take action a t+1 The expectation of the posterior local model estimate, Indicates that in the state-action pair (s t ,a t ) Take action a t The corresponding action-value function is, Indicates that in the state-action pair (s t+1 ,a t ) Take action a t+1 The corresponding estimated value; among them, action a t Does not affect the reward and the next state, action a t Only affects the next action a t+1 ; In the next time slot, action a t Will affect r(s t+1 ,a t+1 ); The state value function of the real-time Markov decision process is expressed as: in, For t and a t The state-value function of the state-action pair; To satisfy a t+1 =a t Conditions, s t+1 and a t The state value of the state-action pair; It means that based on the current strategy π, given a state-action pair (s t ,a t ), the expectation of the state-value function of the local model under .

9. The low-orbit satellite communication network routing method based on federated deep reinforcement learning according to claim 1, It is characterized in that The satellites in the target low-orbit satellite network are clustered to obtain multiple clusters, including: The satellites in the target low-orbit satellite network are clustered to obtain a plurality of clusters through a ground station; the ground station is communicatively connected with the target low-orbit satellite network.

10. The low-orbit satellite communication network routing method based on federated deep reinforcement learning according to claim 5, It is characterized in that The structures of the global model and the local model both include a fully connected layer, an attention layer and a Softmax layer.

Citation Information

Patent Citations

  • Markov decision-based power distribution system elastic strategy construction method

    CN111860611A

  • Low earth orbit satellite network routing method based on reinforcement learning

    CN117792984A

  • Cloud Based Machine Learning Notebook Feature Engineering Command Recommender

    US20230185626A1

Cited By

  • Giant satellite base layered multi-domain routing method and device based on long-chain and short-chain collaboration

    CN120729387A

  • Federal learning-based routing policy network training method and device

    CN121509301A

  • Training Method and Apparatus for Routing Policy Networks Based on Federated Learning

    CN121509301B

  • Task division and unloading method for satellite-ground integrated network k synchronous federal learning

    CN121510147A

  • Service quality sensing routing method and system for large-scale low earth orbit satellites

    CN121567196A