Federal learning-based routing policy network training method and device

Abstract

The invention provides a method and device for training a routing policy network based on federated learning, and the method comprises the steps: receiving a regional feature vector transmitted by a managed first low earth orbit satellite group, the regional feature vectors are generated by low earth orbit satellites in the first low earth orbit satellite group based on graph structure data of a first regional network formed by a locally deployed graph neural network and the first low earth orbit satellite group; training a local routing policy network by adopting a multi-agent reinforcement learning algorithm according to the regional feature vector; network parameters of the trained routing strategy network are sent to a preset aggregation satellite, so that the aggregation satellite fuses the network parameters uploaded by the management satellites of the multiple areas to obtain a global routing strategy model, and global model parameters of the global routing strategy model are sent to the management satellites of the multiple areas; and updating network parameters of a local routing policy network according to the global model parameters sent by the aggregation satellite.

Description

Technical Field

[0001] This invention relates to the field of satellite internet technology, and in particular to a training method and apparatus for a routing policy network based on federated learning. Background Technology

[0002] In large-scale constellation networks composed of thousands of low-Earth orbit satellites, widely distributed and in constant high-speed relative motion, the network topology exhibits highly dynamic characteristics due to changes in satellite positions and fluctuations in link status. Such networks not only have wide coverage and a large number of nodes, but also feature frequent establishment and breakage of inter-satellite links and short communication path lifecycles. Simultaneously, onboard computing, storage, and energy resources are severely limited, and inter-satellite link bandwidth is extremely limited. Given this background, dynamic routing optimization in large-scale constellation networks faces several technical challenges. For example, how to reduce the computing and storage burden on individual satellites to adapt to the reality of limited onboard resources. Another example is how to improve the routing performance and network efficiency of the entire large-scale constellation network. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a training method for a routing policy network based on federated learning, in order to eliminate or improve one or more defects existing in the prior art.

[0004] According to the first aspect, a method for training a routing policy network based on federated learning is provided, wherein the routing policy network is used to route data in a satellite network, the satellite network comprising multiple low Earth orbit satellite groups divided by region, each low Earth orbit satellite group being managed by its respective management satellite, the management satellite being a medium Earth orbit satellite, and the method being executed by a first management satellite; and the method comprising: The system receives regional feature vectors transmitted by the first low Earth orbit satellite group under its management. These regional feature vectors are generated from the graph structure data of the first regional network composed of the low Earth orbit satellites in the first low Earth orbit satellite group and their locally deployed graph neural networks. Based on these regional feature vectors, a local routing policy network is trained using a multi-agent reinforcement learning algorithm. This routing policy network is used to implement routing decisions for the first regional network. The network parameters of the trained routing policy network are sent to a preset aggregation satellite, which then fuses the network parameters uploaded by the management satellites of multiple regions to obtain a global routing policy model. The global model parameters of this global routing policy model are then sent to the management satellites of multiple regions, where the aggregation satellites are medium Earth orbit satellites. The network parameters of the local routing policy network are updated based on the global model parameters sent by the aggregation satellites.

[0005] According to a second aspect, a routing method for a satellite network is provided, wherein the satellite network includes multiple low Earth orbit satellite groups divided by region, each low Earth orbit satellite group is managed by its own management satellite, the management satellite being a medium Earth orbit satellite, and each management satellite is pre-deployed with a routing policy network trained according to any one of the methods in the first aspect, the method being executed by a first management satellite; and the method includes: acquiring a routing task, wherein the routing task includes a source node and a destination node; receiving a region feature vector sent by the managed first low Earth orbit satellite group, wherein the region feature vector is generated by the low Earth orbit satellites in the first low Earth orbit satellite group based on the graph structure data of the graph structure data of the first region network composed of the low Earth orbit satellite group and its locally deployed graph neural network; and, if the source node and the destination node are nodes in the first region network, performing a routing decision using the locally deployed routing policy network according to the region feature vector, calculating the route from the source node to the destination node, and generating a scheduling instruction to control the data forwarding from the source node to the destination node.

[0006] According to a third aspect, a training apparatus for a routing policy network based on federated learning is provided, wherein the routing policy network is used to route data in a satellite network, the satellite network comprising multiple low Earth orbit satellite groups divided by region, each low Earth orbit satellite group being managed by its respective management satellite, the management satellite being a medium Earth orbit satellite, and the apparatus being deployed on a first management satellite; and the apparatus comprising: a receiving unit configured to receive a region feature vector transmitted by the managed first low Earth orbit satellite group, wherein the region feature vector is a graph neural network of the low Earth orbit satellites in the first low Earth orbit satellite group based on their locally deployed graph neural networks and a first region network composed of the first low Earth orbit satellite group. The system includes: a training unit configured to train a local routing policy network using a multi-agent reinforcement learning algorithm based on the aforementioned regional feature vectors, wherein the routing policy network is used to implement routing decisions for the aforementioned first regional network; a sending unit configured to send the network parameters of the trained routing policy network to a preset aggregation satellite, so that the aggregation satellite can fuse the network parameters uploaded by the management satellites of multiple regions to obtain a global routing policy model, and send the global model parameters of the global routing policy model to the management satellites of multiple regions, wherein the aggregation satellite is a medium Earth orbit satellite; and an updating unit configured to update the network parameters of the local routing policy network based on the global model parameters sent by the aggregation satellite.

[0007] According to a fourth aspect, a routing apparatus for a satellite network is provided, wherein the satellite network includes multiple low Earth orbit satellite groups divided by region, each low Earth orbit satellite group is managed by its own management satellite, the management satellite being a medium Earth orbit satellite, and each management satellite is pre-deployed with a routing policy network trained according to any one of the methods in the first aspect, the apparatus being deployed on a first management satellite; and the apparatus includes: a task acquisition unit configured to acquire a routing task, wherein the routing task includes a source node and a destination node; a vector receiving unit configured to receive a regional feature vector sent by the managed first low Earth orbit satellite group, wherein the regional feature vector is generated by the low Earth orbit satellites in the first low Earth orbit satellite group based on the graph structure data of the graph structure data of the first regional network composed of the first low Earth orbit satellite group and its locally deployed graph neural network; and a routing decision unit configured to, when the source node and the destination node are nodes in the first regional network, perform a routing decision using the locally deployed routing policy network according to the regional feature vector, calculate the route from the source node to the destination node, and generate a scheduling instruction to control the data forwarding from the source node to the destination node.

[0008] According to a fifth aspect, a medium Earth orbit satellite is provided, comprising a processor, a memory, and a computer program / instructions stored in the memory, characterized in that the processor is configured to execute the computer program / instructions, and when the computer program / instructions are executed, the medium Earth orbit satellite performs the steps of the method as described in any one of the first or second aspects.

[0009] According to a sixth aspect, a computer-readable storage medium is provided having a computer program / instructions stored thereon, characterized in that, when executed by a processor, the computer program / instructions implement the steps of the method as described in any one of the first and second aspects.

[0010] The present invention discloses a training method and apparatus for a routing policy network based on federated learning. First, the satellite network can be divided into multiple low-Earth orbit (LEO) satellite groups according to regions. Each LEO satellite group is managed by a management satellite, which is a medium-Earth orbit (MEO) satellite. Next, any one of the multiple management satellites is designated as the first management satellite. The first management satellite can receive regional feature vectors sent by the managed first LEO satellite group. These regional feature vectors can be generated by the LEO satellites within the first LEO satellite group based on the graph structure data of the first regional network composed of the LEO satellite group and its locally deployed graph neural network. Then, the first management satellite can train its local routing policy network using a multi-agent reinforcement learning algorithm based on the regional feature vectors. Next, the first management satellite can send the network parameters of the trained local routing policy network to a preset aggregation satellite. The aggregation satellite can then fuse the network parameters uploaded by the management satellites of multiple regions to obtain a global routing policy model, and send the global model parameters of the global routing policy model to the management satellites of multiple regions. Finally, the first management satellite can update the network parameters of its local routing policy network based on the global model parameters sent by the aggregation satellite. Therefore, by dividing the LEO (Low Earth Orbit) constellation into several regions, each autonomously managed by a MEO (Medium Earth Orbit) satellite, the computational and storage pressure on individual nodes can be significantly reduced, adapting to the reality of limited onboard resources. Furthermore, the federated learning framework enables secure aggregation and globally unified updating of network parameters for routing policies across regions, effectively avoiding local optima or policy conflicts that might result from independent training in each region. Simultaneously, the federated learning mechanism balances the needs of regional autonomy and global collaboration, achieving global consistency and collaborative optimization in dynamic routing decisions, thereby improving the overall routing performance and network efficiency of the entire constellation network.

[0011] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0012] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0013] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0014] Figure 1 A schematic diagram illustrating an example of regional partitioning of a large-scale constellation network is shown. Figure 2 A flowchart illustrating a training method for a federated learning-based routing policy network according to one embodiment is shown; Figure 3 A schematic diagram illustrating an example of regional coordination in network training and federated learning is shown. Figure 4 A flowchart of a routing method for a satellite network according to one embodiment is shown; Figure 5 A schematic block diagram of a training apparatus for a federated learning-based routing policy network according to one embodiment is shown. Figure 6 A schematic block diagram of a routing apparatus for a satellite network according to one embodiment is shown. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0016] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0017] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0018] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0019] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0020] As mentioned earlier, routing optimization in large-scale constellation networks faces several technical challenges. For example, it presents the challenge of highly dynamic and rapidly changing networks. The high-speed relative motion of LEO (Low Earth Orbit) constellations causes significant changes in network topology on millisecond to second timescales, leading to frequent establishment and breakage of inter-satellite links and extremely short effective communication paths. This high dynamic nature necessitates continuous updates to routing strategies; otherwise, path failures, data loss, and increased latency can occur, severely impacting service stability and network transmission efficiency. Another challenge is the limited onboard computing power. Low Earth Orbit satellites face strict limitations in computing power, storage capacity, and energy budgets, and inter-satellite link bandwidth is also very limited. Under these conditions, performing highly complex network-wide routing calculations or frequent large-scale state synchronizations can place an excessive computational and communication burden on the onboard system, potentially affecting the normal operation of other critical missions.

[0021] There are also some routing methods for constellation networks in the existing technology.

[0022] For example, some centralized satellite routing algorithms exist in existing technologies. Centralized inter-satellite routing refers to a system where a central control node is responsible for collecting the topology, link status, and task requirements of the entire constellation network, centrally calculating the optimal path, and distributing the calculation results to each satellite node in the form of control commands. Centralized routing has a complete global perspective and can achieve overall optimal path planning and resource scheduling. Existing centralized satellite routing is mostly based on SDN (Software Defined Network) architecture or implemented with a centralized controller. It performs batch path calculations by periodically synchronizing the link status of the entire network and using algorithms such as Dijkstra's algorithm, integer linear programming, or spatiotemporal extended graphs, while maintaining route stability during the topology update cycle. Currently, various research institutions have conducted extensive research on centralized satellite routing algorithms and have produced numerous research results. For example, an SDN-based satellite routing proposed by a university, under a software defined network architecture, combines depth-first search and an improved Dijkstra's algorithm to calculate the static routes of the entire network in time slices, achieving dual optimization of end-to-end latency and packet loss rate, while significantly improving system scalability. Based on this approach, the IADR algorithm incorporates multi-dimensional attributes such as link signal-to-noise ratio, duration, and load into the utility function, simultaneously optimizing throughput and reliability at the controller side through multi-attribute decision-making. For example, the IUDR (Incremental Update-based Dynamic Routing) proposed by a university constructs a spatiotemporal evolution graph, mapping the routing problem to a maximum deviation optimization problem of multi-attribute decision-making, and solves it at the control center using an adaptive maximum deviation algorithm to balance the three key performance indicators of latency, packet loss, and throughput. Furthermore, the deterministic routing based on time-varying graphs proposed by another university introduces a time-spreading graph model, using integer linear programming to ensure the path's continuity within the link's available period, significantly improving resource utilization and path stability. For instance, a college also proposed the DQN-IR algorithm based on Deep Q-Network (DQN). The model input includes features such as the relative positions of neighboring nodes, link distance, and queuing delay. Through training, an optimal policy network is obtained, and each decision only requires forward propagation to select a link, achieving dual optimization of end-to-end latency and communication overhead. For example, a university proposed a multi-target routing model that uses a deep learning model, relying on local feature inputs to jointly optimize end-to-end latency and throughput, while also considering path stability. This is suitable for dynamic scenarios in integrated space-terrestrial networks. Another university proposed Fuzzy-CNN (Fuzzy Convolutional Neural Network) multi-task routing, combining fuzzy logic and convolutional neural networks to achieve multi-task routing optimization within an SDN architecture, improving both flow control and pathfinding flexibility. In large-scale satellite networks, the main problem with centralized routing is primarily the risk of single-point failure.Centralized solutions rely on a small number of master nodes to perform network-wide route calculations and distribution. If these nodes fail, the routing function of the entire network will be severely affected, potentially leading to a complete shutdown. Secondly, centralized routing introduces significant latency overhead in large-scale constellations. When routing decisions require backhauling across multiple satellite or ground links to the center for further distribution, the additional propagation and processing delays are extremely detrimental to low-latency services. Simultaneously, topology state updates and routing requests are concentrated at the master node. As the number of satellites increases, signaling traffic and computational pressure increase exponentially, easily making the central node a bottleneck and causing a decrease in route convergence speed. In rapidly changing, large-scale space network environments, the response speed of centralized systems struggles to keep up with the frequency of change, potentially leading to routing information delays and service interruptions.

[0023] For example, some existing technologies employ distributed satellite routing algorithms. Distributed routing involves each satellite making routing decisions based on local information and neighbor states, without relying on a centralized control node. This type of routing emphasizes network autonomy and can quickly respond to topology changes. Especially in large-scale LEO constellations, as satellites frequently enter and leave the line of sight, distributed protocols can achieve rapid path adjustments and link switching, enhancing system stability. Furthermore, this approach inherently possesses decentralized characteristics, improving network fault tolerance and resilience. Existing distributed routing typically relies on link-state or distance-vector protocols, combined with mechanisms such as area partitioning, load monitoring, or reinforcement learning, to complete path selection and dynamic updates based on local information exchange. Currently, various research institutions have conducted extensive research on distributed satellite routing algorithms. For instance, the ASER algorithm proposed by a university uses a domain-based routing strategy, enabling each satellite to maintain and update its routing table only within its own or adjacent domains. When link states change, only local updates are needed, significantly reducing the overhead of routing computation and table synchronization. For example, a university proposed a dynamic cache allocation strategy that dynamically allocates link cache resources based on traffic prediction and combines real-time comparison of queuing and forwarding latency to determine whether to trigger rerouting, which can balance link utilization and reduce end-to-end latency under high load. Another example is the SALB algorithm proposed by another university, which dynamically monitors link and node load, adjusts path cost weights, and uses the shortest path tree algorithm to generate routes in real time, thereby improving load balancing and fault tolerance. Yet another example is an adaptive routing based on improved dual-Q learning, where nodes maintain two sets of Q tables to evaluate different strategies, design a reward function based on congestion and node state, and autonomously adjust forwarding decisions to adapt to the highly dynamic environment of satellite IoT. A third example is a multi-agent reinforcement learning routing framework proposed by another university, which treats each satellite as an agent, independently trains its model, and exchanges trust values ​​and latency information during communication to achieve distributed collaborative optimization and meet differentiated QoS (Quality of Service) requirements. Finally, a third example is a reinforcement learning dynamic routing algorithm based on Q-learning, which dynamically updates strategies when the topology changes, and optimizes network load balancing and computational resource consumption through long-term rewards. For example, a research institute proposed a satellite network routing optimization method based on deep reinforcement learning and federated learning. Low-Earth orbit (LEO) satellites employ a distributed actor-evaluation network training strategy, while medium-Earth orbit (MEO) satellites periodically aggregate LEO evaluation parameters and distribute the global model. LEO satellites then update their networks accordingly, achieving dynamic optimization and consistency of routing across the entire network. Another example is a university's distributed satellite routing method based on reinforcement learning. This method constructs a static topology model using time slices and designs an objective function that minimizes end-to-end latency. Simultaneously, it builds collaborative and decision networks, training and deploying the decision network through interactive data to achieve collaborative routing decisions, reducing latency and balancing load. Distributed routing also faces significant challenges in large-scale satellite networks.For example, because each satellite relies solely on local and neighbor information for routing decisions, lacking a global view, path selection is often suboptimal, and routing loops may even occur. Furthermore, in highly dynamic topologies, the convergence time of distributed protocols is significantly prolonged, and frequent route fluctuations during convergence cause latency jitter and data loss. Simultaneously, distributed schemes require each satellite to independently run route calculations and store routing tables, placing an additional burden on satellite nodes with limited processing power and energy. More significantly, distributed routing requires continuous exchange of route update information, resulting in signaling overhead that grows linearly or even superlinearly in large-scale networks, consuming additional inter-satellite link bandwidth, and making it difficult to guarantee routing information consistency under highly dynamic conditions, thus affecting service stability and reliability.

[0024] Therefore, the embodiments of this specification provide a training method for a routing policy network based on federated learning, which can reduce the computational and storage pressure on a single node, adapt to the reality of limited onboard resources, and improve the overall routing performance and network efficiency of the entire constellation network.

[0025] Before implementing the federated learning-based routing policy network training method provided in this embodiment, the large-scale constellation network can be partitioned to construct a two-layer architecture of MEO (Medium Earth Orbit) and LEO (Low Earth Orbit) satellites to achieve efficient dynamic routing optimization. For example, the entire LEO constellation can be divided into several regions based on the geographical locations covered by the satellites. Each LEO satellite node within a region is managed by a MEO satellite, which is responsible for training the routing policy network and cross-domain coordination. By decomposing the large-scale constellation into autonomous sub-regions, an efficient hierarchical architecture can be constructed. It can be understood that in this example, Low Earth Orbit (LEO) generally refers to satellites operating at an altitude of 500–2000 km above the Earth's surface. Medium Earth Orbit (MEO) generally refers to satellites at an altitude of 2000–20000 km above the Earth's surface.

[0026] like Figure 1 As shown, Figure 1 A schematic diagram illustrating an example of regional partitioning of a large-scale constellation network is shown. Figure 1 In the example shown, the region can be divided using the following steps 1) and 2), specifically: Step 1), use ground latitude and longitude information to evenly divide the area.

[0027] Specifically, based on ground latitude and longitude information, the ground area with a preset latitude (e.g., 60°, 70°, etc.) can be uniformly divided using fixed longitude and latitude differences. The regions are represented as follows: .

[0028] Step 2), group the LEO satellites and build a two-layer architecture.

[0029] Due to the time-varying nature of the LEO satellite constellation, in this example, it can be determined based on time intervals. The satellite network topology is divided into multiple time slices, and then each time slice is analyzed. First, based on the coordinates mapped from the satellites to the ground, LEO satellites are mapped to the regions defined in step 1). LEO satellites in the same ground region form a group of LEO satellites. Each LEO satellite group is then managed by a MEO satellite covering the LEO satellites within its group; this MEO satellite is the managing satellite corresponding to that LEO satellite group. The satellite network for each time slice is modeled as an undirected graph. ,in, It can represent a satellite node. This can be represented as an ISL (Inter-Satellite Link). As an example, two intra-orbit ISLs and two inter-orbit ISLs can be established for each satellite. The intra-orbit ISLs can include links with the preceding satellite in the same orbit and links with the following satellite in the same orbit. The inter-orbit ISLs can include links established with two satellites across their orbits, for example, links with two satellites in adjacent left and right orbits. Each LEO satellite can obtain status information from other LEO satellites through inter-satellite links. See also... Figure 1 , Figure 1 The curved dashed lines in the diagram represent satellite orbits. A single satellite orbit can include multiple LEO satellites. Each LEO satellite can establish links with the two LEO satellites immediately preceding or following it in the same orbit, and also with LEO satellites in orbits to its left and right. (Illustrative example) Figure 1 The diagram also shows two paths, P1 and P2, between the source (src) satellite node and the destination (des) satellite node.

[0030] At the link level, the link channel model adopts the Free-Space Path Loss (FSPL) model. in, It can represent Time of the first The satellite and the first The distance between satellites' ISLs; It can represent the wavelength of light; It can represent Time of the first The satellite and the first The free-space path loss (ISL) between satellites. According to Shannon's theorem, the formula for calculating the maximum data transmission rate can be further obtained as follows: in, It can represent the noise power spectral density; It can represent link bandwidth; , , These can represent transmit power, transmit antenna gain, and receive antenna gain, respectively.

[0031] After dividing the large-scale constellation network through the above process, the training method of the routing policy network based on federated learning provided in this embodiment can be executed.

[0032] See also Figure 2 , Figure 2 A flowchart illustrating a training method for a federated learning-based routing policy network according to one embodiment is shown. The routing policy network trained by this method can be used to route data in a satellite network. As described above, the satellite network may include multiple low-Earth orbit satellite groups divided by region, each low-Earth orbit satellite group may be managed by a single management satellite, which is a medium-Earth orbit satellite. The above method can be executed by a first management satellite; it is understood that the first management satellite can be any one of multiple management satellites. Figure 2 As shown, the training method for the federated learning-based routing policy network may include the following steps 201-204, specifically: Step 201: Receive the regional feature vector sent by the first low Earth orbit satellite group under management.

[0033] In this embodiment, the first management satellite can manage a first low Earth orbit (LEO) satellite group. The regional network formed by the first LEO satellite group can be called the first regional network. The graph structure data corresponding to the first regional network can include nodes and edges, where nodes can represent LEO satellites and edges can represent inter-satellite links. Pre-trained graph neural networks (GNNs) can be pre-deployed in the LEO satellites of the first LEO satellite group. Thus, the LEO satellites can generate regional feature vectors corresponding to a certain time slice based on the graph structure data of the first regional network at a certain time slice, using their locally deployed GNNs and the graph structure data of the first regional network. The GNN can use pre-trained parameters to receive the graph structure data of the first regional network at a certain time slice (including link states and topology information within the time slice) as input, quickly processing the high-dimensional, complex graph structure data into a low-dimensional, standardized regional feature vector. Here, the GNN can be trained using various methods, such as supervised training algorithms. Since the use and training of graph neural networks are common techniques in this field, they will not be elaborated further here.

[0034] In some implementations, the aforementioned graph neural network may include a graph attention network (GAT), where nodes in the graph structure data can represent low Earth orbit satellites, and edges can represent inter-satellite links. Based on this, the low Earth orbit satellites in the first low Earth orbit satellite group can generate regional feature vectors through the following steps (1) and (2): Step (1): Graph attention network is used to calculate the graph structure data of the first region network to obtain the node feature vector of each node in the first region network.

[0035] Step (2): Generate regional feature vectors based on the node feature vectors corresponding to each low Earth orbit satellite in the first low Earth orbit satellite group.

[0036] Within each regional network, GAT can be used to encode local topology and link-state information; specifically, nodes... In the The feature vector in the next iteration It is obtained from its own features, neighbor node features, and edge features updated in the previous round: , in, It can represent satellite nodes A collection of neighboring satellites; The weight matrix is ​​trainable. It can represent an activation function; It can represent a node To the node The attention weight can be calculated using the following formula: , in, The feature vector that can represent ISL can include channel quality, remaining bandwidth, latency, etc.

[0037] LEO satellites can generate regional feature vectors by aggregating the node feature vectors of all nodes within the region. ,as follows: , in, This can represent a readout function. In this example, the readout function aggregates the node feature vectors of all nodes within a region into a fixed-dimensional vector representing the information of the entire region. The regional feature vector corresponding to a region's regional network is generated by aggregating the feature information of all satellite nodes within that region. It represents a low-dimensional representation of the entire regional network state, indicating the regional state and feature encoding of that region.

[0038] Step 202: Based on the regional feature vectors, a multi-agent reinforcement learning algorithm is used to train the local routing policy network.

[0039] In this embodiment, each management satellite can be treated as an independent intelligent agent, and each agent is equipped with an actor network and an evaluation network. The actor network serves as a local routing policy network for routing decisions. The criterion network is used for local evaluation of the actor network. Based on this, regional feature vectors can be used as environmental state information. A multi-agent reinforcement learning (MARL) algorithm is used to train the respective actor and criterion networks. The actor network, as a routing policy network, can be used for routing decisions in the first regional network. Here, routing refers to the transmission path of data packets from source to destination, including information such as source node, destination node, and number of intermediate hops. It is understood that using the MARL algorithm to train the actor and criterion networks is a well-known technique in the art and will not be elaborated upon here.

[0040] In some implementations, the aforementioned multi-agent reinforcement learning algorithm may include the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm. In this algorithm, each regional management satellite can be considered an independent agent, and each agent can be equipped with an Actor network and a Critic network. The Actor network can serve as a local routing policy network, which can be trained using the MAPPO algorithm. Based on this, step 202 may include: inputting the regional feature vector into the Multi-Agent Proximal Policy Optimization algorithm to train the local routing policy network.

[0041] In this implementation, the first management satellite can input the regional feature vector corresponding to the first regional network into the multi-agent near-end policy optimization algorithm to train the local routing policy network of the first management satellite.

[0042] The MAPPO algorithm is a widely used algorithm in the field of multi-agent reinforcement learning, belonging to the multi-agent extension of the PPO (Proximal Policy Optimization) algorithm. When using the MAPPO algorithm for local routing policy network training, a multi-objective optimization function for the path can be defined, which represents the objective utility function for the transmission path of any source-destination node pair. Due to time delay Packet loss rate With throughput The normalized values ​​are composed of.

[0043] The constraints of the utility function limit the weight coefficients to a sum of 1, the magnitude of the transmission power and actual transmission rate, and the flow conservation constraints that intermediate nodes in the path must adhere to. It can represent the transmission power. It can represent transmission rate and constraints. This can be interpreted as the intermediate nodes of a path adhering to the flow conservation constraint, meaning that the flow into an intermediate node is equal to the flow out of that intermediate node.

[0044] Since satellite routing decisions are made solely based on the current network state, the path selection problem can be modeled as a decentralized partially observable Markov decision process, where each agent is responsible for selecting a path within its region. The Dec-POMDP (Decentralized Partially Observable Markov Decision Process) is modeled as a 7-tuple, as follows: , in, It can represent a set of intelligent agents; It can represent the local state space of each agent α; It can represent the action space of each intelligent agent; It can represent a state transition function; It can represent the observation space of each intelligent agent; It can represent a reward function; It can represent a discount factor.

[0045] In this embodiment, the state space consists of state parameters such as link topology, connectivity, link quality, and resource allocation status within the region. Each state parameter is encoded by a graph neural network to form a low-dimensional representation that can be used for policy learning. The action space is a set of optional paths within the region, dynamically generated by a shortest path algorithm based on the real-time topology. The agent selects the optimal forwarding path to execute data transmission tasks. The observation space manages the local link and node states obtainable by MEO satellites within their coverage area, including the communication quality and link resources of LEO satellites.

[0046] Each managed MEO satellite obtains experience sequences by interacting with LEO nodes within its coverage area and updates its local policy using the MAPPO policy gradient method. The policy objective function is as follows: , in, It can represent a routing policy network; It can represent the dominance function; It can indicate the cropping range; This can represent the ratio of the current policy to the old policy. After training is complete, each managed MEO satellite can extract and retain the key parameter vector of its routing policy network. In this example, the parameter vector can also be used. These are called network parameters.

[0047] Step 203: Send the network parameters of the trained routing policy network to the preset aggregation satellite.

[0048] In this embodiment, a non-managed MEO satellite can be pre-selected as the aggregation satellite. The first managed satellite can then send the network parameters of the trained routing policy network to this aggregation satellite, for example, directly in plaintext. The aggregation satellite can receive the network parameters uploaded by all managed MEO satellites and fuse them to obtain a global routing policy model. Subsequently, the aggregation satellite can send the global model parameters of the global routing policy model to the managed satellites in each region.

[0049] In practice, security issues may arise during model (or network) training. For example, in the training of distributed intelligent routing models for large-scale satellite constellations, training parameters and intermediate model information need to be frequently exchanged and aggregated among multiple satellite nodes. Due to limited inter-satellite link resources and a complex communication environment, without effective encryption and verification mechanisms, transmitted model parameters are highly susceptible to theft, tampering, or forgery, leading to data leakage and model contamination during training. This can severely impact the accuracy and stability of the global model, and even trigger malicious attacks, threatening the intelligent routing security and system reliability of the entire constellation network.

[0050] To enhance security, in some implementations, step 203 above may specifically include the following steps (steps one through four): Step 1) Compress the network parameters using a sparsification method to obtain sparse network parameters.

[0051] In this implementation, the management of MEO satellites can employ sparsification methods to compress the network parameters of the local routing policy network, resulting in sparse network parameters. For example, Top-K sparsification can be used. Top-K sparsification is a dimensionality reduction or feature selection technique that retains the K most important elements in the data (sorted according to a certain metric) while setting the remaining elements to zero.

[0052] For example, after training the regional routing policy network, each regional management MEO satellite can compress the network parameters from its local model (including the local routing policy network) and transmit them uplink to the aggregation satellite. Taking the i-th region as an example, the management MEO satellite can obtain the network parameter vector after training. ,in, It can be a d1-dimensional vector. Managing MEO satellites can also compute the gradient vector of the model on the current dataset. ,in, For parameter dimensions, The original gradient vector The Each component. In practice, the network parameter vector... The dimension d1 and the gradient vector The dimension d2 may be the same or different. To reduce communication load, the Top-K sparsity method is used to manage MEO satellites, compressing the network parameter vector and gradient vector. Taking gradient vector compression as an example, a sparsity threshold can be set. Select the gradient vector with the largest absolute value from the original gradient vector. One component is set to zero, and the remaining components are set to zero to form a sparse gradient vector. For sparse gradient vectors Its mathematical expression is: , Top-K sparsification can reduce the original... -Dimensional gradient compression to contain only There are 1 non-zero elements, and the compression ratio is approximately 1. This can significantly reduce the amount of communication data. Furthermore, to ensure the complete structure can be restored subsequently, the index information of the elements can be recorded and retained. Here, the index information of the elements can refer to their position information within the original gradient vector.

[0053] Similarly, the network parameter vector of the i-th region can also be... Compression is performed to obtain sparse network parameters. Its mathematical expression is: , Among them, sparse network parameters It can include element index information. Network parameter vector The j-th component.

[0054] Step 2) Perform hash calculation on the sparse network parameters to obtain the hash value used for integrity verification.

[0055] Step 3) Encrypt the sparse network parameters and hash value to obtain ciphertext data.

[0056] In this implementation, to ensure the integrity and confidentiality of data transmission, the management of MEO satellites can adjust the sparse network parameters before transmission. Perform hash digests and encryption. For example, managing MEO satellites can use the SHA-256 hash function to hash sparse network parameters. Calculate the hash value as an integrity check. , Subsequently, MEO satellites can be managed using symmetric encryption algorithms and pre-shared keys. The sparse network parameters and their hash values ​​are encrypted to generate ciphertext data. , in, It can represent a series operation. This can represent an encrypted ciphertext packet. This ciphertext packet contains not only sparse network parameter information but also its hash digest. After receiving and decrypting the packet, the aggregated MEO satellite can verify the data integrity by recalculating the hash value and comparing the digest. AES encryption ensures the confidentiality of the update packet during transmission; even if the data is intercepted, the original network parameter information cannot be obtained.

[0057] Step 4) The encrypted data is sent to the preset aggregation satellite, which decrypts and verifies the integrity of the encrypted data. After the integrity verification is passed, a global routing policy model is generated.

[0058] In this implementation, the managing MEO satellite includes compressed and encrypted update packets in a transmission queue, scheduling transmission based on available bandwidth of the inter-satellite link and network topology. For example, the managing MEO satellite can send update packets to the aggregation satellite in batches through multiple time slots during periods of satellite convergence or low communication load to improve link utilization. During transmission, the managing MEO satellite maintains an upload queue, numbering and managing the order of update packets to ensure that the aggregation satellite can assemble and decrypt them in the correct order to obtain complete model update data. After receiving the encrypted data from the managing MEO satellite, the aggregation satellite can decrypt and verify its integrity. After the integrity verification passes, it generates a global routing policy model based on the decrypted network parameters.

[0059] In some implementations, after the aggregation satellite decrypts and obtains the sparse network parameters, it can reconstruct the sparse network parameters based on the element index information to obtain the reconstructed network parameters. Furthermore, the aggregation satellite can use a federated averaging algorithm to fuse multiple reconstructed network parameters to obtain global model parameters. The specific process may include the following 1)-3), specifically: 1) The aggregated MEO satellite can perform AES decryption, ECDSA (Elliptic Curve Digital Signature Algorithm) verification, and sparse restoration on the received encrypted packets.

[0060] Specifically, taking the encrypted packet uploaded from the i-th region as an example, the aggregated MEO satellites can first use a shared symmetric key. For the received ciphertext Perform AES-128-CBC decryption to restore the sparse network parameters. and its hash .

[0061] , Next, the aggregated MEO satellite uses the public key provided by the management MEO satellite to perform ECDSA digital signature verification on the data, ensuring that the data packets originated from a legitimate sender and have not been tampered with. After successful signature verification, the aggregated MEO satellite uses pre-recorded element index information to process the sparse network parameters. Restore to complete d1-dimensional network parameters This means that the network parameters are preserved by filling the corresponding index positions in the original vector, and zeros are filled in the remaining positions, thus completing the restoration of the network parameters.

[0062] 2) The MEO satellites are aggregated using FedAvg (Federated Average Algorithm) to generate a global model by weighting the samples by sample size.

[0063] Specifically, a weighted federated averaging algorithm is used to aggregate the network parameters of each region to calculate the global model. Assume that the regions participating in federated learning... There are N regions, and the network parameters for these N regions are: The training sample size for each region is By aggregating MEO satellite data, the total sample size for all regions can be calculated first, and then the global model parameters can be calculated using a sample size-weighted average method. , in, This can represent the parameters of the global routing strategy model obtained after aggregation. The weighting coefficients reflect the proportion of contribution of each region's data volume to the global model, ensuring that regions with larger sample sizes have a greater impact on the final model, thus achieving a fair fusion of training results from each region.

[0064] 3) The aggregated MEO satellites perform SHA-256 signing and AES encryption on the global model and broadcast it to each management MEO satellite via ISL.

[0065] Specifically, by aggregating MEO satellites to obtain global model parameters Then, the aggregated MEO satellites can use the SHA-256 hash function to calculate the hash value of the model parameter vector: , Subsequently, the MEO satellite uses a symmetric encryption algorithm, followed by a pre-shared key. global model parameters With hash value Concatenation encryption is used to encrypt the global model parameters and their hash digests. , Finally, through ISL The data packet is broadcast to all regional management MEO satellites to ensure that all regions use a unified global routing policy model. Upon receiving the data packet, each regional management MEO satellite can decrypt it using the same symmetric key and perform integrity verification by recalculating the hash value and the accompanying digest, ensuring the security and reliability of the model update process. In this implementation, to address security threats during model training, encryption and verification mechanisms are combined to ensure the integrity and confidentiality of training parameters and model information during transmission and aggregation, preventing data leakage, tampering, and malicious attacks, and ensuring the security of training data and model information. Simultaneously, through a regional autonomous management mechanism, the data independence and privacy of each region are maintained, preventing information leakage and enhancing the security protection capabilities of the constellation's intelligent routing.

[0066] Step 204: Update the network parameters of the local routing policy network based on the global model parameters sent by the aggregated satellites.

[0067] In this embodiment, after receiving the global model parameters sent by the aggregation satellite, each management satellite can update the network parameters of its local routing policy network based on the global model parameters. For example, the global model parameters can be directly used as the network parameters of the local routing policy network. Alternatively, the global model parameters and the network parameters of the local routing policy network can be used to perform calculations, and the network parameters of the local routing policy network can be updated based on the calculation results.

[0068] In some implementations, each region manages MEO satellites upon receiving the encrypted global model. Then, the global model parameters are obtained after decryption and integrity verification. Satellite management can avoid directly using these global model parameters. Instead of replacing the local regional model, a weighted fusion mechanism is adopted to achieve a balance between the regional model and the global model. Based on this, step 204 above can specifically include: linearly weighting and fusing the global model parameters transmitted by the aggregating satellites with the network parameters of the local routing policy network to obtain the updated network parameters of the local routing policy network.

[0069] Specifically, taking the i-th region as an example, the region management MEO satellite can receive the global model parameter vector. The model parameter vector of the current routing decision network in the local area A linearly weighted fusion method is used to balance global consistency and local adaptability. Let the fusion coefficient be... Then the first The updated model parameters for each region are: Among them, the fusion coefficient It can be used to balance the influence of the global model and the local routing decision network. These are the updated regional network parameters. In practice, the fusion coefficient... The parameters can be adjusted according to actual network needs to control the weight ratio of the global model during updates. In this way, the fused network parameters retain the characteristics of local learning while incorporating the results of global optimization, thus improving the model's robustness and generalization ability.

[0070] See also Figure 3 , Figure 3 A schematic diagram illustrating an example of regional coordination in network training and federated learning is shown. Figure 3 In the example shown, LEO satellites can extract features from the graph structure data of their local network using locally pre-deployed Graph Neural Networks (GNNs), obtaining regional feature vectors, which are then sent to the corresponding management MEO satellites. The management MEO satellites train their local routing policy networks based on these regional feature vectors and send the regional network parameters of the trained routing policy networks to the federated aggregation MEO satellites. The federated aggregation MEO satellites aggregate the regional network parameters sent by multiple management MEO satellites to obtain a global routing policy model, and then distribute the global model parameters of this model to multiple management MEO satellites, allowing them to update the network parameters of their local routing policy networks based on these global model parameters.

[0071] Each MEO satellite under management is based on Figure 2 The training method for the federated learning-based routing policy network shown can be used for routing after the local routing policy network is trained.

[0072] Please see Figure 4 , Figure 4 A flowchart illustrating a routing method for a satellite network according to one embodiment is shown. As described above, the satellite network may include multiple low Earth orbit satellite groups divided by region, each low Earth orbit satellite group being managed by its corresponding management satellite, which is a medium Earth orbit satellite. Each management satellite may pre-deploy [unclear - possibly referring to a specific type of satellite]. Figure 2 The routing policy network trained using the method shown is illustrated. The routing method described above for satellite networks can be executed via a first management satellite; it can be understood that the first management satellite can be any one of multiple management satellites. For example... Figure 4 As shown, the routing method for satellite networks may include the following steps 401-403, specifically: Step 401: Obtain the routing task.

[0073] In this embodiment, when data needs to be transmitted from a source node to a destination node in a satellite network, a routing task is generated, which may include a source node and a destination node.

[0074] Step 402: Receive the regional feature vector sent by the first low Earth orbit satellite group under management.

[0075] In this embodiment, the first management satellite can manage a first low Earth orbit (LEO) satellite group. The regional network formed by the first LEO satellite group can be called the first regional network. The graph structure data corresponding to the first regional network can include nodes and edges, where nodes can represent LEO satellites and edges can represent inter-satellite links. Pre-trained graph neural networks (GNNs) can be pre-deployed in the LEO satellites of the first LEO satellite group. Thus, the LEO satellites can generate regional feature vectors based on their locally deployed GNNs and the real-time graph structure data of the first regional network. It can be understood that the graph structure data used by the LEO satellites to generate regional feature vectors includes the current real-time graph structure data. That is, the graph neural network uses pre-trained parameters to receive real-time graph structure data (including real-time link status and topology information) within the first regional network as input, quickly processing the high-dimensional, complex graph structure data into a low-dimensional, standardized regional feature vector. This allows the subsequent routing policy network to more easily understand the current network status.

[0076] Step 403: If the source node and the destination node are nodes in the first regional network, the routing decision is performed using the locally deployed routing policy network based on the regional feature vector, the route from the source node to the destination node is calculated, and a scheduling instruction is generated to control the data forwarding from the source node to the destination node.

[0077] In this embodiment, the regional feature vector received by the first management satellite is generated based on the real-time graph structure data of the first regional network. Therefore, the regional feature vector can characterize the real-time link state information of the first regional network. Combining the real-time link state information of the first regional network and the locally deployed routing policy network, the first management satellite can calculate the optimal hop count and path sequence from the source node to the destination node. The path sequence can include multiple LEO satellite nodes that must be traversed sequentially from the source node to the destination node. It can be understood that the routing policy network locally deployed by the first management satellite is trained according to MAPPO (Multi-Agent Proximal Policy Optimization).

[0078] In some implementations, the routing method for satellite networks described above may also include the following steps 1)-3), specifically: Step 1): If the source node is a node in the first regional network and the destination node is not a node in the first regional network, the shortest path algorithm is used to calculate the cross-domain path from the source node to the destination node. Here, the cross-domain path can be used to indicate at least two regions that the data passes through sequentially during the forwarding process from the source node to the destination node. These at least two regions can include the source region where the source node is located and the destination region where the destination node is located.

[0079] Step 2) Use the locally deployed routing policy network to compute routing decisions from the source node to the boundary node. Here, the boundary node is the boundary node between the source region and the next region.

[0080] Step 3) Generate scheduling instructions from the source node to the boundary node to control the data forwarding from the source node to the boundary node. After the data is forwarded to the next area, the management satellite corresponding to the next area controls the data to continue forwarding until the data is forwarded to the destination node.

[0081] In cross-regional routing scenarios, due to the uniform division of regions, the management MEO satellite in the source region can first use the shortest path algorithm to calculate the cross-regional path, obtaining the regions the data will pass through. Then, it sequentially outputs the specific boundary forwarding nodes through the routing policy network in each of the traversed regions. For example, starting from the source region corresponding to the source node, the management MEO satellite first selects the optimal boundary node for the next region and uses the locally deployed routing policy network to calculate the path from the current node to the boundary node of the next region. After the data arrives in the next region, it extracts the real-time network status of that region and uses the local routing policy network to calculate the path to the next forwarding hop, repeating this process until the destination node in the destination region is reached. This hierarchical routing strategy allows each region to both conform to the global optimization results and maintain local intelligent routing, achieving autonomous, low-latency, and highly reliable routing in large-scale dynamic environments.

[0082] According to another embodiment, a training device for a routing policy network based on federated learning is provided. The routing policy network is used to route data in a satellite network, which includes multiple low Earth orbit satellite groups divided by region. Each low Earth orbit satellite group is managed by its own management satellite, which is a medium Earth orbit satellite. The device is deployed on a first management satellite.

[0083] Figure 5 A schematic block diagram of a training apparatus for a federated learning-based routing policy network according to one embodiment is shown. Figure 5As shown, the training device 500 for the federated learning-based routing policy network may include: a receiving unit 501, configured to receive regional feature vectors sent by a first low Earth orbit satellite group under its management, wherein the regional feature vectors are generated by the low Earth orbit satellites in the first low Earth orbit satellite group based on the graph structure data of the graph neural network deployed locally by the low Earth orbit satellites and the graph structure data of the first regional network composed of the first low Earth orbit satellite group; a training unit 502, configured to train a local routing policy network using a multi-agent reinforcement learning algorithm based on the regional feature vectors, wherein the routing policy network is used to implement the routing decisions of the first regional network; a sending unit 503, configured to send the network parameters of the trained routing policy network to a preset aggregation satellite, so that the aggregation satellite can fuse the network parameters uploaded by the management satellites of multiple regions to obtain a global routing policy model, and send the global model parameters of the global routing policy model to the management satellites of multiple regions, wherein the aggregation satellite is a medium Earth orbit satellite; and an updating unit 504, configured to update the network parameters of the local routing policy network based on the global model parameters sent by the aggregation satellite.

[0084] According to another embodiment, a routing device for a satellite network is provided, wherein the satellite network includes multiple low Earth orbit satellite groups divided by region, each low Earth orbit satellite group is managed by its own management satellite, the management satellite being a medium Earth orbit satellite, and each management satellite is pre-deployed with [missing information - likely related to routing methods]. Figure 2 The routing policy network trained by the method shown is deployed on the first management satellite.

[0085] Figure 6 A schematic block diagram of a routing apparatus for a satellite network according to one embodiment is shown. Figure 6 As shown, the routing device 600 for satellite networks includes: a task acquisition unit 601 configured to acquire a routing task, wherein the routing task includes a source node and a destination node; a vector receiving unit 602 configured to receive a regional feature vector sent by a first low Earth orbit satellite group under its management, wherein the regional feature vector is generated by the low Earth orbit satellites in the first low Earth orbit satellite group based on the graph structure data of the graph structure of the first regional network composed of the first low Earth orbit satellite group and its locally deployed graph neural network; and a routing decision unit 603 configured to, when the source node and the destination node are nodes in the first regional network, perform routing decisions based on the regional feature vector using a locally deployed routing policy network, calculate the route from the source node to the destination node, generate scheduling instructions, and control the data forwarding from the source node to the destination node.

[0086] Corresponding to the above method, the present invention also provides a medium Earth orbit satellite, which includes a processor and a memory. The memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the medium Earth orbit satellite performs the steps of the method described above.

[0087] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned method. The computer-readable storage medium may be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0088] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0089] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0090] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0091] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A training method for a routing policy network based on federated learning, wherein, The routing policy network is used to route data in the satellite network, which includes multiple low Earth orbit satellite groups divided by region. Each low Earth orbit satellite group is managed by its own management satellite, which is a medium Earth orbit satellite. The method is executed by the first management satellite. The method includes the following steps: Receive the regional feature vector sent by the first low Earth orbit satellite group under management, wherein the regional feature vector is generated by the graph structure data of the low Earth orbit satellites in the first low Earth orbit satellite group based on the graph neural network deployed locally and the first regional network composed of the first low Earth orbit satellite group; Based on the region feature vector, a local routing policy network is trained using a multi-agent reinforcement learning algorithm, wherein the routing policy network is used to implement the routing decision of the first region network. The network parameters of the trained routing policy network are sent to a preset aggregation satellite, which then fuses the network parameters uploaded by management satellites in multiple regions to obtain a global routing policy model. The global model parameters of the global routing policy model are then sent to management satellites in multiple regions. The aggregation satellite is a medium Earth orbit satellite. The network parameters of the local routing policy network are updated based on the global model parameters sent by the aggregated satellites.

2. The method according to claim 1, wherein, The graph neural network includes a graph attention network, where nodes in the graph structure data represent low Earth orbit satellites and edges represent inter-satellite links; and the low Earth orbit satellites in the first low Earth orbit satellite group generate the region feature vector in the following manner: A graph attention network is used to calculate the graph structure data of the first region network to obtain the node feature vector of each node in the first region network. A regional feature vector is generated based on the node feature vectors corresponding to each low Earth orbit satellite in the first low Earth orbit satellite group.

3. The method according to claim 1, wherein, The multi-agent reinforcement learning algorithm includes a multi-agent near-end policy optimization algorithm, where the first management satellite, as an independent agent, includes a routing policy network.

4. The method according to claim 1, wherein, The step of sending the network parameters of the trained routing policy network to a preset aggregation satellite includes: The network parameters are compressed using a sparsification method to obtain sparse network parameters, wherein the sparse network parameters include element index information; The sparse network parameters are hashed to obtain hash values ​​used for integrity verification; The sparse network parameters and the hash value are encrypted to obtain ciphertext data; The encrypted data is sent to a preset aggregation satellite, which decrypts and verifies the integrity of the encrypted data. After the integrity verification is passed, a global routing policy model is generated.

5. The method according to claim 4, wherein, After the aggregation satellite decrypts the sparse network parameters, it restores the sparse network parameters according to the element index information to obtain the restored network parameters; and the aggregation satellite uses a federated averaging algorithm to fuse multiple restored network parameters to obtain global model parameters.

6. The method according to claim 1, wherein, The step of updating the network parameters of the local routing policy network based on the global model parameters sent by the aggregated satellites includes: The global model parameters transmitted by the aggregated satellites are linearly weighted and fused with the network parameters of the local routing policy network to obtain the updated network parameters of the local routing policy network.

7. A routing method for satellite networks, wherein, The satellite network comprises multiple low Earth orbit satellite groups divided by region, each low Earth orbit satellite group being managed by a management satellite, wherein the management satellite is a medium Earth orbit satellite, and each management satellite is pre-deployed with a routing policy network trained according to any one of claims 1-6, wherein the method is executed by the first management satellite; the method includes: Obtain a routing task, wherein the routing task includes a source node and a destination node; Receive the regional feature vector sent by the first low Earth orbit satellite group under management, wherein the regional feature vector is generated by the graph structure data of the low Earth orbit satellites in the first low Earth orbit satellite group based on the graph neural network deployed locally and the first regional network composed of the first low Earth orbit satellite group; If the source node and the destination node belong to nodes in the first regional network, a routing decision is performed using a locally deployed routing policy network based on the regional feature vector, the route from the source node to the destination node is calculated, and a scheduling instruction is generated to control the data forwarding from the source node to the destination node.

8. The method according to claim 7, wherein, The method further includes: When the source node is a node in the first regional network and the destination node is not a node in the first regional network, the shortest path algorithm is used to calculate the cross-domain path from the source node to the destination node. The cross-domain path is used to indicate at least two regions that the data passes through sequentially during the forwarding process from the source node to the destination node. The at least two regions include the source region where the source node is located and the destination region where the destination node is located. The routing policy network deployed locally is used to compute routing decisions from source nodes to boundary nodes, where the boundary node is the boundary node between the source region and the next region. The system generates scheduling instructions from the source node to the boundary node, controls the data forwarding from the source node to the boundary node, and after the data is forwarded to the next area, the management satellite corresponding to the next area controls the data to continue forwarding until the data is forwarded to the destination node.

9. A medium Earth orbit satellite, comprising a processor, a memory, and computer programs / instructions stored in the memory, characterized in that, The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the Earth-orbiting satellite implements the steps of the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 8.

Images