A Clustered QoS Routing Method for Large-Scale Satellite Networks Based on Deep Reinforcement Learning

Abstract

This invention discloses a clustered QoS routing method suitable for large-scale satellite networks with LEO-MEO dual-layer constellations. The method constructs a LEO-MEO hierarchical clustered management architecture, with MEO satellites acting as cluster heads responsible for link state acquisition and routing decisions. It categorizes services into three types based on QoS requirements: latency-sensitive, bandwidth-sensitive, and high-reliability, establishing link QoS indicators and a path comprehensive utility model. The routing problem is modeled as a multi-agent partially observable Markov decision process, designing a reward function positively correlated with path utility. A GNN-GRU module extracts topological features and outputs path Q-values, employing a QMIX mechanism to achieve multi-cluster head collaborative decision-making. Cluster heads complete intra-cluster and inter-cluster routing, generating optimal paths and performing forwarding and bandwidth reservation. This invention, using MEO as cluster heads, reduces propagation latency and decision lag in centralized architectures, enabling differentiated QoS guarantees and global routing performance optimization in dynamic satellite networks.

Description

Technical Field

[0001] This invention relates to the field of satellite communication technology, and in particular to a clustered QoS routing method for large-scale satellite networks based on deep reinforcement learning. Background Technology

[0002] With the rapid development of satellite communication technology, large-scale satellite networks composed of hundreds or thousands of Low Earth Orbit (LEO) satellites will become a key infrastructure for realizing ubiquitous 6G connectivity, carrying diverse services such as global broadband access, IoT data transmission, and emergency communication support. However, satellite networks are characterized by highly dynamic topology changes, limited onboard processing resources, and uneven distribution of service traffic, posing a severe challenge to ensuring the Quality of Service (QoS) of multiple services in large-scale satellite networks. Routing, as a fundamental technology for enabling interconnection between network nodes, directly affects the network's transmission capacity and QoS. Therefore, it is necessary to explore adaptive routing mechanisms to improve the QoS assurance capabilities of satellite networks in highly dynamic scenarios.

[0003] Routing in large-scale satellite networks is primarily influenced by factors such as high-speed satellite movement, frequent inter-satellite link (ISL) switching, and diverse service requirements. Existing routing architectures are mainly divided into centralized and distributed types. For example, a centralized architecture based on software-defined networking involves a ground center or geostationary Earth Orbit (GEO) satellites uniformly collecting the network status and calculating the optimal path before distributing routing tables. However, collecting network-wide link status information from the ground center incurs significant signaling overhead, while long-distance LEO-GEO cross-layer links introduce high propagation delays, making it difficult for central nodes to promptly detect link status changes. In contrast, a distributed architecture uses each satellite as the routing decision-maker, independently making next-hop decisions based on locally perceived neighbor status. While this improves the speed of status perception, satellite nodes only have local observation capabilities, making it difficult to anticipate congested areas and plan globally optimal paths.

[0004] To balance real-time performance and global performance in routing decisions, cluster-based routing algorithms offer a good solution. The LEO network is divided into multiple clusters, with a Medium Earth Orbit (MEO) satellite acting as the cluster head, responsible for collecting the status of LEO nodes and links within the cluster and performing routing calculations accordingly. As satellite applications expand from traditional relay communication to areas such as the Industrial Internet, airborne communication, and low-altitude intelligent networks, the types of services carried by the network will exhibit a high degree of diversity, with different services having varying QoS requirements for latency, bandwidth, and packet loss rate. However, existing cluster-based routing methods are insufficient in ensuring differentiated QoS for different services, mostly employing a single path metric or a "best-effort" forwarding model, neglecting the differentiated QoS requirements of multiple services. Summary of the Invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art or related art.

[0006] Therefore, the purpose of this invention is to provide a large-scale satellite network clustered QoS routing method based on deep reinforcement learning, which uses medium-Earth orbit satellites as cluster heads to solve the problem of large propagation delays introduced by long-distance satellite-to-ground links or LEO-GEO cross-layer links in traditional centralized architectures, and the resulting routing decision lag.

[0007] To achieve the above objectives, the present invention provides a large-scale satellite network clustering QoS routing method based on deep reinforcement learning, applicable to a two-layer constellation satellite network composed of low-Earth orbit (LEO) satellites and medium-Earth orbit (MEO) satellites. The method includes the following steps:

[0008] S1. Construct a hierarchical and clustered management architecture for LEO-MEO, dividing the LEO satellite network into several clusters. Each cluster is headed by a corresponding MEO satellite, which is responsible for collecting link status information and making routing decisions within the cluster.

[0009] S2. Satellite network services are classified into three categories according to QoS requirements: latency-sensitive, bandwidth-sensitive, and high-reliability. A link QoS indicator model including latency, remaining bandwidth, and packet loss rate is established. At the same time, a path QoS utility model is constructed, which includes latency utility, bandwidth utility, packet loss rate utility, and a path comprehensive utility obtained by a weighted linear combination of the three.

[0010] S3. The clustered routing problem is modeled as a multi-agent partially observable Markov decision process. A quintuple consisting of state space, action space, transition probability, reward function and discount factor is defined, where the reward function is positively correlated with the overall path utility and a fixed penalty term is given when the data packet transmission fails.

[0011] S4. The cluster head processes the intra-cluster link state information through the GNN-GRU-based topology feature extraction module, and outputs the candidate path Q value after spatial feature aggregation, GRU state update and graph-level readout.

[0012] S5. A multi-agent collaborative decision-making mechanism based on QMIX is adopted to nonlinearly aggregate the local values ​​of each MEO cluster head into the global total value. Monotonicity constraints are used to ensure local optimal decision-making and promote the improvement of global QoS performance.

[0013] S6. The cluster head performs intra-cluster and inter-cluster routing decisions based on the candidate path Q value, generates the end-to-end optimal routing path and sends it to the LEO satellite, and completes packet forwarding and bandwidth resource reservation.

[0014] In the above technical solution, preferably, the LEO-MEO hierarchical clustering management architecture described in step S1 is constructed as follows:

[0015] S11. Divide the global Earth's surface space into a fixed logical grid and abstract it into virtual nodes. LEO satellites are mapped to the corresponding virtual nodes according to their latitude and longitude coordinates.

[0016] S12. Based on the orbital direction and logical adjacency, LEO satellites are divided into several clusters. Each cluster contains a preset number of orbits and a preset number of logically adjacent LEO satellites per orbit. MEO satellites establish connections with LEO satellites within the cluster through inter-layer links to realize the collection of status information and the issuance of commands within the cluster.

[0017] In any of the above technical solutions, preferably, the modeling rules for the link QoS indicator model in step S2 are as follows:

[0018] Link delay is the sum of propagation delay, transmission delay, queuing delay based on the M / M / 1 / m queuing model, and processing delay. The total path delay is the sum of the delays of all links on the path.

[0019] The remaining bandwidth of a link is the difference between the total capacity of the link and the bandwidth already used. The remaining bandwidth of a path is the minimum value of the remaining bandwidth of all links on the path.

[0020] The link packet loss rate is the probability of traffic overload blocking based on the M / M / 1 / m model, while the end-to-end packet loss rate of the path is the product of the failure forwarding probabilities of each hop on the path.

[0021] In any of the above technical solutions, preferably, the calculation rule for the path QoS utility model in step S2 is as follows:

[0022] Time delay effect The theoretical shortest path delay Total delay of the current path The ratio, expressed as: ;

[0023] Bandwidth utility Remaining bandwidth for path bottleneck links with initial link capacity The ratio, expressed as: ;

[0024] Packet loss rate utility To avoid all possible packet loss nodes on the path The rate, expressed as: ;

[0025] Path comprehensive utility The calculation formula is: ,in For time delay effect, For bandwidth efficiency, For packet loss rate utility, , , These are the corresponding weighting factors for latency utility, bandwidth utility, and packet loss rate utility, respectively, satisfying... Furthermore, the weighting factor is dynamically assigned based on the sensitivity of the QoS indicators for the service type.

[0026] In any of the above technical solutions, preferably, the quintuple of the multi-agent partially observable Markov decision process in step S3 is... Defined as:

[0027] The state space S is the set of feature vectors for all links. These feature vectors include link queuing length, remaining bandwidth, packet loss rate, and latency. A single cluster head only obtains local state observations within its managed cluster, expressed as: ,in, The data for the link represents the queue length, reflecting the degree of link congestion; This represents the remaining bandwidth of the link, reflecting its carrying capacity. The packet loss rate of the link reflects the reliability of transmission; For link latency;

[0028] Action space A is a set of candidate paths from the source node to the destination node. A path library is generated by pre-training the K-shortest path algorithm with time delay constraints. The cluster head selects the optimal path from the path library.

[0029] The transition probability P is the probability that the network topology will evolve from the current congestion state to the state at the next moment after the routing decision is made, reflecting the dynamic changes of satellite high-speed movement and service traffic.

[0030] The reward function R is a reward based on the QoS utility function, and the reward value is related to the overall path utility. Positive correlation; if a data packet fails to reach its destination satellite due to congestion or link failure, a fixed penalty is imposed. The calculation formula is:

[0031] ,in, As a reward value, The basic reward constant is a fixed positive reward given when a data packet successfully reaches its destination satellite. It is used to incentivize cluster heads to complete the basic data packet delivery task and ensure basic route reachability. This is a path-wide utility scaling factor used to adjust the ratio of the QoS reward component to the base reward.

[0032] Discount factor The weighting coefficient, which balances the current routing benefits with the future network congestion, ranges from 0 to 1.

[0033] In any of the above technical solutions, preferably, the processing flow of the GNN-GRU-based topology feature extraction module in step S4 is as follows:

[0034] S41. Spatial Feature Aggregation: Execute L rounds of message passing based on links. In the initial phase, when l=0, each link... Hidden state From state space Link feature vector Obtained through mapping via a fully connected layer;

[0035] During the message generation phase, the link From its set of adjacent links In the The next received aggregation message is:

[0036]

[0037] in, and For link and its adjacent links In the The hidden state of the next step. For the message generation function, this paper adopts a weighted aggregation based on an attention mechanism, defined as:

[0038]

[0039] in, This indicates that the feature vectors of the two links are concatenated. It is a single-layer fully connected network. Attention weights;

[0040] S42, GRU State Update: After each round of message passing, the hidden state of the link is updated through the GRU's update gate and reset gate mechanism, i.e., the link's hidden state. In the The hidden state after the next update is:

[0041] ,in, For link In the The next received aggregate message.

[0042] S43. Graph-level reading: First, sum and aggregate all link features within the same subgraph to obtain the subgraph-level representation vector. : ,in, This represents the set of all links within the current subgraph. It is a link go through The final hidden state after the GRU update. Through the non-linear mapping of this network, the Q-value of the current candidate path is finally output. : ,in, , , This is the weight matrix. , , For bias terms; The SELU activation function is used, and Dropout regularization is inserted between layers to prevent overfitting. Final output. As a Q-value estimate of the current state-action pair.

[0043] In any of the above technical solutions, preferably, the implementation rules of the QMIX-based multi-agent collaborative decision-making mechanism in step S5 are as follows:

[0044] Employing a distributed decision-making and centralized training model, the local value Q of the i-th MEO cluster head is integrated through a QMIX hybrid network. i Nonlinear aggregation for global total value Q tot The aggregation formula is:

[0045] in, For the first Each MEO cluster head is based on a local observation sequence. and actions Calculated local value; Nonlinear aggregation functions characterized by hybrid networks; The network state is used as input to the supernetwork to adjust the aggregate weights;

[0046] The weights W of the hybrid network are dynamically generated by the supernetwork and remain non-negative. Through monotonicity constraints Ensuring that the local optimal decisions of each cluster head can promote the overall improvement of QoS performance of the entire network and avoid routing decision oscillations and resource conflicts in the cluster boundary area.

[0047] In any of the above technical solutions, preferably, the specific process of the intra-cluster routing decision in step S6 is as follows:

[0048] S61. After the source LEO node in the cluster generates a service request, it will report the service request message containing the source node ID, destination node ID and bandwidth requirements to the MEO cluster head of this cluster.

[0049] S62. The MEO cluster head determines whether the destination node is located within its cluster. If so, it queries the pre-computed path table for K candidate paths between the source and destination nodes. ;

[0050] S63. For each candidate path Simulated bandwidth requirements Assignment, generating hypothetical states And extract the link state matrix ,in, For the real number space, d represents the total number of links. f For each link, the link state matrix includes remaining bandwidth, bandwidth allocation status, propagation delay, and end-to-end delay as a characteristic dimension.

[0051] S64. Input the link state matrix into the GNN-DQN model and output the Q-value of each candidate path. The candidate path with the largest Q value is selected as the optimal routing path. ;

[0052] The S65 and MEO cluster heads distribute the optimal routing path to the LEO nodes along the path. Each node performs hop-by-hop forwarding according to the path and reserves the corresponding bandwidth resources along the path.

[0053] In any of the above technical solutions, preferably, the specific process of the inter-cluster routing decision in step S6 is as follows:

[0054] S66. After the source cluster MEO cluster head determines that the destination node is located in another cluster, it determines the destination cluster to which the destination node belongs, and generates a cluster sequence on the cluster-level topology graph using the shortest path algorithm. ;

[0055] S67. Determine adjacent clusters sequentially according to the cluster sequence. and For cross-cluster boundary node pairs, select the inter-cluster inter-satellite link with the closest geographical distance and sufficient remaining bandwidth to determine the exit node. and entry node ;

[0056] S68. The MEO clusters of each transit cluster, with the ingress and egress nodes as source-destination nodes, execute the intra-cluster routing decision process of S61-S65 in parallel to generate the optimal sub-path for each cluster. ;

[0057] S69. Concatenate the optimal sub-path of each cluster with the cross-cluster link in sequence to form an end-to-end globally optimal routing path. ;

[0058] S610 and the source cluster MEO cluster head distribute the globally optimal routing path to all LEO nodes along the path. Each node performs hop-by-hop forwarding and reserves the corresponding bandwidth resources along the path.

[0059] In any of the above technical solutions, preferably, the method further includes model training and performance evaluation steps:

[0060] S7. Construct a GNN-DQN routing decision model, obtain training experience by randomly generating business requests and store it in a public experience replay buffer, and update the network parameters using the square of the time difference error as the loss function:

[0061] ,in, , Parameters for the current network and the target network, respectively.

[0062] S8. Each MEO cluster head agent adopts a centralized training and distributed execution paradigm. During the training phase, network parameters and experience buffers are shared, and training is unified with end-to-end routing performance as the joint reward signal. During the inference phase, routing decisions are executed independently based on the local state of the cluster.

[0063] S9. By comparing the average path latency, average minimum remaining bandwidth of the link, and path packet loss rate under different network load rates, the algorithm performance is evaluated and optimized.

[0064] Compared with existing technologies, the advantages of the large-scale satellite network clustered QoS routing method based on deep reinforcement learning provided by this invention are as follows:

[0065] First, it solves the problem that traditional deep reinforcement learning routing models need to be retrained when satellite network topology changes. By introducing graph neural networks to adaptively extract features from the non-Euclidean topology of satellite networks, it significantly improves the generalization ability and scalability of the model among on-board clusters of different sizes and structures.

[0066] Second, it achieves differentiated QoS guarantee for multiple services in a large-scale, highly dynamic satellite network. By combining clustered routing with deep reinforcement learning, it dynamically optimizes path decisions and can still guarantee key service quality requirements such as latency, bandwidth, and packet loss rate in scenarios with frequent topology changes.

[0067] Third, reduce on-board resource overhead and routing computation complexity by adopting a clustering mechanism to divide the large-scale satellite network into multiple management domains. Sub-path optimization and global routing coordination are performed within and between clusters, thereby improving overall routing efficiency and system stability while ensuring QoS. Attached Figure Description

[0068] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0069] Figure 1 This diagram illustrates an application scenario of the large-scale satellite network clustered QoS routing method according to an embodiment of the present invention.

[0070] Figure 2 A flowchart illustrating a large-scale satellite network clustered QoS routing method according to an embodiment of the present invention is shown.

[0071] Figure 3 A flowchart of step S1 as described in an embodiment of the present invention is shown;

[0072] Figure 4 A flowchart of step S4 as described in an embodiment of the present invention is shown;

[0073] Figure 5 A flowchart of step S6 according to an embodiment of the present invention is shown;

[0074] Figure 6 A flowchart of step S6 according to another embodiment of the present invention is shown;

[0075] Figure 7 A flowchart illustrating a large-scale satellite network clustered QoS routing method according to another embodiment of the present invention is shown.

[0076] Figure 8 This paper presents a comparison chart of the average path delay of three routing algorithms under different load rates according to embodiments of the present invention.

[0077] Figure 9 This paper presents a comparison chart of the average link utilization of the entire network under different load rates for three routing algorithms involved in the embodiments of the present invention;

[0078] Figure 10 The diagram shows a comparison of the path packet loss rate of three routing algorithms under different load rates in the embodiments of the present invention. Detailed Implementation

[0079] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0080] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0081] like Figure 1 and 2 As shown, a large-scale satellite network clustering QoS routing method based on deep reinforcement learning according to an embodiment of the present invention is applied to a two-layer constellation satellite network composed of low-Earth orbit (LEO) satellites and medium-Earth orbit (MEO) satellites. The method includes the following steps:

[0082] S1. Construct a hierarchical and clustered management architecture for LEO-MEO, dividing the LEO satellite network into several clusters. Each cluster is headed by a corresponding MEO satellite, which is responsible for collecting link status information and making routing decisions within the cluster.

[0083] S2. Satellite network services are classified into three categories according to QoS requirements: latency-sensitive, bandwidth-sensitive, and high-reliability. A link QoS indicator model including latency, remaining bandwidth, and packet loss rate is established. At the same time, a path QoS utility model is constructed, which includes latency utility, bandwidth utility, packet loss rate utility, and a path comprehensive utility obtained by a weighted linear combination of the three.

[0084] S3. The clustered routing problem is modeled as a multi-agent partially observable Markov decision process. A quintuple consisting of state space, action space, transition probability, reward function and discount factor is defined, where the reward function is positively correlated with the overall path utility and a fixed penalty term is given when the data packet transmission fails.

[0085] S4. The cluster head processes the intra-cluster link state information through the GNN-GRU-based topology feature extraction module, and outputs the candidate path Q value after spatial feature aggregation, GRU state update and graph-level readout.

[0086] S5. A multi-agent collaborative decision-making mechanism based on QMIX is adopted to nonlinearly aggregate the local values ​​of each MEO cluster head into the global total value. Monotonicity constraints are used to ensure local optimal decision-making and promote the improvement of global QoS performance.

[0087] S6. The cluster head performs intra-cluster and inter-cluster routing decisions based on the candidate path Q value, generates the end-to-end optimal routing path and sends it to the LEO satellite, and completes packet forwarding and bandwidth resource reservation.

[0088] In this embodiment, by combining the LEO-MEO two-layer clustering architecture with deep reinforcement learning, distributed routing management and global QoS optimization of large-scale satellite networks are realized. This enables adaptive matching of different service quality requirements under high-speed dynamic satellite topology, thereby improving network throughput, reducing transmission latency, and minimizing routing conflicts.

[0089] like Figure 3 As shown, in the above embodiment, preferably, the LEO-MEO hierarchical clustering management architecture described in step S1 is constructed as follows:

[0090] S11. Divide the global Earth's surface space into a fixed logical grid and abstract it into virtual nodes. LEO satellites are mapped to the corresponding virtual nodes according to their latitude and longitude coordinates.

[0091] S12. Based on the orbital direction and logical adjacency, LEO satellites are divided into several clusters. Each cluster contains a preset number of orbits and a preset number of logically adjacent LEO satellites per orbit. MEO satellites establish connections with LEO satellites within the cluster through inter-layer links to realize the collection of status information and the issuance of commands within the cluster.

[0092] In this embodiment, the use of logical grid mapping and orbital adjacency clustering can reduce the impact of frequent topology switching caused by satellite dynamic motion and improve the stability of the cluster structure. With MEO satellites uniformly serving as cluster heads, the advantages of wide coverage and strong processing capabilities of medium-orbit satellites can be fully utilized to achieve efficient centralized control and lightweight distributed execution of the LEO layer.

[0093] In any of the above embodiments, preferably, the modeling rules for the link QoS indicator model in step S2 are as follows:

[0094] Link delay is the sum of propagation delay, transmission delay, queuing delay based on the M / M / 1 / m queuing model, and processing delay. The total path delay is the sum of the delays of all links on the path.

[0095] The remaining bandwidth of a link is the difference between the total capacity of the link and the bandwidth already used. The remaining bandwidth of a path is the minimum value of the remaining bandwidth of all links on the path.

[0096] The link packet loss rate is the probability of traffic overload blocking based on the M / M / 1 / m model, while the end-to-end packet loss rate of the path is the product of the failure forwarding probabilities of each hop on the path.

[0097] In this embodiment, link QoS indicators are constructed based on the M / M / 1 / m queuing model, which can accurately characterize the real transmission characteristics of satellite network congestion, bandwidth contention and packet loss, providing an objective and quantitative basis for subsequent routing utility evaluation and reinforcement learning reward design, and ensuring that routing decisions are consistent with the actual network operation status.

[0098] In any of the above embodiments, preferably, the calculation rule for the path QoS utility model in step S2 is as follows:

[0099] Time delay effect The theoretical shortest path delay Total delay of the current path The ratio, expressed as: ;

[0100] Bandwidth utility Remaining bandwidth for path bottleneck links with initial link capacity The ratio, expressed as: ;

[0101] Packet loss rate utility To avoid all possible packet loss nodes on the path The rate, expressed as: ;

[0102] Path comprehensive utility The calculation formula is: ,in For time delay effect, For bandwidth efficiency, For packet loss rate utility, , , These are the corresponding weighting factors for latency utility, bandwidth utility, and packet loss rate utility, respectively, satisfying... Furthermore, the weighting factor is dynamically assigned based on the sensitivity of the QoS indicators for the service type.

[0103] In this embodiment, a comprehensive QoS evaluation model is constructed by combining a normalized utility function with a weighted linear combination, which can achieve differentiated adaptation to the QoS preferences of different services. The dynamically adjustable weight mechanism can flexibly switch optimization targets between latency-sensitive, bandwidth-sensitive, and high-reliability services, significantly improving the routing strategy's ability to support multiple types of mixed services.

[0104] In any of the above embodiments, preferably, the quintuple of the multi-agent partially observable Markov decision process described in step S3 is... Defined as:

[0105] The state space S is the set of feature vectors for all links. These feature vectors include link queuing length, remaining bandwidth, packet loss rate, and latency. A single cluster head only obtains local state observations within its managed cluster, expressed as: ,in, The data for the link represents the queue length, reflecting the degree of link congestion; This represents the remaining bandwidth of the link, reflecting its carrying capacity. The packet loss rate of the link reflects the reliability of transmission; For link latency;

[0106] Action space A is a set of candidate paths from the source node to the destination node. A path library is generated by pre-training the K-shortest path algorithm with time delay constraints. The cluster head selects the optimal path from the path library.

[0107] The transition probability P is the probability that the network topology will evolve from the current congestion state to the state at the next moment after the routing decision is made, reflecting the dynamic changes of satellite high-speed movement and service traffic.

[0108] The reward function R is a reward based on the QoS utility function, and the reward value is related to the overall path utility. Positive correlation; if a data packet fails to reach its destination satellite due to congestion or link failure, a fixed penalty is imposed. The calculation formula is:

[0109] ,in, As a reward value, The basic reward constant is a fixed positive reward given when a data packet successfully reaches its destination satellite. It is used to incentivize cluster heads to complete the basic data packet delivery task and ensure basic route reachability. This is a path-wide utility scaling factor used to adjust the ratio of the QoS reward component to the base reward.

[0110] Discount factor The weighting coefficient, which balances the current routing benefits with the future network congestion, ranges from 0 to 1.

[0111] In this embodiment, dynamic clustering routing is modeled as a multi-agent partially observable Markov decision process, which can adapt to the characteristics of satellite networks, such as local observability, distributed decision-making, and time-varying topology. The reward function takes into account both packet reachability and QoS comprehensive utility, ensuring basic routing connectivity while guiding agents to continuously optimize towards higher QoS, thereby improving learning convergence and decision rationality.

[0112] like Figure 4 As shown, in any of the above embodiments, preferably, the processing flow of the GNN-GRU-based topology feature extraction module in step S4 is as follows:

[0113] S41. Spatial Feature Aggregation: Execute L rounds of message passing based on links. In the initial phase, when l=0, each link... Hidden state From state space Link feature vector Obtained through mapping via a fully connected layer;

[0114] During the message generation phase, the link From its set of adjacent links In the The next received aggregation message is:

[0115]

[0116] in, and For link and its adjacent links In the The hidden state of the next step. For the message generation function, this paper adopts a weighted aggregation based on an attention mechanism, defined as:

[0117]

[0118] in, This indicates that the feature vectors of the two links are concatenated. It is a single-layer fully connected network. Attention weights;

[0119] S42, GRU State Update: After each round of message passing, the hidden state of the link is updated through the GRU's update gate and reset gate mechanism, i.e., the link's hidden state. In the The hidden state after the next update is:

[0120] ,in, For link In the The next received aggregate message.

[0121] S43. Graph-level reading: First, sum and aggregate all link features within the same subgraph to obtain the subgraph-level representation vector. : ,in, This represents the set of all links within the current subgraph. It is a link go through The final hidden state after layer GRU update. Then, the subgraph-level representation vector. Input graph-level readout module. This module is a three-layer fully connected network using the SELU activation function, and Dropout regularization is introduced between layers to prevent overfitting. Through the non-linear mapping of this network, the Q-value of the current candidate path is finally output. : ,in, , , This is the weight matrix. , , For bias terms; The SELU activation function is used, and Dropout regularization is inserted between layers to prevent overfitting. Final output. As a Q-value estimate of the current state-action pair.

[0122] In this embodiment, a hybrid GNN-GRU structure is used for topology feature extraction. GNN attention aggregation can effectively capture the spatial correlation and dependency of inter-satellite links, while GRU time-series updates can adapt to the dynamic time-varying characteristics of satellite networks, achieving efficient encoding and representation learning of dynamic topology and link states, and significantly improving the accuracy of Q-value estimation and the rationality of routing decisions.

[0123] In any of the above embodiments, preferably, the implementation rules of the QMIX-based multi-agent collaborative decision-making mechanism in step S5 are as follows:

[0124] Employing a distributed decision-making and centralized training model, the local value Q of the i-th MEO cluster head is integrated through a QMIX hybrid network. i Nonlinear aggregation for global total value Q tot The aggregation formula is:

[0125] in, For the first Each MEO cluster head is based on a local observation sequence. and actions Calculated local value; Nonlinear aggregation functions characterized by hybrid networks; The network state is used as input to the supernetwork to adjust the aggregate weights;

[0126] The weights W of the hybrid network are dynamically generated by the supernetwork and remain non-negative. Through monotonicity constraints Ensuring that the local optimal decisions of each cluster head can promote the overall improvement of QoS performance of the entire network and avoid routing decision oscillations and resource conflicts in the cluster boundary area.

[0127] In this embodiment, multi-cluster head collaborative routing decision-making is implemented based on QMIX. The globally optimal strategy can be learned during the centralized training phase, and each cluster head can make independent and rapid decisions during the distributed execution phase. Monotonicity constraints ensure that local decisions are consistent with the global goal, effectively solving the problems of multi-agent decision-making conflicts, routing oscillations and cross-cluster resource competition, and improving the overall routing performance of large-scale networks.

[0128] like Figure 5 As shown, in any of the above embodiments, preferably, the specific process of the intra-cluster routing decision in step S6 is as follows:

[0129] S61. After the source LEO node in the cluster generates a service request, it will report the service request message containing the source node ID, destination node ID and bandwidth requirements to the MEO cluster head of this cluster.

[0130] S62. The MEO cluster head determines whether the destination node is located within its cluster. If so, it queries the pre-computed path table for K candidate paths between the source and destination nodes. ;

[0131] S63. For each candidate path Simulated bandwidth requirements Assignment, generating hypothetical states And extract the link state matrix ,in, For the real number space, d represents the total number of links. f For each link, the link state matrix includes remaining bandwidth, bandwidth allocation status, propagation delay, and end-to-end delay as a characteristic dimension.

[0132] S64. Input the link state matrix into the GNN-DQN model and output the Q-value of each candidate path. The candidate path with the largest Q value is selected as the optimal routing path. ;

[0133] The S65 and MEO cluster heads distribute the optimal routing path to the LEO nodes along the path. Each node performs hop-by-hop forwarding according to the path and reserves the corresponding bandwidth resources along the path.

[0134] In this embodiment, intra-cluster routing decisions are made by pre-generating candidate paths and selecting the best Q-value, which reduces online computational complexity, meets satellite real-time requirements, and achieves QoS-oriented path optimization through reinforcement learning. Combined with a bandwidth reservation mechanism, it can ensure the stability of service transmission and reduce congestion and retransmission.

[0135] like Figure 6 As shown, in any of the above embodiments, preferably, the specific process of the inter-cluster routing decision in step S6 is as follows:

[0136] S66. After the source cluster MEO cluster head determines that the destination node is located in another cluster, it determines the destination cluster to which the destination node belongs, and generates a cluster sequence on the cluster-level topology graph using the shortest path algorithm. ;

[0137] S67. Determine adjacent clusters sequentially according to the cluster sequence. and For cross-cluster boundary node pairs, select the inter-cluster inter-satellite link with the closest geographical distance and sufficient remaining bandwidth to determine the exit node. and entry node ;

[0138] S68. The MEO clusters of each transit cluster, with the ingress and egress nodes as source-destination nodes, execute the intra-cluster routing decision process of S61-S65 in parallel to generate the optimal sub-path for each cluster. ;

[0139] S69. Concatenate the optimal sub-path of each cluster with the cross-cluster link in sequence to form an end-to-end globally optimal routing path. ;

[0140] S610 and the source cluster MEO cluster head distribute the globally optimal routing path to all LEO nodes along the path. Each node performs hop-by-hop forwarding and reserves the corresponding bandwidth resources along the path.

[0141] In this embodiment, a hierarchical inter-cluster routing mechanism of "cluster-level planning + segmented optimization + splicing into end-to-end path" is adopted to decompose the large-scale global routing problem into multiple small-scale intra-cluster sub-problems, which greatly reduces the computation and signaling overhead; parallel decision-making in each cluster improves the route establishment speed, and the optimal selection of cross-cluster links further guarantees end-to-end QoS performance.

[0142] like Figure 7 As shown, in any of the above embodiments, preferably, the method further includes model training and performance evaluation steps:

[0143] S7. Construct a GNN-DQN routing decision model, obtain training experience by randomly generating business requests and store it in a public experience replay buffer, and update the network parameters using the square of the time difference error as the loss function:

[0144] ,in, , Parameters for the current network and the target network, respectively.

[0145] S8. Each MEO cluster head agent adopts a centralized training and distributed execution paradigm. During the training phase, network parameters and experience buffers are shared, and training is unified with end-to-end routing performance as the joint reward signal. During the inference phase, routing decisions are executed independently based on the local state of the cluster.

[0146] S9. By comparing the average path latency, average minimum remaining bandwidth of the link, and path packet loss rate under different network load rates, the algorithm performance is evaluated and optimized.

[0147] In this embodiment, a centralized training, distributed execution paradigm, and experience replay mechanism are adopted to improve model training efficiency, stability, and sample utilization. Performance evaluation is conducted through multi-dimensional indicators, which can verify the algorithm's comprehensive optimization effect on latency, bandwidth, and reliability under different network loads, ensuring that the method has engineering applicability and robustness in actual satellite network scenarios. Specific Implementation

[0148] This invention proposes a clustered QoS routing method for large-scale satellite networks based on deep reinforcement learning. The specific implementation process includes the following steps:

[0149] S1, Constellation Network Initialization

[0150] A WalkerDelta configuration is used to construct the LEO satellite constellation, comprising 15 orbits with 15 satellites evenly distributed across each orbit, for a total of 225 LEO satellites. The orbital altitude is set at 780 km, and the orbital inclination is set at 55°. Each LEO satellite establishes intra-orbit inter-satellite links with adjacent satellites in the same orbit, and inter-orbit inter-satellite links with in-phase satellites in adjacent orbits to its left and right, with each satellite having 4 inter-satellite links. The initial capacity of each inter-satellite link is set at 200 Gbps. Simultaneously, 9 MEO satellites are deployed above the corresponding orbits as cluster controllers, with an orbital altitude of 8000 km.

[0151] S2, Clustering and Control Architecture Establishment

[0152] The 225 LEO satellites were divided into 9 equal-sized clusters based on their orbital and phase numbers, with each cluster containing 5 × 5 = 25 LEO satellites. The specific division method was as follows: cluster number... ,in For track numbering, Each cluster is assigned a phase number. Each cluster is managed by a corresponding MEO satellite. The MEO satellite collects the status information of each link within the cluster through the MEO-LEO interlayer link and is responsible for the routing decision of that cluster.

[0153] Cluster topology modeling is a directed graph ,in For 25 satellite nodes, This is the set of inter-satellite links within the cluster. For each link... Maintain the following status information: data represents queue length, current remaining bandwidth, bandwidth allocation flag, propagation delay, end-to-end delay, and packet loss rate.

[0154] S3, Candidate Path Pre-calculation

[0155] For each source-destination node pair in the cluster topology graph The K-shortest path algorithm is used for pre-computation. Candidate paths are stored in a path table. In addition, the shortest paths between cluster pairs are pre-calculated for the cluster-level topology, which is used for cluster sequence planning for inter-domain routing.

[0156] S4 and GNN-DQN Model Construction and Training

[0157] Construct a routing decision model that combines GNN and DQN.

[0158] The training process is as follows: In each training round, 100 service requests are randomly generated. Each service request includes a randomly selected source node, destination node, and bandwidth requirement. For each service request, the agent executes the intra-domain routing decision process described in step S5 and obtains an immediate reward. Later experience The data is stored in a shared experience replay buffer. At each training step, a batch of experiences is randomly sampled from the buffer, and the GNN-DQN network parameters are updated using the square of the temporal difference error as the loss function.

[0159]

[0160] in, , These are the parameters for the current network and the target network, respectively. The training process consists of 150 rounds, with 100 service requests per round.

[0161] Each MEO agent adopts a centralized training and distributed execution paradigm: during the training phase, each agent shares network parameters and a common experience replay buffer, and uses end-to-end routing performance as a joint reward signal for unified training; during the inference phase, each MEO controller independently executes decisions based only on the local state of its own cluster.

[0162] S5, Intra-domain Routing Decision

[0163] When a business request is generated within the LEO cluster, the following steps are executed:

[0164] (a) The source node sends the service request message (source node ID, destination node ID, bandwidth requirement) Report to the cluster's MEO controller;

[0165] (b) The MEO controller determines whether the destination node is within the cluster. If so, it executes intra-domain routing steps (c)-(f); otherwise, it proceeds to S6 to execute inter-domain routing.

[0166] (c) Query the path from the source node to the destination node from the path table. Candidate paths ;

[0167] (d) For each candidate path Simulate bandwidth requirements on a replica of the current network state. The allocation;

[0168] (e) will The link-state matrices corresponding to each candidate path are input into the GNN-DQN model in batches. After four rounds of message passing, attention-weighted aggregation, GRU state update, and graph-level readout, the model outputs the Q-value of each candidate path. ;

[0169] (f) Select the path with the largest Q value The MEO controller distributes the path to each LEO node along the route. Each node performs hop-by-hop forwarding and deducts the corresponding bandwidth resources along the path, thus completing the intra-domain routing.

[0170] S6, Inter-domain routing decision

[0171] When the destination node is not in the source cluster, perform the following steps:

[0172] (g) The source cluster MEO controller determines the destination cluster to which the destination node belongs and generates a cluster sequence on the cluster-level topology graph (3×3 grid) using the shortest path algorithm. ;

[0173] (h) Based on the cluster sequence, determine the cross-domain boundary node pairs between adjacent clusters in sequence: for each pair of adjacent clusters, select the cross-domain inter-satellite link with the closest geographical distance and sufficient remaining bandwidth to determine the exit node. and entry node ;

[0174] (i) The MEO controllers of each traversing cluster use the assigned ingress and egress nodes as the source and destination nodes, respectively, and execute the intra-domain routing decision process described in steps (c)-(f) to obtain the optimal sub-path for this cluster. The source cluster starts from the business source node and ends at the exit node; the intermediate cluster starts from the entry node and ends at the exit node; and the destination cluster starts from the entry node and ends at the business destination node. Decisions for each cluster can be executed in parallel.

[0175] (j) Concatenate the sub-paths of each cluster and the cross-domain link in sequence to form a complete end-to-end path. ;

[0176] (k) The source cluster MEO controller distributes the complete path to each LEO node along the path to perform hop-by-hop forwarding, and reserves the corresponding bandwidth resources along the path, thus completing the inter-domain routing.

[0177] S7, Performance Evaluation

[0178] like Figure 8 This invention considers the impact of network load congestion on queuing latency and compares the end-to-end latency performance of the traditional OSPF protocol, the basic DQN algorithm, and the improved DQN algorithm proposed in this invention under different network load conditions. Experimental results show that in the low load phase (0-30%), the latency performance of the three algorithms is basically equivalent, all maintaining a low level of 10-15ms. However, as the load rate increases, when entering the medium load phase (40-70%), the OSPF protocol, lacking dynamic awareness of actual link load, suffers from uneven network traffic distribution due to its shortest path selection mechanism based on static hop count, resulting in a significant increase in latency to 38ms. In contrast, the DQN algorithm and the improved DQN algorithm, with congestion awareness, can effectively control latency within 18ms and 28ms respectively through intelligent path selection. However, with increasing load and entering the high load phase, OSPF, lacking awareness of actual load, experiences a significant increase in latency, while DQN and the proposed improved algorithm, with their congestion awareness capabilities, maintain lower latency. In high-load scenarios with a load rate of 80%-90%, the shortcomings of the OSPF protocol become even more apparent. Due to excessive concentration of traffic on a few shortest paths, severe congestion bottlenecks are formed, and latency spikes dramatically to 60.2ms. The improved DQN algorithm, with its enhanced reward function design, accurately estimates the Q-value of paths to dynamically identify and avoid congested links, stabilizing latency at 17.5ms at 90% load, representing a 23.9% performance improvement compared to the basic DQN algorithm's 28ms.

[0179] like Figure 9 This study compares the average link utilization of three algorithms under different load rates. The average link utilization is defined as the arithmetic mean of the ratio of the actual load carried by all links to their capacity. This indicator reflects the overall efficiency of the routing algorithm in utilizing network link resources. That is, under the same traffic demand, a lower average utilization indicates a more uniform distribution of traffic across the network and more efficient network resource utilization. Experimental results show that in the low-load stage (0-30%), network resources are abundant, and the impact of routing strategies on load distribution is limited; the average link utilization of the three algorithms is basically the same. As the load rate increases, the performance gap between the three algorithms gradually widens. In the medium-load range (40%-70%), the proposed improved DQN algorithm begins to show advantages, with its average utilization rate decreasing by about 1-3 percentage points compared to OSPF and by about 1-2 percentage points compared to the basic DQN algorithm. In the high-load range (80%-90%), the performance gap further widens. When the load rate is 90%, the average link utilization of the improved DQN algorithm is about 25.2%, which is about 20.0% and 10.0% lower than OSPF and the basic DQN algorithm, respectively.

[0180] This invention divides the space into 25 clusters of 5×5. Each cluster is assigned a different load rate based on the population density of the region, and the load rates are in a multiple relationship: the cluster with a load rate code of 1 corresponds to the basic load level and is distributed in relatively sparsely populated areas; the cluster with a load rate code of 2 has a load rate twice that of the cluster with a load rate code of 1 and covers areas with medium population density; the cluster with a load rate code of 3 has a load rate three times that of the cluster with a load rate code of 1 and is concentrated in areas with high population density such as Asia and Africa.

[0181] like Figure 10 As shown, with the increase in packet arrival rate, the packet loss rate of all three schemes shows an upward trend, but the algorithm proposed in this invention maintains the lowest packet loss rate throughout the entire load range. Compared with the traditional static routing algorithm OSPF, the two reinforcement learning-based schemes exhibit superior performance. Compared with the DQN algorithm, the algorithm proposed in this project can promptly identify potential congested nodes within the local topology range and proactively implement traffic rerouting to avoid buffer overflows, thereby effectively reducing packet loss caused by sudden traffic surges.

[0182] Based on the above, Figures 2 to 7 Correspondingly, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the large-scale satellite network cluster QoS routing method based on deep reinforcement learning of any of the above embodiments.

[0183] Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of this application.

[0184] Based on the above, Figures 2 to 7 To achieve the above objectives, this application also provides a computer device, including a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to implement the steps of the deep reinforcement learning-based large-scale satellite network cluster QoS routing method of any of the above embodiments.

[0185] Optionally, the computer device may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB ports, card reader ports, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Bluetooth interfaces, Wi-Fi interfaces), etc.

[0186] Those skilled in the art will understand that the computer device structure provided in this embodiment does not constitute a limitation on the computer device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0187] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages and stores the hardware and software resources of a computer device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software within the physical device.

[0188] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A large-scale satellite network clustering QoS routing method based on deep reinforcement learning, characterized in that, Applied to a two-layer constellation satellite network consisting of low-Earth orbit (LEO) satellites and medium-Earth orbit (MEO) satellites, the method includes the following steps: S1. Construct a hierarchical and clustered management architecture for LEO-MEO, dividing the LEO satellite network into several clusters. Each cluster is headed by a corresponding MEO satellite, which is responsible for collecting link status information and making routing decisions within the cluster. S2. Satellite network services are classified into three categories according to QoS requirements: latency-sensitive, bandwidth-sensitive, and high-reliability. A link QoS indicator model including latency, remaining bandwidth, and packet loss rate is established. At the same time, a path QoS utility model is constructed, which includes latency utility, bandwidth utility, packet loss rate utility, and a path comprehensive utility obtained by a weighted linear combination of the three. S3. The clustered routing problem is modeled as a multi-agent partially observable Markov decision process. A quintuple consisting of state space, action space, transition probability, reward function and discount factor is defined, where the reward function is positively correlated with the overall path utility and a fixed penalty term is given when the data packet transmission fails. S4. The cluster head processes the intra-cluster link state information through the GNN-GRU-based topology feature extraction module, and outputs the candidate path Q value after spatial feature aggregation, GRU state update and graph-level readout. S5. A multi-agent collaborative decision-making mechanism based on QMIX is adopted to nonlinearly aggregate the local values ​​of each MEO cluster head into the global total value. Monotonicity constraints are used to ensure local optimal decision-making and promote the improvement of global QoS performance. S6. The cluster head performs intra-cluster and inter-cluster routing decisions based on the candidate path Q value, generates the end-to-end optimal routing path and sends it to the LEO satellite, and completes packet forwarding and bandwidth resource reservation.

2. The method of claim 1, wherein, The construction method of the LEO-MEO hierarchical cluster management architecture described in step S1 is as follows: S11. Divide the global Earth's surface space into a fixed logical grid and abstract it into virtual nodes. LEO satellites are mapped to the corresponding virtual nodes according to their latitude and longitude coordinates. S12. Based on the orbital direction and logical adjacency, LEO satellites are divided into several clusters. Each cluster contains a preset number of orbits and a preset number of logically adjacent LEO satellites per orbit. MEO satellites establish connections with LEO satellites within the cluster through inter-layer links to realize the collection of status information and the issuance of commands within the cluster.

3. The large-scale satellite network clustered QoS routing method according to claim 1, characterized in that, The modeling rules for the link QoS indicator model described in step S2 are as follows: Link delay is the sum of propagation delay, transmission delay, queuing delay based on the M / M / 1 / m queuing model, and processing delay. The total path delay is the sum of the delays of all links on the path. The remaining bandwidth of a link is the difference between the total capacity of the link and the bandwidth already used. The remaining bandwidth of a path is the minimum value of the remaining bandwidth of all links on the path. The link packet loss rate is the probability of traffic overload blocking based on the M / M / 1 / m model, while the end-to-end packet loss rate of the path is the product of the failure forwarding probabilities of each hop on the path.

4. The large-scale satellite network clustered QoS routing method according to claim 1, characterized in that, The calculation rules for the path QoS utility model in step S2 are as follows: Time delay effect The theoretical shortest path delay Total delay of the current path The ratio, expressed as: ; Bandwidth utility Remaining bandwidth for path bottleneck links with initial link capacity The ratio, expressed as: ; Packet loss rate utility To avoid all possible packet loss nodes on the path The rate, expressed as: ; Path comprehensive utility The calculation formula is: ,in For time delay effect, For bandwidth efficiency, For packet loss rate utility, , , These are the corresponding weighting factors for latency utility, bandwidth utility, and packet loss rate utility, respectively, satisfying... Furthermore, the weighting factor is dynamically assigned based on the sensitivity of the QoS indicators for the service type.

5. The large-scale satellite network clustered QoS routing method according to claim 1, characterized in that, The quintuple of the multi-agent partially observable Markov decision process described in step S3 Defined as: The state space S is the set of feature vectors for all links. These feature vectors include link queuing length, remaining bandwidth, packet loss rate, and latency. A single cluster head only obtains local state observations within its managed cluster, expressed as: ,in, The data for the link represents the queue length, reflecting the degree of link congestion; This represents the remaining bandwidth of the link, reflecting its carrying capacity. The packet loss rate of the link reflects the reliability of transmission; For link latency; Action space A is a set of candidate paths from the source node to the destination node. A path library is generated by pre-training the K-shortest path algorithm with time delay constraints. The cluster head selects the optimal path from the path library. The transition probability P is the probability that the network topology will evolve from the current congestion state to the state at the next moment after the routing decision is made, reflecting the dynamic changes of satellite high-speed movement and service traffic. The reward function R is a reward based on the QoS utility function, and the reward value is related to the overall path utility. Positive correlation; if a data packet fails to reach its destination satellite due to congestion or link failure, a fixed penalty is imposed. The calculation formula is: ,in, As a reward value, The basic reward constant is a fixed positive reward given when a data packet successfully reaches its destination satellite. It is used to incentivize cluster heads to complete the basic data packet delivery task and ensure basic route reachability. This is a path-wide utility scaling factor used to adjust the ratio of the QoS reward component to the base reward. Discount factor The weighting coefficient, which balances the current routing benefits with the future network congestion, ranges from 0 to 1.

6. The large-scale satellite network clustered QoS routing method according to claim 1, characterized in that, The processing flow of the GNN-GRU-based topology feature extraction module in step S4 is as follows: S41. Spatial Feature Aggregation: Execute L rounds of message passing based on links. In the initial phase, when l=0, each link... Hidden state From state space Link feature vector Obtained through mapping via a fully connected layer; During the message generation phase, the link From its set of adjacent links In the The next received aggregation message is: in, and For link and its adjacent links In the The hidden state of the next step. For the message generation function, this paper adopts a weighted aggregation based on an attention mechanism, defined as: in, This indicates that the feature vectors of the two links are concatenated. It is a single-layer fully connected network. Attention weights; S42, GRU State Update: After each round of message passing, the hidden state of the link is updated through the GRU's update gate and reset gate mechanism, i.e., the link's hidden state. In the The hidden state after the next update is: ,in, For link In the The next received aggregate message; S43. Graph-level reading: First, sum and aggregate all link features within the same subgraph to obtain the subgraph-level representation vector. : ,in, This represents the set of all links within the current subgraph. It is a link go through The final hidden state after the GRU update; through the nonlinear mapping of this network, the Q-value of the current candidate path is finally output. : ,in, , , This is the weight matrix. , , For bias terms; The SELU activation function is used, and Dropout regularization is inserted between layers to prevent overfitting. The final output is... As a Q-value estimate of the current state-action pair.

7. The large-scale satellite network clustered QoS routing method according to claim 1, characterized in that, The implementation rules of the QMIX-based multi-agent cooperative decision-making mechanism described in step S5 are as follows: Employing a distributed decision-making and centralized training model, the local value Q of the i-th MEO cluster head is integrated through a QMIX hybrid network. i Nonlinear aggregation for global total value Q tot The aggregation formula is: in, For the first Each MEO cluster head is based on a local observation sequence. and actions Calculated local value; Nonlinear aggregation functions characterized by hybrid networks; The network state is used as input to the supernetwork to adjust the aggregate weights; The weights W of the hybrid network are dynamically generated by the supernetwork and remain non-negative. Through monotonicity constraints Ensuring that the local optimal decisions of each cluster head can promote the overall improvement of QoS performance of the entire network and avoid routing decision oscillations and resource conflicts in the cluster boundary area.

8. The large-scale satellite network clustered QoS routing method according to claim 1, characterized in that, The specific process of intra-cluster routing decision-making in step S6 is as follows: S61. After the source LEO node in the cluster generates a service request, it will report the service request message containing the source node ID, destination node ID and bandwidth requirements to the MEO cluster head of this cluster. S62. The MEO cluster head determines whether the destination node is located within its cluster. If so, it queries the pre-computed path table for K candidate paths between the source and destination nodes. ; S63. For each candidate path Simulated bandwidth requirements Assignment, generating hypothetical states And extract the link state matrix ,in, For the real number space, d represents the total number of links. f For each link, the link state matrix includes remaining bandwidth, bandwidth allocation status, propagation delay, and end-to-end delay as a characteristic dimension. S64. Input the link state matrix into the GNN-DQN model and output the Q-value of each candidate path. The candidate path with the largest Q value is selected as the optimal routing path. ; The S65 and MEO cluster heads distribute the optimal routing path to the LEO nodes along the path. Each node performs hop-by-hop forwarding according to the path and reserves the corresponding bandwidth resources along the path.

9. The large-scale satellite network clustered QoS routing method according to claim 1, characterized in that, The specific process of inter-cluster routing decision-making in step S6 is as follows: S66. After the source cluster MEO cluster head determines that the destination node is located in another cluster, it determines the destination cluster to which the destination node belongs, and generates a cluster sequence on the cluster-level topology graph using the shortest path algorithm. ; S67. Determine adjacent clusters sequentially according to the cluster sequence. and For cross-cluster boundary node pairs, select the inter-cluster inter-satellite link with the closest geographical distance and sufficient remaining bandwidth to determine the exit node. and entry node ; S68. The MEO clusters of each transit cluster, with the ingress and egress nodes as source-destination nodes, execute the intra-cluster routing decision process of S61-S65 in parallel to generate the optimal sub-path for each cluster. ; S69. Concatenate the optimal sub-path of each cluster with the cross-cluster link in sequence to form an end-to-end globally optimal routing path. ; S610 and the source cluster MEO cluster head distribute the globally optimal routing path to all LEO nodes along the path. Each node performs hop-by-hop forwarding and reserves the corresponding bandwidth resources along the path.

10. The large-scale satellite network clustered QoS routing method according to any one of claims 1-9, characterized in that, The method also includes model training and performance evaluation steps: S7. Construct a GNN-DQN routing decision model, obtain training experience by randomly generating business requests and store it in a public experience replay buffer, and update the network parameters using the square of the time difference error as the loss function: ,in, , Parameters for the current network and the target network, respectively. S8. Each MEO cluster head agent adopts a centralized training and distributed execution paradigm. During the training phase, network parameters and experience buffers are shared, and training is unified with end-to-end routing performance as the joint reward signal. During the inference phase, routing decisions are executed independently based on the local state of the cluster. S9. By comparing the average path latency, average minimum remaining bandwidth of the link, and path packet loss rate under different network load rates, the algorithm performance is evaluated and optimized.

Images