A low-orbit satellite network diversified quality of service constraint routing algorithm based on deep reinforcement learning

Abstract

The application discloses a kind of low-orbit satellite network diversification quality of service constraint routing algorithm based on deep reinforcement learning.The method uses an improved multi-agent deep Q learning framework MSDIR, and fuses multi-service classification mechanism and dynamic weighted reward function, significantly improves the adaptability of network to different service type quality of service demand and resource allocation efficiency.In the MSDIR framework, a composite state space including link state, node load and service type is constructed, and end-to-end delay, packet loss rate and throughput are optimized comprehensively.Through the collaborative learning mechanism of "distributed sensing, centralized training and distributed execution", efficient autonomous routing decision of satellite node is realized.The trained routing decision model is deployed to each satellite node, and the network completes the differentiated routing and forwarding of delay-sensitive, high-bandwidth guarantee and best-effort services through local state sensing and distributed decision.The experimental results show that the method is significantly better than traditional satellite routing algorithm in convergence speed, delay control, packet loss rate and throughput.

Description

Technical Field

[0001] This invention belongs to the field of low-Earth orbit satellite network communication technology, and specifically relates to an intelligent routing method based on deep reinforcement learning, which can effectively solve the distributed routing decision optimization problem under the constraints of highly dynamic changes in satellite network topology and diversified service quality. Background Technology

[0002] Low Earth Orbit (LEO) satellite networks, with their wide-area coverage, low transmission latency, and strong resilience, have become the core infrastructure for building next-generation seamless global information networks, demonstrating enormous potential in areas such as broadband internet access, the Internet of Things (IoT), emergency communications, and military communications. With the accelerated construction of global LEO constellations and the deployment and operation of large-scale constellations such as Starlink and OneWeb, the scale and complexity of satellite networks are growing exponentially. These constellations typically consist of hundreds to thousands of satellites, forming a three-dimensional dynamic network topology through dense inter-satellite links, expanding their coverage from traditional regional services to truly global services. However, the periodicity and high dynamic changes in network topology caused by their high-speed motion (typical orbital period of approximately 90-120 minutes, with continuously changing relative satellite positions), coupled with the constraints of limited onboard processing power, inter-satellite link bandwidth, and power resources, make the routing problem of providing differentiated Quality of Service (QoS) guarantees for diversified services extremely challenging. In particular, with the development of 5G / 6G integrated terrestrial and satellite networks, satellite networks need to carry diverse traffic ranging from ultra-low latency real-time interactive services to massive data distribution, which places unprecedented demands on the adaptability, intelligence and reliability of routing algorithms.

[0003] Traditional satellite network routing algorithms are mainly divided into two categories: virtual topology-based and virtual node-based methods. Virtual topology-based methods discretize the continuously changing satellite network into a series of time slices, where the network topology is treated as static, allowing the application of classic terrestrial routing algorithms. While this method has a clear structure, route updates have inherent delays, making it unable to respond promptly to instantaneous state changes such as link congestion and sudden failures. Furthermore, it can cause routing oscillations and packet loss during topology switching. Virtual node-based schemes bind geographical areas to satellites, sending packets to virtual nodes instead of specific satellites, which then forward them. This method simplifies addressing but has limited capabilities in dynamic load balancing and service quality differentiation, making it difficult to adapt to the uneven spatiotemporal distribution of service traffic. Faced with mixed service flows characterized by latency sensitivity, high bandwidth demands, and best-effort behavior, existing methods often differentiate services through static configuration or simple priority queues, lacking the ability to make adaptive and refined routing decisions based on real-time network status and service QoS requirements. This can easily lead to a contradiction between substandard performance of critical services and excessive resource contention by non-critical services, resulting in overall low network resource utilization efficiency.

[0004] In recent years, Deep Reinforcement Learning (DRL) has provided a new approach to solving such dynamic optimization problems. Existing research has attempted to apply DRL to satellite network routing, but most current solutions employ centralized control or optimize only for a single QoS objective, making them difficult to adapt to the distributed characteristics of large-scale constellations and unable to effectively handle complex scenarios with multiple service types. While Multi-Agent Reinforcement Learning (MARL) can achieve distributed decision-making, it faces challenges such as training instability and low collaborative efficiency in some observable environments, and lacks a reward function design that can balance multiple objectives such as latency, bandwidth, and packet loss rate.

[0005] Therefore, in the highly dynamic LEO satellite network environment, overcoming the challenges of partial observability, multi-objective conflicts, and distributed coordination, and designing an intelligent routing algorithm that can operate independently at each satellite node, make rapid decisions based on local information, achieve overall coordination through an effective learning mechanism, and accurately meet the strict QoS constraints of diverse services, has become a key technical challenge that urgently needs to be addressed to break through the limitations of traditional methods and improve the overall service capabilities, reliability, and resource efficiency of future satellite networks. This not only demands innovation in the algorithm itself but also requires in-depth exploration in system modeling, training frameworks, and engineering implementation. Summary of the Invention

[0006] This invention aims to solve the intelligent routing decision-making problem in low-Earth orbit satellite networks under scenarios of dynamic topology and diverse services, proposing a distributed quality-of-service (QoS) constrained routing method based on deep reinforcement learning. Addressing the shortcomings of existing research in distributed collaboration and multi-objective optimization, this study constructs a realistic high-dynamic satellite network simulation environment. By designing a deep Q-network architecture based on multi-agent collaboration and a multi-dimensional weighted reward function, the algorithm's convergence stability is ensured while significantly improving its ability to guarantee differentiated QoS for latency-sensitive, high-bandwidth-guaranteed, and best-effort services. To achieve this goal, the technical solution adopted in this invention includes the following steps:

[0007] Step 1: Establish a system model for the low-Earth orbit (LEO) satellite network. The network adopts a Walker Delta constellation configuration, comprising M orbital planes, each with N satellites evenly distributed, forming a set of M×N satellite nodes. Each satellite node acts as an intelligent routing agent, maintaining its inter-satellite links. Inter-satellite links are divided into two types: intra-orbital links connect adjacent satellites within the same orbital plane, with relatively stable link lengths; inter-orbital links connect satellites within adjacent orbital planes, and their lengths and connectivity status change periodically with the relative motion of the satellites, which is the main cause of the network's high dynamics. Based on this, the satellite network is abstracted as a time-varying weighted undirected graph model, where nodes represent satellites and edges represent inter-satellite links. Each edge has dynamic weight attributes such as propagation delay, available bandwidth, and packet loss rate, providing a formal basis for subsequent intelligent routing decisions.

[0008] Step 2: Establish a multi-service classification mechanism to divide network traffic into three categories: latency-sensitive services, requiring strict latency and jitter control; high-bandwidth guaranteed services, requiring stable high throughput; and best-effort services, with high tolerance for service quality variations. Predefine differentiated QoS constraint thresholds for each service type: set a maximum latency threshold for latency-sensitive services, a minimum bandwidth threshold for high-bandwidth services, and a maximum allowable packet loss rate threshold for all service types. These thresholds constitute the core constraint objectives in the routing optimization process, driving the algorithm to achieve differentiated service quality assurance.

[0009] Step 3: Deploy an autonomous decision-making agent on each satellite node. These agents together constitute a distributed multi-agent system. Each agent is independently responsible for processing data packets arriving at its node and performing hop-by-hop forwarding decisions at the data packet level. This sequential routing decision-making process of each agent is formally modeled as a discrete-time Markov Decision Process (MDP). The model is a quadruple M = (S, A, P, R), where S is the state space, which is the set of all possible environmental states that the agent can perceive; A represents the action space, which contains all legal forwarding actions that the agent can perform in any state; P: S × A × S → [0, 1] is the state transition probability function, which describes the probability of the environment transitioning to the next state after taking a certain action in a specific state. This function is essentially a mathematical abstraction of network dynamics, reflecting the environmental uncertainty in the decision-making process. As a reward function, it provides a scalar instant reward signal after the agent performs an action and causes a state transition. This signal comprehensively evaluates the immediate effects of the selected action on meeting service quality, optimizing network performance, etc., and is a key feedback signal that drives the agent to learn and optimize strategies.

[0010] Based on this MDP framework, the goal of each agent is to learn an optimal policy π through continuous interaction with the environment. * S→A: This strategy can map any observed state to an action, maximizing the expected value of the cumulative discounted reward obtained by the agent in the long run. This modeling method decouples the complex, global satellite network routing optimization problem into a series of parallel, locally observed sequential decision problems, laying the theoretical foundation for subsequent applications of deep reinforcement learning algorithms for distributed and adaptive optimization.

[0011] (1) State-space modeling

[0012] The state space S in the Markov decision process is the set of all environmental information and task context on which the agent makes routing decisions; for any satellite node v in the network i The specific state s that the agent observes and utilizes at decision time t. i t It is a multidimensional feature vector, which is carefully designed to comprehensively capture the key dynamic factors affecting routing performance in a compact form. Its structure can be represented as:

[0013]

[0014] Node body state C i Includes the satellite node v where the current agent is located. iThe inherent attributes and real-time pose information of the node; specifically including: the unique identifier number of the node in the global constellation, the sequence number of the node in the orbital plane, and the three-dimensional position coordinates of the node in the inertial coordinate system or geocentric coordinate system obtained through the onboard navigation system; this dimensional information provides a basic spatial positioning reference for routing decisions.

[0015] Task target state T: contains the final destination node v of the data packet to be forwarded. d The relevant information includes: the unique identifier of the destination node and its real-time location coordinates under predictable ephemeris; this information provides global guidance for routing and is the basis for ensuring the eventual reachability of data packets;

[0016] Link Quality Status (B) i,j ,Pl), representing the current node v i With all its valid neighbor nodes v j The instantaneous transmission capability of inter-satellite links; for each hop candidate link e i，j This dimension includes two key metrics: remaining available bandwidth B. i，j The value is calculated as follows:

[0017] B i，j =B total (e i，j )-B used (e i，j (2)

[0018] That is, the total link bandwidth B total (e i,j Subtract the current bandwidth occupied by B used (e i，j The calculation reflects the instantaneous capacity margin of the link.

[0019] The link packet loss rate Pl, estimated based on recent historical transmission statistics, reflects the link's transmission reliability; this dimension is the core basis for evaluating the direct cost of actions; it can be expressed as:

[0020]

[0021] Node load status Q i , representing the current node v i Its own congestion level; specifically, the real-time depth q of the node's forwarding queue. i This refers to the number of data packets waiting to be processed in the buffer or the total amount of data; this state is directly related to queuing latency and is a key factor affecting end-to-end latency and avoiding local congestion.

[0022] Business semantic state ST: identifies the service type to which the current data packet to be decided belongs; it is a discrete classification variable, and its value comes from a predefined set of service classifications, namely latency-sensitive type - class A, high bandwidth guaranteed type - class B, and best-effort type - class C; this state is the instruction that drives the differentiated routing strategy, and it determines the preference weight of the agent when weighing conflicting objectives such as latency, bandwidth, and packet loss rate.

[0023] Since the aforementioned state components originate from different sensors, measurement modules, or protocol fields, their physical meanings, dimensions, and numerical ranges vary. To ensure the numerical stability of the deep neural network input and accelerate training convergence, the state vector... Before being input into the policy network, normalization is required. Each state variable is scaled by its theoretical maximum or minimum value or historical observation statistics, and is finally mapped to a unified, bounded numerical range. Through this process, heterogeneous multi-source observation data is transformed into a high-dimensional feature representation that is comparable and suitable for neural network processing.

[0024] (2) Action space modeling

[0025] In a low-Earth orbit satellite network routing framework based on deep reinforcement learning, the agent deployed on each satellite node needs to select the appropriate next-hop forwarding direction for the data packets it processes; action space Defined as a discrete, finite set of actions, directly mapped to the physical link interface configuration of the satellite node:

[0026]

[0027] This design strictly maps to the typical physical link layout of a low-Earth orbit satellite constellation, containing four basic geometric directional commands: forward along the orbital plane (front), pointing to the next adjacent satellite in the same orbital plane; backward along the orbital plane (back), pointing to the preceding adjacent satellite in the same orbital plane; to the left adjacent orbital plane (left) and to the right adjacent orbital plane (right), pointing to the satellites in the left and right adjacent orbital planes with the closest latitudes, respectively. Considering the inherent high dynamism of satellite network topology, especially when satellites are operating in high-latitude airspace near polar regions, the relative motion between satellites intensifies, causing drastic changes in the distance and angle of cross-orbital plane links, i.e., the left and right directions, and may even be temporarily interrupted due to antenna pointing limits or physical obstruction; therefore, this method introduces a dynamic action space constraint mechanism. The system dynamically determines the stability of the network environment by monitoring the orbital position of satellite nodes or the real-time signaling status of inter-satellite links in real time. When a node enters a preset polar region or unstable region, this mechanism will automatically and forcibly constrain the effective action space of the agent to a certain limit.

[0028]

[0029] This means that data packets are only allowed to be forwarded in the orbital plane; this design enables the routing strategy to adapt to the physical constraints of the underlying network, fundamentally avoiding invalid decisions to select unstable or unavailable links, thereby significantly improving the reliability of data transmission and the environmental adaptability of the entire routing algorithm.

[0030] (3) Reward function modeling

[0031] In a deep reinforcement learning-based routing decision framework, the reward function is the core signaling mechanism driving the agent to learn the optimal strategy through trial and error. Its design directly determines the convergence direction and final performance of the algorithm. This invention designs a multi-objective weighted composite reward function, aiming to provide the agent with a comprehensive feedback that simultaneously optimizes multiple service quality indicators and global path efficiency; it can be expressed as:

[0032]

[0033] The reward function determines the link e selected by the agent at time t. i，j The calculation is performed after forwarding data packets; firstly, normalized instantaneous observations of four key performance indicators are obtained from the network environment: the transmission delay of the selected link can be expressed as:

[0034]

[0035] Remaining available bandwidth B of the selected link i，j This value can be represented by formula (2), and the larger the value, the better; the current packet loss rate Pl of the selected link can be represented by formula (3), and the smaller the value, the better; and the remaining propagation distance L of the data packet from the current node to the target node. i，d This value should also be as small as possible.

[0036] reward function The core is a linear weighted sum of these indicators, where ω1, ω2, ω3, and ω4 are the corresponding weight coefficients, and satisfy the following:

[0037] ω1+ω2+ω3+ω4=1 (8)

[0038] To meet differentiated service needs, the weighting coefficients are dynamically adjustable: for latency-sensitive services, the latency weight ω1 is significantly increased; for high-bandwidth guaranteed services, the bandwidth weight ω2 is significantly increased; and for best-effort services, a balanced or distance-focused weighting configuration can be adopted to guide the agent to learn routing strategies with different focuses for different types of traffic. In addition, the reward function also includes an important boundary penalty term to ensure the basic effectiveness of the algorithm. To prevent packets from being transmitted indefinitely in loops or from having excessively long paths, a maximum allowed number of hops is set. Once the number of hops for a packet exceeds this threshold, the forwarding of the packet is immediately terminated, and the agent is given a large negative constant penalty.

[0039] The core iterative loop of the algorithm begins with an agent on each satellite node. At each discrete decision time t, the agent first observes its local environment, obtains and constructs its current state st, which integrates multi-dimensional information such as nodes, links, queues and service types. Subsequently, the agent inputs its state into its policy network, or online Q network, which outputs the expected long-term value Q of each optional action a in a given state.

[0040] To balance the use of known good strategies with the exploration of potential better paths, the agent employs an ∈-greedy strategy to select actions: choosing the action with the highest current Q-value with probability 1-∈, i.e.:

[0041]

[0042] Among them, the exploration rate ∈ decays over time according to formula (10), ensuring that the environment is fully explored in the early stage of training, and then stabilizes in the later stage by utilizing the learned strategy.

[0043] ∈ t =∈ min +(∈ max -∈ min )e -λt (10)

[0044] The agent executes the selected action a t Afterwards, the environment will undergo a state transition, entering a new state, and will calculate and provide an immediate reward based on the aforementioned multi-objective weighted reward function. The entire interaction experience is represented by a quadruple (s t a t r t s t+1The data are stored in a shared experience replay pool in the form of [data type name missing]. During training, the algorithm periodically samples a small batch of historical experiences from this replay pool to break the temporal correlation between data and improve the stability of learning. For each sampled experience, the algorithm calculates its temporal difference target value. Then, the mean squared error between the predictions of the policy network is calculated as the loss function.

[0045] Loss(θ) = [r t +γmax a Q'(s t+1 a t ;θ′)-Q(s t a t ;θ)] (11)

[0046] The loss is minimized by stochastic gradient descent, and the parameters of the policy network are updated. To further improve stability, the parameters of the target network are not updated at every step, but are synchronized with the parameters of the policy network periodically and slowly. A soft update rule is usually adopted. Finally, when the cumulative reward curve of the training process tends to a stable plateau and the performance of the policy in the test environment reaches a satisfactory level, the training converges. Attached Figure Description

[0047] Figure 1 This is a diagram of a low-Earth orbit satellite network routing scenario designed in this invention;

[0048] Figure 2 This is a block diagram of the deep reinforcement learning method based on the MSDIR algorithm proposed in this invention;

[0049] Figure 3 This is a comparison chart of reward values ​​during the training process under different learning rates. Detailed Implementation

[0050] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0051] This invention aims to solve the intelligent routing decision-making problem in low-Earth orbit satellite networks under scenarios of dynamic topology and diverse services, proposing a distributed quality-of-service (QoS) constrained routing method based on deep reinforcement learning. Addressing the shortcomings of existing research in distributed collaboration and multi-objective optimization, this study constructs a realistic high-dynamic satellite network simulation environment. By designing a deep Q-network architecture based on multi-agent collaboration and a multi-dimensional weighted reward function, the algorithm's convergence stability is ensured while significantly improving its ability to guarantee differentiated QoS for latency-sensitive, high-bandwidth-guaranteed, and best-effort services. To achieve this goal, the technical solution adopted in this invention includes the following steps:

[0052] Step 1: Establish a system model for the low-Earth orbit satellite network. The network adopts a parameterized Walker Delta constellation configuration, which includes M circular orbital planes. N satellite nodes are uniformly deployed in each orbital plane, forming a satellite node set V with a total size of |V| = M × N. The constellation has a consistent orbital inclination and maintains a fixed phase difference between adjacent orbital planes to ensure global coverage and regular changes in topology.

[0053] Each satellite node acts as an intelligent routing agent in the network, dynamically maintaining its inter-satellite links; each inter-satellite link is represented in the topology as an edge e in the edge set E. i，j Connecting satellite node v i With v j The inter-satellite links are specifically divided into two categories: intra-orbital links, which connect two adjacent satellites in the same orbital plane, and their link length remains approximately constant under orbital dynamic constraints; and inter-orbital links, which connect satellites at similar latitudes in adjacent orbital planes, and whose link length and link connectivity change periodically with the relative positions of the satellites, which is the main factor leading to the high dynamism of the network topology.

[0054] Based on the aforementioned constellation configuration and link model, this paper abstracts the satellite network topology into a time-varying weighted undirected graph model G = (V, E), where V is the set of satellite nodes and E is the set of inter-satellite links. Each link also contains a weight attribute, which includes at least propagation delay, available bandwidth, and packet loss rate information. This model provides a formal descriptive basis for subsequent routing decisions.

[0055] Step Two: Based on the core differences in service quality requirements, establish a multi-service traffic classification mechanism to divide the business traffic flowing through the network into three discrete service categories: The first category is latency-sensitive services, which have strict constraints on end-to-end transmission latency and latency jitter. Typical applications include real-time voice, interactive video communication, and remote control commands. The second category is high-bandwidth guaranteed services, which have clear requirements on continuously available bandwidth and throughput stability. Typical applications include high-definition video streaming, large-scale remote sensing data downlink, and batch file transfer. The third category is best-effort services, which have a high tolerance for changes in service quality parameters such as latency, bandwidth, and packet loss. They are suitable for applications such as email, background synchronization, and non-real-time data reporting.

[0056] For each of the above service categories, predefine differentiated service quality constraint thresholds: set a maximum tolerable end-to-end latency threshold for latency-sensitive services; set a minimum guaranteed bandwidth threshold for high-bandwidth guaranteed services; set a maximum allowable packet loss rate threshold for all service categories, where k∈{A, B, C} represents different service categories; these thresholds serve as hard constraints or optimization target references that must be met during the routing algorithm optimization process.

[0057] Step 3: Deploy an autonomous decision-making agent on each satellite node. These agents together constitute a distributed multi-agent system. Each agent is independently responsible for processing data packets arriving at its node and performing hop-by-hop forwarding decisions at the data packet level. This sequential routing decision-making process of each agent is formally modeled as a discrete-time Markov Decision Process (MDP). The model is a quadruple M = (S, A, P, R), where S is the state space, which is the set of all possible environmental states that the agent can perceive; A represents the action space, which contains all legal forwarding actions that the agent can perform in any state; P: S × A × S → [0, 1] is the state transition probability function, which describes the probability of the environment transitioning to the next state after taking a certain action in a specific state. This function is essentially a mathematical abstraction of network dynamics, reflecting the environmental uncertainty in the decision-making process. As a reward function, it provides a scalar instant reward signal after the agent performs an action and causes a state transition. This signal comprehensively evaluates the immediate effects of the selected action on meeting service quality, optimizing network performance, etc., and is a key feedback signal that drives the agent to learn and optimize strategies.

[0058] Based on this MDP framework, the goal of each agent is to learn an optimal policy π through continuous interaction with the environment. * S→A: This strategy can map any observed state to an action, maximizing the expected value of the cumulative discounted reward obtained by the agent in the long run. This modeling method decouples the complex, global satellite network routing optimization problem into a series of parallel, locally observed sequential decision problems, laying the theoretical foundation for subsequent applications of deep reinforcement learning algorithms for distributed and adaptive optimization.

[0059] (1) State-space modeling

[0060] The state space S in the Markov decision process is the set of all environmental information and task context on which the agent makes routing decisions; for any satellite node v in the network i The specific state s that the agent observes and utilizes at decision time t. i t It is a multidimensional feature vector that is carefully designed to comprehensively capture the key dynamic factors affecting routing performance in a compact form, and its structure can be represented as follows.

[0061]

[0062] Node body state C i Includes the satellite node v where the current agent is located.i The inherent attributes and real-time pose information of the node; specifically including: the unique identifier number of the node in the global constellation, the sequence number of the node in the orbital plane, and the three-dimensional position coordinates of the node in the inertial coordinate system or geocentric coordinate system obtained through the onboard navigation system; this dimensional information provides a basic spatial positioning reference for routing decisions.

[0063] Task target state T: contains the final destination node v of the data packet to be forwarded. d The relevant information includes: the unique identifier of the destination node and its real-time location coordinates under predictable ephemeris; this information provides global guidance for routing and is the basis for ensuring the eventual reachability of data packets.

[0064] Link Quality Status (B) i，j ,Pl), representing the current node v t With all its valid neighbor nodes v j The instantaneous transmission capability of inter-satellite links; for each hop candidate link e i，j This dimension includes two key metrics: remaining available bandwidth B. i，j The value is calculated as follows:

[0065] B i，j =B total (e i，j )-B used (e i，j (13)

[0066] That is, the total link bandwidth B total (e i,j Subtract the current bandwidth occupied by B used (e i，j The calculation reflects the instantaneous capacity margin of the link.

[0067] The link packet loss rate Pl, estimated based on recent historical transmission statistics, reflects the link's transmission reliability. This dimension is the core basis for evaluating the direct cost of actions (selecting the next hop); it can be expressed as:

[0068]

[0069] Node load status Q i , representing the current node v i Its own congestion level; specifically, the real-time depth q of the node's forwarding queue. i This refers to the number of data packets waiting to be processed in the buffer or the total amount of data; this state is directly related to queuing latency and is a key factor affecting end-to-end latency and avoiding local congestion.

[0070] Business semantic state ST: identifies the service type to which the current data packet to be decided belongs; it is a discrete classification variable, and its value comes from a predefined set of service classifications, namely latency-sensitive type - class A, high bandwidth guaranteed type - class B, and best-effort type - class C; this state is the instruction that drives the differentiated routing strategy, and it determines the preference weight of the agent when weighing conflicting objectives such as latency, bandwidth, and packet loss rate.

[0071] Since the aforementioned state components originate from different sensors, measurement modules, or protocol fields, their physical meanings, dimensions, and numerical ranges vary. To ensure the numerical stability of the deep neural network input and accelerate training convergence, the state vector... Before being input into the policy network, normalization is required. Each state variable is scaled by its theoretical maximum or minimum value or historical observation statistics, and is finally mapped to a unified, bounded numerical range. Through this process, heterogeneous multi-source observation data is transformed into a high-dimensional feature representation that is comparable and suitable for neural network processing.

[0072] (2) Action space modeling

[0073] In a low-Earth orbit satellite network routing framework based on deep reinforcement learning, the agent deployed on each satellite node needs to select the appropriate next-hop forwarding direction for the data packets it processes; action space Defined as a discrete, finite set of actions, directly mapped to the physical link interface configuration of the satellite node:

[0074]

[0075] This design strictly maps to the typical physical link layout of a low-Earth orbit satellite constellation, containing four basic geometric directional commands: forward along the orbital plane (front), pointing to the next adjacent satellite in the same orbital plane; backward along the orbital plane (back), pointing to the preceding adjacent satellite in the same orbital plane; to the left adjacent orbital plane (left) and to the right adjacent orbital plane (right), pointing to the satellites in the left and right adjacent orbital planes with the closest latitudes, respectively. Considering the inherent high dynamism of satellite network topology, especially when satellites are operating in high-latitude airspace near polar regions, the relative motion between satellites intensifies, causing drastic changes in the distance and angle of cross-orbital plane links, i.e., the left and right directions, and may even be temporarily interrupted due to antenna pointing limits or physical obstruction; therefore, this method introduces a dynamic action space constraint mechanism. The system dynamically determines the stability of the network environment by monitoring the orbital position of satellite nodes or the real-time signaling status of inter-satellite links in real time. When a node enters a preset polar region or unstable region, this mechanism will automatically and forcibly constrain the effective action space of the agent to a certain limit.

[0076]

[0077] This means that data packets are only allowed to be forwarded in the orbital plane; this design enables the routing strategy to adapt to the physical constraints of the underlying network, fundamentally avoiding invalid decisions to select unstable or unavailable links, thereby significantly improving the reliability of data transmission and the environmental adaptability of the entire routing algorithm.

[0078] (3) Reward function modeling

[0079] In a deep reinforcement learning-based routing decision framework, the reward function is the core signaling mechanism driving the agent to learn the optimal strategy through trial and error. Its design directly determines the convergence direction and final performance of the algorithm. This invention designs a multi-objective weighted composite reward function, aiming to provide the agent with a comprehensive feedback that simultaneously optimizes multiple service quality indicators and global path efficiency; it can be expressed as:

[0080]

[0081] The reward function determines the link e selected by the agent at time t. i，j The calculation is performed after forwarding data packets; firstly, normalized instantaneous observations of four key performance indicators are obtained from the network environment: the transmission delay of the selected link can be expressed as:

[0082]

[0083] Remaining available bandwidth B of the selected link i，j This value can be represented by formula (2), and the larger the value, the better; the current packet loss rate Pl of the selected link can be represented by formula (3), and the smaller the value, the better; and the remaining propagation distance L of the data packet from the current node to the target node. i，d This value should also be as small as possible.

[0084] reward function The core is a linear weighted sum of these indicators, where ω1, ω2, ω3, and ω4 are the corresponding weight coefficients, and satisfy the following:

[0085] ω1+ω2+ω3+ω4=1 (19)

[0086] To meet differentiated service needs, the weighting coefficients are dynamically adjustable: for latency-sensitive services, the latency weight ω1 is significantly increased; for high-bandwidth guaranteed services, the bandwidth weight ω2 is significantly increased; and for best-effort services, a balanced or distance-focused weighting configuration can be adopted to guide the agent to learn routing strategies with different focuses for different types of traffic. In addition, the reward function also includes an important boundary penalty term to ensure the basic effectiveness of the algorithm. To prevent packets from being transmitted indefinitely in loops or from having excessively long paths, a maximum allowed number of hops is set. Once the number of hops for a packet exceeds this threshold, the forwarding of the packet is immediately terminated, and the agent is given a large negative constant penalty.

[0087] The core iterative loop of the algorithm begins with an agent on each satellite node; at each discrete decision time t, the agent first observes its local environment, obtains and constructs its current state s. t This state integrates multi-dimensional information such as nodes, links, queues, and service types; subsequently, the agent inputs its state into its policy network, or online Q network, which outputs the expected long-term value Q of each optional action a in a given state.

[0088] To balance utilizing known good strategies with exploring potential better paths, the agent employs an ∈-greedy strategy to select actions: choosing the action with the highest current Q-value with probability 1-∈, i.e.:

[0089]

[0090] Among them, the exploration rate ε decays with time according to formula (10), ensuring that the environment is fully explored in the early stage of training, and then stabilizes at the learned strategy in the later stage.

[0091] ∈ t =∈ min +(∈ max -∈ min )e -λt (twenty one)

[0092] The agent executes the selected action a t Afterwards, the environment will undergo a state transition, entering a new state, and will calculate and provide an immediate reward based on the aforementioned multi-objective weighted reward function. The entire interaction experience is represented by a quadruple (s t a t r t s t+1The data are stored in a shared experience replay pool in the form of [data type name missing]. During training, the algorithm periodically samples a small batch of historical experiences from this replay pool to break the temporal correlation between data and improve the stability of learning. For each sampled experience, the algorithm calculates its temporal difference target value. Then, the mean squared error between the predictions of the policy network is calculated as the loss function.

[0093] Loss(θ) = [r t +γmax a Q′(s t+1 a t ;θ′)-Q(s t a t ;θ)] (22)

[0094] The loss is minimized by stochastic gradient descent, and the parameters of the policy network are updated. To further improve stability, the parameters of the target network are not updated at every step, but are synchronized with the parameters of the policy network periodically and slowly. A soft update rule is usually adopted. Finally, when the cumulative reward curve of the training process tends to a stable plateau and the performance of the policy in the test environment reaches a satisfactory level, the training converges.

[0095] To fully verify the effectiveness and superiority of the proposed deep reinforcement learning-based routing algorithm for diversified quality of service constraints in low-Earth orbit satellite networks, this study compares the proposed multi-service dynamic intelligent routing (MSDIR) with two classic or cutting-edge benchmark routing algorithms, SPF and ELB, in a constructed high-dynamic satellite network simulation environment.

[0096] The MSDIR algorithm demonstrated significant advantages across four key dimensions of system performance, comprehensively validating its effectiveness in addressing the highly dynamic topology and diverse service requirements of low-Earth orbit satellite networks. For example... Figure 3As shown, in terms of algorithm convergence, MSDIR employs a dual-network deep Q-learning framework, combining an experience replay pool with an adaptive ε-greedy exploration strategy. By balancing exploration and utilization through dynamically decaying exploration rates, it ensures that the algorithm quickly converges to a stable high-reward strategy within approximately 4200 training cycles, providing a reliable training efficiency foundation for practical deployment. Regarding quality of service assurance, the algorithm achieves precise differentiated routing optimization for different types of services through a carefully designed multi-objective weighted reward function: For latency-sensitive services, the algorithm dynamically increases the latency factor weight, and the agent prioritizes paths with low propagation latency and low queue congestion, resulting in end-to-end latency significantly lower than traditional shortest path first and explicit load balancing algorithms under a 2.9Mbps network load; for high-bandwidth guaranteed services, the algorithm focuses on bandwidth factor optimization, driving the agent to select links with high remaining bandwidth, thereby achieving higher network throughput compared to the comparison algorithms; for best-effort services, the algorithm guides the process through packet loss rate penalties and remaining propagation distance, achieving the lowest packet loss rate among all service types while ensuring basic connectivity. In summary, the MSDIR algorithm, through its distributed intelligent agent architecture, physically constrained dynamic action space, and service-adaptive multi-objective optimization mechanism, has successfully achieved synergistic improvements in core indicators such as latency, packet loss rate, and throughput, demonstrating its enormous potential for efficient, reliable, and differentiated services.

[0097] The contents not described in detail in this application are existing technologies known to those skilled in the art.

Claims

1. A multi-service quality-constrained routing method for low-Earth orbit satellite networks based on deep reinforcement learning, characterized in that, The steps used are: Step 1: Establish a system model for the low-Earth orbit satellite network. The network adopts a parameterized Walker Delta constellation configuration, which includes M circular orbital planes. N satellite nodes are uniformly deployed in each orbital plane, forming a satellite node set V with a total size of |V| = M × N. The constellation has a consistent orbital inclination and maintains a fixed phase difference between adjacent orbital planes to ensure global coverage and regular changes in topology. Each satellite node acts as an intelligent routing agent in the network, dynamically maintaining its inter-satellite links; each inter-satellite link is represented in the topology as an edge e in the edge set E. i，j Connecting satellite node v i With v j The inter-satellite links are specifically divided into two categories: intra-orbital links, which connect two adjacent satellites in the same orbital plane, and their link length remains approximately constant under orbital dynamic constraints; and inter-orbital links, which connect satellites at similar latitudes in adjacent orbital planes, and whose link length and connection status change periodically with the relative positions of the satellites, which is the main factor leading to the high dynamism of the network topology. Based on the aforementioned constellation configuration and link model, this paper abstracts the satellite network topology into a time-varying weighted undirected graph model G = (V, E), where V is the set of satellite nodes and E is the set of inter-satellite links. Each link also contains a weight attribute, which includes at least propagation delay, available bandwidth, and packet loss rate information. This model provides a formal descriptive basis for subsequent routing decisions. Step Two: Based on the differences in core service quality requirements, establish a multi-service traffic classification mechanism to divide the business traffic flowing through the network into three discrete service categories: The first category is latency-sensitive services, which have strict constraints on end-to-end transmission latency and latency jitter. Typical applications include real-time voice, interactive video communication, and remote control commands. The second category is high-bandwidth guaranteed services, which have clear requirements on continuously available bandwidth and throughput stability. Typical applications include high-definition video streaming, large-scale remote sensing data downlink, and batch file transfer. The third category is best-effort services, which have a high tolerance for changes in service quality parameters such as latency, bandwidth, and packet loss. They are suitable for applications such as email, background synchronization, and non-real-time data reporting. For each of the above service categories, predefine its differentiated service quality constraint thresholds: set the maximum tolerable end-to-end latency threshold for latency-sensitive services; set the minimum guaranteed bandwidth threshold for high-bandwidth guaranteed services; set the maximum allowable packet loss rate threshold for all service categories, where k∈{A, B, C} represents different service categories; These thresholds serve as hard constraints or optimization targets that must be met during the routing algorithm optimization process. Step 3: Deploy an autonomous decision-making agent on each satellite node. These agents together constitute a distributed multi-agent system. Each agent is independently responsible for processing data packets arriving at its node and performing hop-by-hop forwarding decisions at the data packet level. This sequential routing decision-making process of each agent is formally modeled as a discrete-time Markov Decision Process (MDP), which is a quadruple M = (S, A, P, R), where S is the state space, which is the set of all possible environmental states (including network state and task context) that the agent can perceive; A represents the action space, which contains all legal forwarding actions that the agent can execute in any state. P: S×A×S→[0,1] is the state transition probability function, which describes the probability of the environment (i.e., the satellite network) transitioning to the next state after taking a certain action in a specific state. This function is essentially a mathematical abstraction of network dynamics (such as topology changes, queue updates, and link quality fluctuations), reflecting the environmental uncertainty in the decision-making process. As a reward function, it provides a scalar instant reward signal after the agent performs an action and causes a state transition. This signal comprehensively evaluates the immediate effects of the selected action on meeting service quality, optimizing network performance, etc., and is the key feedback signal that drives the agent to learn optimization strategies. Based on this MDP framework, the goal of each agent is to learn an optimal policy π through continuous interaction with the environment. * S→A: This strategy can map any observed state to an action, maximizing the expected value of the cumulative discounted reward obtained by the agent in the long run. This modeling method decouples the complex, global satellite network routing optimization problem into a series of parallel, locally observed sequential decision problems, laying the theoretical foundation for subsequent applications of deep reinforcement learning algorithms for distributed and adaptive optimization. (1) State-space modeling The state space S in the Markov decision process is the set of all environmental information and task context on which the agent makes routing decisions; for any satellite node v in the network i The specific state s that the agent observes and utilizes at decision time t. i t It is a multidimensional feature vector, which is carefully designed to comprehensively capture the key dynamic factors affecting routing performance in a compact form. Its structure can be represented as: Node body state C i Includes the satellite node v where the current agent is located. i The inherent attributes and real-time pose information of the node; specifically including: the unique identifier of the node in the global constellation, the node's sequence number in the orbital plane, and the three-dimensional position coordinates of the node in the inertial coordinate system or geocentric coordinate system obtained through the onboard navigation system; this dimensional information provides a basic spatial positioning reference for routing decisions; Task target state T: contains the final destination node v of the data packet to be forwarded. d The relevant information includes: the unique identifier of the destination node and its real-time location coordinates under predictable ephemeris; this information provides global guidance for routing and is the basis for ensuring the eventual reachability of data packets; Link Quality Status (B) i,j ,Pl), representing the current node v i With all its valid neighbor nodes v j The instantaneous transmission capability of inter-satellite links; for each hop candidate link e i，j This dimension includes two key metrics: remaining available bandwidth B. i,j The value is calculated as follows: B i，j =B total (e i，j )-B used (e i，j ) (2) That is, the total link bandwidth B total (e i，j Subtract the current bandwidth occupied by B used (e i，j This is calculated and reflects the instantaneous capacity margin of the link; The link packet loss rate Pl, estimated based on recent historical transmission statistics, reflects the link's transmission reliability. This dimension is the core basis for evaluating the direct cost of actions (selecting the next hop); it can be expressed as: Node load status Q i , representing the current node v i Its own congestion level; specifically, the real-time depth q of the node's forwarding queue. i This refers to the number of data packets waiting to be processed in the buffer or the total amount of data; this state is directly related to queuing latency and is a key factor affecting end-to-end latency and avoiding local congestion. Business semantic state ST: identifies the service type to which the current data packet to be decided belongs; it is a discrete classification variable whose value comes from a predefined set of service classifications (e.g., latency-sensitive type - class A, high bandwidth guaranteed type - class B, best-effort type - class C); this state is the instruction that drives the differentiated routing strategy, and it determines the agent's preference weight when weighing conflicting objectives such as latency, bandwidth, and packet loss rate. Because the aforementioned state components originate from different sensors, measurement modules, or protocol fields, their physical meanings, dimensions, and numerical ranges vary (e.g., location coordinates, bandwidth megabits, packet loss percentage, queue length, etc.); to ensure the numerical stability of the deep neural network input and accelerate training convergence, the state vector... Before being input into the policy network, normalization is required. Each state variable is scaled by its theoretical maximum or minimum value or historical observation statistics, and is finally mapped to a unified, bounded numerical range (such as [0, 1] or [-1, 1]). Through this process, heterogeneous multi-source observation data is transformed into a high-dimensional feature representation that is comparable and suitable for neural network processing. (2) Action space modeling In a low-Earth orbit satellite network routing framework based on deep reinforcement learning, the agent deployed on each satellite node needs to select the appropriate next-hop forwarding direction for the data packets it processes; action space Defined as a discrete, finite set of actions, directly mapped to the physical link interface configuration of the satellite node: This design strictly maps to the typical physical link layout of a low-Earth orbit satellite constellation, containing four basic geometric directional commands: forward along the orbital plane (front), pointing to the next adjacent satellite in the same orbital plane; backward along the orbital plane (back), pointing to the preceding adjacent satellite in the same orbital plane; left and right adjacent orbital planes, respectively pointing to the satellites in the left and right adjacent orbital planes with the closest latitudes. Considering the inherent high dynamism of satellite network topology, especially when satellites are operating in high-latitude airspace near polar regions, the relative motion between satellites intensifies, causing drastic changes in the distance and angle of cross-orbital plane links (i.e., the left and right directions), and may even be temporarily interrupted due to antenna pointing limits or physical obstruction; therefore, this method introduces a dynamic action space constraint mechanism. The system dynamically determines the stability of the network environment by monitoring the orbital position (such as latitude information) of satellite nodes or the real-time signaling status of inter-satellite links. When a node enters a preset polar region or unstable region, this mechanism will automatically and forcibly constrain the effective action space of the agent to a certain limit. This means that data packets are only allowed to be forwarded in the orbital plane; this design enables the routing strategy to adapt to the physical constraints of the underlying network, fundamentally avoiding invalid decisions to select unstable or unavailable links, thereby significantly improving the reliability of data transmission and the environmental adaptability of the entire routing algorithm. (3) Reward function modeling In a deep reinforcement learning-based routing decision framework, the reward function is the core signaling mechanism driving the agent to learn the optimal strategy through trial and error. Its design directly determines the convergence direction and final performance of the algorithm. This invention designs a multi-objective weighted composite reward function, aiming to provide the agent with a comprehensive feedback that simultaneously optimizes multiple service quality indicators and global path efficiency; it can be expressed as: The reward function determines the link e selected by the agent at time t. i，j The calculation is performed after forwarding data packets; firstly, normalized instantaneous observations of four key performance indicators are obtained from the network environment: the transmission delay of the selected link can be expressed as: Remaining available bandwidth B of the selected link i,j This value can be represented by formula (2), and the larger the value, the better; the current packet loss rate Pl of the selected link can be represented by formula (3), and the smaller the value, the better; and the remaining propagation distance L of the data packet from the current node to the target node. i，d This value should also be as small as possible; reward function r it The core is a linear weighted sum of these indicators, where ω1, ω2, ω3, and ω4 are the corresponding weight coefficients, and satisfy the following: ω1+ω2+ω3+ω4=1 (8) To meet differentiated service needs, the weighting coefficients are dynamically adjustable: for latency-sensitive services, the latency weight ω1 is significantly increased; for high-bandwidth guaranteed services, the bandwidth weight ω2 is significantly increased; and for best-effort services, a balanced or distance-focused weighting configuration can be used to guide the agent to learn routing strategies with different focuses for different types of traffic. In addition, the reward function also includes an important boundary penalty term to ensure the basic effectiveness of the algorithm. To prevent packets from being transmitted indefinitely in loops or from having excessively long paths, a maximum allowed number of hops is set. Once the number of hops for a packet exceeds this threshold, the forwarding of the packet is immediately terminated, and the agent is given a large negative constant penalty. The core iterative loop of the algorithm begins with an agent on each satellite node; at each discrete decision time t, the agent first observes its local environment, obtains and constructs its current state s. t This state integrates multi-dimensional information such as nodes, links, queues, and service types; subsequently, the agent inputs its state into its policy network (or online Q network), which outputs the expected long-term value Q of each optional action a in a given state; To balance utilizing known good strategies with exploring potential better paths, the agent employs an ε-greedy strategy to select actions: choosing the action with the highest current Q-value with probability 1-ε, i.e.: Among them, the exploration rate ε decays with time according to formula (10), ensuring that the environment is fully explored in the early stage of training, and then stabilizes at the learned strategy in the later stage. e t =e min +(e max --e min )e -λt (10) The agent executes the selected action a t Subsequently, the environment (i.e., the satellite network) will undergo a state transition, entering a new state, and will calculate and feed back an immediate reward based on the aforementioned multi-objective weighted reward function. The entire interaction experience is represented by a quadruple (s t a t r t s t+1 The data are stored in a shared experience replay pool in the form of [data type name missing]. During training, the algorithm periodically samples a small batch of historical experiences from this replay pool to break the temporal correlation between data and improve the stability of learning. For each sampled experience, the algorithm calculates its temporal difference target value. Then, the mean squared error between the predictions of the policy network is calculated as the loss function. Loss(θ)=[r t +γmax a Q′(s t+1 ,a t ;θ′)-Q(s t ,a t (11) The loss is minimized by stochastic gradient descent, and the parameters of the policy network are updated. To further improve stability, the parameters of the target network are not updated at every step, but are synchronized with the parameters of the policy network periodically and slowly. A soft update rule is usually adopted. Finally, when the cumulative reward curve of the training process tends to a stable plateau, and the performance of the policy in the test environment (such as end-to-end latency and packet loss rate) reaches a satisfactory level, the training converges.

Images