Satellite network distributed routing method, system and device and storage medium

Abstract

The invention relates to a satellite network distributed routing method, a satellite network distributed routing system, satellite network distributed routing equipment and a storage medium, belongs to the technical field of wireless communication, and solves the problems of dynamic topological structure change and overlarge load of part of nodes caused by non-uniform service volume distribution in the existing satellite network routing method. The satellite network distributed routing method comprises the following steps: firstly, modeling a routing problem, constructing a problem model, then defining a state space, an observation space, an action space and a reward function of a Markov decision process, constructing a DDQN-based satellite network routing algorithm training framework based on the Markov decision process, and constructing a satellite network routing algorithm training framework based on the DDQN. And then a satellite network routing algorithm training framework based on DDQN is utilized to realize satellite network routing decision, and the network load is reduced with the purpose of minimizing end-to-end delay. The method is not influenced by the dynamic change of the network topology, and can adapt to the frequent network topology change of the low-orbit satellite, thereby ensuring the real-time performance and robustness of data transmission.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technology, and in particular to a satellite network distributed routing method, system, device and storage medium. Background Art

[0002] With the advent of 6G wireless networks, advances in wireless communications and mobile network technologies are driving increasing user demand for high-speed data transmission applications. This places stringent demands on networks for efficient, seamless coverage, massive connectivity, and high capacity to meet exponentially growing data traffic. However, today's traditional terrestrial networks struggle to meet these unprecedentedly stringent requirements due to limited and costly network infrastructure deployment and operations. Recent developments and advancements in Low Earth Orbit (LEO) satellite networks offer a solution for extending internet coverage to remote and oceanic areas. Typically, constellations consisting of hundreds of thousands of LEO satellites are deployed at altitudes between 500 and 2,000 kilometers to achieve global communications coverage. LEO satellite constellations offer several advantages over geosynchronous and medium Earth orbit (GEO) satellite constellations. First, their closer proximity to the Earth's surface reduces signal transmission latency. Second, LEO constellations can deploy a large number of small satellites, providing greater overall bandwidth. Furthermore, due to the larger number of satellites in LEO constellations, they can provide more comprehensive global coverage. These advantages enable low-Earth orbit satellites to provide a wide range of services to the ground, including high-speed internet access, real-time communications, distance education, medical support, and intelligent transportation. They are ideally suited for real-time communications, large-scale data transmission, global coverage, and future communications needs. The deployment of low-Earth orbit satellites in space has increased significantly over the past few years, making the design of viable network solutions for low-Earth orbit satellite constellations imperative. Currently, most satellite routing algorithms are extensions of terrestrial network routing schemes, primarily relying on shortest-path strategies. While these satellite routing schemes are effective and easy to deploy under light loads, they become insufficient to meet user needs as network topology changes and traffic loads increase. This is because low-Earth orbit satellite networks have distinct characteristics compared to terrestrial networks, such as extended inter-orbit link propagation delays and frequent topology changes driven by the high-speed motion of satellites. Furthermore, the uneven distribution of traffic demand leads to unbalanced traffic loads and significant variations in buffer queue backlogs, which can cause severe congestion with high packet loss rates, especially when using shortest-path routing schemes. Therefore, the dynamic topology and unbalanced traffic loads of low-Earth orbit satellite networks make it challenging to develop appropriate routing methods that guarantee end-to-end delay requirements. Summary of the Invention

[0003] In order to solve the problems of dynamic topology changes in existing satellite network routing methods and excessive load on some nodes due to uneven distribution of traffic, the present invention provides a satellite network distributed routing method, system, device and storage medium based on deep reinforcement learning.

[0004] The technical solution adopted by the present invention is as follows:

[0005] A satellite network distributed routing method comprises the following steps:

[0006] Step 1: Construct a low-orbit satellite network model and set model parameters, including orbit altitude, orbit inclination, number of orbital planes, and total number of satellites;

[0007] Step 2: Model the routing algorithm problem of the low-orbit satellite network model and obtain three problem models: communication delay model, packet loss rate model, and node load model;

[0008] Step 3: Setting an optimization goal, wherein the optimization goal is to optimize latency while ensuring load balancing of the low-orbit satellite network;

[0009] Step 4: Model the distributed routing process as a Markov decision process, defining the observation space, action space, state transition space, and reward function;

[0010] Step 5: Based on the Markov decision process created in step 4, build a DDQN-based satellite network routing algorithm training framework;

[0011] Step 6: Use the DDQN-based satellite network routing algorithm training framework established in Step 5 to implement satellite network routing decisions with the goal of minimizing end-to-end latency while reducing network load.

[0012] Accordingly, the present invention further proposes a satellite network distributed routing system, comprising:

[0013] A model building module is used to build a low-orbit satellite network model and set model parameters, including orbit altitude, orbit inclination, number of orbit planes and total number of satellites;

[0014] The modeling module is used to model the routing algorithm problem of the low-orbit satellite network model, and obtain three problem models: communication delay model, packet loss rate model and node load model;

[0015] A setting module is used to set an optimization goal, wherein the optimization goal is to optimize the latency while ensuring the load balance of the low-orbit satellite network;

[0016] The decision modeling module is used to model the distributed routing process as a Markov decision process, defining the observation space, action space, state transition space, and reward function;

[0017] The training framework module is used to build a DDQN-based satellite network routing algorithm training framework based on the Markov decision process created by the decision modeling module;

[0018] The routing decision module is used to implement satellite network routing decisions using the DDQN-based satellite network routing algorithm training framework established by the training framework module, with the goal of minimizing end-to-end delay while reducing network load.

[0019] A processing device includes a memory and a processor, wherein the memory stores a computer program executable by the processor, and the processor implements the satellite network distributed routing method described above when executing the computer program.

[0020] A storage medium stores a computer program, which, when read and executed, implements the satellite network distributed routing method as described above.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] This paper proposes a satellite network distributed routing method, system, device, and storage medium based on a Double Deep Q-Network (DDQN). This satellite network distributed routing method first models the routing problem and constructs a problem model. Next, a distributed deep reinforcement learning routing framework is designed based on a Markov decision process, defining the state space, observation space, and action space. A reward function is designed based on the satellite node queue state and load growth characteristics. Finally, a deep reinforcement learning (DDQN) architecture and a prioritized experience replay mechanism are employed to accelerate convergence through value network separation and sample weight adjustment strategies, thereby achieving low end-to-end transmission latency and reducing node load in the network. This method is unaffected by dynamic changes in network topology and can adapt to frequent network topology changes in low-orbit satellites, ensuring real-time and robust data transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1This is a flow chart of a satellite network distributed routing method proposed in an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of the Walker-Delta constellation;

[0026] Figure 3 This is the training flow chart of the DDQN-based satellite network routing algorithm;

[0027] Figure 4 This is a decision flow chart of the satellite network distributed routing method proposed by the present invention. DETAILED DESCRIPTION

[0028] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0029] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] like Figure 1 As shown, this embodiment provides a satellite network distributed routing method based on DDQN. In step 1 of the method, a low-orbit satellite network model is first constructed. The specific construction process includes the following steps 1.1 to 1.3.

[0031] Step 1.1: Based on Figure 2 The Walker-Delta constellation shown in the figure constructs a satellite network model. According to the characteristics of the Walker-Delta constellation, the Walker-Delta constellation is represented as ,in Represents the total number of constellation satellites, represents the number of orbital planes, represents the orbit height parameter, is the orbital inclination, is the phase factor, .

[0032] Step 1.2: Abstract the satellite network model according to graph theory , where the vertex set Corresponding to LEO satellite nodes, edge sets Indicates intersatellite link.

[0033] Step 1.3: Discretize the satellite network topology according to the time series, and keep the satellite topology and network resources relatively stable within each time slice.

[0034] Step 2: Model the routing algorithm problem of the low-orbit satellite network model. It can be modeled into three main categories of problem models, namely communication delay model, packet loss rate model and node load model.

[0035] Specifically, the communication delay model is established through the following steps:

[0036] Step 2.1: In a low-orbit satellite network, two satellites and The Euclidean distance between them can be expressed as:

[0037] ;

[0038] in, and They respectively represent the coordinate positions of the two satellites in three-dimensional space.

[0039] Step 2.2: Based on the Euclidean distance between the two satellites, the data packet is sent to the satellite. and The propagation delay between can be expressed as:

[0040] ;

[0041] in, Indicates the speed at which a signal propagates in space.

[0042] Step 2.3: The buffer capacity of the low-orbit satellite is limited. Data packets are forwarded in a first-in-first-out manner. The arrival interval of the data packets follows an exponential distribution, while the service time may be close to an exponential distribution, which conforms to the M / M / 1 / m queuing model, where M indicates that the arrival interval follows an exponential distribution, 1 indicates that there is a single queue in the satellite node, and m represents the maximum capacity of the buffer. The queue refers to the first-in-first-out data structure in the buffer of the low-orbit satellite for temporarily storing data packets to be forwarded. This invention assumes that the arrival of data packets follows the parameter The time required for the satellite to forward a packet follows a Poisson distribution characterized by For the sake of generality, the present invention assumes that the time each packet spends in the queue is independently and identically distributed. Therefore, based on the established findings of the M / M / 1 / m queue model, the number of packets waiting in the queue is The expected time for each packet follows The Erlang distribution of order is used The probability density function of this distribution is defined as:

[0043] ;

[0044] in, = The above formula shows that the waiting time of the data packets in the queue follows the parameter exponential distribution.

[0045] Step 2.4: Data Packet The expected queuing delay It can be expressed as ,in Satellite node The queue is The capacity of time, represents the average waiting time, If the queue reaches its maximum capacity before the next packet arrives, the satellite node will drop the incoming packet. Therefore, the process of queue capacity change can be described by considering the dynamics of packet arrival and departure in the queue. The queue capacity at any given time can be modeled as:

[0046] ;

[0047] in, express Satellite node at the moment The queue capacity; and Respectively expressed in Moment and Satellite nodes between The number of packets sent and received by the queue; Satellite node The maximum capacity of the queue.

[0048] Step 2.5: Average end-to-end delay is the average time it takes for a data packet to travel from the source node to the destination node. It is calculated as:

[0049] ;

[0050] in, is the average end-to-end delay, Indicates the total number of packets successfully transmitted, Indicates the The start time of sending a data packet, Indicates the The time it takes to complete the reception of a data packet.

[0051] Step 2.6: If the queue reaches its maximum capacity before the next packet arrives, the satellite node will drop the incoming packet. Defined as:

[0052] ;

[0053] in, Indicates the number of packets discarded. Indicates the total number of packets sent on the network.

[0054] Create a node load model by following these steps:

[0055] Step 2.7: Average Full Node Percentage It represents the average proportion of fully loaded nodes among the working nodes. The calculation formula is:

[0056] ;

[0057] in, Indicates in The number of nodes in the fully loaded state in each time step, Indicates in The number of nodes working in each time step, Indicates the total time steps or the time range of the statistics.

[0058] Step 2.8: Average queue length Indicates the average length of the task queue in the network during the statistical time. The calculation formula is:

[0059] ;

[0060] in, Indicates in The length of the task queue for each time step, Indicates the total time step.

[0061] Step 3: According to the problem model, set the optimization goal, which is to optimize the latency while ensuring the load balance of the low-orbit satellite network.

[0062] Step 4: Model the distributed routing process as a Markov decision process, including observation space, action space, state transition space, and reward function.

[0063] Specifically, the Markov decision process is created by the following steps:

[0064] Step 4.1: Define the observation space. Each satellite can establish connections with up to 4 satellites around it. In order to more accurately describe the local network status, the satellite nodes are defined as follows: The local observation of is defined as the state of the neighboring satellites that can be connected to it, denoted as , which can be expressed as:

[0065] ;

[0066] in, Satellite node The send queue size; Satellite node The receive queue size; Satellite node The expected queuing delay of packets on adjacent satellites is estimated based on the queue capacity of the adjacent satellites; Indicates satellite The propagation delay of each link to the satellite; Satellite node Is it the destination node of the data packet? If so, The value is 1 if yes, otherwise it is 0.

[0067] Step 4.2: Define the state space, as a satellite node decision process, the satellite node routing decision should take into account the satellite node At the moment All local environment information, so at any moment The state space of is defined as: ,in is the total number of satellites in the constellation, that is, the total number of satellite nodes.

[0068] Step 4.3: Define the action space, since each satellite can establish up to 4 inter-orbit links Defining Satellite Nodes At the moment The actions taken are:

[0069] ;

[0070] in, ∈{0,1} represents the A choice of inter-track links, where .

[0071] Step 4.4: Define a reward function that dynamically calculates the difference between the satellite node's queue length and the equilibrium queue length, and calculates a penalty proportionally based on the queue growth factor. When the queue length exceeds a threshold determined by the weight parameter and the maximum queue length, the penalty is dynamically adjusted based on the queue size difference and the queue growth factor, guiding the agent to select nodes with lower load and a more gradual growth trend. The reward function formula is as follows:

[0072] ;

[0073] in, It is the sum of the node's sending queue and receiving queue; is the maximum queue length; is the weight of the queue growth factor; is the queue growth factor.

[0074] Step 5: Based on the Markov decision process created in step 4, build a DDQN-based satellite network routing algorithm training framework.

[0075] Train the DDQN-based satellite network routing algorithm, such as Figure 3 As shown, the specific steps include:

[0076] Step 5.1: Each satellite node is considered an independent agent, capable of perceiving the state of its network and making routing decisions accordingly. After the agent executes an action, the environment provides feedback on the next state and corresponding reward. The satellite node must select the optimal next-hop node based on the current network state. The decision space is a discrete set of neighboring nodes, with the goal of minimizing end-to-end transmission delay and reducing network load.

[0077] Step 5.2: During training, use the experience replay pool to store information about each interaction with the environment, including state ,action and immediate rewards from environmental feedback , next state For each decision, the generated samples The data is stored in the experience replay pool. During training, batches of data are extracted using a prioritized sampling mechanism for network updates to reduce the correlation of training data and prevent overfitting. Prioritized experience replay is used during sampling, assigning different priorities based on the temporal difference error of samples. High-error samples are extracted with a higher probability to accelerate the learning of key experiences.

[0078] Step 5.3: The target Q value calculation process is divided into two stages: First, the estimation network determines the next state The best action , and then the target network evaluates the long-term value of the action ( ),in, To estimate the network parameters, is the target network parameter. The Q estimate of the estimated network output represents the real-time prediction of the state-action value. Refers to the next state The maximum Q value of all possible actions. Indicates choosing the action that maximizes the Q value , used to decide the optimal next-hop node.

[0079] Step 5.4: The loss function is used to measure the error between the Q value predicted by the estimation network, that is, the Q estimate, and the target Q value output by the target network. Specifically, the mean square error can be used as the loss function, and its calculation formula is:

[0080] ;

[0081] in, is the priority sampling capability, which indicates the number of samples sampled in each training; It is the parameter of the estimated network, which is used to predict the Q value of the current state-action pair in the model; is the target Q value, which is calculated by the target network.

[0082] Step 5.5: Next, update the estimated network parameters and the target network parameters by gradient descent method. Then, the estimated network parameters are directly copied every C steps, and the stability of the target Q value is maintained through a periodic hard update strategy.

[0083] Step 6: Use the DDQN-based satellite network routing algorithm training framework established in Step 5 to implement satellite network routing decisions with the goal of minimizing end-to-end latency while reducing network load.

[0084] In this step, the routing decision process of the present invention is constructed, such as Figure 4 As shown, the specific steps include:

[0085] Step 6.1: Initialize the experience replay pool, estimation network, and target network, and then train.

[0086] Step 6.2: Randomly generate a pair of start and target nodes of the satellite topology constellation as the decision starting point of the agent. The routing decision is made by the start node as an agent with an initialized state.

[0087] Step 6.3: The agent acts as the starting node and has an initialized state. At each decision, the agent selects an action based on its current state and the pre-set dynamic greedy strategy. If the action results in the packet reaching the target node, it proceeds to step 6.8; otherwise, it proceeds to step 6.4.

[0088] Step 6.4: Predict node load and calculate rewards. After each decision, the agent uses a lightweight neural network to predict the load of the next hop node, which is the Q value, and returns a reward based on the load.

[0089] Step 6.5: The agent chooses the best action based on the reward, updates the parameters of its estimation network, and stores the experience.

[0090] Step 6.6: The experience replay pool stores the experience tuple for subsequent experience replay and network update. After extracting experience from the experience replay pool, determine whether the action in the extracted experience causes the data packet to reach the target node. If so, ,in is the target Q value, which represents the agent’s estimate of the future cumulative reward after performing an action in the current state. represents the immediate reward at the current moment, then execute step 6.7; otherwise, ,in represents the discount factor (0≤ <1), used to balance the importance of current rewards and future rewards, Denote the maximum Q-value of the next state, which represents the estimate of the optimal future cumulative reward, and then execute step 6.7.

[0091] Step 6.7: The estimation network updates its weights using gradient descent to minimize the prediction error. The target network updates its weights every predetermined number of steps to maintain weight stability and avoid inconsistent weight updates between the target and estimation networks.

[0092] Step 6.8: When the packet reaches the destination node, the routing decision is complete, the environment rewards the packet, and the cumulative reward is calculated. This cumulative reward is used to evaluate routing performance and provide a basis for subsequent path optimization.

[0093] Step 6.9: Randomly generate a new pair of starting nodes and target nodes, return to step 6.3, and repeat the above training process.

[0094] The DDQN-based distributed routing method for satellite networks proposed in this paper first models the routing problem and constructs a problem model. Next, a distributed deep reinforcement learning routing framework is designed based on the Markov decision process, defining the state space, observation space, and action space. A reward function is designed based on the satellite node queue state and load growth characteristics. Next, a deep reinforcement learning (DDQN) architecture and a prioritized experience replay mechanism are employed to accelerate convergence through value network separation and sample weight adjustment strategies, thereby achieving lower end-to-end transmission latency and reducing node load in the network. This method is unaffected by dynamic changes in network topology and can adapt to frequent network topology changes in low-orbit satellites, thereby ensuring real-time and robust data transmission.

[0095] The satellite network distributed routing system, processing equipment and storage medium used to execute the satellite network distributed routing method provided by the present invention are described below. The specific implementation process and technical effects thereof are referred to the relevant content of the above-mentioned satellite network distributed routing method, which will not be repeated below.

[0096] In another embodiment, the present invention provides a satellite network distributed routing system, the system comprising:

[0097] The model building module is used to build a low-orbit satellite network model and set model parameters, including orbit altitude, orbit inclination, number of orbit planes, and total number of satellites;

[0098] The modeling module is used to model the routing algorithm problem of the low-orbit satellite network model, and obtain three problem models: communication delay model, packet loss rate model and node load model;

[0099] The setting module is used to set the optimization target, which is to optimize the latency while ensuring the load balance of the low-orbit satellite network;

[0100] The decision modeling module is used to model the distributed routing process as a Markov decision process, defining the observation space, action space, state transition space, and reward function;

[0101] The training framework module is used to build a DDQN-based satellite network routing algorithm training framework based on the Markov decision process created by the decision modeling module;

[0102] The routing decision module is used to implement satellite network routing decisions using the DDQN-based satellite network routing algorithm training framework established by the training framework module, with the goal of minimizing end-to-end delay while reducing network load.

[0103] The above-mentioned satellite network distributed routing system is used to execute the satellite network distributed routing method provided by the above-mentioned embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0104] The above modules may be one or more integrated circuits configured to implement the above satellite network distributed routing method, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented by scheduling program code on a processing element, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling program code. For another example, the modules may be integrated together and implemented as a system-on-a-chip (SOC).

[0105] In another embodiment, the present invention provides a processing device comprising a memory and a processor. The memory is used to store a computer program, and the processor invokes the computer program stored in the memory to execute the steps of the satellite network distributed routing method provided in the aforementioned embodiment. The specific implementation methods and technical effects are similar and will not be further elaborated here.

[0106] In another embodiment, the present invention provides a program product, such as a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the various steps of the satellite network distributed routing method provided in the aforementioned embodiment. It should be noted that the computer-readable storage medium in this embodiment may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof, such as a USB flash drive, a removable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, among other media capable of storing program code. The computer-readable storage medium may be included in a computing device or may exist independently and not incorporated into the computing device.

[0107] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0108] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A satellite network distributed routing method, characterized in that: The following steps are involved: Step 1: Construct a low-orbit satellite network model and set model parameters, including orbit altitude, orbit inclination, number of orbital planes, and total number of satellites; Step 2: Model the routing algorithm problem of the low-orbit satellite network model and obtain three problem models: communication delay model, packet loss rate model, and node load model; Step 3: Setting an optimization goal, wherein the optimization goal is to optimize latency while ensuring load balancing of the low-orbit satellite network; Step 4: Model the distributed routing process as a Markov decision process, defining the observation space, action space, state transition space, and reward function; Step 5: Based on the Markov decision process created in step 4, build a DDQN-based satellite network routing algorithm training framework; Step 6: Use the DDQN-based satellite network routing algorithm training framework established in Step 5 to implement satellite network routing decisions with the goal of minimizing end-to-end latency while reducing network load.

2. The satellite network distributed routing method according to claim 1, wherein: The low-orbit satellite network model is constructed by the following steps: Step 1.1: Build a satellite network model based on the Walker-Delta constellation; Step 1.2: Abstract the satellite network model according to graph theory , where the vertex set Corresponding to LEO satellite nodes, edge sets represents an intersatellite link; Step 1.3: Discretize the satellite network topology according to the time series, and keep the satellite topology and network resources relatively stable within each time slice.

3. The satellite network distributed routing method according to claim 1 or 2, characterized in that: Use the following steps to build a communication delay model and packet loss rate model: Step 2.1: Calculate the number of satellites in the low-orbit satellite network and Euclidean distance between ; Step 2.2: Based on the Euclidean distance between the two satellites, calculate the distance between the two satellites. and The propagation delay between , the calculation formula is: ； in, Indicates the speed of signal propagation in space; Step 2.3: Based on the M / M / 1 / m queue model, determine the waiting time of the data packets in the queue according to the parameters where M indicates that the inter-arrival time follows an exponential distribution, 1 indicates that there is a single queue in the satellite node, and m indicates the maximum capacity of the buffer; Step 2.4: Define the data package The expected queuing delay for: ； in, Satellite node The queue is capacity of the moment; The queue capacity at any given time is: ； in, express Satellite node at time The queue capacity; and Respectively expressed in Moment and Satellite nodes between The number of packets sent and received by the queue; Satellite node The maximum capacity of the queue; Step 2.5: Calculate the average end-to-end delay using the formula: ； in, is the average end-to-end delay, Indicates the total number of packets successfully transmitted, Indicates the The start time of sending a data packet, Indicates the The time it takes to complete the reception of a data packet; Step 2.6: Define the end-to-end packet loss rate as: ； in, Indicates the number of packets discarded. Indicates the total number of packets sent on the network.

4. The satellite network distributed routing method according to claim 1 or 2, characterized in that: Create a node load model by following these steps: Step 2.7: Calculate the average percentage of fully loaded nodes using the following formula: ； in, is the average percentage of fully loaded nodes, Indicates in The number of nodes in the fully loaded state in each time step, Indicates in The number of nodes working in each time step, Indicates the total time step or the time range of statistics; Step 2.8: Calculate the average queue length using the following formula: ； in, is the average queue length, Indicates in The length of the task queue for each time step.

5. The satellite network distributed routing method according to claim 1 or 2, characterized in that: Create a Markov decision process by following these steps: Step 4.1: Define the observation space. Each satellite can establish a connection with up to 4 satellites around it. The local observation of is defined as the state of the neighboring satellites that can be connected to it: ； in, Satellite node The send queue size; Satellite node The receive queue size; Satellite node The expected queuing delay of packets on adjacent satellites is estimated based on the queue capacity of the adjacent satellites; Indicates satellite The propagation delay of each link to the satellite; Satellite node Is it the destination node of the data packet? If so, The value of is 1, otherwise it is 0; Step 4.2: Define the state space; ； in, is the total number of satellite nodes; Step 4.3: Define the action space; ； in, ∈{0,1} represents the A choice of inter-track links, where ; Step 4.4: Define the reward function. ； in, It is the sum of the node's sending queue and receiving queue; is the maximum queue length; is the weight of the queue growth factor; is the queue growth factor.

6. The satellite network distributed routing method according to claim 1 or 2, characterized in that: The following steps are used to build a DDQN-based satellite network routing algorithm training framework: Step 5.1: Treat each satellite node as an independent intelligent agent and select the optimal next-hop node based on the current network status; Step 5.2: Use the experience replay pool to store samples during training ,During training, batch data is extracted according to the priority sampling mechanism for network update; Step 5.3: The calculation of the target Q value is divided into two stages: the estimation network The best action , and then by the target network ( ); Step 5.4: Use the loss function to calculate the error between the Q estimate predicted by the estimation network and the target Q value; Step 5.5: Update the estimated network parameters by gradient descent, copy the estimated network parameters to the target network every C steps, and maintain the stability of the target Q value through a periodic hard update strategy.

7. The satellite network distributed routing method according to claim 1 or 2, characterized in that: Satellite network routing decisions are made through the following steps: Step 6.1: Initialize the experience replay pool, estimation network, and target network; Step 6.2: Randomly generate a pair of start and target nodes of the satellite topology constellation; Step 6.3: The agent acts as the starting node. Each time it makes a decision, it selects an action based on its current state and the preset dynamic greedy strategy. If the action causes the data packet to reach the target node, it jumps to step 6.

8. Otherwise, proceed to step 6.4; Step 6.4: Predict node load and calculate rewards; Step 6.5: The agent selects the best action based on the reward, updates the parameters of its estimation network, and stores the experience; Step 6.6: Extract experience from the experience replay pool and determine whether the action in the extracted experience causes the data packet to reach the target node; Step 6.7: The estimation network uses the gradient descent method to update the network weights, and the target network updates the network weights every preset number of steps; Step 6.8: When the data packet reaches the destination node, the routing decision is completed, the environment gives a reward, and the cumulative reward value is calculated; Step 6.9: Randomly generate a new pair of starting nodes and target nodes, return to step 6.3, and repeat the above training process.

8. A satellite network distributed routing system, characterized in that: include: A model building module is used to build a low-orbit satellite network model and set model parameters, including orbit altitude, orbit inclination, number of orbit planes and total number of satellites; The modeling module is used to model the routing algorithm problem of the low-orbit satellite network model, and obtain three problem models: communication delay model, packet loss rate model and node load model; A setting module is used to set an optimization goal, wherein the optimization goal is to optimize the latency while ensuring the load balance of the low-orbit satellite network; The decision modeling module is used to model the distributed routing process as a Markov decision process, defining the observation space, action space, state transition space, and reward function; The training framework module is used to build a DDQN-based satellite network routing algorithm training framework based on the Markov decision process created by the decision modeling module; The routing decision module is used to implement satellite network routing decisions using the DDQN-based satellite network routing algorithm training framework established by the training framework module, with the goal of minimizing end-to-end delay while reducing network load.

9. A processing device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, the satellite network distributed routing method according to any one of claims 1 to 7 is implemented.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is read and executed, the satellite network distributed routing method according to any one of claims 1 to 7 is implemented.

Images