A Fully Distributed Routing Method and System Based on Deep Reinforcement Learning

Abstract

This invention relates to a fully distributed routing method and system based on deep reinforcement learning, belonging to the field of satellite network dynamics technology. The method uses the remaining energy of neighboring nodes, the coordinates of the destination satellite, the length of the neighboring node's receive queue, the coordinates of the neighboring node, and the length of the current node's send queue as state inputs to intelligently decide the next hop. Its main steps include: First, establishing a partially observable Markov decision process model for each satellite, requiring only the spatial location, queueing state, and remaining energy of neighboring satellites. Second, proposing a fully distributed dynamic routing method based on multi-agent deep reinforcement learning to derive the optimal routing strategy.

Description

Technical Field

[0001] This invention belongs to the field of satellite network dynamic technology, and relates to a fully distributed routing method and system based on deep reinforcement learning. Background Technology

[0002] Satellite internet provides essential support for information forwarding, processing, and fusion, and is a key project in the national "new infrastructure" initiative. Routing is the core of satellite internet. Compared to geostationary and medium Earth orbit satellites, low Earth orbit (LEO) satellite networks offer advantages such as wide coverage, low latency, low launch costs, and low transmission power, making them an important component of communication network architecture. However, the high-dimensional and highly dynamic nature of LEO satellite constellations makes traditional terrestrial network routing algorithms ill-suited to their routing performance. Furthermore, the limited size of satellites restricts the power and lifespan of batteries and other equipment. Therefore, designing efficient dynamic routing algorithms for LEO satellite constellations is one of the current technological challenges they face. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a fully distributed routing method and system based on deep reinforcement learning. A graph attention neural network model is established based on the state of a fully distributed low-Earth orbit (LEO) satellite network, and a reward function is set according to the optimization objective function. The low-dimensional features obtained from the graph attention neural network model are input into a Dueling DQN network to obtain the Q-values ​​of each action. An ∈-greedy algorithm is used to select an action as the routing decision. An action in the action space is selected based on the state information in the state space. After the environment executes the action, it returns an immediate reward and the state at the next time step. The state, routing decision, reward function, and state at the next time step are stored in an experience pool. A small batch of samples is selected from the experience pool for training to obtain the Q-values ​​and the target Q-value. The parameters of the Q-estimation network and the Q-target network of the deep reinforcement learning model are updated according to the loss function and gradient descent. Through multiple iterations until the model converges, the update is completed. The updated deep reinforcement learning model is used to select the optimal routing path for the LEO satellite network. Data transmission is performed based on the optimal routing path, which increases the average amount of data that can be transmitted per unit of energy, thereby improving the energy utilization efficiency of the LEO satellite network.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] A fully distributed routing method based on deep reinforcement learning, comprising the following steps:

[0006] Step 1: Establish a system model and communication model for fully distributed routing in the LEO satellite network;

[0007] Step 2: Establish energy consumption, propagation delay, and queuing delay models for fully distributed routing in the LEO satellite network;

[0008] Step 3: Construct the objective function that minimizes the energy consumption, propagation delay, and queuing delay of the LEO satellite network;

[0009] Step 4: Calculate the estimated remaining propagation delay of each data packet from the candidate next hop to the destination satellite;

[0010] Step 5: Construct a state space based on user business requirements, with the lengths of the four transmission queues of the current satellite, the lengths of the receiving queues of neighboring satellite nodes, the coordinates of the destination satellite of the current data packet, the remaining estimated delay between the neighboring satellite nodes and the destination satellite node, the remaining energy of the current node, and the remaining energy of the neighboring satellite nodes as states;

[0011] Step 6: Use the GAT architecture to merge the relationship information between each agent, thereby obtaining the global hidden features;

[0012] Step 7: Construct the reward function for each action in the action space by referring to the objective function, where the action space represents the next hop candidate satellite node of the current satellite, i.e., neighboring satellites;

[0013] Step 8: At each time step, the agent interacts with the environment and obtains a local observation. Using GAT, the local observation is transformed into a global information feature representation by merging agent information. The obtained feature representation is then fed into the Q-estimation network of the deep reinforcement learning model to calculate the Q-value of each action in the action space. The value function of each action is evaluated based on the expected reward, and intelligent decision-making is performed through an ∈-greedy strategy, that is, an action is randomly selected as the next-hop satellite node with ∈ probability, and the action with the largest Q-value is selected as the next-hop satellite node with 1-∈ probability.

[0014] Step 9: The environment reacts to this action and receives an immediate reward and the next-step state. The state, routing decision, reward function, and next-step state are stored in the experience pool. A small batch of samples is randomly selected from the experience pool for training to obtain the estimated Q-value and the target Q-value. The parameters of the Q-estimation network and the Q-target network of the deep reinforcement learning model are updated according to the loss function and gradient descent. The above process is repeated until the deep reinforcement learning model converges. The converged deep reinforcement learning model is used to intelligently decide the optimal next hop for the LEO satellite network.

[0015] Optionally, in S3, the objective function that minimizes the energy consumption, propagation delay, and queuing delay of the LEO satellite network is calculated as follows:

[0016] min cost = (1-μ)·E l (i,j)+μ·t y

[0017]

[0018]

[0019]

[0020] Where μ represents the weighting factor, R i B represents the satellite's remaining energy; B represents the satellite's initial energy, i.e., battery capacity; E l (i, j) represents the energy state of link ij; t y This represents the average delivery delay of the link; n represents the number of neighboring satellite nodes.

[0021] Optionally, the propagation delay is the time required for the current satellite to propagate to the next hop; the queuing delay is the time it takes for data packets to queue in the satellite's input and output queues; and the energy consumption is the energy consumed by the satellite to forward data, including data transmission energy consumption, data reception energy consumption, and energy consumption for the satellite to maintain normal operation.

[0022] Optionally, step 6 specifically includes:

[0023] The state of the low-Earth orbit satellite network is obtained, including the length of the four transmission queues of the current satellite, the length of the receiving queue of the neighboring satellite nodes, the coordinates of the destination satellite of the current data packet, the remaining estimated delay of the neighboring satellite nodes from the destination satellite node, the remaining energy of the current node, the remaining energy of the neighboring satellite nodes, and the remaining estimated propagation delay from the neighboring nodes to the destination node. These states are combined by associating them with different elements of the input graph and considering how these elements are connected in the graph. The state of the elements is updated by an iterative message passing algorithm, and the output is generated using the resulting state.

[0024] The messages computed for the same node and its neighbors are aggregated using element-wise summation. The hidden state is updated using the aggregated information. The same process is repeated for all linked neighbors to obtain the global hidden information through iteration.

[0025] Optionally, the reward function is:

[0026]

[0027]

[0028]

[0029]

[0030] E i (t)=E o +E s +E r =(P o +P s +P r )×τ

[0031] G i (t)=P c (t)×max{0,(τ-T i )}

[0032] Where ω1, ω2, ω3 represent weighting factors, and τ y Indicates the total transmission delay. Indicates queuing delay. E represents the propagation delay. l R represents the energy of link i,j. i (t) represents the remaining link energy of satellite i, E i (t) represents the energy consumption of satellite i, G i (t) represents the energy gained by satellite i.

[0033] Optionally, the routing decision selects actions from the action space based on the ∈-greedy policy, including:

[0034] Calculate the probability of each action in the action space, randomly select an action from the action space with probability ∈ as the routing decision, and select the action with the largest Q value from the action space with probability 1-∈ as the routing decision.

[0035] Optionally, in step 9, after each action is performed, a four-tuple is obtained consisting of the state, routing decision, reward function, and next-time state. This four-tuple is stored in the experience pool, and four-tuples are randomly selected from the experience pool as a batch. Specifically:

[0036] A batch of samples is randomly selected from the experience pool for training to break the correlation between data. The current state information is used as the input of the Q estimation network, and the output is obtained through three fully connected layers, which is the Q value of the routing decision taken in this state. The next state information is used as the input of the Q estimation network, and the output is obtained through three fully connected layers, which is the action when the Q value of the Q estimation network reaches its maximum. This action and the next state are used as the input of the Q target network to output the target Q value.

[0037] The target network is fixed, and only the prediction network that interacts with the environment is updated; the difference between the TD-target value calculated by the target network and the output of the prediction network is used to optimize and update the model; after updating the prediction network several times, the target network is replaced with the updated network; the above steps are repeated until the model converges, and the optimal next hop is obtained by making a decision using the converged model; specifically:

[0038] The variance between the target value and the predicted value is called the loss function. We try to minimize the loss by updating the weights, and then calculate the gradient based on the loss function.

[0039] After every k steps of updates, copy the actual network weights to the target network weights, and repeat the above steps until the model converges.

[0040] A fully distributed routing system based on deep reinforcement learning, comprising:

[0041] A system model module is built to establish communication models, propagation delay models, queuing delay models, and energy consumption models for low-Earth orbit satellite networks.

[0042] The objective function establishment module is used to establish an objective function that minimizes the energy consumption, propagation delay, and queuing delay of the low-Earth orbit satellite network.

[0043] The remaining estimated propagation delay module is used to calculate the estimated propagation delay from the satellite where the current data packet is located to the destination node;

[0044] The graph attention network feature extraction module uses the GAT architecture to merge the relationship information between each agent, thereby obtaining global hidden features;

[0045] The state space module is used to construct a state space based on user business requirements, with the lengths of the four transmission queues of the current satellite, the lengths of the receiving queues of neighboring satellite nodes, the coordinates of the destination satellite of the current data packet, the remaining estimated delay of the neighboring satellite node from the destination satellite node, the remaining energy of the current node, the remaining energy of the neighboring satellite node, and the remaining estimated propagation delay from the neighboring node to the destination node as states.

[0046] The action space module represents the number of neighboring nodes in a fully distributed routing problem. When selecting an action, an action is chosen from the action space based on the ∈-greedy strategy as the routing decision.

[0047] The reward function module constructs the reward function by referring to the objective function;

[0048] The deep reinforcement learning module is used to input the low-dimensional features into the Q-estimation network of the deep reinforcement learning model to calculate the Q-value of each action in the action space; and select an action according to the action space module, the environment reacts to this action, and obtains an immediate reward and the next time step state. The state, routing decision, reward function and the next time step state are stored in the experience pool. A small batch of samples is randomly selected from the experience pool for training to obtain the estimated Q-value and the target Q-value. The parameters of the Q-estimation network and the Q-target network of the deep reinforcement learning model are updated according to the loss function and gradient descent. The above process is repeated until the deep reinforcement learning model converges. The converged deep reinforcement learning model is used to intelligently decide the optimal next hop of the LEO satellite network.

[0049] Optionally, the graph attention network feature extraction module includes a graph module, an input layer, an attention layer, and an output layer;

[0050] The graph module is used to connect satellite nodes and edges to form a network topology graph;

[0051] The input layer consists of a set of node features and produces a new set of node features as output, transforming the input node features into learnable linear features;

[0052] Attention layer: After feature transformation, attention is applied in the network by parameterizing the output of the input layer using a weight matrix. Self-attention is applied to each node by applying this weight matrix. A single-layer feedforward neural network is used as the attention layer, providing normalized attention coefficients.

[0053] The output layer, after obtaining normalized attention coefficients, calculates the feature set corresponding to the coefficients and uses it as the final feature from the network; to stabilize the attention process, multi-head attention is used, applying various independent attention to perform the transformation and connection of the output features.

[0054] The beneficial effects of this invention are as follows:

[0055] 1. To address the challenges of high-dimensionality, high dynamics, high overhead, and non-stationarity caused by local observations, this invention proposes a spatial localization-assisted fully distributed routing algorithm based on multi-agent deep reinforcement learning to achieve efficient data forwarding in large-scale LEO satellite networks. To reduce computational complexity and communication overhead, a partially observable Markov decision process model is established for each satellite, requiring only the transmission of the spatial position, queuing state, and remaining estimated propagation delay of one hop of neighboring satellites. To effectively capture the dynamic characteristics of the satellite network, multi-agent deep reinforcement learning is used to solve the optimization problem through distributed training and distributed execution. To avoid local optimization and non-stationarity issues caused by local observations, the remaining estimated propagation delay is incorporated into the reward function to guide route selection.

[0056] 2. This invention uses graph neural networks to establish the topology of satellite networks as graph structure information. The high dynamism of satellite networks can be described by graph changes, which perfectly matches the powerful generalization ability of graph neural networks.

[0057] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0058] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0059] Figure 1 This is a flowchart illustrating the ubiquitous sensing routing optimization method for low-Earth orbit satellite networks provided in Embodiment 1 of the present invention.

[0060] Figure 2 This is a general framework diagram of the ubiquitous sensing routing optimization method for low-Earth orbit satellite networks provided in an embodiment of the present invention.

[0061] Figure 3 A schematic diagram illustrating the remaining estimated delay provided in an embodiment of the present invention;

[0062] Figure 4 This is a structural block diagram of the ubiquitous sensing routing optimization system for low-Earth orbit satellite networks provided in Embodiment 2 of the present invention. Detailed Implementation

[0063] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0064] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0065] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0066] Example 1

[0067] Existing algorithms mainly include dynamic routing algorithms for LEO satellite networks and energy-efficient routing algorithms for LEO satellite networks. The design of dynamic routing algorithms for LEO satellite networks is often considered from four aspects: First, LEO satellite networks typically consist of thousands of satellites, each with various characteristics that may involve routing choices, leading to high computational complexity and even the "curse of dimensionality." Second, each satellite is constantly moving at high speed, resulting in intermittent connections and frequent changes in network topology. Routing planning algorithms need to accurately capture these dynamic states. Third, especially for large-scale LEO satellite networks, frequently collecting global link states to calculate optimal routing strategies will lead to high communication overhead and significant information acquisition delays. Fourth, due to the limited weight and size of constellation satellites, battery capacity is restricted. Energy conservation is a pressing issue for such large-scale constellations due to the large number of data packets that need to be routed. A well-designed energy system is essential to ensure power supply to satellite components (such as routers) in communication. More importantly, once an older satellite runs out of power, a new satellite needs to be launched, but satellite launches are very costly. Therefore, in response to the problems of diverse user needs and limited satellite battery capacity, this embodiment provides a ubiquitous sensing routing optimization method for low-orbit satellite networks, which can reduce satellite communication latency and energy consumption.

[0068] like Figure 1 As shown, the route optimization methods include:

[0069] Establish a system model and communication model for fully distributed routing in the LEO satellite network;

[0070] Establish energy consumption, propagation delay, and queuing delay models for fully distributed routing in the LEO satellite network;

[0071] The objective function for constructing a LEO satellite network is to minimize energy consumption, propagation delay, and queuing delay.

[0072] Calculate the estimated remaining propagation delay for each data packet to reach the destination satellite via the candidate next hop;

[0073] Construct a state space based on user business requirements, with the lengths of the four transmission queues of the current satellite, the lengths of the receiving queues of neighboring satellite nodes, the coordinates of the destination satellite of the current data packet, the remaining estimated delay of the neighboring satellite node from the destination satellite node, the remaining energy of the current node, and the remaining energy of the neighboring satellite nodes as states;

[0074] The GAT architecture is used to merge the relationship information between each agent, thereby obtaining global hidden features;

[0075] A reward function is constructed for each action in the action space in accordance with the objective function, wherein the action space represents the next hop candidate satellite node of the current satellite, i.e., neighboring satellites;

[0076] At each moment, the agent's interaction with the environment yields a local observation. Using GAT (Global Action Technology), this local observation can be transformed into a global information feature representation by merging agent information. This feature representation is then fed into the Q-estimation network of a deep reinforcement learning model to calculate the Q-value of each action in the action space. The value function of each action is evaluated based on the expected reward, and intelligent decision-making is performed using an ∈-greedy strategy: an action is randomly selected as the next-hop satellite node with probability ∈, and the action with the largest Q-value is selected as the next-hop satellite node with probability 1-∈.

[0077] The environment reacts to this action and receives an immediate reward and the next-step state. The state, routing decision, reward function, and next-step state are stored in an experience pool. A small batch of samples is randomly selected from the experience pool for training to obtain the estimated Q-value and the target Q-value. The parameters of the Q-estimation network and the Q-target network of the deep reinforcement learning model are updated according to the loss function and gradient descent. The above process is repeated until the deep reinforcement learning model converges. The converged deep reinforcement learning model is used to intelligently decide the optimal next hop for the LEO satellite network.

[0078] In this embodiment, as Figure 2As shown, a state space is constructed based on user service requirements, with the lengths of the four transmission queues of the current satellite, the lengths of the receiving queues of neighboring satellite nodes, the coordinates of the destination satellite of the current data packet, the remaining estimated delay between the neighboring satellite node and the destination satellite node, the remaining energy of the current node, and the remaining estimated propagation delay between the neighboring satellite node and the destination node as states. Then, a graph attention neural network model is used to construct a network topology attribute graph using the obtained local LEO low-Earth orbit satellite network information. The graph attention neural network model iteratively extracts the low-dimensional global hidden features of the attribute graph, and then inputs them into a fully connected network to output the Q-values ​​of different routing decisions. An action is selected from the action space as the routing decision according to the ∈-greedy policy. Since each action execution allows transition to the next state and a reward, a quadruple of state, routing decision, reward function, and next-time state is obtained after each action execution. This quadruple is stored in the experience pool, and a small number of quadruples are randomly selected from the experience pool as a batch. This ensures that the training samples are independent and identically distributed, and also makes the sample size of each batch small, which can speed up the training process.

[0079] The target network is fixed, and only the prediction network that interacts with the environment is updated. The difference between the TD-target value calculated by the target network and the output of the prediction network is used to optimize and update the model. After updating the prediction network several times, the target network is replaced with the updated network. The above steps are repeated until the model converges. The converged model is then used to make a decision to obtain the optimal next hop.

[0080] In some feasible schemes, the objective function that minimizes the energy consumption, propagation delay, and queuing delay of the low-Earth orbit satellite network is calculated as follows:

[0081] min cost = (1-μ)·E l (i,j)+μ·t y

[0082]

[0083]

[0084]

[0085] Where μ represents the weighting factor, R i B represents the satellite's remaining energy; B represents the satellite's initial energy, i.e., battery capacity; E l (i, j) represents the energy state of link ij; t u This represents the average delivery delay of the link; n represents the number of neighboring satellite nodes.

[0086] In this embodiment, since inter-satellite communication in the LEO satellite network occurs in free space, it is primarily affected by free space path loss (FSPL) and (thermal) noise power. This noise is assumed to be additive white Gaussian noise (AWGN). i，t v j，t Spatial coordinates can be expressed as (x i，t y i，t , z i，t ) and (x j，t y j，t , z j，t Therefore, their spatial distance can be defined as:

[0087] ||v i，t v j，t ||=|x i，t -x j，t |+|y i，t -y j，t |+|z i，t -z j，t |

[0088] FSPL can be represented as follows:

[0089]

[0090] In the formula, f is the carrier frequency, c is the speed of light, and I * (v i，t v j，t ) is the maximum line-of-sight (LoS) distance.

[0091] Please note that interference can be avoided if the inter-plane ISL antennas in all satellites combine sufficiently narrow beams and precise beam steering or antenna pointing capabilities. Therefore, in an interference-free environment, assuming the radio channel is symmetrical, v i，t You can choose with v j，t The maximum data rate of communication is

[0092]

[0093]

[0094]

[0095] In the formula, P tr and G tr Here, Grc represents the transmit power and antenna gain, and k represents the receive antenna gain. B Where is Boltzmann constant, B is channel bandwidth, T is thermal noise, D represents antenna diameter, and η is... a This indicates the antenna efficiency of the receiver.

[0096] In some possible implementation schemes, the communication delay includes propagation delay and queuing delay;

[0097] In this embodiment, data transmission from the low-Earth orbit satellite (SRC) to the low-Earth orbit satellite (DST) is considered, while the ground end is not considered. The communication delay model of the low-Earth orbit satellite network will be introduced from the following two aspects: propagation delay and queuing delay.

[0098] (1) Propagation delay

[0099] like Figure 1 As shown, each satellite maintains one receive queue and four directed transmit queues. The receive queue is for temporary storage of arriving data packets. The four directed transmit queues, namely the n-link transmit queue, s-link transmit queue, w-link transmit queue, and E-link transmit queue, are associated with one of the four ISLs (In-Link Systems). Data packets in the receive queues are transmitted to one of the four transmit queues according to the routing policy, and then propagated to the next-hop satellite through the corresponding antenna. Each queue follows a first-in, first-out (FIFO) principle.

[0100] From satellite v i，t to v j，t The propagation delay of transmitted data packets is determined by spatial distance. In free space, the link e ij Propagation delay of packet k on It can be defined as:

[0101]

[0102] Here, c is the speed of light. Propagation delay can effectively reflect the spatial length of the transmission path. Therefore, routing algorithms based on propagation delay perform well when network traffic is low.

[0103] (2) Queuing delay

[0104] As traffic load increases, more data packets will queue in the transmission queue due to limited transmission rates. Queuing delay has a more significant impact on overall end-to-end delay than propagation delay. Therefore, this paper also considers queuing delay to achieve load balancing and congestion control. In this context, we assume a time-slot system, with time slots indexed by t. For simplicity, we assume data packets travel on satellite v. i In the N-link transmission queue, the queuing delay follows an M / M / 1 / m queuing model, the probability of arrival at a node follows a Poisson distribution with parameter λ, the node processing follows an exponential distribution with parameter μ, and the maximum queue capacity is m. Therefore, the queuing delay... It can be represented as

[0105]

[0106] qi (t)=min{q i (t-1)+g i (t-1)-u i (t-1), m}

[0107] Where θ is the size of each packet, Ψ is the number of time intervals for transmission, and q i (t) is the time length of the satellite transmission queue, where q i (t-1) is the length of the satellite transmission queue, g i (t-1) and u i (t-1) represents the number of packets received and sent during this time period, respectively.

[0108] Based on the above two points, let's assume the path of data packet k from the source node to the destination node is denoted as D. k =(V′) k E′ k ), where V′ k and E′ k This represents the set of nodes and links through which data packet k passes. Therefore, the total delay for data packet f from its source LEO satellite node src to its destination LEO satellite node dst is:

[0109]

[0110] A satellite contains several components that contribute to energy consumption, including router modules, control modules, and antenna modules. These can be broadly categorized into three parts. The first part is the router module used for packet transmission. The total power consumption of this part depends on the traffic load on the node. Within a time interval Δt, the data transmission speed across the satellite node v... i The total data packet is Pkt i The calculation method is as follows:

[0111]

[0112] The power consumption of buffer queuing and routing table lookup is determined by α. i ·Pkt i Calculate, where α i It's about v i The coefficient. Therefore, the total energy consumed can be obtained by processing these data packets:

[0113]

[0114] Where P s and P r These represent the transmission power of the sent and received packets, respectively. This is the power consumed by sending data packets to the satellite link (i, j); The first part is the power consumed by receiving data packets from the satellite link (j, i). The second part is the power consumed during normal operation, which is a constant P. norm The third part is the power consumption of the processor, which can be used... Calculate, where μ i and σ i It is a constant. It is also positively correlated with traffic flow.

[0115] Focusing our attention on the satellites located within the eclipse zone, we divide continuous time into discrete slots to maintain the stability of the satellite routing strategy. Within each time slot, the routing strategy for each satellite remains unchanged. From the above equation, we can derive the following energy consumption for satellite operation:

[0116]

[0117] In some possible implementation schemes, a network topology attribute graph is constructed using the obtained local LEO low-Earth orbit satellite network information. A graph attention neural network model is then used to iteratively extract low-dimensional global hidden features from this attribute graph. Specifically:

[0118] The state of the low-Earth orbit satellite network is obtained, including the length of the four transmission queues of the current satellite, the length of the receiving queue of the neighboring satellite nodes, the coordinates of the destination satellite of the current data packet, the remaining estimated delay of the neighboring satellite nodes from the destination satellite node, the remaining energy of the current node, the remaining energy of the neighboring satellite nodes, and the remaining estimated propagation delay from the neighboring nodes to the destination node. These states are combined by associating them with different elements of the input graph and considering how these elements are connected in the graph. The state of the elements is updated by an iterative message passing algorithm, and the output is generated using the resulting state.

[0119] The messages computed for the same node and its neighbors are aggregated using element-wise summation. The hidden state is updated by aggregating the aggregated information. The same process is repeated for all linked neighbors to obtain the global hidden information through iteration.

[0120] Firstly, based on the transformation process from feature quantity F to feature quantity F', only a feature transformation matrix W is needed to complete this feature transformation. Therefore, we first define a weight matrix W∈R. F’*F This is used to complete the feature transformation process for all nodes. The transformation formula is:

[0121] z j =Wh j

[0122] Secondly, an attention mechanism is introduced. In GAT, a self-attention mechanism is used, employing a shared attention calculation function α, the formula of which is:

[0123] eij =α(W(h) i |h j ))

[0124] This represents the contribution of node j's features to node i. Throughout the calculation, the contribution of each neighbor node k of node i to i needs to be calculated. Note that "||" indicates vector concatenation.

[0125] Then, this weight is converted into a corresponding weight, which is to normalize the contribution of each neighboring node k. The calculation formula is as follows:

[0126]

[0127] For linear layer operations, a non-linear activation function is generally applied. This paper uses the LeakyReLU activation function with a slope of 0.2. The final attention calculation formula is then...

[0128]

[0129] Finally, after calculating the contribution of each neighboring node of node i, the features of all neighboring nodes of node i are summed and updated according to their weights. The final output of node i is calculated using the following formula:

[0130]

[0131] In some possible implementation schemes, the reward function is calculated as follows:

[0132]

[0133]

[0134]

[0135]

[0136] E i (t)=E o +E s +E r =(P o +P s +P r )×τ

[0137] G i (t)=P c (t)×max{0,(τ-T i )}

[0138] Where ω1, ω2, ω3 represent weighting factors, and τy Indicates the total transmission delay. Indicates queuing delay. E represents the propagation delay. l R represents the energy of link i,j. i (t) represents the remaining link energy of satellite i, E i (t) represents the energy consumption of satellite i, G i (t) represents the energy gained by satellite i.

[0139] Specifically, in this embodiment, referring to the calculation formula of the objective function in the above embodiments, the reward function is obtained based on the optimization problem of the objective function.

[0140] 5. In some implementations, an action is selected from the action space as the routing decision based on the ∈-greedy policy, including:

[0141] Calculate the probability of each action in the action space, randomly select an action from the action space with probability ∈ as the routing decision, and select the action with the largest Q value from the action space with probability 1-∈ as the routing decision.

[0142] In some implementations, since each action transitions to the next state and receives a reward, a four-tuple (state, routing decision, reward function, and next-time state) is obtained after each action. This four-tuple is stored in an experience pool, and a small number of four-tuples are randomly selected from the experience pool as a batch. This ensures that the training samples are independent and identically distributed, while also keeping the sample size of each batch small, thus accelerating training. Specifically:

[0143] A batch of samples is randomly selected from the experience pool for training to break the correlation between data. The current state information is used as the input of the Q estimation network, and the output is obtained through three fully connected layers, which is the Q value of the routing decision taken in this state. The next state information is used as the input of the Q estimation network, and the output is obtained through three fully connected layers, which is the action when the Q value of the Q estimation network reaches its maximum. This action and the next state are used as the input of the Q target network to output the target Q value.

[0144] In some implementations, the target network is fixed, and only the prediction network that interacts with the environment is updated. The difference between the TD-target value calculated by the target network and the output of the prediction network is used to optimize and update the model. After updating the prediction network several times, the target network is replaced with the updated network. The above steps are repeated until the model converges, and the converged model is used to make a decision to obtain the optimal next hop. Specifically:

[0145] The variance between the target value and the predicted value is called the loss function. The loss is minimized by updating the weights, and then the gradient is calculated based on the loss function.

[0146] After every k steps of updates, copy the actual network weights to the target network weights, and repeat the above steps until the model converges.

[0147] Specifically, in this example, agent i forwards the current packet k to the corresponding neighbor agent based on the selected action, and calculates the reward r. i，t Agent i's current state si,t will be transitioned to the next state s. i，t+1 After receiving feedback from this information, agent i will perform this transformation (s i，t a i，t r i，t s i，t+1 The data is recorded in the replay memory R, ​​and then a random batch of samples is obtained for learning, thereby breaking the correlation of the training data. This is achieved by minimizing the objective Q-value y. i，t Update Q using the mean square error between the current Q network output and the current Q network output. i (θ i，t The parameters of the loss function Li,t are...

[0148] L i，t =(y j，t -Q i ( si，t a i，t ;θ i，t )) 2

[0149]

[0150] Where y i，t The target value is γ, which is the discount factor, and θ is the θ value. i，t It is the Q-estimation network parameter. These are the parameters of the target network in Q. Target network parameters Use Q to estimate network parameters θ in each k-step. i，t The parameters θ of the network can be updated and remain constant between updates. i，t The update is as follows:

[0151]

[0152] Where α is the learning rate.

[0153] Example 2

[0154] like Figure 4 As shown, Embodiment 2 of this application provides a fully distributed ubiquitous sensing routing optimization system for low-Earth orbit satellite networks, based on Embodiment 1, comprising:

[0155] A system model module is built to establish communication models, propagation delay models, queuing delay models, and energy consumption models for low-Earth orbit satellite networks.

[0156] The objective function establishment module is used to establish an objective function that minimizes the energy consumption, propagation delay, and queuing delay of the low-Earth orbit satellite network.

[0157] The remaining estimated propagation delay module is used to calculate the estimated propagation delay from the satellite where the current data packet is located to the destination node.

[0158] The graph attention network feature extraction module is used to construct a network topology attribute graph using the obtained local LEO low-Earth orbit satellite network information, and to continuously extract low-dimensional global hidden features of the attribute graph using a graph attention neural network model.

[0159] The state space module is used to construct a state space based on user business requirements, with the lengths of the four transmission queues of the current satellite, the lengths of the receiving queues of neighboring satellite nodes, the coordinates of the destination satellite of the current data packet, the remaining estimated delay of the neighboring satellite node from the destination satellite node, the remaining energy of the current node, the remaining energy of the neighboring satellite node, and the remaining estimated propagation delay from the neighboring node to the destination node as states.

[0160] The action space module represents the number of neighboring nodes in a fully distributed routing problem. When selecting an action, an action is chosen from the action space based on the ∈-greedy strategy as the routing decision.

[0161] The reward function module constructs the reward function by referring to the objective function;

[0162] The deep reinforcement learning module is used to input the low-dimensional features into the Q-estimation network of the deep reinforcement learning model to calculate the Q-value of each action in the action space; and select an action according to the action space module, the environment reacts to this action, and obtains an immediate reward and the next time step state. The state, routing decision, reward function and the next time step state are stored in the experience pool. A small batch of samples is randomly selected from the experience pool for training to obtain the estimated Q-value and the target Q-value. The parameters of the Q-estimation network and the Q-target network of the deep reinforcement learning model are updated according to the loss function and gradient descent. The above process is repeated until the deep reinforcement learning model converges. The converged deep reinforcement learning model is used to intelligently decide the optimal next hop of the LEO satellite network.

[0163] In some implementations, the graph attention network feature extraction module includes a graph module, an input layer, an attention layer, and an output layer; the graph module is used to connect satellite nodes and edges to form a network topology graph;

[0164] The input layer consists of a set of node features and should be able to produce a new set of node features as output. These layers should also be able to transform the input node features into learnable linear features.

[0165] Attention layers, after feature transformation, can be applied to the network. They can be parameterized using the weight matrix of the input layer's output. By applying this weight matrix to each node, self-attention can be applied to that node. Mechanically, this can be implied as a single-layer feedforward neural network acting as the attention layer, which can be given a normalized attention coefficient.

[0166] The output layer, after obtaining normalized attention coefficients, can use them to compute the feature set corresponding to the coefficients, and use these as the final features from the network. To stabilize the attention process, multi-head attention can be used, so that various independent attention mechanisms can be applied to perform the transformation and concatenation of the output features.

[0167] It should be understood that the functions implemented by each module provided in this embodiment two have been described in each method step of embodiment one, so there will be no further description.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A fully distributed routing method based on deep reinforcement learning, characterized in that: The method comprises the following steps: Step 1: establishing a system model and a communication model of the fully distributed routing in the LEO satellite network; Step 2: establishing an energy consumption, propagation delay and queuing delay model of the fully distributed routing in the LEO satellite network; Step 3: constructing an objective function of energy consumption, propagation delay and queuing delay minimization of the LEO satellite network; the calculation formula is: wherein, represents a weighting factor, represents a satellite residual energy; B represents a satellite initial energy, i.e. battery capacity; represents a link ij energy status; represents a link average delivery latency; n represents a number of neighbor nodes of a satellite; represents a residual energy of a satellite j , represents a set of data packets, k represents a data packet index, represents a weight coefficient for adjusting the contribution of the queuing latency term in the latency calculation; represents a weight coefficient for adjusting the contribution of the propagation latency term in the latency calculation; v i represents a node i in a satellite network, represents a set of nodes on the path from a source to a destination of a data packet k , represents a queuing latency of a data packet k at a node t at time i ; represents an edge or link i in a satellite network, represents a set of links on the path from a source to a destination of a data packet k , represents a propagation latency of a data packet k on a link t at time ij ; represents a sum of residual energies of all neighbor satellites R i The propagation latency is the time taken by the current satellite to propagate to the next hop; the queuing latency is the time for a data packet to queue in the input queue and the output queue of a satellite; the energy consumption is the energy consumed by a satellite to forward data, including the energy consumed for data transmission, data reception, and the energy consumed for the satellite to maintain normal operation. Step 4: calculating the estimated residual propagation delay of each data packet to the next hop to the destination satellite; Step 5: constructing a state space with the current satellite four sending queue lengths, the neighbor satellite node receiving queue lengths, the destination satellite coordinates of the current data packet, the residual estimated delay of the neighbor satellite node to the destination satellite node, the current node residual energy and the neighbor satellite node residual energy as states according to the user service demand; Step 6: using the GAT architecture to combine the relationship information between each agent, so as to obtain the global hidden feature; specifically, the states in the state space are obtained, wherein the states include the current satellite four sending queue lengths, the neighbor satellite node receiving queue lengths, the destination satellite coordinates of the current data packet, the residual estimated delay of the neighbor satellite node to the destination satellite node and the current node residual energy and the neighbor satellite node residual energy, the states are associated with different elements of the input graph, the states of the elements are updated through the iterative message passing algorithm, and the output is generated using the result state; the messages calculated for the same node and its neighbors are aggregated using element summation, the hidden state is updated through the aggregated information, and the same process is repeated to obtain the global hidden information through iteration for all linked neighbors; Step 7: constructing a reward function corresponding to each action in the action space according to the objective function, wherein the action space represents the next hop candidate satellite node of the current satellite, that is, the neighbor satellite; the reward function is: wherein , denotes a weighting factor, denotes the total transmission delay, denotes the queuing delay, denotes the propagation delay, denotes the link i , j energy, denotes the link residual energy of the satellite i at the time instant t , denotes the energy consumption of the satellite i at the time instant t , denotes the energy harvested by the satellite i at the time instant t ; denotes a distance-dependent cost term, denotes the energy consumption for maintaining normal operation of the satellite, denotes the energy consumed for data transmission by the satellite, denotes the energy consumed for data reception by the satellite, denotes the normal operation power of the satellite, denotes the data transmission power of the satellite, denotes the data reception power of the satellite, denotes the time slot length, denotes the charging power of the satellite i at the time instant t , denotes a time threshold; Step 8: At each time, the agent interacts with the environment to obtain a local observation, and the local observation is converted into a global information feature representation by merging the agent information through the GAT, and the obtained feature representation is input into the Q estimation network of the deep reinforcement learning model to calculate the Q value of each action in the action space; the value function of each action is evaluated based on the expected return, and the intelligent decision is made through the greedy strategy, that is, an action is randomly selected as the next hop satellite node with a probability probability, and the action with the maximum Q value is selected as the next hop satellite node with a probability probability. Step 9: the environment reacts to the action and obtains an instant reward and a next time state, and the state, the routing decision, the reward function and the next time state are stored in an experience pool, a small batch of samples batch are randomly selected from the experience pool for training, estimated Q values and target Q values are obtained, parameters of a Q estimation network and a Q target network of the deep reinforcement learning model are updated according to a loss function and gradient descent, and the above process is continuously cycled until the deep reinforcement learning model converges, and the converged deep reinforcement learning model is used to intelligently decide the optimal next hop of the LEO satellite network. 2.The fully distributed routing method based on deep reinforcement learning according to claim 1, characterized in that: The routing decision is based on The policy selects an action from the action space, comprising: probabilities of each action in the action space are calculated, and a probability is randomly selected from the action space as the routing decision, and the action with the maximum Q value is selected from the action space as the routing decision. 3.The fully distributed routing method based on deep reinforcement learning according to claim 1, wherein: In the step 9, a four-tuple of the state, the routing decision, the reward function and the next time state is obtained after each action is executed, and each time the four-tuple obtained is stored in the experience pool, and the four-tuple is randomly extracted from the experience pool as a batch, specifically: A batch of samples are randomly selected from the experience pool for training, breaking the correlation between data, taking the state information at the current time as the input of the Q estimation network, obtaining the output through three fully connected layers, and taking the next time state information as the input of the Q estimation network, obtaining the output through three fully connected layers, and taking the action when the Q value of the Q estimation network reaches the maximum as the input of the Q target network, and outputting the target Q value; The target network is fixed, and only the prediction network interacting with the environment is updated; the difference between the TD-target value calculated by the target network and the prediction network output is used to optimize and update the model; after updating the prediction network several times, the updated network is used to replace the target network; repeat the above steps until the model converges, and use the converged model to make decisions to obtain the optimal next hop; specifically: The variance between the target value and the prediction value is called the loss function, which is updated by reducing the loss as much as possible, and then the gradient is calculated according to the loss function; Every update k After the step, copy the actual network weights to the target network weights, repeat the above steps until the model converges.

4. A fully distributed routing system based on deep reinforcement learning, characterized in that: The system comprises: The system model module is used to establish a communication model of the low-orbit satellite network, a propagation delay model, a queuing delay model and an energy consumption model; The target function establishment module is used to establish a target function for minimizing the energy consumption, propagation delay and queuing delay of the low-orbit satellite network; the calculation formula is: wherein, represents a weighting factor, represents a satellite residual energy; B represents a satellite initial energy, i.e. battery capacity; represents an energy state of a link ij ; represents an average delivery latency of a link; n represents a number of neighbor satellites of a satellite; represents a residual energy of a satellite j , represents a set of data packets, k represents a data packet index, represents a weight coefficient for adjusting the contribution of the queuing latency term in the latency calculation; represents a weight coefficient for adjusting the contribution of the propagation latency term in the latency calculation; v i represents a node i in a satellite network, represents a set of nodes on the path from a source to a destination of a data packet k , represents a queuing latency of a data packet k at a node t at time i , represents an edge or link i in a satellite network, represents a set of links on the path from a source to a destination of a data packet k , represents a propagation latency of a data packet k on a link t at time ij ; represents a sum of residual energies of all neighbor satellites R i The propagation latency is the time needed for the current satellite to propagate to the next hop; the queuing latency is the time for a data packet to queue in the input queue and the output queue of a satellite; the energy consumption is the energy consumed by a satellite to forward data, including the energy consumed for data transmission, data reception, and the energy consumed for the satellite to maintain normal operation. The residual estimated propagation delay module is used to calculate the estimated propagation delay from the next hop of the satellite where the current data packet is located to the destination node; The graph attention network feature extraction module uses the GAT architecture to combine the relationship information between each agent to obtain global hidden features; specifically: the states in the state space are obtained, wherein the states include the lengths of the four sending queues of the current satellite, the lengths of the receiving queues of the neighbor satellite nodes, the coordinates of the destination satellite of the current data packet, the residual estimated delay of the neighbor satellite nodes from the destination satellite node, and the residual energy of the current node and the residual energy of the neighbor satellite nodes; by associating these states with different elements of the input graph, updating the state of the elements through an iterative message passing algorithm, and using the result state to generate an output; the messages calculated for the same node and its neighbors are aggregated using element summation, the hidden state is updated through the aggregated information, and the same process is repeated to obtain global hidden information through iteration for all linked neighbors; The state space module is used to construct a state space with the lengths of the four sending queues of the current satellite, the lengths of the receiving queues of the neighbor satellite nodes, the coordinates of the destination satellite of the current data packet, the residual estimated delay of the neighbor satellite nodes from the destination satellite node, the residual energy of the current node and the residual energy of the neighbor satellite nodes as states according to user business requirements; Action space module, the action space of fully distributed routing problem is the number of neighbor nodes, and the action selection is based on the strategy selects an action from the action space as the routing decision; The reward function module constructs a reward function according to the target function; the reward function is: wherein , denotes a weighting factor, denotes the total transmission delay, denotes the queuing delay, denotes the propagation delay, denotes the link i , j energy, denotes the link residual energy of the satellite i at the time instant t , denotes the energy consumption of the satellite i at the time instant t , denotes the energy harvested by the satellite i at the time instant t ; denotes a distance-dependent cost term, denotes the energy consumption for maintaining normal operation of the satellite, denotes the energy consumed for data transmission by the satellite, denotes the energy consumed for data reception by the satellite, denotes the normal operation power of the satellite, denotes the data transmission power of the satellite, denotes the data reception power of the satellite, denotes the time slot length, denotes the charging power of the satellite i at the time instant t , denotes a time threshold; The deep reinforcement learning module is configured to input the low-dimensional features obtained by the graph attention network feature extraction module into a Q estimation network of a deep reinforcement learning model, calculate Q values of each action in the action space, select an action according to an action space module, and make the environment react to the action to obtain an instant reward and a next time state, store the state, the routing decision, a reward function and the next time state into an experience pool, randomly select a small batch of samples from the experience pool for training, obtain estimated Q values and target Q values, update parameters of the Q estimation network and a Q target network of the deep reinforcement learning model according to a loss function and gradient descent, and repeatedly perform the above process until the deep reinforcement learning model converges, and use the converged deep reinforcement learning model to intelligently determine an optimal next hop of the LEO satellite network.

5. The fully distributed routing system based on deep reinforcement learning according to claim 4, characterized in that: The graph attention network feature extraction module comprises a graph module, an input layer, an attention layer and an output layer. The graph module is configured to connect satellite nodes and edges to form a network topology graph. The input layer is composed of a group of node features, and generates a group of new node features as output, and converts the input node features into learnable linear features. The attention layer is applied in the network after the conversion of the features, and is parameterized by using the output of the input layer through a weight matrix, and applies self-attention to the nodes by applying the weight matrix to each node. The output layer calculates a feature set corresponding to the normalized attention coefficients after obtaining the normalized attention coefficients, and takes the feature set as the final features from the network. In order to stabilize the attention process, multi-head attention is used, and various independent attentions are applied to perform the conversion and connection of the output features.

Images