Satellite network multipath routing method based on deep reinforcement learning
By applying a multi-path routing method of deep reinforcement learning in satellite networks, the convolutional dual-delay deep deterministic strategy gradient algorithm is used to optimize traffic allocation, and the problem of load imbalance in satellite networks is solved, load balancing and rapid model convergence are achieved, and chains are reduced Road congestion and transmission delay.
Patent Information
- Application Number
- CN202411277110.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-12-03
AI Technical Summary
Due to the uneven distribution of global users and unbalanced traffic density, the load imbalance is caused by satellite networks. Traditional multi-path routing algorithms cannot effectively alleviate link congestion, and the deep reinforcement learning model training is complex and has poor convergence.
Using a multi-path routing method based on deep reinforcement learning, the convolutional dual-delay deep deterministic strategy gradient algorithm (CTD3) collects inter-star link information under the software-defined network (SDN) structure, and trains the intelligent body through convolutional neural network. , output link weights and calculate the shortest weighted path using Freud algorithm, introduce the maximum link bandwidth utilization reward function, optimize traffic allocation, and realize load balancing.
The load balancing of satellite networks is realized, the model convergence speed is improved, link congestion and transmission delay are reduced, and the real-time control capability of the network is improved.
Smart Images

Figure CN119070889A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of satellite network routing and deep reinforcement learning, and specifically to a satellite network multipath routing method based on deep reinforcement learning. Background Art
[0002] Satellite networks have attracted much attention due to their wide coverage, lack of impact from ground disasters, and ability to achieve ubiquitous communications. Compared with satellites in geosynchronous orbit and medium earth orbit, low-orbit satellites have lower propagation delay and propagation loss. Low-orbit satellites can better provide data transmission for global users. Routing is the technology responsible for transmission path selection and data forwarding between satellites. However, due to the uneven distribution of global users and the imbalanced distribution of traffic density at different times, the satellite load is unbalanced and link congestion occurs, thus affecting data transmission efficiency. The above problems have triggered discussions and research on load balancing routing algorithms for satellite networks.
[0003] The proposed load balancing routing algorithms are divided into two categories: single-path and multi-path. The single-path load balancing routing algorithm selects the path with the lowest link cost for transmission according to the network load status or migrates the traffic to other paths with the lowest link cost when the traffic is too large. However, if all traffic is concentrated on one path, it may cause a link in the path to be overloaded and other links to be idle. Therefore, the single-path load balancing routing algorithm cannot achieve load balancing for the entire network. Multi-path routing disperses the transmission data by establishing multiple paths between the source and destination nodes. It alleviates congestion, fault tolerance and improves throughput by making full use of idle network resources. Therefore, it is an effective method to achieve network load balancing. However, the traditional multi-path routing algorithm finds multiple shortest paths to evenly distribute the traffic for transmission. Evenly distributing the traffic on multiple paths for transmission may cause one of the paths to be overloaded and cause link congestion, which cannot solve the problem of network load imbalance.
[0004] The application of deep reinforcement learning to satellite network routing has been a research hotspot in recent years. Deep reinforcement learning methods can capture multi-dimensional information of the network and greatly improve the real-time control capability of the network. However, deep reinforcement learning models are usually more complex and require a lot of computing resources and time for training. In addition, the policy gradient method in deep reinforcement learning may cause the gradient to disappear or explode, affecting the convergence of the model. Summary of the invention
[0005] To sum up, in order to solve the problem of satellite network load imbalance, the present invention proposes a satellite network multipath routing method based on deep reinforcement learning to achieve satellite network load balancing and rapid convergence of the deep reinforcement learning model.
[0006] The technical solution adopted by the present invention is as follows:
[0007] A satellite network multipath routing method based on deep reinforcement learning comprises the following steps:
[0008] A. Collecting intersatellite link information
[0009] In the software defined network (SDN) structure, the data plane is composed of low-orbit satellites, and the control plane is composed of controllers. The controller is an SDN controller, and the controller is installed with a convolutional double-delay deep deterministic policy gradient algorithm (CTD3), and the convolutional double-delay deep deterministic policy gradient algorithm is referred to as an agent;
[0010] The controller collects intersatellite link information of low-orbit satellites as state input to the agent, and the steps are as follows:
[0011] A1. The controller collects inter-satellite link information of low-orbit satellites, where the inter-satellite link information includes link remaining bandwidth, transmission delay, packet loss rate, and time-space level;
[0012] A2. The controller inputs the link packet loss rate matrix, the spatiotemporal level matrix, the normalized residual bandwidth matrix, the transmission delay matrix, and the traffic matrix into the agent as the state space S; the neural network in the agent adopts a convolutional neural network;
[0013] B. Training the Agent
[0014] B1. The action output by the agent is used as the link weight
[0015] The action is expressed as action = [ω1, ω2, ···, ω m ,···,ω E ], where ω m represents the weight of the mth link, |E| represents the number of visible links in the network, and is assigned to the links in set E in turn.
[0016] B2. Use the Floyd algorithm to calculate the two shortest weighted paths between any source and destination nodes according to the link weights as the path set P, and then use the path weight ratio as the traffic distribution ratio to split the traffic according to the traffic distribution ratio.
[0017] B3. Introducing the reward function into the maximum link bandwidth utilization rate represents the load of the link. The smaller the maximum link bandwidth utilization rate, the more balanced the satellite network load is.
[0018] B4. Training the Neural Network in the Agent
[0019] The agent includes 6 neural networks, namely Actor network, TargetActor network, Critic1 network, Critic2 network, TargetCritic1 network and TargetCritic2 network, where Actor network is denoted as π φ , parameter is φ, TargetActor network is π′ φ , the parameter is φ′, the Critic1 network and the Critic2 network are and The parameters are θ1 and θ2, and the Target Critic1 network and Target Critic2 network are and The parameters are θ1′ and θ2′ respectively.
[0020] C. Output multipath routing strategy
[0021] After the training of the intelligent agent converges, a multipath routing strategy is generated according to the final output traffic distribution ratio. The controller forwards the multipath routing strategy to the low-orbit satellite node to realize multipath transmission in the low-orbit satellite network.
[0022] Furthermore, the method for the controller in step A1 to collect inter-satellite link information of low-orbit satellites comprises the following steps:
[0023] A11. Use the satellite toolkit (STK) to obtain satellite parameters, and then analyze the change pattern of link distance. The simulation time is one day. Since the distance between satellites on the same orbit changes very little, the distance between satellites on different orbits changes periodically and has a regular change over time. The model of the distance between adjacent orbits changing over time is as follows:
[0024]
[0025] In the formula, r u,v (t) represents the distance between satellite u and satellite v at time t, B is half of the difference between the maximum and minimum distances between adjacent orbits, A is the difference between B and the minimum distance between adjacent orbits, T is the period of distance change between adjacent orbits, is the initial phase.
[0026] A12. Use the visibility matrix G t represents the satellite network topology that changes with time t; V represents the set of satellites, E represents the set of links between satellites, E = {e(u,v)}, u≠v, u∈V, v∈V, where u∈[1,n], v∈[1,n], and n represents the total number of satellites. e(u,v) represents the link between satellite u and satellite v, and satellite v is a visible neighbor node of satellite u, then the visibility matrix G t It is expressed as follows:
[0027]
[0028] Among them, r u ' ,v (t) indicates whether satellite u and satellite v are visible at time t. u ' ,v When (t) = 0, it means that satellite u and satellite v are not visible at time t; when r u ' ,v When (t)≠0, it means that satellite u and satellite v are visible at time t; and r u ' ,v (t) = r u,v (t).
[0029] A13, calculating the link transmission delay; the link transmission delay d uv (t) includes propagation delay and average queuing delay The formula is as follows:
[0030]
[0031] In the formula, c represents the speed of light, num represents the number of packets in the queue at time t, and pk avg Indicates the average packet length.
[0032] The remaining bandwidth b of link e(u,v) uv The bandwidth capacity bw of the link uv Subtract the used bandwidth of the link Get, expressed as follows:
[0033]
[0034] The used bandwidth of the link is obtained from the flow rate of the satellites at both ends of the link and is calculated as follows:
[0035]
[0036] The flow rate of satellite u is based on the total number of data bytes received within a period of time ΔT. and the total number of bytes transferred The formula is as follows:
[0037]
[0038] Similarly, the velocity of satellite v is obtained:
[0039]
[0040] The time and space level of the link is the maximum time and space level of the satellite nodes at both ends of the link. The time and space level of the satellite node considers the traffic demand of the coverage area from both time and space aspects; the time level of satellite i at time t is Y i , i∈[1,n], the spatial level of the area covered by satellite i is TD i , the spatiotemporal level of traffic in the area covered by satellite i i The calculation formula is as follows:
[0041]
[0042] The spatiotemporal level of link e(u,v) uv Set to the value with the larger spatiotemporal level between satellite u and satellite v, calculated as follows:
[0043] level uv =max{level u ,level v} (11)
[0044] In a satellite network, there is a certain amount of data traffic between satellites. uv represents the flow between satellites u and v, and the flow is normalized using the following formula:
[0045]
[0046] In the formula, f u ' v represents the normalized flow rate between satellites u and v.
[0047] Use formulas (13) and (14) to normalize the link residual bandwidth and transmission delay respectively:
[0048]
[0049] Where b u ' v represents the normalized link residual bandwidth, d u ' v Indicates the normalized link transmission delay.
[0050] l uv It represents the packet loss rate of link e(u,v), and is calculated as follows:
[0051]
[0052] In the formula, pk v Indicates the number of packets received by satellite v, pk u Indicates the number of data packets sent by satellite u.
[0053] Furthermore, the state space S in step A2 is expressed as follows:
[0054] S = [FBDL Lev] (16)
[0055] In the formula, F represents the traffic matrix, B represents the remaining bandwidth matrix of the satellite network, D represents the transmission delay matrix of the satellite network, L represents the packet loss rate matrix of the satellite network, and Lev represents the spatiotemporal level matrix of the satellite network. The specific expressions of each matrix are as follows:
[0056]
[0057]
[0058] Furthermore, the method of introducing the reward function into the maximum link bandwidth utilization in step B3 is as follows:
[0059] The maximum link bandwidth utilization RU is calculated as follows:
[0060]
[0061] The weighted sum of the maximum link bandwidth utilization, the satellite network packet loss rate, and the average end-to-end delay (i.e., the average value of the transmission delay from the source to the destination and the transmission delay from the destination to the source) is used as the reward function to adjust the action until the reward value converges to the interval [0.6, 0.7]. The reward function is expressed as follows:
[0062]
[0063] Where D avg , Loss represent the average end-to-end delay and packet loss rate of the satellite network, respectively; α1, β1, and λ1 are the weight parameters of the maximum link bandwidth utilization, average end-to-end delay, and satellite network packet loss rate when the satellite network load is relatively balanced; α2, β2, and λ2 are the weight parameters of the maximum link bandwidth utilization, average end-to-end delay, and satellite network packet loss rate when the satellite network load is relatively unbalanced; α3, β3, and λ3 are the weight parameters of the maximum link bandwidth utilization, average end-to-end delay, and satellite network packet loss rate when the satellite network load is unbalanced. Use RU as the segmentation condition and set different weight parameters for each segment function to increase the penalty for poor actions.
[0064] Average end-to-end delay D avg is the average transmission delay of the path set P, and the calculation formula is as follows:
[0065]
[0066] Using formula (25) for D avgNormalized, where D max represents the maximum path delay in the path set P, D min represents the minimum path delay in the path set P, and the formula is as follows:
[0067]
[0068] In the formula, Represents the normalized average end-to-end delay.
[0069] The total number of packets received by the destination is denoted as f accept , the calculation formula is as follows:
[0070]
[0071] Network packet loss rate L sum The calculation formula is as follows:
[0072]
[0073] Use formula (28) to normalize the network packet loss rate L sum , where L max , L min They represent the maximum and minimum packet loss rates of the shortest weighted path in the network respectively.
[0074]
[0075] Where Loss represents the normalized network packet loss rate.
[0076] Furthermore, the steps of training the neural network in the agent in step B4 are as follows:
[0077] B41. In the sampling phase, the state s is input into the Actor network, the action a is output as the link weight, the action is executed to obtain the next state s′ and reward value r, and experience = (s, a, r, s′) is stored as a historical experience in the experience playback buffer.
[0078] B42. In the training phase, N data are sampled from the experience replay buffer pool. Assuming that one data experience = (s, a, r, s′) is sampled, the update process of all networks is as follows:
[0079] B421. Use the TargetActor network to calculate the action a′ in state s′. Based on the target strategy smoothing regularization, add exploration noise to a′:
[0080]
[0081] where ε is the exploration noise, clip is the clipping function, is the random noise function, σ represents the standard deviation, and c is the limit of the clipping function.
[0082] B422. Calculate the target value:
[0083]
[0084] Where y represents the target value and γ represents the discount factor.
[0085] B423. Calculate the evaluation value The error between the target value y is:
[0086]
[0087] B424, minimize the average error between the evaluation value and the target value to update the parameters θ1 and θ2 in the Critic1 network and the Critic2 network:
[0088]
[0089] B425, after Critic1 and Critic2 networks are updated d steps, the Actor network is used to calculate the action a under state s, and then the Critic1 network is used to calculate the evaluation value Using the gradient ascent algorithm to maximize Complete the update of the Actor network parameter φ:
[0090]
[0091] B426. Finally, the target network is updated by soft update. An update rate τ is introduced to make a weighted average of the old target network parameters and the new corresponding network parameters, and then assigned to the target network:
[0092] θ i ′←τθ i +(1-τ)θ i ′,i=1,2 (34)
[0093] φ′←τφ+(1-τ)φ′(35).
[0094] Compared with the prior art, the present invention has the following beneficial effects:
[0095] 1. The present invention takes into account the time and space levels of the link, and can avoid selecting links with heavy loads and try to select idle links, which is conducive to achieving load balancing of the satellite network.
[0096] 2. The neural network of the intelligent agent in the present invention adopts a convolutional neural network, which can greatly improve the convergence speed of the model.
[0097] 3. The present invention introduces the maximum link bandwidth utilization into the reward function of the intelligent agent to guide the selection of links with low bandwidth utilization, which can avoid link congestion, reduce transmission delay, and further achieve satellite network load balancing. BRIEF DESCRIPTION OF THE DRAWINGS
[0098] Figure 1 This is the SDN architecture diagram of the present invention.
[0099] Figure 2 It is the training flowchart of the intelligent agent.
[0100] Figure 3 This is the 24h traffic demand map of the satellite network.
[0101] Figure 4 It is the density map of user traffic distribution on Earth.
[0102] Figure 5 This is a comparison chart of traffic distribution index between the present invention and other satellite network routing methods under different traffic intensities. DETAILED DESCRIPTION
[0103] The present invention will be further described below in conjunction with the accompanying drawings. Figure 1 In the SDN architecture shown in the figure, the data plane is composed of low-orbit satellites, and the control plane is composed of controllers. The controller is equipped with an intelligent agent, namely the Convolutional Twin Delayed Deep Deterministic Policy Gradient (CTD3). The controller is deployed on a high-orbit satellite and is responsible for collecting information and forwarding routing strategies. The intelligent agent uses the input network status and traffic information, and after training and convergence, outputs actions as link weights to generate multi-path routing strategies.
[0104] The agent CTD3 includes 6 neural networks, and the Actor network is denoted as π φ , the parameter is φ, and the TargetActor network is π′ φ , the parameter is φ′, the Critic1 network and the Critic2 network are and The parameters are θ1 and θ2, and the Target Critic1 network and Target Critic2 network are and The parameters are θ1′ and θ2′ respectively.
[0105] The Actor network and the TargetActor network have the same neural network structure. The input state s is passed through three layers of convolution, and then nonlinearly transformed by the activation function, flattened to vectorize the features, two fully connected layers and activation functions, and finally outputs the action a as an array.
[0106] Critic1 network, Critic2 network, Target Critic1 network and Target Critic2 network have the same neural network structure. The input state s and action a are transformed through three layers of convolution, and then nonlinearly transformed through activation functions, flattened to vectorize the features, two fully connected layers and activation functions, and finally output an action value function Q(s,a).
[0107] The controller implements multipath routing in three stages, as follows.
[0108] Phase 1: Collect link information.
[0109] Step 1: The controller collects inter-satellite link information of low-orbit satellites, wherein the inter-satellite link information includes link remaining bandwidth, transmission delay, packet loss rate and time-space level;
[0110] Step2: The controller takes the link packet loss rate matrix, spatiotemporal level matrix, normalized residual bandwidth matrix, transmission delay matrix, and traffic matrix as state space inputs to CTD3.
[0111] The second stage is to train the agent.
[0112] Step 1: The action output by the agent is used as the link weight
[0113] Step 2: Use the Floyd algorithm to calculate the two shortest weighted paths between any source and destination nodes as the path set P according to the link weights, and then use the path weight ratio as the traffic distribution ratio to split the traffic according to the traffic distribution ratio.
[0114] Step 3: Introduce the reward function into the maximum link bandwidth utilization to represent the link load. The smaller the maximum link bandwidth utilization, the more balanced the satellite network load.
[0115] Step 4: Train the neural network in the agent. The specific process is as follows Figure 2 shown.
[0116] Step 41: In the sampling phase, the information of the first phase is input as the state s into the Actor network, the action a is output as the link weight, the action is executed to obtain the state s′ and reward value r at the next moment, and experience = (s, a, r, s′) is stored as a historical experience in the experience playback buffer.
[0117] Step 42: During the training phase, N data are sampled from the experience replay buffer pool. Assuming that one data experience = (s, a, r, s′) is sampled, the update process of all networks is as follows.
[0118] Step 421: Use the TargetActor network to calculate the action a′ under the state s′. The Target Actor network has the same neural network structure as the Actor network.
[0119] Step 422: Based on the target strategy smoothing regularization, add exploration noise to a′:
[0120]
[0121] where ε is the exploration noise, clip is the clipping function, is the random noise function, σ represents the standard deviation, and c is the limit of the clipping function.
[0122] Step 423: Then calculate the target value:
[0123]
[0124] Where y represents the target value and γ represents the discount factor.
[0125] Step 424: Calculate the evaluation value The error between the target value y is:
[0126]
[0127] Step 425: Minimize the average error between the evaluation value and the target value to update the parameters θ1 and θ2 in the Critic1 network and the Critic2 network:
[0128]
[0129] Step 426: After the Critic1 network and the Critic2 network are updated for d steps, the Actor network is used to calculate the action a under state s, and then the Critic1 network is used to calculate the evaluation value Using the gradient ascent algorithm to maximize Complete the update of the Actor network parameter φ:
[0130]
[0131] Step 427: Finally, the target network is updated by soft update, an update rate τ is introduced, the old target network parameters and the new corresponding network parameters are weighted averaged, and then assigned to the target network:
[0132] θ i ′←τθ i +(1-τ)θ i ′,i=1,2 (34)
[0133] φ′←τφ+(1-τ)φ′(35)
[0134] In the third stage, the multipath routing strategy is output. After the agent training converges, the multipath routing strategy is generated according to the final output traffic allocation ratio. The controller forwards the obtained multipath routing strategy to the low-orbit satellite node to realize multipath transmission in the low-orbit satellite network.
[0135] The satellite link time and space level considers the traffic demand of the coverage area from both time and space aspects. Figure 3 As shown in Figure 1, in terms of time, users have different demands due to their work and daily routines. Figure 3 The user demand at different times varies greatly, so segmented fitting of the data in the figure is more in line with the actual situation. Divide a day into four sections, namely 0:00-6:00, 6:00-14:00, 14:00-18:00, and 18:00-23:00. Fit the traffic demand in the four time periods. The function y(t) of the traffic demand ratio changing with time is expressed as follows:
[0136]
[0137] Normalize y(t) to get the time level Y of satellite i at time t i .
[0138] like Figure 4 As shown in the figure, in terms of space, due to the uneven distribution of the earth's population and the different economic development levels of various regions, the user demand in different regions on the earth at the same time is different. The earth is divided into 6×12 regions, and the number in each region represents the user traffic distribution density TDD in that region. The spatial level TD of the area covered by satellite i is obtained by normalizing TDD i .
[0139] The spatiotemporal level of traffic in the area covered by satellite i i The calculation is as follows:
[0140]
[0141] The spatiotemporal level of link e(u,v) uv is the value with the largest spatiotemporal level among satellites u and v, and is calculated as follows:
[0142] level uv=max{level u ,level v} (38)
[0143] The experiment uses four different levels of traffic intensity, which occupy 25%, 50%, 75% and 100% of the intersatellite link bandwidth respectively. 100 traffic matrices are generated for different traffic intensities and satellite link spatiotemporal levels. In order to evaluate the load balancing performance of the present invention, the multipath routing method based on convolutional twin delayed deep deterministic policy gradient (CTD3-MR) proposed in the present invention is compared with TD3, ECMP, and TMR routing methods under different traffic intensities. The traffic distribution index TDI is compared.
[0144]
[0145] The TD3 routing algorithm uses the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm to obtain the optimal path to transmit traffic; ECMP is an equal-cost multi-path routing algorithm, which evenly distributes traffic between the two shortest paths between the same source-destination node pairs; TMR is a three-constraint multi-path routing method that comprehensively considers link delay, bandwidth and node load.
[0146] like Figure 5As shown in the figure, the traffic distribution index of four routing algorithms under different traffic intensities is compared. The TDI of the CTD3-MR algorithm is the highest under all four traffic intensities, and this method performs well in achieving network load balancing. When the traffic intensity is in the range of 75%-100%, the TDI of the TD3 algorithm decreases as the traffic intensity increases, and the other three multipath routing algorithms increase as the traffic intensity increases. Because multipath routing makes all links in the network utilized when the network traffic is large, and there are still links with light loads when the traffic is relatively small, the multipath routing algorithm can better balance the network load when the network traffic is large. The ECMP algorithm has the lowest traffic distribution index. The average split traffic cannot adapt to the link load situation, resulting in a large load difference between the links in the network, and the network load balancing is not achieved through multipath routing, which proves the importance of the traffic distribution ratio of multipath routing. Since the TMR algorithm takes into account the node load and adjusts the distribution ratio according to the link load situation, it is better than the TD3 algorithm and the ECMP algorithm in load balancing. The CTD3-MR algorithm takes into account the spatial and temporal levels of satellite links, and adjusts the traffic distribution ratio in advance according to the link load conditions, splitting more traffic to lightly loaded links for transmission, thereby achieving network load balancing. Compared with the other three routing algorithms under different traffic intensities, the traffic distribution index of CTD3-MR is at least 5.82% higher.
[0147] The present invention is not limited to this embodiment, and any equivalent concepts or changes within the technical scope disclosed by the present invention are included in the protection scope of the present invention.
Claims
1. A satellite network multipath routing method based on deep reinforcement learning, characterized in that: The following steps are involved: A. Collecting intersatellite link information In a software defined network (SDN) structure, the data plane is composed of low-orbit satellites, and the control plane is composed of a controller; the controller is an SDN controller, and the controller is installed with a convolutional double-delay deep deterministic policy gradient algorithm (CTD3), and the convolutional double-delay deep deterministic policy gradient algorithm is referred to as an agent; The controller collects intersatellite link information of low-orbit satellites as state input to the agent, and the steps are as follows: A1. The controller collects inter-satellite link information of low-orbit satellites, where the inter-satellite link information includes link remaining bandwidth, transmission delay, packet loss rate, and time-space level; A2. The controller inputs the link packet loss rate matrix, the spatiotemporal level matrix, the normalized residual bandwidth matrix, the transmission delay matrix, and the traffic matrix into the agent as the state space S; the neural network in the agent adopts a convolutional neural network; B. Training the Agent B1. The action output by the agent is used as the link weight The action is expressed as action = [ω1, ω2, ···, ω m ,···,ω E ], where ω m represents the weight of the mth link, |E| represents the number of visible links in the network, and is assigned to the links in set E in turn; B2. Use the Floyd algorithm to calculate the two shortest weighted paths between any source and destination nodes according to the link weights as the path set P, and then use the path weight ratio as the traffic distribution ratio to split the traffic according to the traffic distribution ratio; B3. Introducing the reward function into the maximum link bandwidth utilization rate to represent the link load. The smaller the maximum link bandwidth utilization rate, the more balanced the satellite network load. B4. Training the Neural Network in the Agent The agent includes 6 neural networks, namely Actor network, TargetActor network, Critic1 network, Critic2 network, TargetCritic1 network and TargetCritic2 network, where Actor network is denoted as π φ , parameter is φ, TargetActor network is π′ φ , the parameter is φ′, the Critic1 network and the Critic2 network are and The parameters are θ1 and θ2, and the Target Critic1 network and Target Critic2 network are and The parameters are θ1′ and θ2′; C. Output multipath routing strategy After the training of the intelligent agent converges, a multipath routing strategy is generated according to the final output traffic distribution ratio. The controller forwards the multipath routing strategy to the low-orbit satellite node to realize multipath transmission in the low-orbit satellite network.
2. According to claim 1, a satellite network multipath routing method based on deep reinforcement learning is characterized in that: The method of collecting inter-satellite link information of low-orbit satellites by the controller in step A1 comprises the following steps: A11. Use the satellite toolkit (STK) to obtain satellite parameters, and then analyze the change pattern of link distance. The simulation time is one day. Since the distance between satellites on the same orbit changes very little, the distance between satellites on different orbits changes periodically and has a regular change over time. The model of the distance between adjacent orbits changing over time is as follows: In the formula, r u,v (t) represents the distance between satellite u and satellite v at time t, B is half of the difference between the maximum and minimum distances between adjacent orbits, A is the difference between B and the minimum distance between adjacent orbits, T is the period of distance change between adjacent orbits, is the initial phase; A12. Use the visibility matrix G t Represents the satellite network topology that changes with time t; V represents the set of satellites, and E represents the set of links between satellites, E = {e(u,v)}, u≠v, u∈V, v∈V, where u∈[1,n], v∈[1,n], and n represents the total number of satellites; e(u,v) represents the link between satellite u and satellite v, and satellite v is a visible neighbor node of satellite u, then the visibility matrix G t It is expressed as follows: Among them, r u ' ,v (t) indicates whether satellite u and satellite v are visible at time t. u ' ,v When (t) = 0, it means that satellite u and satellite v are not visible at time t; when r u ' ,v When (t)≠0, it means that satellite u and satellite v are visible at time t; and r u ' ,v (t) = r u,v (t); A13, calculating the link transmission delay; the link transmission delay d uv (t) includes propagation delay and average queuing delay The formula is as follows: In the formula, c represents the speed of light, num represents the number of packets in the queue at time t, and pk avg Indicates the average packet length; The remaining bandwidth b of link e(u,v) uv The bandwidth capacity bw of the link uv Subtract the used bandwidth of the link Get, expressed as follows: The used bandwidth of the link is obtained from the flow rate of the satellites at both ends of the link and is calculated as follows: The flow rate of satellite u is based on the total number of data bytes received within a period of time ΔT. and the total number of bytes transferred The formula is as follows: Similarly, the velocity of satellite v is obtained: The time and space level of the link is the maximum time and space level of the satellite nodes at both ends of the link. The time and space level of the satellite node considers the traffic demand of the coverage area from both time and space aspects; the time level of satellite i at time t is Y i , i∈[1,n], the spatial level of the area covered by satellite i is TD i , the spatiotemporal level of traffic in the area covered by satellite i i The calculation formula is as follows: The spatiotemporal level of link e(u,v) uv Set to the value with the larger spatiotemporal level between satellite u and satellite v, calculated as follows: level uv =max{level u ,level v } (11) In a satellite network, there is a certain amount of data traffic between satellites. uv represents the flow between satellites u and v, and the flow is normalized using the following formula: In the formula, f u ' v represents the normalized flow rate between satellites u and v; Use formulas (13) and (14) to normalize the link residual bandwidth and transmission delay respectively: Where b u ' v represents the normalized link residual bandwidth, d u ' v represents the normalized link transmission delay; l uv It represents the packet loss rate of link e(u,v), and is calculated as follows: In the formula, pk v Indicates the number of packets received by satellite v, pk u Indicates the number of data packets sent by satellite u.
3. According to claim 1, a satellite network multipath routing method based on deep reinforcement learning is characterized in that: The state space S in step A2 is represented as follows: S = [FBDL Lev] (16) In the formula, F represents the traffic matrix, B represents the remaining bandwidth matrix of the satellite network, D represents the transmission delay matrix of the satellite network, L represents the packet loss rate matrix of the satellite network, and Lev represents the spatiotemporal level matrix of the satellite network. The specific expressions of each matrix are as follows:
4. According to claim 1, a satellite network multipath routing method based on deep reinforcement learning is characterized in that: The method of introducing the reward function into the maximum link bandwidth utilization described in step B3 is as follows: The maximum link bandwidth utilization RU is calculated as follows: Use the weighted sum of the maximum link bandwidth utilization, the satellite network packet loss rate, and the average end-to-end delay, i.e., the average value of the transmission delay from the source to the destination and the transmission delay from the destination to the source, as the reward function to adjust the action until the reward value converges to the interval [0.6, 0.7]. The reward function is expressed as follows: Where D avg , Loss represent the average end-to-end delay and packet loss rate of the satellite network, respectively; α1, β1, and λ1 are the weight parameters of the maximum link bandwidth utilization, average end-to-end delay, and satellite network packet loss rate when the satellite network load is relatively balanced; α2, β2, and λ2 are the weight parameters of the maximum link bandwidth utilization, average end-to-end delay, and satellite network packet loss rate when the satellite network load is relatively unbalanced; α3, β3, and λ3 are the weight parameters of the maximum link bandwidth utilization, average end-to-end delay, and satellite network packet loss rate when the satellite network load is unbalanced; Use RU as the segmentation condition and set different weight parameters for each segment function to increase the penalty for poor actions; Average end-to-end delay D avg is the average transmission delay of the path set P, and the calculation formula is as follows: Using formula (25) for D avg Normalized, where D max represents the maximum path delay in the path set P, D min represents the minimum path delay in the path set P, and the formula is as follows: In the formula, represents the normalized average end-to-end delay; The total number of packets received by the destination is denoted as f accept , the calculation formula is as follows: Network packet loss rate L sum The calculation formula is as follows: Use formula (28) to normalize the network packet loss rate L sum , where L max , L min They represent the maximum and minimum packet loss rates of the shortest weighted path in the network respectively; Where Loss represents the normalized network packet loss rate.
5. According to claim 1, a satellite network multipath routing method based on deep reinforcement learning is characterized in that: The steps of training the neural network in the agent described in step B4 are as follows: B41. In the sampling phase, the state s is input into the Actor network, the action a is output as the link weight, the action is executed to obtain the next state s′ and reward value r, and experience = (s, a, r, s′) is stored as a historical experience in the experience playback buffer; B42. In the training phase, N data are sampled from the experience replay buffer pool. Assuming that one data experience = (s, a, r, s′) is sampled, the update process of all networks is as follows: B421. Use the TargetActor network to calculate the action a′ in state s′; based on the target strategy smoothing regularization, add exploration noise to a′: where ε is the exploration noise, clip is the clipping function, is the random noise function, σ represents the standard deviation, and c is the limit of the clipping function; B422. Calculate the target value: Where y represents the target value and γ represents the discount factor; B423. Calculate the evaluation value The error between the target value y is: B424, minimize the average error between the evaluation value and the target value to update the parameters θ1 and θ2 in the Critic1 network and the Critic2 network: B425, after Critic1 and Critic2 networks are updated d steps, the Actor network is used to calculate the action a under state s, and then the Critic1 network is used to calculate the evaluation value Using the gradient ascent algorithm to maximize Complete the update of the Actor network parameter φ: B426. Finally, the target network is updated by soft update. An update rate τ is introduced to make a weighted average of the old target network parameters and the new corresponding network parameters, and then assigned to the target network: i i ′←τθ i +(1-τ)θ i ′,i=1.2 (34) φ′←τφ+(1-τ)φ′(35).
Citation Information
Cited By
Private domain live broadcast bandwidth resource allocation and routing method and system based on reinforcement learning
CN119383380A
Low-orbit inclined orbit constellation network routing method
CN119675758A
Route selection method, system, device and equipment suitable for electric power communication service
CN119854198A
Space-space heterogeneous network cross-domain link selection and message distribution method
CN120321732A
Space-air heterogeneous network cross-domain link selection and message distribution method
CN120321732B
