A low-orbit satellite communication network networking method and system

Abstract

The present invention discloses a low-orbit satellite communication network networking method and system, relating to the field of communication technology. The present invention proposes a novel satellite view design, through which all possible connection paradigm sets between satellites are obtained, and a networking decision model is established simultaneously, in which the congestion states of all satellites are set as the state space of an intelligent agent, and the connection paradigm set is set as the behavior space of the intelligent agent. At the same time, a reward function of end-to-end delay is considered to train the networking decision model. The process eliminates the problems of many transmission hops and a single structure in traditional technologies by establishing a connection paradigm set based on the satellite view, optimizes the service quality of the satellite network according to the delay during satellite communication, selects the best connection paradigm as the overall networking decision of the satellite network, and enables the networking decision to be reasonably adjusted according to the current overall network state of the satellite network, so as to build a better global satellite network communication system.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a low-orbit satellite communication network networking method and system. Background Art

[0002] Low-orbit satellite communication systems, through ultra-dense deployment in very low Earth orbit (LEO), can provide seamless, low-latency, and high-throughput services and are considered a key technology for 6G. In recent years, humanity has invested a lot of resources in building low-orbit satellite constellations. It is expected that by 2029, more than 57,000 satellites will be launched into low-Earth orbit. In the low-orbit satellite network, through networking technology, low-orbit satellites can be interconnected to build a flexible, efficient, and stable global WeChat network system, providing support for global communication construction.

[0003] Mesh networking technology is a relatively traditional and classic networking method at this stage. Typically, each satellite establishes four links (Inter-satellite Links, ISLs) with its neighboring satellites, two of which are intra-orbit inter-satellite links and the other two are inter-orbit inter-satellite links. As low-orbit satellite constellations continue to develop towards ultra-dense, traditional mesh networking technology faces problems such as excessive number of transmission hops and a single structure. In addition, it is difficult for satellites to flexibly adjust networking strategies according to network status to achieve better Quality of Service (QoS). At the same time, when faced with sudden satellite failures, fault detection and rerouting are generally used to solve the problem, but this type of approach is difficult to meet the needs of some tasks with high real-time requirements.

[0004] Therefore, at the current stage, when low-orbit satellites are networked through grid networking technology, grid networking technology often has problems such as many transmission hops and a single structure, making it difficult for low-orbit satellites to flexibly adjust their networking strategies according to the current network status, thereby building a better global satellite network system. Summary of the Invention

[0005] The embodiments of the present invention provide a low-orbit satellite communication network networking method and system, which can solve the problem in the existing technology that when low-orbit satellites are currently networked through grid networking technology, the grid networking technology often has problems such as a large number of transmission hops and a single structure, making it difficult for low-orbit satellites to flexibly adjust their networking strategies according to the current network status.

[0006] An embodiment of the present invention provides a method for establishing a low-orbit satellite communication network, comprising the following steps:

[0007] Build a single-layer ultra-dense LEO satellite network and model the LEO satellite network as a topology graph;

[0008] Determine the satellite view based on the relative positions of the satellite nodes in the topology map; and form a set of connection paradigms of the LEO satellite network based on the satellite view;

[0009] According to the operating status of the LEO satellite network, the link delay of the LEO satellite end-to-end communication is obtained, and the optimization target of the delay is determined to establish the objective function of minimizing the delay;

[0010] Based on the similarity of satellite views in the LEO satellite network and the similarity of the connection paradigms of each satellite, a virtual object agent is abstracted for the entire LEO satellite network to make networking decisions. The agent selects a connection paradigm and applies it to all satellites. The agent then interacts with the environment to obtain the congestion status of all satellites.

[0011] Based on the deep reinforcement learning algorithm DQN, a networking decision model is established. The intelligent agent trains the networking decision model according to the connection paradigm set of the LEO satellite network, the objective function of minimizing delay, and the congestion status of all satellites. The trained networking decision model is obtained for the networking of the LEO satellite network.

[0012] Preferably, the step of building a single-layer ultra-dense LEO satellite network and modeling the LEO satellite network as a topology graph includes:

[0013] A single-layer ultra-dense LEO satellite network is built based on the Walker constellation configuration, and the LEO satellite network is modeled as a topology graph G(V,E). This graph includes n satellite nodes, several inter-satellite links, and end-to-end links. V represents the set of satellite nodes, and E represents the set of inter-satellite links.

[0014] Then the satellite node set is expressed as:

[0015] V={v1,v2,...,v n};

[0016] The set of intersatellite links is expressed as:

[0017] E={e i,j |i,j∈{1,2,...,n}};

[0018] For end-to-end paths, such as satellite v src To satellite v dst The end-to-end path is expressed as:

[0019] path src,dst ={v src ,e src,j ,v j ,...,v dst},

[0020] Among them: e src,jIndicates the satellite v src To satellite v j optical link.

[0021] Preferably, the determining of the satellite view comprises:

[0022] When only the current satellite exists in the satellite view, the satellite view is 1×1;

[0023] In traditional grid networking, each satellite needs to establish four inter-satellite links with its neighboring satellites. Two intra-orbit links connect two adjacent satellites in the same orbit, and two inter-orbit links connect the two closest satellites in adjacent orbits. This is used to determine the satellite view.

[0024] When the satellite view is 3×3, for a single satellite, it can choose to connect to two adjacent satellites on the same orbit. x To establish a link, you can also choose the nearest satellite v in the adjacent orbit y Establish a link, and you can also choose to connect to the satellite v y Two adjacent satellites in the same orbit;

[0025] When the satellite view is n×n (n must be an odd number), the current satellite can select the satellites above and below it in the same orbit. You can also choose to establish a link with the satellites on the left and right of the adjacent orbits. For the n-1 satellites on the left and right, the current satellite can also establish links with the n-1 satellites above and below the corresponding orbits. A link is established between the satellites, thus forming an n×n view of the satellite.

[0026] Preferably, the set of connection paradigms forming the LEO satellite network includes:

[0027] In the Walker constellation configuration, once the satellite's view is determined, the satellite can select any satellite within its view that does not exceed the number of APT terminals to establish a link;

[0028] If the inter-satellite networking mode is set to center-symmetric, then only half of the links corresponding to half of the satellites in the satellite view need to be considered, and the other half of the links are symmetric about the corresponding satellite centers;

[0029] Based on the similarity between satellite views and connection paradigms in a fault-free state, the connection paradigm set of the entire satellite network is obtained. At the same time, by evaluating one by one whether the satellite topology graph to which the connection paradigm is applied is a strongly connected graph, the connection paradigms in the connection paradigm set are filtered to form the available connection paradigm set of the LEO satellite network.

[0030] Preferably, after obtaining the link delay during end-to-end communication of the LEO satellite, the end-to-end link delay within the LEO satellite network is used as an optimization target;

[0031] Satellite v i To satellite v j The single-hop delay of is composed of sending delay, propagation delay and queuing delay, which can be expressed as:

[0032]

[0033] in: Indicates satellite v i and satellite v j The distance between them; ι represents the data transmission rate in free space; l represents the size of the data packet; Indicates satellite v i The sending rate; Indicates satellite v i The number of cached packets;

[0034] The end-to-end link delay is obtained by adding up the delays of multiple single hops and is expressed as:

[0035]

[0036] Where: T represents the path src,dst The end-to-end delay is Indicates the satellite v i To satellite v j One-hop delay;

[0037] The objective function of the optimization target is expressed as:

[0038]

[0039] stx i ≤M i ,

[0040] N i ≤N i ,

[0041] Where: x i Indicates satellite v i The number of APT terminals; M i Indicates satellite v i The upper limit of APT terminals that can be owned; N i and N i They represent the satellite's data queue length and buffer size respectively.

[0042] Preferably, obtaining the trained networking decision model includes:

[0043] A networking decision model based on the deep reinforcement learning algorithm DQN is constructed. The satellite networking decision process is set as a Markov decision process with a state space, behavior space, and reward function.

[0044] The satellite networking strategy depends on the satellite state, so the state space of the agent is represented by the states of all satellites s = {s1, s2, ..., s n The state space is further represented as the congestion state of all satellites. The congestion state of a satellite is the ratio of the satellite's data queue size to the buffer size. The agent interacts with the environment to obtain the congestion state of the satellite.

[0045] The connection paradigm set of the LEO satellite network is set as the behavior space of the agent; the inverse of the objective function of the optimization target is set as the reward function of the agent; the networking decision model is trained, and the agent interacts with the environment to obtain the environmental state at time t, that is, the congestion status s of the satellite network t , input the state into the neural network to obtain the value corresponding to each behavior in the behavior space, use the ò-greedy strategy to make decisions to obtain the networking paradigm, apply it to each satellite, and obtain the reward value. Repeat this process for training to obtain the trained networking decision model.

[0046] Preferably, the networking decision model takes into account satellite failures during training and when using the model to make networking decisions, and updates the values ​​corresponding to the behaviors in the behavior space of the agent;

[0047] The agent evaluates the Q corresponding to the action in the action space through the neural network θ (s t ,a t ) value, considering the case of satellite failure, in order to solve the change of action space size caused by satellite failure, it is necessary to adjust the value Q of the connection paradigm θ (s t ,a t ) Perform AND operation using mask;

[0048] In the case of satellite failure, the networking connection paradigm adopted cannot form a connected graph for the entire satellite network due to the satellite failure. The mask of this type of connection paradigm is set to 0, and the mask of the networking paradigm that can still be connected in the case of satellite failure is set to 1; the value of the connection paradigm that cannot form a strong connected graph when there is a faulty satellite is masked. The intelligent agent selects the connection paradigm corresponding to the action with high value in the utilization stage, and randomly selects the connection paradigm corresponding to the action with non-zero value in the exploration stage.

[0049] An embodiment of the present invention further provides a low-orbit satellite communication network system, comprising:

[0050] Design modules for building a single-layer ultra-dense LEO satellite network and modeling the LEO satellite network as a topology graph;

[0051] A paradigm module is used to determine the satellite view based on the relative positions of the satellite nodes in the topology map; and to form a set of connection paradigms of the LEO satellite network based on the satellite view;

[0052] An optimization module is used to obtain the link delay of LEO satellite end-to-end communication according to the operating status of the LEO satellite network, and determine the optimization target of the delay to establish an objective function to minimize the delay;

[0053] The algorithm module is used to abstract a virtual object agent for the entire LEO satellite network to make networking decisions based on the similarity of satellite views and the similarity of the connection paradigms of each satellite. The agent selects a connection paradigm and applies it to all satellites. The agent interacts with the environment to obtain the congestion status of all satellites.

[0054] Based on the deep reinforcement learning algorithm DQN, a networking decision model is established;

[0055] In the training module, the intelligent agent trains the networking decision model according to the connection paradigm set of the LEO satellite network, the objective function of minimizing delay, and the congestion status of all satellites, and obtains the trained networking decision model for the LEO satellite network.

[0056] The embodiments of the present invention provide a low-orbit satellite communication network construction method and system. Compared with the prior art, the advantages thereof are as follows:

[0057] The present invention proposes a novel satellite view design, which obtains a set of all possible connection paradigms between satellites through the satellite view design, and simultaneously establishes a networking decision model based on the deep reinforcement learning algorithm DQN. The congestion status of all satellites is set as the state space of the intelligent agent in the networking decision model, and the final connection paradigm set is set as the behavior space of the intelligent agent. At the same time, the networking decision model is trained by considering the reward function related to end-to-end delay. The process eliminates the problems of many transmission hops and single structure existing in traditional technologies by establishing a connection paradigm set of satellite views, optimizes the service quality of the satellite network according to the delay during satellite communication, and selects the best connection paradigm as the overall networking decision of the satellite network. The network decision can be reasonably adjusted according to the current overall network status of the satellite network, and the satellite network can make networking decisions on demand according to different network statuses, thereby building a more optimized global satellite network communication system. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1A schematic diagram of a low-orbit satellite communication network construction method provided by an embodiment of the present invention;

[0059] Figure 2 A schematic diagram of a selection process for a low-orbit satellite communication network networking method using symmetry to simplify the connection paradigm provided by an embodiment of the present invention;

[0060] Figure 3 A schematic diagram of a networking decision process based on a connection paradigm for a low-orbit satellite communication network networking method provided by an embodiment of the present invention;

[0061] Figure 4 A schematic diagram comparing the average data packet delays of a traditional grid networking method and the method of the present invention under different satellite view sizes for a low-orbit satellite communication network networking method provided by an embodiment of the present invention;

[0062] Figure 5 A schematic diagram comparing the average data packet delay between a low-orbit satellite communication network networking method provided by an embodiment of the present invention and a traditional grid networking method in the absence of satellite failures;

[0063] Figure 6 A schematic diagram showing a comparison of average data packet delays between a low-orbit satellite communication network networking method provided by an embodiment of the present invention and a traditional grid networking method when the satellite failure rate is 3%;

[0064] Figure 7 A schematic diagram showing a comparison of average data packet delays between a low-orbit satellite communication network networking method provided by an embodiment of the present invention and a traditional grid networking method when the satellite failure rate is 6%;

[0065] Figure 8 A schematic diagram comparing the average data packet delay between a low-orbit satellite communication network networking method provided by an embodiment of the present invention and a traditional grid networking method when the satellite failure rate is 9%;

[0066] Figure 9 A schematic diagram comparing the cumulative distribution function of the number of hops between a low-orbit satellite communication network networking method provided by an embodiment of the present invention and a traditional grid networking method when there is no satellite failure;

[0067] Figure 10 A schematic diagram comparing the cumulative distribution function of the number of hops between a method of establishing a low-orbit satellite communication network provided by an embodiment of the present invention and a traditional grid networking method when the satellite failure rate is 3%;

[0068] Figure 11A schematic diagram comparing the cumulative distribution function of the number of hops between a method of establishing a low-orbit satellite communication network provided by an embodiment of the present invention and a traditional grid networking method when the satellite failure rate is 6%;

[0069] Figure 12 A schematic diagram comparing the cumulative distribution function of the number of hops between a low-orbit satellite communication network networking method provided by an embodiment of the present invention and a traditional grid networking method when the satellite failure rate is 9%. DETAILED DESCRIPTION

[0070] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. The following description sets forth many specific details to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar modifications without violating the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0071] See also Figure 1 The embodiment of the present invention provides a method for establishing a low-orbit satellite communication network, comprising the following steps:

[0072] Step S1: Build a single-layer ultra-dense LEO satellite network based on the Walker constellation and model the dense LEO satellite network.

[0073] Step S2: Based on the satellite view, a certain number of satellites are selected from the view to establish intersatellite links. All possible combinations of satellites form a connection paradigm set, which is further simplified and filtered using symmetry and connectivity.

[0074] Step S3: Based on the established single-layer LEO satellite network model, the service quality indicators are modeled; in response to the needs of low-orbit satellite communications, the optimization objectives related to delay are defined based on the service quality indicators, and an objective function for minimizing delay is established for optimization.

[0075] Step S4: Based on the similarity of satellite views and connection paradigms, an intelligent agent is abstracted for the entire satellite network as a networking decision model, and combined with a deep reinforcement learning algorithm, a single-agent deep reinforcement learning networking decision algorithm is constructed.

[0076] Step S5: Use the single-agent deep reinforcement learning networking decision algorithm to train the agent to obtain a networking decision model.

[0077] Step S6: Use the trained networking decision model to make networking decisions, input the status of the satellite network, output the networking connection paradigm, and apply it to each satellite.

[0078] Specifically:

[0079] Step S1: Build a single-layer ultra-dense LEO satellite network and model it.

[0080] The present invention uses the Walker constellation and models the ultra-dense LEO satellite network under consideration as a graph G(V,E), which specifically includes n satellite nodes, where V represents the set of satellite nodes and E represents the set of inter-satellite links; each satellite node is equipped with four APT terminals, which means that each satellite can establish links with up to four other satellites; end-to-end communication between satellites often requires multiple satellite relays, so the present invention only focuses on the inter-satellite communication part and ignores the satellite-to-ground communication part.

[0081] The satellite node set is represented as:

[0082] V={v1,v2,...,v n};

[0083] The set of intersatellite links is expressed as:

[0084] E={e i,j |i,j∈{1,2,...,n}};

[0085] For end-to-end paths, such as satellite v src To satellite v dst The end-to-end path can be expressed as:

[0086] path src,dst ={v src ,e src,j ,v j ,...,v dst},

[0087] Among them: e src,j Indicates the satellite v src To satellite v j optical link.

[0088] Step S2: Obtain a set of connection paradigms based on satellite views, and simplify and filter them using symmetry and connectivity. Specifically, the following steps are performed:

[0089] Step S21: In the traditional grid networking mode, each satellite needs to establish four inter-satellite links with its adjacent satellites, of which two intra-orbit links connect two adjacent satellites on the same orbit, and the other two inter-orbit links connect the two nearest satellites on the adjacent orbit. Therefore, the satellite view can be determined based on this; when there is only the current satellite in the satellite view, the satellite view is 1×1; when the satellite view is 3×3, for a single satellite, it can choose to connect to two adjacent satellites on the same orbit. x To establish a link, you can also choose the nearest satellite v in the adjacent orbity Establish a link, and you can also choose to connect to the satellite v y Two adjacent satellites on the same orbit; when the satellite view is n×n (n must be an odd number), the current satellite can select the satellites above and below it on the same orbit. You can also choose to establish a link with the satellites on the left and right of the adjacent orbits. For the n-1 satellites on the left and right, the current satellite can also establish links with the n-1 satellites above and below the corresponding orbits. A link is established between the satellites, thus forming an n×n view of the satellite.

[0090] Step S22: In the Walker constellation, traditional satellites are usually equipped with four APT terminals. Assuming that the satellite view is m×m, the networking based on the connection paradigm allows each satellite to select any satellite within its own view that does not exceed the number of APT terminals to establish a link. Therefore, for a satellite with a fixed number of APT terminals, networking is equivalent to a combinatorial problem with a time complexity of O(C(m 2 ,4))≈O(m 8 ), depends on the size of the satellite's view; although in practice the satellite's view is limited and m is not particularly large (usually 3, 5, 7, etc.), the time complexity of the algorithm is still relatively high.

[0091] Step S23: Taking into account the dynamics of the satellite network and the necessity of maintaining a balance in satellite communication resources, while simplifying the networking algorithm, the present invention designs the networking connection to be centrally symmetric, that is, only the method of establishing links with half of the satellites in the satellite view needs to be considered, and the other half of the links are symmetric about the center of this satellite; therefore, the time complexity is reduced to like Figure 2 As shown, the present invention designs the networking connection to be centrally symmetric. The central satellite is the satellite to be networked. In its 5×5 view, half of the satellites are pruned due to central symmetry. After the central satellite selects an optional satellite, it then performs central symmetry on the link to find the corresponding pruned satellite. This method significantly reduces the time complexity of obtaining a set consisting of all possible connection paradigms.

[0092] Step S24: The intelligent agent independently selects the connection paradigm of the satellite, and cannot guarantee that any pair of satellites in the global topology are connected; therefore, by evaluating whether the satellite topology graph G of the application paradigm is a strongly connected graph, the connection paradigm is further filtered, and the filtered connection paradigm can be used as an alternative set for dynamic switching links of LEO satellites; in addition, a faulty satellite may also cause some connection paradigms in the filtered set to become unavailable. The dimensions of the available connection paradigm set are also different under different fault conditions, but the output dimension of a single neural network must be fixed. Therefore, in this step, for the purpose of fixing the output dimension of the neural network, the present invention does not consider the situation of a faulty satellite, thereby reducing the training cost.

[0093] Step S3: Define the optimization goal related to the delay and establish an objective function for minimizing the delay.

[0094] In satellite communication scenarios, latency is one of the most important service quality indicators, directly affecting satellite communication efficiency and user experience. Therefore, this paper analyzes satellite networks from the perspective of latency indicators, defines optimization goals related to latency, and establishes an objective function for minimizing latency to meet higher real-time requirements, thereby improving user experience.

[0095] Specifically, satellite v i To satellite v j The one-hop delay of can be composed of sending delay, propagation delay and queuing delay, which can be calculated by the following formula:

[0096]

[0097] in: Indicates satellite v i and satellite v j The distance between them; ι represents the data transmission rate in free space; l represents the size of the data packet; Indicates satellite v i The sending rate; Indicates satellite v i The number of packets buffered.

[0098] The end-to-end delay is obtained by adding up multiple single-hop delays and can be expressed as:

[0099]

[0100] Where: T represents the path src,dst The end-to-end delay is Indicates the satellite v i To satellite v j One-hop delay.

[0101] According to the above formula, the optimization goal of the present invention is to minimize the sum of end-to-end delays, and its objective function is:

[0102]

[0103] stx i ≤M i ,

[0104] N i ≤N i ,

[0105] Where: x i Indicates satellite v i The number of APT terminals; M i Indicates satellite v i The upper limit of APT terminals that can be owned; N i and N i They represent the satellite's data queue length and buffer size respectively.

[0106] Step S4: Based on the similarity of satellite views and connection paradigms, an intelligent agent is abstracted for the entire satellite network as a networking decision model.

[0107] Since there are a large number of satellite nodes in the real satellite network topology, and each satellite needs to be networked, it is unrealistic to set up a networking decision model for each satellite. Therefore, based on the idea of ​​connection paradigm, for the purpose of simplification, the views of each satellite in the present invention are the same. Without considering the faulty satellite, the connection paradigm set obtained based on step S2 should also be the same. Therefore, it is only necessary to abstract a virtual object, namely the intelligent agent, for the entire satellite network to make networking decisions. It selects a connection paradigm and applies it to all satellites.

[0108] In the present invention, the intelligent agent first uses deep reinforcement learning to fit the Q table using a neural network to solve the problem of an overly large state-action space. The intelligent agent continuously explores and learns to minimize the objective function and optimize the networking decision. The intelligent agent selects the appropriate networking connection paradigm according to the satellite network status through the neural network and obtains reward feedback. This process is repeated and trained, and finally a decision network with superior performance is trained, which is the networking decision model.

[0109] Step S5: Construct a deep reinforcement learning networking decision algorithm.

[0110] The DQN algorithm in deep reinforcement learning is used to optimize the objective function defined in step S3. The agent interacts with the environment to make networking decisions, such as Figure 3As shown, the set of connection paradigms obtained in step S2 is used as the behavior space to make decisions using the intelligent agent. The selected connection paradigm is the networking strategy to be adopted. All satellites are networked according to this networking paradigm and receive a reward value. This step is repeated and training is performed continuously to minimize the end-to-end delay of the satellite network.

[0111] Specifically, in the satellite networking decision-making process, the state at the next moment is only related to the state at the current moment, and has nothing to do with the previous state. Therefore, it can be regarded as a Markov decision process with a state space, a behavior space, and a reward function. The state space, behavior space, and reward function can be expressed as:

[0112] ① State space: The satellite networking strategy depends on the satellite state, so the state space can be expressed as the state of all satellites s = {s1, s2, ..., s n}; At the same time, because the congestion level of the satellite has a great impact on the delay, the satellite v i The state can be represented as For a neural network, the dimension of the neural network input should not change, so in order to account for the change in state space size caused by satellite failure, the state of the faulty satellite is set to 1.

[0113] ② Behavior space: The filtered set of connection paradigms obtained in step S2 is the behavior space of the agent. For a neural network, the dimension of the neural network output should not change, that is, the dimension of the behavior space should not change. However, in the case of satellite failure, different satellite failures may cause different connection paradigms to be unavailable, and the dimension of the behavior space may change. Therefore, it is necessary to address the changes in the dimension of the behavior space caused by satellite failure.

[0114] ③ Reward function: The reward function is defined as the inverse of the objective function in step S3 and normalized. The objective function represents the size of the network delay after adopting a certain networking strategy. The longer the delay, the lower the corresponding reward value should be. Correspondingly, the shorter the delay, the higher the corresponding reward value should be.

[0115] In order to solve the delay optimization in step S3, the agent obtains the current state s by interacting with the environment. t , take a certain method to select action a t And apply this networking paradigm to get the reward value r t and the next state s t+1 Composition<s t ,a t ,r t ,s t+1>Store in the experience pool, then randomly sample from the experience pool and calculate the loss function, continuously update and iterate the network to minimize end-to-end delay.

[0116] The loss function of the algorithm is:

[0117]

[0118] Where: N represents the size of a batch of data samples randomly drawn from the experience replay pool D; θ represents the policy network parameters; y i It represents the decision of the target network for sample i, which can be expressed as:

[0119]

[0120] Where: γ represents the discount function; θ - Indicates the target network parameters.

[0121] At the same time, considering the situation of satellite failure, the change of action space size caused by satellite failure is solved; the agent interacts with the environment to obtain the environmental state at time t, that is, the congestion status s of the satellite network t , according to the state, use the ò-greedy greedy strategy to make a decision and get behavior a t The greedy strategy ò-greedy is to balance exploration and utilization. In the early stage of network training, as much exploration as possible is performed; in the later stage of training, almost only utilization is performed; the agent evaluates the Q corresponding to the action in the action space through the neural network θ (s t ,a t ) value, considering the case of satellite failure, in order to solve the change of action space size caused by satellite failure, it is necessary to adjust the value Q of the connection paradigm θ (s t ,a t ) uses a mask to perform AND operations; in the case of a satellite failure, it is possible that the networking connection paradigm adopted cannot form a connected graph for the entire network due to the satellite failure. The mask of this connection paradigm is set to 0, and the mask of the networking paradigm that can still be connected in the case of satellite failure is set to 1; therefore, through this method, the value of the connection paradigm that cannot form a strong connected graph when there is a faulty satellite is masked. The intelligent agent selects the connection paradigm corresponding to the action with the highest value in the utilization phase, and randomly selects the connection paradigm corresponding to the action with a non-zero value in the exploration phase.

[0122] Step S6: The agent is trained using a deep reinforcement learning network decision algorithm; specifically, the following steps are involved:

[0123] Step S61: Initialize the satellite network topology G(V,E), the experience replay pool D, the batch size N and the discount function γ.

[0124] Step S62: Input the number of training episodes max The number of slots required for networking in each training round is 1, and each round of training uses the Open Shortest Path First (OSPF) routing algorithm to perform one round of data packet forwarding.

[0125] Step S63: Initialize the policy network Q with random network parameters θ θ (s,a), using the same parameters θ - ←θ initialize the target network Q θ- (s,a).

[0126] Step S64: Use the connection paradigm set obtained by the method in step S2 as the behavior space.

[0127] Step S65: Determine whether the current number of training times is less than episode max If it is less than, jump to step S66; if it is greater than or equal to, jump to step S616.

[0128] Step S66: The round number t is set to 0.

[0129] Step S67: Determine whether the current networking round t is less than slot. If it is less, jump to step S68; if it is greater than or equal to slot, jump to step S612.

[0130] Step S68: Get the state s of wheel t t , input the policy network and perform mask operation to obtain Q θ (s t ,a t ).

[0131] Step S69: Select the action a corresponding to the highest reward value in the case of utilization t As the networking strategy for each satellite, randomly select an available action a in the exploration situation t As the networking strategy for each satellite.

[0132] Step S610: Apply a t , get reward r t and the next state s t+1 , and <s t ,a t ,r t ,s t+1 >Deposit into experience replay pool D.

[0133] Step S611: perform t+1 and jump to step S67.

[0134] Step S612: Determine whether the amount of data in the experience pool is greater than or equal to the batch size. If so, jump to step S613. If less than, increase the number of training times by one and jump to step S65.

[0135] Step S613: Randomly extract samples of size N from the experience replay pool D and calculate the loss function.

[0136] Step S614: Update the policy network Q by minimizing the loss function θ , and update the target network parameter Q after a certain period θ- .

[0137] Step S615: The number of training times is increased by one and the process jumps to step S65.

[0138] Step S616: The training is completed, and a trained neural network is obtained, which is the networking strategy.

[0139] Step S617: Use the trained neural network to obtain the Q value, and the intelligent agent selects the action with the largest Q value after masking as the networking paradigm of each satellite.

[0140] Step S7: Testing the networking strategy of the dense low-orbit satellite network.

[0141] like Figure 4 As shown in the figure, the average data packet delay of the traditional grid method and the method of the present invention under different satellite views when there is no satellite failure. As the data generation rate continues to increase, the network congestion continues to increase, and therefore the average data packet delay increases accordingly; when the view size is 3×3 or 5×5, it can be seen from the figure that the method proposed by the present invention performs better than the traditional grid networking method; in addition, since a larger view means more available connection paradigms, multiple satellites may no longer be required for relay between satellites at longer distances, so the average data packet delay decreases with increasing view size; however, since a larger view size will result in higher connection paradigm calculation costs, the view size cannot be increased indefinitely, and therefore the present invention adopts a view size of 5×5 in the specific analysis.

[0142] like Figure 5 As shown in the figure, the average packet delay of the method of the present invention and the traditional grid method when there is no satellite failure is compared. When the data generation rate is low, the average packet delay of the two methods is similar, which shows that all data packets can be transmitted smoothly under the two methods; however, as the data generation rate increases, the performance of the traditional grid method decreases rapidly, while the performance of the method of the present invention decreases more slowly. This can be attributed to the inherent limitations of the traditional method, including a single form, a large number of hops, and inadaptability to changes in satellite network status; Figure 6 、 Figure 7 and Figure 8As shown in the figure, as the satellite failure rate increases, the performance of the traditional grid method decreases rapidly. The performance of the method proposed in the present invention is still better than the traditional algorithm. This can be attributed to the fact that the traditional grid networking method lacks the ability to cope with satellite failures. In contrast, the method proposed in the present invention can provide more networking options according to the changes in satellite status and reduce the performance loss caused by satellite failures.

[0143] like Figure 9 As shown in the figure, the cumulative distribution function of the routing hop count of the method of the present invention and the traditional grid networking method is compared when there is no satellite failure. It can be seen that the method of the present invention helps to reduce the number of satellite routing relay hops, and the data packet can reach the destination through fewer relay nodes; Figure 10 、 Figure 11 and Figure 12 As shown in the figure, as the satellite failure rate increases, the number of hops of the traditional grid networking method increases significantly, while the method of the present invention shows a slower increase and the cumulative distribution function of the hop number is always better than the traditional algorithm. This advantage can be attributed to the method proposed in the present invention. Compared with the grid networking algorithm, it not only selects a networking strategy with fewer routing hops, but also has better robustness against satellite failures.

[0144] This invention innovatively proposes the concept of a satellite view. Based on the satellite view, the set of all possible connection paradigms is calculated, simplified, and filtered. Then, a single-agent deep reinforcement learning method is used, so that the agent can make reasonable networking decisions based on the state of the overall satellite network in combination with satellite failure conditions, continuously optimize the network's service quality according to the objective function, reduce the number of hops required for end-to-end transmission, and thus improve the robustness, flexibility, and QoS indicators of the satellite network.

[0145] The present invention can construct a set of connection paradigms for satellite networking by proposing the concept of "view" when the number of APT terminals and the size of the buffer zone are limited. The networking strategy can be adjusted according to the status of the satellite network, thereby optimizing the service quality index. This can be used to solve the problem of making networking decisions on demand for different network states in dense low-orbit satellite network application scenarios.

[0146] An embodiment of the present invention also provides a networking decision system for a dense low-orbit satellite network, comprising a design module, a paradigm module, an optimization module, an algorithm module, a training module, and a decision module.

[0147] Design module: Build a single-layer Walker constellation and model a dense low-orbit satellite network.

[0148] Paradigm module: Based on the satellite view, a certain number of satellites are selected from the view to establish intersatellite links. All possible combinations of satellites form a connection paradigm set, which is further simplified and filtered using symmetry and connectivity.

[0149] Optimization Module: Based on the established single-layer LEO satellite network model, service quality indicators are modeled. Targeted at the needs of low-orbit satellite communications, an objective function is established based on service quality indicators for optimization.

[0150] Algorithm module: Based on the similarity of satellite views and connection paradigms, an intelligent agent is abstracted for the entire satellite network as a networking decision model. Combined with the deep reinforcement learning algorithm, it is a single-agent deep reinforcement learning networking method.

[0151] The training module uses the constructed single-agent deep reinforcement learning networking method to train the agent and obtain the networking decision model.

[0152] Decision module: Uses the networking decision model obtained from the training module to make networking decisions, inputs the status of the satellite network, outputs the networking connection paradigm, and applies it to each satellite.

[0153] In the specific experiments of the present invention, an 8×8 satellite network topology is adopted, that is, there are 8 satellite orbits, 8 satellites are evenly distributed on each orbit, and each satellite is equipped with 4 transceivers; the orbital inclination is set to 55°, and the orbital altitude is 550 km; in addition, it is assumed that the data transmission rate of each satellite is 5 Gbps, the cache size is 1875 MB, and the size of a single data packet is 20 KB; the satellite routing algorithm uniformly adopts the Open Shortest Path First (OSPF) algorithm.

[0154] Therefore, the method of the present invention can adjust the networking strategy based on the status of the satellite network to achieve the goal of minimizing latency, even when the number of APT terminals and buffer size are limited. Specifically, the present invention first builds and models a single-layer ultra-dense LEO satellite network. Second, a set of connection paradigms is obtained based on satellite views, and simplified and filtered using symmetry and connectivity. Next, an optimization goal related to latency is defined, and an objective function for minimizing latency is established. Based on the similarity of satellite views and connection paradigms, a virtual object, namely an agent, is abstracted for the entire satellite network to make networking decisions. Next, a deep reinforcement learning networking decision-making algorithm is constructed, and its detailed elements are defined. The agent is trained using the proposed networking algorithm. Finally, the networking strategy of a dense LEO satellite network is tested. Experimental results show that compared with traditional grid networking methods, the present invention effectively reduces the number of routing hops and the average packet delay by 30%. This can reduce communication latency, reduce the number of hops required for satellite routing, optimize the service quality of satellite communications, and enhance networking flexibility. It can also cope with emergencies such as satellite failures, significantly improving the robustness of the satellite network while optimizing QoS indicators.

[0155] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A method for establishing a low-orbit satellite communication network, characterized in that: The following steps are involved: Build a single-layer ultra-dense LEO satellite network and model the LEO satellite network as a topology graph; Determine the satellite view according to the relative positions of the satellite nodes in the topology map; And based on the satellite view, a set of connection paradigms of the LEO satellite network is formed; According to the operating status of the LEO satellite network, the link delay of the LEO satellite end-to-end communication is obtained, and the optimization target of the delay is determined to establish the objective function of minimizing the delay; Based on the similarity of satellite views in the LEO satellite network and the similarity of the connection paradigms of each satellite, a virtual object agent is abstracted for the entire LEO satellite network to make networking decisions. The agent selects a connection paradigm and applies it to all satellites. The agent then interacts with the environment to obtain the congestion status of all satellites. Based on the deep reinforcement learning algorithm DQN, a networking decision model is established. The intelligent agent trains the networking decision model according to the connection paradigm set of the LEO satellite network, the objective function of minimizing delay, and the congestion status of all satellites. The trained networking decision model is obtained for the networking of the LEO satellite network.

2. A low-orbit satellite communication network networking method according to claim 1, characterized in that: The method of building a single-layer ultra-dense LEO satellite network and modeling the LEO satellite network as a topology graph includes: Build a single-layer ultra-dense LEO satellite network based on the Walker constellation configuration, and model the LEO satellite network as a topology graph ; which includes satellite nodes, several intersatellite links and end-to-end links, represents the set of satellite nodes, represents the set of intersatellite links; Then the satellite node set is expressed as: ； The set of intersatellite links is expressed as: ； For end-to-end paths, such as satellite to satellite The end-to-end path is expressed as: in: Indicates that from the satellite to satellite optical link.

3. The method for establishing a low-orbit satellite communication network according to claim 1, wherein: The determining of the satellite view comprises: When only the current satellite exists in the satellite view, the satellite view is 1×1; In traditional grid networking, each satellite needs to establish four inter-satellite links with its neighboring satellites. Two intra-orbit links connect two adjacent satellites in the same orbit, and two inter-orbit links connect the two closest satellites in adjacent orbits. This is used to determine the satellite view. When the satellite view is 3×3, for a single satellite, it can choose to connect to two adjacent satellites on the same orbit. To establish a link, you can also choose the nearest satellite in the adjacent orbit Establish a link and optionally connect to the satellite Two adjacent satellites in the same orbit; When the satellite view is n×n, n must be an odd number to ensure the center symmetry of the satellite view. The current satellite can be selected to be above and below the satellite in the same orbit. You can also choose to establish a link with the satellites on the left and right of the adjacent orbits. For the n-1 satellites on the left and right, the current satellite can also establish links with the n-1 satellites above and below the corresponding orbits. A link is established between the satellites, thus forming an n×n view of the satellite.

4. The method for establishing a low-orbit satellite communication network according to claim 1, wherein: The set of connection paradigms forming the LEO satellite network includes: In the Walker constellation configuration, once the satellite's view is determined, the satellite can select any satellite within its view that does not exceed the number of APT terminals to establish a link; If the inter-satellite networking mode is set to center-symmetric, then only half of the links corresponding to half of the satellites in the satellite view need to be considered, and the other half of the links are symmetric about the corresponding satellite centers; Based on the similarity between satellite views and connection paradigms in a fault-free state, the connection paradigm set of the entire satellite network is obtained. At the same time, by evaluating one by one whether the satellite topology graph to which the connection paradigm is applied is a strongly connected graph, the connection paradigms in the connection paradigm set are filtered to form the available connection paradigm set of the LEO satellite network.

5. The method for establishing a low-orbit satellite communication network according to claim 1, wherein: After obtaining the link delay during the end-to-end communication of the LEO satellite, the end-to-end link delay within the LEO satellite network is used as an optimization target; Satellite to satellite The single-hop delay of is composed of sending delay, propagation delay and queuing delay, which can be expressed as: in: Indicates satellite and satellite the distance between them; Indicates the data transmission rate in free space; Indicates the size of the data packet; Indicates satellite The sending rate; Indicates satellite The number of cached packets; The end-to-end link delay is obtained by adding up the delays of multiple single hops and is expressed as: in: Indicates the path The end-to-end delay is Indicates that from the satellite to satellite One-hop delay; The objective function of the optimization target is expressed as: ； in: Indicates satellite The number of APT terminals; Indicates satellite The upper limit of APT terminals that can be owned; N i and They represent the satellite's data queue length and buffer size respectively.

6. A low-orbit satellite communication network networking method according to claim 5, characterized in that: The obtaining of the trained networking decision model includes: A networking decision model based on the deep reinforcement learning algorithm DQN is constructed. The satellite networking decision process is set as a Markov decision process with a state space, behavior space, and reward function. The satellite networking strategy depends on the satellite state, so the state space of the agent is represented by the state of all satellites ,The state space is further represented as the congestion state of all satellites;,the congestion state of a satellite is the ratio of the satellite’s,data queue size to the buffer size, and the agent interacts with the,environment to obtain the congestion state of the satellite; The connection paradigm set of the LEO satellite network is set as the behavior space of the agent; the opposite of the objective function of the optimization target is set as the reward function of the agent; the networking decision model is trained, and the agent interacts with the environment to obtain The environmental status at that time is the congestion status of the satellite network , input the state to the neural network to obtain the value corresponding to each behavior in the behavior space, and use A greedy strategy is used to make decisions to obtain a networking paradigm, which is applied to each satellite and a reward value is obtained. This process is repeated for training to obtain a trained networking decision model.

7. A low-orbit satellite communication network networking method according to claim 6, characterized in that: The networking decision model takes into account satellite failures during training and when using the model to make networking decisions, and updates the value corresponding to the behavior in the behavior space of the intelligent agent; The agent evaluates the corresponding action in the action space through the neural network After considering the satellite failure, in order to solve the change of action space size caused by satellite failure, the value of the connection paradigm needs to be adjusted. Perform AND operations using masks; In the case of satellite failure, the networking connection paradigm adopted cannot form a connected graph for the entire satellite network due to the satellite failure. The mask of the networking paradigm that cannot be connected in the case of satellite failure is set to 0, and the mask of the networking paradigm that can still be connected in the case of satellite failure is set to 1; the value of the connection paradigm that cannot form a strong connected graph when there is a faulty satellite is masked. The intelligent agent selects the connection paradigm corresponding to the action with high value in the utilization stage, and randomly selects the connection paradigm corresponding to the action with non-zero value in the exploration stage.

8. A low-orbit satellite communication network system, characterized in that: include: Design modules for building a single-layer ultra-dense LEO satellite network and modeling the LEO satellite network as a topology graph; The paradigm module is used to determine the satellite view based on the relative positions of satellite nodes in the topology map; And based on the satellite view, a set of connection paradigms of the LEO satellite network is formed; An optimization module is used to obtain the link delay of LEO satellite end-to-end communication according to the operating status of the LEO satellite network, and determine the optimization target of the delay to establish an objective function to minimize the delay; The algorithm module is used to abstract a virtual object agent for the entire LEO satellite network to make networking decisions based on the similarity of satellite views and the similarity of the connection paradigms of each satellite. The agent selects a connection paradigm and applies it to all satellites. The agent interacts with the environment to obtain the congestion status of all satellites. Based on the deep reinforcement learning algorithm DQN, a networking decision model is established; In the training module, the intelligent agent trains the networking decision model according to the connection paradigm set of the LEO satellite network, the objective function of minimizing delay, and the congestion status of all satellites, and obtains the trained networking decision model for the LEO satellite network.

Images