Networking method and system for low-orbit satellite communication network

Abstract

The invention discloses a networking method and system for a low-orbit satellite communication network, relates to the technical field of communication, and provides a novel satellite view design, all possible connection paradigm sets among satellites are obtained through the satellite view design, and a networking decision model is established at the same time; the congestion states of all satellites are set as state spaces of the intelligent agents, the connection normal form set is set as behavior spaces of the intelligent agents, meanwhile, a reward function of end-to-end time delay is considered, and a networking decision model is trained; according to the process, a connection normal form set based on a satellite view is established, the phenomena of multiple transmission hops and single structure in the traditional technology are eliminated, the service quality of the satellite network is optimized according to the time delay during satellite communication, and the optimal connection normal form is selected as the overall networking decision of the satellite network. And the networking decision can be reasonably adjusted according to the current overall network state of the satellite network so as to construct a better global satellite network communication system.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a low-orbit satellite communication network networking method and system. Background Art

[0002] The low-orbit satellite communication system can provide seamless, low-latency and high-throughput services through ultra-dense deployment in extremely low Earth Orbit (LEO), and is regarded as a key technology for 6G. In recent years, humans have invested a lot of resources to build low-orbit satellite constellations. It is expected that by 2029, more than 57,000 satellites will be launched into low-Earth orbit. In the low-orbit satellite network, through networking technology, low-orbit satellites can be interconnected in a coordinated manner, and a flexible, efficient and stable global WeChat network system can be built to provide guarantees for global communication construction.

[0003] Grid networking technology is a relatively traditional and classic networking method at this stage. Usually, each satellite establishes four links (Inter-satellite Link, ISL) with its adjacent satellites, two of which are intra-orbit inter-satellite links and the other two are inter-orbit inter-satellite links. As the low-orbit satellite constellation continues to develop towards ultra-dense, traditional grid networking technology faces problems such as too many transmission hops and a single structure. It is also difficult for satellites to flexibly adjust networking strategies according to network status to obtain better quality of service (Quality of Service, QoS). At the same time, when faced with sudden satellite failures, fault detection and rerouting are generally used to solve the problem, but this type of method is difficult to meet some tasks with high real-time requirements.

[0004] Therefore, at the current stage, when low-orbit satellites are networked through grid networking technology, grid networking technology often has problems such as large number of transmission hops and single structure, making it difficult for low-orbit satellites to flexibly adjust their networking strategies according to the current network status, thereby building a better global satellite network system. Summary of the invention

[0005] The embodiments of the present invention provide a low-orbit satellite communication network networking method and system, which can solve the problem in the prior art that, when low-orbit satellites are currently networked through grid networking technology, the grid networking technology often has the problems of many transmission hops and a single structure, making it difficult for low-orbit satellites to flexibly adjust their networking strategies according to the current network status.

[0006] An embodiment of the present invention provides a low-orbit satellite communication network networking method, comprising the following steps:

[0007] Build a single-layer ultra-dense LEO satellite network and model the LEO satellite network as a topology graph;

[0008] Determine the satellite view according to the relative positions of the satellite nodes in the topology map; and form a set of connection paradigms of the LEO satellite network according to the satellite view;

[0009] According to the operation status of the LEO satellite network, the link delay of the LEO satellite end-to-end communication is obtained, and the optimization target of the delay is determined to establish the objective function of minimizing the delay;

[0010] Based on the similarity of satellite views in the LEO satellite network and the similarity of the connection paradigms of each satellite, a virtual object agent is abstracted for the entire LEO satellite network to make networking decisions. The agent selects a connection paradigm to apply to all satellites; the agent interacts with the environment to obtain the congestion status of all satellites;

[0011] Based on the deep reinforcement learning algorithm DQN, a networking decision model is established. The intelligent agent trains the networking decision model according to the connection paradigm set of the LEO satellite network, the objective function of minimizing the delay and the congestion status of all satellites, and the trained networking decision model is obtained for the networking of the LEO satellite network.

[0012] Preferably, the step of building a single-layer ultra-dense LEO satellite network and modeling the LEO satellite network as a topology graph includes:

[0013] A single-layer ultra-dense LEO satellite network is built based on the Walker constellation configuration, and the LEO satellite network is modeled as a topology graph G(V,E), which includes n satellite nodes, several inter-satellite links and end-to-end links, where V represents the set of satellite nodes and E represents the set of inter-satellite links.

[0014] Then the satellite node set is expressed as:

[0015] V={v 1 ,v 2 ,...,v n};

[0016] The set of intersatellite links is expressed as:

[0017] E={e i,j |i,j∈{1,2,...,n}};

[0018] For an end-to-end path, such as a satellite v src To satellite v dst The end-to-end path is expressed as:

[0019] path src,dst = {v src ,e src,j ,v j ,...,v dst},

[0020] Where: e src,j Indicates the satellite v src To satellite v j optical link.

[0021] Preferably, the determining of the satellite view comprises:

[0022] When there is only the current satellite in the satellite view, the satellite view is 1×1;

[0023] In the traditional grid networking method, each satellite needs to establish four inter-satellite links with its adjacent satellites, two of which are intra-orbit links connecting two adjacent satellites in the same orbit, and the other two inter-orbit links connecting the two nearest satellites in adjacent orbits, based on which the satellite view is determined;

[0024] When the satellite view is 3×3, for a single satellite, it can choose to be connected to two adjacent satellites on the same orbit. x To establish a link, you can also choose the nearest satellite v in an adjacent orbit y Establish a link and optionally connect to the satellite v y Two adjacent satellites in the same orbit;

[0025] When the satellite view is n×n (n must be an odd number), the current satellite can select the satellites above and below it in the same orbit. You can also choose to establish links with satellites on the left and right sides of the adjacent orbits. For the n-1 satellites on the left and right, the current satellite can also establish links with the n-1 satellites above and below the corresponding orbits. Links are established with 1 satellite, thus forming an n×n view of the satellite.

[0026] Preferably, the connection paradigm set for forming the LEO satellite network includes:

[0027] In the Walker constellation configuration, once the satellite’s view is determined, the satellite can select any satellite within its view that does not exceed the number of APT terminals to establish a link;

[0028] If the networking mode between satellites is set to center symmetry, then only half of the links corresponding to half of the satellites in the satellite view need to be considered, and the other half of the links are symmetric about the corresponding satellite center;

[0029] Based on the similarity between satellite views and connection paradigms in a fault-free state, the connection paradigm set of the entire satellite network is obtained. At the same time, the connection paradigms in the connection paradigm set are filtered by evaluating whether the satellite topology graph to which the connection paradigm is applied is a strongly connected graph, thus forming a set of available connection paradigms for the LEO satellite network.

[0030] Preferably, after obtaining the link delay of the LEO satellite end-to-end communication, the end-to-end link delay in the LEO satellite network is used as an optimization target;

[0031] Satellite v i To satellite v j The single-hop delay of is composed of sending delay, propagation delay and queuing delay, which can be expressed as:

[0032]

[0033] in: Indicates satellite v i and satellite v j The distance between them; ι represents the data transmission rate in free space; l represents the size of the data packet; Indicates satellite v i The sending rate; Indicates satellite v i The number of packets cached;

[0034] The delay of an end-to-end link is obtained by adding up multiple single-hop delays and is expressed as:

[0035]

[0036] Where: T represents the path src,dst The end-to-end delay is Indicates the satellite v i To satellite v j One-hop delay;

[0037] The objective function of the optimization target is expressed as:

[0038]

[0039] stx i ≤M i ,

[0040] N i ≤N i ,

[0041] Where: x i Indicates satellite v i The number of APT terminals; M i Indicates satellite v i The upper limit of APT terminals that can be owned; N i and N i They represent the satellite's data queue length and buffer size respectively.

[0042] Preferably, obtaining the trained networking decision model includes:

[0043] A networking decision model based on the deep reinforcement learning algorithm DQN is constructed. The satellite networking decision process is set as a Markov decision process with state space, behavior space and reward function.

[0044] The satellite networking strategy depends on the satellite state, so the state space of the agent is represented by the state of all satellites s = {s 1 ,s 2 ,...,s n}, the state space is further represented as the congestion state of all satellites; the congestion state of a satellite is the ratio of the satellite's data queue size to the buffer size, and the agent interacts with the environment to obtain the congestion state of the satellite;

[0045] The connection paradigm set of the LEO satellite network is set as the behavior space of the agent; the inverse of the objective function of the optimization target is set as the reward function of the agent; the networking decision model is trained, and the agent interacts with the environment to obtain the environmental state at time t, that is, the congestion status s of the satellite network t , input the state into the neural network to obtain the value corresponding to each behavior in the behavior space, use the ò-greedy strategy to make decisions to obtain the networking paradigm, apply it to each satellite, and obtain the reward value. Repeat this process for training to obtain the trained networking decision model.

[0046] Preferably, the networking decision model takes into account satellite failures during training and when using the model to make networking decisions, and updates the value corresponding to the behavior in the behavior space of the agent;

[0047] The agent evaluates the Q corresponding to the action in the action space through a neural network θ (s t ,a t ) value, considering the case of satellite failure, in order to solve the change of action space size caused by satellite failure, it is necessary to adjust the value Q of the connection paradigm θ (s t ,a t ) Perform AND operation using mask;

[0048] In the case of satellite failure, the networking connection paradigm adopted cannot make the entire satellite network form a connected graph due to the satellite failure. The mask of this type of connection paradigm is set to 0, and the mask of the networking paradigm that can still be connected in the case of satellite failure is set to 1; then the value of the connection paradigm that cannot form a strong connected graph when there is a faulty satellite is masked, and the intelligent agent selects the connection paradigm corresponding to the action with high value in the utilization stage, and randomly selects the connection paradigm corresponding to the action with non-zero value in the exploration stage.

[0049] The embodiment of the present invention further provides a low-orbit satellite communication network system, including:

[0050] Design modules for building a single-layer ultra-dense LEO satellite network and modeling the LEO satellite network as a topology graph;

[0051] A paradigm module is used to determine the satellite view according to the relative positions of the satellite nodes in the topology map; and to form a set of connection paradigms of the LEO satellite network according to the satellite view;

[0052] An optimization module is used to obtain the link delay of the LEO satellite end-to-end communication according to the operation status of the LEO satellite network, and determine the optimization target of the delay to establish an objective function for minimizing the delay;

[0053] The algorithm module is used to abstract a virtual object agent for the entire LEO satellite network to make networking decisions based on the similarity of satellite views in the LEO satellite network and the similarity of the connection paradigms of each satellite. The agent selects a connection paradigm to apply to all satellites; the agent interacts with the environment to obtain the congestion status of all satellites;

[0054] Based on the deep reinforcement learning algorithm DQN, a networking decision model is established;

[0055] In the training module, the intelligent agent trains the networking decision model according to the connection paradigm set of the LEO satellite network, the objective function of minimizing the delay and the congestion status of all satellites, and obtains the trained networking decision model for the LEO satellite network.

[0056] The embodiment of the present invention provides a low-orbit satellite communication network networking method and system, which has the following beneficial effects compared with the prior art:

[0057] The present invention proposes a novel satellite view design, through which all possible connection paradigm sets between satellites are obtained, and a networking decision model is established according to a deep reinforcement learning algorithm DQN, the congestion states of all satellites are set as the state space of an intelligent agent in the networking decision model, the final connection paradigm set is set as the behavior space of the intelligent agent, and a reward function related to end-to-end delay is considered to train the networking decision model; the process eliminates the phenomenon of many transmission hops and a single structure existing in traditional technologies by forming a connection paradigm set of satellite views, optimizes the service quality of the satellite network according to the delay during satellite communication, selects the best connection paradigm as the overall networking decision of the satellite network, so that the networking decision can be reasonably adjusted according to the current overall network state of the satellite network, and the satellite network can make networking decisions on demand according to different network states, so as to build a better global satellite network communication system. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A schematic diagram of a low-orbit satellite communication network networking method provided by an embodiment of the present invention;

[0059] Figure 2 A schematic diagram of a selection process of a low-orbit satellite communication network networking method using symmetry to simplify the connection paradigm provided by an embodiment of the present invention;

[0060] Figure 3 A schematic diagram of a networking decision process based on a connection paradigm for a low-orbit satellite communication network networking method provided by an embodiment of the present invention;

[0061] Figure 4 A schematic diagram showing a comparison of average data packet delays of a traditional grid networking method of a low-orbit satellite communication network networking method provided by an embodiment of the present invention and the method of the present invention under different satellite view sizes;

[0062] Figure 5 A schematic diagram showing a comparison of average data packet delays between a low-orbit satellite communication network networking method provided by an embodiment of the present invention and a traditional grid networking method when there is no satellite failure;

[0063] Figure 6 A schematic diagram showing a comparison of average data packet delays between a method of networking a low-orbit satellite communication network provided by an embodiment of the present invention and a traditional grid networking method when the satellite failure rate is 3%;

[0064] Figure 7 A schematic diagram showing a comparison of average data packet delays between a method of networking a low-orbit satellite communication network provided by an embodiment of the present invention and a traditional grid networking method when the satellite failure rate is 6%;

[0065] Figure 8 A schematic diagram showing a comparison of average data packet delays between a method of networking a low-orbit satellite communication network provided by an embodiment of the present invention and a traditional grid networking method when the satellite failure rate is 9%;

[0066] Fig. 9 A schematic diagram showing a comparison of the cumulative distribution function of the number of hops between a method of establishing a low-orbit satellite communication network provided by an embodiment of the present invention and a traditional grid networking method when there is no satellite failure;

[0067] Fig.10 A schematic diagram showing a comparison of the cumulative distribution function of the number of hops between a method of networking a low-orbit satellite communication network provided by an embodiment of the present invention and a traditional grid networking method when the satellite failure rate is 3%;

[0068] Fig.11A schematic diagram showing a comparison of the cumulative distribution function of the number of hops between a method of networking a low-orbit satellite communication network provided by an embodiment of the present invention and a traditional grid networking method when the satellite failure rate is 6%;

[0069] Fig.12 A schematic diagram comparing the cumulative distribution function of the number of hops of a low-orbit satellite communication network networking method provided by an embodiment of the present invention and a traditional grid networking method when the satellite failure rate is 9%. DETAILED DESCRIPTION

[0070] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific embodiments disclosed below.

[0071] See also Figure 1 The embodiment of the present invention provides a low-orbit satellite communication network networking method, comprising the following steps:

[0072] Step S1: Build a single-layer ultra-dense LEO satellite network based on the Walker constellation and model the dense low-orbit satellite network.

[0073] Step S2: Based on the satellite view, a certain number of satellites are selected from the view to establish intersatellite links. All possible combinations of satellites form a connection paradigm set, which is further simplified and filtered using symmetry and connectivity.

[0074] Step S3: Based on the constructed single-layer LEO satellite network model, the service quality index is modeled; according to the needs of low-orbit satellite communications, the optimization objectives related to the delay are defined based on the service quality index, and the objective function of minimizing the delay is established for optimization.

[0075] Step S4: Based on the similarity of satellite views and connection paradigms, an intelligent agent is abstracted for the entire satellite network as a networking decision model, and combined with a deep reinforcement learning algorithm, a single-agent deep reinforcement learning networking decision algorithm is constructed.

[0076] Step S5: Use a single-agent deep reinforcement learning networking decision algorithm to train the agent to obtain a networking decision model.

[0077] Step S6: Use the trained networking decision model to make networking decisions, input the status of the satellite network, output the networking connection paradigm, and apply it to each satellite.

[0078] Specifically:

[0079] Step S1: Build a single-layer ultra-dense LEO satellite network and model it.

[0080] The present invention selects the Walker constellation, and models the ultra-dense LEO satellite network under consideration as a graph G(V,E), which specifically includes n satellite nodes, where V represents a set of satellite nodes, and E represents a set of inter-satellite links; each satellite node is equipped with four APT terminals, which means that each satellite establishes links with at most four other satellites; end-to-end communication between satellites often requires multiple satellite relays, so the present invention only focuses on the inter-satellite communication part and ignores the satellite-to-ground communication part.

[0081] The satellite node set is represented as:

[0082] V={v 1 ,v 2 ,...,v n};

[0083] The set of intersatellite links is expressed as:

[0084] E={e i,j |i,j∈{1,2,...,n}};

[0085] For an end-to-end path, such as a satellite v src To satellite v dst The end-to-end path can be expressed as:

[0086] path src,dst = {v src ,e src,j ,v j ,...,v dst},

[0087] Where: e src,j Indicates the satellite v src To satellite v j optical link.

[0088] Step S2: Obtain a set of connection paradigms based on satellite views, and simplify and filter them using symmetry and connectivity; specifically, the following steps are included:

[0089] Step S21: In the traditional grid networking mode, each satellite needs to establish four inter-satellite links with its adjacent satellites, of which two intra-orbit links connect two adjacent satellites on the same orbit, and the other two inter-orbit links connect the nearest two satellites in the adjacent orbit. Therefore, the satellite view can be determined based on this; when there is only the current satellite in the satellite view, the satellite view is 1×1; when the satellite view is 3×3, for a single satellite, it can choose to connect to two adjacent satellites on the same orbit. xTo establish a link, you can also choose the nearest satellite v in an adjacent orbit y Establish a link and optionally connect to the satellite v y Two adjacent satellites on the same orbit; when the satellite view is n×n (n must be an odd number), the current satellite can select the satellites above and below it on the same orbit. You can also choose to establish links with satellites on the left and right sides of the adjacent orbits. For the n-1 satellites on the left and right, the current satellite can also establish links with the n-1 satellites above and below the corresponding orbits. Links are established with 1 satellite, thus forming an n×n view of the satellite.

[0090] Step S22: In the Walker constellation, traditional satellites are usually equipped with four APT terminals. Assuming that the satellite view is m×m, the networking based on the connection paradigm allows each satellite to select any satellite within its own view that does not exceed the number of APT terminals to establish a link; therefore, for a satellite with a certain number of APT terminals, networking is equivalent to a combinatorial problem with a time complexity of O(C(m 2 ,4))≈O(m 8 ), depends on the size of the satellite’s view; although in practice the satellite’s view is limited and m is not particularly large (usually 3, 5, 7, etc.), the time complexity of the algorithm is still relatively high.

[0091] Step S23: Considering the dynamics of the satellite network and the necessity of maintaining a balance in satellite communication resources, and simplifying the networking algorithm, the present invention designs the networking connection to be centrally symmetrical, that is, only the way of establishing links with half of the satellites in the satellite view needs to be considered, and the other half of the links are centrally symmetrical about the satellite; therefore, the time complexity is reduced to like Figure 2 As shown, the present invention designs the networking connection to be centrally symmetrical; the central satellite is the satellite that needs to be networked, and in its 5×5 view, half of the satellites are pruned due to the central symmetry. After the central satellite selects the optional satellite, the link is centrally symmetrically found to find the corresponding pruned satellite. This method significantly reduces the time complexity of obtaining a set consisting of all possible connection paradigms.

[0092] Step S24: The intelligent agent independently selects the connection paradigm of the satellite, and cannot guarantee that any pair of satellites in the global topology are connected; therefore, by evaluating whether the satellite topology graph G of the application paradigm is a strongly connected graph, the connection paradigm is further filtered, and the filtered connection paradigm can be used as an alternative set for dynamic switching links of LEO satellites; in addition, a faulty satellite may also cause some connection paradigms in the filtered set to become unavailable, and the dimensions of the available connection paradigm sets are different under different fault conditions, but the output dimension of a single neural network must be fixed. Therefore, in this step, for the purpose of fixing the output dimension of the neural network, the present invention does not consider the situation of a faulty satellite, thereby reducing the training cost.

[0093] Step S3: define the optimization target related to the delay and establish the objective function of minimizing the delay.

[0094] In satellite communication scenarios, latency is one of the most important service quality indicators, which directly affects satellite communication efficiency and user experience. Therefore, the present invention analyzes the satellite network from the perspective of latency indicators, defines the optimization goals related to latency, and establishes an objective function for minimizing latency, so that it meets higher real-time requirements, thereby improving user experience.

[0095] Specifically, satellite v i To satellite v j The one-hop delay of can be composed of sending delay, propagation delay and queuing delay, which can be calculated by the following formula:

[0096]

[0097] in: Indicates satellite v i and satellite v j The distance between them; ι represents the data transmission rate in free space; l represents the size of the data packet; Indicates satellite v i The sending rate; Indicates satellite v i The number of packets buffered.

[0098] The end-to-end delay is obtained by adding up multiple single-hop delays, which can be expressed as:

[0099]

[0100] Where: T represents the path src,dst The end-to-end delay is Indicates the satellite v i To satellite v j One-hop delay.

[0101] According to the above formula, the optimization goal of the present invention is to minimize the sum of end-to-end delays, and its objective function is:

[0102]

[0103] stx i ≤M i ,

[0104] N i ≤N i ,

[0105] Where: x i Indicates satellite v i The number of APT terminals; M i Indicates satellite v i The upper limit of APT terminals that can be owned; N i and N i They represent the satellite's data queue length and buffer size respectively.

[0106] Step S4: Based on the similarity of satellite views and connection paradigms, an intelligent agent is abstracted for the entire satellite network as a networking decision model.

[0107] In the real satellite network topology, there are a large number of satellite nodes, and each satellite needs to be networked, but it is unrealistic to set up a networking decision model for each satellite; therefore, based on the idea of ​​connection paradigm, for the purpose of simplification, the views of each satellite in the present invention are the same, and without considering the faulty satellite, the connection paradigm set obtained based on step S2 should also be the same; therefore, it is only necessary to abstract a virtual object, namely an intelligent agent, for the entire satellite network to make a networking decision, which selects a connection paradigm and applies it to all satellites.

[0108] In the present invention, the intelligent agent first uses deep reinforcement learning to fit the Q table using a neural network to solve the problem of too large state-action space. The intelligent agent continuously explores and learns to minimize the objective function and optimize the networking decision. The intelligent agent selects a suitable networking connection paradigm according to the satellite network status through the neural network and obtains reward feedback. This is repeated interaction and training, and finally a decision network with superior performance is trained, which is the networking decision model.

[0109] Step S5: Construct a deep reinforcement learning networking decision algorithm.

[0110] The DQN algorithm in deep reinforcement learning is used to optimize the objective function defined in step S3, and the agent interacts with the environment to make networking decisions, such as Figure 3As shown, the set of connection paradigms obtained in step S2 is used as the agent in the behavior space to make decisions. The selected connection paradigm is the networking strategy to be adopted. All satellites are networked according to this networking paradigm and obtain a reward value. This step is repeated and training is performed to minimize the end-to-end delay of the satellite network.

[0111] Specifically, in the process of satellite networking decision-making, the state at the next moment is only related to the state at the current moment, but not to the previous state. Therefore, it can be regarded as a Markov decision process with state space, behavior space and reward function. The state space, behavior space and reward function can be expressed as:

[0112] ① State space: The satellite networking strategy depends on the satellite state, so the state space can be expressed as the state of all satellites s = {s 1 ,s 2 ,...,s n}; At the same time, since the congestion level of the satellite has a great impact on the delay, the satellite v i The state can be represented as For a neural network, the dimensionality of the neural network input should not change, so in order to account for the change in state space size caused by satellite failure, the state of the failed satellite is set to 1.

[0113] ② Behavior space: The set of filtered connection paradigms obtained in step S2 is the behavior space of the agent. For a neural network, the dimension of the neural network output should not change, that is, the dimension of the behavior space should not change; but in the case of satellite failure, different satellite failures may cause different connection paradigms to be unavailable, and the dimension of the behavior space may change; therefore, it is necessary to solve the change in the dimension of the behavior space caused by satellite failure.

[0114] ③ Reward function: The reward function is defined as the opposite of the objective function in step S3 and normalized; the objective function represents the size of the network delay after adopting a certain networking strategy, and the greater the delay, the lower the corresponding reward value should be; correspondingly, the smaller the delay, the higher the corresponding reward value should be.

[0115] In order to solve the delay optimization in step S3, the agent obtains the current state s by interacting with the environment. t , take a certain method to select action a t And apply this networking paradigm to get the reward value r t and the next state s t+1 Composition t ,a t ,r t ,s t+1>Store in the experience pool, then randomly sample from the experience pool and calculate the loss function, continuously update and iterate the network to minimize the end-to-end delay.

[0116] The loss function of the algorithm is:

[0117]

[0118] Where: N represents the size of a batch of data samples randomly drawn from the experience replay pool D; θ represents the policy network parameters; y i represents the decision of the target network for sample i, which can be expressed as:

[0119]

[0120] Where: γ represents the discount function; θ - Represents the target network parameters.

[0121] At the same time, considering the situation of satellite failure, the change of action space size caused by satellite failure is solved; the agent interacts with the environment to obtain the environmental state at time t, that is, the congestion status s of the satellite network t , according to the state, use the ò-greedy strategy to make a decision and get behavior a t The greedy strategy ò-greedy is to balance exploration and utilization. In the early stage of network training, as much exploration as possible is performed; in the later stage of training, almost only utilization is performed; the agent evaluates the Q corresponding to the action in the action space through the neural network θ (s t ,a t ) value, considering the case of satellite failure, in order to solve the change of action space size caused by satellite failure, it is necessary to adjust the value Q of the connection paradigm θ (s t ,a t ) uses a mask to perform AND operations; in the case of satellite failure, it is possible that the networking connection paradigm used cannot form a connected graph for the entire network due to the satellite failure. The mask of this connection paradigm is set to 0, and the mask of the networking paradigm that can still be connected in the case of satellite failure is set to 1; therefore, through this method, the value of the connection paradigm that cannot form a strong connected graph when there is a faulty satellite is masked, and the intelligent agent selects the connection paradigm corresponding to the action with the highest value in the utilization stage, and randomly selects the connection paradigm corresponding to the action with a value not equal to 0 in the exploration stage.

[0122] Step S6: The agent is trained using a deep reinforcement learning network decision algorithm; specifically, the training includes:

[0123] Step S61: Initialize the satellite network topology G(V,E), the experience replay pool D, the batch size N and the discount function γ.

[0124] Step S62: Input the number of training episodes max The number of slots required for networking for each training round is 1, and each round of each training round uses the Open Shortest Path First (OSPF) routing algorithm to perform a round of data packet forwarding.

[0125] Step S63: Initialize the policy network Q with random network parameters θ θ (s,a), using the same parameters θ - ←θ Initialize the target network Q θ- (s,a).

[0126] Step S64: Use the connection paradigm set obtained by the method in step S2 as the behavior space.

[0127] Step S65: Determine whether the current training number is less than episode max If it is less than, jump to step S66; if it is greater than or equal to, jump to step S616.

[0128] Step S66: The round number t is set to 0.

[0129] Step S67: Determine whether the current networking round t is less than slot. If so, jump to step S68; if so, jump to step S612.

[0130] Step S68: Get the state s of wheel t t , input the policy network and perform mask operation to obtain Q θ (s t ,a t ).

[0131] Step S69: Select the action a corresponding to the highest reward value in the case of utilization t As the networking strategy for each satellite, randomly select an available action a in the exploration case t As the networking strategy for each satellite.

[0132] Step S610: Apply a t , get reward r t and the next state s t+1 , and <s t ,a t ,r t ,s t+1 >Deposit into experience replay pool D.

[0133] Step S611: perform t+1 and jump to step S67.

[0134] Step S612: Determine whether the amount of data in the experience pool is greater than or equal to the batch size. If so, jump to step S613. If less than, increase the number of training times by one and jump to step S65.

[0135] Step S613: Randomly extract a sample of size N from the experience replay pool D and calculate the loss function.

[0136] Step S614: Update the policy network Q by minimizing the loss function θ , and update the target network parameter Q after a certain period of time θ- .

[0137] Step S615: After the number of training times is increased by one, jump to step S65.

[0138] Step S616: After the training is completed, a trained neural network is obtained, which is the networking strategy.

[0139] Step S617: Use the trained neural network to obtain the Q value, and the intelligent agent selects the action with the largest Q value after masking as the networking paradigm of each satellite.

[0140] Step S7: Testing the networking strategy of the dense low-orbit satellite network.

[0141] like Figure 4 As shown in the figure, the average data packet delay of the traditional grid method and the method of the present invention under different satellite views when there is no satellite failure. As the data generation rate continues to increase, the network congestion level continues to increase, so the average data packet delay increases accordingly; when the view size is 3×3 or 5×5, it can be seen from the figure that the performance of the method proposed by the present invention is better than the traditional grid networking method; in addition, since a larger view means more available connection paradigms, multiple satellites may no longer be required for relaying between satellites at a longer distance, so the average data packet delay decreases as the view size increases; however, since a larger view size will result in a higher connection paradigm calculation cost, the view size cannot be increased indefinitely, and thus the present invention adopts a view size of 5×5 in specific analysis.

[0142] like Figure 5 As shown in FIG. 1 , the average packet delay of the method of the present invention and the traditional grid method when there is no satellite failure is compared. When the data generation rate is low, the average packet delay of the two methods is similar, which indicates that all data packets can be transmitted smoothly under the two methods; however, as the data generation rate increases, the performance of the traditional grid method decreases rapidly, while the performance of the method of the present invention decreases more slowly, which can be attributed to the inherent limitations of the traditional method, including a single form, a large number of hops, and inadaptability to changes in satellite network status; Figure 6 , Figure 7 and Figure 8As shown in the figure, as the satellite failure rate increases, the performance of the traditional grid method decreases rapidly. The performance of the method proposed in the present invention is still better than the traditional algorithm. This can be attributed to the fact that the traditional grid networking method lacks the ability to deal with satellite failures. In contrast, the method proposed in the present invention can provide more networking options according to the changes in satellite status and reduce the performance loss caused by satellite failures.

[0143] like Fig. 9 As shown in FIG. 1 , the cumulative distribution function of the routing hops of the method of the present invention and the traditional grid networking method when there is no satellite failure is compared. It can be seen that the method of the present invention helps to reduce the number of satellite routing relay hops, and the data packet can reach the destination through fewer relay nodes; Fig.10 , Fig.11 and Fig.12 As shown in the figure, with the increase of satellite failure rate, the number of hops of the traditional grid networking method increases significantly, while the method of the present invention shows a slower increase and the cumulative distribution function of the number of hops is always better than the traditional algorithm. This advantage can be attributed to the method proposed in the present invention. Compared with the grid networking algorithm, it not only selects a networking strategy with fewer routing hops, but also has better robustness against satellite failure conditions.

[0144] The present invention innovatively proposes a concept of satellite view. Based on the satellite view, all possible connection paradigm sets are calculated, simplified and filtered. Then, a single-agent deep reinforcement learning method is used, so that the agent can reasonably make networking decisions based on the state of the overall satellite network in combination with satellite failure conditions, continuously optimize the network service quality according to the objective function, and reduce the number of hops required for end-to-end transmission, thereby improving the robustness, flexibility and QoS indicators of the satellite network.

[0145] The present invention can construct a set of connection paradigms for satellite networking by proposing the concept of "view" when the number of APT terminals and the size of the buffer zone are limited, and adjust the networking strategy according to the state of the satellite network, thereby optimizing the service quality index. It can be used to solve the problem of making networking decisions on demand for different network states in dense low-orbit satellite network application scenarios.

[0146] An embodiment of the present invention also provides a networking decision system for a dense low-orbit satellite network, including a design module, a paradigm module, an optimization module, an algorithm module, a training module and a decision module.

[0147] Design module: Build a single-layer Walker constellation to model a dense low-orbit satellite network.

[0148] Paradigm module: Based on the satellite view, a certain number of satellites are selected from the view to establish intersatellite links. All possible combinations of satellites form a connection paradigm set, which is further simplified and filtered using symmetry and connectivity.

[0149] Optimization module: Based on the built single-layer LEO satellite network model, the service quality indicators are modeled. In view of the needs of low-orbit satellite communications, the objective function is established based on the service quality indicators for optimization.

[0150] Algorithm module: Based on the similarity of satellite views and connection paradigms, an intelligent agent is abstracted for the entire satellite network as a networking decision model. Combined with the deep reinforcement learning algorithm, it is a single-agent deep reinforcement learning networking method.

[0151] The training module uses the constructed single-agent deep reinforcement learning networking method to train the agent and obtain the networking decision-making model.

[0152] Decision module: Use the networking decision model obtained from the training module to make networking decisions, input the status of the satellite network, output the networking connection paradigm, and apply it to each satellite.

[0153] In the specific experiment of the present invention, the present invention adopts an 8×8 satellite network topology, that is, there are 8 satellite orbits, 8 satellites are evenly distributed in each orbit, and each satellite is equipped with 4 sets of transceivers; the orbital inclination is set to 55°, and the orbital altitude is 550km; in addition, it is assumed that the data transmission rate of each satellite is 5Gbps, the cache size is 1875MB, and the size of a single data packet is 20KB; the satellite routing algorithm uniformly adopts the Open Shortest Path First (OSPF) algorithm.

[0154] Therefore, the method of the present invention can adjust the networking strategy according to the state of the satellite network under the condition that the number of APT terminals and the buffer size are limited, so as to achieve the goal of minimizing the delay. Specifically, the present invention first builds a single-layer ultra-dense LEO satellite network and models it. Secondly, a connection paradigm set is obtained based on the satellite view, and symmetry and connectivity are used to simplify and screen it. Then, the optimization goal of the delay is defined, and the objective function of minimizing the delay is established. Based on the similarity of the satellite view and the connection paradigm, a virtual object, namely an intelligent agent, is abstracted for the entire satellite network to make a networking decision. After that, a deep reinforcement learning networking decision algorithm is constructed, and the detailed elements therein are defined, and the intelligent agent is trained using the proposed networking algorithm. Finally, the networking strategy of the dense low-orbit satellite network is tested. The experimental results show that compared with the traditional grid networking method, the present invention effectively reduces the number of routing hops, the average data packet delay is reduced by 30%, the communication delay can be reduced, the number of hops required for satellite routing is reduced, the service quality of satellite communication is optimized, and the flexibility of networking is improved. In addition, it can also cope with emergencies such as satellite failures, and significantly improve the robustness of the satellite network while optimizing the QoS index.

[0155] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for establishing a low-orbit satellite communication network, characterized in that: The following steps are involved: Build a single-layer ultra-dense LEO satellite network and model the LEO satellite network as a topology graph; Determine the satellite view according to the relative positions of the satellite nodes in the topology map; And based on the satellite view, a set of connection paradigms for the LEO satellite network is formed; According to the operation status of the LEO satellite network, the link delay of the LEO satellite end-to-end communication is obtained, and the optimization target of the delay is determined to establish the objective function of minimizing the delay; Based on the similarity of satellite views in the LEO satellite network and the similarity of the connection paradigms of each satellite, a virtual object agent is abstracted for the entire LEO satellite network to make networking decisions. The agent selects a connection paradigm to apply to all satellites; the agent interacts with the environment to obtain the congestion status of all satellites; Based on the deep reinforcement learning algorithm DQN, a networking decision model is established. The intelligent agent trains the networking decision model according to the connection paradigm set of the LEO satellite network, the objective function of minimizing the delay and the congestion status of all satellites, and the trained networking decision model is obtained for the networking of the LEO satellite network.

2. A low-orbit satellite communication network networking method according to claim 1, characterized in that: The single-layer ultra-dense LEO satellite network is constructed, and the LEO satellite network is modeled as a topology graph, including: A single-layer ultra-dense LEO satellite network is built based on the Walker constellation configuration, and the LEO satellite network is modeled as a topology graph G(V,E), which includes n satellite nodes, several inter-satellite links and end-to-end links, where V represents the set of satellite nodes and E represents the set of inter-satellite links. Then the satellite node set is expressed as: V={v1,v2,...,v n }; The set of intersatellite links is expressed as: E={e i,j |i,j∈{1,2,...,n}}; For an end-to-end path, such as a satellite v src To satellite v dst The end-to-end path is expressed as: path src,dst ={v src ,e src,j ,v j ,...,v dst }, Where: e src,j Indicates the satellite v src To satellite v j optical link.

3. A low-orbit satellite communication network networking method according to claim 1, characterized in that: The determining of the satellite view comprises: When there is only the current satellite in the satellite view, the satellite view is 1×1; In the traditional grid networking method, each satellite needs to establish four inter-satellite links with its adjacent satellites, two of which are intra-orbit links connecting two adjacent satellites in the same orbit, and the other two inter-orbit links connecting the two nearest satellites in adjacent orbits, based on which the satellite view is determined; When the satellite view is 3×3, for a single satellite, it can choose to be connected to two adjacent satellites on the same orbit. x To establish a link, you can also choose the nearest satellite v in an adjacent orbit y Establish a link and optionally connect to the satellite v y Two adjacent satellites in the same orbit; When the satellite view is n×n (n must be an odd number), the current satellite can select the satellites above and below it in the same orbit. You can also choose to establish links with satellites on the left and right sides of the adjacent orbits. For the n-1 satellites on the left and right, the current satellite can also establish links with the n-1 satellites above and below the corresponding orbits. Links are established with 1 satellite, thus forming an n×n view of the satellite.

4. A low-orbit satellite communication network networking method according to claim 1, characterized in that: The set of connection paradigms forming the LEO satellite network includes: In the Walker constellation configuration, once the satellite’s view is determined, the satellite can select any satellite within its view that does not exceed the number of APT terminals to establish a link; If the networking mode between satellites is set to center symmetry, then only half of the links corresponding to half of the satellites in the satellite view need to be considered, and the other half of the links are symmetric about the corresponding satellite center; Based on the similarity between satellite views and connection paradigms in a fault-free state, the connection paradigm set of the entire satellite network is obtained. At the same time, the connection paradigms in the connection paradigm set are filtered by evaluating whether the satellite topology graph to which the connection paradigm is applied is a strongly connected graph, thus forming a set of available connection paradigms for the LEO satellite network.

5. A low-orbit satellite communication network networking method according to claim 1, characterized in that: After obtaining the link delay during the end-to-end communication of the LEO satellite, the end-to-end link delay in the LEO satellite network is used as an optimization target; Satellite v i To satellite v j The single-hop delay of is composed of sending delay, propagation delay and queuing delay, which can be expressed as: in: Indicates satellite v i and satellite v j The distance between them; ι represents the data transmission rate in free space; l represents the size of the data packet; Indicates satellite v i The sending rate; Indicates satellite v i The number of packets cached; The delay of an end-to-end link is obtained by adding up multiple single-hop delays and is expressed as: Where: T represents the path src,dst The end-to-end delay is Indicates the satellite v i To satellite v j One-hop delay; The objective function of the optimization target is expressed as: Where: x i Indicates satellite v i The number of APT terminals; M i Indicates satellite v i The upper limit of APT terminals that can be owned; N i and N i They represent the satellite's data queue length and buffer size respectively.

6. A method for establishing a low-orbit satellite communication network according to claim 5, characterized in that: The step of obtaining the trained networking decision model includes: A networking decision model based on the deep reinforcement learning algorithm DQN is constructed. The satellite networking decision process is set as a Markov decision process with state space, behavior space and reward function. The satellite networking strategy depends on the satellite state, so the state space of the agent is represented by the states of all satellites s = {s1, s2, ..., s n }, the state space is further represented as the congestion state of all satellites; the congestion state of a satellite is the ratio of the satellite's data queue size to the buffer size, and the agent interacts with the environment to obtain the congestion state of the satellite; The connection paradigm set of the LEO satellite network is set as the behavior space of the agent; the inverse of the objective function of the optimization target is set as the reward function of the agent; the networking decision model is trained, and the agent interacts with the environment to obtain the environmental state at time t, that is, the congestion status s of the satellite network t , input the state into the neural network to obtain the value corresponding to each behavior in the behavior space, use the ò-greedy strategy to make decisions to obtain the networking paradigm, apply it to each satellite, and obtain the reward value. Repeat this process for training to obtain the trained networking decision model.

7. A method for establishing a low-orbit satellite communication network according to claim 6, characterized in that: The networking decision model takes into account satellite failures during training and when using the model to make networking decisions, and updates the value corresponding to the behavior in the behavior space of the agent; The agent evaluates the Q corresponding to the action in the action space through a neural network θ (s t ,a t ) value, considering the case of satellite failure, in order to solve the change of action space size caused by satellite failure, it is necessary to adjust the value Q of the connection paradigm θ (s t ,a t ) Perform AND operation using mask; In the case of satellite failure, the networking connection paradigm adopted cannot make the entire satellite network form a connected graph due to the satellite failure. The mask of this type of connection paradigm is set to 0, and the mask of the networking paradigm that can still be connected in the case of satellite failure is set to 1; then the value of the connection paradigm that cannot form a strong connected graph when there is a faulty satellite is masked, and the intelligent agent selects the connection paradigm corresponding to the action with high value in the utilization stage, and randomly selects the connection paradigm corresponding to the action with non-zero value in the exploration stage.

8. A low-orbit satellite communication network system, characterized in that: include: Design modules for building a single-layer ultra-dense LEO satellite network and modeling the LEO satellite network as a topology graph; The paradigm module is used to determine the satellite view based on the relative positions of the satellite nodes in the topology map; And based on the satellite view, a set of connection paradigms for the LEO satellite network is formed; An optimization module is used to obtain the link delay of the LEO satellite end-to-end communication according to the operation status of the LEO satellite network, and determine the optimization target of the delay to establish an objective function for minimizing the delay; The algorithm module is used to abstract a virtual object agent for the entire LEO satellite network to make networking decisions based on the similarity of satellite views in the LEO satellite network and the similarity of the connection paradigms of each satellite. The agent selects a connection paradigm to apply to all satellites; the agent interacts with the environment to obtain the congestion status of all satellites; Based on the deep reinforcement learning algorithm DQN, a networking decision model is established; In the training module, the intelligent agent trains the networking decision model according to the connection paradigm set of the LEO satellite network, the objective function of minimizing the delay and the congestion status of all satellites, and obtains the trained networking decision model for the LEO satellite network.

Images