Intelligent load balancing method and system for double-layer mega constellation

Abstract

The application discloses a kind of intelligent load balancing methods and systems for double-layer giant constellation, generate double-layer giant constellation network topology;Establish double-layer giant constellation clustering management mechanism;Cluster head periodically collects each node information in cluster and its remaining cluster head information;Based on the remaining cluster head information, the cluster head of this cluster uses long short-term memory network to complete the remaining cluster load prediction;Construct multi-agent deep reinforcement learning load balancing model, complete state space, action space, reward function, observation space mapping;Satellite on-board intelligent agent completes routing decision according to part observation;Cluster head periodically collects experience information in cluster, completes multi-agent reinforcement learning centralized network training;Cluster head periodically issues intelligent agent network to each satellite, each satellite completes online routing decision strategy update, and completes routing decision according to this strategy.The application realizes the intelligent load balancing of double-layer low-orbit giant constellation.

Description

Technical Field

[0001] This invention belongs to the field of wireless communication technology, specifically relating to an intelligent load balancing method and system for a two-layer mega-constellation. Background Technology

[0002] The next-generation communication system, 6G (6-Generation), aims to provide globally covered, low-latency, and high-bandwidth services to achieve high-speed interconnection across air, space, and sea. Low Earth orbit (LEO) satellite constellations, with their wide coverage and low latency, are becoming an important component of future networks. With the reduction in satellite launch and equipment costs, mega-constellations in LEO are rapidly developing. The most significant characteristic of mega-constellations in LEO is their large number of satellites, typically reaching tens of thousands. For example, SpaceX's Starlink constellation had deployed over 3,600 satellites as of February 2023, with plans to eventually deploy over 42,000. While the expansion of LEO satellite constellations increases the transmission capacity and expands communication coverage, it also presents significant challenges to the system design and management of satellite networks, especially in terms of load balancing strategies.

[0003] Due to the large number of satellites, mega-constellations of low-Earth orbit (LEO) satellites experience a significantly higher number of hops compared to traditional LEO satellite constellations, posing a substantial challenge to routing planning. As the number of satellites increases, the number of relays for transmission tasks gradually rises, leading to a rapid increase in the computational burden of routing decisions. The overhead of routing decisions consumes a significant amount of resources in mega-constellations, including computational resources for route calculation and transmission resources for data packets. However, mega-constellations are often constrained by equipment size limitations, resulting in insufficient computational and transmission resources. Furthermore, frequent changes in satellite network topology also pose a significant challenge to routing calculations for satellite network transmission. Additionally, the sheer size and limited equipment resources of mega-constellations prevent them from performing global routing calculations.

[0004] Deep reinforcement learning (DRL) is considered one of the best solutions for Artificial General Intelligence (AGI). Agents using DRL algorithms can dynamically perceive changes in their environment and acquire learning experience through interaction with the environment. DRL typically uses deep neural networks (DNNs) to approximate value functions or policy functions, and can adapt to large-scale environments and sample data. DRL has achieved great success in fields such as games, autonomous driving, and robot control. Simultaneously, DRL has also been widely used in wireless communication, such as dynamic intelligent routing path selection.

[0005] Multi-Agent Deep Reinforcement Learning (MADRL) represents a further development in deep reinforcement learning. It primarily addresses the collaboration problem among multiple agents in complex environments to maximize global reward. In more complex environments, agents often lack awareness of the global state, only observing information within a limited radius, thus hindering optimal decision-making. MADRL achieves this by coordinating the decision-making strategies of each agent to maximize global reward. It is anticipated that MADRL will have significant advantages in handling decision-making tasks in large-scale complex networks.

[0006] In actual deployment, multiple constellations and multi-level constellations are usually used. For example, the Starlink satellite constellation uses five satellite constellations distributed at four orbital altitudes, while the domestic GW satellite constellation uses seven satellite constellations distributed at four orbital altitudes. By combining constellations at different altitudes and inclinations, more effective global coverage and better communication performance can be achieved.

[0007] The use of a two-layer satellite constellation enables two-layer transmission assistance, preventing excessive load on any single satellite constellation and achieving global load balancing in the satellite network through inter-layer assistance. Currently, no load balancing algorithm for two-layer mega-constellations has been proposed in domestic and international research. Research on two-layer constellation routing algorithms mainly focuses on the two-layer MEO-LEO architecture, where the upper-layer MEO satellites are responsible for calculating and distributing routing tables for LEO satellites within their coverage area and also handle some task transmission offloading. However, this routing algorithm cannot be applied to mega-constellations. Mega-constellation equipment is simple, and its computing resources cannot handle the massive routing calculations required by the MEO, nor can its transmission resources handle large-capacity data forwarding tasks. Secondly, due to the enormous scale of mega-constellations, collecting global topology information is too difficult, making it impossible to achieve virtual topology routing based on the global topology for the MEO. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide an intelligent load balancing method and system for two-layer mega-constellations, which addresses the shortcomings of the prior art and solves the technical problem of routing planning for two-layer mega-constellations, and achieves intelligent load balancing of mega-constellations based on their characteristics.

[0009] The present invention adopts the following technical solution:

[0010] A smart load balancing method for two-layer mega-constellations includes the following steps:

[0011] S1. Establish a two-layer mega-constellation network and obtain the topology of the two-layer mega-constellation satellite network;

[0012] S2. Based on the two-layer mega-constellation satellite network topology obtained in step S1, establish a two-layer mega-constellation cluster management mechanism;

[0013] S3. Establish a cluster load prediction mechanism based on the two-layer giant constellation cluster management mechanism established in step S2;

[0014] S4. Based on the two-layer giant constellation cluster management mechanism established in step S2 and the cluster load prediction mechanism in step S3, complete the mapping between the multi-agent deep reinforcement learning model and the two-layer giant constellation model, and construct the two-layer giant constellation multi-agent deep reinforcement learning load balancing model.

[0015] S5. Based on the two-layer giant constellation multi-agent deep reinforcement learning load balancing model constructed in step S4, the onboard agent observes the environment to obtain the observation space, selects the next-hop satellite for the transmission task based on the observation space, and completes the action execution process.

[0016] S6. Based on the two-layer giant constellation cluster management mechanism obtained in step S2 and the onboard intelligent agent action execution process obtained in step S5, the cluster head periodically completes the collection of satellite experience values ​​within the cluster; based on the cluster load prediction mechanism in step S3, the current cluster head periodically collects the load information of the remaining clusters, inputs it into the long short-term memory network, and obtains the predicted load information of the remaining clusters.

[0017] S7. Based on the two-layer giant constellation multi-agent deep reinforcement learning load balancing model constructed in step S4, and the cluster information collected by the cluster head in step S6 and the load information of the remaining clusters after prediction, the multi-agent deep reinforcement learning model completes centralized training; then the cluster head sends the trained onboard agent network to each satellite in the cluster to complete the agent network update, and completes online action decision-making based on the updated agent network.

[0018] Specifically, in step S1, in the dual-layer mega-constellation satellite network topology, each satellite in each layer of the mega-constellation is a topology node, and the intra-layer inter-satellite links and inter-layer inter-satellite links of each layer of the mega-constellation are topology edges.

[0019] Specifically, in step S2, the two-layer giant constellation cluster management mechanism is as follows:

[0020] Hierarchical and clustered management is used, with each layer of the giant constellation being managed by a cluster, and the number of satellites in each cluster being equal. Each cluster includes a cluster head and cluster members, with the cluster head managing the members within the cluster.

[0021] The cluster head is responsible for issuing routing decisions to members within the cluster and collecting information. Cluster members are responsible for executing the cluster head's routing decisions and periodically uploading their experience values.

[0022] Inter-layer inter-satellite links exist between the upper and lower cluster heads, deployed at the cluster heads of each cluster, and exchange information periodically to complete inter-layer cluster management.

[0023] Specifically, in step S3, the cluster load prediction mechanism is as follows:

[0024] The current cluster head completes the collection of load information for the remaining clusters and uses the long short-term memory network deployed at the cluster head to predict the load information of each cluster in the two-layer mega-constellation network.

[0025] Specifically, in step S4, the two-layer giant constellation multi-agent deep reinforcement learning load balancing model includes a central network and an agent network.

[0026] The centralized network is deployed at the cluster head and is responsible for periodically collecting information within the cluster and between clusters, completing the training of the agent network, and distributing the trained agent network to each satellite within the cluster.

[0027] The agent network is deployed among the other cluster members. The onboard agent network selects the next hop transmission action based on the observation space determined by the transmission mission.

[0028] Furthermore, a centralized network specifically refers to:

[0029] The input to the central network is the reward value r of each agent, and the state space s is the load information of each cluster and the predicted load information of the other clusters. The prediction of the current load information of the other clusters is completed by using a long short-term memory network deployed at the cluster head.

[0030] After completing the forward execution based on the input and state space, the centralized network completes the back propagation update based on the global reward value, and periodically transmits the updated agent network to members within the cluster.

[0031] Furthermore, the agent network specifically refers to:

[0032] The observation space o contains the transmission tasks of the current satellite node and the satellite payload information within the cluster;

[0033] Action space a is the next hop transmission direction of the transmission task, including two intra-orbit inter-satellite links and two inter-orbit inter-satellite links within the layer, and one inter-layer inter-satellite link connecting the cluster head to another layer.

[0034] The reward function r is determined by the distance to the target satellite and the load information of each layer.

[0035] Furthermore, the observation space is:

[0036] o(t)={l t ,l i ,p s ,o s ,p d ,o d ,b u ,b d ,b l ,b r}

[0037] Among them, l t l is the feature number of the current layer of the transmission task. i The layer feature number for the target satellite of the transmission mission; p i For the orbital number of the current mission satellite node, o i b is the satellite number in orbit for the current mission satellite node. i This includes the payload information of its four adjacent satellites;

[0038] The motion space is:

[0039] a(t) = {layer, l, r, u, d}

[0040] Where layer represents inter-layer transmission, l,r represents the transmission direction between orbits of the satellite constellation in this layer, and u,d represents the transmission direction within the orbit of the satellite constellation in this layer;

[0041] The reward function is:

[0042]

[0043] Where, diff is the distance from the current transmission mission to the target satellite, MaxHop is the initial distance of the transmission mission, and B... u and B d Let rp be the network load of each of the two satellite networks, and rp be the penalty factor of the reward function.

[0044] Specifically, in step S5, the next hop for the transmission task of the onboard intelligent agent can be selected in addition to the four fixed inter-satellite links within the current layer, or the inter-layer inter-satellite link can be used for transmission; the inter-layer inter-satellite link is deployed at the cluster head, and the current satellite needs to complete the transmission task to the cluster head first.

[0045] Secondly, embodiments of the present invention provide an intelligent load balancing system for a two-layer mega-constellation, comprising:

[0046] The network module is used to establish a two-layer mega-constellation network and obtain the topology of the two-layer mega-constellation satellite network;

[0047] The management module is used to establish a two-layer mega-constellation cluster management mechanism based on the two-layer mega-constellation satellite network topology obtained from the network module.

[0048] The prediction module is used to establish a cluster load prediction mechanism based on the two-layer mega-constellation cluster management mechanism established by the management module.

[0049] The construction module is used to complete the mapping between the multi-agent deep reinforcement learning model and the two-layer giant constellation model based on the two-layer giant constellation cluster management mechanism established by the management module and the cluster load prediction mechanism of the prediction module, and to construct the two-layer giant constellation multi-agent deep reinforcement learning load balancing model.

[0050] The selection module is used to construct a two-layer giant constellation multi-agent deep reinforcement learning load balancing model based on the construction module. The onboard agent observes the environment to obtain the observation space, selects the next-hop satellite for the transmission task based on the observation space, and completes the action execution process.

[0051] The collection module is used to collect satellite experience values ​​within the cluster periodically, based on the two-layer giant constellation cluster management mechanism obtained from the management module and the onboard intelligent agent action execution process obtained from the selection module. Based on the cluster load prediction mechanism of the prediction module, the current cluster head periodically collects the load information of the remaining clusters, inputs it into the long short-term memory network, and obtains the predicted load information of the remaining clusters.

[0052] The output module is used for the two-layer giant constellation multi-agent deep reinforcement learning load balancing model built based on the building module, as well as the collection module. The cluster head collects the intra-cluster information and the load information of the remaining clusters after prediction. The multi-agent deep reinforcement learning model completes the central training. Then, the cluster head sends the trained onboard agent network to each satellite in the cluster to complete the agent network update, and completes the online action decision based on the updated agent network.

[0053] Compared with the prior art, the present invention has at least the following beneficial effects:

[0054] An intelligent load balancing method for two-layer mega-constellations is proposed. It employs multi-agent deep reinforcement learning to achieve inter-layer load balancing, minimizing transmission task latency. A centralized training and distributed execution network management strategy is used, effectively reducing the computational load onboard agents and lowering network overhead. A mega-network clustering and management mechanism is established to achieve distributed clustering management of the two-layer mega-constellation, effectively reducing network management overhead. A cluster load prediction mechanism is proposed, using a long short-term memory network to predict cluster load information, effectively improving the transmission success rate of the mega-constellation network.

[0055] Furthermore, in the two-layer mega-constellation satellite network topology, each satellite is a topology node and the inter-satellite links are topology edges; the two intra-orbit inter-satellite links do not change over time; the two inter-orbit inter-satellite links change over time due to the relative motion of the satellites; the inter-layer inter-satellite links are only deployed at the cluster head, which effectively reduces the complexity of the network topology and can effectively complete the information transmission between and within layers.

[0056] Furthermore, the hierarchical cluster management model for giant constellations uses hierarchical distributed management, and the number of satellites in each cluster is the same, which effectively reduces the additional overhead caused by network management; the cluster head is only responsible for the management of satellites within the cluster, and the network management overhead is reduced from the network scale to the cluster scale, thus effectively reducing the network management overhead.

[0057] Furthermore, cluster heads periodically exchange cluster load information, reducing the flooding of network information from the network scale to the cluster head quantity scale, effectively reducing network information collection overhead; the long short-term memory network deployed at the cluster heads completes the prediction of information for the remaining cluster heads, providing more effective environmental information for the centralized network, further improving the transmission success rate of the mega-constellation network.

[0058] Furthermore, the centralized network deployed at the cluster head can effectively facilitate the collaboration of various agents within the cluster. Based on the centralized network, the decision-making strategies of each onboard agent within the cluster are guided, thereby facilitating the collaboration of intra-cluster and inter-layer transmission tasks and achieving load balancing to reduce transmission task latency. Onboard agents only need to observe a small surrounding space and make action decisions accordingly, effectively reducing the execution overhead of the agent network and extending the working life of the two-layer mega-constellation network.

[0059] Furthermore, the centralized network uses the cluster load information predicted by the Long Short-Term Memory network to complete the training and updating of the centralized network. The predicted cluster load information eliminates interference on the time scale, provides more effective environmental information, effectively improves the decision-making accuracy of the agent network, and effectively reduces the latency of transmission tasks.

[0060] Furthermore, the reward function of the agent network is jointly determined by the transmission task delay and inter-layer load. A nonlinear function is introduced into the reward function. When the distance between the current satellite and the target satellite of the transmission task is relatively large (i.e., much smaller than the initial distance MaxHop of the transmission task), the reward value changes slowly, allowing the agent to explore various cooperative possibilities. When the transmission task is close to the target satellite, the reward value changes rapidly to ensure that data packets arrive as quickly as possible and avoid interfering with other transmission tasks within the current cluster. Finally, to ensure inter-layer load balancing, two layers of network load B are introduced into the reward function. u and B d The ratio is used to prevent transmission tasks from being singled out to a particular network layer, causing excessive load on that layer. Additionally, when the next hop's transmission direction moves away from the target satellite, the reward value is defined as a penalty factor rp to prevent transmission tasks from backing up.

[0061] Furthermore, when using inter-layer inter-satellite links for transmission, the data must first be transmitted to the cluster head of the current cluster. Inter-satellite links are deployed only on the cluster head satellites, effectively reducing the equipment deployment costs of a two-layer mega-constellation network and improving its economic efficiency.

[0062] It is understandable that the beneficial effects of the second aspect mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0063] In summary, this invention effectively reduces the network management overhead of a two-layer mega-constellation by implementing hierarchical and clustered management, and provides inter-layer transmission links. Through inter-layer transmission, inter-layer traffic load balancing is achieved, allowing satellite constellations with higher loads to dynamically offload data tasks to another layer. The two-layer satellite constellation load balancing algorithm, while ensuring the delivery success rate of lower-load layers, significantly improves the overall network transmission task delivery success rate through task offloading. The introduction of the LSTM load prediction mechanism fully extracts the temporal characteristics of load information, making the environmental information obtained by the cluster head during center training more reliable, improving the performance of cooperative transmission, and increasing the delivery success rate of transmission tasks.

[0064] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0065] Figure 1 This is a diagram of the dual-layer mega-constellation satellite network structure of the present invention;

[0066] Figure 2 This is a topology diagram of the two-layer mega-constellation satellite network of the present invention;

[0067] Figure 3 This is a schematic diagram of the multi-agent deep reinforcement learning architecture of the present invention;

[0068] Figure 4 This is a flowchart of the giant constellation intelligent load balancing based on multi-agent deep reinforcement learning in this invention.

[0069] Figure 5 This is a comparison chart of the transmission success rates of giant constellation network transmission tasks when the upper-level constellation network size is 6*6 and the lower-level constellation network size is 12*12. Detailed Implementation

[0070] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0072] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0073] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this invention generally indicates that the preceding and following objects have an "or" relationship.

[0074] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0075] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0076] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0077] This invention provides an intelligent load balancing method for a two-layer mega-constellation, generating a two-layer mega-constellation network topology; establishing a two-layer mega-constellation cluster management mechanism; the cluster head periodically collects information from each node within its cluster and other cluster heads; based on the information from other cluster heads, the cluster head uses a long short-term memory network to predict the load of other clusters; a multi-agent deep reinforcement learning load balancing model is constructed, completing the mapping of state space, action space, reward function, and observation space; satellite-borne agents complete routing decisions based on partial observations; the cluster head periodically collects experiential information within its cluster to complete the training of the multi-agent reinforcement learning central network; the cluster head periodically distributes the agent network to each satellite, and each satellite updates its online routing decision strategy and completes routing decisions based on this strategy. The research object is a two-layer mega-constellation low-Earth orbit satellite network, addressing the local congestion problem and inter-layer load imbalance problem during the operation of the two-layer low-Earth orbit mega-constellation satellite network, and implementing a distributed load balancing strategy for the two-layer low-Earth orbit mega-constellation.

[0078] Please see Figure 4 This invention discloses an intelligent load balancing method for two-layer mega-constellations, comprising the following steps:

[0079] S1. Establish a two-layer giant constellation network and obtain the topology of the two-layer giant constellation network;

[0080] Please see Figure 1 This invention uses a tilted low Earth orbit (LEO) mega-constellation model. For each satellite constellation layer, each satellite is considered a communication node, and communication is achieved using four fixed inter-satellite links and inter-layer inter-satellite links. For a two-layer mega-constellation network, communication links are established using inter-layer inter-satellite links deployed at the cluster head.

[0081] Please see Figure 2In a two-layer mega-constellation satellite network topology, each satellite in each mega-constellation layer is a topology node, and the intra-layer inter-satellite links and inter-layer inter-satellite links in each mega-constellation layer are topology edges.

[0082] Due to the relative motion of satellites in the upper and lower layers, using a giant constellation with the same inclination angle for both layers reduces the difficulty of antenna tracking. It is worth noting that the application scenario of this invention can be extended to two-layer giant constellations with different inclination angles, requiring only periodic extrapolation of the giant constellation's orbit.

[0083] S2. Based on the two-layer mega-constellation network topology, establish a two-layer mega-constellation cluster management mechanism, with the cluster head responsible for the management of satellites within the cluster, to complete the distributed management of the two-layer mega-constellation;

[0084] Please see Figure 2 The two-layer mega-constellation cluster management mechanism employs hierarchical cluster management. Each layer of the mega-constellation is managed as a cluster, with an equal number of satellites within each cluster to minimize management overhead. Each cluster consists of a cluster head and cluster members, with the cluster head managing the members. The cluster head is responsible for issuing routing decisions and collecting information from its members. The issued information includes agent network parameter values, while the collected information includes reinforcement learning experience values ​​and load information for each member. Cluster members are responsible for periodically updating the agent network information issued by the cluster head, executing local online routing decisions, and periodically uploading their experience values ​​to the cluster head.

[0085] Inter-layer inter-satellite links exist between the upper and lower cluster heads and are deployed at the cluster heads of each cluster. Each cluster head exchanges information periodically via these inter-layer inter-satellite links to complete inter-layer cluster management.

[0086] S3. Establish a cluster load prediction mechanism. Based on the two-layer giant constellation cluster management mechanism established in step S2, the current cluster head completes the collection of load information of the remaining clusters and uses the long short-term memory network to predict the load information of each cluster in the two-layer giant constellation network.

[0087] Each cluster head periodically collects load information from its members and exchanges this information with other cluster heads periodically to complete the global load information collection. The Long Short-Term Memory (LSTM) network deployed at the cluster head is responsible for predicting the global load information based on the global load information it has collected.

[0088] The training of the Long Short-Term Memory network is completed using expired previous values ​​and current true values. The pre-training process is first completed on the ground network, and the cluster head only needs to complete the training update periodically.

[0089] S4. Construct a two-layer giant constellation multi-agent deep reinforcement learning load balancing model. Based on the two-layer giant constellation cluster management mechanism established in step S2 and the cluster load prediction mechanism in step S3, complete the mapping between the multi-agent deep reinforcement learning model and the two-layer giant constellation model.

[0090] The multi-agent deep reinforcement learning model consists of a central network and agent networks. For the specific structure of the multi-agent deep reinforcement learning model, please refer to [link / reference needed]. Figure 3 The centralized network consists of a parameter generation network and a hybrid network, enabling agents within a cluster to coordinate based on environmental conditions. The agent network comprises two layers of perceptrons and one layer of gated recurrent units, making online action decisions based on environmental observations. The centralized network is deployed at the cluster head, responsible for periodically collecting intra-cluster and inter-cluster load information, training the agent network, and distributing the trained agent network to each satellite within the cluster. The agent networks are deployed in the remaining cluster members, and the onboard agent networks select the next-hop transmission action based on the observation space determined by the transmission mission.

[0091] The specific steps for constructing an agent network are as follows:

[0092] This involves mapping the agent network's observation space, reward function, action space, and the actual problem. The observation space *o* represents the current satellite node's transmission task and the satellite load information within the cluster; the action space *a* represents the next-hop transmission direction of the transmission task, including two intra-orbit inter-satellite links and two inter-orbit inter-satellite links within the layer, as well as one inter-layer inter-satellite link connecting the cluster head to another layer; the reward function *r* is determined by the distance to the target satellite and the load information of each layer.

[0093] The observation space is:

[0094] o(t)={l t ,l i ,p s ,o s ,p d ,o d ,b u ,b d ,b l ,b r}

[0095] Among them, l t l is the feature number of the current layer of the transmission task. i The layer feature number for the target satellite of the transmission mission; p i For the orbital number of the current mission satellite node, o i b is the satellite number in orbit for the current mission satellite node. i This provides payload information for its four neighboring satellites. The satellite uses an onboard agent to make action decisions based on its observed space, selecting the next-hop transmission satellite.

[0096] The motion space is:

[0097] a(t) = {layer, l, r, u, d}

[0098] There are two types and four choices in the action space. `layer` represents inter-layer transmission, `l` and `r` represent the transmission directions between orbits of the satellite constellation within the same layer, and `u` and `d` represent the transmission directions within the orbit of the satellite constellation within the same layer. When the agent's action decision is inter-layer transmission, the transmission task must first be transmitted to the cluster head of the current cluster. At this point, the transmission target temporarily changes to the cluster head, and then the cluster head uses the inter-layer inter-satellite link with the corresponding cluster head of another satellite constellation to complete the inter-layer transmission.

[0099] The reward function is:

[0100]

[0101] It is determined by two parts of information: latency and load.

[0102] Where, diff is the distance from the current transmission mission to the target satellite, MaxHop is the initial distance of the transmission mission, and B... u and B d Let rp be the network load of each of the two satellite networks, and rp be the penalty factor of the reward function.

[0103] The construction of a centralized network specifically involves:

[0104] The mapping of the input and state space of the centralized network is completed. The input of the centralized network is the reward value r of each agent, and the state space s is the load information of each agent in the cluster and the predicted load information of the other clusters. The Long Short-Term Memory network deployed at the cluster head is used to predict the current load information of the other clusters. After the centralized network completes the forward execution based on the input and state space, it completes the back propagation update of the network based on the global reward value, and periodically transmits the updated agent network to the members in the cluster.

[0105] State space definition:

[0106] s(t)={b1,b2,...,b i C1, C2, ..., C i}

[0107] It consists of two parts. First, the load information of each satellite within the cluster is used by the cluster head to establish a load balancing strategy. Second, the load information of another layer of satellite constellation is used, which determines the performance of transmission tasks using inter-layer transmission.

[0108] In addition to the four fixed inter-satellite links within the current layer, the next hop for the onboard intelligent agent's transmission mission can also utilize inter-layer inter-satellite links. Inter-layer inter-satellite links are only deployed at the cluster head, and the current satellite must first complete the transmission mission to the cluster head.

[0109] S5. Based on the two-layer giant constellation multi-agent deep reinforcement learning load balancing model constructed in step S4, the onboard agent observes the environment to obtain the observation space, selects the next-hop satellite for the transmission task based on the observation space, and completes the action execution process.

[0110] In addition to the four fixed inter-satellite links within the current layer, the next hop for the onboard intelligent agent's transmission mission can also utilize inter-layer inter-satellite links. Inter-layer inter-satellite links are only deployed at the cluster head, and the current satellite must first complete the transmission mission to the cluster head.

[0111] The onboard agent acquires the state space and inputs it into the agent network. After completing the forward computation, the agent network obtains the action to be executed. Based on the action of the agent network, the satellite selects the next hop satellite and transmits the data to the selected target satellite node.

[0112] S6. Based on the distributed cluster management structure of step S2 and the onboard intelligent agent action execution process of step S5, the cluster head periodically completes the collection of satellite experience values ​​within the cluster; based on the cluster load prediction mechanism of step S3, the current cluster head periodically collects the load information of the remaining clusters, inputs it into the long short-term memory network, and obtains the predicted load information of the remaining clusters.

[0113] Cluster heads only collect information within the cluster at specific times to reduce network management overhead.

[0114] The pre-training of the Long Short-Term Memory network is completed by the ground network and then deployed to the cluster head. The cluster head is updated only at specific times to reduce on-board resource consumption.

[0115] S7. Based on the two-layer giant constellation multi-agent deep reinforcement learning load balancing model constructed in step S4, and the cluster information collected by the cluster head in step S6 and the predicted load information of the remaining clusters, the multi-agent deep reinforcement learning model completes centralized training. Based on the completed centralized training, the cluster head distributes the trained onboard agent network to each satellite in the cluster. Each satellite in the cluster updates its agent network accordingly and makes online action decisions based on the updated agent network.

[0116] In another embodiment of the present invention, an intelligent load balancing system for two-layer mega-constellations is provided. This system can be used to implement the above-mentioned intelligent load balancing method for two-layer mega-constellations. Specifically, the intelligent load balancing system for two-layer mega-constellations includes a network module, a management module, a prediction module, a construction module, a selection module, a collection module, and an output module.

[0117] Among them, the network module is used to establish a two-layer mega-constellation network and obtain the topology of the two-layer mega-constellation satellite network;

[0118] The management module is used to establish a two-layer mega-constellation cluster management mechanism based on the two-layer mega-constellation satellite network topology obtained from the network module.

[0119] The prediction module is used to establish a cluster load prediction mechanism based on the two-layer mega-constellation cluster management mechanism established by the management module.

[0120] The construction module is used to complete the mapping between the multi-agent deep reinforcement learning model and the two-layer giant constellation model based on the two-layer giant constellation cluster management mechanism established by the management module and the cluster load prediction mechanism of the prediction module, and to construct the two-layer giant constellation multi-agent deep reinforcement learning load balancing model.

[0121] The selection module is used to construct a two-layer giant constellation multi-agent deep reinforcement learning load balancing model based on the construction module. The onboard agent observes the environment to obtain the observation space, selects the next-hop satellite for the transmission task based on the observation space, and completes the action execution process.

[0122] The collection module is used to collect satellite experience values ​​within the cluster periodically, based on the two-layer giant constellation cluster management mechanism obtained from the management module and the onboard intelligent agent action execution process obtained from the selection module. Based on the cluster load prediction mechanism of the prediction module, the current cluster head periodically collects the load information of the remaining clusters, inputs it into the long short-term memory network, and obtains the predicted load information of the remaining clusters.

[0123] The output module is used for the two-layer giant constellation multi-agent deep reinforcement learning load balancing model built based on the building module, as well as the collection module. The cluster head collects the intra-cluster information and the load information of the remaining clusters after prediction. The multi-agent deep reinforcement learning model completes the central training. Then, the cluster head sends the trained onboard agent network to each satellite in the cluster to complete the agent network update, and completes the online action decision based on the updated agent network.

[0124] In another embodiment of the present invention, a terminal device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or corresponding function. The processor described in this embodiment can be used for the operation of an intelligent load balancing method for a two-layer mega-constellation, including:

[0125] A two-layer mega-constellation network is established, and its topology is obtained. Based on this topology, a two-layer mega-constellation cluster management mechanism is established. A cluster load prediction mechanism is then established based on this mechanism. Following both the cluster management and load prediction mechanisms, a mapping is completed between a multi-agent deep reinforcement learning model and the two-layer mega-constellation model, constructing a two-layer mega-constellation multi-agent deep reinforcement learning load balancing model. Based on this model, onboard agents observe the environment to acquire the observation space, select the next-hop satellite for the transmission task, and complete the corresponding actions. The execution process involves the following steps: Based on a two-layer mega-constellation cluster management mechanism and the onboard agent action execution process, the cluster head periodically collects experience values ​​from satellites within the cluster. Based on a cluster load prediction mechanism, the current cluster head periodically collects load information from other clusters, inputs it into a long short-term memory network, and obtains the predicted load information from the remaining clusters. Based on the two-layer mega-constellation multi-agent deep reinforcement learning load balancing model, the cluster head collects the onboard information, and the predicted load information from the remaining clusters, the multi-agent deep reinforcement learning model completes centralized training. Then, the cluster head distributes the trained onboard agent network to each satellite within the cluster to update the agent network, and makes online action decisions based on the updated agent network.

[0126] In another embodiment of the present invention, a storage medium is also provided, specifically a computer-readable storage medium (memory). This computer-readable storage medium is a memory device in a terminal device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device.

[0127] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the intelligent load balancing method for a two-layer mega-constellation in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor to perform the following steps:

[0128] A two-layer mega-constellation network is established, and its topology is obtained. Based on this topology, a two-layer mega-constellation cluster management mechanism is established. A cluster load prediction mechanism is then established based on this mechanism. Following both the cluster management and load prediction mechanisms, a mapping is completed between a multi-agent deep reinforcement learning model and the two-layer mega-constellation model, constructing a two-layer mega-constellation multi-agent deep reinforcement learning load balancing model. Based on this model, onboard agents observe the environment to acquire the observation space, select the next-hop satellite for the transmission task, and complete the corresponding actions. The execution process involves the following steps: Based on a two-layer mega-constellation cluster management mechanism and the onboard agent action execution process, the cluster head periodically collects experience values ​​from satellites within the cluster. Based on a cluster load prediction mechanism, the current cluster head periodically collects load information from other clusters, inputs it into a long short-term memory network, and obtains the predicted load information from the remaining clusters. Based on the two-layer mega-constellation multi-agent deep reinforcement learning load balancing model, the cluster head collects the onboard information, and the predicted load information from the remaining clusters, the multi-agent deep reinforcement learning model completes centralized training. Then, the cluster head distributes the trained onboard agent network to each satellite within the cluster to update the agent network, and makes online action decisions based on the updated agent network.

[0129] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0130] Please see Figure 5 The paper presents a comparison of transmission success rates for two-layer mega-constellation network transmission tasks with an upper-layer constellation size of 6×6 and a lower-layer constellation size of 12×12. The mega-constellation intelligent load balancing method based on multi-agent deep reinforcement learning achieves a delivery success probability improvement of over 20% compared to single-layer mega-constellation load balancing.

[0131] In summary, the intelligent load balancing method and system for two-layer mega-constellations of this invention has the following characteristics:

[0132] (1) A multi-agent deep reinforcement learning model was adopted to achieve intelligent load balancing of a two-layer giant satellite network. A distributed cluster management strategy was used to achieve intelligent load balancing by coordinating the behavior of agents within the cluster and the transmission tasks between layers.

[0133] (2) A centralized training distributed execution strategy is used, and a centralized network is deployed only on certain satellites to reduce network load, taking into account the limited resources of giant constellation equipment.

[0134] (3) Using a long short-term memory network to complete global load information prediction eliminates the time interference of network load information and provides more effective environmental load information.

[0135] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0136] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0137] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0138] In the embodiments provided by this invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0139] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0141] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0142] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0143] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0144] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0145] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A smart load balancing method for a two-layer mega-constellation, characterized in that, Includes the following steps: S1. Establish a two-layer mega-constellation network and obtain the topology of the two-layer mega-constellation satellite network; S2. Based on the two-layer mega-constellation satellite network topology obtained in step S1, establish a two-layer mega-constellation cluster management mechanism; S3. Establish a cluster load prediction mechanism based on the two-layer giant constellation cluster management mechanism established in step S2; S4. Based on the two-layer giant constellation cluster management mechanism established in step S2 and the cluster load prediction mechanism in step S3, complete the mapping between the multi-agent deep reinforcement learning model and the two-layer giant constellation model, and construct the two-layer giant constellation multi-agent deep reinforcement learning load balancing model. The two-layer giant constellation multi-agent deep reinforcement learning load balancing model includes a central network and an agent network. The centralized network is deployed at the cluster head and is responsible for periodically collecting information within the cluster and between clusters, completing the training of the agent network, and distributing the trained agent network to each satellite within the cluster. The agent network is deployed among the other cluster members. The onboard agent network selects the next hop transmission action based on the observation space determined by the transmission mission. S5. Based on the two-layer giant constellation multi-agent deep reinforcement learning load balancing model constructed in step S4, the onboard agent observes the environment to obtain the observation space, selects the next-hop satellite for the transmission task based on the observation space, and completes the action execution process. S6. Based on the two-layer giant constellation cluster management mechanism obtained in step S2 and the onboard intelligent agent action execution process obtained in step S5, the cluster head periodically completes the collection of satellite experience values ​​within the cluster; based on the cluster load prediction mechanism in step S3, the current cluster head periodically collects the load information of the remaining clusters, inputs it into the long short-term memory network, and obtains the predicted load information of the remaining clusters. S7. Based on the two-layer giant constellation multi-agent deep reinforcement learning load balancing model constructed in step S4, and the cluster information collected by the cluster head in step S6 and the load information of the remaining clusters after prediction, the multi-agent deep reinforcement learning model completes centralized training; then the cluster head sends the trained onboard agent network to each satellite in the cluster to complete the agent network update, and completes online action decision-making based on the updated agent network.

2. The intelligent load balancing method for a two-layer mega-constellation according to claim 1, characterized in that, In step S1, in the dual-layer mega-constellation satellite network topology, each satellite in each layer of the mega-constellation is a topology node, and the intra-layer inter-satellite links and inter-layer inter-satellite links of each layer of the mega-constellation are topology edges.

3. The intelligent load balancing method for a two-layer mega-constellation according to claim 1, characterized in that, In step S2, the two-layer giant constellation cluster management mechanism is specifically as follows: Hierarchical and clustered management is used, with each layer of the giant constellation being managed by a cluster, and the number of satellites in each cluster being equal. Each cluster includes a cluster head and cluster members, with the cluster head managing the members within the cluster. The cluster head is responsible for issuing routing decisions to members within the cluster and collecting information. Cluster members are responsible for executing the cluster head's routing decisions and periodically uploading their experience values. Inter-layer inter-satellite links exist between the upper and lower cluster heads, deployed at the cluster heads of each cluster, and exchange information periodically to complete inter-layer cluster management.

4. The intelligent load balancing method for a two-layer mega-constellation according to claim 1, characterized in that, In step S3, the cluster load prediction mechanism is as follows: The current cluster head completes the collection of load information for the remaining clusters and uses the long short-term memory network deployed at the cluster head to predict the load information of each cluster in the two-layer mega-constellation network.

5. The intelligent load balancing method for a two-layer mega-constellation according to claim 1, characterized in that, The centralized network is specifically as follows: The input to a centralized network is the reward value of each agent. state space For the load information of each cluster and the predicted load information of the remaining clusters, the Long Short-Term Memory network deployed at the cluster head is used to predict the current load information of the remaining clusters. After completing the forward execution based on the input and state space, the centralized network completes the back propagation update based on the global reward value, and periodically transmits the updated agent network to members within the cluster.

6. The intelligent load balancing method for a two-layer mega-constellation according to claim 1, characterized in that, The intelligent agent network is specifically as follows: Observation space This includes the current satellite node's transmission tasks and the satellite load information within the cluster. Action space The next-hop transmission direction for the transmission mission includes two intra-orbit inter-satellite links and two inter-orbit inter-satellite links within the layer, as well as one inter-layer inter-satellite link connecting the cluster head to another layer. reward function The distance to the target satellite and the load information of each layer are determined.

7. The intelligent load balancing method for a two-layer mega-constellation according to claim 6, characterized in that, The observation space is: in, The feature number of the current layer where the transmission task is located. Assign a layer feature number to the target satellite of the transmission mission; The orbital number of the current mission satellite node. The satellite number in orbit for the current mission satellite node. This includes the payload information of its four adjacent satellites; The motion space is: in, For inter-layer transmission, This indicates the direction of transmission between orbits of the satellite constellation at this level. This indicates the transmission direction within the orbit of this satellite constellation. The reward function is: in, The distance from the current transmission mission to the target satellite. The initial distance for the transmission task. and For the respective network loads of the two-layer satellite network, This is the penalty factor for the reward function.

8. The intelligent load balancing method for a two-layer mega-constellation according to claim 1, characterized in that, In step S5, the next hop for the transmission task of the onboard intelligent agent can be selected in addition to the four fixed inter-satellite links within the current layer, or the inter-layer inter-satellite link can be used for transmission. The inter-layer inter-satellite link is deployed at the cluster head, and the current satellite needs to complete the transmission task to the cluster head first.

9. An intelligent load balancing system for a two-layer mega-constellation, characterized in that, include: The network module is used to establish a two-layer mega-constellation network and obtain the topology of the two-layer mega-constellation satellite network; The management module is used to establish a two-layer mega-constellation cluster management mechanism based on the two-layer mega-constellation satellite network topology obtained from the network module. The prediction module is used to establish a cluster load prediction mechanism based on the two-layer mega-constellation cluster management mechanism established by the management module. The construction module is used to complete the mapping between the multi-agent deep reinforcement learning model and the two-layer giant constellation model based on the two-layer giant constellation cluster management mechanism established by the management module and the cluster load prediction mechanism of the prediction module, and to construct the two-layer giant constellation multi-agent deep reinforcement learning load balancing model, which includes a central network and an agent network. The centralized network is deployed at the cluster head and is responsible for periodically collecting information within the cluster and between clusters, completing the training of the agent network, and distributing the trained agent network to each satellite within the cluster. The agent network is deployed among the other cluster members. The onboard agent network selects the next hop transmission action based on the observation space determined by the transmission mission. The selection module is used to construct a two-layer giant constellation multi-agent deep reinforcement learning load balancing model based on the construction module. The onboard agent observes the environment to obtain the observation space, selects the next-hop satellite for the transmission task based on the observation space, and completes the action execution process. The collection module is used to collect satellite experience values ​​within the cluster periodically, based on the two-layer giant constellation cluster management mechanism obtained from the management module and the onboard intelligent agent action execution process obtained from the selection module. Based on the cluster load prediction mechanism of the prediction module, the current cluster head periodically collects the load information of the remaining clusters, inputs it into the long short-term memory network, and obtains the predicted load information of the remaining clusters. The output module is used for the two-layer giant constellation multi-agent deep reinforcement learning load balancing model built based on the building module, as well as the collection module. The cluster head collects the intra-cluster information and the load information of the remaining clusters after prediction. The multi-agent deep reinforcement learning model completes the central training. Then, the cluster head sends the trained onboard agent network to each satellite in the cluster to complete the agent network update, and completes the online action decision based on the updated agent network.

Images