Multi-satellite cooperative task migration method and device for low-orbit satellites and electronic equipment

Abstract

The invention relates to the technical field of edge computing of satellites, and provides a multi-satellite cooperative task migration method and device for low-orbit satellites and electronic equipment. The method comprises the following steps: after obtaining the total task load and load of each user cluster served by each satellite in a low earth orbit satellite network under the current time slot and the load variance of the corresponding satellite, determining a first satellite needing task migration and a corresponding migratable second satellite under the current time slot based on the load of each satellite; calculating a propagation distance and a channel state between the first satellite and each corresponding second satellite to obtain the state of each second satellite in the current time slot; and respectively inputting the states of the second satellites in the current time slot into the trained multi-agent reinforcement learning model to obtain decision-making actions of the second satellites representing the migration strategy in the current time slot. According to the method, load balancing between low earth orbit satellites is realized, and task processing time delay is reduced.

Description

Technical Field

[0001] The present application relates to the field of satellite edge computing technology, and more specifically, to a multi-satellite collaborative task migration method, device and electronic equipment for low-orbit satellites. Background Art

[0002] Low-orbit satellite networks can break through spatial dimensions and geographical limitations and provide communication services with global coverage, which is of great help in providing Internet access and emergency rescue in remote areas. With the widespread application of edge computing technology, edge servers are carried on low-orbit satellites to realize on-board edge computing, which fully utilizes the advantages of edge computing, can quickly respond to users' business processing needs, and improve network performance and user service experience. However, due to the uneven distribution of ground user computing tasks and the limited heterogeneity of satellite resources, direct offloading to the nearest satellite is prone to uneven load, affecting user experience and system energy consumption. Against this background, task migration came into being. However, the high dynamics of low-orbit satellite network topology increases the difficulty of migration decisions, making it difficult to determine a migration plan that ensures service continuity.

[0003] Therefore, studying efficient computing task migration strategies is crucial to improving the edge computing performance of low-orbit satellites. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a multi-satellite collaborative task migration method, device and electronic equipment for low-orbit satellites, which solves the problems of load imbalance among low-orbit satellites and dynamic changes in low-orbit satellite networks, achieves load balancing among low-orbit satellites, and reduces task processing delays.

[0005] In a first aspect, a multi-satellite collaborative task migration method for low-orbit satellites is provided, and the method may include:

[0006] Obtain the total task volume, load and load variance of each user cluster served by each satellite in the low-orbit satellite network in the current time slot; wherein adjacent satellites in the low-orbit satellite network are connected by inter-satellite links, and each satellite corresponds to an edge server; each user cluster is a set of users covered by the satellite beam of the corresponding satellite; each satellite has the same configuration information;

[0007] Based on the loads of the satellites, determining a first satellite that needs to perform task migration and a corresponding second satellite that can be migrated in the current time slot;

[0008] Calculating the propagation distance and channel status between the first satellite and the corresponding second satellites;

[0009] The data set of the total task amount of each user cluster of the first satellite, the load of the first satellite, the load of each corresponding second satellite, the available computing resources of the corresponding second satellite, the propagation distance and the channel state is determined as the state of each second satellite in the current time slot; the available computing resources are obtained from the edge server of the corresponding second satellite;

[0010] The states of each second satellite in the current time slot are respectively input into the trained multi-agent reinforcement learning model to obtain the decision actions of each second satellite representing the migration strategy in the current time slot; the decision actions are determined by the multi-agent reinforcement learning model based on the load variance and total migration delay of the first satellite in the current time slot.

[0011] In a possible implementation, the method for obtaining the load of any satellite in the current time slot includes:

[0012] Obtaining the total computing resources of the satellite from the edge server corresponding to the satellite;

[0013] The load of the satellite in the current time slot is determined based on the ratio of the product of the accumulated sum of the total task amounts of each user cluster served by the satellite and the preset coefficient to the total computing resources.

[0014] In a possible implementation, determining a first satellite that needs to perform task migration and a corresponding second satellite that can be migrated in a current time slot based on the load of each satellite includes:

[0015] For any satellite, if the load of the satellite is greater than a preset overload threshold value, determining that the load state of the satellite is an overload state;

[0016] If the load of the satellite is less than a preset light load threshold value, determining that the load state of the satellite is a light load state;

[0017] Determine a satellite in an overloaded state in the low-orbit satellite network as the first satellite that needs to perform mission migration in the current time slot;

[0018] Determine other satellites in a lightly loaded state in the low-orbit satellite network except the first satellite as primary second satellites that can be migrated;

[0019] For any user cluster of any first satellite, obtaining a time slot difference between a start visible time slot of each primary second satellite and the user cluster and a current time slot t;

[0020] Based on the radius of the earth, the satellite altitude of the primary second satellite, the minimum geocentric angle between the primary second satellite and the user cluster, and the satellite motion speed of the primary second satellite, obtaining the visible time of the primary second satellite and the user cluster;

[0021] Determine the sum of the visible time and the time slot difference as the remaining visible time between the primary second satellite and the user cluster;

[0022] A primary second satellite that meets preset screening conditions is determined as a migratable second satellite corresponding to the first satellite; the preset screening conditions include that the time slot difference is not greater than a preset maximum upcoming visible time interval, and the remaining visible time is not less than a preset minimum remaining visible time.

[0023] In a possible implementation, each satellite has the same configuration information; the configuration information includes transmit power, transmit antenna gain, receive antenna gain, carrier frequency between adjacent satellites, total noise of the satellite system, and ratio of received energy required per bit to noise density;

[0024] Before calculating the total migration delay between the first satellite and each second satellite, the method further includes:

[0025] Based on the configuration information, determining an information transmission rate between adjacent satellites;

[0026] For any user cluster of the first satellite, based on the task amount of the user cluster, the information transmission rate between adjacent satellites and the propagation delay between adjacent satellites, the communication time between adjacent satellites is obtained;

[0027] Using the Dijkstra algorithm, based on the distances between adjacent satellites, the shortest path from the first satellite to any second satellite is determined; the shortest path includes a plurality of shortest subpaths from the first satellite to adjacent satellites within the corresponding second satellite;

[0028] Determine the total transmission time from the first satellite to any second satellite based on the shortest path information and the communication time between adjacent satellites;

[0029] Determine the execution time of the task amount of the user cluster on the second satellite based on the computing resources that can be allocated to any second satellite in the current time slot and the task amount of the user cluster;

[0030] The sum of the total transmission time from the first satellite to any second satellite and the execution time of the task volume of the user cluster on the satellite is determined as the total migration delay between the first satellite and the corresponding second satellite.

[0031] In one possible implementation, calculating the propagation distance and channel status between the first satellite and each second satellite includes:

[0032] Determining a propagation distance between a first satellite and any second satellite based on the shortest path and distances between adjacent satellites;

[0033] A channel state between a first satellite and any second satellite is determined based on an information transmission rate between adjacent satellites and the number of paths between adjacent satellites in a shortest path.

[0034] In one possible implementation, the multi-agent reinforcement learning model includes a policy network and an evaluation network; the training process of the multi-agent reinforcement learning model includes:

[0035] Initialize the policy networks to be trained, the evaluation networks to be trained, and the experience replay pools of all agents deployed on satellites in the historical low-orbit satellite network; one agent is deployed on each satellite;

[0036] Each agent inputs its observed states at different historical time slots into the strategy network to be trained, and obtains decision actions at corresponding historical time slots, wherein the decision actions include a determination decision representing a migration satellite and a selection decision representing a user cluster;

[0037] Input the states at different historical time slots into the evaluation network to be trained to obtain the state values ​​at the corresponding historical time slots;

[0038] Based on the load variance of each satellite that needs to migrate tasks in different historical time slots, the total migration delay between any migratable satellites corresponding to the corresponding satellites, and the decision action, the instant reward of each satellite that needs to migrate tasks in the historical time slot is determined;

[0039] The states, decision actions, state values, immediate rewards and the states of the next historical time slots at different historical time slots are stored in the experience replay pool;

[0040] Using a preset loss function, multiple batches of samples are randomly selected from the experience replay pool, and the parameters of the policy network to be trained and the evaluation network to be trained are optimized respectively to obtain a trained policy network and a trained evaluation network, so as to obtain a trained multi-agent reinforcement learning model; each batch of samples is the state, decision action, state value, immediate reward and state of the next time slot of any historical time slot.

[0041] In one possible implementation, the method further includes:

[0042] Based on the states and immediate rewards at different historical time slots, as well as the states of the next historical time slot of the corresponding historical time slot, the generalized advantage estimate at the corresponding historical time slot is determined and stored in the experience replay pool;

[0043] The preset loss function includes a first loss function of the policy network and a second loss function of the evaluation network; the first loss function is determined based on the state, action and generalized advantage estimation at different historical time slots; the second loss function is determined based on the state at different historical time slots.

[0044] In a second aspect, a multi-satellite collaborative task migration device for low-orbit satellites is provided, and the device may include:

[0045] An acquisition unit is used to acquire the total task volume, load and load variance of each user cluster served by each satellite in the low-orbit satellite network in the current time slot, wherein adjacent satellites in the low-orbit satellite network are connected by inter-satellite links, and each satellite corresponds to an edge server; each user cluster is a set of users covered by the satellite beam of the corresponding satellite;

[0046] A determination unit, configured to determine, based on the loads of the satellites, a first satellite that needs to perform task migration and a corresponding second satellite that can be migrated in a current time slot;

[0047] A calculation unit, used to calculate the propagation distance and channel status between the first satellite and the corresponding second satellites;

[0048] The determination unit is further used to determine the data set of the total task amount of each user cluster of the first satellite, the load of the first satellite, the load of each corresponding second satellite, the available computing resources of the corresponding second satellite, the propagation distance and the channel state as the state of each second satellite in the current time slot; the available computing resources are obtained from the edge server of the corresponding second satellite;

[0049] The input unit is used to input the state of each second satellite in the current time slot into the trained multi-agent reinforcement learning model, so as to obtain the decision action of each second satellite representing the migration strategy in the current time slot; the decision action is determined by the multi-agent reinforcement learning model based on the load variance and total migration delay of the first satellite in the current time slot.

[0050] In a third aspect, an electronic device is provided, the electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0051] Memory, used to store computer programs;

[0052] The processor is used to implement any method step described in the first aspect when executing the program stored in the memory.

[0053] In a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any method step described in the first aspect is implemented.

[0054] The multi-satellite collaborative task migration method for low-orbit satellites provided in the present application obtains the total task volume, load and load variance of each user cluster served by each satellite in the low-orbit satellite network in the current time slot. Adjacent satellites in the low-orbit satellite network are connected by inter-satellite links, and each satellite corresponds to an edge server; each user cluster is a set of users covered by the satellite beam of the corresponding satellite, and based on the load of each satellite, the first satellite that needs to perform task migration and the corresponding second satellite that can be migrated in the current time slot are determined; the propagation distance and channel state between the first satellite and the corresponding second satellites are calculated; the The total task volume of each user cluster, the load of the first satellite, the load of each corresponding second satellite, the available computing resources of the corresponding second satellite, the propagation distance and the data set of the channel state are determined as the state of each second satellite in the current time slot; the available computing resources are obtained from the edge server of the corresponding second satellite; the state of each second satellite in the current time slot is input into the trained multi-agent reinforcement learning model respectively to obtain the decision action of each second satellite to characterize the migration strategy in the current time slot; the decision action is determined by the multi-agent reinforcement learning model based on the load variance and total migration delay of the first satellite in the current time slot. This method solves the problem of load imbalance between low-orbit satellites and dynamic changes in low-orbit satellite networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 A schematic diagram of the structure of a low-orbit satellite network provided in an embodiment of the present application;

[0057] Figure 2 A schematic diagram of a flow chart of a multi-satellite collaborative task migration method for low-orbit satellites provided in an embodiment of the present application;

[0058] Figure 3 A schematic diagram of the structure of a multi-satellite collaborative mission migration device for low-orbit satellites provided in an embodiment of the present application;

[0059] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application. Unless otherwise defined, the technical terms or scientific terms used in this application should be the common meanings understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in this application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "including" and other similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connected", "coupled" or "connected" and other similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0061] The multi-satellite collaborative task migration method for low-orbit satellites provided in the embodiment of the present application can be applied to Figure 1 In the low-orbit satellite network shown in Figure 1 As shown, the low-orbit satellite network may include: S moving low-orbit satellites (hereinafter referred to as "satellites"), where S is an integer not less than 1.

[0062] Intersatellite links are constructed between adjacent satellites for mutual communication. Intersatellite links include two adjacent satellites in the same orbit and two adjacent satellites in adjacent orbits. Each satellite carries an edge server that can provide computing services to users on the ground. All users served by each satellite are divided into P user clusters, and each user cluster is a set of users covered by the satellite beam of the corresponding satellite. P is an integer not less than 1. For example, the set of user clusters corresponding to satellite s is expressed as A user The amount of tasks generated in time slot t is w u (t).

[0063] The configuration information may include the transmission power P tran , Transmitting antenna gain G tran , receiving antenna gain G rece and the carrier frequency f between adjacent satellites c , the total noise of the satellite system U s The ratio of the received energy required per bit to the noise density E b / N0, etc. Divide the continuous time into discrete time slots, expressed as Each satellite in the low-orbit satellite network may have the same configuration information or different configuration information, which is not limited in this application. This application is described in detail below using the same configuration information as an example.

[0064] In the low-orbit satellite network, when some low-orbit satellites are overloaded, the task migration strategy of these overloaded low-orbit satellites is determined by a pre-trained multi-agent reinforcement learning model. The purpose is to minimize the weighted sum of the load balance between satellites and the total task migration delay, so as to solve the problems of load imbalance between low-orbit satellites and dynamic changes in the low-orbit satellite network.

[0065] This application models the problem of load imbalance between low-orbit satellites and dynamic changes in low-orbit satellite networks as an integer programming problem. In order to solve the optimization problem, the multi-satellite collaborative task migration scenario is modeled as a multi-agent Markov decision process, and a multi-agent task processing algorithm based on centralized training and distributed execution architecture MAPPO is used to solve it.

[0066] The optimization problem is modeled as an integer programming problem. Under the conditions of meeting the visibility time constraint and load state constraint, a migration strategy is obtained that minimizes the weighted sum of inter-satellite load balance and total task migration delay.

[0067] The multi-agent Markov decision process model is established according to the optimization problem, where the state includes the task volume of the user cluster, the load state of the overloaded satellite and all its migratable satellite nodes that meet the constraints, the available resources, and the propagation distance and channel state between the migratable satellite nodes. The action is the combination of the migratable satellite nodes and user clusters that meet the constraints corresponding to each low-orbit satellite. The reward is set to the negative of the optimization target.

[0068] Among them, the multi-agent Markov decision process can be summarized as a tuple, namely Wherein, N is the number of intelligent agents. In this application, one intelligent agent is deployed on each satellite, so N=S (n=s).

[0069] is the state space Among them, O n is the state space of the nth agent, and represents the state observed by agent n in time slot t. Represents the state set of all agents in time slot t.

[0070] Define the global state as G t , since all agents are homogeneous, the global state can be expressed as a vector consisting of the states observed by all agents in time slot t, that is

[0071] is the action space, Among them, A n is the action space of the nth agent, and Represents the action of the agent in time slot t. represents the set of actions of all agents in time slot t.

[0072] is the state transition probability, satisfying

[0073] is the reward function, Indicates that the multi-agent is in state O t Next, take action A t Instant feedback from the system environment when the system is in operation.

[0074] The multi-agent reinforcement learning MAPPO algorithm is a policy-based method. Each agent consists of a policy network, an evaluation network, and an experience buffer pool. For each agent, the evaluation network is used to fit the global state value function to evaluate the state of the environment. The strategy outputs actions based on the local observations of each agent to obtain decisions. The experience buffer pool is used to store the "experience" of the interaction between the agent and the environment, including state, action, reward, etc., and use these experiences to update the parameters of the network.

[0075] In this application, all agents share the parameters of the policy network and the evaluation network, and store the experience in the same shared experience pool. MAPPO adopts a centralized training-distributed execution training framework, that is, all observations, actions and global information of the system can be obtained during the training phase, while during the execution phase, the agent can only make decisions based on the local information it observes.

[0076] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0077] Figure 2 A flowchart of a multi-satellite collaborative task migration method for low-orbit satellites provided in an embodiment of the present application. Figure 2 As shown, the method may include:

[0078] Step S210: Obtain the total task volume, load and load variance of each user cluster served by each satellite in the low-orbit satellite network in the current time slot.

[0079] (1) The total task load of each user cluster served by any satellite in the current time slot t is the sum of the task loads of all users in the corresponding user cluster. For example, Total amount of tasks

[0080] (2) Considering the impact of the total number of tasks and computing power of the server on the load, the load is defined as the ratio of the total number of tasks on the corresponding server of the satellite in the current time slot to its computing power. Therefore, the method for obtaining the load of any satellite in the current time slot includes: obtaining the total computing resources of the satellite from the edge server corresponding to the satellite; based on the product of the cumulative sum of the total task volume of each user cluster served by the satellite and the preset coefficient, and the ratio of the total computing resources, the load L of the satellite in the current time slot t is determined. s (t).

[0081] For example, the load L of satellite s at the current time slot t s (t) can be expressed as:

[0082] Among them, F s is the total computing resource of satellite s, ρ is a coefficient, and p is any user cluster among P user clusters.

[0083] (3) The load variance of any satellite at the current time slot t can be expressed as: Wherein, Ω(t) is the load variance at the current time slot t.

[0084] Step S220: Based on the load of each satellite, determine the first satellite that needs to perform task migration and the corresponding second satellite that can be migrated in the current time slot.

[0085] The satellite load state is divided into three types: light load state, moderate state and overload state. The preset light load threshold value L low and the preset overload threshold value L high They are light load threshold and overload threshold respectively. Among them, a server in light load state can receive computing tasks migrated from other satellites; a server in overload state needs to migrate computing tasks to other satellites to complete; if the server is in a moderate state, it will not receive migration tasks from other satellites, nor request to migrate its own computing tasks.

[0086] In a specific implementation, for any satellite, if the load L of the satellite at the current time slot t is s (t) is greater than the preset overload threshold value, the load state of the satellite is determined to be an overload state; if the load L of the satellite at the current time slot t s(t) is not greater than the preset overload threshold value and not less than the preset light load threshold value, then the load state of the satellite is determined to be moderate; if the load L of the satellite at the current time slot t s (t) is less than the preset light-load threshold value, the load state of the satellite is determined to be a light-load state.

[0087] A satellite in an overloaded state in a low-orbit satellite network is determined as a first satellite that needs to perform mission migration in a current time slot t; and other satellites in a lightly loaded state in the low-orbit satellite network except the first satellite are determined as primary second satellites that can be migrated.

[0088] It can be understood that the first satellite refers to a type of satellite that needs to perform task migration in the current time slot t, and the primary second satellite refers to a type of satellite that is in a light-load state in the current time slot t.

[0089] Furthermore, since the range of the user cluster is small, the visible time between the user cluster center and the corresponding satellite is regarded as the visible time between the entire user cluster and the satellite. For any first satellite, the corresponding migratable satellite should be a satellite that is currently visible or will be visible to any user cluster of the first satellite. Therefore, it is necessary to select satellites that are currently visible or will be visible to each user cluster of the first satellite from the primary second satellites. Specifically:

[0090] For any user cluster of any first satellite, obtain a time slot difference between a start visible time slot of each primary second satellite and the user cluster of the first satellite and a current time slot t, that is, a time slot difference between each primary second satellite and the user cluster;

[0091] Based on the radius of the earth, the satellite altitude of the primary second satellite, the minimum geocentric angle between the primary second satellite and the user cluster, and the satellite motion speed of the primary second satellite, obtaining the visible time between the primary second satellite and the user cluster;

[0092] Afterwards, determining the sum of the visible time and the time slot difference as the remaining visible time between the primary second satellite and the user cluster;

[0093] A primary second satellite that meets preset screening conditions is determined as a migratable second satellite corresponding to the first satellite; wherein the preset screening conditions include that the time slot difference is not greater than a preset maximum upcoming visible time interval, and the remaining visible time is not less than a preset minimum remaining visible time.

[0094] In one example, the first satellite is satellite s, and the user cluster is the user cluster of satellite s. Take the satellite s' as an example for the primary second satellite, specifically:

[0095] Get real-time data of each primary second satellite and user cluster The start visible time slot and the start of the visible time slot The time slot difference ΔT with the current time slot t, that is

[0096] Satellite s′ and user cluster The calculation formula of the visible time can be expressed as:

[0097] Among them, R e is the radius of the earth and h is the altitude of the satellite. is the satellite s and the user cluster The minimum geocentric angle between the satellite s and the user cluster Minimum elevation angle between Calculated, Among them, v s′ is the satellite motion speed.

[0098] Therefore, satellite s′ and user cluster The calculation formula for the remaining visible time can be expressed as:

[0099] Furthermore, in order to ensure the continuity of user services, considering that the migrating satellite s′ will directly transmit the results back to the user after completing the computing task, the migratable satellite s′ corresponding to satellite s should meet the following two conditions:

[0100] 1) Migrate satellite s′, which should be a user cluster Satellites that are currently visible or will soon be visible, so: Where ΔT max is the maximum upcoming visible time interval.

[0101] 2) If satellite s′ and user cluster If the remaining visible time is too short, the result cannot be successfully transmitted back and task migration needs to be continued, which greatly increases the end-to-end latency of the user. Therefore: in is the minimum remaining visible time.

[0102] Thus, we can obtain the relocatable satellites s' corresponding to satellite s and the number of corresponding relocatable satellites s'. These relocatable satellites and satellite s form a set

[0103] It should be noted that this application uses binary variables Represents the decision to migrate satellites, a binary variable Represents the user cluster decision. If satellite s clusters user All tasks of are migrated to satellite s′ for execution, which can be expressed as: like This indicates that satellite s does not migrate at this time.

[0104] Step S230: Calculate the total migration delay, propagation distance and channel status between the first satellite and the corresponding second satellites.

[0105] (1) The total migration delay between the first satellite s and any second satellite s′:

[0106] Based on the configuration information, the information transmission rate between adjacent satellites is determined; wherein the information transmission rate R between adjacent satellites i and j is i,j The calculation formula of (t) can be expressed as:

[0107]

[0108] Among them, P tran is the satellite’s transmission power, G tran is the transmitting antenna gain, G rece is the receiving antenna gain, c is the speed of light, d i,j (t) is the distance between satellite i and satellite j at time slot t, f c is the carrier frequency for communication between satellite i and satellite j, k is the Boltzmann constant, U s is the total noise temperature of the system, E b / N0 represents the ratio of the received energy required per bit to the noise density, and A is a preset value.

[0109] For any user cluster of the first satellite, based on the task volume of the user cluster, the information transmission rate between adjacent satellites and the propagation delay between adjacent satellites, the communication time between adjacent satellites is obtained; the communication time is the sum of the information transmission time and the propagation delay.

[0110] Among them, the user cluster of satellite s in the current time slot t is The migration task, the communication time between adjacent satellites i and j The calculation formula can be expressed as:

[0111]

[0112] The Dijkstra algorithm is used to determine the shortest path from the first satellite to any second satellite based on the distances between adjacent satellites; the shortest path may include multiple shortest subpaths from the first satellite to the adjacent satellites within the corresponding second satellite. s,s′ It can be expressed as Among them, s1,…,i,j,…,s E is the E satellites passed by the shortest path from satellite s to satellite s′.

[0113] Based on the shortest path and the communication time between adjacent satellites, the total transmission time from the first satellite to any second satellite is determined; wherein the calculation formula of the total transmission time can be expressed as:

[0114]

[0115] Based on the computing resources that can be allocated to any second satellite in the current time slot t and the task volume of the user cluster, the execution time of the task volume of the user cluster on the second satellite is determined; wherein, in the current time slot t, the user cluster The execution time of the migration task on the second satellite s′ is:

[0116] The total migration delay between the first satellite s and the corresponding second satellite s′ is determined as the sum of the total transmission time between the first satellite s and any second satellite s′ and the execution time of the task of the user cluster on the second satellite s′.

[0117] (2) The propagation distance between the first satellite s and any second satellite s′:

[0118] Based on the shortest path and the distance d between adjacent satellites i,j (t), determine the propagation distance between the first satellite s and any second satellite s′. The calculation formula of the propagation distance can be expressed as:

[0119] is the propagation distance between the current satellite s and its migrating satellite s′.

[0120] (3) Channel status between the first satellite s and any second satellite s′:

[0121] Determine a channel state between a first satellite s and any second satellite s′ based on an information transmission rate between adjacent satellites and the number of paths between adjacent satellites in a shortest path;

[0122] The channel state between the access satellite and the migratable satellite node can be formed by the average information propagation rate, that is, the average information propagation rate between the satellite s and its migrating satellite s′ in the current time slot The calculation formula can be expressed as:

[0123] Step S240: Determine the state of each first satellite in the current time slot as a data set including the total task amount of each user cluster of the first satellite, the load of the first satellite, the load of each corresponding second satellite, the available computing resources of the corresponding second satellite, the propagation distance and the channel state.

[0124] The available computing resources are obtained from the edge server of the corresponding second satellite.

[0125] Since each satellite is deployed with an agent, the state observed by each agent is defined as: Specifically:

[0126] represents the total task volume of all user clusters of the first satellite s in the current time slot t.

[0127] L s (t) represents the load of the first satellite at the current time slot t.

[0128] use Represents the number of migratable satellites of the first satellite in the current time slot t.

[0129] Represents the load status of the migratable satellite of the first satellite s in the current time slot t.

[0130] Represents the available resources of the migratable satellite of the first satellite s in the current time slot t.

[0131] Represents the propagation distance between the first access satellite s and the migratable satellite.

[0132] The channel status between the first access satellite and the migratable satellite is represented by the average information propagation rate.

[0133] Furthermore, the action space of each agent is defined as: Therefore, the action space of agent n is The size is

[0134] Step S250: Input the states of each second satellite in the current time slot into the trained multi-agent reinforcement learning model to obtain the decision action representing the migration strategy of the corresponding second satellite in the current time slot.

[0135] Among them, the decision action is determined by the multi-agent reinforcement learning model based on the load variance and total migration delay of the first satellite in the current time slot.

[0136] In some embodiments, the multi-agent reinforcement learning model includes a policy network and an evaluation network; the training process of the multi-agent reinforcement learning model may include:

[0137] Initialize the policy networks to be trained, the evaluation networks to be trained, and the experience replay pools of all agents deployed on satellites in the historical low-orbit satellite network;

[0138] Each agent inputs its observed states at different historical time slots into the strategy network to be trained, and obtains the decision actions at the corresponding historical time slots. The decision actions may include the determination decision representing the migration satellite and the selection decision representing the user cluster.

[0139] Input the states at different historical time slots into the evaluation network to be trained to obtain the state values ​​at the corresponding historical time slots;

[0140] Based on the load variance of each satellite that needs to migrate tasks in different historical time slots, the total migration delay between any migratable satellites corresponding to the corresponding satellites, and the decision action, the instant reward of each satellite that needs to migrate tasks in the historical time slot is determined;

[0141] The states, decision actions, state values, immediate rewards and the states of the next historical time slots at different historical time slots are stored in the experience replay pool;

[0142] Using a preset loss function, multiple batches of samples are randomly selected from the experience replay pool, and the parameters of the policy network to be trained and the evaluation network to be trained are optimized respectively to obtain a trained policy network and a trained evaluation network, so as to obtain a trained multi-agent reinforcement learning model; each batch of samples is the state, decision action, state value, immediate reward and state of the next time slot of any historical time slot.

[0143] Among them, the immediate reward represents the task migration problem of multi-satellite collaboration, which can be expressed as:

[0144]

[0145]

[0146] Among them, ξ1 and ξ2 are hyperparameters that balance the load balancing degree and the total task migration delay.

[0147] Constraints C1-C3 restrict: for each satellite, only one user cluster's tasks can be migrated, and they can only be migrated to one satellite for execution;

[0148] Constraints C4-C5 restrict: the visible time between the migrating node and the user, where C4 requires that the migrating node is a satellite that is currently visible or will be visible to the user, and C5 requires that the visible time between the migrating node and the user cannot be too short;

[0149] Constraint C6 restricts that the migration node must be in a light-load state.

[0150] In short, the goal is to obtain a migration strategy that minimizes load balancing and total task migration delay under the above constraints.

[0151] Furthermore, based on the states and immediate rewards at different historical time slots, and the states of the next historical time slot of the corresponding historical time slot, a generalized advantage estimate at the corresponding historical time slot is determined and stored in the experience replay pool;

[0152] Among them, the calculation method of generalized advantage estimation can be expressed as:

[0153]

[0154] Where γ is the discount factor, λ is the generalized advantage estimation parameter,

[0155] The preset loss function may include a first loss function of the policy network and a second loss function of the evaluation network;

[0156] The first loss function is determined based on the state, action and generalized advantage estimation at different historical time slots;

[0157] The first loss function can be expressed as:

[0158]

[0159] in, represents the old strategy before parameter update, ψ is the strategy entropy, and σ is the hyperparameter of entropy.

[0160] The second loss function is determined based on the states at different historical time slots.

[0161] The second loss function can be expressed as: Among them, V φ is the fitted state value function for evaluating the network output.

[0162] Corresponding to the above method, the embodiment of the present application also provides a multi-satellite collaborative task migration device for low-orbit satellites, such as Figure 3 As shown, the device comprises:

[0163] The acquisition unit 310 is used to acquire the total task volume, load and load variance of each user cluster served by each satellite in the low-orbit satellite network in the current time slot, wherein adjacent satellites in the low-orbit satellite network are connected by inter-satellite links, and each satellite corresponds to an edge server; each user cluster is a set of users covered by the satellite beam of the corresponding satellite;

[0164] A determination unit 320 is used to determine a first satellite that needs to perform task migration and a corresponding second satellite that can be migrated in a current time slot based on the load of each satellite;

[0165] A calculation unit 330, configured to calculate a propagation distance and a channel state between the first satellite and corresponding second satellites;

[0166] The determination unit 320 is further configured to determine the data set of the total task amount of each user cluster of the first satellite, the load of the first satellite, the load of each corresponding second satellite, the available computing resources of the corresponding second satellite, the propagation distance and the channel state as the state of each second satellite in the current time slot; the available computing resources are obtained from the edge server of the corresponding second satellite;

[0167] The input unit 340 is used to input the state of each second satellite in the current time slot into the trained multi-agent reinforcement learning model, and obtain the decision action of each second satellite characterizing the migration strategy in the current time slot; the decision action is determined by the multi-agent reinforcement learning model based on the load variance and total migration delay of the first satellite in the current time slot.

[0168] The functions of each functional unit of the multi-satellite collaborative mission migration device for low-orbit satellites provided in the above-mentioned embodiments of the present application can be achieved through the above-mentioned method steps. Therefore, the specific working process and beneficial effects of each unit in the multi-satellite collaborative mission migration device for low-orbit satellites provided in the embodiments of the present application will not be repeated here.

[0169] The present application also provides an electronic device, such as Figure 4 As shown, it includes a processor 410 , a communication interface 420 , a memory 430 and a communication bus 440 , wherein the processor 410 , the communication interface 420 , and the memory 430 communicate with each other via the communication bus 440 .

[0170] Memory 430, for storing computer programs;

[0171] The processor 410 is used to execute the program stored in the memory 430 to implement the following steps:

[0172] Obtain the total task volume, load and load variance of each user cluster served by each satellite in the low-orbit satellite network in the current time slot; wherein adjacent satellites in the low-orbit satellite network are connected by inter-satellite links, and each satellite corresponds to an edge server; each user cluster is a set of users covered by the satellite beam of the corresponding satellite;

[0173] Based on the loads of the satellites, determining a first satellite that needs to perform task migration and a corresponding second satellite that can be migrated in the current time slot;

[0174] Calculating the propagation distance and channel status between the first satellite and the corresponding second satellites;

[0175] The data set of the total task amount of each user cluster of the first satellite, the load of the first satellite, the load of each corresponding second satellite, the available computing resources of the corresponding second satellite, the propagation distance and the channel state is determined as the state of each second satellite in the current time slot; the available computing resources are obtained from the edge server of the corresponding second satellite;

[0176] The states of each second satellite in the current time slot are respectively input into the trained multi-agent reinforcement learning model to obtain the decision actions of each second satellite representing the migration strategy in the current time slot; the decision actions are determined by the multi-agent reinforcement learning model based on the load variance and total migration delay of the first satellite in the current time slot.

[0177] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0178] The communication interface is used for communication between the above electronic device and other devices.

[0179] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0180] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0181] The implementation methods and beneficial effects of the components of the electronic device in the above embodiments to solve the problems can be seen in Figure 2The various steps in the illustrated embodiment are implemented, therefore, the specific working process and beneficial effects of the electronic device provided by the embodiment of the present application are not repeated here.

[0182] In another embodiment provided in the present application, a computer-readable storage medium is also provided, which stores instructions. When the computer-readable storage medium is executed on a computer, the computer executes the multi-satellite collaborative mission migration method for low-orbit satellites described in any of the above embodiments.

[0183] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute the multi-satellite collaborative mission migration method for low-orbit satellites described in any of the above embodiments.

[0184] Those skilled in the art will appreciate that the embodiments in the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt a complete hardware embodiment, a complete software embodiment, or a form of an embodiment combining software and hardware. Moreover, the present application may adopt a form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0185] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0186] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0187] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0188] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.

[0189] Obviously, those skilled in the art can make various changes and modifications to the embodiments in the present application without departing from the spirit and scope of the embodiments in the present application. Thus, if these modifications and variations of the embodiments in the present application fall within the scope of the claims and their equivalents in the embodiments of the present application, the embodiments of the present application are also intended to include these modifications and variations.

Claims

1. A multi-satellite collaborative task migration method for low-orbit satellites, characterized in that: The method comprises: Obtain the total task volume, load and load variance of each user cluster served by each satellite in the low-orbit satellite network in the current time slot; wherein adjacent satellites in the low-orbit satellite network are connected by inter-satellite links, and each satellite corresponds to an edge server; each user cluster is a set of users covered by the satellite beam of the corresponding satellite; Based on the loads of the satellites, determining a first satellite that needs to perform task migration and a corresponding second satellite that can be migrated in the current time slot; Calculating the propagation distance and channel status between the first satellite and the corresponding second satellites; The data set of the total task amount of each user cluster of the first satellite, the load of the first satellite, the load of each corresponding second satellite, the available computing resources of the corresponding second satellite, the propagation distance and the channel state is determined as the state of each second satellite in the current time slot; the available computing resources are obtained from the edge server of the corresponding second satellite; The states of each second satellite in the current time slot are respectively input into the trained multi-agent reinforcement learning model to obtain the decision actions of each second satellite representing the migration strategy in the current time slot; the decision actions are determined by the multi-agent reinforcement learning model based on the load variance and total migration delay of the first satellite in the current time slot.

2. The method according to claim 1, characterized in that The method of obtaining the load of any satellite in the current time slot includes: Obtaining the total computing resources of the satellite from the edge server corresponding to the satellite; The load of the satellite in the current time slot is determined based on the ratio of the product of the accumulated sum of the total task amounts of each user cluster served by the satellite and the preset coefficient to the total computing resources.

3. The method according to claim 1, characterized in that Determining a first satellite that needs to perform task migration and a corresponding second satellite that can be migrated in a current time slot based on the load of each satellite includes: For any satellite, if the load of the satellite is greater than a preset overload threshold value, determining that the load state of the satellite is an overload state; If the load of the satellite is less than a preset light load threshold value, determining that the load state of the satellite is a light load state; Determine a satellite in an overloaded state in the low-orbit satellite network as the first satellite that needs to perform mission migration in the current time slot; Determine other satellites in a lightly loaded state in the low-orbit satellite network except the first satellite as primary second satellites that can be migrated; For any user cluster of any first satellite, obtaining a time slot difference between a start visible time slot of each primary second satellite and the user cluster and a current time slot t; Based on the radius of the earth, the satellite altitude of the primary second satellite, the minimum geocentric angle between the primary second satellite and the user cluster, and the satellite motion speed of the primary second satellite, obtaining the visible time of the primary second satellite and the user cluster; Determine the sum of the visible time and the time slot difference as the remaining visible time between the primary second satellite and the user cluster; A primary second satellite that meets preset screening conditions is determined as a migratable second satellite corresponding to the first satellite; the preset screening conditions include that the time slot difference is not greater than a preset maximum upcoming visible time interval, and the remaining visible time is not less than a preset minimum remaining visible time.

4. The method according to claim 1, characterized in that Each satellite has the same configuration information; the configuration information includes transmit power, transmit antenna gain, receive antenna gain, carrier frequency between adjacent satellites, total noise of the satellite system, and ratio of received energy required per bit to noise density; Before calculating the total migration delay between the first satellite and each second satellite, the method further includes: Based on the configuration information, determining an information transmission rate between adjacent satellites; For any user cluster of the first satellite, based on the task amount of the user cluster, the information transmission rate between adjacent satellites and the propagation delay between adjacent satellites, the communication time between adjacent satellites is obtained; Using the Dijkstra algorithm, based on the distances between adjacent satellites, the shortest path from the first satellite to any second satellite is determined; the shortest path includes a plurality of shortest subpaths from the first satellite to adjacent satellites within the corresponding second satellite; Determine the total transmission time from the first satellite to any second satellite based on the shortest path information and the communication time between adjacent satellites; Determine the execution time of the task amount of the user cluster on the second satellite based on the computing resources that can be allocated to any second satellite in the current time slot and the task amount of the user cluster; The sum of the total transmission time from the first satellite to any second satellite and the execution time of the task volume of the user cluster on the satellite is determined as the total migration delay between the first satellite and the corresponding second satellite.

5. The method according to claim 4, characterized in that Calculating the propagation distance and channel status between the first satellite and each second satellite, including: Determining a propagation distance between a first satellite and any second satellite based on the shortest path and distances between adjacent satellites; Based on the information transmission rate between adjacent satellites and the number of paths between adjacent satellites in the shortest path, a channel state between a first satellite and any second satellite is determined.

6. The method according to claim 1, characterized in that The multi-agent reinforcement learning model includes a policy network and an evaluation network; The training process of the multi-agent reinforcement learning model includes: Initialize the policy networks to be trained, the evaluation networks to be trained, and the experience replay pools of all agents deployed on satellites in the historical low-orbit satellite network; one agent is deployed on each satellite; Each agent inputs its observed states at different historical time slots into the strategy network to be trained, and obtains decision actions at corresponding historical time slots, wherein the decision actions include a determination decision representing a migration satellite and a selection decision representing a user cluster; Input the states at different historical time slots into the evaluation network to be trained to obtain the state values ​​at the corresponding historical time slots; Based on the load variance of each satellite that needs to migrate tasks in different historical time slots, the total migration delay between any migratable satellites corresponding to the corresponding satellites, and the decision action, the instant reward of each satellite that needs to migrate tasks in the historical time slot is determined; The states, decision actions, state values, immediate rewards and the states of the next historical time slots at different historical time slots are stored in the experience replay pool; Using a preset loss function, multiple batches of samples are randomly selected from the experience replay pool, and the parameters of the policy network to be trained and the evaluation network to be trained are optimized respectively to obtain a trained policy network and a trained evaluation network, so as to obtain a trained multi-agent reinforcement learning model; each batch of samples is the state, decision action, state value, immediate reward and state of the next time slot of any historical time slot.

7. The method according to claim 1, characterized in that The method further comprises: Based on the states and immediate rewards at different historical time slots, as well as the states of the next historical time slot of the corresponding historical time slot, the generalized advantage estimate at the corresponding historical time slot is determined and stored in the experience replay pool; The preset loss function includes a first loss function of the policy network and a second loss function of the evaluation network; the first loss function is determined based on the state, action and generalized advantage estimation at different historical time slots; the second loss function is determined based on the state at different historical time slots.

8. A multi-satellite collaborative mission migration device for low-orbit satellites, characterized in that: The device comprises: An acquisition unit is used to acquire the total task volume, load and load variance of each user cluster served by each satellite in the low-orbit satellite network in the current time slot, wherein adjacent satellites in the low-orbit satellite network are connected by inter-satellite links, and each satellite corresponds to an edge server; each user cluster is a set of users covered by the satellite beam of the corresponding satellite; each satellite has the same configuration information; A determination unit, configured to determine, based on the loads of the satellites, a first satellite that needs to perform task migration and a corresponding second satellite that can be migrated in a current time slot; A calculation unit, used to calculate the propagation distance and channel status between the first satellite and the corresponding second satellites; The determination unit is further used to determine the data set of the total task amount of each user cluster of the first satellite, the load of the first satellite, the load of each corresponding second satellite, the available computing resources of the corresponding second satellite, the propagation distance and the channel state as the state of each second satellite in the current time slot; the available computing resources are obtained from the edge server of the corresponding second satellite; The input unit is used to input the state of each second satellite in the current time slot into the trained multi-agent reinforcement learning model, so as to obtain the decision action of each second satellite representing the migration strategy in the current time slot; the decision action is determined by the multi-agent reinforcement learning model based on the load variance and total migration delay of the first satellite in the current time slot.

9. An electronic device, characterized in that: The electronic device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-7 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images