Multi-satellite autonomous cooperative scheduling method based on distributed multi-agent reinforcement learning

Abstract

The huge progress of the autonomous technology promotes the rapid development of imaging satellites, and the satellites have advanced functions such as on-satellite autonomous decision making, inter-satellite link communication and multi-satellite collaborative planning in the future. High timeliness, autonomy and intelligentization become an important trend of development of a satellite task planning system. The invention provides a multi-satellite autonomous cooperative scheduling algorithm based on distributed multi-agent reinforcement learning aiming at the characteristics of limited on-satellite computing power, simple inter-satellite cooperative mechanism, unstable communication link and the like in on-satellite autonomous planning, and supports each satellite to carry out distributed autonomous decision based on local communication information. According to the method, a good task planning effect can be achieved while low communication consumption is kept, and the method has quick response capability and good system survivability.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-satellite autonomous collaborative scheduling of distributed multi-agent reinforcement learning, and enables satellites to make distributed autonomous decisions based on local communication information. Background Art

[0002] With the rapid improvement of satellite technology, future satellites will have advanced functions such as on-board automatic processing, on-board autonomous planning, inter-satellite link communication, and data broadcast distribution. High timeliness, autonomy, and intelligence are becoming important trends in the future development of satellite mission planning systems. However, the current imaging satellite mission planning is usually carried out on a daily basis, using the "ground decision-making, on-board execution" model, which is not flexible enough and has poor timeliness. Since the satellite-ground link cannot be constantly interconnected, the ground cannot grasp the real-time resource status on the satellite, and thus cannot give full play to the satellite's observation efficiency and is difficult to respond quickly to dynamically changing environments. This traditional model has become increasingly difficult to meet the needs of rapid response to future missions. There is an urgent need to explore new models for imaging satellite mission planning to more effectively respond to the complex challenges faced in future aerospace development.

[0003] The design of on-board algorithms is usually based on a multi-satellite collaborative architecture. In current research, collaborative architectures can be classified into two categories: centralized architecture and distributed architecture. The centralized architecture usually relies on a central node that can obtain information from all sub-nodes and make comprehensive plans based on it. Its advantages are simple system operation and accurate solutions, but it places high demands on the computing power of the central node and the inter-satellite communication environment. Once the central node fails, the performance of the entire system will be greatly reduced. In order to cope with more complex communication environments and improve system anti-destruction capabilities, distributed scheduling methods have been favored. In a distributed architecture, each sub-node undertakes certain computing work, rather than just being an executor of instructions. That is, each node in the system has a certain degree of autonomy, and each node obtains consistent planning results through continuous negotiation, such as the contract network protocol (CNP) and the consensus-based bundle algorithm (CBBA). However, such algorithms are often accompanied by high communication overhead. Considering the risks of inter-satellite link connectivity failure, external interference, communication delay, data loss, etc., when the communication quality is poor, the on-board system requires the algorithm to rely less on inter-satellite communication. Multi-agent deep reinforcement learning (MADRL) algorithm shows great potential in this scenario.

[0004] MADRL allows multiple agents to learn interactively in the environment, and each agent maximizes its cumulative reward by interacting with the environment and other agents. Centralized training & decentralized execution (CTDE) is an algorithm framework commonly used by MADRL. It equips each agent with a decision network, introduces a central control network in the training phase, and integrates all agent networks for joint training with the assistance of global state information, thereby avoiding the non-stationary problem of the environment and the problem of too large state-action space, and ensuring the training effect of each agent's decision network; in the execution phase, each agent only needs to make independent decisions based on the local information it observes, achieving rapid task response. Lowe et al. proposed the multi-agent deep deterministic policy gradient algorithm (MADDPG), which is an extension of the classic deep deterministic policy gradient (DDPG) algorithm in a multi-agent environment. Foerster et al. proposed the counterfactual multi-agent policy gradients (COMA) algorithm, which aims to solve the credit allocation problem in a multi-agent environment. By keeping the actions of other agents unchanged, the expected return of the agent is compared with the counterfactual baseline to estimate the contribution of a single agent's action to the overall task. Sunehag et al. proposed the value decomposition network (VDN), which is a method of decomposing the joint action value function of multiple agents into separate action value functions of each agent. The core idea is to maximize the joint action value function while ensuring that the action value function of each agent is maximized. A large number of improvement studies have been carried out based on this algorithm, such as the QMIX algorithm and the QTRAN algorithm, which effectively improve the versatility and solution effect of the algorithm.

[0005] At present, the research on applying MADRL to imaging satellite mission planning is still preliminary. In response to this problem, Wang Haijiao proposed a satellite distributed online scheduling algorithm based on MADDGP. ​​Its application background is defined as observation requirements are submitted to the system one by one, and each satellite in the system needs to respond quickly to the received tasks based on its own observation information. In similar fields, Prasad and Dusparic applied the MADRL model to energy distribution problems, and Yun et al. used the MADRL algorithm for autonomous collaboration of UAVs in distributed environments. The MADRL method still has great research potential in the field of satellite mission planning.

[0006] In order to meet the development needs of future satellites with high timeliness, autonomy and intelligence, and give full play to the potential brought by technological changes such as satellite autonomous decision-making and multi-satellite collaboration, the present invention proposes a multi-satellite autonomous collaborative scheduling algorithm based on distributed multi-agent reinforcement learning, which effectively breaks through the current limitations of limited on-board computing power and unstable inter-satellite communication links, and supports multi-satellite distributed autonomous decision-making in a local communication environment. Summary of the invention

[0007] The purpose of the present invention is to provide a multi-satellite autonomous collaborative scheduling algorithm based on distributed multi-agent reinforcement learning, which supports each satellite to make distributed autonomous decisions based on local communication information. The present invention can achieve good mission planning effects while maintaining low communication consumption, and has rapid response capabilities and good system anti-destruction performance.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] In a first aspect, the present invention provides a multi-satellite autonomous collaborative scheduling algorithm based on distributed multi-agent reinforcement learning, which specifically includes the following steps:

[0010] S1. Task planning is performed based on target attribute information, satellite attribute information, and multiple constraints. Task planning is used to solve how to effectively allocate and schedule multiple satellite resources and formulate satellite observation plans to maximize the completion of user-submitted tasks.

[0011] S2. Construct a partially observable Markov decision process model adapted to the multi-satellite autonomous collaborative planning mode based on satellite mission planning;

[0012] S3. Considering the limited autonomous planning capabilities of satellites, a distributed multi-agent reinforcement learning algorithm QTRAN is used to build a separate decision network for each satellite.

[0013] S4, using centralized training and distributed execution framework for ground training and on-board execution application mode;

[0014] S5, centralized training phase, centralized training of all satellites, coordinated training by combining all satellite networks;

[0015] S6, distributed execution phase, each satellite makes independent decisions based on the local network and only interacts with some satellites.

[0016] Furthermore, the task planning in step S1 includes an objective function and constraints; specifically, the objective function:

[0017]

[0018] Constraints:

[0019]

[0020] Formula (1-1) is the objective function, which represents maximizing the total revenue of the task. ijk is a decision variable, when x ijk =1, it means that task t j Arrange for satellitesat i The kth visible time window is executed, at this time ts j =ws ijk ,te j =we ijk , on the contrary x ijk =0; time window benefit r ijk The definition is as follows:

[0021]

[0022] The mechanism of benefit decay over time is introduced, which takes into account the importance of the task and the need for timely response. β is the decay factor. is the attenuation coefficient, formula (1-2) is the storage constraint. During the planning period, the storage occupied by the satellite to perform the task shall not exceed the maximum storage capacity of the satellite; formula (1-3) is the task execution uniqueness constraint, which means that each task can be executed at most once; formula (1-4) is the attitude transition time constraint, which means that the interval time between two adjacent tasks performed by the satellite must be greater than its attitude transition time; formula (1-5) is the task continuous observation time constraint, which means that the actual observation time of the task must meet the task imaging duration requirement; formula (1-6) defines the value range of the decision variable.

[0023] Furthermore, the step S2 specifically includes the following steps:

[0024] Multiple satellites can be regarded as a fully cooperative multi-agent system. The multi-agent system is modeled as a decentralized partially observable Markov decision process Dec-POMDP, represented as a tuple in, Represents the real environment in which the agent is located; represents the set of agents {1,...,N}; the action selected by the i-th agent is expressed as The actions selected by all agents constitute a joint action u = [u 1 ,u 2 ,...,u N ]; represents the local observation of the ith agent; Z represents the observation function, P(s′|s,u) represents the state transition probability; R represents the reward function. In a fully cooperative environment, all agents share the same reward function, expressed as r=R(s,u); γ is the discount factor, which is used to balance immediate rewards and future rewards.

[0025] Furthermore, the step S2 further includes the following steps:

[0026] In the multi-satellite autonomous collaborative mission planning system, after receiving the mission requirements, each satellite in the system makes decisions in a distributed manner and generates its own mission planning plan. This method does not require the entire system to share information. Each satellite makes independent decisions based on the information it has, thereby achieving rapid response to requirements. The satellite set is represented by SAT = {sat 1 ,sat 2 ,...,sat N}, including N satellites, sat i represents the i-th satellite; the set of tasks to be planned is represented by T = {t 1 ,t 2 ,...,t |T|}, including |T| tasks, t j represents the jth task in the task set.

[0027] Furthermore, the step S3 specifically includes the following steps: The multi-agent reinforcement learning algorithm VDN based on the value function decomposition idea adopts the CTDE framework, and each agent deploys a decision network Q i , can learn and construct their own action value functions. The idea of ​​value function decomposition is to construct a hybrid network to fit the joint action value function Q in the centralized training stage. jt , through training, the action value function Q of each agent is i and the joint action value function Q jt Satisfies the following relationship:

[0028]

[0029] Formula (2-1) is called the individual-total maximum condition, where represents the action-observation history of the agent. When this condition is met, the optimal action selected by each agent according to its own decision network is equivalent to the joint optimal action of the entire system, so that each agent can ensure the overall optimality of the system under independent decision-making; let represents a vector consisting of the Q-value functions of all agents; when [Q i ] jt When the IGM condition is met, it is said that [Q i ] is Q jtTo construct a value decomposition relation that satisfies IGM, VDN proposes an additive decomposition method:

[0030]

[0031] This constraint can achieve value decomposition that satisfies the IGM condition, but it also imposes structural restrictions on the problem. On this basis, the QMIX algorithm expands this additive decomposition and proposes a monotonic decomposition method, see formula (2-3), so that it can fit a more complex relationship between the joint Q value and the Q value of each agent;

[0032]

[0033] The QTRAN algorithm proposes a decomposition method to decompose the original joint action value function Q jt Transformed into a new function Q′ that is easier to decompose jt , by ensuring that the joint optimal actions of the two are the same, the IGM condition is satisfied;

[0034] definition represents the optimal action of the ith agent, and the set of optimal actions of each agent is expressed as The QTRAN algorithm gives [Q i ] satisfies a sufficient condition for IGM:

[0035]

[0036] in

[0037]

[0038] The QTRAN algorithm directly converts the transfer function Q′ jt The definition is as follows:

[0039]

[0040] The construction method defined by formula (2-6) is similar to the VDN algorithm. Obviously, [Q i ] satisfies Q′ jt The IGM condition, and because Therefore, [Q i ] can be regarded as Q′ jt The value decomposition of

[0041] Formula (2-4) will be used as the training basis for the QTRAN algorithm, which means [Q i ] and Q′ jt The additive decomposition relationship between them will be represented by formula (2-4); in the algorithm training process, three types of functions are involved: the Q value function of each agent [Qi ], joint action value function Q jt And the function V jt , due to the conversion function Q′ jt Can be directly obtained by [Q i ], so no additional definition is needed. In order to show Q more clearly jt and its conversion function Q′ jt The relationship between , combining formula (2-3) and formula (2-6) to obtain formula (2-7):

[0042]

[0043] From the above analysis, we can see that [Qi] and Q′ jt The relationship between jt and V jt To fit; during the training process, the QTRAN algorithm additionally introduces the function V jt , V jt Can be regarded as a correction term to correct Q jt and Q′ jt The differences between them can characterize the complex relationships in multi-agent systems.

[0044] Furthermore, the step S4 specifically includes: under the CTDE framework, each satellite will deploy a decision network. During the training phase, an additional hybrid network is constructed to assist learning. During the execution phase, each satellite only needs to make independent decisions based on local observation information.

[0045] Furthermore, the step S5 specifically includes: in order to improve the generalization ability of the algorithm in different scenarios, the training process adopts an algorithm training mechanism for random initial scenarios: several training scenarios are generated, and one scenario is randomly selected for training in each iteration; in the predetermined scenario, each satellite makes decisions independently in a distributed manner to obtain its own mission planning plan; the training data generated by the decision-making process will be put into the experience pool for subsequent algorithm training.

[0046] Furthermore, the step S6 specifically includes: for a homogeneous multi-satellite system, in the distributed execution process, each imaging satellite can be regarded as having the same decision-making process. i The trained model Q is pre-deployed i , after receiving the mission set, the satellite sat i Based on the model Q i Make decisions independently on the entire set of tasks.

[0047] This application is aimed at the development needs of future satellite autonomy, intelligence, and rapid response. Combined with the current development status of limited satellite computing power, simple inter-satellite coordination mechanism, and unstable communication links, this application innovatively proposes a multi-satellite autonomous collaborative scheduling algorithm based on distributed multi-agent reinforcement learning. First, a partially observable Markov decision process model adapted to the multi-satellite autonomous collaborative planning mode is constructed, and each component of the model is designed in a targeted manner based on full consideration of the actual conditions on the satellite. Considering the current limited autonomous planning capabilities of satellites, this application introduces a low-complexity distributed multi-agent reinforcement learning algorithm-QTRAN. The algorithm constructs a separate decision network for each satellite and adopts a centralized training and distributed execution framework, which is suitable for ground training and on-board execution. In the centralized training stage, collaborative training is carried out by combining all satellite networks to cultivate the global optimization awareness of each satellite and ensure the overall mission planning effect. In the distributed execution stage, each satellite makes independent decisions based on the local network and only needs to interact with some satellites. Finally, this application constructs multi-satellite mission planning scenarios of different scales to verify the effectiveness of the algorithm. The simulation results show that the algorithm can effectively reduce inter-satellite communication consumption while ensuring the planning effect, and has faster response speed and higher system anti-destruction capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 Schematic diagram of the local observation space for fusing multi-source information in the present invention;

[0050] Figure 2 It is a single-star network structure in the present invention;

[0051] Figure 3 It is the hybrid network structure in the present invention;

[0052] Figure 4 The algorithm operation process at the decision step t in the present invention is

[0053] Figure 5 Analysis of the QTRAN algorithm training and evaluation process. DETAILED DESCRIPTION

[0054] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0055] Imaging satellites need to consider many factors when completing imaging tasks. When a user makes an imaging request, task planning needs to be carried out based on target attribute information, satellite attribute information, and multiple constraints. Task planning mainly solves how to effectively allocate and schedule multiple satellite resources and formulate satellite observation plans to maximize the completion of user-submitted tasks. The effect of task planning directly affects the effectiveness of the imaging satellite earth observation system in performing tasks and is a key link in ensuring the efficient operation of the system.

[0056] Since the imaging satellite mission planning problem contains many complex constraints, in order to highlight the research focus, this application makes the following assumptions and simplifications:

[0057] (1) All satellites have sufficient power.

[0058] (2) The image download process is not considered.

[0059] (3) All targets are point targets and can be observed by a single satellite within a single time window.

[0060] The relevant symbols and descriptions of the imaging satellite mission planning problem are shown in Table 1.

[0061] Table 1 Symbols and descriptions of imaging satellite mission planning problems

[0062]

[0063] Mission planning includes objective functions and constraints; specifically,

[0064] Objective function:

[0065]

[0066] Constraints:

[0067]

[0068] Formula (1-1) is the objective function, which represents maximizing the total revenue of the task. ijk is a decision variable, when x ijk =1, it means that task t j Arrange for satellitesat iThe kth visible time window is executed, at this time ts j =ws ijk ,te j =we ijk , on the contrary x ijk = 0. Time window benefit r ijk The definition is as follows:

[0069]

[0070] This function introduces a mechanism in which the benefits decay over time, taking into account the importance of the task and the need for timely response. β is called the decay factor. is the attenuation coefficient. Formula (1-2) is the storage constraint. During the planning period, the storage occupied by the satellite to perform the task must not exceed the maximum storage capacity of the satellite. Formula (1-3) is the task execution uniqueness constraint, which means that each task can be executed at most once. Formula (1-4) is the attitude transition time constraint, which means that the interval time between two adjacent tasks performed by the satellite must be greater than its attitude transition time. Formula (1-5) is the task continuous observation time constraint, which means that the actual observation time of the task must meet the task imaging duration requirement. Formula (1-6) defines the value range of the decision variable.

[0071] In the process of multi-satellite autonomous collaborative scheduling, multiple satellites can be regarded as a fully cooperative multi-agent system, which share the same environment and influence each other. Since the agents are often unable to observe the entire environment, the multi-agent system is usually modeled as a decentralized partially observable Markov decision process (Dec-POMDP), represented by a tuple in, Represents the real environment in which the agent is located; represents the set of agents {1,...,N}; the action selected by the i-th agent is expressed as The actions selected by all agents constitute a joint action u = [u 1 ,u 2 ,...,u N ]; represents the local observation of the ith agent; Z represents the observation function, P(s′|s,u) represents the state transition probability; R represents the reward function. In a fully cooperative environment, all agents share the same reward function, expressed as r=R(s,u); γ is the discount factor, which is used to balance immediate rewards and future rewards.

[0072] In 2017, the DeepMind team proposed a multi-agent reinforcement learning algorithm based on the value function decomposition idea, VDN, which achieved good results in solving multi-agent problems. Subsequently, many improved algorithms were derived based on this idea, becoming an important research direction in the field of multi-agent reinforcement learning. The VDN algorithm uses the CTDE framework, and each agent deploys a decision network Q i , which can learn and construct their own action-value functions. The idea of ​​value function decomposition is to construct a hybrid network to fit the joint action-value function Q in the centralized training phase. jt , through training, the action value function Q of each agent is i and the joint action value function Q jt Satisfies the following relationship:

[0073]

[0074] Formula (2-1) is called the individual-global-max (IGM) condition, where represents the action-observation history of the agent. When this condition is met, the optimal action selected by each agent based on its own decision network is equivalent to the joint optimal action of the entire system, so that each agent can ensure the overall optimality of the system under independent decision-making. represents a vector consisting of the Q-value functions of all agents. i ] jt When the IGM condition is met, it is said that [Q i ] is Q jt To construct a value decomposition relation that satisfies IGM, VDN proposes an additive decomposition method:

[0075]

[0076] This constraint can achieve value decomposition that satisfies the IGM conditions, but it also imposes structural restrictions on the problem. On this basis, the QMIX algorithm expands this additive decomposition and proposes a monotonic decomposition method, see formula (2-3), which can fit a more complex relationship between the joint Q value and the Q value of each agent.

[0077]

[0078] The QTRAN algorithm proposes a more general decomposition method, which transforms the original joint action value function Q jt Transformed into a new function Q′ that is easier to decompose jt , the IGM condition is satisfied by ensuring that the joint optimal actions of the two are the same.

[0079] definition represents the optimal action of the ith agent, and the set of optimal actions of each agent is expressed as The QTRAN algorithm gives [Q i ] satisfies a sufficient condition for IGM:

[0080]

[0081] in

[0082]

[0083] The QTRAN algorithm directly converts the transfer function Q′ jt The definition is as follows:

[0084]

[0085] The construction method defined by formula (2-6) is similar to the VDN algorithm. Obviously, [Q i ] satisfies Q′ jt The IGM condition, and because Therefore, [Q i ] can be regarded as Q′ jt The value decomposition of .

[0086] Formula (2-4) will be used as the training basis for the QTRAN algorithm, which means [Q i ] and Q′ jt The additive decomposition relationship between them will be represented by formula (2-4). Therefore, in the algorithm training process, there are three types of functions involved: the Q value function of each agent [Q i ], joint action value function Q jt And the function V jt , due to the conversion function Q′ jt Can be directly obtained by [Q i ], so no additional definition is needed. In order to show Q more clearly jt and its conversion function Q′ jt The relationship between , combining formula (2-3) and formula (2-6) to obtain formula (2-7):

[0087]

[0088] From the above analysis, we can see that [Q i ] and Q′ jt The relationship between jt and V jt Therefore, although the conversion function Q′ jt The form is similar to the VDN algorithm, but during the training process, the QTRAN algorithm additionally introduces the function V jt , Vjt Can be regarded as a correction term to correct Q jt and Q′ jt The differences between them can represent more complex relationships in multi-agent systems.

[0089] In the scenario of multi-satellite autonomous collaborative scheduling, each satellite can only communicate with neighboring satellites due to communication constraints; if it needs to obtain information from more distant satellites, it needs to use intermediate satellites for information transmission. This method is not only time-consuming when it comes to long-distance communication, but may also reduce the robustness of task planning due to problems such as signal interference and data loss. Therefore, the algorithm strives to reduce communication consumption while ensuring the effect of task planning. The CTDE framework adopted by the multi-agent reinforcement learning algorithm shows its applicability in the context of this problem. Under the CTDE framework, each satellite has an independent decision network. The centralized training phase can be carried out on the ground, using the pre-collected data on the satellite to integrate all satellite networks for joint training. After the training is completed, the decision network is deployed to the corresponding satellite. In the distributed execution phase, each satellite makes independent plans based on its decision model, and only needs to communicate with some satellites to make better decisions, thereby greatly reducing the inter-satellite communication consumption. Therefore, the above-mentioned multi-agent reinforcement learning algorithm based on the value decomposition idea can be effectively applied to the scenario of multi-satellite autonomous collaborative scheduling.

[0090] In imaging satellite mission planning, a uniqueness constraint is set to avoid wasting resources. This means that from the perspective of a single satellite, in order to ensure the overall mission planning effect, the best action is not necessarily to choose to execute the mission, which makes the relationship between the joint Q value and the Q value of each satellite more complicated. Therefore, this application chooses the QTRAN algorithm with a more general value decomposition method to solve the multi-satellite autonomous mission planning problem.

[0091] In the multi-satellite autonomous collaborative mission planning system, after receiving the mission requirements, each satellite in the system makes decisions in a distributed manner and generates its own mission planning plan. This method does not require the entire system to share information. Each satellite makes independent decisions based on the information it has, thereby enabling rapid response to requirements. The satellite set is represented by SAT = {sat 1 ,sat 2 ,...,sat N}, including N satellites, sat i represents the i-th satellite; the set of tasks to be planned is represented by T = {t 1 ,t 2 ,...,t |T|}, including |T| tasks, t j represents the jth task in the task set.

[0092] Since the design of multi-satellite autonomous mission planning algorithms is closely related to the characteristics of the satellites themselves, this application defines the characteristics of the research object in detail, as follows:

[0093] (1) All satellites are homogeneous. This means that the imaging functions of each satellite are similar, they follow the same mission planning rules, and can be solved using the same decision model.

[0094] (2) Intersatellite communication is limited. Considering many issues such as signal interference, data loss, energy limitations, and the time cost of communication, algorithm design needs to minimize the consumption of intersatellite communication.

[0095] From the perspective of a single satellite, the following features are set:

[0096] (1) Satellites have multiple channels to obtain mission planning information, such as ground transmission, local observation, and intersatellite communication.

[0097] (2) Due to communication limitations, satellites can only obtain partial system information through inter-satellite communications.

[0098] (3) During the entire mission planning process, the behavior of the satellite can be summarized as: receiving the mission set - collecting information - mission planning - program execution. During this process, the behavior of the satellite is completely independent of other satellites. Only in the information collection stage does inter-satellite interaction occur, but this behavior is initiated autonomously by the satellite for the sole purpose of obtaining information. The so-called inter-satellite coordination refers to coordination at the strategy level. This strategy network is pre-trained and deployed on each satellite.

[0099] For a fully cooperative multi-agent system, at a certain decision step, after all satellites take actions on the environment, the environment will return a joint action reward. In order to train each satellite to achieve tacit coordination under distributed decision-making, the model is set so that within the predetermined decision step, all satellites make decisions for the same task. In this way, the decision objects faced by all satellites during training are consistent, so that the information on which the decision is based is more similar. Under this setting, after receiving the task set, each satellite first sorts the tasks according to predetermined rules to obtain a unified decision order. This application sets the sorting rule to sort from high to low according to task priority, and the sorted task decision sequence is expressed as T′={t′ 1 ,t′ 2 ,...,t′ |T|}. i For task t′ j The action space for decision making is represented as:

[0100]

[0101] The size of the decision space for each decision step is “the number of satellites + 1”. Indicates not to select any satellite. Indicates satellite i The task t′ j Assigned to the satellite with the corresponding index.

[0102] Action selection must strictly meet the constraints in the mathematical model in Section 1.1.3. Considering the constraint coupling relationship between tasks, a constrained single-step check mechanism is designed to screen the optional space of actions. Specifically:

[0103] (1) For the onboard storage constraints, a forward storage check mechanism is designed. The specific operation is: for the current decision task t′ j , assuming that its occupied storage is sm′ j , satellite i Before making a decision, the satellites in the saved status information will be pre-screened. k Remaining storage Does it meet the requirements? Action Not selectable.

[0104] (2) For time window constraints, when two time windows overlap or do not meet the conversion time requirements, the two time windows are said to conflict. A time window single-step troubleshooting mechanism is designed for this purpose. The specific operation is as follows: i Before the initial decision, set a 0-1 variable to mark task t′ j Is there a time window on each satellite that satisfies the constraints: in, Indicates satellite i Consider task t′ j On satellitesat k There is a time window, action Satisfy the time window constraint. On the contrary, if It indicates action Not optional. Every time a decision is made, the flag variables of the remaining tasks need to be updated.

[0105] (3) In particular, when multiple satellites make decisions based on local observation information, it is inevitable that tasks will be repeated, that is, the uniqueness constraint (Formula (1-3)) is difficult to satisfy. Since this constraint is intended to avoid the waste of satellite resources and is not due to the limitation of satellite execution capabilities, it is defined as a soft constraint and used as the learning goal of the algorithm to minimize the repeated execution of tasks.

[0106] Under the multi-agent reinforcement learning algorithm, each satellite independently conducts mission planning to develop its own mission execution plan. Therefore, although each satellite can generate a plan for the entire system during the decision-making process, it will ultimately only retain the part of the plan related to itself. i The set of tasks to be performed on this star after the decision is made can be expressed as:

[0107]

[0108] Since mission planning is for the entire system, and each satellite in the system makes independent decisions in a distributed manner, in order to ensure the overall mission planning effect, the more fully the satellite has mastered the global information, the better. Satellites have multiple channels for obtaining information, including ground uploads, local satellite perception, and inter-satellite interaction. The ground can predict the status of the entire on-board system, but errors are inevitable; on-board information is highly real-time, but due to communication limitations, all information cannot be obtained in a timely manner. Based on the above analysis, the model designs a local observation space that integrates multi-source information, using satellite sat i Take it as an example, and express its local observation space as:

[0109] O i = {O i1 ,O i2 ,...,O iN} 2-10

[0111] satellite i The information of all satellites in the system is maintained. The difference is that the information sources of different satellites are different. Figure 1 Taking four satellites as an example, the information source composition of the satellite local observation space is demonstrated. Assuming that the four satellites are located in the same orbital plane and distributed in a ring shape, each satellite can establish a communication link with two adjacent satellites (for example, satellite sat 1 The adjacent satellites are sat 2 and sat 4 ). When the ground uploads the mission set to the satellite, it also uploads the mission planning related information (prediction data) of all satellites in the corresponding orbital plane. After receiving this information, the satellite can first update its own mission planning related information based on local perception; then, through inter-satellite communication, the satellite can further update the mission planning related information of surrounding satellites. However, due to the limitation of inter-satellite communication capabilities, it is difficult for satellites to obtain information about all satellites in the entire orbital plane in a timely manner. Therefore, the information maintained by each satellite will eventually include information from the ground, the satellite itself, and adjacent satellites. Figure 1 (c).

[0112] The decision step j satellite sat i Maintained about satellitesat k The information is represented as:

[0113]

[0114] Indicates satellite k Execute task t′ j The benefits that can be obtained (see formula (1-7)); Indicates satellite k For task t′ j The conflict degree of the visible time window is further divided into two categories: in Indicates the number of time windows that conflict with this time window, referred to as the conflict number. It represents the sum of the benefits corresponding to the time windows that conflict with this time window, referred to as the conflict value; Indicates satellite k Available storage at decision step j; Indicates satellite k The number of tasks that have been assigned at decision step j. It is a static feature and remains unchanged throughout the decision-making process; It is a dynamic feature and needs to be updated after each decision.

[0115] The global state space s∈S is consistent with the local observation space representation maintained by a single satellite. The difference is that the information represented by the global state space all comes from the on-board environment, that is, it represents the real-time information of the entire system.

[0116] In the QTRAN algorithm design, after all agents have chosen their actions, the environment feeds back the joint action reward, and all satellites in the decision step j are recorded as task t′ j After the decision is made, the joint action reward of the environment feedback is r j , the corresponding joint action reward function is expressed as:

[0117]

[0118] From formula (2-9), we can see that when calculating the joint action reward, we only need to consider whether each satellite chooses to transfer the current task t′ j Assign to itself, thereby setting the decision variable Used to represent satellites i Whether to choose to execute task t′ j , specifically expressed as:

[0119]

[0120] Formula (2-12) means: When multiple satellites select observation tasks t′ j When , the joint action reward is the ratio of the maximum benefit obtained by each observation satellite to the total number of observation satellites. This is because in the problem context where the task only needs to be observed once, even if the same task is observed multiple times, only one benefit can be obtained. At the same time, in order to avoid wasting resources, repeated observation behavior is punished by dividing by the total number of observation satellites, thereby guiding multi-satellite decision-making behavior. On this basis, the cumulative reward obtained in a scenario can be expressed as:

[0121]

[0122] Under the CTDE framework, each satellite will deploy a decision network. During the training phase, additional hybrid networks will be constructed to assist learning. During the execution phase, each satellite only needs to make independent decisions based on local observation information. This application gives the specific structure of each satellite decision network and hybrid network and their update methods, and elaborates on the centralized training and distributed execution processes in detail in combination with the specific context of satellite mission planning.

[0123] Since each satellite makes decisions under partial observation, a deep recurrent Q-learning (DRQN) suitable for partially observable Markov decision processes is configured on each satellite. The specific network structure is shown in Figure 2 . MLP represents a basic neural network structure, with hidden states It is a key component of the DRQN ​​algorithm, which provides a memory mechanism for the agent, which retains past observation information by updating it at each decision step, so that the agent can better use historical information to make decisions. Gating mechanisms in the network, such as LSTM or GRU, are often used to promote learning on a longer time scale.

[0124] Each satellite i There is a single DRQN ​​network that receives individual observations at each decision step and the action at the previous time step Output Q of all actions i (τ i ,·) values, and select actions through the ε-greedy strategy during the training phase, and output the Q value of the corresponding action.

[0125] The QTRAN algorithm needs to be constructed as Figure 3 Three independent neural networks are used to fit Q′ jt and [Q i ] is the value decomposition relationship between them.

[0126] Q network of each satellite: All satellite network outputs Q i The sum of the values ​​constitutes Q′ jt , Figure 3 (c) shows this process.

[0127] United Q Network: Used to calculate the joint action value Q of all satellites jt The network is based on the sum of all individual network hidden layer features The hidden state parameters h of each satellite are shared in the form of t , and at the same time with the joint action u of all satellites t and the global state s t As input, the process is as follows Figure 3 (a) shown.

[0128] State value network: Calculate the scalar state value for the modified Q jt and Q′ jt The network also shares the characteristic parameters of the hidden layer of the individual network. From formula (2-5), we can see that the calculation of the state value is independent of the action selection of each satellite. Therefore, in addition to the hidden layer state parameters, the network only uses the global state s t As input, Figure 3 (b) shows this process.

[0129] The QTRAN algorithm uses the CTDE framework, which differs between the training phase and the execution phase. The centralized training phase and the distributed execution phase will be described in detail.

[0130] The centralized training process is shown in Algorithm 2.1. In order to improve the generalization ability of the algorithm in different scenarios, the training process adopts an algorithm training mechanism for random initial scenarios: several training scenarios are generated, and one scenario is randomly selected for training in each iteration. Under the predetermined scenario, each satellite makes its own decision in a distributed manner to obtain its own mission planning solution. The training data generated by this decision-making process will be put into the experience pool for subsequent algorithm training.

[0131]

[0132]

[0133] The specific decision-making steps for each satellite are as follows: when receiving the task set, obtain and update local observation information, which can come from multiple channels such as ground upload, local satellite perception, and inter-satellite interaction. Then, determine the task decision order according to the established rules, and make decisions on the tasks one by one. Algorithm 2.1, lines 7-11, show this sequential decision process. First, a constraint check is performed on the current decision task to determine the set of feasible actions. Then, on this basis, actions are selected according to the ε-greedy strategy and the state is updated. When all tasks are decided, the task planning scheme for the satellite can be obtained. In addition, in the centralized training stage, it is necessary to save the global state information. This global information is irrelevant to the decision-making process of each satellite and is only used as part of the input of the hybrid network in the training stage to assist model training.

[0134] Since the training process of deep reinforcement learning algorithms requires a lot of computing resources, the limited computing power on board cannot support it. Therefore, the centralized training process will be carried out on the ground, simulating the actual operation scenario by pre-collecting satellite and ground data. After the training is completed, the model will be deployed on each satellite to support the distributed decision-making of the onboard system.

[0135] The distributed execution process is shown in Algorithm 2.2. For a homogeneous multi-satellite system, each imaging satellite can be considered to have the same decision-making process during the distributed execution process. Therefore, Algorithm 2.2 only shows this process from the perspective of a single satellite. i The trained model Q is pre-deployed i , after receiving the mission set, the satellite sat i Based on the model Q i The decision-making process for the entire task set is completed independently. Except for selecting the action with the largest Q value at each decision step, the rest of the decision-making process is exactly the same as the single-star decision-making process shown in Algorithm 2.1. This design allows the model to be directly applied to distributed execution scenarios after training.

[0136]

[0137] Regarding the on-board distributed autonomous scheduling process, the following points need to be emphasized:

[0138] (1) This process belongs to pre-planning. When the satellite receives the task set T, it starts to make decisions and formulate a plan for the satellite in a certain period of time in the future. Then it will perform the observation mission according to the plan.

[0139] (2) Since it is pre-planned, after selecting the execution satellite for the current task in line 7 of Algorithm 2.2, the corresponding satellite will not be ordered to execute the task immediately. Instead, it is assumed that the task can be successfully executed and the set reward can be obtained.

[0140] (3) Considering the communication constraints, it is stipulated that the satellite will only obtain information through intersatellite communication when receiving the task set, and no further communication activities are involved in the decision-making process of lines 5-8 of Algorithm 2.2. The state transition is designed to be updated according to the existing data under the premise that the action can be executed smoothly.

[0141] (4) Since the information about each satellite is real-time, this decision-making model can ensure that the mission planning scheme of the satellite is not affected by ground prediction data and strictly complies with actual constraints.

[0142] The training process and execution process in a certain predetermined scenario can be found in Figure 4 .

[0143] In the training and evaluation process of the model in the 12 satellites - 120 missions scenario, in order to simplify the expression, the scenario configuration will be expressed as "number of satellites - number of missions", such as 12-120. There are two indicators for evaluating the effectiveness of the QTRAN algorithm. One is the cumulative reward in the training phase represented by formula (2-14), and the other is the overall mission planning goal represented by formula (1-1). Among them, formula (2-14) can be used to observe the training effect of the algorithm, and formula (1-1) is the final evaluation indicator of this application. In order to distinguish the two, in the subsequent content, formula (2-14) will be referred to as "total reward" and formula (1-1) will be referred to as "total benefit".

[0144] During the training process, the algorithm records the total rewards and total benefits obtained in each iteration. Figure 5 (a) shows the curves of total reward and total revenue changing with the training process. Since the model interacts with random scenarios, in order to more clearly observe the overall trend of the curve, the curve is smoothed and the smoothness is defined as ξ, which means taking the average of the data of each ξ generation. In the experiment of this application, ξ=200 is set. It can be seen that the curve finally converges to a relatively stable state after a period of rise. The rising trends of total reward and total revenue are similar, which shows that there is consistency between the design of the reward function and the final total revenue. For the evaluation process, it is set to perform an overall test on the evaluation example every 200 generations of training, and the curve smoothness is set to ξ=50, which is equivalent to taking the average of the entire evaluation set. In order to more clearly observe the changes in the effect of the evaluation results as the training process progresses, Figure 5 (b) When displaying the evaluation curve trend, the horizontal axis is enlarged proportionally to match the number of iterations of the training curve. It can be seen that the evaluation curve rises rapidly in the early stage, and then the rise slows down and is accompanied by large fluctuations. This may be due to random decisions during the exploration process, and the final curve achieves a relatively stable effect. Unless otherwise specified in subsequent experiments, the training and evaluation processes will be displayed using the above processing method.

[0145] Figure 5 (c) and Figure 5 (d) shows the changes in the number of repeated tasks during training and evaluation. In the multi-satellite distributed decision-making process based on local observations, repeated task claims are inevitable. However, in the case of limited resources, repeated claims will cause resource waste. To ensure the planning effect, the algorithm expects that the task is claimed only once. Figure 5 (c) and Figure 5 As can be seen from (d), the number of tasks that were claimed only once changed the most significantly, with a clear upward trend. A small number of tasks were claimed twice, and their corresponding curves decreased slightly during the training process. A very small number of tasks were claimed three times, and after training, they almost reached a state close to 0.

[0146] This application is aimed at the development needs of future satellite autonomy, intelligence, and rapid response. Combined with the current development status of limited satellite computing power, simple inter-satellite coordination mechanism, and unstable communication links, this application innovatively proposes a multi-satellite autonomous collaborative scheduling algorithm based on distributed multi-agent reinforcement learning. First, a partially observable Markov decision process model adapted to the multi-satellite autonomous collaborative planning mode is constructed, and each component of the model is designed in a targeted manner based on full consideration of the actual conditions on the satellite. Considering the current limited autonomous planning capabilities of satellites, this application introduces a low-complexity distributed multi-agent reinforcement learning algorithm-QTRAN. The algorithm constructs a separate decision network for each satellite and adopts a centralized training and distributed execution framework, which is suitable for ground training and on-board execution. In the centralized training stage, collaborative training is carried out by combining all satellite networks to cultivate the global optimization awareness of each satellite and ensure the overall mission planning effect. In the distributed execution stage, each satellite makes independent decisions based on the local network and only needs to interact with some satellites. Finally, this application constructs multi-satellite mission planning scenarios of different scales to verify the effectiveness of the algorithm. The simulation results show that the algorithm can effectively reduce inter-satellite communication consumption while ensuring the planning effect, and has faster response speed and higher system anti-destruction capability.

[0147] Another embodiment of the present invention further provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned method. If the specific technical details are not disclosed, please refer to the method part of the above-mentioned embodiment of the present invention.

[0148] Another embodiment of the present invention further provides a computer-readable storage medium, on which computer instructions are stored. The computer instructions are used to enable the computer to execute the above method. If the specific technical details are not disclosed, please refer to the method part of the above embodiment of the present invention.

[0149] The medium in the present invention can adopt any combination of one or more computer-readable media. The medium can be a computer-readable signal medium or a computer-readable storage medium. The medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of the medium (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, the medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0150] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A multi-satellite autonomous collaborative scheduling method based on distributed multi-agent reinforcement learning, specifically comprising the following steps: S1. Task planning is performed based on target attribute information, satellite attribute information, and multiple constraints. Task planning is used to effectively allocate and schedule multiple satellite resources and formulate satellite observation plans to maximize the completion of user-submitted tasks. S2. Construct a partially observable Markov decision process model adapted to the multi-satellite autonomous collaborative planning mode based on satellite mission planning; S3. Considering the limited autonomous planning capabilities of satellites, a distributed multi-agent reinforcement learning algorithm QTRAN is used to build a separate decision network for each satellite. S4, using centralized training and distributed execution framework for ground training and on-board execution application mode; S5, centralized training phase, centralized training of all satellites, coordinated training by combining all satellite networks; S6, distributed execution phase, each satellite makes independent decisions based on the local network and only interacts with some satellites.

2. The method according to claim 1, characterized in that The task planning in step S1 includes an objective function and constraints; specifically, the objective function: Constraints: Formula (1-1) is the objective function, which means maximizing the total benefit of the task; where x ijk is a decision variable, when x ijk =1, it means that task t j Arrange for satellitesat i The kth visible time window is executed, at this time ts j =ws ijk ,te j =we ijk , on the contrary x ijk =0; time window benefit r ijk The definition is as follows: Among them, the satellite set SAT = {sat1, sat2, ..., sat |S| }, |S| is the number of satellites, and sat i represents a satellite, i is the satellite identifier, M i For satellitesat i Maximum storage capacity within the planning period; m j To perform the task j Satellite storage required; d j For task t j Required observation duration; ts j and j They are task t j The actual start and end time of the observation; p j For task t j The priority of the task set T = {t1, t2, ..., t |T| }, |T| is the number of tasks, t j represents a mission, j is the mission identifier, and satellite sat i For task t j The visible time window set |W ij | indicates satellite sat i For task t j The number of visible time windows. In one orbital period, a satellite can have at most one visible time window for the same mission. ijk Indicates satellite i For task t j The kth visible time window of ws ijk and we ijk Represent the visible time window w ijk The start and end time of ijk represents the time window w ijk Observation mission j of income; Indicates satellite i Execute adjacent tasks t in time windows w and w′ respectively j ,t j‘ The required attitude conversion time; similarly, W can be obtained ij′ and x ij′k′ The object represented by ; among them, a mechanism is introduced that the benefits decay over time, taking into account the two factors of task importance and timely response requirements, β is the decay factor, is the attenuation coefficient, formula (1-2) is the storage constraint. During the planning period, the storage occupied by the satellite to perform the task shall not exceed the maximum storage capacity of the satellite; formula (1-3) is the task execution uniqueness constraint, which means that each task can be executed at most once; formula (1-4) is the attitude transition time constraint, which means that the interval time between two adjacent tasks performed by the satellite must be greater than its attitude transition time; formula (1-5) is the task continuous observation time constraint, which means that the actual observation time of the task must meet the task imaging duration requirement; formula (1-6) defines the value range of the decision variable.

3. The method according to claim 1, characterized in that The step S2 specifically includes the following steps: Multiple satellites can be regarded as a fully cooperative multi-agent system. The multi-agent system is modeled as a decentralized partially observable Markov decision process Dec-POMDP, represented as a tuple in, Represents the real environment in which the agent is located; represents the set of agents {1,...,N}; the action selected by the i-th agent is expressed as The actions selected by all agents constitute a joint action u = [u 1 ,u 2 ,...,u N ]; represents the local observation of the ith agent; Z represents the observation function, P(s′|s,u) represents the state transition probability; R represents the reward function. In a fully cooperative environment, all agents share the same reward function, expressed as r=R(s,u); γ is the discount factor, which is used to balance immediate rewards and future rewards.

4. The method according to claim 1, characterized in that: The step S2 further comprises the following steps: In the multi-satellite autonomous collaborative mission planning system, after receiving the mission requirements, each satellite in the system makes decisions in a distributed manner and generates its own mission planning plan. This method does not require the entire system to share information. Each satellite makes independent decisions based on the information it has, thereby achieving rapid response to requirements. The satellite set is represented by SAT = {sat1, sat2, ..., sat N }, including N satellites, sat i represents the i-th satellite; the set of tasks to be planned is represented by T = {t1, t2, ..., t |T| }, including |T| tasks, t j represents the jth task in the task set.

5. The method according to claim 3, characterized in that: The step S3 specifically includes the following steps: The multi-agent reinforcement learning algorithm VDN based on the value function decomposition idea adopts the CTDE framework. Each agent deploys a decision network Qi, which can learn and construct its own action value function. The idea of ​​value function decomposition is to construct a hybrid network to fit the joint action value function Q in the centralized training stage. jt , through training, the action value function Q of each agent is i and the joint action value function Q jt Satisfies the following relationship: Formula (2-1) is called the individual-total maximum condition, where represents the action-observation history of the agent. When this condition is met, the optimal action selected by each agent according to its own decision network is equivalent to the joint optimal action of the entire system, so that each agent can ensure the overall optimality of the system under independent decision-making; let represents a vector consisting of the Q-value functions of all agents; when [Q i ] jt When the IGM condition is met, it is said that [Q i ] is Q jt To construct a value decomposition relation that satisfies IGM, VDN proposes an additive decomposition method: This constraint can achieve value decomposition that satisfies the IGM condition, but it also imposes structural restrictions on the problem. On this basis, the QMIX algorithm expands this additive decomposition and proposes a monotonic decomposition method, see formula (2-3), so that it can fit a more complex relationship between the joint Q value and the Q value of each agent. The QTRAN algorithm proposes a decomposition method to decompose the original joint action value function Q jt Transformed into a new function Q′ that is easier to decompose jt , by ensuring that the joint optimal actions of the two are the same, the IGM condition is satisfied; definition represents the optimal action of the ith agent, and the set of optimal actions of each agent is expressed as The QTRAN algorithm gives a sufficient condition for [Qi] to satisfy IGM: in The QTRAN algorithm directly converts the transfer function Q′ jt The definition is as follows: In formula (2-6) [Q i ] satisfies Q′ jt The IGM condition, and because Therefore, [Q i ] can be regarded as Q′ jt The value decomposition of Formula (2-4) will be used as the training basis for the QTRAN algorithm, which means [Q i ] and Q′ jt The additive decomposition relationship between them will be represented by formula (2-4); in the algorithm training process, three types of functions are involved: the Q value function of each agent [Q i ], joint action value function Q jt And the function V jt , due to the conversion function Q′ jt Can be directly obtained by [Q i ] indicates that in order to more clearly show Q jt and its conversion function Q′ jt The relationship between , combining formula (2-3) and formula (2-6) to obtain formula (2-7): From the above analysis, we can see that [Q i ] and Q′ jt The relationship between jt and V jt To fit; during the training process, the QTRAN algorithm additionally introduces the function V jt , V jt Can be regarded as a correction term to correct Q jt and Q′ jt The differences between them can characterize the complex relationships in multi-agent systems.

6. The method according to claim 1, characterized in that The step S4 specifically includes: under the CTDE framework, each satellite will deploy a decision network. During the training phase, an additional hybrid network is constructed to assist learning. During the execution phase, each satellite only needs to make independent decisions based on local observation information.

7. The method according to claim 1, characterized in that The step S5 specifically includes: in order to improve the generalization ability of the algorithm in different scenarios, the training process adopts an algorithm training mechanism for random initial scenarios: a number of training scenarios are generated, and one scenario is randomly selected for training in each iteration; in the predetermined scenario, each satellite makes its own decision in a distributed manner to obtain its own mission planning plan; the training data generated by the decision-making process will be put into the experience pool for subsequent algorithm training.

8. The method according to claim 1, characterized in that The step S6 specifically includes: for a homogeneous multi-satellite system, in the distributed execution process, each imaging satellite can be regarded as having the same decision-making process. In the execution stage, the satellite sat i The trained model Qi is pre-deployed. After receiving the task set, the satellite sat i The decision on the entire task set will be made independently based on model Qi.

Images