Satellite space target cooperative observation distributed planning method based on multi-agent reinforcement learning

Abstract

A satellite space target cooperative observation distributed planning method based on multi-agent reinforcement learning, comprising: constructing an on-board Actor network and a ground centralized Critic network to form a centralized training and distributed execution architecture; designing a structured mixed space action, training the Actor-Critic network through a large number of ground simulations and using an improved MAHPPO algorithm and a curriculum learning technique; deploying the trained Actor network on each satellite of the constellation executing the cooperative observation task to realize distributed execution; after uploading the list of space targets to be observed to each satellite through satellite-ground communication, each satellite does not need to communicate with other satellites, each satellite executes the observation target of each time slice decided by the on-board Actor network, and controls the satellite to implement the observation task, thereby realizing the space target cooperative observation task. The present application enables each satellite to complete the cooperative observation without communication during the on-orbit execution, and can be extended to large-scale constellation applications.

Description

Technical Field

[0001] This invention belongs to the field of distributed satellite system collaborative observation planning technology, specifically involving a distributed planning method for collaborative observation of satellite space targets based on multi-agent reinforcement learning. Background Technology

[0002] With the development of aerospace technology and the increasing demands of space-based applications, space debris and satellites have significantly increased, making the observation of space targets far more important. Due to the large number and wide distribution of satellites in orbit, an observation system composed of satellite constellations is needed to traverse the targets to be observed in the shortest possible time. Furthermore, due to the characteristics of space targets such as their material composition and distance, important space targets often require coordinated observation from multiple satellites to achieve high-precision monitoring and identification, and to obtain detailed characteristic information.

[0003] There are two main types of collaborative observation mission planning for space targets: centralized and distributed approaches.

[0004] The traditional control model is a centralized ground-based approach, which requires the ground to generate observation plans first, and then upload the observation plans to each satellite. This method is difficult to cope with sudden emergencies and changes in the operating environment, and is also constrained by the telemetry and control time window.

[0005] Centralized decision-making on satellites requires satellites to act as management nodes. These management nodes send planning results to each satellite, placing very high demands on their communication capabilities, computing power, and reliability.

[0006] On-board distributed decision-making allows each satellite to make independent decisions based on its own operational status and the list of targets to be observed, offering advantages such as high reliability, flexibility, and rapid response. Traditional on-board distributed decision-making requires communication between satellites to coordinate observation plans, placing high demands on inter-satellite communication links.

[0007] Patent document publication number CN113128828 discloses a distributed online planning method for satellite observation based on multi-agent reinforcement learning, which uses an on-board decision model to decide whether to accept the current input observation task. However, the decision content of this method is simple and cannot cope with complex scenarios where there are a large number of observation targets at the same time. Summary of the Invention

[0008] To address the aforementioned problems, this invention proposes a satellite planning method based on multi-agent reinforcement learning. In a multi-agent system, the actions of each agent impact the environment, and each agent's decisions are influenced by the actions of other agents. Agents need to coordinate and learn effective strategies, especially when communication between agents is impossible. Therefore, this multi-agent reinforcement learning method employs a centralized training and distributed execution architecture, enabling each agent to learn a coordinated strategy during training, thereby eliminating the dependence on communication during execution. This not only overcomes the dependence of traditional distributed satellite system collaborative observation on inter-satellite communication capabilities but also overcomes the problem of simplistic decision-making in traditional multi-agent reinforcement learning-based distributed satellite observation planning methods.

[0009] The technical solution of this invention is:

[0010] A distributed planning method for collaborative observation of space targets based on multi-agent reinforcement learning is proposed. This method includes a centralized training and distributed execution architecture based on multi-agent deep reinforcement learning, constructing an on-board Actor network and a ground-based centralized Critic network. Structured hybrid space actions are designed for collaborative observation of space targets, including local observations by the Actor network and global observations by the Critic network, as well as a collaborative observation reward function. The Actor-Critic network is centrally trained through extensive ground simulations using an improved MAHPPO algorithm and curriculum learning technology. The trained Actor network is deployed on various satellites within the constellation executing the collaborative observation mission to achieve distributed execution. After uploading the list of space targets to be observed to each satellite via satellite-to-ground communication, each satellite does not need to communicate with other satellites. Each satellite executes the on-board Actor network to decide the observation targets for each time slice and controls the satellite to carry out the observation mission, thus achieving collaborative observation of space targets.

[0011] The centralized training and distributed execution architecture based on multi-agent deep reinforcement learning includes several shared-parameter Actor networks and a centralized Critic network. Both networks consist of several recurrent neural network layers and fully connected layers. The Actor network, or policy network, takes local observation information from each satellite as input and the turntable direction of each satellite in the current time slice as output. The Critic network, or evaluation network, takes global observation information as input and the quality of each satellite's decision in the current time slice as output.

[0012] Structured hybrid space manipulation transforms the decision about the observed target into a decision about the turntable's pointing direction. Hybrid space refers to dividing the range of turntable pointing values ​​into several intervals. After selecting an interval, a specific pointing angle is chosen within that interval, thus constructing a discrete + continuous hybrid space. Since the turntable is movable on both the pitch and azimuth axes, structured manipulation links the decisions on these two axes, ensuring that the celestial region represented by the selected turntable pointing interval combination necessarily contains a target from the target list to be observed.

[0013] Local observation refers to information available to each satellite solely through satellite-to-ground communication and its own calculations, including the position information of each observed target and the satellite's own status information. Global observation is the synthesis of local observation information from all satellites, including the position information of each observed target, the status information of all satellites, and observation mission completion information.

[0014] The collaborative observation reward function consists of two aspects: traversing low-priority targets individually and observing high-priority targets together. The reward for low-priority targets is only obtained when the constellation completes the observation of the low-priority target for the first time; the reward for high-priority targets is obtained when two or more satellites observe the high-priority target simultaneously, and the longer the joint observation time and the more satellites participating in the observation, the higher the reward.

[0015] The MAHPPO algorithm is suitable for centralized training of Actor and Critic networks in multi-agent hybrid action spaces, and its improvements allow it to adapt to structured actions. Extensive ground simulation training is conducted until the global total reward converges. Using a curriculum learning technique, models trained on smaller constellations are used as pre-trained models to directly train networks on larger constellations.

[0016] In distributed execution, no Critic network is needed. Instead, an Actor network is deployed on each satellite. Each satellite inputs its local observation information into its respective Actor network to obtain its observation plan. Each satellite then mobilizes its turntable to execute the observation task, thus achieving collaborative observation.

[0017] The advantages of this invention compared to the prior art are:

[0018] (1) A multi-agent reinforcement learning architecture based on centralized training and distributed execution is adopted to solve the distributed planning problem of satellite space target collaborative observation, so that the satellites can complete collaborative observation without communication during the on-orbit execution process; and since the spatiotemporal complexity of the on-board Actor network inference is low, it is feasible to deploy it on-orbit, which is conducive to enhancing the autonomy and intelligence of the satellites; the course learning technology is conducive to extending the method to the application of large-scale constellations.

[0019] (2) A structured hybrid space action is designed for the collaborative observation of space targets. By deciding the direction of each satellite turntable in the current time slice, the observation target of each satellite in the current time slice is represented. This ensures that the meaning of the action remains consistent when facing different lists of space targets to be observed, which is conducive to the stability of the decision-making effect. Moreover, unlike the traditional multi-agent reinforcement learning method that only decides whether to accept the current task, this method can greatly increase the flexibility of decision-making and improve the planning effect. At the same time, the structured and hybrid space greatly reduces the space of optional actions for each decision, which is conducive to improving the efficiency of training and the planning effect. Attached Figure Description

[0020] Figure 1 Schematic diagram of structured hybrid space action;

[0021] Figure 2 A schematic diagram of the centralized training architecture for a distributed planning strategy for collaborative observation of space targets;

[0022] Figure 3 A schematic diagram of the distributed execution architecture of a distributed planning strategy for collaborative observation of space targets;

[0023] Figure 4 Schematic diagram of structured action generation;

[0024] Figure 5 A schematic diagram of the structure of a neural network. Detailed Implementation

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this should not be construed as limiting the scope of protection of the present invention.

[0026] 1. Modeling of Distributed Planning Problem for Cooperative Observation of Space Targets

[0027] 1.1 Definition of Planning Problems

[0028] Given an observation network S = {s1, s2, ..., sn} with N observation satellites N}, a set J = {J1, J2, ..., Jn} with M objects to be observed. M}, the attribute J of each observed target m Defined as in Let p be the coordinates of target m in the J2000 system at the start of time slice t. m Prioritize this objective, plan the observation targets of each observation satellite at each time step, and maximize the overall global observation benefits.

[0029] 1.2 Constraints

[0030] The main constraints of this planning problem include:

[0031] 1.2.1 Pointing Constraint. The pointing range of the turntable must be within the maximum range of rotation of the turntable.

[0032] 1.2.2 Exclusivity constraint. At any given time, a satellite can only observe one target.

[0033] 1.3 Assumptions

[0034] 1.3.1 Storage and Energy Adequacy Assumptions. It is assumed that the satellite's storage space and energy will not be depleted during the mission, and that there is no need for data transmission or sun orientation.

[0035] 1.3.2 Homogeneous Agent Assumption. It is assumed that all satellites in this constellation are homogeneous, meaning they have identical functions and attributes. 1.3.3 Communication-Limited Assumption. It is assumed that the satellites in this constellation lack inter-satellite communication capabilities and cannot obtain information about the observation missions of other satellites.

[0036] 1.3.4 Turntable Turning Speed ​​Assumption. It is assumed that when the satellite switches observation tasks, the turntable turning speed is always the maximum turning angular velocity of the turntable.

[0037] 1.3.5 Priority Assumptions. It is assumed that there are only two priority levels: high and low. Low-priority targets only need to be observed by one satellite during the simulation; high-priority targets require observation by at least two satellites simultaneously, and the more satellites observing simultaneously and the longer the observation period, the greater the benefit.

[0038] 1.4 Multi-agent reinforcement learning model

[0039] Mainly observed by intelligent agents intelligent agent actions Global reward r t and the states s used to assess the global value. t composition.

[0040] 1.4.1 Observation of Satellite Intelligent Agents This represents the known input information used by satellite i for decision-making at the start of the current time slice t. Defined as

[0041] The coordinates of each satellite in the constellation and the Sun at the beginning of time slice t are in the J2000 system.

[0042] These are the attributes of the target object at the beginning of time slice t. in Let m be the coordinates of the target m in the J2000 system. p represents the number of remaining visible time slices for the target relative to satellite i. m The priority of target m.

[0043] in and These represent the elevation and azimuth angles of satellite i as pointed at the start of the current time slice t, respectively. This indicates the target number observed by satellite i in the previous time slice. This represents a record of targets observed by satellite i after the previous time slice ended.

[0044] 1.4.2 Actions of Satellite Intelligent Agents This represents the decision action given by satellite i at the start of the current time slice t. This action is a structured hybrid space action. Defined as like Figure 1 As shown, This indicates the interval number selected from the discretized range of pitch angle values; This indicates the interval number selected from the discretized range of azimuth angle values. A real number in the range [-1, 1], representing the angle value further selected from the selected pitch angle range, offset from the starting point of the range; Let be a real number in the range [-1, 1], representing the angle value offset from the starting point of the selected azimuth angle interval. Since the decision-making action contains both discrete and continuous values, it is called a hybrid space action. Structured actions require the agent to select... There is at least one target in the corresponding celestial region, that is, the target m has a remaining visible time slice relative to the current satellite i. and Change to the turntable pointing of the current satellite i Falling within the area.

[0045] Depend on The formula for calculating the direction of the turntable is:

[0046]

[0047]

[0048]

[0049] Among them Pitch min With Azimuth min These represent the minimum pitch angle and minimum azimuth angle of the turntable, respectively. width With Azimuth width These represent the size of each interval of the pitch and azimuth angles, respectively.

[0050] The selected target is the one with the smallest angle between the observed targets and the planned turntable direction.

[0051] 1.4.3 Global Reward r t r represents the sum of the benefits of each satellite's decision in the current time slice t. t Defined as

[0052] in The reward value for the first observed low-priority target j is set to 1. The reward value for the high-priority target i observed in the current time slice t. To determine the number of satellites simultaneously observing high-priority target i, The value is

[0053]

[0054] 1.4.4 States s used to evaluate global value t It is composed of global observations and the local states of each agent, s t Defined as Map t This represents a record of all targets observed by satellites after the previous time slice ended.

[0055] 1.5 Objective Function

[0056] The collaborative observation problem of space targets requires observing high-priority targets simultaneously with multiple satellites, while traversing as many low-priority targets as possible. The objective function of this planning problem is to maximize the total benefit of completing the observation mission:

[0057]

[0058] 2. Centralized Training Distributed Execution Architecture and Improved MAHPPO

[0059] 2.1 Centralized Training Distributed Execution Architecture

[0060] The centralized training distributed execution architecture is based on a decentralized partially observable Markov decision process (DEC-POMDP) ​​with shared rewards. DEC-POMDP is defined by <S, A, O, R, P, n, γ>, where S is the state space, A is the action space shared by each agent, and so on. i =O(s;i) represents the local observation of agent i in the global state s. P(s′|s,A) represents the joint action A = (a1,…,a) of n agents. n Let R(s,A) represent the transition probability from S to S′. Let R(s,A) denote the shared reward function. Let γ represent the reward discount factor. For homogeneous agent problems, shared parameters are used: each agent uses a shared policy π9(a) parameterized by 9 parameters. i|o i Based on its local observations i The action a produces i And optimize its discount accumulation rewards

[0061] By employing a centralized approach during training, and then allowing agents to make distributed decisions based solely on their local observations using a pre-trained policy network after training, this method can, to some extent, overcome the problems of unstable environments and large-scale agents, while also eliminating the need for communication between agents during execution.

[0062] The training process is as follows Figure 2 As shown, in the simulation environment, each agent's Actor network takes its own local observations as input, calculates its own decision action, and stores the local observations, actions, and global reward information of each agent in an experience replay buffer. After a fixed number of calculations, a batch of data is sampled from the experience replay buffer, and the local observations of each agent are combined to generate a state for evaluating the global value, which is then used as input to the Critic network. The Critic network outputs the value evaluation of a given state. Then, the Actor network and the Critic network are updated according to the improved MAHPPO algorithm.

[0063] To extend this method to large-scale constellation applications, a course learning technique can be used during training. When training small constellations, the input size is reserved to match that of large-scale constellations, with unused portions represented by 0. Then, when training large-scale constellations, the pre-trained small constellation model can be directly used as a pre-trained model, and training can continue on top of it, thereby accelerating the convergence of the large-scale constellation model.

[0064] Execution process as follows Figure 3 As shown, only the Actor network is needed during execution, without a Critic network. Each satellite uses the local observation data required by the Actor network as input, and the decision results of each satellite are obtained through the calculation of the Actor network. Since a global Critic network is used during training, the agents do not need to communicate with each other during execution, and the effect of collaborative observation can still be achieved.

[0065] 2.2 Improved MAHPPO

[0066] The MAHPPO algorithm follows the structure of the single-agent HPPO algorithm, where each agent learns a policy π in a hybrid action space. θ and a value function V φ (s). V φ(s) is used to reduce variance and is only used during training, thus providing additional global information in multi-agent scenarios. Therefore, MAHPPO is suitable for training in centralized training distributed execution architectures.

[0067] MAHPPO follows methods commonly used in PPO implementations, including: Generalized Advantage Estimation (GAE) with advantage normalization, observation data normalization, gradient clipping, value clipping, layer normalization, and ReLU activation function with orthogonal initialization.

[0068] MAHPPO was further improved to meet the requirements of structured actions. To generate two pointing intervals simultaneously, two decision heads were used to select the elevation and azimuth intervals respectively. To ensure that the celestial region represented by the combination of these two intervals contains at least one target, a method was adopted as follows... Figure 4 The proposed scheme is as follows: First, consider the pitch angle dimension. Based on the characteristics of each interval calculated by the pitch angle decision head, set the feature value of the interval without a target to MIN. Then, use the Softmax function to obtain the selection probability of each interval with a target. For each pitch angle interval with a target, perform the following calculations: Assuming the pitch angle interval has been selected, based on the characteristics of each interval calculated by the azimuth angle decision head, set the feature value of the azimuth angle interval without a target to MIN. Use the Softmax function to obtain the selection probability of each azimuth angle interval with a target, and temporarily store the results. Based on the pitch angle interval obtained from the final sampling, call the corresponding temporarily stored results and sample again to obtain the azimuth angle interval. At this point, the celestial region represented by the combination of these two intervals contains at least one target.

[0069] 3. Neural Network Structure

[0070] Neural network structure such as Figure 5 As shown. The entire network structure contains two independent neural networks: a satellite-based distributed Actor network and a ground-based centralized Critic network.

[0071] 3.1 On-board Distributed Actor Network Due to the need for both discrete and continuous outputs in the hybrid action space, the Actor network has two parallel sub-networks. Each sub-network uses a multi-layer recurrent neural network to extract features, consisting of four recurrent neural network hidden layers with 256 neurons each. For discrete actions, two decision heads are used, and the extracted features are fed to these two decision heads, specifically a fully connected layer called the Discrete Policy. This layer has 256 neurons and uses the Softmax activation function, outputting discrete values ​​after structured processing. For continuous actions, the extracted features are fed to a fully connected layer called the Parameter Policy. This layer has 256 neurons and uses the Tanh activation function, thus outputting continuous values.

[0072] The network input is a local observation of a satellite, expanded into a vector.

[0073] The network output represents the decision for the current time slice of the satellite. The output scale of both discrete strategies is 1, representing the selected elevation and azimuth intervals, respectively. The continuous strategy outputs continuous values ​​in the range [-1, 1], with an output scale of 2, representing the offset within the elevation and azimuth intervals, respectively.

[0074] 3.2 Ground-based centralized Critic network

[0075] The Critic network, like the Actor network, uses a multi-layer recurrent neural network of the same specifications to extract features. The features are fed into a fully connected layer called the Value layer. This layer has 256 neurons and uses the ReLU activation function. The network's input is the state s, unfolded into a vector, used to evaluate the global value. t .

[0076] The network output is a value assessment of the input state, with an output size of 1. This value is used to train the Actor network.

Claims

1. A distributed planning method for collaborative observation of satellite space targets based on multi-agent reinforcement learning, characterized in that, The method includes: Step 1) Construct an on-board Actor network and a ground-based centralized Critic network for each satellite in the satellite space target collaborative observation network to form a centralized training and distributed execution architecture; Step 2) Design structured hybrid spatial actions, including local observations from the Actor network and global observations from the Critic network, as well as a collaborative observation reward function; Step 3) The Actor-Critic network is trained intensively through extensive ground simulations and using the improved MAHPPO algorithm and curriculum learning techniques. Step 4) Deploy the trained Actor network on each satellite of the constellation that will perform the collaborative observation mission to achieve distributed execution; Step 5) After uploading the list of space targets to be observed to each satellite via satellite-to-ground communication, each satellite does not need to communicate with other satellites. Each satellite executes the on-board Actor network to decide the observation targets for each time slice and controls the satellite to carry out the observation tasks, thereby realizing the collaborative observation task of space targets. The structured hybrid space action transforms the decision on the observed target into a decision on the turntable direction. The hybrid space refers to dividing the range of turntable direction values ​​into several intervals. After selecting an interval, a specific direction angle is further selected within that interval, thereby constructing a discrete + continuous hybrid space. The structured hybrid spatial action refers to linking the decisions of the turntable on the pitch and azimuth axes so that the selected turntable pointing interval combination must contain targets from the target list to be observed in the celestial sphere region. The aforementioned local observation refers to the information that each satellite can obtain solely through satellite-to-ground communication and its own calculations, including the position information of each target to be observed and the satellite's own status information; The global observation mentioned above is the integration of local observation information from all satellites, including the position information of each target to be observed, the status information of all satellites, and the observation task completion information; The collaborative observation reward function consists of two aspects: traversing low-priority targets separately and observing high-priority targets together. The reward for low-priority targets is obtained when the constellation completes the observation of the low-priority target for the first time. The reward for high-priority targets is obtained when two or more satellites observe the high-priority target at the same time. The longer the joint observation time and the more satellites participating in the observation, the higher the reward. The improved MAHPPO algorithm uses two decision heads to select the pitch angle interval and the azimuth angle interval respectively, and ensures that the celestial sphere region represented by the combination of these two intervals contains at least one target. The aforementioned learning technique involves using a pre-trained model of a smaller constellation as a pre-trained model to directly train a network of a larger constellation.

2. The distributed planning method for collaborative observation of satellite space targets based on multi-agent reinforcement learning according to claim 1, characterized in that, The centralized training distributed execution architecture includes several shared parameter Actor networks and a centralized Critic network. Both networks consist of several recurrent neural network layers and fully connected layers. The Actor network is the policy network, which takes local observation information of each satellite as input and the turntable direction of each satellite in the current time slice as output. The Critic network is the evaluation network, which takes global observation information as input and the quality of each satellite's decision in the current time slice as output.

3. The distributed planning method for collaborative observation of satellite space targets based on multi-agent reinforcement learning according to claim 1 or 2, characterized in that, The multi-agent reinforcement learning model includes: Satellite intelligent agent observation This represents the known input information used by satellite i for decision-making at the start of the current time slice t; Satellite Structured Hybrid Space Actions This represents the decision action given by satellite i at the start of the current time slice t; Global reward r t , representing the sum of the benefits of each satellite's decisions in the current time slice t; and The state used to assess the overall value. t It is composed of global observations and the local states of each agent.

4. The distributed planning method for collaborative observation of satellite space targets based on multi-agent reinforcement learning according to claim 3, characterized in that, The aforementioned satellite intelligent agent observation in, The coordinates of each satellite in the constellation and the Sun at the beginning of time slice t are in the J2000 system. These are the attributes of the target object at the beginning of time slice t. Let m be the coordinates of the target m in the J2000 system. p represents the number of remaining visible time slices for the target relative to satellite i. m The priority of target m.

5. The distributed planning method for collaborative observation of satellite space targets based on multi-agent reinforcement learning according to claim 4, characterized in that, The structured hybrid space actions of the satellite intelligent agent in, This indicates the interval number selected from the discretized range of pitch angle values. This indicates the interval number selected from the discretized range of azimuth angle values. A real number in the range [-1, 1] represents a further selected angle value offset from the starting point of the selected pitch angle range. A real number in the range [-1, 1] represents the angle value further selected from the selected azimuth interval, offset from the starting point of the interval; There is at least one target in the corresponding celestial region, that is, the target m has a remaining visible time slice relative to the current satellite i. and Change to the turntable pointing of the current satellite i Falling Within the region, and These represent the elevation and azimuth angles of satellite i as pointed at the start of the current time slice t, respectively. These represent the elevation and azimuth angles of satellite i as it points towards target m at the start of the current time slice t.

6. The distributed planning method for collaborative observation of satellite space targets based on multi-agent reinforcement learning according to claim 3, characterized in that, The states s used to evaluate the global value t , Map t This represents a record of all targets observed by satellites after the previous time slice ended.

Images