Distributed decision-making method based on multi-agent deep reinforcement learning

Abstract

The application discloses a kind of distributed decision-making methods based on multi-agent deep reinforcement learning, constructs the integrated scheduling problem model of satellite observation and data download based on decentralized partially observable Markov decision process, each satellite is regarded as intelligent agent with autonomous decision-making ability, each intelligent agent can make autonomous decision on observation task, determine the observation time of ground target and the download time of observation data, so that the total income in scheduling period is maximum, in online phase, the decision result of satellite cluster is obtained by satellite cluster scheduling network trained in real time according to observation task data, whether each satellite distributed decision-making each observation task is executed, execution time, the download time of observation data and download ground station are realized.The application can dynamically carry out satellite observation data download while carrying out satellite observation task planning, and significantly improve the observation efficiency of satellite cluster.

Description

Technical Field

[0001] This invention relates to a technology in the field of satellite cluster collaborative mission planning, specifically a distributed decision-making method for satellite cluster observation and data downlink based on multi-agent deep reinforcement learning. Background Technology

[0002] With the rapid development of aerospace technology, the number of satellites in orbit is constantly increasing. Earth Observation Satellites (EOS), as an important remote sensing data acquisition platform, can observe ground targets in their orbits using imaging payloads and transmit the acquired observation data to ground stations to generate image data products that meet various applications. As Earth observation needs become increasingly diverse and complex, how to more efficiently schedule satellite resources has become an urgent problem to be solved. EOS mission scheduling mainly refers to allocating satellite resources according to observation tasks and determining the tasks to be completed within the scheduling cycle, the order of task completion, and the completion time of each task based on specific optimization objectives. A complete mission process typically includes observation of the target and transmission of observation data. Summary of the Invention

[0003] This invention addresses the shortcomings of existing satellite resource scheduling technologies, which do not consider the data downlink process and user remote sensing image requirements. For integrated satellite-ground scheduling scenarios, it proposes a distributed decision-making method based on multi-agent deep reinforcement learning. This method treats each satellite as an autonomous agent capable of making independent decisions. Each agent can autonomously decide on observation tasks, determining the observation time for ground targets and the downlink time for observation data, thereby maximizing the total benefit within the scheduling cycle. This approach enables dynamic satellite observation data downlinking while simultaneously planning satellite observation tasks, significantly improving the observation efficiency of satellite constellations.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a distributed decision-making method for satellite cluster observation and data downlink based on multi-agent deep reinforcement learning. It constructs an integrated scheduling problem model for satellite observation and data downlink based on a decentralized, partially observable Markov decision process. In the offline phase, the integrated scheduling model is simulated using the action value network of each satellite in the satellite cluster and a hybrid network of the satellite cluster. Based on the observation tasks and information about the satellite cluster and ground stations, the simulation is performed to obtain the visible time windows of the satellites to the ground observation targets and the communicable time windows of the satellites to the ground stations as training samples to train the satellite cluster scheduling network. In the online phase, the trained satellite cluster scheduling network obtains the decision results of the satellite cluster in real time based on the observation task data, realizing distributed decision-making by each satellite regarding whether to execute each observation task, the execution time, the downlink time of the observation data, and the downlink to the ground station.

[0006] The action value network comprises: an input layer, two multilayer perceptrons (MLPs), and a gated recurrent neural network (GRU), wherein: the input layer is based on satellite local observation vectors. and action information from the previous time window The input vector is obtained by concatenation. The first multilayer perceptron integrates the features of the input vector. The gated recurrent neural unit processes historical information based on the feature information. The second multilayer perceptron obtains the action value function value based on the historical information.

[0007] The local observation vector Including the time window data ω corresponding to the current moment y state vector and the satellite's state vector i is the serial number of the observed satellite. For ω t The corresponding observation task, and For the time window ω t The start and end times, These are the minimum observation time, required on-board storage, observation benefits, and remote sensing image type for the current mission. and Satellite S respectively i Remaining and maximum storage of onboard memory. For satellite S i The imaging payload is encoded using a single-heat encoding method.

[0008] All fully connected layers have a dimension of 64.

[0009] The recurrent neural network layer is used to process historical sequence data.

[0010] The actions include: selecting the current window for observation, selecting the current window for downloading, and not selecting the current time window; the action value function value is... Where: τ i For satellite S i Historical observation of motion The environmental feedback reward, where γ is the discount factor for the reward, l is the total number of time windows, and E is the mathematical expectation, is used to measure the long-term benefits of the satellite's actions at the current time step.

[0011] The satellite constellation hybrid network comprises an input layer, two supernetworks, and an output layer. The input layer performs monotonically mixed processing based on the action value function value output by the action value network. The supernetwork layer processes the data based on the global state information of the environment. The weights and biases of the hybrid network are obtained, and the output layer obtains the joint action value based on the information from the supernetwork and the input layer.

[0012] The global state information of the environment Where: p is the number of ground stations, and m is the number of observation missions. One-hot encoding of the frequency bands for the data transmission antennas of p ground stations. The completion status of m tasks.

[0013] The value of the joint action in: Let τ represent the joint actions of the satellite swarm, and let τ be the observation history of the joint actions of the satellite swarm. This formula is used to measure the long-term benefits of the joint actions of the satellite swarm at the current time step.

[0014] The training mentioned refers to: initializing the parameters θ and θ' of the action value network and the satellite cluster hybrid network; and the training of n satellites for a time window ω. t The actions taken constitute a combined action. It acts on the environment and receives rewards from the environment in return. Save tuples To memorize and replay a library D and randomly sample b_s data points from it, a network is trained based on a loss function. Where: b_s is the batch size, y tot =r+γmaxQ tot (τ′,u′,s′;θ - ) represents the output value of the target value network, which has the same structure as the action value network, r represents the immediate reward of the environment, γ∈[0,1] represents the discount factor of the reward, and θ represents the output value of the target value network. - The target value network parameters.

[0015] Technical effect

[0016] This invention dynamically plans satellite observation data downlink while simultaneously planning satellite observation missions, effectively solving the integrated scheduling problem of satellite observation and mission downlink. Furthermore, as a distributed decision-making method, it is a space-ground integrated resource joint scheduling technology that effectively improves the observation efficiency of satellite constellations and the management efficiency of ground station systems. Attached Figure Description

[0017] Figure 1 This is a schematic diagram illustrating satellite observation and data downlink as an example.

[0018] Figure 2 This is a time-sharing diagram of the satellite downlink process in an example embodiment;

[0019] Figure 3 Here is a diagram of the satellite cluster decision network architecture for an example.

[0020] Figure 4 The following is a diagram illustrating the value network and hybrid network structures in an embodiment.

[0021] Figure 5 This is a flowchart of the network training process for an example. Detailed Implementation

[0022] This embodiment relates to a distributed decision-making method for satellite cluster observation and data downlink based on multi-agent deep reinforcement learning, including the following steps:

[0023] Step 1, construct as follows Figure 1 The integrated scheduling model for satellite observation and data downlink illustrated is as follows: Based on a certain number of observation tasks within a scheduling period, the scheduling system determines the executing satellite and execution time for each task according to the current satellite and task status. Simultaneously, it dynamically generates downlink tasks and, based on the status of each satellite and ground station, downlinks the observation data stored on the satellite's memory to the ground station. Here, satellite observation and data downlink are interleaved; task scheduling includes both observation tasks and dynamically generated downlink tasks. Completing different tasks will bring different benefits, and the scheduling objective is to maximize the total benefit of tasks completed within the scheduling period.

[0024] Based on the integrated scheduling problem model of satellite observation and data downlink, it is modeled as a decentralized partially observable Markov decision process model. In this decentralized partially observable Markov decision process, each satellite is considered an intelligent agent with autonomous decision-making capabilities, using a plural group...<S,U,P,r,Z,O,n,γ> Let U be the number of agents in the system, i.e., the number of satellites; s∈S, representing the state of the satellite system environment, including the satellite cluster, observation missions, and ground station states; and U, representing the action space for each agent when making decisions, including data downlink, Earth observation, and not selecting the current time window. The actions of all agents in the system constitute a joint action u∈U≡U. n P is the transition function of the system environment state, that is, the probability of transitioning from one state to another, P(s′|s,u):S×U×S→[0,1]; r is the reward function of all agents in the system, that is, the feedback signal of the environment to the joint actions taken by all agents, used to evaluate the merits of the system in making a certain decision in a certain state; Z is the local observation of each agent, that is, the information that the satellite can observe in its own environment; γ is the reward discount coefficient, used to balance short-term and long-term benefits.

[0025] The decentralized partially observable Markov decision process described above is solved using the QMIX multi-agent deep reinforcement learning algorithm, which is based on centralized training and distributed execution. Centralized training means that global information from the environment is used during network training, enabling satellites in the constellation to learn to cooperate with each other. Distributed execution means that for the trained network, each satellite only uses its own locally observed information during actual decision-making, thus achieving cooperation without the need for inter-satellite communication. This not only better aligns with practical engineering scenarios but also reduces communication overhead between satellites.

[0026] In the integrated scheduling problem model described above: all observation tasks can be decomposed into point imaging targets that can be completed in a single operation; all satellites have sufficient energy, and their recharging process with the sun is not considered; the scanning range of the imaging payloads of all satellites is equal; the time spent by satellites adjusting their attitude and restarting sensors when performing adjacent tasks is considered constant; only time constraints and frequency band constraints of the data transmission antenna are considered during data downlink; when satellites and ground stations perform data downlink, the preparation time, acquisition time, and release time spent establishing a communication link are all considered constant, such as... Figure 2 As shown; when the satellite transmits data, it only considers the storage and forwarding form, and does not consider the actual shooting and transmission; the speed of satellite observation data is constant during storage and transmission, that is, the amount of data stored and transmitted is directly proportional to the observation time and transmission time, respectively.

[0027] In the integrated scheduling problem model described above: the scheduling time period [SS, SE], where SS is the start time of the plan and SE is the end time of the plan. Observation task set. m represents the number of observation tasks. in: A unique identifier for the task. The geographical location information of the target point is described using longitude and latitude. To minimize the continuous observation time for the mission, The on-board storage space required for mission observation data. The reward for completing this observation task and This refers to the type of remote sensing image (e.g., panchromatic image, hyperspectral image, multispectral image, etc.) and its spatial resolution. The satellite constellation Sat = {s1, ..., s2} n}, where n is the number of observation satellites. in: A unique identifier for the satellite. and This represents the maximum and remaining storage capacity of the satellite's onboard memory. This refers to the transfer time required for the satellite to perform adjacent tasks. and This consists of the satellite's payload set used for imaging and the antenna set used for data transmission. The main parameters of the former are imaging type and spatial resolution, while the main parameter of the latter is frequency band. The ground station set G = {g1, ..., g...} p}, where p is the number of ground stations. in: A unique identifier for the ground station. These are the location parameters of the ground station, also described using longitude and latitude. This is the set of data transmission antennas for the ground station, whose main parameter is the frequency band. The time window set W = {ω1, ..., ω...} is visible. l Let} represent the number of visible time windows for all observation missions' target points and ground stations within the satellite constellation. These visible time windows include both the visible time windows for the target points submitted by the satellite constellation for the observation missions and the communicable time windows for all ground stations; for ease of description, they are collectively referred to as visible time windows. The visible time windows primarily depend on the satellite constellation's orbital parameters and the positional parameters of the target points and ground stations within the scheduling period. in: A unique identifier for the time window. The identifier for the satellite corresponding to this time window. This refers to the symbol or identifier of the ground station for the observation target corresponding to that time window. and Let R be the start and end times of this time window. The scheduling result set R = {r1, ..., r...} q}, where q represents the number of times the satellite constellation performs observations or downloads. in: A unique identifier for the scheduling result. This is the identifier for the time window corresponding to the result. and This refers to the start time, end time, and duration of the observation or download corresponding to this result.

[0028] The constraints of the integrated scheduling problem model include:

[0029] 1. Observation Demand Constraints: The primary goal of satellite cluster scheduling is to fulfill observation requirements, which are mainly constrained by the type of remote sensing image and its spatial resolution. If satellite s i Select to perform observation tasks The corresponding imaging payload is The constraints that the remote sensing image type and spatial resolution must meet are: in: For satellites s i Spatial resolution of the corresponding type of imaging payload. The first item indicates that the satellite must include the corresponding type of imaging payload, while the second item indicates that the spatial resolution of the corresponding type of imaging payload on the satellite should be lower than the spatial resolution required for observation.

[0030] 2. Observation time constraint: When satellite s i Continuous observation mission as well as The corresponding scheduling results are r k as well as The corresponding visible time windows are ω p and The constraints to be satisfied are: This means ensuring the timely execution of observation tasks, executing them as early as possible within the allotted execution time. For satellites s i The shortest transfer time required to execute consecutive tasks; That is, the continuous observation time of the target point needs to meet the requirement of the shortest observation time; That is, the end time of the observation needs to meet the constraint of the visible time window.

[0031] 3. Onboard memory constraints: The satellite's onboard memory resources are limited; therefore, the satellite cannot conduct observations without restriction before transmitting data. i Select to perform observation tasks The corresponding scheduling result is r k Its storage constraint is: This means that before a satellite can perform a mission, it needs sufficient onboard storage space to store the image data of the mission to be observed. Where: v u The storage speed of image data is positively correlated with the amount of data generated by the observation and the duration of the observation.

[0032] 4. Data downlink constraints: When a satellite enters the communication range with a ground station, in addition to meeting the following requirements... Figure 2 The time constraints shown also require ensuring that the data transmission antenna and the ground station's data transmission antenna are in the same frequency band. When satellite s i With ground station g j When establishing a communication link for data downlink, the corresponding scheduling result is r. k The time window is ω p The scheduling result satisfies: Where: t pre The preparation time required for the ground station to transmit data, t cap t is the acquisition time required for the ground station to establish a communication link with the satellite. rel This is the release time required for the ground station after it finishes receiving data transmitted from the satellite, such as... Figure 2 As shown; v represents the amount of data transmitted by the satellite within the current time window. d The data transmission speed during the data downlink process is positively correlated with the amount of data transmitted by the satellite in a single transmission.

[0033] The objective function of the integrated scheduling problem model is as follows: For the integrated scheduling problem of satellite observation and data downlink, the goal of scheduling is to complete as many observation and downlink tasks as possible and maximize the total scheduling benefit.

[0034] in: and Tasks Whether observation and transmission have been achieved is indicated by a value of 1 (yes) and 0 (no); k i It is used to adjust tasks. A coefficient indicating the urgency of the download.

[0035] Step 2, construct as follows Figure 4 The diagram shows the action value network for each satellite and the hybrid network for the satellite constellation, where: the value network is based on satellite observations. And the actions taken in the previous time window The Q-values ​​of each satellite action are output, and the final decision action is randomly output via ∈-greedy; the satellite cluster hybrid network outputs the Q-values ​​of each satellite action based on environmental information. The combined action value is output along with the Q-value of each action.

[0036] The action value network comprises: an input layer, two multilayer perceptrons (MLPs), and a gated recurrent neural network (GRU), wherein: the input layer is based on satellite local observation vectors. and action information from the previous time window The input vector is obtained by concatenation. The first multilayer perceptron integrates the features of the input vector. The gated recurrent neural unit processes historical information based on the feature information. The second multilayer perceptron obtains the action value function value based on the historical information.

[0037] All the multilayer perceptrons described are implemented using fully connected layers with a dimension of 64.

[0038] The environmental information mentioned For the environment during the processing time window ω t The status at any given time includes the current mission completion status, the satellite status, the ground station status, and the status of the corresponding mission.

[0039] The Q values ​​for each satellite action are as follows: This is used to measure the long-term return of a decision made at the current moment, where E is the expected value and τ is the satellite's historical observation data. The value network considers not only the immediate return of the decision but also its long-term returns.

[0040] The value of the aforementioned joint action is specifically as follows: This is used to measure the long-term return of joint decision-making across all current satellites, where τ is the joint action observation history. This refers to the coordinated actions of a satellite constellation. The satellites within a constellation cooperate, and the quality of individual decisions is always positively correlated with the quality of the joint decision. Hybrid networks are used to fit the contribution of individual decisions to the joint decision.

[0041] The term ∈-greedy refers to randomly selecting an action with a probability of ∈, otherwise selecting the action with the largest Q value as the decision for this time.

[0042] The satellite constellation hybrid network comprises an input layer, two supernetworks, and an output layer. The input layer performs monotonically mixed processing based on the action value function value output by the action value network. The supernetwork layer processes the data based on the global state information of the environment. The weights and biases of the hybrid network are obtained, and the output layer obtains the joint action value based on the information from the supernetwork and the input layer. During the training phase, the overall environmental information is transmitted to the hybrid network through the supernetwork.

[0043] Step 3, as follows Figure 5 As shown, the action value network of each satellite and the satellite constellation hybrid network are trained based on historical information generated by the interaction between the satellite constellation and the environment during decision-making. Specifically, this includes:

[0044] 3.1) Initialize training parameters: scheduling network parameters θ and corresponding target value network parameters θ′, number of training rounds epi_max, experience replay unit and its size D, update frequency of target value network k, batch size bs;

[0045] The network structure of the target value network is the same as that of the action value network of each satellite.

[0046] 3.2) Initialize the environment and read the parameter information of the observation mission, satellite, and ground station;

[0047] 3.3) In each round of training, each visible time window is traversed sequentially to obtain the observations of all satellite agents, the state of the environment, and the available actions for the current time window, and then the current decision action is selected in the ∈-greedy manner;

[0048] 3.4) All satellites execute the actions of the current time window and store the information of the current time step, along with the obtained environmental feedback reward, in the experience replay unit as historical information; when the number of data entries stored in the experience replay unit exceeds the batch size b_s, b_s data entries are randomly sampled from it, and the loss function is calculated. And update the network parameters θ using gradient descent;

[0049] The historical information includes: observation information at each time step, environmental state, available actions, actual actions taken, and reward information from environmental feedback.

[0050] 3.5) Every k steps, copy the value network parameters to the target value network;

[0051] 3.6) Repeat the above steps until the specified number of training iterations are completed.

[0052] Step 4: In the online phase, based on the trained scheduling network, the observation task and visible time window data are input to obtain the distributed strategy for each satellite, including the task received by each satellite, the selected visible time window, the observation time, and the data downlink time and ground station.

[0053] During the execution phase, only the value network parameters of each satellite and the state of each satellite are needed, without the need for global information and mixed network parameters, thus the strategy is obtained in a distributed manner.

[0054] like Figure 3As shown, this invention relates to a distributed decision-making system based on multi-agent deep reinforcement learning for implementing the above-mentioned method, comprising: a data storage unit, a data input unit, an agent unit, a network training unit, and a decision generation unit, wherein: the data storage unit is used to store data of each satellite in the satellite cluster of the satellite ground system and data of each ground station in the ground station system; the data input unit performs preprocessing based on the observation task information submitted by the user to obtain task data conforming to the input parameter format; the agent unit initializes each satellite agent and its corresponding action value network based on the satellite and ground station information involved in this planning; the network training unit trains the action value network of each satellite and the hybrid network of the satellite cluster based on the observation task, satellite, and ground station data to obtain the trained network parameters; the decision generation unit obtains the distributed decision results of each satellite, including the task received by each satellite, the selected visible time window, the observation time, the data downlink time, and the ground station, based on the trained action value network of each satellite and the observation task information.

[0055] Based on specific practical experiments, this embodiment establishes a simulation dataset containing the following Earth observation elements: the number of observation tasks is set to 100, 200, and 500; observation gains are randomly generated from 1 to 10; the shortest observation time is 5 to 10 seconds; storage consumption is 4 to 8 Gb; and the image type is panchromatic, hyperspectral, or multispectral. The required spatial resolution for panchromatic images is 2 to 10 meters, while the required spatial resolution for spectral and multispectral images is 100 to 200 meters. The number of satellites is set to two types: 3 and 6; the onboard storage is set to 130 to 170 Gb; the imaging payload includes a panchromatic camera, a multispectral camera, and a hyperspectral camera, with spatial resolution parameters set to 2 to 120 meters; the number of data transmission antenna frequency bands is 2 to 4; the number of ground stations is set to 5; and the number of data transmission antenna frequency bands is set to 2 to 4. The visible time window between the satellite and the ground target is 10 to 15 seconds, and the communication time window between the satellite and the ground station is 8 to 15 minutes.

[0056] The evaluation metrics used are total scheduling benefit, observation benefit, data downlink benefit, and the number of tasks completed. Total scheduling benefit and the number of tasks completed are used to evaluate the overall performance of the algorithm, while observation benefit and data downlink benefit are used to evaluate the scheduling performance of ground observation and observation data downlink, respectively. The comparison algorithms are the classic intelligent optimization algorithms, genetic algorithm and simulated annealing algorithm. The Earth observation satellite scheduling scenarios are set as 3 satellites with 100 tasks, 3 satellites with 200 tasks, and 6 satellites with 500 tasks.

[0057] The results of different algorithms for different scheduling scenarios are shown in Table 1:

[0058] Table 1 Comparison between the present invention and existing methods

[0059]

[0060] Compared with existing technologies, this method outperforms the comparative methods in almost all aspects, including total scheduling benefit, observation benefit, data downlink benefit, and the number of tasks completed. Furthermore, the advantages of this invention become increasingly apparent compared to traditional intelligent optimization algorithms as the scale of satellite scheduling tasks increases. The results in a 3-satellite, 200-task scenario show that although this invention completes fewer tasks than the simulated annealing algorithm, its observation benefit and total scheduling benefit are higher. This indicates that this invention tends to accept observation tasks with higher benefits, and the average benefit of all completed tasks is higher, suggesting that higher-priority (higher-benefit) observation tasks have a greater chance of being completed. This has significant practical value in satellite scheduling applications, addressing the problem of traditional intelligent optimization algorithms easily getting trapped in local optima when dealing with large-scale scheduling problems. It is well-suited to the scheduling problem of satellite cluster collaborative Earth observation in the context of the continuously increasing number of satellites in orbit.

[0061] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A distributed decision-making method for satellite cluster observation and data downlink based on multi-agent deep reinforcement learning, characterized in that, An integrated scheduling problem model for satellite observation and data downlink based on a decentralized partially observable Markov decision process is constructed. In the offline stage, the integrated scheduling problem model is simulated through the action value network of each satellite in the satellite cluster and the hybrid network of the satellite cluster. Based on the observation task and the information of the satellite cluster and ground station, the observation task and the satellite cluster and ground station are simulated. The visible time window of the satellite to the ground observation target and the communicable time window of the satellite to the ground station are obtained as training samples to train the action value network of each satellite and the hybrid network of the satellite cluster. During the online phase, the action value network of each trained satellite and the satellite cluster hybrid network are used to obtain the decision results of the satellite cluster in real time based on the observation task data, so as to realize the distributed decision-making of each satellite on whether to execute each observation task, the execution time, the downlink time of the observation data, and the downlink to the ground station; The satellite constellation hybrid network comprises an input layer, two supernetworks, and an output layer. The input layer performs monotonically mixed processing based on the action value function value output by the action value network. The supernetwork layer processes the data based on the global state information of the environment. The weights and biases of the hybrid network are obtained, and the output layer obtains the joint action value based on the information from the supernetwork and the input layer. The action value network comprises: an input layer, two multilayer perceptrons (MLPs), and a gated recurrent neural network (GRU), wherein: the input layer is based on satellite local observation vectors. and action information from the previous time window The input vector is obtained by concatenating the input vector information. The first multilayer perceptron integrates the features of the input vector information. The gated recurrent neural unit processes historical information based on the feature information. The second multilayer perceptron obtains the action value function value based on the historical information.

2. The distributed decision-making method for satellite cluster observation and data downlink based on multi-agent deep reinforcement learning as described in claim 1, characterized in that, The local observation vector Including the time window data corresponding to the current moment. state vector ( , ) and S i state vector ( , ); i is the serial number of the observed satellite, for The corresponding observation task, and For time window The start and end times, , , , These are the minimum observation time, required on-board storage, observation benefits, and remote sensing image type for the current mission. and Satellite S i Remaining and maximum storage of onboard memory. For satellite S i One-hot encoding of the imaging payload; The actions include: selecting the current window for observation, selecting the current window for downloading, and not selecting the current time window; the action value function value is... ,in: For satellite S i Historical observation of motion Rewards for environmental feedback, Here, l is the discount factor for the reward, l is the total number of time windows, and E is the expected value. This formula is used to measure the long-term benefits of the satellite's actions at the current time step.

3. The distributed decision-making method for satellite cluster observation and data downlink based on multi-agent deep reinforcement learning as described in claim 1, characterized in that, The global state information of the environment Where: p is the number of ground stations, and m is the number of observation missions. , ..., One-hot encoding of the frequency bands for the data transmission antennas of p ground stations. , ..., The completion status of m tasks; The value of the joint action ,in: For the coordinated operation of the satellite constellation, This formula represents the observation history of the joint actions of the satellite constellation and is used to measure the long-term benefits of the joint actions of the satellite constellation at the current time step.

4. The distributed decision-making method for satellite cluster observation and data downlink based on multi-agent deep reinforcement learning as described in claim 1, characterized in that, The training mentioned refers to: initializing the parameters θ and θ' of the action value network and the satellite constellation hybrid network; and the time window for n satellites. The actions taken constitute a combined action. It acts on the environment and receives rewards from the environment in return. Save tuples To memorize and replay a library D and randomly sample b_s data points from it, a network is trained based on a loss function. Where: b_s is the batch size, The output value of the target value network is given. The target value network has the same structure as the action value network described above. r represents the immediate reward from the environment. The discount factor for the reward. The target value network parameters.

5. The distributed decision-making method for satellite cluster observation and data downlink based on multi-agent deep reinforcement learning according to claim 4, characterized in that, The aforementioned integrated scheduling model for satellite observation and data downlink based on a decentralized, partially observable Markov decision process refers to the following: Based on a certain number of observation tasks within a scheduling period, the scheduling system determines the executing satellite and execution time of each task according to the current satellite and task status. Simultaneously, it dynamically generates downlink tasks and then downlinks the observation data from the onboard storage to the ground station based on the status of each satellite and the ground station. Here, satellite observation and data downlink are interleaved, and task scheduling includes both observation tasks and dynamically generated downlink tasks. Completing different tasks will bring different benefits, and the scheduling objective is to maximize the total benefit of tasks completed within the scheduling period. In the decentralized, partially observable Markov decision-making process, each satellite is considered an intelligent agent with autonomous decision-making capabilities, using a plural group... It means that, among them: This represents the number of intelligent agents in the system, i.e., the number of satellites. The status of the satellite system environment, including the satellite constellation, observation missions, and ground stations; The action space for each agent when making decisions includes data download, ground observation, and not selecting the current time window. The actions of each agent in the system constitute a joint action. P is the transition function of the system's environmental state, that is, the probability of transitioning from one state to another. ; r is the reward function for all agents in the system, which is the feedback signal of the environment to the joint actions taken by all agents, used to evaluate the merits of the system's decision in a certain state; Z is the local observation of each agent, which is the information that the satellite can observe in its own environment; This is the discount factor for rewards, used to balance short-term and long-term gains.

6. The distributed decision-making method for satellite cluster observation and data downlink based on multi-agent deep reinforcement learning according to claim 5, characterized in that, The training, namely the training of the action value network of each satellite and the satellite cluster hybrid network based on historical information generated by the interaction between the satellite cluster and the environment during decision-making, specifically includes: 3.1) Initialize training parameters: Schedule network parameters and the corresponding target value network parameters The number of training epochs, epi_max, experience replay units and their sizes, the update frequency k of the target value network, and the batch size b_s; The network structure of the target value network is the same as the action value network of each satellite. 3.2) Initialize the environment and read the parameter information of the observation mission, satellite, and ground station; 3.3) In each training round, iterate through each visible time window, acquiring all satellite agents' observations, the state of the environment, and the available actions for the current time window, and then... The -greedy method is used to select the current decision action; 3.4) All satellites execute the actions of the current time window and store the information of the current time step, along with the obtained environmental feedback reward, in the experience replay unit as historical information; when the number of data entries stored in the experience replay unit exceeds the batch size b_s, b_s data entries are randomly sampled from it, and the loss function is calculated. And update the network parameters using gradient descent. ; The historical information includes: observation information at each time step, environmental state, available actions, actual actions taken, and reward information from environmental feedback; 3.5) Every k steps, copy the value network parameters to the target value network; 3.6) Repeat the above steps until the specified number of training iterations are completed.

7. A distributed decision-making system based on multi-agent deep reinforcement learning that implements the method of any one of claims 1-6, characterized in that, include: The system comprises a data storage unit, a data input unit, an agent unit, a network training unit, and a decision generation unit. Specifically: the data storage unit stores data from each satellite in the satellite constellation of the satellite ground system and data from each ground station in the ground station system; the data input unit preprocesses the observation task information submitted by the user to obtain task data conforming to the input parameter format; the agent unit initializes each satellite agent and its corresponding action value network based on the satellite and ground station information involved in this plan; the network training unit trains the action value networks of each satellite and the hybrid network of the satellite constellation based on the observation tasks, satellite data, and ground station data to obtain the trained network parameters; and the decision generation unit, based on the trained action value networks of each satellite and the observation task information, obtains the distributed decision results for each satellite, including the task received, the selected visibility window, the observation time, the data download time, and the ground station's distributed decision-making process.

Images