Large-scale complex correlation earth observation task planning method based on reinforcement learning

Abstract

The invention discloses a large-scale complex correlation earth observation task planning method based on reinforcement learning, and relates to the technical field of task resource planning. The method comprises the following steps: acquiring historical data of earth observation task planning, and constructing a sample data set; constructing a graph neural network and a reinforcement learning network; using the sample data set to train the graph neural network and the reinforcement learning network at the same time; acquiring a demand information list, a satellite information list, a data transmission task information list and an observation information list, and processing the information lists into demand feature vectors, satellite feature vectors, data transmission task feature vectors and observation feature vectors; and inputting the four feature vectors into a trained graph neural network to obtain an environment feature vector, inputting the environment feature vector and the demand feature vector into a trained reinforcement learning network, and outputting an earth observation task planning result. The method can solve the scheduling optimization problem under the condition of multiple tasks and limited resources, improves the task planning efficiency, and can be widely applied to the fields of satellite task scheduling, intelligent manufacturing, automatic driving and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of task resource planning, and in particular to a large-scale complex correlation earth observation task planning method based on reinforcement learning. BACKGROUND

[0002] Currently, task resource scheduling is widely used in various fields such as satellite task scheduling, intelligent manufacturing, and autonomous driving. In satellite task scheduling, how to reasonably allocate and utilize limited resources, especially in high-dynamic and complex constraint resource allocation, is a challenging problem. Traditional scheduling algorithms, such as heuristic algorithms and integer programming algorithms, can provide solutions for certain specific task scheduling problems, but they have the following shortcomings in dealing with dynamic changes, complex correlations, and multi-constraint task scheduling environments.

[0003] Firstly, traditional algorithms such as heuristic scheduling algorithms and round-robin scheduling mainly rely on pre-set rules for task scheduling. Although these methods can play a certain role in specific scenarios, they have slow response speed and lack flexibility when the demand for tasks and resources changes. Especially in the face of dynamic tasks and real-time demand, traditional algorithms often fail to adjust in time, resulting in reduced resource utilization.

[0004] Secondly, the efficiency of task scheduling decreases. Greedy algorithms can quickly generate feasible solutions, but they are often affected by greedy strategies and easily fall into local optimal solutions, failing to achieve global optimality. At the same time, these methods have poor adaptability, and when the type of task or constraint condition changes, manual adjustment or reconfiguration of scheduling rules is required, which weakens the generalization ability of the algorithm.

[0005] Based on the above problems, it is particularly urgent to develop a large-scale complex correlation earth observation task planning technology based on reinforcement learning. This technology, with its adaptive and intelligent characteristics, can effectively overcome these limitations and provide more flexible and efficient solutions for large-scale, dynamic, and complex task scheduling problems. SUMMARY

[0006] Therefore, the present application proposes a large-scale complex correlation earth observation task planning method based on reinforcement learning. The present application aims to solve the scheduling optimization problem in large-scale task resource planning, especially the resource and task allocation problem in the field of satellite task scheduling, while improving the dynamic adaptability to the environment.

[0007] The technical solution adopted by the present application is as follows:

[0008] The large-scale complex correlation earth observation task planning method based on reinforcement learning comprises the following steps:

[0009] Step 1, obtain historical data of earth observation task planning, process the historical data, and construct a sample data set;

[0010] Step 2, construct a graph neural network and a reinforcement learning network;

[0011] Step 3, use the sample data set to train the graph neural network and the reinforcement learning network simultaneously;

[0012] Step 4, obtain four types of information lists of requirements, satellites, data transmission tasks, and observations, process them into requirement feature vectors, satellite feature vectors, data transmission task feature vectors, and observation feature vectors; input the four feature vectors into the trained graph neural network to obtain an environment feature vector, and then input the environment feature vector and the requirement feature vector into the trained reinforcement learning network to output the earth observation task planning result.

[0013] Further, the specific way of step 1 is:

[0014] Step 101, obtain earth observation task planning historical data, including four types of information lists of requirements, satellites, data transmission tasks, and observation tasks, and planning results: the requirement list records observation time and position, the satellite list records orbit and constraint, the data transmission task list records satellite data transmission time to the ground station, the observation task list records satellite observable time window, and the planning result is a satellite scheduling scheme; the requirement information list contains the start time and end time , task priority, and target area coordinates (ф, λ); the satellite information list contains satellite ID, satellite orbit height, remaining storage state, and various constraint conditions under the satellite, including observation constraint, data transmission constraint, time interval constraint, sliding circle constraint, and fixed storage constraint; the data transmission information list includes satellite ID, data transmission rate , start time and end time of the data transmission task; the observation task information includes satellite ID, observation start time and end time , and orbit number ;

[0015] Step 102, convert all times in each list into Unix timestamp seconds and normalize them to the [0, 1] interval;

[0016] Step 103, for each requirement information task, generate a candidate satellite set, and traverse all satellites; if the satellite is visible within the task time window and the payload matches, the satellite is added to the candidate satellite set;

[0017] Step 104, the interval tree task number, the average task duration, the scheduled orbit circle times, the average orbit working duration, and the remaining storage state in the satellite information are numerically valued and spliced by column, and zero-mean unit variance standardization is performed according to the dimension, to obtain a satellite feature vector ;

[0018] Step 105, the field information in the demand information and the {0, 1} field of whether the task has been planned are numerically valued and spliced by column, and zero-mean unit variance standardization is performed according to the dimension, to obtain a demand feature vector generated by demand ; the field information in the observation task information and the {0, 1} field of whether the task has been planned are numerically valued and spliced by column, and zero-mean unit variance standardization is performed according to the dimension, to obtain an observation feature vector ; the field information in the data transmission task and the {0, 1} field of whether the task has been planned are numerically valued and spliced by column, and zero-mean unit variance standardization is performed according to the dimension, to obtain a data transmission feature vector ;

[0019] Step 106, a task-satellite executable candidate matrix C is constructed , the value of the matrix element is 0 or 1, and each sample data is composed of the task-satellite executable candidate matrix and the corresponding historical scheduling result.

[0020] Further, the graph neural network transforms the normalized tensor features into trainable node representations, and constructs a heterogeneous graph based on the node representations; the node mutual information is captured by relationship message passing in a multi-layer heterogeneous convolution layer, and after normalization, activation and dropout processing in each layer, a graph-level environment embedding is finally obtained by pooling. The embedding is spliced with the target local feature to serve as the state input of the reinforcement learning network.

[0021] The graph neural network includes a node encoder module, a heterogeneous graph construction module, a heterogeneous convolution layer module, a graph-level pooling module, and a graph-level output to an Actor-Critic network.

[0022] The node encoder module is implemented by a multi-layer feedforward neural network, which inputs the normalized features of various nodes, and sequentially performs linear transformation, batch normalization and ReLU activation, to output a feature matrix of a unified embedding dimension after encoding each type of node;

[0023] The heterogeneous graph construction module combines the encoding feature matrices of the four types of nodes with the edge index relationship provided by the environment, to generate a heterogeneous graph object containing demand, observation, satellite, and data transmission nodes and their associated edges, for subsequent GNN processing.

[0024] The heterogeneous convolution layer module includes 3 layers of heterogeneous convolution units, each layer performing message passing and aggregation on different relations, and outputting updated node features after GraphNorm normalization, ReLU activation and Dropout processing;

[0025] The graph-level pooling module performs global pooling on each type of node feature and merges the results to generate a graph representation vector g that integrates all node and edge information, as the global feature input of the Actor-Critic.

[0026] Further, the reinforcement learning network is based on an Actor-Critic framework;

[0027] The reinforcement learning network takes as input a state vector s that is a concatenation of a graph neural network environment feature vector and a current candidate target local feature: the Actor network adopts a hierarchical policy network: a top-level decision network determines the macro action type, and after a shared sub-network generates shared features, the top-level decision enters the corresponding secondary branch network according to the top-level decision to output specific planning requirements; the Critic network adopts a standard multi-layer MLP to obtain a state value from the input state vector s, which is used to measure the TD error;

[0028] The top-level decision network takes as input a state feature vector s: first, a linear layer, LayerNorm, and ReLU are combined to perform feature transformation, and the number of cycles is set to be one less than the number of linear layers; the last linear layer directly outputs a four-dimensional vector, which is converted to a probability distribution by softmax, and then the four types of top-level action decisions of observation, data transmission, erasure, and skip are obtained by multinomial sampling;

[0029] If the top-level decision result is a skip action, the network directly returns a skip identifier and a null value, ending the current decision;

[0030] If the top-level decision is one of the other three task types, a shared feature extraction network is started to process the same state feature input, and the shared feature extraction network adopts the same structure as the top-level network to obtain a shared feature vector;

[0031] When the top-level decision is to perform an observation task, the observation task sub-network expands the shared feature vector to the same batch dimension as the observation task feature, and then concatenates it with the observation task feature in the feature dimension, and outputs the selection scores of each observation task after linear layer-LayerNorm-ReLU processing;

[0032] When the top-level decision is to perform a data transmission task, the data transmission task sub-network performs a similar processing flow to the observation task, and finally outputs the data transmission task number;

[0033] When the top-level decision is to perform an erasing task, the satellite sub-network performs a similar processing flow as the observation task, and finally outputs the satellite number, selecting which satellite to perform the erasing task.

[0034] Further, the specific manner of step 3 is:

[0035] An experience replay mechanism is adopted, historical task data trajectories are stored in a replay buffer, and a small batch of samples are randomly extracted for training the network; the objective function is defined as maximizing the long-term discounted cumulative reward obtained by the agent in the environment, and the target is achieved by constructing the policy loss and value loss of the reinforcement learning network;

[0036] In the training loop, first, a number of trajectories are randomly extracted from the replay buffer, and are spliced into a sample sequence in chronological order; after random rearrangement of the sample sequence, the sample sequence is divided into multiple training small batches according to the batch size m; for each small batch: the Actor network calculates the policy distribution (|s) in parallel, determines the action set a according to the policy sampling, and records the action pair logarithmic probability log (a|s); the Critic network calculates the value prediction (s) in parallel, and records the predicted value set;

[0037] The actual immediate reward of the small batch and the value prediction of the next state are used to perform the temporal difference calculation, and the generalized advantage estimation is recursively calculated in the order from back to front, specifically:

[0038]

[0039] Wherein, represents the immediate reward returned by the environment after performing the action at time step t, represents the discount factor, represents the value function of the next state at time t+1, (1− ) represents a termination flag correction term, represents the value function of the current state

[0040] The temporal difference error is calculated, wherein, represents the advantage function estimate at time step t, represents the temporal difference error at time step t, represents a smoothing parameter of the generalized advantage estimation, represents the advantage estimate at the next time t+1;

[0041] The advantage estimate is recursively obtained, and then The value of each step is calculated; after the advantage and target value calculation is completed, all samples are summarized into a set and randomly shuffled, and then divided into several small batches according to the set small batch size, for subsequent training;

[0042] Parameter updating is performed in mini-batches: first, calculate the advantage and package the samples, and perform multiple rounds of gradient descent for each mini-batch; the Actor is updated using PPO clipping targets and backpropagated to the shared encoder and graph network, and the Critic is updated using mean square error; at the end of each round, the current policy is copied to the old policy for the next round of ratio calculation; training is terminated after the maximum number of rounds, and the task planning result is output.

[0043] The beneficial effects of the present application are:

[0044] 1. The present application can solve the scheduling optimization problem under the condition of multi-task and limited resources, improve the efficiency of task planning, and can be widely applied in the fields of satellite task scheduling, intelligent manufacturing, automatic driving, etc.

[0045] 2. In view of the limitations of traditional heuristic and planning algorithms in dynamic environment, such as response lag, local optimum and insufficient generalization ability, the present application designs a modular reinforcement learning framework: the state space integrates task attributes (type, priority, time window), satellite state (position, load, storage), time constraints and multi-task correlation and dependence; the action space adopts a two-level decision mechanism (top-level action type selection + secondary resource fine-grained allocation), supporting observation, data transmission, erasure and skipping operations; the Actor-Critic architecture combines the PPO optimization algorithm, generates scheduling actions through the policy network (Actor), and evaluates the state-action pair value through the value network (Critic); the reward function integrates task completion, constraint satisfaction and resource utilization, driving the agent to balance efficiency and compliance.

[0046] 3. The present application models the global state through heterogeneous graph convolution, supports multi-frequency observation, inter-satellite cooperation and complex patterns, and realizes dynamic resource conflict avoidance and priority optimization in training. Experiments show that this method significantly improves the task completion rate and resource utilization efficiency of large-scale satellite clusters in dynamic environments, and can be extended to intelligent manufacturing, automatic driving and similar scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 The system module structure diagram of the present application.

[0048] Figure 2 The core algorithm flowchart of the present application.

[0049] Figure 3 The system timing diagram of the present application. DETAILED DESCRIPTION

[0050] A method for planning large-scale, complex, correlated Earth observation missions based on reinforcement learning includes the following steps:

[0051] Step 1: Obtain historical data on Earth observation mission planning, process the historical data, and construct a sample dataset; the specific method is as follows:

[0052] Step 101: Obtain historical data for Earth observation mission planning, including a list of requirements information, a list of satellite information, a list of data transmission mission information, a list of observation mission information, and planning results for each mission planning. The requirements information list records when to observe a certain location, the satellite information list records the satellite's orbit and constraints, the data transmission mission information list records when the satellite can transmit data to the ground station, the observation mission information list records the observable time window of the satellite, and the planning results are the satellite scheduling scheme.

[0053] Step 102: Represent the demand information list, satellite information list, data transmission task information list, and observation task information list in JSON format. The demand information list includes the start time of the target to be observed. and end time The mission priority and target area coordinates (ф, λ) are specified. The satellite information list includes the satellite ID, orbital altitude, remaining storage status, payload status, and various constraints imposed on the satellite, including observation constraints, data transmission constraints, time interval constraints, sliding circle constraints, and storage constraints. The data transmission information list includes the satellite ID and data transmission rate. Start time of data transmission task and end time The observation mission information includes the satellite ID and the observation start time. With end time , the circle Hash the satellite IDs to generate consecutive integer indices, and save the mapping table as mapID;

[0054] Step 103, set the task start time. Mission End Time Data transmission task start time Data transmission end time Observation mission start time Observation mission end time The timestamps are uniformly converted to Unix timestamps in seconds and normalized to the [0,1] range.

[0055] Step 104: For each requirement information task, generate a candidate satellite set, traverse all satellites, and if a satellite is visible within the task time window and the payload matches, add the satellite to the candidate satellite set.

[0056] Step 105, the interval tree task number, the average task duration, the scheduled orbit lap number, the orbit average working duration, and the remaining storage state in the satellite information are numerically connected in columns; the start time, the end time, the task priority, the target area coordinates, and whether the task has been planned are numerically connected in columns; the start time, the end time, the satellite ID, the lap number, and whether the task has been planned are numerically connected in columns; the start time, the end time, the satellite ID, the data transmission rate, and whether the task has been planned are numerically connected in columns; the satellite orbit height, the remaining storage, and the load state are connected in columns;

[0057] Step 106, all feature vectors are executed by zero-mean unit variance standardization according to the dimension, and the processed results are organized into five tensors, which are:

[0058] For each satellite, dynamic indicators are extracted from its static field and satellite management statistical data generated during sample construction: interval tree task number, average task duration, scheduled orbit lap number, orbit average working duration, and remaining storage state. These dynamic indicators are normalized according to scaler_params, and then concatenated with the satellite static field in columns to form a satellite feature vector ;

[0059] A demand feature vector is generated for the demand ;

[0060] An observation feature vector is generated for the observation task ;

[0061] A data transmission feature vector is generated for each data transmission task .

[0062] Task-satellite executable candidate matrix C .

[0063] Each sample data is composed of the above feature tensors and the corresponding historical scheduling results, which is used to train the network.

[0064] Step 2, construct a graph neural network and a reinforcement learning network;

[0065] The graph neural network first receives heterogeneous graph data containing four types of nodes as input, including target nodes, observation nodes, satellite nodes, and data transmission nodes, each carrying a 64-dimensional feature vector. In addition, the graph neural network also receives connection relationships of four edge types, which are:

[0066] (target, requires, observation) edge from demand node to observation node;

[0067] observation, executed, satellite edges from observation nodes to satellite nodes;

[0068] satellite, executes, transmission edges from satellite nodes to transmission nodes;

[0069] target, self, target edges from target nodes to target nodes.

[0070] The network inputs the heterogeneous graph data into the first layer of heterogeneous convolution modules, which are configured with independent SAGEConv convolution operators for the four different edge types. The SAGEConv for the requires edge processes the information transmission between the target nodes and the observation nodes, the SAGEConv for the executed edge processes the information transmission between the observation nodes and the satellite nodes, the SAGEConv for the executes edge processes the information transmission between the satellite nodes and the transmission nodes, and the SAGEConv for the self edge processes the self-loop information transmission within the target nodes.

[0071] After the four SAGEConv operators perform convolution operations in parallel, the information from different edge types is fused through the mean aggregation method to generate updated node feature representations.

[0072] The node features output by the first layer of heterogeneous convolution modules are then input into the corresponding graph normalization modules. The network is configured with independent GraphNorm normalizers for the four node types. The target node features are normalized by the target type GraphNorm, the observation node features are normalized by the observation type GraphNorm, the satellite node features are normalized by the satellite type GraphNorm, and the transmission node features are normalized by the transmission type GraphNorm.

[0073] The normalized feature vectors are then subjected to nonlinear transformation by the ReLU activation function and random inactivation regularization processing by the Dropout module, with an inactivation probability of 0.3.

[0074] The node features processed by the first layer are input into the second layer of heterogeneous convolution module, which adopts the same four edge type configurations and SAGEConv convolution structure as the first layer, but the input dimension is adjusted to 128 dimensions of the hidden layer. The convolution, normalization, activation and Dropout operation processes of the second layer are exactly the same as those of the first layer. The processed feature vector enters the third layer of heterogeneous convolution module, and the third layer also maintains the same processing flow and module configuration.

[0075] After the three-layer convolution processing is completed, the network performs global average pooling operation on the features of each node type respectively. The network first checks the size of the feature tensor of each node type. If the number of nodes is greater than zero, the global_mean_pool operation is performed on the node feature of this type to generate a fixed-dimensional vector representation. If the number of nodes is zero, a zero vector of the corresponding dimension is generated as the representation vector of this type. The pooled result vectors of the four node types are spliced in dimension 0, and the spliced tensor is again subjected to an average operation to obtain the final graph-level embedding vector.

[0076] The entire network outputs a graph-level representation vector with a dimension of [1, 128], which integrates the information of all node types and edge types in the heterogeneous graph and can be used as input for downstream reinforcement learning.

[0077] The reinforcement learning network adopts a hierarchical decision-making architecture, including a top-level decision network, a shared feature extraction network, and three secondary decision networks.

[0078] The reinforcement learning network first receives the graph-level embedding vector output by the graph neural network, i.e., the environment state feature as input. The environment state feature is simultaneously transmitted to the top-level decision network and the shared feature extraction network for parallel processing.

[0079] After receiving the environment state feature, the top-level decision network sequentially passes through a combination of linear layers and LayerNorm normalization layers for feature transformation. Each linear layer is followed by a LayerNorm normalization layer, and then a ReLU activation function is applied. The network repeatedly performs this linear transformation-normalization-activation cycle operation, with the number of cycles equal to the set number of layers minus one. The last linear transformation directly outputs a four-dimensional vector without normalization and activation operations. The four-dimensional vector is converted into a probability distribution by the softmax function, and the top-level action decision is obtained by multinomial sampling. This decision includes four options: performing an observation task, performing a data transmission task, performing an erasure task, or skipping the current decision.

[0080] If the top-level decision result is a skip action, the network directly returns a skip identifier and a null value, ending the current decision process.

[0081] If the top-level decision is one of the other three task types, the network initiates a shared feature extraction network to process the same state feature input. The shared feature extraction network adopts the same structural pattern as the top-level network, extracts shared feature representations through a cycle combination of multiple linear transformations, LayerNorm normalization and ReLU activation, and finally obtains a shared feature vector through ReLU activation in the last layer.

[0082] When the top-level decision is to perform an observation task, the network expands and replicates the shared feature vector to the same batch dimension as the observation task features, and then concatenates the expanded shared features with the observation task features in the feature dimension. The concatenated feature vector is input into the observation task selection network, which also adopts a cycle structure of linear layers-LayerNorm-ReLU to process the features, and finally outputs the selection scores of each observation task.

[0083] The network constructs a mask tensor according to the observation task arrangement in the current environment state and the target demand association information, sets the positions corresponding to the tasks that have been arranged and do not meet the current target demand to True, and sets the remaining positions to False. The selection score tensor and the mask tensor are subjected to a mask filling operation, and the True positions are filled with negative infinity. The masked scores are converted into a probability distribution through a softmax function, and if the probability sum is below a threshold, a skip flag is returned, otherwise a specific observation task number is obtained through polynomial sampling.

[0084] When the top-level decision is to perform a data transmission task, the network performs a similar processing procedure as the observation task, concatenates the shared features with the data transmission task features, and inputs them into the data transmission task selection network for feature processing. The network constructs a mask according to the data transmission task arrangement state, sets the positions of the tasks that have been arranged to True and fills them with negative infinity in subsequent mask operations. The masked scores are converted into a probability distribution through a softmax function and a data transmission task number is obtained through polynomial sampling.

[0085] When the top-level decision is to perform an erasure task, the network concatenates the shared features with the satellite features, and inputs them into the satellite selection network for feature transformation. Since the erasure task does not need to consider the task arrangement state restriction, the network directly converts the output scores of the satellite selection network into a probability distribution through a softmax function, and then performs polynomial sampling to obtain the target satellite number.

[0086] Step 3, simultaneously train the graph neural network and the reinforcement learning network using the sample data set; the specific method is:

[0087] The data of historical tasks, including task requirements, satellite states, observation information, and data transmission information, are stored in the replay buffer in the form of records, and samples are extracted from the replay buffer in a random small batch manner to input the network for training. The objective function is defined as maximizing the long-term discounted cumulative reward obtained by the agent in the environment. The policy loss and value loss of the reinforcement learning network are constructed to achieve this goal. The policy loss and value loss are combined to form the overall loss L(Θ) = L(θ) + L(φ).

[0088] When entering a training cycle, a number of trajectories are extracted from the replay buffer in a random sampling manner and spliced into a sample sequence in chronological order. Then, random rearrangement is performed on the sample sequence, and the sample sequence is divided into multiple training small batches according to the small batch size m. For each time step sample, a state construction operation is performed: first, the target requirements, observation tasks, and original attributes of the satellite and data transmission tasks corresponding to the time step are input into the node encoder in turn to obtain the node feature tensor. Then, the heterogeneous graph is constructed using the node feature tensor and the task-satellite association edge, and the graph neural network is input to perform multi-layer neighbor feature aggregation. The graph neural network outputs the global environment feature vector . The local task features corresponding to the time step are spliced to obtain the state vector , and the is stored in the state set of the current small batch for parallel forward inference.

[0089] The Actor network forward inference operation is performed on the current small batch: the state set in the small batch is input to calculate the policy distribution (π(.|s)) in parallel, and then the action set a corresponding to the small batch is determined according to the principle of policy sampling or the highest probability, and the action pair log (a|s) probability is recorded.

[0090] The Critic network forward inference operation is performed on the current small batch: the state set in the small batch is input to calculate the value prediction (V(s)) in parallel, and the predicted value set is recorded.

[0091] The temporal difference is calculated using the actual immediate reward of the small batch and the value prediction of the next state, and the generalized advantage estimation (GAE) is recursively calculated in the order from back to front. Specifically:

[0092]

[0093] where represents the immediate reward returned by the environment after performing the action at time step t, represents the discount factor, represents the value function of the next state , and (1− () indicates a termination flag modification item. Indicates the current state The value function of .

[0094] Calculate the timing difference error and follow the formula.

[0095]

[0096] in, This represents the estimate of the dominance function at time step t. This represents the timing difference error at time step t. The smoothing parameter represents the generalized dominance estimation. This represents the advantage estimate for the next time step t+1.

[0097] Advantage estimation is obtained recursively. Then press Calculate the value of each step; after completing the calculation of the advantage and target value, form a sample set by recording all samples and perform random rearrangement on the set to generate an unordered sample sequence. Then, divide the unordered sample sequence into several small batches according to the batch size for subsequent training. This completes the data input preparation.

[0098] The data input method is as follows: "First, the trajectory is sampled stepwise in the environment in chronological order. Then, the trajectory data is randomly shuffled and input into the network in small batches for training" (i.e., sampling is stepwise input, and training is batch input). After the small batch division is completed, the training update phase begins. The training update phase performs the following operations in several training rounds: At the beginning of each round, a copy of the current Actor network parameters is stored in the old policy parameters for use in ratio calculation. Then, the following sub-operations are performed on each small batch:

[0099] Using all states in the mini-batch as input, parallel forward inference is performed under the current policy to obtain the log probabilities of each action under the current policy. And what was obtained under the old strategy The probability ratio is calculated based on the difference between the two:

[0100]

[0101] by With advantages Construct two candidate options , , and according to Calculate the Actor loss for this mini-batch, and then use the state of this mini-batch as input to perform parallel inference on the Critic network to obtain the predicted value. and with target value Calculate the mean squared error loss of Critic .

[0102] After obtaining the Actor loss and Critic loss, parameter update operations are performed. Specifically, the Actor loss is first backpropagated to calculate the gradients with respect to the Actor network parameters and the shared encoder and GCN parameters, and then a parameter update is performed on the Actor optimizer, and the gradient is norm-clipped before the update to limit the gradient amplitude. The Critic loss is backpropagated to calculate the gradients with respect to the Critic network parameters and the shared encoder and GCN parameters, and a parameter update is performed on the Critic optimizer, and the gradient is norm-clipped before the update. The parameters of the shared encoder and GCN are updated according to the optimizer design rules within or between mini-batches, so that the parameters of the shared module are simultaneously affected by the gradients from the Actor and the Critic. After completing the update of all mini-batches in the current round, the current Actor network parameters are copied to the old policy parameters for use in the next round. After a predetermined number of epochs, the inner loop of the current training batch is ended, and the training loss and average return indicators of the batch are recorded. At the end of each training batch, if the episode count meets the model saving strategy, the current Actor network parameters and Critic network parameters are serialized and saved as a checkpoint file.

[0103] After completing a complete sampling and updating cycle, the training episode number is incremented and the above sampling, advantage calculation, mini-batch division, and multi-round update are repeated in the next sampling cycle until the training termination condition is met. The training termination conditions include but are not limited to the following judgments: if the training episode count reaches the predefined maximum number of episodes, the training is stopped, if the sliding average return reaches or exceeds the predefined performance target, the training is stopped, or if there is no improvement for a continuous number of episodes that reaches the early stopping threshold, the training is stopped. When the training is terminated, the historical saved optimal model parameters are loaded or retained, and the model is exported as the final strategy for inference. The final training result information is a task planning scheme.

[0104] Step 4: Obtain the demand information list, satellite information list, data transmission task list, and observation information list, process the four lists into four feature vectors, input the four feature vectors into the trained graph neural network to obtain the environment feature vector, and then input the environment feature vector and the feature vector of the demand information list into the trained reinforcement learning network to obtain the result of the Earth observation task planning.

[0105] As shown in Figure 1 , the system implementing the method comprises:

[0106] 1. State space module: input of the system, containing task information, satellite information, time information, and constraint information.

[0107] (1) The task information module contains all the basic information of the task, including the type of the task (such as a shooting task, a transmission task, etc.), resource requirements, task target area, priority, time window, special constraints (for example, a specific task must be completed within a specified period), etc.

[0108] (2) The satellite information module records the state information of all on-orbit satellites, such as position, flight orbit, payload status (such as whether the shooting device and communication device are available), power consumption, storage space, etc. It also contains the working status of all current satellites (such as whether they are busy, whether they can be scheduled, etc.).

[0109] (3) The time information module provides information such as the current time of the system, the scheduling time limit of the task, and the time window, to ensure that the task scheduling is carried out within the time constraints. It also considers the feasibility of future time periods, such as the frequency requirements of the task in a certain period.

[0110] (4) The constraint information module is responsible for collecting all the constraints in the task execution, such as the dependency between tasks, priority, satellite task conflict restrictions, time window restrictions, multi-frequency task requirements, etc. These constraints ensure that there are no resource contention, conflict, or unmet demand issues during task execution.

[0111] The task information module and the satellite information module cooperate to determine whether each task has a suitable satellite for execution, and work with the constraint information module to ensure that the task meets the time window and resource constraints during scheduling.

[0112] 2. Action space module: defines all possible actions that the agent can take, including task assignment, payload selection, time selection, etc. The action space is determined by the top-level decision network and can be divided into the following types:

[0113] Execute observation task: select an unscheduled observation task for scheduling.

[0114] Execute data transmission task: select an unscheduled data transmission task for scheduling.

[0115] Execute erase task: select a satellite for file erasure.

[0116] Skip: do nothing.

[0117] In the ActorNetwork, the top-level decision network outputs a probability distribution of 4 categories, corresponding to the above four actions. The specific selection of which task or satellite is further decided by the secondary network.

[0118] (1) Satellite Assignment Submodule: Responsible for selecting suitable satellites from the set of available satellites to execute the task. The assignment is determined based on the task requirements provided by the Task Information Module and the state of the satellites in the Satellite Information Module. The goal of satellite assignment is to optimize the execution of the task by assigning the most suitable satellites, ensuring efficient use of resources while adhering to constraints.

[0119] (2) Payload Selection Submodule: In some cases, a task may require a specific type of payload, such as infrared imaging or SAR imaging, to be completed. The Payload Selection Submodule will identify the best matching payload based on the task requirements and available options on the satellite. This module is responsible for selecting the appropriate payload or sensor type for the task to meet the task requirements. Different payload types (e.g., optical imaging, SAR imaging) can provide different data types, and the goal of this module is to improve observation quality by matching the payload with the task requirements.

[0120] (3) Time Selection Submodule: Responsible for selecting an appropriate time window for the task execution to ensure it is completed within the specified time frame. The key objective of the Time Selection Submodule is to ensure that the task can be executed within the required observation time, especially for multi-frequency tasks and time synchronization requirements for collaborative tasks.

[0121] 3. Actor Module: Generates the policy network and selects the optimal action based on the state information

[0122] Input Layer: Inputs all the information obtained from the State Space Module. The input layer converts the state information into a tensor or other suitable format for deep learning processing and may perform simple normalization or standardization to make the data more suitable for feature extraction by the model.

[0123] Feature Extraction Layer: Extracts features of the task and satellites by passing the input state information through a series of neural network layers (such as convolutional layers or fully connected layers) to generate representative feature vectors. These feature vectors serve as the basis for decision-making in the subsequent Action Selection Layer.

[0124] Action Selection Layer: Based on the feature vectors generated by the Feature Extraction Layer, this layer outputs the optimal action strategy for the current state. The output of this layer is the action selected by the model, which is the operation that the system should perform in the current state.

[0125] Scores each action option through a fully connected layer or a specific scoring network, reflecting the suitability of each option. Outputs a probability distribution of actions, indicating the likelihood of each action being selected. The model will sample based on this probability distribution to determine the final action. The specific action can be executed based on a greedy strategy (selecting the action with the highest probability) or a random sampling strategy (sampling randomly according to the probability distribution).

[0126] 4. Critic module: evaluates the action selected by the Actor, outputs the value estimate of the state through the value network.

[0127] (1) Input layer: receives state information and action information selected by the Actor, and converts them into a format that the model can process. The Critic module usually uses state-action pairs as input, so it includes not only the state information of the current system, but also the action selected by the Actor module.

[0128] (2) Feature extraction layer: extracts features from the state and action for value estimation. The feature extraction layer contains a series of neural network layers (such as fully connected layers or convolutional layers) that convert input data into high-dimensional feature vectors, allowing the model to find the most effective information for value estimation from the state and action.

[0129] (3) Value estimation layer: the value estimation layer calculates the value estimate of the current state and action based on the feature vector generated by the feature extraction layer. The output is a scalar or vector representing the expected value of the current state-action pair, which serves as the basis for policy optimization by the Actor module. Through the backpropagation algorithm, the Critic module uses the error between the actual return and the estimated value to update the model parameters, making the estimate more accurate.

[0130] 5. Reward module: calculates the effect of the action taken by the Actor and provides a reward signal to the Actor and Critic.

[0131] (1) Task completion degree calculation module: checks whether the task is completed within the specified time and whether the Agent's action has achieved the expected goal. For example, whether a specific satellite has successfully observed the target area. In the case of tasks with multiple stages or dimensions, the module quantifies the degree of task completion, such as the time window of the task, the integrity of data transmission, and the execution of the satellite's task. If completed, give positive reward, otherwise punish.

[0132] (2) Constraint satisfaction degree calculation module: evaluates whether the task meets various constraint conditions, including time constraints, resource constraints, and data transmission constraints. If satisfied, give a reward, otherwise punish.

[0133] (3) Resource utilization rate calculation module: evaluates the use of resources and encourages efficient resource utilization. In the case of multiple satellites and multiple payloads, check whether resources are reasonably allocated to achieve efficient use of overall resources. By optimizing the resource utilization rate calculation module, the Agent can learn to use resources more conservatively, achieving a balance between task completion and resource efficiency.

[0134] 6. PPO optimization module: update the parameters of the Actor and Critic networks.

[0135] (1) Policy loss calculation module: calculate the update loss of the policy according to the output of the Actor and the actual feedback.

[0136] (2) Value loss calculation module: calculate the update loss of the value network according to the output of the Critic and the actual reward.

[0137] (3) Optimization control module: PPO limits the update range of the policy, controls the distance between the new and old policies to ensure the stability of the algorithm. By setting different learning rates, the update step of the Actor and Critic is controlled to avoid rapid or drastic updates that can cause training divergence. Gradient clipping is used to avoid gradient explosion problems during the update process, especially in cases of large losses, to ensure smooth updates. The policy is only allowed to update after a certain number of steps to reduce unnecessary frequent updates, allowing the Agent to have time to fully explore.

[0138] Figure 2 The core architecture of the Actor-Critic network is described. The environment receives actions from the Actor network, outputs a new state and immediate reward. The Actor network inputs the current state and outputs an action policy to determine the action performed by the agent in the environment. Experience storage stores the triplets (state, action, reward) generated during the interaction process in the cache for subsequent sampling and training. The Critic network samples state-action pairs from the experience cache to estimate the corresponding value function, i.e., the Critic scores the performance of the current policy.

[0139] The advantage function compares the value estimated by the Critic network with the output of the old Actor network (used as a "baseline" during stable training) to calculate the advantage value, which measures how much better a certain action is than the average level. The reverse update weight uses the advantage function to update the Actor network with a policy gradient, and simultaneously updates the Critic network with a regression of the value function, completing one training iteration.

[0140] Overall, the Actor-Critic method combines policy optimization and value evaluation, and through the continuous alternation of updating the parameters of the two, the agent learns to make optimal decisions in the environment.

[0141] Figure 3 The timing diagram of the PPO training process is shown in the following figure, which shows the information flow between the state space, Actor, Critic, environment, reward, and PPO optimization.

[0142] The principle of this method is as follows:

[0143] (1) Data Preparation and Preprocessing

[0144] Before formally entering the reinforcement learning training, first need to unify the "demand information", "meta task strip", "satellite information", "satellite constraints", "data transmission resources" data preprocessing. This step contains the following sub-steps:

[0145] Satellite and resource ID mapping: Because the satellite, ground station and even inter-satellite link in the input often appear as string ID or mixed type, it is necessary to map them to continuous integer values for subsequent discrete coding in neural networks. At this time, the original and mapped correspondence is preserved to facilitate reverse mapping when the final output is displayed.

[0146] Timestamp to float: All time fields (target observation time window, strip observation start and end, data transmission start and end, etc.) are normalized to a unified and comparable floating point number (such as Unix seconds or second offset), which facilitates subsequent constraint checking and feature normalization when calculating time intervals or relative order.

[0147] For multi-frequency, cooperative guidance, inter-satellite link, multi-strip splicing and other mode requirements, it is necessary to check and fill in the corresponding "frequency", "cooperative rule", "cluster information" and other fields in the target and strip during preprocessing. If some targets lack necessary fields (such as multi-frequency tasks lacking "observation interval time"), warnings need to be thrown or default values need to be filled in.

[0148] After the above preprocessing steps, a complete set of structured internal data for reinforcement learning environment is obtained. Including:

[0149] target_info_list: target objects arranged by index, each object carrying observation window, priority, mode information (whether multi-frequency / cooperative / splicing, etc.).

[0150] observation_task_list: strip objects arranged by index, each object containing the spatio-temporal coverage area of the strip, the sensor mode used, the turn number, the cluster ID, etc.

[0151] satellite_list: satellite static attribute list, including orbit type, recording rate, available working mode list, etc.

[0152] transmission_task_list: data transmission window list arranged by index, including satellite ID, start / end time, transmission rate, ground station or relay satellite ID.

[0153] constraint_info_list: constraint setting of each satellite, covering all elements such as time interval, sliding circle, storage, erasure, upper limit of data transmission duration, etc.

[0154] (2) Environment modeling and state representation

[0155] In the reinforcement learning framework, the entire satellite scheduling problem needs to be defined as an iterative process of interaction between an "environment" and an "agent". The environment is responsible for maintaining the current state, responding to the agent's actions, returning the next state and reward; the agent outputs an action based on the existing state to schedule tasks. In order to enable the agent to efficiently evaluate complex temporal and spatial constraints and pattern information, the following methods are used for environment modeling and state representation:

[0156] State object: saves the list of current remaining goals to be met and the index, the "arrangement flag" of the current observation task and data transmission task (i.e. which bands have been arranged, which data transmission windows have been used), and each satellite corresponds to a satellite manager for maintaining the interval tree of the arranged tasks of the satellite, the orbit dictionary and the file management system. Mainly used to define global static information (constraints) and dynamic state (which tasks are arranged, the latest available time of the satellite, file system occupation, etc.) that change with time iteration, and will be extracted by the agent for features at each decision-making step.

[0157] (3) Network architecture design

[0158] After the environment and state representation are prepared, a typical Actor-Critic PPO (Proximal Policy Optimization) framework is used for training and reasoning.

[0159] The Acotr policy network needs to make two-level decisions:

[0160] Top-level action (macro type), the agent first decides which macro operation to perform: "meet current goal demand" -> select a specific observation task; "execute data transmission task" -> select a data transmission task of a satellite; "execute erasure task" -> select a satellite to trigger file erasure; "skip this goal" -> abandon the current goal and move directly to the next goal.

[0161] Secondary action (micro selection), if the top-level selection is "observation", a filtering and sampling is done in the candidate observation band set - only the bands that can meet the current target can be selected, and the satellite corresponding to the band must meet the timing constraints, storage capacity, load mode, mode (multi-frequency / collaboration / concatenation), etc. The agent will calculate the score distribution of each selectable band, and then randomly sample an ID as the observation task for this time.

[0162] If the top-level selection is "Data Transmission", the agent calculates the data transmission task priority distribution for each file in the current satellite task set corresponding to all files to be transmitted, and then samples the data transmission task to be executed.

[0163] If the top-level selection is "Erasure", the agent calculates a probability distribution for each satellite in the satellite list with files to be erased and samples the satellite to be executed.

[0164] If the top-level selection is "Skip", no sub-action is performed and the current target is directly marked as "Skipped".

[0165] Before making a decision, the agent also refers to the target priority, mode progress (remaining number of times for multi-frequency, coordination progress, and splicing progress), satellite resource state (capacity, circle count statistics, number of tasks already scheduled, and number of files to be transmitted and erased) carried in the current "global graph embedding", etc. After two-level network processing, the agent gives a macro selection and its corresponding sub-action, triggering the corresponding rules and resource updates. At the same time, the system needs to design appropriate reward functions. Generally, rewards can include: a positive reward is given according to the target priority when a target demand is successfully met, and a full reward is given if the multi-frequency mode is met each time or the coordination is met; a certain positive reward is given according to the transmission rate ratio after successfully executing a data transmission task; a certain reward is given when successful erasure reduces the occupied storage of the stored file; a penalty proportional to the target priority is deducted for each skipped demand, encouraging the agent to meet high-priority demands as much as possible; if a conflict is arranged or a certain constraint is violated (such as insufficient interval or insufficient capacity but still attempting to observe), a large negative penalty is given.

[0166] Through this design, the agent will learn in constant trial and error: when should observation be prioritized, when should the satellite perform data transmission / erasure to free up capacity, how to balance each target for multi-frequency / coordination / splicing mode, and when to skip a low-priority target to reserve resources for more important demands.

[0167] (4) Environment interaction and training process

[0168] a) Start of loop: the environment queues all targets in a certain order (usually according to priority or time order). The agent resets internal counters, including the queue for each satellite, the file system state, and the erasure task ID, etc.

[0169] b) Current target index: the agent checks the current target ID to be processed.

[0170] c) State Encoding: The environment extracts the latest features from various data structures (target list, observation task list, satellite manager, telemetry list), converts them into a tensor of four types of nodes through "node encoding", and then constructs a heterogeneous graph with these features and corresponding relationships. After multi-layer graph convolution and global pooling, a graph-level state vector is output.

[0171] d) Agent Action Selection: Based on the graph-level vector, the agent first makes a classification decision to perform "observation / telemetry / erase / skip" in the top-level network, and then calculates the score distribution of each candidate action (specific strip or task) in the "shared features + corresponding node type features" space for the secondary network and samples an action.

[0172] e) Execution and Update: After receiving the macro action and sub-action, the environment performs the corresponding update:

[0173] If it is "observation", the corresponding observation task is inserted into the interval tree of the satellite, the file system capacity and the pending file queue are updated;

[0174] If it is "telemetry", the transmission operation is performed in the file system of the corresponding satellite according to the priority queue, and the file transmission ratio and the pending / erased queue are updated;

[0175] If it is "erase", all transmitted files are deleted in the satellite file system and the capacity is updated;

[0176] If it is "skip", the current target is directly marked as skipped, and no task update is performed.

[0177] At the same time, the environment calculates the immediate reward obtained in this step and returns it to the agent. If the target index is skipped or the observation is completed, the index is automatically incremented by one, and the next target is entered.

[0178] f) Data Storage: The current state, action, reward, next state, done flag, and required feature tensor are recorded in the internal trajectory buffer of the agent. This facilitates the subsequent calculation of the advantage function (GAE) and the update of the policy and value networks.

[0179] g) Check End: When all targets are traversed or the maximum number of steps is reached, the "Episode" ends. If further training is needed, the advantage is calculated in the buffer and the Actor and Critic network parameters are updated according to the PPO algorithm, completing a training round. Otherwise, the trained policy model is saved to disk.

[0180] h) Multiple Rounds of Training: The entire process is repeated for several "Episodes" until the sliding average reward or validation set performance no longer improves.

[0181] (5) Inference and scheduling result output

[0182] After training, we save the learned best policy model. In the inference stage, the environment and state representation are basically the same as in the training stage, but the network parameters are no longer updated, and only single or multiple decision samples are generated to generate the final scheduling scheme. In inference, the process is roughly as follows:

[0183] Load environment and model: Initialize the same environment, read the same format of target list, strip, satellite constraints and other data, and then load the trained Actor network parameters.

[0184] Loop decision: From the first target, repeat the steps of "state encoding → agent action selection → environment execution → record results" until all targets are processed or there are no available actions. Each step will save the decision result (including observation task ID, satellite ID, task start / end time) to a scheduling result list.

[0185] In this way, the final scheduling result can meet the demand constraints of "multi-frequency", "cooperative guidance", "multi-strip splicing" and "inter-satellite cooperation" modes, and can maximize the use of satellite resources - balance observation coverage, file transmission and storage erasure. The whole process from data input to feature extraction, to heterogeneous graph modeling, graph convolution to extract global state, two-level policy decision and environment execution, forms a closed loop, which can adapt to large-scale, multi-mode, multi-satellite, multi-resource joint scheduling and optimization tasks.

Claims

1. A method for planning a large-scale complex correlated ground observation task based on reinforcement learning, characterized in that, The method comprises the following steps: Step 1, obtaining historical data of earth observation task planning, processing the historical data, and constructing a sample data set; Step 2, constructing a graph neural network and a reinforcement learning network; Step 3, training the graph neural network and the reinforcement learning network simultaneously using the sample data set; Step 4, obtaining a list of four types of information of requirements, satellites, data transmission tasks and observations, processing them into requirement feature vectors, satellite feature vectors, data transmission task feature vectors and observation feature vectors; inputting the four feature vectors into the trained graph neural network to obtain an environment feature vector, and then inputting the environment feature vector and the requirement feature vector into the trained reinforcement learning network to output an earth observation task planning result.

2. The method of claim 1, wherein the method is based on reinforcement learning. The specific manner of step 1 is: Step 101, obtaining the historical data of the earth observation task planning, including four categories of information list of demand, satellite, data transmission task and observation task and planning results: the demand list records the observation time and position, the satellite list records the orbit and constraint, the data transmission task list records the satellite data transmission time to the ground station, the observation task list records the satellite observable time window, and the planning result is the satellite scheduling scheme; the demand information list contains the start time and end time , task priority, target area coordinates (ф, λ); the satellite information list contains satellite ID, satellite orbit height, remaining storage state, and various constraint conditions under the satellite, including observation constraint, data transmission constraint, time interval constraint, sliding ring constraint and fixed storage constraint; the data transmission information list includes satellite ID, data transmission rate , data transmission task start time and end time ; the observation task information includes satellite ID, observation start time and end time , and the ring number ; Step 102, converting all the time in each list into Unix timestamp seconds and normalizing them to the [0, 1] interval; Step 103, for each requirement information task, generating a candidate satellite set, and traversing all satellites, if the satellite is visible within the task time window and the payload matches, the satellite is added to the candidate satellite set; Step 104, the interval tree task number, the average task duration, the arranged orbit lap number, the orbit average working duration, and the remaining storage state in the satellite information are numerically connected by column, and zero-mean unit variance standardization is performed according to the dimension, to obtain a satellite feature vector ; Step 105, the field information in the demand information and the {0, 1} field of whether the task has been planned are valued and spliced by column, and zero-mean unit variance standardization is performed according to the dimension, to obtain a demand generation demand feature vector ; the field information in the observed task information and the {0, 1} field of whether the task has been planned are valued and spliced by column, and zero-mean unit variance standardization is performed according to the dimension, to obtain an observation feature vector ; the field information in the data transmission task and the {0, 1} field of whether the task has been planned are valued and spliced by column, and zero-mean unit variance standardization is performed according to the dimension, to obtain a data transmission feature vector ; Step 106, constructing a task-satellite executable candidate matrix C The value of the matrix element is 0 or 1, and each sample data is composed of the task-satellite executable candidate matrix and the corresponding historical scheduling result.

3. The method of claim 1, wherein, The graph neural network transforms the normalized tensor features into trainable node representations, and constructs a heterogeneous graph accordingly; in a multi-layer heterogeneous convolution layer, the node mutual information is captured by message passing according to the relationship, and after normalization, activation and dropout processing in each layer, the graph-level environment embedding is finally obtained by pooling; after the embedding and the target local feature are spliced, they are input into the state of the reinforcement learning network; The graph neural network comprises a node encoder module, a heterogeneous graph construction module, a heterogeneous convolution layer module, a graph-level pooling module and a graph-level output to an Actor-Critic network; The node encoder module is realized by a multi-layer feedforward neural network, and the normalized features of various nodes are input, sequentially passing through linear transformation, batch normalization and ReLU activation, and outputting a feature matrix of a unified embedding dimension after encoding each type of node; The heterogeneous graph construction module combines the encoding feature matrices of the four types of nodes with the edge index relationship provided by the environment to generate a heterogeneous graph object containing the requirements, observations, satellites and data transmission nodes and their associated edges for subsequent GNN processing; The heterogeneous convolution layer module comprises three heterogeneous convolution units, each of which performs message passing and aggregation on different relationships, and outputs updated node features after GraphNorm normalization, ReLU activation and Dropout processing; The graph-level pooling module performs global pooling on the node features of each type, and combines the results to generate a graph representation vector g that integrates all node and edge information, which is input into the Actor-Critic as a global feature.

4. The method of claim 3, wherein, The reinforcement learning network is based on the Actor-Critic framework; The reinforcement learning network takes the state vector s which is the concatenation of the graph neural network environment feature vector and the current candidate target local feature as input: the Actor network adopts a hierarchical policy network: the top-level decision network determines the macro action type, and after the shared sub-network generates shared features, it enters the corresponding secondary branch network according to the top-level decision to output specific planning requirements; the Critic network adopts a standard multi-layer MLP to obtain the state value from the input state vector s, which is used to measure the TD error; The top-level decision network takes the state feature vector s as input: first, it performs feature transformation through a combination of linear layers, LayerNorm, and ReLU, and the number of loop operations is one less than the number of specified linear layers; the last linear layer directly outputs a four-dimensional vector, which is converted to a probability distribution through softmax, and then the four types of top-level action decisions of observation, data transmission, erasure, and skip are obtained through multinomial sampling; If the top-level decision result is a skip action, the network directly returns a skip identifier and a null value, ending the current decision; If the top-level decision is one of the other three task types, the shared feature extraction network is started to process the same state feature input, and the shared feature extraction network adopts the same structure as the top-level network to obtain a shared feature vector; When the top-level decision is to execute an observation task, the observation task sub-network expands the shared feature vector to the same batch dimension as the observation task feature, and then concatenates it with the observation task feature in the feature dimension. After linear layer-LayerNorm-ReLU processing, the selection scores of each observation task are output. When the top-level decision is to execute a data transmission task, the data transmission task sub-network performs a similar processing flow to the observation task, and finally outputs the data transmission task number. When the top-level decision is to execute an erasure task, the satellite sub-network performs a similar processing flow to the observation task, and finally outputs the satellite number to select which satellite to execute the erasure task.

5. The method of claim 4, wherein, The specific way of step 3 is: An experience replay mechanism is used to store historical task data trajectories in a replay buffer, and small batches of samples are randomly extracted for network training; the objective function is defined as maximizing the long-term discounted cumulative reward obtained by the agent in the environment, and this goal is achieved by constructing the policy loss and value loss of the reinforcement learning network; In the training cycle, first randomly extract several trajectories from the replay buffer, splice into sample sequences in chronological order; after random rearrangement of the sample sequences, divide them into multiple training mini-batches according to the batch size m; for each mini-batch: the Actor network calculates the strategy distribution in parallel (|s), determines the action set a according to the strategy sampling, and records the action pair logarithmic probability log (a|s); the Critic network calculates the value prediction in parallel (s), and records the predicted value set; Temporal difference is calculated using the actual immediate reward and the value prediction of the next state, and the generalized advantage estimation is calculated recursively in the order from back to front, which is: wherein, represents the immediate reward returned by the environment after performing action at time step t, represents a discount factor, represents the value function of the state at the next time step; (1− ) represents a terminal flag modifier term, represents the value function of the state at the current time step; Computing the TD error where, denotes the advantage function estimate at time step t, denotes the temporal difference error at time step t, denotes the smoothing parameter for the generalized advantage estimate, denotes the advantage estimate at the next time step t+1; recursive advantage estimates , then compute the value of each step; after the advantage and target value computations are complete, pool all the samples into a set and shuffle randomly, then split into small batches of a set size for use in subsequent training; compute the value of each step; after the advantage and target value computations are complete, pool all the samples into a set and shuffle randomly, then split into small batches of a set size for use in subsequent training; Parameter updating is performed in mini-batches: first, calculate the advantage and package the samples, and perform multiple rounds of gradient descent for each mini-batch; the Actor uses PPO clipping target update and back propagation to the shared encoder and graph network, and the Critic uses mean square error update; at the end of each round, the current policy is copied as the old policy for the next round of ratio calculation; training is terminated after the maximum number of rounds, and the task planning result is output.

Images