Multi-satellite on-orbit cooperative scheduling method based on deep reinforcement learning and heuristic rule fusion

Abstract

The application discloses a kind of multi-satellite on-orbit collaborative scheduling method based on depth reinforcement learning and heuristic rule fusion, it is related to multi-satellite on-orbit intelligent scheduling field.The method includes: based on multi-agent depth deterministic policy gradient algorithm to establish collaborative decision model;Introduce priority experience replay mechanism to enhance learning efficiency;Design heuristic rule is used to guide agent decision-making;Through dynamic mixing coefficient, the adaptive fusion of reinforcement learning strategy and heuristic rule is realized;Reward shaping technology is used to accelerate agent learning convergence;According to satellite resource constraint and task priority, dynamically adjust scheduling scheme.The application can realize the intelligent collaborative scheduling of multi-satellite task under complex constraint condition, improve task completion rate and resource utilization efficiency, and has important significance for improving the autonomous decision-making ability of satellite system and the robustness of responding to unexpected situations.

Description

Technical Field

[0001] This invention relates to the field of intelligent scheduling of multiple satellites in orbit, and in particular to a multi-satellite in-orbit collaborative scheduling method based on the fusion of deep reinforcement learning and heuristic rules. Background Technology

[0002] With the continuous development of aerospace technology, multi-satellite systems have been widely used in Earth observation, communication relay, and scientific exploration. The multi-satellite cooperative scheduling problem refers to the rational allocation of observation tasks to multiple satellites under various constraints to achieve optimal overall performance. This problem is characterized by multiple agents, multiple constraints, large scale, and dynamic nature, making it a typical NP-hard problem.

[0003] In existing technologies, multi-satellite mission scheduling methods are mainly divided into two categories: heuristic methods and optimization methods. Heuristic methods obtain feasible solutions quickly by designing specific rules and strategies, but they often struggle to guarantee the quality of the solutions and lack flexibility when facing complex constraints. Optimization methods, such as genetic algorithms and particle swarm optimization, can obtain better solutions, but their high computational complexity makes them unsuitable for large-scale dynamic scheduling scenarios.

[0004] In recent years, deep reinforcement learning technology has demonstrated strong learning capabilities in complex decision-making problems, providing a new solution for multi-satellite collaborative scheduling. However, pure deep reinforcement learning methods have the following shortcomings: (1) low training efficiency, requiring a large number of trial and error samples to converge; (2) easy to fall into suboptimal strategies in the early stages of training; (3) difficulty in fully utilizing domain expert knowledge; and (4) may produce infeasible actions when faced with constraints.

[0005] In summary, the aforementioned techniques suffer from low training efficiency, slow convergence speed, and inability to fully utilize domain knowledge, making it difficult to quickly obtain high-quality scheduling strategies in practical applications. Summary of the Invention

[0006] One object of the present invention is to solve at least the above-mentioned problems and to provide at least the advantages that will be described later.

[0007] This invention discloses a multi-satellite on-orbit collaborative scheduling method based on the fusion of deep reinforcement learning and heuristic rules, relating to the field of intelligent on-orbit scheduling of multiple satellites. The method includes: establishing a collaborative decision-making model based on a multi-agent deep deterministic policy gradient algorithm; introducing a priority experience replay mechanism to enhance learning efficiency; designing heuristic rules to guide agent decision-making; achieving adaptive fusion of reinforcement learning strategies and heuristic rules through dynamic mixing coefficients; employing reward shaping technology to accelerate agent learning convergence; and dynamically adjusting the scheduling scheme according to satellite resource constraints and task priorities. This invention enables intelligent collaborative scheduling of multi-satellite missions under complex constraints, improving mission completion rate and resource utilization efficiency, and is of great significance for enhancing the autonomous decision-making capability and robustness of satellite systems in responding to emergencies.

[0008] To achieve these objectives and other advantages according to the present invention, a multi-satellite on-orbit cooperative scheduling method based on the fusion of deep reinforcement learning and heuristic rules is provided, comprising the following steps: Based on the satellite resources of the distributed multi-satellite Earth observation system, the planned observation tasks of each satellite toward the Earth observation target are obtained as the original scheduling scheme; a multi-satellite collaborative decision-making model is constructed based on the multi-agent deep deterministic policy gradient algorithm, and the collaborative decision-making model is used for distributed decision-making and collaboration among multiple satellites; A priority experience replay buffer is constructed to store joint experience samples of multiple agents, and a priority is assigned to each experience sample according to the temporal difference error. During training, weighted sampling is performed according to the priority, so that experience samples with greater learning value are used more frequently for policy updates. Design heuristic scheduling rules, which include: a prior probability allocation strategy based on satellite remaining resources, a task feasibility constraint masking strategy, and a hard rule post-processing strategy. The heuristic scheduling rules are used to provide domain knowledge guidance for reinforcement learning agents. Set dynamic mixing coefficient The dynamic mixing coefficient Used to control the fusion ratio between reinforcement learning policy output and heuristic rule prior; a larger fusion coefficient is set in the early stage of training to accelerate learning using heuristic rules, and the fusion coefficient is gradually reduced in the later stage of training to give full play to the autonomous decision-making ability of reinforcement learning, so as to achieve a smooth transition from heuristic guidance to fully autonomous learning. The reward shaping technique is used to enhance the original task reward. The reward shaping includes: giving an additional positive reward when the agent chooses to accept the task and the decision is consistent with the heuristic rule suggestion, and imposing a negative penalty when the agent chooses a decision that strongly conflicts with the heuristic rule. The reward shaping is used to smooth the reward signal and accelerate policy convergence during training. Based on the current satellite resource status and mission characteristics, the fused decision-making strategy is used to allocate tasks; the satellite resource status is updated and experience samples are stored in the priority experience replay buffer; training batches are sampled from the buffer according to priority, and the parameters of the actor network and critic network of each agent are updated; the above process is repeated until training converges to obtain the cooperative scheduling strategy.

[0009] Preferably, the construction of the multi-agent deep deterministic policy gradient algorithm includes the following steps: Each satellite agent is configured with an independent actor network and a critic network; the actor network takes into account the current local observation state of the satellite and outputs the probability distribution of the satellite's actions for the current task; the critic network takes into account the state and action information of all satellites and outputs a value assessment of the joint actions. A centralized training and distributed execution framework is adopted; during the training phase, the critic network can access global information to provide accurate value estimates; during the execution phase, each agent makes independent decisions based only on its own actor network and local observations, achieving distributed cooperation. The actor network uses the Gumbel-Softmax technique to transform the discrete action space into a differentiable form, enabling the policy gradient to be calculated via backpropagation; the critic network uses temporal difference learning to minimize the Bellman error of the value function.

[0010] Preferably, the implementation of the priority experience replay mechanism specifically includes: Store joint experience samples of multiple agents. Each experience sample contains: the state vector of all agents, the action vector of all agents, the reward value of all agents, the next state vector of all agents, the termination flag, and the heuristic prior probability. A priority is calculated for each empirical sample, based on the absolute value of the temporal difference error; samples with larger temporal difference errors indicate that the model predictions are inaccurate and have higher learning value, and are therefore assigned a higher sampling priority. Priority-weighted importance sampling is used; the priority is defined as... ,in For the first The temporal difference error of each sample, To prevent small constants with zero priority, Hyperparameters that control the priority level; the sampling probability is proportional to the priority. Importance sampling weights are introduced to correct sampling bias; the weights are... ,in For buffer size, Gradually increase the initial value to 1 to eliminate the bias. The sampling probability is used; when calculating the loss function, the loss for each sample is multiplied by the corresponding importance sampling weight. After each network update, the temporal difference error of the sampled experience is recalculated, and its priority in the buffer is updated so that the priority can dynamically reflect the learning value of the experience.

[0011] Preferably, the design of the heuristic scheduling rule specifically includes: Prior probability allocation strategy based on satellite remaining resources: When multiple satellites compete for the same task, resource scores are calculated based on the remaining time and storage resources of each satellite, and the resource scores are normalized to a prior acceptance probability; satellites with sufficient resources are assigned a higher acceptance probability to achieve load balancing. Mission feasibility constraint masking strategy: For invisible satellites or satellites with insufficient resources, mask the actions of accepting the mission in their action space, and retain a very small probability of exploration. This allows for limited exploration; the strategy ensures that the actions chosen by the agent satisfy physical constraints. Hard rule post-processing strategy: After the agent outputs an action, hard rules are applied based on task priority and resource scarcity; for example, when satellite resources are below a threshold, low-priority tasks are forcibly rejected; when a task is high-priority and feasible, its probability of being accepted is increased. The prior probability is The outputs of the formal and reinforcement learning policy networks are weighted and fused; the fusion formula is: fusion ,in For dynamic mixing coefficients, is the log-odds value, representing the unnormalized preference score for action selection, is the original action preference value output by the reinforcement learning policy network, is the original action preference value output by the heuristic rule, and is the fused decision preference value.

[0012] Preferably, the dynamic mixing coefficient The settings specifically include: setting the initial mixing coefficient. and final mixing coefficient The initial mixing coefficient is set to a relatively large value (e.g., 0.5 to 1.0), so that the early training mainly relies on heuristic rule guidance; the final mixing coefficient is set to a relatively small value (e.g., 0.5 to 0.2), so that the later training mainly relies on reinforcement learning autonomous decision-making. A linear annealing strategy is used to dynamically adjust the mixing coefficients; in the total number of training rounds In the middle, the first The mixing coefficient of the wheel is calculated as follows: This strategy enables a smooth transition from heuristic guidance to autonomous learning. The mixing coefficient can also be adjusted individually according to the learning progress of different satellite agents. For agents that learn slowly, a higher mixing coefficient is maintained to obtain more heuristic guidance. For agents that learn quickly, the mixing coefficient is reduced in advance to enhance autonomy. During the inference phase, the mixing coefficients can be dynamically selected based on task complexity and confidence level; for complex scenarios, a certain amount of heuristic fusion is retained to improve decision robustness.

[0013] Preferably, the adoption of the reward shaping technology specifically includes: Define the original task reward The original task reward is calculated based on the task completion status and resource consumption; when the agent successfully accepts and completes the task, it receives a positive reward, the size of which is related to the task priority and benefits. Calculate shaping items based on heuristic rules. An additional reward is given when the agent's decision matches the heuristic prior probability; specifically, if the heuristic suggestion has a probability of acceptance... If 0.7 and the agent chooses to accept, then If the probability of accepting a heuristic suggestion is... If 0.3 and the agent still chooses to accept, then ; in, The plasticity coefficient; For reward bonus parameters; For penalty parameters, This represents the heuristic prior acceptance probability.

[0014] Calculate the reward after shaping The shaped reward is used to train a reinforcement learning model to guide the agent toward rational decision-making while maintaining the original optimization objective. The original reward and the shaping reward are stored simultaneously in the experience replay buffer; the shaping reward is used during training, and the original reward is used for performance evaluation to ensure the authenticity of the algorithm optimization. The shaping coefficient, reward bonus parameter, and penalty parameter can be dynamically adjusted according to the training progress; a larger shaping coefficient is used in the early stage of training to accelerate convergence, and the shaping coefficient is gradually reduced in the later stage of training to avoid deviating from the true optimization goal.

[0015] Preferably, the execution of the cooperative scheduling strategy specifically includes: The state vectors of each satellite agent are constructed based on the satellite's current position, remaining resources, and planned tasks. The state vectors include: normalized remaining time, normalized remaining storage, execution duration of the current task, storage consumption of the current task, reward value of the current task, and normalized current time. The heuristic scheduler is queried to obtain the prior acceptance probability and feasibility mask for each satellite for the current task; the prior acceptance probability is calculated based on the satellite resource status and task characteristics; the feasibility mask identifies which satellites can physically perform the task. The state vector is input into the actor network of each agent to obtain the reinforcement learning policy output; the policy output is then mixed with the heuristic prior according to the current mixing coefficient. Perform fusion; apply a feasibility mask to constrain the action space; use Gumbel-Softmax to sample the final action; Apply heuristic post-processing rules to correct actions that violate hard constraints; execute the final action to update the satellite resource status; calculate the original reward and shaping reward; and store the joint experience samples in the priority experience replay buffer. Once the number of samples in the buffer reaches the minimum training batch size, training batches are sampled from the buffer according to priority; the sampled batches are used to update the actor and critic networks of all agents; the target network parameters are updated; the temporal difference error of the sampled experience is recalculated and its priority is updated.

[0016] The present invention has at least the following beneficial effects: 1. This invention introduces a priority experience replay mechanism, enabling experience samples with large temporal difference errors to be used for training more frequently, significantly improving sample utilization efficiency and learning convergence speed. Compared to traditional uniform sampling, priority experience replay allows the algorithm to learn the value function of key state-action pairs faster, reducing the number of training samples required.

[0017] 2. This invention designs heuristic rules based on resource states, providing domain knowledge guidance for reinforcement learning agents. In the early stages of training, the heuristic rules guide the agent to avoid obviously erroneous actions through prior probability allocation and feasibility constraint masks, effectively narrowing the exploration space and accelerating policy convergence. This fusion approach fully utilizes the experience and knowledge of human experts, compensating for the blindness of pure reinforcement learning methods in the early stages of training.

[0018] 3. The dynamic mixing coefficient mechanism proposed in this invention achieves a smooth transition from heuristic guidance to fully autonomous learning. Through a linear annealing strategy, the mixing coefficient gradually decreases from a high value in the early stages of training to a low value in the later stages. This allows the algorithm to gain acceleration from heuristic rules while ultimately breaking free from the limitations of the rules, leveraging the autonomous learning capabilities of reinforcement learning, and obtaining an optimal strategy that may surpass heuristic rules.

[0019] 4. The reward shaping technique employed in this invention smooths the reward signal and alleviates the sparse reward problem by adding additional terms related to the consistency of heuristic rules to the original reward. This technique provides the agent with denser learning signals without changing the optimal policy, further accelerating policy convergence.

[0020] 5. The collaborative decision-making model constructed based on the multi-agent deep deterministic policy gradient algorithm of this invention adopts a centralized training and distributed execution framework. During training, it utilizes global information to provide accurate value estimation, while during execution, each agent relies only on local observations to make decisions, achieving true distributed collaboration. This design ensures both the efficiency of policy learning and meets the distributed deployment requirements of practical systems.

[0021] 6. The method of this invention is significantly superior to existing technologies in several aspects: compared to purely heuristic methods, it can acquire better strategies through learning; compared to purely reinforcement learning methods, it greatly improves training efficiency and convergence speed; compared to simple rule + learning combinations, it achieves more intelligent adaptive fusion. Experimental results show that this method has achieved significant improvements in indicators such as task completion rate, resource utilization rate, and decision quality.

[0022] Other advantages, objectives and features of the present invention will become apparent in part from the following description, and in part from those skilled in the art through study and practice of the invention. Attached Figure Description

[0023] Figure 1 A flowchart illustrating the overall process of a multi-satellite on-orbit collaborative scheduling method based on the fusion of deep reinforcement learning and heuristic rules, provided for embodiments of the present invention. Figure 2 This is a schematic diagram of the multi-agent deep deterministic policy gradient algorithm structure provided in an embodiment of the present invention; Figure 3 A flowchart illustrating the priority experience replay mechanism provided in this embodiment of the invention; Figure 4 This is a schematic diagram illustrating the fusion of heuristic rules and reinforcement learning provided in an embodiment of the present invention. Detailed Implementation

[0024] The present invention will now be described in further detail with reference to the accompanying drawings, so that those skilled in the art can implement it based on the description.

[0025] It should be understood that terms such as “having,” “comprising,” and “including” as used herein do not imply the presence or addition of one or more other elements or combinations thereof.

[0026] See Figures 1 to 4This invention provides a multi-satellite on-orbit collaborative scheduling method based on the fusion of deep reinforcement learning and heuristic rules, comprising the following steps: Step 1: Construct a multi-satellite collaborative scheduling decision model To address the multi-agent, multi-constraint characteristics of the multi-satellite cooperative scheduling problem proposed in the background, a multi-satellite cooperative scheduling decision model is first constructed. The satellite set and its resource attributes (remaining time resources, remaining storage resources, planned task set, time window list) are defined, as are the task set and its attributes (arrival time, execution duration, storage consumption, task reward, priority, observable satellite set). The original scheduling scheme refers to the set of tasks planned by each satellite before the current scheduling time, including the occupation of satellite time windows and storage resources by each task. This scheme serves as a constraint on the current decision, determining the feasibility of each satellite accepting newly arriving tasks. Based on this, a state space, action space, and reward function are designed, formalizing the scheduling problem as a Markov decision process, providing a unified problem framework for subsequent reinforcement learning algorithms.

[0027] This step outputs: a multi-star cooperative scheduling Markov decision process model.

[0028] The specific steps are explained in detail below: S1.1: Given a set of satellites ,in Number of satellites. Each satellite It has the following attributes: (1) Remaining time resources : indicates satellite The remaining time available to execute the task within the current planning period; (2) Remaining storage resources : indicates satellite Currently available data storage capacity; (3) Set of planned tasks : Indicates that it has been allocated to a satellite The task set; (4) Time window list Recording satellites The time period already occupied is used for conflict detection.

[0029] S1.2: Given a set of tasks ,in The number of tasks. Each task It has the following attributes: (1) Arrival time :Task The time of arrival; (2) Execution duration Complete the task Time required; (3) Storage consumption : Execute the task The required storage capacity; (4) Task Rewards Complete the task Initial earnings that can be obtained (before being converted into actual earnings through a reward function); (5) Priority : Indicates task The importance level is 0 (high priority) or 1 (normal priority). (6) Set of observable satellites : Able to perform tasks A subset of satellites to be observed.

[0030] S1.3: Define the state space. For each satellite Its state vector contains 6 elements: in, For the current scene time, all satellites are saved and updated synchronously; Input task at current time The execution time and storage consumption are calculated, and if the satellite ultimately chooses to accept the mission, its remaining time and storage resources are deducted. The normalized resource ratio reflects the satellite's resource status, and the mission characteristics reflect the mission information for the current decision.

[0031] S1.4: Define the action space. For each mission, each satellite has two discrete action options: (1) Accepting a task: The action is represented as [1, 0]; (2) Reject task: The action is represented as [0, 1].

[0032] S1.5: Define the reward function. When the satellite Accept the mission Upon receiving the original reward: in, To accept the task at the same time The number of satellites This is the multi-star coordinated decrease coefficient. This represents the resource consumption penalty coefficient. The reward function design considers the diminishing returns of multi-star collaborative task completion and the resource consumption costs.

[0033] Step 2: Constructing the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) Algorithm Based on the decision-making model constructed in the first step, a collaborative decision-making framework is established using a multi-agent deep deterministic policy gradient algorithm. Each satellite agent is configured with an independent actor network and a critic network, employing a centralized training and distributed execution framework. The actor network uses the Gumbel-Softmax technique to achieve differentiable sampling of discrete actions, while the critic network receives global information for value evaluation. This step addresses the multi-agent collaborative decision-making problem proposed in the background art; however, the algorithm still suffers from low training efficiency at this stage.

[0034] Output: The MADDPG algorithm framework with distributed collaborative capabilities.

[0035] The specific details are as follows: S2.1: Construct an independent actor-critic network structure for each satellite agent.

[0036] (1) Actor Network : The input is a satellite. state vector After passing through two fully connected neural networks (with hidden layer dimensions of 128 and 100 respectively), the output action space is... Vectors. The actor network uses the ReLU activation function and employs the Gumbel-Softmax technique to achieve differentiable motion sampling.

[0037] (2) Critics Network The input is a concatenation of the state and action vectors of all satellites, passed through two fully connected neural networks (with hidden layer dimensions of 128 and 100 respectively), and the output is a scalar Q-value. The critic network uses a centralized training method, enabling it to access global information.

[0038] S2.2: Construct the target network. Create a corresponding target network for each agent's actor network and critic network. and This is used to stabilize the training process. The target network parameters are gradually updated to track the main network parameters using a soft update method. in, This is the soft update factor, which is usually set to 0.01.

[0039] S2.3: Define the loss function for the critic network. For the agent... The critics' losses are: Among them, the target value , This is the discount factor.

[0040] For the target critic network; The state at the next moment; For the next action generated by the target actor network, The reward actually received by the agent is what is subsequently set. .

[0041] S2.4: Define the loss function for the actor network. The actor loss consists of two parts: (1) Policy gradient term: (2) Regularization term: Used to prevent The value is too large, which leads to the instability of the value.

[0042] (3) Total loss: in, The actions output by the current actor online; The actions of other agents are obtained from buffer sampling. Regularization coefficient, with a default value of 1e-3.

[0043] S2.5: Applying the Gumbel-Softmax sampling technique. During the forward propagation process, the actor network output... Convert to one-hot action via Gumbel-Softmax: (1) Sampling Gumbel noise: in, These are uniformly distributed random numbers ranging from 0 to 1.

[0044] (2) Calculate the softening probability: in, For stable parameters, A higher value indicates a more uniform output distribution and a stronger exploratory nature. A lower value indicates that the output is closer to one-hot and more deterministic.

[0045] (3) Applying the straight-through technique: using one-hot(argmax( )), used during backward The gradient.

[0046] Because the argmax operation is not differentiable, this technique allows the policy gradient in the discrete action space to be computed via backpropagation.

[0047] Step 3: Build a Priority Experience Replay (PER) mechanism To address the low training efficiency of the MADDPG algorithm in the second step, a priority experience replay mechanism is introduced. A priority experience replay buffer is constructed to store joint experience samples from multiple agents, and a priority is assigned to each experience sample based on the temporal difference error. During training, priority-weighted sampling is performed, ensuring that experience samples with greater learning value are used more frequently for policy updates. Simultaneously, importance sampling weights are introduced to correct sampling bias. This step directly solves the low training efficiency problem mentioned in the background technology.

[0048] Output: The MADDPG+PER algorithm with efficient sample utilization.

[0049] The specific details are as follows: S3.1: Define the empirical sample structure. Each empirical sample... Include: in, For satellite To perform an action of accepting or rejecting the task. For satellite Is the resource level too low, causing the task to be stopped?

[0050] S3.2: Initialize the priority buffer. Create a circular buffer with capacity `capacity`, and maintain a priority array `priorities` to store the priority of each empirical sample. Initially, the priority of all samples is set to the maximum priority `max_priority` to ensure that new samples are sampled.

[0051] S3.3: Define the priority calculation method. For empirical samples Its priority Defined as: in For timing difference error, =1e-6 is a small constant to prevent it from having zero priority. =0.6 is a hyperparameter that controls the priority level. When =0, it degenerates into uniform sampling. When =1, sampling is performed entirely according to the TD error.

[0052] S3.4: Implement priority sampling. When sampling batch_size samples from the buffer: (1) Calculate the sampling probability: (2) Sample indexing based on probability distribution.

[0053] S3.5: Calculate importance sampling weights. To correct for the bias introduced by priority sampling, for each sampled... Calculate the importance sampling weights: in, The current size of the buffer. The importance sampling index is initially set to 0.4 and increases linearly to 1.0 during training. The weights are normalized to... .

[0054] S3.6: Apply importance sampling weights in the loss function calculation. For the loss of the commentator network: High-priority samples (with large TD errors) have a larger weight and have a greater impact on parameter updates; at the same time, the importance sampling weight corrects the sampling bias and ensures an unbiased estimate of the expected value.

[0055] S3.7: Update Sample Priority. After each network parameter update, recalculate the TD error of the sampled experience and update its priority in the buffer: in, The updated temporal difference error allows the prioritization to dynamically reflect the learning value of experience, as the network improves. Samples with large errors will be sampled and updated more frequently.

[0056] Step 4: Design Heuristic Scheduling Rules To address the problem of insufficient utilization of domain expert knowledge mentioned in the background technology, a heuristic scheduling rule is designed to provide domain knowledge guidance for reinforcement learning agents. The heuristic rule consists of three levels: (1) a priori probability allocation strategy based on the remaining satellite resources, which calculates the acceptance probability of each satellite according to the resource status; (2) a task feasibility constraint masking strategy to ensure that the agent's actions meet physical constraints; and (3) a hard rule post-processing strategy, which applies hard constraints according to task priority and resource scarcity. This step encodes domain expert knowledge into a rule form that can be used by computers.

[0057] Output: Heuristic scheduling rule set (prior probabilities, feasibility mask, post-processing rules).

[0058] The specific details are as follows: S4.1: Prior probability allocation strategy based on satellite remaining resources. The goal of this strategy is to achieve load balancing by rationally allocating acceptance probabilities based on resource status when multiple satellites compete for the same mission.

[0059] (1) Regarding the task traverse its set of observable satellites Calculate each satellite Resource score: in, For satellite Current remaining time resources For satellite Current remaining storage resources, This represents the initial time resource value for the satellite. This represents the initial storage resource value for the satellite.

[0060] (2) Normalize the resource score to the prior acceptance probability: Where Tanh is the hyperbolic tangent function.

[0061] (3) Set the base acceptance probability = 0.5, and adjust according to resource status: in, For the task Its observable satellite set The quantity.

[0062] (4) For satellites with limited resources, reduce their acceptance probability: (5) For high-priority missions, increase the acceptance probability of all feasible satellites: S4.2: Task Feasibility Constraint Masking Strategy. This strategy uses an action space mask to ensure that the actions selected by the agent satisfy physical constraints.

[0063] (1) For each mission-satellite pair ( Please check the following conditions: ① Satellite Is it in the task? Within the observable satellite set; ② Satellite Is there enough time remaining? ; ③ Satellite Is there enough remaining storage? ; ④ Are there any time window conflicts? Check [ , Whether or not The time periods already occupied overlap.

[0064] (2) If all conditions are met, set the feasibility mask to [1, 1], allowing both accept and reject actions; (3) If any condition is not met, set the feasibility mask as [ , 1], where =0.05 represents an extremely low probability of exploration, which essentially prohibits accepting actions but retains a tiny space for exploration.

[0065] (4) When selecting subsequent actions, apply the mask. Integration phase.

[0066] S4.3: Heuristic Priors Formal representation. To fuse with the output of the RL policy network, the prior probabilities are transformed into... form: in, It is a constant that approaches 0 to prevent log0 from occurring in logarithmic operations.

[0067] S4.4: Hard Rule Post-Processing Strategy. After the agent outputs an action, the following hard rules are applied: (1) Ensure the action format is correct: Each action must be [ , A binary list of the form ] has a sum of probabilities of 1; (2) For actions that are not feasible to accept (violation of physical constraints), force them to be changed to actions that reject [0, 1]; (3) For high-priority tasks, if satellite resources are sufficient and the prior probability is high, increase the acceptance probability; (4) For satellites with extremely scarce resources (below the threshold), low-priority tasks will be forcibly rejected.

[0068] Step 5: Implement the dynamic mixing coefficient mechanism To effectively integrate the reinforcement learning algorithm in the third step with the heuristic rules in the fourth step, a dynamic mixing coefficient mechanism is designed. The dynamic mixing coefficient controls the fusion ratio between the reinforcement learning policy output and the heuristic rule prior. A linear annealing strategy is adopted: a larger mixing coefficient is set in the early stages of training to accelerate learning using heuristic rules and avoid getting trapped in suboptimal policies; the mixing coefficient is gradually reduced in the later stages of training to leverage the autonomous decision-making ability of reinforcement learning. This step addresses the problem of easily getting trapped in suboptimal policies in the early stages of training, as mentioned in the background art.

[0069] Output: Fusion strategy.

[0070] The specific details are as follows: S5.1: Set the initial and final values ​​for the mixing coefficient. Based on experimental experience: # Early training relied entirely on heuristic rules # Some heuristic guidance is still retained in the later stages of training. higher (e.g., 0.5) can maintain a certain degree of decision robustness in the later stages of training, especially when facing scenarios outside the training set.

[0071] S5.2: Calculate the mixing coefficients for the current round using a linear annealing strategy. Assume the total number of training rounds is... The current training round is ,but: This linear annealing strategy enables a smooth transition from heuristic guidance to autonomous learning.

[0072] S5.3: Integrates RL policies and heuristic rules in action selection. The agent's actor network output. Heuristic rules provide The fusion method is as follows: Then, the fused Feasibility masking and Gumbel-Softmax sampling are applied.

[0073] S5.4: Personalized adjustments based on the agent's learning progress (optional). Monitor the average reward or TD error change rate of each agent, maintaining a higher rate for agents that learn more slowly. Reduce the risk of fast-learning agents in advance. : in, Represents intelligent agents The learning speed function has a value range of [0,1] and can be set according to the actual project scenario. This personalized adjustment can provide adaptive guidance intensity for agents with different learning abilities.

[0074] S5.5: Selection of mixing coefficients during the inference phase. After training, select appropriate coefficients based on the application scenario. value: (1) For routine scenarios, use = To achieve optimal performance; (2) For high-risk scenarios or scenarios outside the training set, increase (like = 0.7) to improve decision robustness; (3) For scenarios that pursue ultimate performance, reduce (like = 0.2) to fully leverage the self-learning ability of reinforcement learning.

[0075] Step Six: Employing Reward Shaping Techniques To further accelerate policy convergence, a reward shaping technique is employed based on the policy fusion in step five. A shaping term is calculated based on the consistency between the agent's decisions and heuristic priors: an additional positive reward is given when the decision aligns with the heuristic suggestion, and a negative penalty is imposed when the decision strongly conflicts with the heuristic suggestion. The shaped reward is used for training, while the original reward is used for evaluation, ensuring that the optimization objective does not deviate from the true benefit. This step further addresses the slow convergence problem while ensuring the optimality of the final policy.

[0076] Output: Shaping Reward .

[0077] The specific details are as follows: S6.1: Define the original task rewards The original reward, defined in step one, reflects the actual benefits and resource costs of completing the task.

[0078] S6.2: Design reward shaping term. The shaping term is calculated based on the consistency between the agent's decisions and heuristic priors: (1) Obtain the heuristic prior acceptance probability ; (2) Determine the actual action chosen by the agent. (0 indicates acceptance, 1 indicates rejection); (3) If 0.7 (heuristic strongly recommended) and = 0 (Agent chooses to accept): (4) If 0.3 (heuristics strongly recommends against) and = 0 (The agent still chooses to accept): (5) Other situations, = 0.

[0079] Among them, among them, The plasticity coefficient; For reward bonus parameters; This is a penalty parameter that can be adjusted based on the experimental results.

[0080] S6.3: Calculate the reward after shaping: . The shaped reward is used to train the reinforcement learning model, guiding the agent to learn towards reasonable decision-making.

[0081] S6.4: Store both types of rewards simultaneously in the experience replay buffer. Each experience sample contains: : [ , ..., # Used for training : [ , ..., # Used for evaluation Use during training Calculate TD error and update network parameters; use in performance evaluation We need to analyze the actual returns to ensure that the optimization goals are not deviated from.

[0082] S6.5: Dynamically adjust reward shaping parameters (optional). The shaping coefficient can be dynamically adjusted according to the training progress: in, Control the decay rate. Use a larger shaping coefficient in the early stages of training to accelerate convergence, and gradually decrease it in the later stages of training to avoid deviating from the true optimization objective.

[0083] Step 7: Execute the coordinated scheduling strategy Combining all the above steps, a complete collaborative scheduling strategy execution flow is formed: initializing the environment and agent network parameters; traversing the task set according to task arrival time; for each task, constructing a state vector, obtaining heuristic priors, fusing strategy selection actions, applying post-processing rules, executing actions, and storing experience; when the buffer samples are sufficient, sampling and training the network according to priority; repeating the above process until training converges. After training, a strategy model capable of achieving efficient multi-satellite collaborative scheduling under complex constraints is obtained.

[0084] Final output: A multi-star cooperative scheduling strategy that has been successfully trained and converged.

[0085] The specific details are as follows: S7.1: Initialize the environment and agent network parameters.

[0086] S7.2: For each episode in the training set: (1) Reset the environment and initialize the resource status of all satellites.

[0087] S7.3: Traverse the task set in order of task arrival time. : (1) Update the current environment time to the arrival time of the current task; (2) Update the state vectors of all satellites, including current mission information; (3) Call the heuristic scheduler to obtain the prior probability and feasibility mask; (4) Use the fusion strategy to select actions: for j in 1 to n: (5) Apply heuristic post-processing rules to correct actions; (6) Execute the action, and the environment returns. (7) Store the experience samples in the priority experience replay buffer: (8) Accumulated original rewards are used for performance statistics; (9) Update status = S7.4: When the number of samples in the buffer reaches the minimum training batch size, update the network parameters: (1) Sample batch_size empirical samples from the buffer according to priority and their importance sampling weights; (2) For each agent j: ① Calculate critic loss and update the critic network: Target Q value: in, for A concise representation.

[0088] TD error: loss: Gradient calculation: Parameter update: in, For importance sampling weights, For critics' learning rate.

[0089] renew parameter ② Calculate actor losses and update the actor network: Gradient calculation: renew parameter in, For actors' online learning rate, For the actions of other intelligent agents.

[0090] ③ Soft update target network: (3) Recalculate the sampling experience Error, update the priority in the buffer; (4) Update parameter: in, Incremental step size S7.5: Every fixed number of rounds, use the validation set to evaluate the performance of the current strategy and save the optimal model.

[0091] S7.6: After training is complete, use the trained actor network to perform inference on the test set and evaluate the final performance.

[0092] A specific example is as follows: The following example, which involves the coordinated scheduling of 200 tasks by 7 satellites, illustrates the method in this application in detail.

[0093] Experimental setup: (1) Number of satellites: n = 7, numbered from Satellite1 to Satellite7; (2) Initial time resources for each satellite: = 3600 seconds; (3) Initial storage resources for each satellite: = 150 units; (4) Number of tasks: 500 batches, each batch contains 100 / 200 / 300 tasks, and the tasks arrive dynamically within a 3600-second planning period; (5) Task execution time: Between [3, 6] seconds; (6) Task storage consumption: Between [3,6] units; (7) Task reward: The base reward is set between [1, 10], and the actual task reward is calculated based on priority and resource consumption; (8) Number of training rounds: = 500 episodes.

[0094] Hyperparameter settings: Buffer capacity: capacity = 1,000,000; Batch size: batch_size = 128; Minimum training batch size: minimal_size = 512; Actor learning rate: = 1e-3; Critics' learning rate: = 1e-3; Discount factor: = 0.95; Soft update coefficient: = 0.01; PER hyperparameter: = 0.6, Initial value = 0.4 Increment = 1e-4; Mixing coefficient: = 0.7, = 0.2; Bonus shaping parameter: = 0.5, = 0.5, = 0.5.

[0095] Performance comparison experiment: 1. In a scenario with 300 tasks per batch: (1) Pure MADDPG (no PER, no heuristics): After 500 training rounds, the task completion rate was 68%, the average reward was 532, and the overall variance was 180.7465. (2) This method (MADDPG+PER+heuristic fusion): task completion rate 78%, average reward 579, overall variance: 119.8655.

[0096] 2. In a scenario with 200 tasks per batch: (1) Pure MADDPG (no PER, no heuristics): After 500 training rounds, the task completion rate was 78%, the average reward was 495, and the overall variance was 455.9171. (2) This method (MADDPG+PER+heuristic fusion): task completion rate 90%, average reward 515, overall variance: 76.9889.

[0097] 2. In a scenario with 100 tasks per batch: (1) Pure MADDPG (no PER, no heuristics): After 500 training rounds, the task completion rate was 98%, the average reward was 285, and the overall variance was 314.5788. (2) This method (MADDPG+PER+heuristic fusion): task completion rate 98%, average reward 288, overall variance: 65.7387.

[0098] Experimental results show that the method of this invention significantly outperforms the comparative methods in terms of task completion rate, average reward, and overall variance. The priority experience replay mechanism improves sample utilization efficiency, leading to faster training convergence. The fusion of heuristic rules enables the agent to achieve better performance early in training, avoiding a large amount of ineffective exploration. The dynamic hybrid coefficient mechanism achieves a smooth transition from heuristic guidance to autonomous learning, and the final strategy retains the robustness of heuristic rules while possessing the optimization capabilities of reinforcement learning.

[0099] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A multi-satellite on-orbit collaborative scheduling method based on the fusion of deep reinforcement learning and heuristic rules, characterized in that, Includes the following steps: Step 1: Based on the satellite resources of the distributed multi-satellite Earth observation system, obtain the planned observation tasks of each satellite toward the Earth observation target as the original scheduling scheme, and construct a multi-satellite collaborative scheduling decision model; Step 2: Based on the multi-star collaborative scheduling decision model, a collaborative decision framework is established using the multi-agent deep deterministic policy gradient algorithm; Step 3: Construct a priority experience replay buffer to store joint experience samples of multiple agents, and assign priority to each experience sample according to the temporal difference error; Step 4: Design heuristic scheduling rules to provide domain knowledge guidance for reinforcement learning agents; Step 5: Control the fusion ratio between the reinforcement learning strategy output and the heuristic rule prior, and dynamically mix the reinforcement learning strategy and the heuristic rule; Step Six: Enhance the original task rewards using reward shaping techniques to optimize the speed and results of the final decision output; Step 7: Based on the current satellite resource status and mission characteristics, use the fused decision-making strategy to allocate tasks; In step four, the design of the heuristic scheduling rules includes the following steps: Prior probability allocation strategy based on satellite remaining resources: When multiple satellites compete for the same task, resource scores are calculated based on the remaining time and storage resources of each satellite, and the resource scores are normalized to a prior acceptance probability; satellites with sufficient resources are assigned a higher acceptance probability to achieve load balancing. Mission feasibility constraint masking strategy: For invisible satellites or satellites with insufficient resources, the actions to accept the mission are masked in their action space, and a very small probability of exploration is retained to allow limited exploration; this mission feasibility constraint masking strategy ensures that the actions selected by the agent satisfy physical constraints. Hard rule post-processing strategy: After the agent outputs an action, hard rules are applied according to task priority and resource scarcity. The prior probability is The outputs of the formal and reinforcement learning policy networks are weighted and fused; the fusion formula is: fusion ,in For dynamic mixing coefficients, The log-odds value represents the unnormalized preference score for action selection. To reinforce the original action preference values ​​output by the learning strategy network, The original action preference values ​​output by the heuristic rules. The fused decision preference value; In step five, the dynamic mixing coefficient is set. The steps include: Set the initial mixing coefficient and final mixing coefficient This means that early training relies primarily on heuristic rules, while later training relies primarily on reinforcement learning and autonomous decision-making. A linear annealing strategy is used to dynamically adjust the mixing coefficients; in the total number of training rounds In the middle, the first The mixing coefficient of the wheel is calculated as follows: This linear annealing strategy enables a smooth transition from heuristic guidance to autonomous learning. During the inference phase, the mixing coefficients can be dynamically selected based on task complexity and confidence level; for complex scenarios, a certain amount of heuristic fusion is retained to improve decision robustness.

2. The multi-satellite on-orbit collaborative scheduling method based on the fusion of deep reinforcement learning and heuristic rules as described in claim 1, characterized in that, In step two, the construction of the multi-agent deep deterministic policy gradient algorithm includes the following steps: Each satellite agent is configured with an independent actor network and a critic network; the actor network takes into account the current local observation state of the satellite and outputs the probability distribution of the satellite's actions for the current task; the critic network takes into account the state and action information of all satellites and outputs a value assessment of the joint actions. A centralized training and distributed execution framework is adopted; during the training phase, the critic network accesses global information to provide accurate value estimates; during the execution phase, each agent makes independent decisions based solely on its own actor network and local observations, achieving distributed collaboration. The actor network uses the Gumbel-Softmax technique to transform the discrete action space into a differentiable form, enabling the policy gradient to be calculated via backpropagation; the critic network uses temporal difference learning to minimize the Bellman error of the value function.

3. The multi-satellite on-orbit collaborative scheduling method based on the fusion of deep reinforcement learning and heuristic rules as described in claim 2, characterized in that, In step three, the implementation of the priority experience replay mechanism includes the following steps: Store joint experience samples of multiple agents. Each experience sample contains: the state vector of all agents, the action vector of all agents, the reward value of all agents, the next state vector of all agents, the termination flag, and the heuristic prior probability. Priorities are calculated for each empirical sample, and the priority is evaluated based on the absolute value of the time-series difference error. Priority-weighted importance sampling is used; the priority is defined as... ,in For the first The temporal difference error of each sample, To prevent small constants with zero priority, Hyperparameters that control the priority level; the sampling probability is proportional to the priority. Importance sampling weights are introduced to correct sampling bias; the weights are... ,in For buffer size, Gradually increase the initial value to 1 to eliminate the bias. The sampling probability is used; when calculating the loss function, the loss for each sample is multiplied by the corresponding importance sampling weight. After each network update, the temporal difference error of the sampled experience is recalculated, and its priority in the buffer is updated so that the priority can dynamically reflect the learning value of the experience.

4. The multi-satellite on-orbit collaborative scheduling method based on the fusion of deep reinforcement learning and heuristic rules as described in claim 3, characterized in that, Step six, the reward shaping, includes the following steps: Define the original task reward The original task reward is calculated based on the task completion status and resource consumption; when the agent successfully accepts and completes the task, it receives a positive reward, the size of which is related to the task priority and benefits. Calculate shaping items based on heuristic rules. When an agent's decision matches the heuristic prior probability, an additional reward is given. Calculate the reward after shaping The shaped reward is used to train a reinforcement learning model to guide the agent toward rational decision-making while maintaining the original optimization objective. The original reward and the shaping reward are stored simultaneously in the experience replay buffer; the shaping reward is used during training, and the original reward is used for performance evaluation to ensure the authenticity of the algorithm optimization. The shaping coefficient, reward bonus parameter, and penalty parameter can be dynamically adjusted according to the training progress. The shaping coefficient is increased in the early stage of training to accelerate convergence, and gradually decreased in the later stage of training to avoid deviating from the true optimization goal.

5. The multi-satellite on-orbit collaborative scheduling method based on the fusion of deep reinforcement learning and heuristic rules as described in claim 4, characterized in that, In step seven, the execution of the cooperative scheduling strategy includes the following steps: The state vectors of each satellite agent are constructed based on the satellite's current position, remaining resources, and planned tasks. The state vectors include: normalized remaining time, normalized remaining storage, execution duration of the current task, storage consumption of the current task, reward value of the current task, and normalized current time. The heuristic scheduler is queried to obtain the prior acceptance probability and feasibility mask for each satellite for the current task; the prior acceptance probability is calculated based on the satellite resource status and task characteristics; the feasibility mask identifies which satellites can physically perform the task. The state vectors are input into the actor network of each agent to obtain the reinforcement learning policy output; the policy outputs are then mixed with the heuristic priors according to the current mixing coefficients. Perform fusion; apply a feasibility mask to constrain the action space; use Gumbel-Softmax to sample the final action; Apply heuristic post-processing rules to correct actions that violate hard constraints; execute the final action to update the satellite resource status; calculate the original reward and shaping reward; and store the joint experience samples in the priority experience replay buffer. Once the number of samples in the buffer reaches the minimum training batch size, training batches are sampled from the buffer according to priority; the sampled batches are used to update the actor and critic networks of all agents; the target network parameters are updated; the temporal difference error of the sampled experience is recalculated and its priority is updated.

Images