Multi-agile earth observation satellite scheduling method based on deep q-learning and ensemble heuristic

Abstract

The application discloses an agile earth observation satellite planning method based on deep Q learning and heuristic rules, uses a DQN learning task allocation strategy, improves the efficiency and quality of task allocation, combines various heuristic rules, dynamically adjusts a scheduling strategy, and can adapt to different scene requirements; through a time relaxation task sliding insertion strategy, the execution time of the arranged task is allowed to be adjusted, task conflicts are reduced, and scheduling flexibility is improved; through an interaction mechanism between upper task allocation and lower task scheduling, the efficiency and stability of task allocation and scheduling are ensured.

Description

Technical Field

[0001] This invention belongs to the field of satellite mission planning technology, specifically relating to a method for planning agile Earth observation satellites, which aims to optimize satellite mission allocation and scheduling through deep reinforcement learning and integrated heuristic rules. Background Technology

[0002] Traditional methods for scheduling multiple agile Earth observation satellites are mainly based on exact algorithms, heuristic algorithms, and metaheuristic algorithms. These methods suffer from high computational costs, poor adaptability, and long solution times when dealing with large-scale missions and multi-satellite systems. For example, exact algorithms such as mixed-integer programming can only solve small-scale instances, while metaheuristic algorithms such as Adaptive Large Neighborhood Search (A-ALNS) and Greedy Stochastic Local Search (GRILS), although capable of handling larger-scale problems, have long solution times and inconsistent solution quality.

[0003] Existing methods struggle to adapt quickly to changes in mission and satellite numbers while maintaining solution quality. Furthermore, these methods often require complex calculations when handling time-dependent transformation time constraints. This invention aims to address the problems of low computational efficiency, poor adaptability, and unstable solution quality in existing multi-agile Earth observation satellite scheduling methods when dealing with large-scale missions and multi-satellite systems. Summary of the Invention

[0004] This invention provides an agile Earth observation satellite planning method based on deep Q-learning and heuristic rules to solve the problems of low computational efficiency, poor adaptability and unstable solution quality of existing multi-agile Earth observation satellite scheduling methods when dealing with large-scale missions and multi-satellite systems.

[0005] To achieve the above objectives, the technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows: An agile Earth observation satellite planning method based on deep Q-learning and heuristic rules, the planning method including task allocation, task scheduling and scheduling result feedback; The task allocation includes constructing an upper-layer task allocation model (TAM). The TAM is based on a deep Q-learning network (DQN) for task allocation. The TAM selects a satellite for task allocation based on the current task and satellite status. The selection is based on the Q-value predicted by the DQN network. The optimal satellite is selected for task allocation through a satellite greedy action selection strategy (SGAS). The task scheduling includes constructing a lower-level task scheduling model ECH. The ECH receives tasks assigned by TAM, selects the optimal visible time window (VTW) by combining multiple heuristic rules, and inserts the tasks into the satellite's execution sequence through a time-relaxed task sliding insertion strategy. The multiple heuristic rules include maximum time window (MTW), best imaging opportunity (BIO), and minimum insertion cost (MIC). The scheduling result feedback includes feeding the task scheduling results back to TAM for calculating rewards and updating the DQN network. Through the experience replay mechanism, TAM optimizes and adjusts the task allocation strategy.

[0006] Furthermore, the DQN employs fully connected layers, takes a state vector as input, and outputs the Q-value for each action. It continuously updates parameters through an experience replay mechanism to approximate the true Q-value. The mean squared error (MSE) is used to measure the loss between the network output and the target Q-value, and the network parameters are updated through a stochastic gradient descent (SGD) optimizer. The experience replay mechanism stores experience in an experience pool and trains the network by randomly sampling samples.

[0007] Furthermore, the SGAS includes Strategies and taboo strategies, as described Strategy is based on probability Randomly select actions, with probability Select the current optimal action; the taboo policy is to mark an action as taboo when it causes task scheduling failure, and temporarily prohibit the selection of that action in subsequent selections until a successful scheduling method is found.

[0008] Furthermore, the time-relaxed task sliding insertion strategy involves inserting task i at position j+1, sliding adjacent tasks to the left or right to create idle time for task i, and if there is sufficient idle time, the task insertion is successful, and the transition time of all tasks in the sequence is updated.

[0009] Furthermore, before task allocation, there is an initialization phase, which includes setting up the task set, the satellite set, and the VTW set for each task on each satellite.

[0010] The method of this invention utilizes DQN to learn task allocation strategies, thereby improving the efficiency and quality of task allocation. It integrates heuristic methods and combines multiple heuristic rules to dynamically adjust the scheduling strategy to adapt to different scenario requirements. Through a time-relaxed task sliding insertion strategy, it allows adjustment of the execution time of scheduled tasks, reducing task conflicts and improving scheduling flexibility. Through the interaction mechanism between upper-level task allocation and lower-level task scheduling, it ensures the efficiency and stability of task allocation and scheduling. Attached Figure Description

[0011] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of the planning method of the present invention; Figure 2 This is a flowchart illustrating the ECH integration heuristic rule of the present invention; Figure 3 This is a schematic diagram of the time-relaxed task sliding insertion strategy of the present invention. Detailed Implementation

[0013] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0014] This invention provides an agile Earth observation satellite planning method based on deep Q-learning and heuristic rules, such as... Figure 1 As shown, the planning method includes task allocation, task scheduling, and scheduling result feedback; The task allocation includes constructing an upper-layer task allocation model (TAM). The TAM is based on a deep Q-learning network (DQN) for task allocation. The TAM selects a satellite for task allocation based on the current task and satellite status. The selection is based on the Q-value predicted by the DQN network. The optimal satellite is selected for task allocation through a satellite greedy action selection strategy (SGAS). The task scheduling includes constructing a lower-level task scheduling model ECH. The ECH receives tasks assigned by TAM, selects the optimal visible time window (VTW) by combining multiple heuristic rules, and inserts the tasks into the satellite's execution sequence through a time-relaxed task sliding insertion strategy. The multiple heuristic rules include maximum time window (MTW), best imaging opportunity (BIO), and minimum insertion cost (MIC). The scheduling result feedback includes feeding the task scheduling results back to TAM for calculating rewards and updating the DQN network. Through the experience replay mechanism, TAM optimizes and adjusts the task allocation strategy.

[0015] Furthermore, the DQN employs fully connected layers, takes a state vector as input, and outputs the Q-value for each action. It continuously updates parameters through an experience replay mechanism to approximate the true Q-value. The mean squared error (MSE) is used to measure the loss between the network output and the target Q-value, and the network parameters are updated through a stochastic gradient descent (SGD) optimizer. The experience replay mechanism stores experience in an experience pool and trains the network by randomly sampling samples.

[0016] Furthermore, the SGAS includes Strategies and taboo strategies, as described Strategy is based on probability Randomly select actions, with probability Select the current optimal action; the taboo policy is to mark an action as taboo when it causes task scheduling failure, and temporarily prohibit the selection of that action in subsequent selections until a successful scheduling method is found.

[0017] Furthermore, the time-relaxed task sliding insertion strategy involves inserting task i at position j+1, sliding adjacent tasks to the left or right to create idle time for task i, and if there is sufficient idle time, the task insertion is successful, and the transition time of all tasks in the sequence is updated.

[0018] Furthermore, before task allocation, there is an initialization phase, which includes setting up the task set, the satellite set, and the VTW set for each task on each satellite.

[0019] The planning method of the embodiments of the present invention is as follows: Figure 1 As shown, the model comprises a two-stage process. In a Markov Decision Process (MDP), an action is selected in the Task Assignment Model (TAM) via a DQN at each step, corresponding to assigning a task. Then, the lower layer applies an ECH to schedule the task and compute the reward. During training, the model iteratively optimizes the TAM-ECH through feedback and evaluation.

[0020] TAM selects a satellite for mission allocation based on the current mission and satellite status. The selection is based on the Q-value predicted by the DQN network, and the exploration and utilization are balanced through the SGAS strategy.

[0021] Markov decision processes (MDPs) include: State Space (S): The state space includes static and dynamic elements of the task. Static elements include the task index, profit, and duration; dynamic elements include the satellite's current task load and the number of conflicting tasks.

[0022] Action space (A): Action space is the selection of satellites, that is, which satellite to choose to perform the current task.

[0023] Reward Function (R): The reward function is based on the success or failure of task scheduling. Successfully scheduled tasks receive a positive reward, while failed tasks are penalized.

[0024] Transition probability matrix (T): The probability of transitioning from the current task to the next task.

[0025] Deep Q-learning networks (DQNs) include: Network Structure: DQN uses fully connected layers, taking a state vector as input and outputting the Q-value for each action. The network continuously updates its parameters through an experience replay mechanism to approximate the true Q-value.

[0026] Q-value function optimization: The mean squared error (MSE) is used to measure the loss between the network output and the target Q-value, and the network parameters are updated through a stochastic gradient descent (SGD) optimizer.

[0027] Experience replay mechanism: Store experience in an experience pool and train the network by randomly selecting samples to improve the stability and generalization ability of the network.

[0028] The Satellite Greedy Action Selection Strategy (SGAS) includes: Strategy: Based on probability Randomly select actions, with probability Choose the best action at present, balancing exploration and exploitation.

[0029] Taboo Strategy: When an action causes task scheduling to fail, it is marked as taboo and temporarily prohibited from being selected in subsequent choices until a successful scheduling solution is found. The ECH integration heuristic process provided by the embodiments of the present invention is as follows: Figure 2 As shown, ECH selects the optimal virtual time window for executing tasks, and then uses the insertion function to place tasks into the sequence at the appropriate positions, thus arriving at a solution.

[0030] The task scheduling model ECH receives tasks assigned by TAM, combines multiple heuristic rules including maximum time window, best imaging opportunity and minimum insertion cost, selects the optimal visible time window (VTW), and inserts the task into the satellite's execution sequence through a time-relaxed task sliding insertion strategy.

[0031] ECH is responsible for efficiently scheduling the tasks assigned to each satellite. By combining various heuristic rules, ECH selects the best virtual time window for executing tasks and then uses the insertion function to place tasks into the sequence at the appropriate position, thereby arriving at a solution.

[0032] Among them, the heuristic rules include: Maximum Time Window (MTW): Select the VTW with the longest duration to increase the range of execution time options for the task.

[0033] Best Imaging Opportunity (BIO): Select the VTW with the shortest idle time to optimize the use of idle time.

[0034] Minimum Insertion Cost (MIC): Select the VTW with the minimum insertion cost to reduce the impact of task insertion on the overall scheduling.

[0035] The ECH process may include: Use rules in stages: Use MTW rules in the early stages of scheduling, and use BIO and MIC rules in the later stages to adapt to the scheduling needs of different stages.

[0036] Weighted composite rule: The evaluation indices of different heuristic rules are weighted and summed to generate a new composite index, and the VTW with the highest composite index is selected.

[0037] The time-relaxed task sliding insertion strategy of embodiments of the present invention is as follows: Figure 3 As shown, the time-relaxed task sliding insertion strategy involves inserting task i at position j+1, sliding adjacent tasks to the left or right to create idle time for task i. If there is sufficient idle time, the task insertion is successful, and the transition times of all tasks in the sequence are updated. Insertion position detection can be based on the FIFO (First-In, First-Out) and TI (Triangle Inequality) principles to detect all feasible insertion positions. The insertion position can be chosen to minimize the task sliding time, thus reducing the impact on the overall scheduling.

[0038] The planning method of the present invention can be implemented through the following steps: (1) Initialization: Set up the task set, satellite set and VTW set for each task on each satellite.

[0039] (2) Task allocation: TAM uses DQN to select satellites based on MDP for task allocation and selects actions through SGAS strategy.

[0040] (3) Task scheduling: The ECH receives the tasks assigned by the TAM, selects the optimal VTW, and inserts the tasks into the satellite's execution sequence through a time-relaxed task sliding insertion strategy.

[0041] (4) Reward calculation: Calculate the reward based on the task scheduling result and update the DQN network.

[0042] (5) Experience replay: Store the experience in the experience pool and train the DQN network through the experience replay mechanism.

[0043] (6) Iterative optimization: Repeat the above steps until all tasks are assigned and scheduled. TAM and ECH continuously optimize task assignment and scheduling strategies through feedback mechanisms.

[0044] The planning method of this invention has the advantages of high efficiency, adaptability and stability. TAM-ECH outperforms existing algorithms in both optimization speed and quality, with an average profit margin 11.7% higher than GRILS and a shorter solution time. TAM-ECH can adapt to missions of different sizes and the number of satellites, and performs particularly well in large-scale mission scenarios. By integrating heuristic methods, TAM-ECH can provide stable and high-quality scheduling solutions in different scenarios.

[0045] The method of this invention utilizes DQN to learn task allocation strategies, improving the efficiency and quality of task allocation. It integrates heuristic methods and combines multiple heuristic rules to dynamically adjust the scheduling strategy to adapt to different scenario requirements. Through a time-relaxed task sliding insertion strategy, it allows adjustment of the execution time of scheduled tasks, reduces task conflicts, and improves scheduling flexibility. The interaction mechanism between upper-level task allocation and lower-level task scheduling ensures the efficiency and stability of task allocation and scheduling.

[0046] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An agile Earth observation satellite planning method based on deep Q-learning and heuristic rules, characterized in that, The planning method includes task allocation, task scheduling, and scheduling result feedback; The task allocation includes constructing an upper-layer task allocation model (TAM). The TAM is based on a deep Q-learning network (DQN) for task allocation. The TAM selects a satellite for task allocation based on the current task and satellite status. The selection is based on the Q-value predicted by the DQN network. The optimal satellite is selected for task allocation through a satellite greedy action selection strategy (SGAS). The task scheduling includes constructing a lower-level task scheduling model ECH. The ECH receives tasks assigned by TAM, selects the optimal visible time window (VTW) by combining multiple heuristic rules, and inserts the tasks into the satellite's execution sequence through a time-relaxed task sliding insertion strategy. The multiple heuristic rules include maximum time window (MTW), best imaging opportunity (BIO), and minimum insertion cost (MIC). The scheduling result feedback includes feeding the task scheduling results back to TAM for calculating rewards and updating the DQN network. Through the experience replay mechanism, TAM optimizes and adjusts the task allocation strategy.

2. The method according to claim 1, characterized in that, The DQN employs fully connected layers, takes a state vector as input, and outputs the Q-value for each action. It continuously updates parameters through an experience replay mechanism to approximate the true Q-value. The mean squared error (MSE) is used to measure the loss between the network output and the target Q-value, and the network parameters are updated through a stochastic gradient descent (SGD) optimizer. The experience replay mechanism stores experience in an experience pool and trains the network by randomly sampling samples.

3. The method according to claim 1, characterized in that, The SGAS includes Strategies and taboo strategies; the aforementioned Strategy is based on probability Randomly select actions, with probability Select the current optimal action; the taboo policy is to mark an action as taboo when it causes task scheduling failure, and temporarily prohibit the selection of that action in subsequent selections until a successful scheduling method is found.

4. The method according to claim 1, characterized in that, The time-relaxed task sliding insertion strategy involves inserting task i at position j+1, sliding adjacent tasks to the left or right to create idle time for task i, and if there is enough idle time, the task insertion is successful, and the transition time of all tasks in the sequence is updated.

5. The method according to claim 1, characterized in that, Before task allocation, there is an initialization phase, which includes setting up the task set, satellite set, and VTW set for each task on each satellite.

Images