A reinforcement and neighborhood search method and system for scheduling satellite observation missions

Abstract

The application relates to the technical field of satellite scheduling, and particularly discloses a reinforcement and neighborhood search method and system for scheduling satellite observation tasks, wherein the method comprises the following steps: preprocessing satellite observation tasks; canceling original scheduling plans after task screening is completed; training a neural network by using a deep reinforcement learning training model; inputting the preprocessed satellite observation tasks into the neural network to output a scheduling scheme; and delivering the scheduling scheme output by the neural network as an initial solution to a large neighborhood search algorithm to output an optimized scheme.

Description

Technical Field

[0001] This invention relates to the field of satellite scheduling technology, and specifically to an enhanced method and system for scheduling satellite observation missions and performing neighborhood search. Background Technology

[0002] The rapid development of Earth observation satellites can provide richer services to an increasing number of enterprise, government, and military users. The rational scheduling of satellite Earth observation missions is crucial to meeting user needs and has already seen strong progress. However, online scheduling of large-scale emergency missions remains a complex and ongoing problem. Although the academic community has made significant efforts to solve the satellite Earth observation mission scheduling problem, large-scale emergency missions still present a dual challenge: on the one hand, user-submitted missions typically arrive in a dense, short period, and their arrival time and quantity cannot be predicted in advance. This high dynamism leads to a constantly changing optimization environment, requiring algorithms to have strong adaptability; on the other hand, as the scale of missions and satellites grows, the solution space explodes exponentially, but the allowable delay for scheduling decision optimization is extremely short.

[0003] Therefore, how to efficiently schedule large-scale emergency missions online using satellites that cannot observe the Earth is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] To achieve the objectives of this invention, this application provides an enhancement and neighborhood search method for scheduling satellite observation tasks, comprising: Step S1: Preprocess the satellite observation task; Step S2: Train the neural network using a deep reinforcement learning training model, including transforming the process of mapping tasks to resources in the deep reinforcement learning model into a task serialization subprocess and a resource selection subprocess, in order to compress the search space that grows exponentially with the increase of tasks. Step S3: Input the preprocessed satellite observation task into the neural network and output a scheduling scheme; Step S4: Pass the scheduling scheme output by the neural network as the initial solution to the large neighborhood search algorithm to output the optimized scheme; The large neighborhood search algorithm includes the following: for two Earth observation tasks on a satellite, if the visible time windows of the two Earth observation tasks overlap, a new feasible execution sequence can be obtained by exchanging the execution order of the tasks; conversely, if the end time of the visible time window of one task is earlier than the start time of the visible time window of another task, the former is executed before the latter.

[0005] In some specific embodiments, step S2 includes: Construct an encoder-decoder-based policy network; The policy network is trained using the REINFORCE method. Update network parameters during the training process; Through multiple iterations of training, the neural network learns the optimal task scheduling strategy.

[0006] In some specific embodiments, in step S3, the deep reinforcement learning training model sequentially creates a scheduling scheme based on an encoder-decoder policy network to map satellite status and waiting task information into an embedded representation.

[0007] In some specific embodiments, the preprocessing includes: adding new satellite observation tasks; updating the satellite scheduling time limit; filtering out tasks whose execution instruction upload time is later than the current time limit; and canceling their original scheduling plans after the task filtering is completed.

[0008] In some specific embodiments, the destruction ratio, repair ratio, and exchange ratio of the large neighborhood search algorithm are all set to 0.1.

[0009] To achieve the objectives of this invention, this application also provides an enhancement and neighborhood search system for scheduling satellite observation missions, comprising: Preprocessing module: Used to preprocess satellite observation tasks; Neural Network Training Module: Used to train neural networks using deep reinforcement learning training models, including transforming the process of mapping tasks to resources in deep reinforcement learning models into task serialization subprocesses and resource selection subprocesses, in order to compress the search space that grows exponentially with the increase of tasks. Scheduling scheme output module: used to input the preprocessed satellite observation tasks into the neural network and output the scheduling scheme; Scheduling scheme optimization module: used to pass the scheduling scheme output by the neural network as the initial solution to the large neighborhood search algorithm, and output the optimized scheme; The large neighborhood search algorithm includes the following: for two Earth observation tasks on a satellite, if the visible time windows of the two Earth observation tasks overlap, a new feasible execution sequence can be obtained by exchanging the execution order of the tasks; conversely, if the end time of the visible time window of one task is earlier than the start time of the visible time window of another task, the former is executed before the latter.

[0010] In some specific embodiments, the neural network training module is used to perform the following steps: Construct an encoder-decoder-based policy network; The policy network is trained using the REINFORCE method. Update network parameters during the training process; Through multiple iterations of training, the neural network learns the optimal task scheduling strategy.

[0011] In some specific embodiments, in the neural network training module, the deep reinforcement learning training model sequentially creates scheduling schemes based on an encoder-decoder policy network to map satellite status and waiting task information into embedded representations.

[0012] In some specific embodiments, the preprocessing includes: adding new satellite observation tasks; updating the satellite scheduling time limit; filtering out tasks whose execution instruction upload time is later than the current time limit; and canceling their original scheduling plans after the task filtering is completed.

[0013] In some specific embodiments, the destruction ratio, repair ratio, and exchange ratio of the large neighborhood search algorithm are all set to 0.1.

[0014] The beneficial effects of the above technical solution are as follows: This invention combines deep reinforcement learning with large neighborhood search to effectively adapt to the dynamic and resource-constrained characteristics of satellite mission scheduling environments. A novel emergency scheduling framework based on rolling time domain is developed, transforming dynamic online scheduling into a series of static sub-problems. This method enhances optimization decisions based on current needs and system state. Furthermore, a novel decoding mechanism is designed within deep reinforcement learning to simplify the scheduling process and significantly reduce the output dimensionality caused by the exponential growth of the search space. Additionally, three novel destruction and repair operators are designed to further improve the optimization efficiency of large neighborhood search and adapt to the highly dynamic nature of the environment. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of the method for deep enhancement and neighborhood search for scheduling satellite observation tasks provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the deep enhancement and neighborhood search device for scheduling satellite observation tasks provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the operational framework of an Earth observation satellite system. Figure 4 This is a diagram illustrating satellite attitude adjustment; Figure 5 This is a diagram illustrating the problem-solving process provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the deep reinforcement learning construction solution provided in an embodiment of the present invention; Figure 7 This is a trend graph of the revenue as a function of the search process provided in the first implementation of the present invention. Figure 8 This is a trend graph of the revenue as a function of the search process provided in the second embodiment of the present invention; Figure 9 This is a trend graph of the revenue as a function of the search process provided in the embodiments of the present invention; Figure 10 This is a trend graph of the revenue provided by the fourth implementation of the search process in this invention. Detailed Implementation

[0017] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0018] Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar symbols denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the invention, and should not be construed as limiting the invention.

[0019] Example 1 One embodiment of the present invention provides an enhancement and neighborhood search method for scheduling satellite observation missions, referring to... Figure 1 , Figure 3 As shown, it includes: Step S1: Preprocess the satellite observation task; Preprocessing includes: Add new satellite observation missions; Update the upper limit of satellite scheduling time; Filter out tasks whose execution command upload time is later than the current time limit; Once the task selection is complete, cancel its original scheduling plan.

[0020] Specifically, when a new task arrives, the framework will preprocess the newly arrived task and waiting tasks: 1) Add the new task to... ;2) Update the scheduling time limit;3) Filter out tasks whose execution instruction upload time is later than the current time limit, including the set Used to store the filtered tasks, 4) After the task selection is completed, cancel its original scheduling plan.

[0021] The satellite scheduling problem in this application focuses on the scheduling of observation missions performed by satellite constellations. The increasing demand for observations creates a conflict between the available observation missions and limited satellite resources. The research objective is to obtain feasible solutions (scheduling schemes) that maximize certain objective functions while adhering to key physical and operational constraints.

[0022] Unlike existing online scheduling methods, which immediately upload execution commands to the corresponding satellites once an Earth observation mission is scheduled, the IDRLNS method proposed in this application places all newly arriving and pending missions into a rolling time domain. This allows for further optimization of the scheduling plans for these missions, potentially leading to higher overall performance. By utilizing the rolling time domain, this application transforms the online Earth observation satellite scheduling problem into a series of static problems. This framework also ensures that each static problem can be solved by fully utilizing all available information during each optimization iteration.

[0023] Table 1 Algorithm 1 The overall framework of the proposed algorithm is shown in Algorithm 1 in Table 1. For emergency mission scenarios, the task scheduling problem can be transformed into a Markov decision process: the state space represents the feature sets of the task and the satellite, and the action space corresponds to the selection between the task and the satellite. Based on the current state, this application utilizes a deep reinforcement learning policy network to perform action selection. The policy network is trained offline using a training algorithm (row 1), and the specific training process is shown in Algorithm 2.

[0024] In the proposed scheduling framework, a rolling time domain (denoted as...) is adopted. This stores the set of tasks corresponding to execution instructions that have not yet been uploaded to the satellite. The rolling time domain is initialized to an empty set (line 2). The parameters in line 3... This indicates the deadline for the next round of scheduling optimization, which is determined by the upload time of the execution instructions for newly arrived tasks.

[0025] When a new task arrives, the framework will preprocess the newly arrived task and waiting tasks: 1) Add the new task to... (Line 5); 2) Update the scheduling time limit (Line 6); 3) Filter out tasks whose execution instruction upload time is later than the current time limit (Lines 8-10). The set... Used to store the filtered tasks, 4) After task filtering is completed, cancel its original scheduling plan (line 11).

[0026] After preprocessing, the framework enters the optimization phase. First, it calls a deep reinforcement learning-based training model to quickly generate a scheduling scheme for the selected tasks (line 12). The training model's latency for scheduling a single task is in the millisecond range. Then, the solution obtained from the training model is passed as the initial solution to the large neighborhood search algorithm (line 13). To improve the efficiency of the LNS search task execution sequence, a heuristic rule is designed to filter the set of exchangeable tasks for each task (line 14), the specific process of which is shown in Algorithm 3. The Middle The element records the elements that can be compared with the first element. The framework then swaps the execution order of a subset of tasks. Following this, it enters the main loop for searching the large neighborhood, continuing until the current time. Reaching the time limit (Lines 15-20). In each iteration, a function containing three destruction and repair operators is called to generate a new scheduling scheme (line 16), which then updates the current scheduling scheme and the optimal scheduling scheme. (Lines 17-20). Among them... and These represent random numbers between 0 and 1, and a pre-set threshold, respectively.

[0027] Step S2: Train the neural network using a deep reinforcement learning training model, including transforming the process of mapping tasks to resources in the deep reinforcement learning model into a task serialization subprocess and a resource selection subprocess, in order to compress the search space that grows exponentially with the increase of tasks.

[0028] After preprocessing, the framework enters the optimization phase. First, it calls a deep reinforcement learning-based training model to quickly generate scheduling schemes for the selected tasks. The trained model's latency for scheduling a single task is in the millisecond range.

[0029] In one specific embodiment of the present invention, step S2 includes: Construct an encoder-decoder-based policy network; The policy network is trained using the REINFORCE method. Update network parameters during the training process; Through multiple iterations of training, the neural network learns the optimal task scheduling strategy.

[0030] In one specific embodiment of the present invention, the process of mapping tasks to resources in a deep reinforcement learning model is transformed into two sub-processes: task serialization and resource selection, thereby compressing the search space that grows exponentially with the increase of tasks. The deep reinforcement learning training model includes: an input layer, a hidden layer, and an output layer; Set the hidden layers to two 128-dimensional layers; The learning rate for the deep reinforcement learning training model is set to 0.001; Add Dropout to each hidden layer.

[0031] Specifically, the scheduling of satellite Earth observation missions is a typical sequential decision-making process, which is modeled as a Markov decision process (MDP) in this application. Figure 5 The solution process for this problem is demonstrated. In each decision-making process, the agent determines the solution based on the current state (…). ) through strategy Select the action to perform ( The execution of an action will trigger an environmental shift from the current state, consisting of the satellite's state and the tasks to be performed. ) to the next state ( The system transfers data, and simultaneously, the intelligent system receives a reward determined by the impact of its current action on the environment. ).

[0032] When solving the Earth observation mission scheduling problem, Markov Decision Processes (MDPs) mainly involve the following elements: state, action, transition rule, and reward, which can be described as a quadruple. : At a certain point in time The satellite's status and pending mission information. Satellite status includes each satellite's scheduled but not yet completed missions, remaining storage and energy, and attitude adjustment rate. Pending mission information includes their attributes and the visible time window for each satellite.

[0033] This application defines the action as two parts: selecting a waiting task and selecting a satellite. At a given time... These two sub-actions are executed sequentially, and then converted into a task execution sequence and a mapping between the task and the satellite.

[0034] The state transition function determines the state transition. Next joint action This leads to a transition to a state. The probability of.

[0035] : It is a reward function. This application defines time points. Rewards The priority of successfully scheduling tasks, i.e. Then, the cumulative reward is calculated as follows: ,in It is the stopping time step.

[0036] To solve the above model, this application employs an encoder-decoder-based policy network to sequentially create scheduling schemes. The encoder maps satellite state and waiting task information into an embedded representation. Specifically, the encoder performs embedding operations through multiple attention layers, including a series of multi-head attention layers and feedforward layers, as well as adding skip connections and batch normalization operations. The task selection decoder and satellite selection decoder map the encoder's embedded representation into a scheduling scheme.

[0037] The goal of an intelligent agent is to learn an optimal policy using a deep neural network. To maximize its long-term cumulative benefits, this application uses the REINFORCE method to train the policy network. For a case study of scheduling Earth observation missions for satellites... Generate solutions The probability is This application will Negative cumulative reward is defined as cost The loss function of the deep reinforcement learning model is defined as the expected value of the cost, as shown below: This application uses a baseline to optimize cost expectations: in, The baseline indicates that the cumulative return is negative.

[0038] The main process of training a deep neural network is shown in Algorithm 2 in Table 2.

[0039] Table 2 Algorithm 2 As described in Algorithm 2, the parameters of the policy network It is randomly initialized (line 1). Then, parameters are assigned to the baseline network. (Line 2). Randomly generated in each cycle. For each batch of cases, the algorithm generates a scheduling solution using the current policy network (lines 6-11). Then, the algorithm optimizes the parameters of the policy network based on the difference between the loss values ​​of the policy network and the baseline network (lines 12-15). If the current policy network is better, the baseline network is updated (lines 16-17).

[0040] Unlike previous optimization searches performed in an exponentially growing combinatorial space, the deep reinforcement learning-based scheduling framework proposed in this application divides the optimization of task sequences and task-to-satellite mappings into two steps: task selection and satellite selection. Tasks are selected progressively to construct the task sequence, and tasks and satellites are selected in one step to form their mapping. Figure 6 As shown, the construction of a solution can be transformed into a sequential decision-making process. In each step, the policy determines a task and a satellite based on the current state. At this point, the selected task is added to the end of the sequence queue, and the task and satellite selected in the same stage constitute a partial solution for scheduling. Simultaneously, the system state is updated. The above process is repeated until a complete scheduling scheme is constructed.

[0041] Step S3: Input the preprocessed satellite observation task into the neural network and output the scheduling scheme.

[0042] Step S4: Pass the scheduling scheme output by the neural network as the initial solution to the large neighborhood search algorithm to output the optimized scheme.

[0043] In one specific embodiment of the present invention, the large neighborhood search algorithm includes: For two Earth observation missions on a satellite, if the visible time windows of the two Earth observation missions overlap, a new feasible execution sequence can be obtained by exchanging the execution order of the missions; conversely, if the end time of the visible time window of one mission is earlier than the start time of the visible time window of another mission, the former takes precedence over the latter.

[0044] In one specific embodiment of the present invention, the destruction ratio, repair ratio, and exchange ratio of the large neighborhood search algorithm are all set to 0.1.

[0045] Subsequently, the solutions obtained from the trained model are used as initial solutions and passed to the large neighborhood search algorithm. To improve the efficiency of the LNS search task execution sequence, a heuristic rule is designed to filter the commutative task set for each task, as shown in Algorithm 3. The Middle The element records the elements that can be compared with the first element. The framework then swaps the execution order of a subset of tasks. Following this, it enters the main loop for searching the large neighborhood, continuing until the current time. Reaching the time limit In each iteration, a function containing three destruction and repair operators is called to generate a new scheduling scheme, which in turn updates the current scheduling scheme and the optimal scheduling scheme. .in and These represent random numbers between 0 and 1, and a pre-set threshold, respectively.

[0046] Specifically, for two Earth observation missions on a satellite, if their visible time windows overlap, a new feasible execution sequence can be obtained by exchanging the execution order of the missions; conversely, if the end time of the visible time window of one mission is earlier than the start time of the visible time window of another mission, the former must be executed before the latter. Based on the above characteristics, this application selects a corresponding set of interchangeable missions for each Earth observation mission on each satellite.

[0047] As shown in Algorithm 3 of Table 3, initialize a A matrix is ​​used to record the exchangeable task set for each Earth observation mission on each satellite (row 1). Elements Recorded the first The first Earth observation mission was in the A set of exchangeable tasks on each satellite is determined. Then, for each satellite, the following operations are performed to filter the set of exchangeable tasks. First, tasks are sorted in non-decreasing order based on the start time of the visible time window between the task and the satellite under consideration. Note that tasks without a visible time window are directly excluded. Second, for each task… (Line 4), if its visible time window for the satellite coincides with subsequent missions. If the visible time windows overlap (line 6), then update the exchangeable task sets for these two tasks (lines 7-8), and then continue with the task... Compare with other subsequent tasks until the above conditions are no longer met (lines 9-10). express The end time of the visible time window, and express The start time of the visible time window.

[0048] Table 3 Algorithm 3 Table 4 outlines the basic process of the proposed three destruction and repair operators. It consists of four modules. The first module identifies successfully scheduled tasks (rows 1-4), whose indexes are recorded in a set. In the middle. If a task is not scheduled, its corresponding coded position is assigned a value of -1. The second module disrupts the mapping between tasks and satellites according to the disruption ratio. First, the index of the unscheduled task is identified (line 5). Symbol and These represent the number of satellites to which the disrupted missions are mapped and their upper limit (line 6). In each iteration, each mission is assigned a dynamic probability based on the number of its visible time windows. (Lines 8-10). Symbols Indicates task The number of visible time windows. Then, a task is selected using a roulette wheel and its mapping to satellites is removed (line 11).

[0049] The third module is the repair operator, which is responsible for remapping the mission onto the satellite. This module includes two mechanisms. The first mechanism is to randomly select a satellite from the candidate satellite set, denoted as... (Lines 18-19). The second mechanism first calculates the probability of selection based on the payload of each candidate satellite (lines 21-23), and then uses a roulette wheel to select a satellite (line 24). [Symbol] Indicates the first The payload of each satellite (line 23).

[0050] The fourth module is a repair operator based on the exchange ratio. Reverse the task execution order. Parameters The number of tasks whose execution order was swapped is recorded (line 27). A task is randomly selected (line 29). If the selected task is mapped to a satellite (line 31), a task is randomly selected from the set of swappable tasks (line 32). Afterwards, the execution order of the two tasks is swapped (line 33).

[0051] Table 4 Algorithm 4 Performance Evaluation: This application validates the effectiveness of the proposed method through numerical experiments. First, the experimental setup is introduced, including the parameters of the satellite constellation, the Earth observation mission, and deep reinforcement learning. Then, the experimental results are summarized and analyzed.

[0052] Experimental Setup: Referring to existing research, this application constructs a simulation scenario of a Walker constellation consisting of six orbital planes observing global ground targets. The simulation duration is 24 hours (from 00:00 to 24:00 on October 5, 2024). Among them, the High Resolution Multi-Mode Agile Earth Observation Satellite (GFDM-1) is my country's first optical remote sensing satellite with civilian sub-meter resolution. Based on the orbital parameters of the GFDM-1 satellite (see Table 5), the applicant deploys this type of satellite in each of the six orbital planes. During the simulation, satellite parameters are imported using STK software, and the visibility analysis module is used to calculate the visible time window between the target and the satellite.

[0053] Table 5. Main parameters of the Gaofen multi-mode satellite In the test scenario, Earth observation requests are generated in the following areas: latitude 0-60 degrees and longitude 0-180 degrees. The request arrives via the following parameters: The model is based on a Poisson distribution. For each emergency Earth observation target, the following attributes are also included: observation duration, priority, and geographic location.

[0054] To evaluate the effectiveness of the proposed IDRLNS algorithm, the applicant compared it with three baseline algorithms: HADRT, IDRL-MD, and IALNS-TSF. HADRT is essentially a greedy algorithm that sorts all waiting tasks in ascending order based on their earliest execution completion time and selects the earliest imaging opportunity for each task that satisfies all constraints. The second comparison algorithm, IDRL-MD, is an agile satellite scheduling algorithm based on deep reinforcement learning. The agile Earth observation satellite scheduling problem is formulated as a Markov decision process, and two backup operators are introduced for a Monte Carlo tree search mechanism. IALNS-TSF includes a task assignment operator and a task sorting operator in each iteration. The former is a heuristic mechanism that decomposes the multi-satellite scheduling problem into multiple scheduling problems, where each satellite corresponds to one problem. The latter is an adaptive large neighborhood search method for a specific problem, used to optimize the task sequence on each satellite.

[0055] Deep reinforcement learning neural networks consist of an input layer, hidden layers, and an output layer. The input layer, which uses an attention mechanism for each neuron, can also be called the attention layer.

[0056] The deep reinforcement learning model is a neural network architecture consisting of fully connected layers and attention layers. The hidden layers are two 128-dimensional layers. The applicant trained the model for 10 epochs, each containing 500,000 instances, using the Adam optimizer with a learning rate of 0.001. To mitigate overfitting, Dropout was added to each hidden layer. Furthermore, each hidden layer employed the ReLU activation function. The network's hyperparameters are summarized in Table 6. For large neighborhood search, the destruction, repair, and swap ratios were all set to 0.1.

[0057] All experiments were written in Python 3.11, and the deep reinforcement learning framework was imported using PyTorch 1.9.0. The experimental code was run on a computer equipped with a Windows 10 operating system, an NVIDIA RTX 4090 graphics card, and 128 GB of RAM.

[0058] Table 6. Main hyperparameters of deep neural networks Convergence analysis The experiments in this section examine the convergence properties of large neighborhood search over an evolutionary process. The applicant constructed four different scenarios based on varying arrival rates and simulation durations. The arrival rates were set to 0.007 and 0.013, and the simulation durations were set to 35,000 and 65,000 seconds. When the arrival rate was 0.007 and the simulation duration was 35,000 seconds, the task quantity was approximately 241. Similarly, when the arrival rate was 0.013 and the simulation duration was 65,000 seconds, the task quantity was approximately 842. Simultaneously, the applicant maintained an average execution time of 15 seconds for Earth observation missions and a satellite attitude transition time of 20 seconds between adjacent missions. Since both the proposed IDRLNS and the baseline IALNS-TSF utilize large neighborhood search, the applicant performed 30 comparisons for each scenario and calculated the average results. Figure 7 The performance comparison of IDRLNS and IALNS-TSF in four different scenarios is presented.

[0059] from Figure 7 In the study, the applicant observed that the reward values ​​of both IDRLNS and IALNS-TSF increased with the number of evaluations, indicating a gradual improvement in performance as the search process progresses. Notably, the IDRLNS algorithm achieved a higher reward value than IALNS-TSF with the same number of evaluations. Initially, IDRLNS's reward value was superior to IALNS-TSF. This can be attributed to IDRLNS utilizing a trained deep reinforcement learning model to generate initial solutions, while IALNS-TSF relies on stochastic methods. This demonstrates the effectiveness of deep reinforcement learning models in generating high-quality solutions. Overall, IDRLNS's growth rate is greater than that of IALNS-TSF, indicating that IDRLNS has a faster convergence speed. Especially when handling more tasks or higher arrival rates, IDRLNS exhibits a significantly higher reward value. This can be attributed to the efficiency and adaptability of the proposed destruction and repair operators in scheduling large-scale emergency Earth observation missions. Fast convergence and a higher final reward value suggest that the proposed IDRLNS is an effective choice for solving such scheduling problems.

[0060] Reference Figures 8-10 As shown, the experiments in this application aim to measure the effectiveness of the proposed IDRLNS by comparing it with HADRT, IDRL-MD and IALNS-TSF under different conditions: (a) number of tasks, (b) arrival rate, (c) execution time, and (d) transmission time.

[0061] Table 7 illustrates how the number of Earth observation missions affects these four scheduling algorithms. In this set of experiments, the applicant varied the number of missions from 150 to 450, increasing by 60 each time. Simultaneously, the applicant kept the arrival rate, mission execution time, and transmission time constant at 0.007, 15, and 20, respectively. The payoff values ​​of the four algorithms were then averaged over 30 repeated experiments. A key observation is that the payoff of all four algorithms increases with the number of missions. This is because, with a constant arrival rate and an increasing number of missions, the satellite constellation operates for a longer period, allowing it to complete more missions, thus resulting in higher payoffs for each algorithm. Compared to HADRT and IDRL-MD, IDRLNS consistently achieved higher payoffs, especially with an increasing number of missions. This suggests that the combination of deep reinforcement learning and large neighborhood search in IDRLNS is more effective than the Markov tree search method used in IDRL-MD. Furthermore, IALNS-TSF, which combines heuristic rules with large neighborhood search, performs comparably to IDRLNS with a small number of missions but falls behind as the number of missions increases. These results demonstrate that IDRLNS achieves the highest gains across all mission numbers, particularly when the number of missions is large (e.g., 450 missions). This showcases IDRLNS' superior ability to schedule large-scale, urgent Earth observation missions.

[0062] Table 7 illustrates the impact of arrival rate on the rewards obtained by the four algorithms. The applicant varied the arrival rate from 0.005 to 0.015 in steps of 0.002, while keeping the task execution time, transmission time, and simulation duration constant at 15, 20, and 35,000 seconds, respectively. As the arrival rate increased, the number of tasks arriving within the fixed simulation duration increased, and the rewards obtained by the four algorithms naturally increased accordingly. The results in Table 7 revealed an interesting phenomenon: with the same number of tasks, the rewards of the four algorithms were lower at higher arrival rates. For example, when the arrival rate was 0.13 and the number of tasks was approximately 450, the rewards of the four algorithms were significantly lower than when the arrival rate was 0.007 and the number of tasks was 450. This can be explained by the fact that as the arrival rate increases, the number of tasks that the satellite constellation needs to execute simultaneously increases, and resource conflicts between tasks become more severe, thus reducing the number of tasks completed. Baselines HADRT and IDRL-MD performed satisfactorily at lower arrival rates but deteriorated significantly at higher arrival rates. Baseline IALNS-TSF performed well at medium arrival rates but also experienced performance degradation at extremely high arrival rates. Compared to the three baselines, IDRLNS maintained a high reward value across all arrival rates, demonstrating its stability and effectiveness under different arrival conditions.

[0063] To investigate the impact of average task execution time on performance, the applicant increased this parameter from 15 seconds to 40 seconds in 5-second increments, while maintaining the number of tasks, arrival rate, and transmission time at 241, 0.007, and 20, respectively. The average gain of the four baseline algorithms across 30 repeated tests at different task execution times is presented. Similarly, the applicant investigated how satellite transmission time affects the average gain of the four scheduling algorithms and summarized the comparative results. The applicant observed that task execution time and satellite transmission time have a small impact on algorithm performance. Furthermore, IDRLNS consistently achieves high gain values ​​across all execution and satellite transmission times, demonstrating its stability and effectiveness across various scenarios. This is achieved by integrating deep reinforcement learning and large neighborhood search to derive high-quality initial solutions, allowing for the reorganization of more Earth observation task mappings and execution sequences for satellite completion.

[0064] Overall, the proposed IDRLNS outperforms the three baseline algorithms under various conditions of task quantity, arrival rate, execution time, and transmission time. This is mainly attributed to its strategy, which integrates deep reinforcement learning to generate initial solutions and uses a large neighborhood search operator to iteratively optimize these solutions, making them adaptable to dynamic scheduling environments and requirements. Therefore, IDRLNS has significant advantages and potential in scheduling large-scale emergency Earth observation missions.

[0065] Table 7 Performance comparison of the four algorithms on 24 instances Table 7 shows the mean and variance (in parentheses) of the experimental results in this section. The case name consists of four elements: number of tasks, arrival rate, average duration, and satellite attitude adjustment time. For example, the case name 150-0.007-15-20 in the first row of Table 3 indicates that the number of tasks was 150, the arrival rate was 0.007, the average duration was 15 seconds, and the satellite attitude adjustment time was 20 seconds.

[0066] This application proposes a novel online scheduling method for large-scale emergency missions of Earth observation satellites, namely, a deep reinforcement learning and neighborhood search method for scheduling satellite observation missions. It combines deep reinforcement learning with large neighborhood search to effectively adapt to the dynamic and resource-constrained characteristics of satellite mission scheduling environments. A novel emergency scheduling framework based on rolling time domain is developed, transforming dynamic online scheduling into a series of static sub-problems. This method enhances optimization decisions based on current needs and system state. Furthermore, a novel decoding mechanism is designed within deep reinforcement learning to simplify the scheduling process and significantly reduce the output dimensionality caused by the exponential growth of the search space. In addition, three novel destruction and repair operators are designed to further improve the optimization efficiency of large neighborhood search and adapt to the highly dynamic environment. Extensive comparative results validate the superior overall performance of IDRLNS, which consistently outperforms three baseline algorithms (HADRT, IDRL-MD, and IALNS-TSF) in various experimental settings. Moreover, the experimental results highlight the significant contribution of its functional modules to overall performance.

[0067] Based on data collected from real-world orbits and applications, this invention compares the proposed ensemble algorithm with three representative algorithms. Experimental results show that the proposed algorithm achieves a 20.1% increase in reward while maintaining a faster convergence speed.

[0068] This invention combines deep reinforcement learning with large neighborhood search to effectively adapt to the dynamic and resource-constrained characteristics of satellite mission scheduling environments. A novel emergency scheduling framework based on rolling time domain is developed, transforming dynamic online scheduling into a series of static sub-problems. This method enhances optimization decisions based on current needs and system state. Furthermore, a novel decoding mechanism is designed within deep reinforcement learning to simplify the scheduling process and significantly reduce the output dimensionality caused by the exponential growth of the search space. Additionally, three novel destruction and repair operators are designed to further improve the optimization efficiency of large neighborhood search and adapt to the highly dynamic nature of the environment.

[0069] Example 2 One embodiment of the present invention provides an enhancement and neighborhood search system for scheduling satellite observation missions, referring to... Figure 2 As shown, it includes: Preprocessing module 10: Used to preprocess satellite observation tasks; Neural Network Training Module 20: Used to train neural networks using deep reinforcement learning training models, including transforming the process of mapping tasks to resources in deep reinforcement learning models into task serialization subprocesses and resource selection subprocesses to compress the search space that grows exponentially with the increase of tasks. Scheduling scheme output module 30: used to input the preprocessed satellite observation task into the neural network and output the scheduling scheme; Scheduling scheme optimization module 40: used to pass the scheduling scheme output by the neural network as the initial solution to the large neighborhood search algorithm and output the optimized scheme; The large neighborhood search algorithm includes the following: for two Earth observation tasks on a satellite, if the visible time windows of the two Earth observation tasks overlap, a new feasible execution sequence can be obtained by exchanging the execution order of the tasks; conversely, if the end time of the visible time window of one task is earlier than the start time of the visible time window of another task, the former is executed before the latter.

[0070] In one specific embodiment of the present invention, the neural network training module 20 is used to perform the following steps: Construct an encoder-decoder-based policy network; The policy network is trained using the REINFORCE method. Update network parameters during the training process; Through multiple iterations of training, the neural network learns the optimal task scheduling strategy.

[0071] In one specific embodiment of the present invention, in the neural network training module 20, the deep reinforcement learning training model sequentially creates a scheduling scheme based on an encoder-decoder policy network to map satellite status and waiting task information into an embedded representation.

[0072] In one specific embodiment of the present invention, the preprocessing includes: adding new satellite observation tasks; updating the satellite scheduling time limit; filtering out tasks whose execution instruction upload time is later than the current time limit; and canceling their original scheduling plans after the task filtering is completed.

[0073] In one specific embodiment of the present invention, the destruction ratio, repair ratio, and exchange ratio of the large neighborhood search algorithm are all set to 0.1.

[0074] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0075] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the functions specified in one or more boxes. Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the invention. Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0076] The methods and apparatus provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

[0077] In the description of this specification, references to terms such as "an embodiment," "some embodiments," "example," "specific example," or "a specific embodiment" or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An enhancement and neighborhood search method for scheduling satellite observation tasks, characterized in that, include: Step S1: Preprocess the satellite observation task; Step S2: Train the neural network using a deep reinforcement learning training model, including transforming the process of mapping tasks to resources in the deep reinforcement learning model into a task serialization subprocess and a resource selection subprocess, in order to compress the search space that grows exponentially with the increase of tasks. Step S3: Input the preprocessed satellite observation task into the neural network and output a scheduling scheme; Step S4: Pass the scheduling scheme output by the neural network as the initial solution to the large neighborhood search algorithm to output the optimized scheme; The large neighborhood search algorithm includes the following: for two Earth observation tasks on a satellite, if the visible time windows of the two Earth observation tasks overlap, a new feasible execution sequence can be obtained by exchanging the execution order of the tasks; conversely, if the end time of the visible time window of one task is earlier than the start time of the visible time window of another task, the former is executed before the latter.

2. The enhancement and neighborhood search method for scheduling satellite observation tasks according to claim 1, characterized in that, Step S2 includes: Construct an encoder-decoder-based policy network; The policy network is trained using the REINFORCE method. Update network parameters during the training process; Through multiple iterations of training, the neural network learns the optimal task scheduling strategy.

3. The enhancement and neighborhood search method for scheduling satellite observation tasks according to claim 1, characterized in that, In step S3, the deep reinforcement learning training model sequentially creates a scheduling scheme based on an encoder-decoder policy network to map satellite status and waiting task information into an embedded representation.

4. The enhancement and neighborhood search method for scheduling satellite observation tasks according to claim 1, characterized in that, The preprocessing includes: adding new satellite observation tasks; updating the satellite scheduling time limit; filtering out tasks whose execution instruction upload time is later than the current time limit; and canceling their original scheduling plans after the task filtering is completed.

5. The enhancement and neighborhood search method for scheduling satellite observation tasks according to claim 1, characterized in that, The destruction ratio, repair ratio, and exchange ratio of the large neighborhood search algorithm are all set to 0.

1.

6. An enhancement and neighborhood search system for scheduling satellite observation missions, characterized in that, include: Preprocessing module: Used to preprocess satellite observation tasks; Neural Network Training Module: Used to train neural networks using deep reinforcement learning training models, including transforming the process of mapping tasks to resources in deep reinforcement learning models into task serialization subprocesses and resource selection subprocesses, in order to compress the search space that grows exponentially with the increase of tasks. Scheduling scheme output module: used to input the preprocessed satellite observation tasks into the neural network and output the scheduling scheme; Scheduling scheme optimization module: used to pass the scheduling scheme output by the neural network as the initial solution to the large neighborhood search algorithm, and output the optimized scheme; The large neighborhood search algorithm includes the following: for two Earth observation tasks on a satellite, if the visible time windows of the two Earth observation tasks overlap, a new feasible execution sequence can be obtained by exchanging the execution order of the tasks; conversely, if the end time of the visible time window of one task is earlier than the start time of the visible time window of another task, the former is executed before the latter.

7. The enhancement and neighborhood search method for scheduling satellite observation tasks according to claim 6, characterized in that, The neural network training module is used to perform the following steps: Construct an encoder-decoder-based policy network; The policy network is trained using the REINFORCE method. Update network parameters during the training process; Through multiple iterations of training, the neural network learns the optimal task scheduling strategy.

8. The enhancement and neighborhood search method for scheduling satellite observation tasks according to claim 6, characterized in that, In the neural network training module, the deep reinforcement learning training model uses an encoder-decoder policy network to sequentially create scheduling schemes to map satellite status and waiting task information into embedded representations.

9. The enhancement and neighborhood search method for scheduling satellite observation tasks according to claim 6, characterized in that, The preprocessing includes: adding new satellite observation tasks; updating the satellite scheduling time limit; filtering out tasks whose execution instruction upload time is later than the current time limit; and canceling their original scheduling plans after the task filtering is completed.

10. The enhancement and neighborhood search method for scheduling satellite observation tasks according to claim 6, characterized in that, The destruction ratio, repair ratio, and exchange ratio of the large neighborhood search algorithm are all set to 0.1.

Images