Multi-satellite task allocation strategy adaptive selection method based on deep reinforcement learning

Abstract

The invention discloses a multi-satellite task allocation strategy adaptive selection method based on deep reinforcement learning, and aims to solve the problems that the satellite task scheduling problem is difficult to solve due to the fact that the satellite attitude maneuver ability is improved, and the task number and the satellite scale are explosively increased, and the solving limitation of a traditional method is solved. According to the method, the problem is divided into an upper-layer multi-satellite task allocation part and a lower-layer single-satellite scheduling part, iterative optimization of a scheme is realized through multiple interactions of the two parts, a self-adaptive search method based on reinforcement learning is provided for the upper-layer task allocation problem, and task allocation strategy self-adaptive selection based on deep reinforcement learning is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of satellite mission allocation, and in particular, to a method for adaptively selecting a multi-satellite mission allocation strategy based on deep reinforcement learning. Background Art

[0002] With the continuous transformation of satellite technology, imaging satellites are playing an increasingly important role in terrain exploration, disaster warning, military intelligence acquisition, etc. In recent years, the continuous development of on-board hardware, the construction of large constellations, and the continuous emergence of application requirements have posed new challenges to current satellite mission scheduling methods. First, a new generation of agile satellites with high attitude maneuverability has been widely used. Compared with non-agile satellites, agile satellites have the characteristic of time dependence, the decision-making dimension of the satellite mission scheduling problem is further increased, and the complexity of a single unit is improved. Second, the number of tasks and the scale of satellites are continuously increasing, and the scale of the problem solution space is further increased. In addition, emergency requirements occur frequently, and the requirement for the solution timeliness of the method becomes higher. Since the integrated optimization framework needs to traverse the entire solution space of the problem, once the problem scale expands, the contradiction between "solution quality - solution speed" faced by the method becomes more prominent.

[0003] Facing the challenge of the continuously increasing solution space in the multi-satellite mission scheduling problem, the present invention adopts a two-layer optimization framework, divides the problem into two parts: upper-layer multi-satellite mission allocation and lower-layer single-satellite mission scheduling, realizes "divide and conquer and dimension reduction" at the problem level, and further realizes the rapid response of the method. However, this framework will limit the method to explore the entire solution space, thus unable to guarantee the quality of the solution. To effectively reduce the impact of the two-layer optimization framework on the method and coordinate the in-orbit satellites and management and control resources in our country, the present invention proposes an adaptive search method based on reinforcement learning, aiming to improve the solution performance of the method in a data-driven manner. Given that there are already a large number of studies on the single-satellite mission scheduling problem and many mature methods with excellent performance have been produced, the present invention will focus on the design of the method for the upper-layer multi-satellite mission allocation problem and explore the interaction mode between the upper-layer mission allocation method and the lower-layer single-satellite scheduling method under the two-layer optimization architecture. First, for the upper-layer mission allocation problem, according to the knowledge of the problem domain, a variety of mission allocation strategies are designed, and the selection of the mission allocation strategy is processed into a sequential decision-making problem, and an adaptive selection mechanism for the allocation strategy based on deep reinforcement learning is proposed. This mechanism uses the DQN model to decide the mission allocation strategy to be selected in the next round of interaction, which not only ensures the low complexity of the method but also improves the quality of the mission allocation plan. Then, the current advanced single-satellite scheduling method is directly adopted in the lower layer, and the final mission planning plan is quickly generated through the parallel operation of the single-satellite scheduling methods on multiple satellites. Summary of the Invention

[0004] The present invention provides a method for adaptively selecting a multi-satellite task allocation strategy based on deep reinforcement learning to solve the problem of high complexity of a single satellite task scheduling problem. Through the method of the present invention, the complexity of tasks is reduced, and the quality of the task allocation scheme is improved.

[0005] To achieve the above object, the technical solution adopted by the present invention to solve the above technical problem provides a method for adaptively selecting a multi-satellite task allocation strategy based on deep reinforcement learning. The adaptive search method RLASA based on reinforcement learning is used to implement multi-satellite task scheduling, and the RLASA is applied to a multi-satellite task scheduling model to implement adaptive selection of a multi-satellite task allocation strategy; the method includes the following steps:

[0006] Set three types of satellite scheduling task attribute parameters, satellite parameters, and visible time window attribute parameters. The satellite scheduling task set T = {t i | 1 ≤ i ≤ N}, where N represents the number of tasks. The attribute of each task t i is represented by a triple (p i , l i , c i ). p i is the priority of task t i , l i is the execution time of task t i , and c i is the minimum imaging quality specified by task t i .

[0007] The satellite set SR = {sr j | 1 ≤ j ≤ M}, where M represents the satellite scale; the attribute of each satellite sr j is represented by a triple . γ jt represents the roll angle of the j-th satellite at time t, π jt represents the pitch angle of the j-th satellite at time t, represents the yaw angle of the j-th satellite at time t.

[0008] The visible time window set W = {w ijk | 1 ≤ i ≤ N, 1 ≤ j ≤ M, 1 ≤ k ≤ |w ij |}, where w ijk represents the k-th visible time window of task t i on the j-th satellite, |w ij | is the number of visible time windows of task t i on the j-th satellite, and w ijk is represented by a triple (st ijk , et ijk , d ijk ). stijk is the start time of time window w ijk , et ijk is the end time of time window w ijk , d ijk is the time length of time window w ijk .

[0009] Establish a multi-satellite task scheduling model, and set the decision variables of the multi-satellite task scheduling problem as (x ijk , u ijk ). x ijk is a binary variable. When x ijk = 1, it means that task t i is executed within the k-th visible time window of the j-th satellite. Otherwise, x ijk = 0. The decision variable u ijk represents the start time of task t i within time window w ijk .

[0010] The constructed multi-satellite task scheduling model includes the following steps:

[0011] S1. Set the objective function of the multi-satellite task scheduling problem, which represents maximizing the revenue of the scheduled tasks. Specifically:

[0012]

[0013] S2. Set the task execution uniqueness constraint, which means that each task can be executed at most once and cannot be executed repeatedly. Specifically:

[0014]

[0015] S3. Set that the task must be executed within the visible time window to ensure the effectiveness of task execution. Specifically:

[0016]

[0017] S4. Set that different tasks executed on the same satellite need to meet the conversion time constraint. Specifically:

[0018]

[0019] S5. Set the calculation formula for the attitude conversion time between tasks, where a1, a2, a3, a4 represent the conversion speeds of four different attitude angles, and Δg represents the attitude angle difference between tasks. Specifically:

[0020]

[0021] S6. Determine the imaging quality of the task and set the imaging quality q of the taski Higher than the specified minimum imaging quality, where is the moment of the best imaging quality in the time window w ijk and can be obtained through the formula The specific calculation is as follows:

[0022]

[0023] Furthermore, the RLASA method includes a satellite task scheduling stage and a reinforcement learning stage; the satellite task scheduling stage includes: obtaining tasks to be assigned, generating task assignment tasks, single-line scheduling, and generating a scheduling result; the reinforcement learning stage includes: scheme revenue, calculation reporting, state update, input to the neural network, and determining the task assignment strategy.

[0024] In the satellite task scheduling stage, an assignment scheme is generated based on the task assignment strategy of this round. Subsequently, each satellite independently conducts task scheduling in parallel, and finally, the revenue of the scheduling scheme is obtained.

[0025] In the reinforcement learning execution stage, the revenue of the previous round of scheduling scheme is used to calculate the return, update the current state, input it to the neural network, and select the action with the largest Q value as the task assignment strategy for the next round.

[0026] Specifically, it includes the following steps:

[0027] S1. In the initial solution solving stage, set the task set T, satellite set SR, task assignment strategy set A, neighborhood construction operator O N , the parameters θ of the trained deep Q-network DQN, the objective function F(X), and the number of task assignments MA; input the task set and satellite resource set into the established task scheduling model to generate an initial solution operation, which specifically includes: sorting the tasks in descending order according to the revenue value, then randomly assigning the sorted tasks to different satellites, and then generating an initial solution using the precedence arrangement method, and then entering the main loop.

[0028] S2. The main loop includes the satellite task scheduling stage. An assignment scheme is generated based on the task assignment strategy of this round. Subsequently, each satellite independently conducts task scheduling in parallel, and finally, the revenue of the scheduling scheme is obtained.

[0029] S3. Reinforcement learning stage. In the reinforcement learning execution stage, the revenue of the previous round of scheduling scheme is used to calculate the return, update the current state, input it to the neural network, and select the action with the largest Q value as the task assignment strategy for the next round.

[0030] S4. Generate a task assignment scheme according to the task assignment strategy and enter the next round of task execution.

[0031] Furthermore, the RLASA integrates a Deep Reinforcement Learning-based Task Assignment Strategy Adaptive Selection Mechanism (DRL-TASASM) and a Frequent Pattern-based Adaptive Local Search Approach (ALSA), which are respectively oriented to the task assignment problem and the single-satellite scheduling problem, and specifically include:

[0032] In the DRL-TASASM, the action is defined as selecting different task assignment strategies, and the state is designed as metrics related to the single-satellite or overall scheduling scheme. The agent adopts the DQN method in deep reinforcement learning. In each iteration, the agent takes different state components as the input of the neural network, outputs the Q-values under different assignment strategies, and then selects the best task assignment strategy according to the greedy strategy, thus completing the solution of the task assignment problem.

[0033] In the ALSA, for each satellite, single-satellite scheduling is performed to obtain the initial solution of the current single-satellite scheduling. If the single-satellite scheduling termination condition is not reached, a new solution is generated, and the initial solution is updated to the new solution. If the conditions of the new solution are met, the solution set is updated to the optimal solution set, thereby achieving adaptive solution set update based on frequent patterns.

[0034] An adaptive search method based on reinforcement learning of the present invention adopts a two-layer optimization framework, and through multiple interactions between upper-layer task assignment and lower-layer single-satellite scheduling, iterative optimization of the scheme is achieved. A Deep Reinforcement Learning-based Task Assignment Strategy Adaptive Selection Mechanism (DRL-TASASM) is proposed. This mechanism uses DQN to select the best assignment strategy for each interaction, which not only ensures the low complexity of the method but also significantly improves the quality of the task assignment scheme. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0036] Figure 1 is the framework diagram of the RLASA method of the present invention;

[0037] Figure 2 is the data flow diagram of the RLASA method of the present invention;

[0038] Figure 3 is the working flow diagram of the DRL-TASASM mechanism of the present invention;

[0039] Figure 4 is the training situation diagram of DQN under different scenarios of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0040] The following describes exemplary embodiments of the present application with reference to the accompanying drawings. Various details of the embodiments of the present application are included to facilitate understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, descriptions of well-known functions and structures are omitted in the following description for clarity and conciseness.

[0041] The present invention provides a method for adaptively selecting a multi-satellite task allocation strategy based on deep reinforcement learning. The multi-satellite task scheduling is implemented based on the reinforcement learning-based adaptive search method RLASA, and the RLASA is applied to the multi-satellite task scheduling model to realize the adaptive selection of the multi-satellite task allocation strategy. The method includes the following steps:

[0042] Set three types of parameters: satellite scheduling task attribute parameters, satellite parameters, and visible time window attribute parameters. The satellite scheduling task set T = {t i |1 ≤ i ≤ N}, where N represents the number of tasks. The attribute of each task t i is represented by a triple (p i , l i , c i ). p i is the priority of task t i , l i is the execution time of task t i , and c i is the minimum imaging quality required by task t i .

[0043] The satellite set SR = {sr j |1 ≤ j ≤ M}, where M represents the scale of the satellites. The attribute of each satellite sr j is represented by a triple to represent. γ jt represents the roll angle of the j-th satellite at time t, π jt represents the pitch angle of the j-th satellite at time t, represents the yaw angle of the j-th satellite at time t.

[0044] The visible time window set W = {w ijk |1 ≤ i ≤ N, 1 ≤ j ≤ M, 1 ≤ k ≤ |w ij |}, where w ijk represents the k-th visible time window of task t i on the j-th satellite, and |w ij | is the number of visible time windows of task t i on the j-th satellite. w ijkUse a triple (st ijk ,et ijk ,d ijk ) to indicate that st ijk is the time window w ijk The start time, et ijk is the time window w ijk End time, d ijk is the time window w ijk length of time.

[0045] Establish a multi-satellite task scheduling model and set the decision variables of the multi-satellite task scheduling problem as (x ijk ,u ijk ), x ijk is a binary variable, when x ijk =1 indicates task t i Executed within the kth visible time window on the jth satellite, otherwise x ijk =0, decision variable u ijk Represents task t i In the time window w ijk The start time within.

[0046] The multi-satellite mission scheduling model constructed includes the following steps:

[0047] S1. Set the objective function of the multi-satellite task scheduling problem to maximize the benefits of the scheduled tasks. Specifically:

[0048]

[0049] S2. Set the task execution uniqueness constraint, which means that each task can be executed at most once and cannot be executed repeatedly. Specifically:

[0050]

[0051] S3. Set the task to be executed within the visible time window to ensure the effectiveness of task execution. Specifically:

[0052]

[0053] S4. Setting the same satellite to perform different tasks requires satisfying the switching time constraints, specifically:

[0054]

[0055] S5. Set the calculation formula for the posture conversion time between tasks, where a1, a2, a3, and a4 represent the conversion speeds of four different posture angles, and Δg represents the difference in posture angles between tasks. Specifically:

[0056]

[0057] S6. Determine the imaging quality of the task and set the imaging quality q of the task i Higher than the specified minimum imaging quality, where is the best imaging quality moment in the time window w, which can be calculated by the formula ijk and specifically is

[0058]

[0059] Furthermore, the RLASA method includes a satellite task scheduling stage and a reinforcement learning stage; the satellite task scheduling stage includes: obtaining tasks to be assigned, producing task assignment tasks, single-line scheduling, and generating a scheduling result; the reinforcement learning stage includes: plan revenue, calculation report, state update, input to the neural network, and determination of the task assignment strategy.

[0060] In the satellite task scheduling stage, an assignment plan is generated based on the task assignment strategy of this round. Subsequently, each satellite independently conducts task scheduling in parallel, and finally, the revenue of the scheduling plan is obtained.

[0061] In the reinforcement learning execution stage, the revenue of the previous round of scheduling plan is used to calculate the return, update the current state, input it to the neural network, and select the action with the largest Q value as the task assignment strategy for the next round.

[0062] Specifically, it includes the following steps

[0063] S1. In the initial solution solving stage, set the task set T, the satellite set SR, the task assignment strategy set A, the neighborhood construction operator O N , the parameters θ of the trained deep Q network DQN, the objective function F(X), and the number of task assignments MA; input the task set and the satellite resource set into the established task scheduling model to generate an initial solution operation, specifically including: sorting the tasks in descending order according to the revenue value, then randomly assigning the sorted tasks to different satellites, and then generating an initial solution using the pre-predecessor arrangement method, and then entering the main loop.

[0064] S2. The main loop includes the satellite task scheduling stage, in which an assignment plan is generated based on the task assignment strategy of this round. Subsequently, each satellite independently conducts task scheduling in parallel, and finally, the revenue of the scheduling plan is obtained.

[0065] S3. Reinforcement learning stage. In the reinforcement learning execution stage, the revenue of the previous round of scheduling plan is used to calculate the return, update the current state, input it to the neural network, and select the action with the largest Q value as the task assignment strategy for the next round.

[0066] ​S4. Generate a task assignment plan according to the task assignment strategy and enter the next round of task execution.

[0067] Furthermore, the RLASA integrates a Deep Reinforcement Learning-based Task Assignment Strategy Adaptive Selection Mechanism (DRL-TASASM) and a Frequent Pattern-based Adaptive Local Search Algorithm (ALSA), which are respectively oriented to the task assignment problem and the single-satellite scheduling problem, and specifically include:

[0068] In the DRL-TASASM, the action is defined as selecting different task assignment strategies, and the state is designed as an index related to the single-satellite or overall scheduling plan. The agent uses the DQN method in deep reinforcement learning. In each iteration of the method, the agent takes different state components as the input of the neural network, outputs the Q values under different assignment strategies, and then selects the best task assignment strategy according to the greedy strategy, thus completing the solution of the task assignment problem;

[0069] In the ALSA, for each satellite, single-satellite scheduling is performed to obtain the initial solution of the current single-satellite scheduling. If the single-satellite scheduling termination condition is not reached, a new solution is generated, and the initial solution is updated to the new solution. If the conditions of the new solution are met, the solution set is updated to the optimal solution set, thereby realizing the adaptive update of the solution set based on frequent patterns.

[0070] On the premise of following the business processing flow of actual satellite task management and control, to highlight the research focus, the present invention makes the following assumptions to simplify the single-satellite scheduling problem:

[0071] (1) All targets are point targets and can be observed by a single satellite within a single time window;

[0072] (2) Tasks cannot be preempted during execution. Once a task starts execution, it is not allowed to be terminated midway;

[0073] (3) Each satellite can only execute one task at the same time;

[0074] (4) Each task is executed at most once and cannot be executed repeatedly;

[0075] (5) Special usage constraints of satellites and special situations such as cloud cover are not considered;

[0076] (6) Considering the on-orbit charging of satellites facing the sun and the occasional erasure of stored data during downlink, the power and storage limitations of satellites are not considered.

[0077] According to the prior art, the methods for solving the satellite task assignment problem mainly focus on three aspects: (1) Single heuristic strategies, such as task random assignment strategy, minimum conflict degree assignment strategy, maximum observation opportunity assignment strategy, and load balancing assignment strategy, etc. Although the complexity of this solution method is relatively low, the solution quality is poor. (2) Based on the single heuristic strategy, the method evaluates the contribution degree of different assignment strategies according to the quality of the solution, and updates their usage probabilities in the next round of search. Although this method can improve the adaptive ability of the method to a certain extent, the way of guiding the subsequent search of the method by only updating the assignment strategy probability according to the contribution degree is a bit "rough". Essentially, both of the above two methods ignore the relationship between task assignment and single-satellite scheduling, so there is still a large room for improvement in the quality of the final solution. (3) Using meta-heuristic methods to solve the task assignment problem, specifically: the method generates different task assignment schemes in each round of iteration, calls the lower-layer single-satellite scheduling method to calculate the scheduling scheme revenue corresponding to this assignment scheme, and the upper and lower layers interact continuously until the termination condition of the method is reached. Although this method has good solution results, it takes a lot of time.

[0078] Although the solution qualities of the above first two methods are not ideal, they bring new inspirations to the solution of the task assignment problem. If the mapping relationship between the task assignment scheme and the task scheduling scheme can be established, and then the method is guided to select the optimal task assignment strategy at different stages based on this mapping relationship, then excellent solution results can be obtained on the premise of ensuring the low complexity of the method. Based on the above analysis, an RL-based Adaptive Search algorithm (RLASA) is proposed to solve the multi-satellite task scheduling problem, and a DRL-based task assignment strategy adaptive selection mechanism (DRL-TASASM) is designed to improve the low solution quality situation that occurs when traditional heuristic strategies solve the task assignment problem. The lower layer adopts an advanced method proposed in the current single-satellite task scheduling field - the adaptive local search algorithm (ALSA), and the entire solution time is greatly reduced by parallelly solving each single-satellite scheduling problem.

[0079] The method framework of RLASA is shown in Figure 1 Figure. It can be seen from the overall process that the RLASA method integrates the DRL-TASASM mechanism and the ALSA method, which are respectively oriented to the task assignment problem and the single-satellite scheduling problem.

[0080] The pseudo-code of the RLASA method is shown in Method 1: The input of the RLASA method is a set of tasks and a set of satellite resources. Among them, the second line of Method 1 is the operation of generating an initial solution. The specific approach is to sort the tasks in descending order according to the revenue value, then randomly assign the sorted tasks to different satellites, and then generate an initial solution using the precedence arrangement method. Subsequently, the method enters the main loop. Lines 7-9 of the method describe the working process of the DRL-TASASM mechanism. The RLASA method designs various task allocation strategies, and DRL-TASASM uses the trained Deep Q Network (DQN) to select the best task allocation strategy for this round, corresponding to line 8 of the method. Subsequently, a task allocation plan is generated, corresponding to line 9 of the method. Lines 10-24 of the method describe the working process of the ALSA method.

[0081]

[0082]

[0083] The data flow of the RLASA method is as Figure 2 , in the satellite task scheduling stage, the method generates an allocation plan based on the task allocation strategy for this round. Subsequently, each satellite independently conducts task scheduling in parallel, and finally obtains the revenue of the scheduling plan. In the reinforcement learning execution stage, the revenue of the previous round of scheduling plan is used to calculate the return, update the current state, input it to the neural network, and select the action with the largest Q value as the task allocation strategy for the next round.

[0084] The working process of the DRL-TASASM mechanism is as Figure 3 shown. In this mechanism, the action is defined as selecting different task allocation strategies, and the state is designed as a series of metrics related to the single-satellite / overall scheduling plan. The agent uses the DQN method in deep reinforcement learning. In each iteration of the method, the agent takes different state components as the input of the neural network and outputs the Q values under different allocation strategies. Then, the best task allocation strategy is selected according to the greedy strategy, thus completing the solution of the task allocation problem.

[0085] In each iteration, the RLASA method selects the best strategy from the set of task allocation strategies according to the results of the previous stage of task scheduling and combines the tasks to be allocated in this stage. The task allocation strategy selected by the method in the t i+1 -th generation only depends on the t iis related to the current generation and has nothing to do with the earlier states, that is, there is no correlation between the methods in multiple iterations. The above solution process satisfies the conditions for constructing a Finite Markov Decision Process (MDP). Generally, an MDP model can be represented by a quadruple <S, A, R, V>, where S represents the set of states of the agent, A represents the set of actions of the agent, R represents the reward value, and V represents the value function. During the learning process of MDP, in each round of interaction, the agent is in a certain state s t ∈S, and based on the policy, the action a t ∈A for this round is selected. Then, the environment, according to the current state s t and the selected action a t , feedbacks the reward R t ∈R to the agent and calculates the state s t+1 for the next round. The agent continuously repeats this process for learning until the termination condition is met. The entire learning process generates an interaction sequence τ.

[0086] τ = {s0, a0, R0, …, s t , a t , R t , …, s T , a T , R T} (10)

[0087] In the formula, T represents the maximum step length of the finite MDP.

[0088] Once the MDP model is trained, the task of solving the task allocation problem by the RLASA method can be described as follows: Assume that the maximum number of iterations of the method is set to t max , at the t i -th (t i < t max ) generation, first, according to the state s i , the best task allocation strategy a i for this round is selected to generate a task allocation plan. Then, each satellite, according to the assigned tasks, independently enables the single-satellite scheduling method to obtain the satellite task scheduling result of the t i -th generation and calculates the reward value r i . At the t i+1 -th (t i+1 ≤ t max ) generation, the method, according to the new state s i+1 , selects the best task allocation strategy a i+1 for this round and continuously repeats this process until the method reaches the termination condition.

[0089] The solution to the task allocation problem depends on the results of the lower-layer single-satellite scheduling. Therefore, the MDP model designed in the present invention requires the upper and lower layers to cooperate with each other and interact continuously. Next, the various elements and functional modules of the MDP model will be introduced in detail.

[0090] An action is the way an agent interacts with the environment and also a means of transferring from one state to another. Which action to choose is jointly determined by the current state and the value function. In the task allocation problem, an action is defined as "selecting a certain task allocation strategy". A total of eight task allocation strategies are designed, as follows.

[0091] (1) Random allocation strategy (RA): Randomly arrange tasks to satellites with visible time windows, aiming to enhance the randomness of the method.

[0092] (2) Satellite position allocation strategy (SPA): Allocate tasks to the satellite with the smallest attitude maneuvering angle, aiming to reduce the satellite's attitude conversion time, thereby reserving more time to arrange subsequent tasks. The attitude maneuvering angle is obtained by summing the roll angle, pitch angle, and yaw angle. When observing target A with satellite A, the attitude maneuvering angle of satellite A is equal to 0. When observing target A with satellite B, the attitude maneuvering angle is greater than 0. Therefore, according to the SPA strategy, the task of "observing target A" is preferentially allocated to satellite A.

[0093] (3) Minimum quantity allocation strategy (MQA): This strategy preferentially allocates tasks to satellites with fewer tasks already arranged according to the number of tasks arranged on different satellites. Generally speaking, the fewer the number of tasks, the more remaining observable resources, and the lower the conflict degree between tasks, and the higher the probability of task execution.

[0094] (4) Maximum opportunity allocation strategy (MOA): The opportunity for observation also refers to the visible time window. Usually, the more visible time windows a satellite has for a target, the greater the probability of task execution. This strategy preferentially allocates tasks to satellites with a larger number of visible time windows for the target according to the number of visible time windows of different satellites for the target.

[0095] (5) Early time allocation strategy (ETA): This strategy preferentially assigns tasks to the satellite with the earliest start time, aiming to reserve more time windows for arranging the remaining tasks.

[0096] (6) Maximum VTW allocation strategy (MVA): Generally, the longer the visible time window length of a task on a satellite, the higher the probability of the task being executed.

[0097] (7) Minimum unit profit allocation strategy (MUPA): The unit profit of a scheme can reflect the quality of the scheme to a certain extent. Generally speaking, the lower the unit profit, the worse the quality of the scheme. Therefore, this strategy preferentially assigns tasks to the satellite with a lower unit profit to improve the quality of the scheme.

[0098] (8) Historical allocation strategy (HA): This strategy randomly selects one from the set of task allocation strategies not selected in the previous round, aiming to help the method explore more unknown solution spaces and prevent the method from falling into local optima.

[0099] At time step t, the action a taken by the agent t is represented by the following formula.

[0100]

[0101] The state is an overview of the environment at the current moment and is the basis for the agent to take actions. The decision-making performance of reinforcement learning depends to a large extent on the design of the state. From the perspective of the overall method process, after solving the task allocation problem, the satellite task scheduling problem actually turns into multiple single-satellite scheduling sub-problems. Therefore, the design of the state is closely related to the scheduling scheme of each satellite. At time step t, the state s of the agent t can be represented by the following formula.

[0102] s t = {ss t , as t}} (12)

[0103] s t consists of two parts, where ss t is characterized by the scheduling scheme of each satellite, as shown in the following formula.

[0104]

[0105] in, Represents the state component of the j-th satellite, represented by an octet:

[0106]

[0107] Assume that at time step t, the scheduling plan for the jth satellite is is the number of scheduled tasks; is the total revenue of the scheduled tasks; is the total execution time of the scheduled tasks; It represents the mission completion rate, which is expressed as the ratio of the number of scheduled missions to the number of missions assigned to the satellite; It represents the mission profitability, expressed as the ratio of the total revenue of scheduled missions to the total revenue of missions assigned to the satellite; It is the execution time ratio, which is expressed as the ratio of the total execution time of the scheduled tasks to the scheduling period. Indicates the conflict degree of the scheme. This invention uses the average number of time windows where different tasks overlap to represent the conflict degree of the scheme. The calculation formula is as follows:

[0108]

[0109] Where, Represents task t i With the task The number of overlapping visible time windows; It represents the number of tasks included in the j-th satellite scheduling plan at time step t.

[0110] It is a piecewise function used to characterize the changes in the benefits of a single satellite scheduling solution.

[0111]

[0112] Where, It represents the benefit of the j-th satellite scheduling scheme at time step t.

[0113] as t It is t Another state component of , which is used to reflect the change in the benefits of the entire satellite mission scheduling plan, is also represented by a piecewise function.

[0114]

[0115] Where: P t ——Satellite scheduling plan at time step t;

[0116] F(P t)——The revenue of the entire satellite scheduling plan at time step t.

[0117] The reward is an evaluation of the agent's choice of an action in the current state. In the present invention, the reward is equal to the revenue of the satellite scheduling plan at time step t minus the revenue of the satellite scheduling plan at time step t - 1, that is

[0118] R t =F(P t ) - F(P t-1 ) (18)

[0119] DQN is the pioneering work of deep reinforcement learning, first proposed by the DeepMind team in 2013, and then an improved version was proposed in 2015. DQN gives full play to the representation ability of deep learning and the learning ability of reinforcement learning. Currently, it has been widely used to solve combinatorial optimization problems and has achieved excellent solving performance. DQN is developed on the basis of Q-learning. The traditional Q-learning method uses a Q-table to store the long-term rewards of different actions in each state, aiming to find the "state-action" pair value (i.e., the Q-value) to characterize the rationality of action selection. However, limited by the limited capacity of the Q-table, Q-learning can only handle low-dimensional and discrete state and action spaces. In actual situations, most problems have high-dimensional and continuous state or action spaces. At this time, if the Q-learning method is continued to be used, on the one hand, it is a huge test for the computer memory, and on the other hand, traversing the Q-table to find a certain state will also waste a lot of time. Therefore, DQN uses a neural network to approximately replace the Q-table in Q-Learning and finds the most suitable parameter θ by training the neural network, realizing the effective processing of complex high-dimensional problems.

[0120] The training method of the task allocation strategy based on DQN is as follows.

[0121]

[0122]

[0123] To verify the effectiveness of the proposed RLASA method for solving the multi-satellite task scheduling problem, since there is no publicly available test set for this problem, the present invention uses a simulator to generate multiple experimental scenarios. To create a fair experimental environment, both the method and the comparison method are written in Python 3.7.6. The operating environment of the method is an Intel(R) Core(TM) i7-1360P CPU 2.20GHz, Windows 11 system with 32GB of memory.

[0124] The present invention generates a total of 3 major categories and 10 minor categories of experimental scenarios, including small-scale, medium-scale, and large-scale scenarios. Additionally, to fully verify the performance of the method, the present invention designs an ultra-large-scale experimental scenario consisting of 10,000 tasks and 100 agile satellites, which is also the largest experimental example in the publicly available materials in the field of agile satellite task scheduling. The specific elements of the scenario include scenario category, scenario number, number of tasks, satellite scale, scenario start time, scenario end time, and task distribution. The task distribution is divided into two categories: regional distribution and global distribution. Among them, the region is set to 3N - 53N and 74°E - 133°E. The scenario time is set to a certain time interval between 2013 / 04 / 20 / 00:00:00 and 2013 / 04 / 21 / 00:00:00. The specific elements of the scenario are shown in the following table.

[0125] Table 1 Elements of Experimental Scenarios

[0126]

[0127]

[0128] To test the performance of the RLASA method and highlight its advantages in solving the multi-satellite task scheduling problem, the present invention sets two comparison methods for it, namely the large neighborhood search method based on the adaptive assignment mechanism (Adaptive task assignment based ALNS, A-ALNS), and the adaptive large neighborhood search method based on random task assignment (Random task assignment based ALNS, R-ALNS).

[0129] (1) Large neighborhood search method based on the adaptive assignment mechanism: Based on the ALNS / TPF method for solving the single-satellite scheduling problem, by defining multiple task assignment strategies and introducing an adaptive task assignment mechanism, the A-ALNS method is designed. This is also one of the most prominent methods for solving the multi-satellite task scheduling problem currently.

[0130] (2) Adaptive large neighborhood search method based on random task assignment: The ALNS method has been proven to be able to efficiently solve the single-satellite scheduling problem. On this basis, a random task assignment strategy is added as the second comparison method of the present invention.

[0131] The experimental results statistically analyzed by the present invention include four parts: First, the return rate and solution time of each method are statistically analyzed; second, the training process of the method is analyzed; third, the impact of adding reinforcement learning on the return rate of the method is statistically analyzed.

[0132] Table 2 shows the performance of the RLASA method and other comparison methods in terms of the return rate.

[0133] Table 2 Yields of Each Method (%)

[0134]

[0135] It can be found from Table 2 that among all methods, the RLASA method has the most prominent solution quality. In all small-scale, medium-scale, and large-scale experiments, the RLASA method achieved the best yield rate (tied for first with the A-ALNS method in Scenario 2). Moreover, the RLASA method can almost schedule all tasks in some small-scale scenarios, with a yield rate of over 99%. Compared with the RLASA method, the optimization effects of the A-ALNS method and the R-ALNS method are slightly worse, but they can also achieve almost the same yield rate as RLASA in some scenarios, which also indicates that the two methods also have good performance in solving the multi-satellite task scheduling problem.

[0136] Under the same single-satellite scheduling method, the yield rates of the DRL-TASASM mechanism (RL) and the task random allocation mechanism (Random) were respectively counted. It can be found that under the action of the DRL-TASASM mechanism, the RLASA method achieved higher yield rates in all experimental scenarios, which fully indicates that the DRL-TASASM mechanism can effectively improve the performance of the method in solving the multi-satellite task scheduling problem.

[0137] Figure 4 Describes the training situation of DQN in different scenarios. It can be seen from the figure that the network model shows an oscillating and unstable phenomenon at the initial stage of training, but as the training progresses, the loss function value gradually converges and the network model tends to be stable.

[0138] For the multi-satellite task scheduling problem, the present invention proposes an adaptive search method RLASA based on reinforcement learning. The method adopts a two-layer optimization framework and realizes iterative optimization of the solution through multiple interactions between the upper-layer task allocation and the lower-layer single-satellite scheduling. The design focuses on the upper-layer task allocation part. According to the problem domain knowledge, eight task allocation strategies are designed, an MDP model for task allocation strategy selection is constructed, and a task allocation strategy adaptive selection mechanism DRL-TASASM based on deep reinforcement learning is designed. This mechanism uses DQN to select the best allocation strategy for each interaction, which not only ensures the low complexity of the method but also significantly improves the quality of the task allocation solution. Finally, small-scale, medium-scale, and large-scale experimental scenarios are designed to verify the optimization performance of the method. The experimental results show that compared with the comparative methods, the RLASA method has outstanding solution quality and solution speed in both small-scale, medium-scale, and large-scale scenarios.

[0139] The above specific embodiments do not constitute a limitation to the protection scope of this application. Those skilled in the art should understand that various modifications, combinations, sub - combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principle of this application shall be included within the protection scope of this application.

Claims

1. A method for adaptively selecting a multi-satellite task allocation strategy based on deep reinforcement learning, characterized in that, The RLASA (Reinforcement Learning-based Adaptive Search Algorithm) for multi-satellite task scheduling realizes the adaptive selection of multi-satellite task allocation strategies by applying the RLASA to a multi-satellite task scheduling model. The specific steps are as follows: There are three types of parameters: satellite scheduling task attribute parameters, satellite parameters, and visible time window attribute parameters. The satellite scheduling task set T = {t i | 1 ≤ i ≤ N}, where N represents the number of tasks. The attribute of each task t i is represented by a triple (p i , l i , c i ). p i is the priority of task t i , l i is the execution time of task t i , and c i is the minimum imaging quality specified by task t i . The satellite set SR = {sr j | 1 ≤ j ≤ M}, where M represents the satellite scale; the attributes of each satellite sr j are represented by a triple , where γ jt represents the roll angle of the j-th satellite at time t, π jt represents the pitch angle of the j-th satellite at time t, represents the yaw angle of the j-th satellite at time t; The set of visible time windows \(W = \{w ijk |1\leq i\leq N, 1\leq j\leq M, 1\leq k\leq |w ij |\}\), where \(w ijk represents the \(k\)-th visible time window of task \(t i on the \(j\)-th satellite, \(|w ij |\) is the number of visible time windows of task \(t i on the \(j\)-th satellite, and \(w ijk is represented by a triple \((st ijk , et ijk , d ijk )\). Here, \(st ijk is the start time of time window \(w ijk , et ijk is the end time of time window \(w ijk , and \(d ijk is the time length of time window \(w ijk ;\) Establish a multi-satellite mission scheduling model, and set the decision variables of the multi-satellite mission scheduling problem as (x ijk , u ijk ). x ijk is a binary variable. When x ijk = 1, it means that task t i is executed within the k-th visible time window of the j-th satellite. Otherwise, x ijk = 0. The decision variable u ijk represents the start time of task t i within the time window w ijk ; The constructed multi-satellite task scheduling model includes the following steps: S1. Set the objective function of the multi-satellite task scheduling problem, which represents maximizing the revenue of the scheduled tasks. Specifically: S2. Set the task execution uniqueness constraint, which means that each task can be executed at most once and cannot be executed repeatedly. Specifically: S3. Set that tasks must be executed within the visible time window to ensure the effectiveness of task execution. Specifically: S4. Set that different tasks executed by the same satellite need to meet the transition time constraint. Specifically: S5. Set the calculation formula for the attitude transition time between tasks, where a1, a2, a3, and a4 represent the transition speeds of four different attitude angles, and Δg represents the attitude angle difference between tasks. Specifically: S6. Determine the imaging quality of the task and set the imaging quality q of the task i Higher than the specified minimum imaging quality, where is the best imaging quality moment in the time window w ijk and can be calculated through the formula The specific calculation is as follows:

2. The method according to claim 1, wherein The RLASA method includes a satellite task scheduling stage and a reinforcement learning stage; The satellite task scheduling stage includes: obtaining tasks to be allocated, generating task allocation tasks, single-line scheduling, and generating a scheduling result; the reinforcement learning stage includes: calculating the revenue of the plan, reporting, updating the state, inputting to the neural network, and determining the task allocation strategy; In the satellite task scheduling stage, an allocation plan is generated based on the task allocation strategy of this round. Subsequently, each satellite independently conducts task scheduling in parallel, and finally, the revenue of the scheduling plan is obtained; In the reinforcement learning execution stage, the revenue of the previous round of scheduling plan is used to calculate the return, update the current state, input it to the neural network, and select the action with the largest Q value as the task allocation strategy for the next round; Specifically, it includes the following steps: S1. In the initial solution solving stage, set the task set T, satellite set SR, task assignment strategy set A, neighborhood construction operator O N , the parameters θ of the trained deep Q-network DQN, the objective function F(X), and the number of task assignments MA; input the task set and satellite resource set into the established task scheduling model to generate an initial solution operation, which specifically includes: sorting the tasks in descending order according to the revenue value, then randomly assigning the sorted tasks to different satellites, and then generating the initial solution by using the method of arranging predecessors, and then entering the main loop; S2. The main loop includes the satellite task scheduling stage. An allocation plan is generated based on the task allocation strategy of this round. Subsequently, each satellite independently conducts task scheduling in parallel, and finally, the revenue of the scheduling plan is obtained; S3. The reinforcement learning stage. In the reinforcement learning execution stage, the revenue of the previous round of scheduling plan is used to calculate the return, update the current state, input it to the neural network, and select the action with the largest Q value as the task allocation strategy for the next round; S4. Generate a task allocation plan according to the task allocation strategy and enter the next round of task execution.

3. The method according to claim 1, wherein The RLASA integrates a DRL-TASASM (Deep Reinforcement Learning-based Task Allocation Strategy Adaptive Selection Mechanism) for task allocation problems and an ALSA (Adaptive Local Search Algorithm) based on frequent patterns for single-satellite scheduling problems. Specifically, it includes: In the DRL-TASASM, the action is defined as selecting different task allocation strategies, and the state is designed as an index related to the single-satellite or overall scheduling plan. The agent adopts the DQN method in deep reinforcement learning. In each iteration of the method, the agent takes different state components as the input of the neural network, outputs the Q values under different allocation strategies, and then selects the best task allocation strategy according to the greedy strategy, thus completing the solution of the task allocation problem; In the ALSA, single-satellite scheduling is performed for each satellite to obtain the initial solution of the current single-satellite scheduling. If the termination condition of the single-satellite scheduling is not reached, a new solution is generated, and the initial solution is updated to the new solution. If the conditions for the new solution are met, the solution set is updated to the optimal solution set, thereby realizing adaptive update of the solution set based on the frequent pattern.

Images