A master-slave satellite task allocation method based on double-layer backtracking mechanism

Abstract

The application discloses a master-slave satellite task allocation method based on a double-layer backtracking mechanism, constructs a master-slave satellite cooperative task planning model with a target function of maximizing the efficiency generated by the master-slave satellite in executing tasks according to obtained master-slave satellite task planning requirements and constraint conditions, performs task planning conversion based on a Markov decision process (MDP), defines a state space, an action space and a reward function, solves the master-slave satellite cooperative task planning model by using a deep reinforcement learning (DQN) algorithm, performs task planning optimization through a double-layer backtracking mechanism, and obtains a task allocation strategy of the master-slave satellite based on the solved master-slave satellite cooperative task planning model. The double-layer backtracking mechanism is designed, a high-cost and low-yield planning decision is dynamically adjusted in a task planning process in a manner of combining global backtracking with local backtracking, resource waste and task conflicts are avoided, and the rationality and robustness of task allocation are improved.

Description

TECHNICAL FIELD

[0001] The application relates to satellite task planning, in particular to a master-slave satellite task allocation method based on a double-layer backtracking mechanism. BACKGROUND

[0002] Under the background of increasingly complex multi-target task requirements, how to realize efficient planning and intelligent collaboration of tasks in a satellite constellation has become an important research topic in the field of satellite task planning.

[0003] Traditional satellite task planning methods usually rely on centralized planning architecture, in which a ground station uniformly schedules the task allocation and execution of all satellites in the constellation. However, this method has limitations such as centralized calculation load, delayed information update, and low resource utilization efficiency. To solve the above problems, distributed task planning methods have gradually become a research hotspot in recent years. Among them, the collaborative task planning mode based on master-slave structure is considered as a relatively effective solution. In this mode, the master satellite in the constellation is responsible for receiving task requests from the ground station and allocating tasks to the slave satellites; the slave satellites make independent decisions based on task requirements and their own resource states and feedback to the master satellite. This distributed architecture can significantly reduce the computational load of a single node, improve the flexibility and robustness of task planning.

[0004] Deep Q network (DQN) combines neural networks and reinforcement learning and can efficiently solve task allocation problems in high-dimensional state space. In master-slave satellite collaborative task planning, DQN can dynamically evaluate task costs and benefits and generate optimal allocation strategies. Combined with the master-slave structure, DQN alleviates the bottleneck of centralized planning, realizes intelligent and distributed optimization of task planning, and improves the operation efficiency of low-orbit remote sensing satellite constellation systems. However, existing task planning still has problems such as resource waste, task conflict, and low efficiency. SUMMARY

[0005] The application provides a master-slave satellite task allocation method based on a double-layer backtracking mechanism to improve the rationality and robustness of task allocation.

[0006] Technical scheme: To solve the above problems, the application adopts a master-slave satellite task allocation method based on a double-layer backtracking mechanism, which includes the following steps:

[0007] (1) Obtain the master-slave satellite task planning requirements;

[0008] (2) According to the master-slave satellite task planning requirements and constraint conditions, construct a master-slave satellite collaborative task planning model with the objective function of maximizing the efficiency of the master-slave satellite executing tasks; the efficiency of the master-slave satellite executing tasks is the weighted sum of the cost required by the master-slave satellite to execute the planning tasks and the benefits generated by the master-slave satellite executing the planning tasks.

[0009] (3) Task planning transformation based on Markov decision process (MDP), define state space, action space and reward function; solve the master-slave satellite cooperative task planning model by using deep reinforcement learning (DQN) algorithm, and optimize the task planning through double backtracking mechanism, which includes local backtracking and global backtracking, wherein the local backtracking is: in each step of the task planning process, if the backtracking condition is met, the state space is rolled back to the previous state to reselect the action, and the reward function is updated; the global backtracking is: periodically evaluate the global income of the task allocation strategy, if the global income is significantly lower than the expected threshold, then re-optimize the task planning strategy;

[0010] (4) Obtain the task allocation strategy of the master-slave satellite based on the solved master-slave satellite cooperative task planning model.

[0011] Further, the master-slave satellite cooperative task planning model is:

[0012]

[0013] Wherein, F represents the objective function, f1(s, a) represents the cost of the master-slave satellite executing the planned task, f2(s, a) represents the urgency income of the master-slave satellite, f3(s, a) represents the load balancing income of the master-slave satellite, λ1 represents the weight of the sub-target resource cost, λ2 represents the weight of the sub-target task income, λ3 represents the weight of the sub-target load balancing, T represents the total task planning time, k represents the time step in the task planning, e j represents the end time of the idle time window, o j represents the start time of the idle time window, d task represents the task working time, o task represents the task start execution time, c i represents the remaining storage capacity of satellite i, c task represents the storage required by the task, g i represents the execution frequency of satellite i in the planning period, g max represents the maximum on-off frequency of satellite in a certain period.

[0014] Further, the cost of the master-slave satellite executing the planned task includes the slave satellite task working time cost and the slave satellite task storage cost, specifically:

[0015]

[0016] Wherein, d task (t i ) represents the task working time, c task (t i ) represents the storage capacity required by the task, dtask_max denotes the maximum working duration of a task, c task_max denotes the maximum storage capacity of a task; a1 denotes the weight of the task execution duration, and a2 denotes the weight of the storage capacity required by the task;

[0017] The benefits generated by the master-slave satellite task planning include emergency degree benefits and load balancing benefits.

[0018] The master-slave satellite emergency degree benefits are calculated according to the task emergency degree and the slave satellite decision variable, and are specifically as follows:

[0019]

[0020] Wherein, p task (t i ) denotes the task emergency degree, p task_max denotes the maximum emergency degree of a task, x i denotes the slave satellite decision variable;

[0021] The master-slave satellite load balancing degree benefits are calculated according to the number of tasks executed by the satellite and the storage capacity, and are specifically as follows:

[0022]

[0023] Wherein, g max (w i ) denotes the maximum number of tasks executed by the satellite w i in the task planning period, g(w i ) denotes the number of tasks executed by the satellite w i from the beginning to now, c(w i ) denotes the remaining storage capacity of the satellite w i , and c max (w i ) denotes the maximum storage capacity of the satellite w i .

[0024] Further, the backtracking condition of the local backtracking is that any one of the following conditions is met:

[0025] λ1f1(s,a)>C max

[0026] λ2f2(s,a)+λ3f3(s,a)<R min

[0027] Wherein, C max denotes the expected cost threshold, and R min denotes the expected benefit threshold.

[0028] The update formula of the reward function is as follows:

[0029] R adjustedR(s t ,a t )+δ·ΔK(s t ,a t )

[0030] wherein, δ represents a knowledge base correction coefficient, ΔK(s t ,a t ) represents an incremental change of action reward after backtracking, and R(s t ,a t ) represents a reward function before updating.

[0031] Further, the DQN algorithm in the step (3) is optimized, the DQN algorithm combines a brain-like cognitive mechanism to establish a hybrid enhanced knowledge base, and the hybrid enhanced knowledge base is used to solve the master-slave satellite cooperative task planning model, and the DQN algorithm combining the brain-like cognitive mechanism comprises:

[0032] The brain-like cognitive mechanism introduces a cognitive model, dynamically adjusts an action selection strategy, and combines a Q value of the DQN algorithm with a brain weight to adjust an action selection probability;

[0033] Under the brain framework, priorities are given to experience samples in an experience replay module of the DQN algorithm in combination with task values and learning importance;

[0034] The brain mechanism analyzes the similarity of a new task, associates a current allocation strategy with historical experience, and dynamically updates the hybrid enhanced knowledge base.

[0035] Further, the formula for adjusting the action selection probability is:

[0036]

[0037] wherein, π(a|s) represents a probability of selecting an action a under a state s, Q(s,a) represents an action value of the action a under the state s predicted by the DQN algorithm, and W(s,a) represents a weight given by the brain-like cognitive mechanism;

[0038] W(s,a)=f(φ(s),H)

[0039] wherein, φ(s) represents a current state feature vector, H represents historical task data stored in the hybrid enhanced knowledge base, and f represents a brain weight generation function.

[0040] Further, the priority given to the experience samples in the experience replay module of the DQN algorithm is:

[0041]

[0042] wherein, P(i) represents a probability of selecting a sample i, β represents an adjustment parameter, and p ipriority of sample i is represented;

[0043] p i defined as the absolute value of the time difference error TD-error:

[0044]

[0045] wherein gamma represents a discount factor, R(s,a) represents an immediate reward in the current state, representing the maximum action value.

[0046] Further, the update formula of the hybrid enhanced knowledge base is:

[0047] H new = H old ∪ {(s,a,r,s'} current}

[0048] wherein H new represents the updated knowledge base content, H old represents the existing knowledge base content,

[0049] (s,a,r,s') current represents the experience data of the current task.

[0050] The application also adopts a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0051] The application also adopts a computer readable storage medium, which stores a computer program, wherein the computer program is executed by the processor to implement the steps of the above method.

[0052] Beneficial effects: Compared with the prior art, the application has the following advantages: based on the master-slave satellite distributed collaborative architecture, the distributed task planning and optimization strategy is adopted, the calculation load of centralized planning is significantly reduced, the real-time performance of task allocation and the utilization efficiency of satellite resources are improved, a new type of solution is provided for complex task requirements. A double-backtracking mechanism is designed, which dynamically adjusts the planning decision of high cost and low benefit in the task planning process by combining global backtracking and local backtracking, avoids resource waste and task conflict, and improves the rationality and robustness of task allocation. A hybrid enhanced knowledge base is established by combining the brain-like cognitive mechanism and the deep reinforcement learning (DQN) algorithm to realize optimal task planning, avoid the inefficiency problem caused by random exploration in traditional methods, and effectively improve the rationality of task planning and the intelligence of the system. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1The principle structure block diagram of the master-slave satellite task allocation method in the application.

[0054] Figure 2 The specific flow chart of the master-slave satellite task allocation method in the application. DETAILED DESCRIPTION

[0055] As shown in Figure 1 and Figure 2 , the master-slave satellite task allocation method based on the double-layer backtracking mechanism in the embodiment comprises the following steps:

[0056] Step 1: Obtain the master-slave satellite task planning requirement, and construct a master-slave satellite cooperative task planning model with the maximum efficiency generated by the master-slave satellite executing the task as the objective function according to the master-slave satellite task planning requirement and the constraint condition; the efficiency generated by the master-slave satellite executing the task is the weighted sum of the cost required by the master-slave satellite executing the planning task and the benefit generated by the master-slave satellite executing the planning task.

[0057] Step 1.1: Determine the cost required by the master-slave satellite task planning: the cost required by the master-slave satellite task planning includes the from-satellite task working time cost and the from-satellite task required storage cost, and is specifically:

[0058]

[0059] In the formula, d task (t i ) represents the task working time, c task (t i ) represents the task required storage, d task_max represents the maximum working time of the task, c task_max represents the maximum storage capacity of the task; α1 represents the weight of the task execution time, and α2 represents the weight of the task required storage capacity.

[0060] Step 1.2: Determine the benefit generated by the master-slave satellite task planning, which includes the urgency benefit and the load balancing benefit.

[0061] (1) The master-slave satellite urgency benefit is calculated according to the task urgency and the from-satellite decision variable, and is specifically:

[0062]

[0063] In the formula, p task (t i ) represents the task urgency, p task_max represents the maximum urgency of the task, and x i represents the from-satellite decision variable.

[0064] (2) The master-slave satellite load balancing degree benefit is calculated according to the number of tasks executed by the satellite and the storage capacity, and is specifically:

[0065]

[0066] wherein g max (w i ) represents the maximum number of tasks executed by the satellite w i in the task planning period, g(w i ) represents the number of tasks executed by the satellite w i from the beginning to now, c(w i ) represents the remaining storage capacity of the satellite w i , and c max (w i ) represents the maximum storage capacity of the satellite w i ;

[0067] Step 1.3: Determine the multi-master-slave satellite task planning objective function, which is specifically:

[0068]

[0069] wherein λ1 represents the weight of the sub-target resource cost, λ2 represents the weight of the sub-target task benefit, λ3 represents the weight of the sub-target load balancing, and T represents the total duration of the task planning;

[0070] Step 1.4: Determine the constraint condition, only considering the slave satellite constraint, which is specifically:

[0071] e j -o j ≥d task (5)

[0072] o j ≤o task (6)

[0073] c i ≥c task (7)

[0074] g i ≤g max (8) wherein e j represents the end time of the idle time window, o j represents the start time of the idle time window, d task represents the task working duration, o task represents the task start execution time, c i represents the remaining storage capacity of the satellite i, c task represents the storage required by the task, g i represents the execution number of the satellite i in the planning period, and gmax This indicates the maximum number of times a satellite can be switched on and off within a certain period.

[0075] The master-slave satellite mission planning model is as follows:

[0076]

[0077] Step 2: Based on Markov Decision Process (MDP), perform task planning transformation, defining the state space, action space, and reward function; use Deep Reinforcement Learning (DQN) algorithm to solve the master-slave satellite collaborative task planning model, and optimize task planning through a two-layer backtracking mechanism, including local backtracking and global backtracking. The local backtracking is: for each step of task planning, if the backtracking condition is met, the state space is rolled back to the previous state, actions are reselected, and the reward function is updated; the global backtracking is: periodically evaluate the global benefit of the task allocation strategy, and if the global benefit is significantly lower than the expected threshold, the task planning strategy is re-optimized.

[0078] A two-layer backtracking mechanism is introduced for task planning optimization. By combining global and local backtracking, unreasonable decisions in the task planning process are dynamically adjusted. On the one hand, local backtracking quickly corrects high-cost, low-return decisions in single-step task allocation; on the other hand, global backtracking comprehensively analyzes the overall planning effect, avoiding task planning from getting stuck in local optima and ensuring the global optimality and robustness of task allocation.

[0079] (1) For local backtracking, if any of the following conditions are met, the local backtracking mechanism is triggered, and the state is rolled back to the previous state s. t-1 And then select a new action. The trigger condition is:

[0080] λ1f1(s,a)>C max (9)

[0081] λ2f2(s,a)+λ3f3(s,a)<R min (10)

[0082] In the formula, C max R represents the expected cost threshold. min This represents the expected return threshold.

[0083] In the case of backtracking, the update formula for the reward function is:

[0084] R adjusted =R(s) t ,a t )+δ·ΔK(s t ,a t (11)

[0085] where δ represents the knowledge base correction coefficient, ΔK(s t ,a t represents the incremental change in action reward after backtracking.

[0086] (2) For global backtracking, global backtracking mainly targets the overall strategy failure problem that may occur in the task planning process. By re-evaluating the global reward of the overall planning, the global optimality of task allocation is ensured. Specifically, it includes the following steps:

[0087] a. Global reward monitoring: At different stages of task planning, the comprehensive reward of all task allocation schemes is regularly evaluated and compared with the optimal threshold set by the current system.

[0088] b. Trigger backtracking condition: When the global reward is significantly lower than the expected threshold, trigger the global backtracking mechanism; for example, when multiple satellite tasks conflict or resource utilization is uneven, resulting in a decrease in reward.

[0089] c. Overall strategy reconstruction: Combine the mixed enhanced intelligent framework, use the brain-like knowledge base experience and the prediction ability of deep reinforcement learning to re-optimize the task planning strategy.

[0090] d. Global feedback update: Store the feedback information collected during the backtracking process in the mixed enhanced knowledge base to provide guidance for subsequent task planning and improve the adaptive ability of the system.

[0091] Optimize the deep reinforcement learning DQN algorithm, and combine the brain-like cognitive mechanism and deep reinforcement learning (DQN) algorithm to establish a mixed enhanced knowledge base to achieve optimal task planning.

[0092] Based on the DQN algorithm, introduce the mixed enhanced knowledge base, combine the logical reasoning ability of brain-like mechanism and the efficient learning ability of machine intelligence, and thus optimize the action selection and experience storage strategy. The mixed enhanced knowledge base integrates human decision-making experience and machine learning models to efficiently update knowledge and intelligently infer task allocation.

[0093] The mixed enhanced knowledge base optimizes the action selection strategy of DQN by integrating the features of brain-like cognitive mechanism and machine intelligence automatic learning, avoiding the randomness and inefficiency of exploration behavior in traditional algorithms.

[0094] The DQN algorithm combined with brain-like cognitive mechanism includes:

[0095] The brain-like mechanism dynamically adjusts the action selection strategy by introducing a cognitive model, making exploration more intelligent. The Q value is combined with the brain weight to adjust the action probability:

[0096]

[0097] In the formula, π(a|s) represents the probability of selecting action a in state s, Q(s, a) represents the action value predicted by DQN, and W(s, a) represents the weight given by the brain-like cognitive mechanism;

[0098] The brain-like weight is generated by analyzing the current state feature vector φ(s) and the historical task data H stored in the knowledge base:

[0099] W(s, a) = f(φ(s), H) (13)

[0100] In the formula, f represents the brain-like weight generation function.

[0101] The hybrid enhanced knowledge base stores efficient experience samples in real time in task allocation, and improves the training efficiency through the priority experience replay module of DQN. In the brain-like framework, the priority weight is given to the experience samples by combining task value and learning importance, and the storage and sampling strategies are optimized:

[0102]

[0103] In the formula, P(i) represents the probability of selecting sample i, β represents the adjustment parameter, and p i represents the priority of sample i, which is defined as the absolute value of time difference error TD-error:

[0104]

[0105] In the formula, γ represents the discount factor, and R(s, a) represents the immediate reward in the current state.

[0106] The brain-like mechanism analyzes the similarity of new tasks, associates the current allocation strategy with historical experience, and dynamically updates the hybrid enhanced knowledge base.

[0107] The update formula of the knowledge base is:

[0108] H new = H old ∪{(s,a,r,s′) current} (16)

[0109] In the formula, H new represents the updated knowledge base content, H old represents the existing knowledge base content, and (s, a, r, s′) current represents the experience data of the current task.

[0110] Step 3: Based on the solved master-slave satellite cooperative task planning model, the task allocation strategy of the master-slave satellite is obtained.

Claims

1. A master-slave satellite task allocation method based on a two-layer backtracking mechanism, characterized in that, Includes the following steps: (1) Obtain master-slave satellite mission planning requirements; (2) Based on the master-slave satellite mission planning requirements and constraints, construct a master-slave satellite collaborative mission planning model with the objective function of maximizing the efficiency generated by the master-slave satellites in performing missions; the efficiency generated by the master-slave satellites in performing missions is the weighted sum of the cost required for the master-slave satellites to perform the planned missions and the benefits generated by the master-slave satellites in performing the planned missions. (3) Based on Markov Decision Process (MDP), the task planning transformation is carried out, and the state space, action space and reward function are defined. The master-slave satellite collaborative task planning model is solved by Deep Reinforcement Learning (DQN) algorithm. The task planning is optimized through a two-layer backtracking mechanism, which includes local backtracking and global backtracking. The local backtracking is: for each step of the task planning process, if the backtracking condition is met, the state space is rolled back to the previous state to reselect the action and update the reward function. The global backtracking is: periodically evaluate the global benefit of the task allocation strategy. If the global benefit is significantly lower than the expected threshold, the task planning strategy is re-optimized. (4) Based on the solved master-slave satellite collaborative mission planning model, obtain the mission allocation strategy for master-slave satellites; The master-slave satellite collaborative mission planning model is as follows: in, Describe the objective function. This indicates the cost required for the master and slave satellites to perform the planned mission. Indicates the urgency benefits of master and slave satellites. This indicates the benefit of master-slave satellite load balancing. The weights represent the resource costs of the sub-objectives. This indicates the weight of the sub-target task's reward. This indicates the weight of the sub-target load balancer. Indicates the total duration of the task planning. This represents the time step in task planning. Indicates the end time of the idle time window. Indicates the start time of the idle time window. Indicates the task duration. Indicates the start time of task execution. Indicates satellite Remaining storage capacity, This indicates the storage required for the task. Indicates satellite Number of executions within the planning period This indicates the maximum number of times a satellite can be switched on and off within a certain period. The backtracking condition for the local backtracking is to satisfy any one of the following: in, Indicates the expected cost threshold. This represents the expected return threshold; The update formula for the reward function is: in, This represents the knowledge base correction coefficient. This indicates the incremental change in the reward of the action after backtracking. This represents the reward function before the update.

2. The master-slave satellite task allocation method according to claim 1, characterized in that, The costs required for the master and slave satellites to perform the planned tasks include the slave satellite mission's working time cost and the storage cost required by the slave satellite mission, specifically: in, Indicates the task duration. Indicates the storage capacity required for the task. Indicates the maximum working time of the task. Indicates the maximum storage capacity of the task; The weights representing task execution time The weight representing the storage capacity required by the task; The benefits generated by master and slave satellites performing planned tasks include benefits from urgency and load balancing. The master-slave satellite urgency benefit is calculated based on mission urgency and slave satellite decision variables, specifically as follows: in, Indicates the urgency of the task. Indicates the maximum urgency of the task. Indicates the decision variable from the star; The benefits of master-slave satellite load balancing are calculated based on the number of missions performed by the satellites and their storage capacity, specifically: in, Indicates the satellite within the mission planning period The most times the task was executed. Indicates satellite The number of tasks performed from the beginning to the present. Indicates satellite Remaining storage capacity, Indicates satellite Maximum storage capacity.

3. The master-slave satellite task allocation method according to claim 2, characterized in that, The DQN algorithm in step (3) is optimized by combining the DQN algorithm with a brain-like cognitive mechanism to establish a hybrid augmented knowledge base. The hybrid augmented knowledge base is then used to solve the master-slave satellite collaborative mission planning model. The DQN algorithm combined with the brain-like cognitive mechanism includes: A brain-like cognitive mechanism is introduced into the cognitive model to dynamically adjust the action selection strategy. The Q-value of the DQN algorithm is combined with brain-like weights to adjust the action selection probability. Within the brain-like framework, priority is assigned to experience samples in the DQN algorithm's experience replay module, taking into account task value and learning importance. The brain-like mechanism analyzes the similarity of new tasks, associates the current allocation strategy with historical experience, and dynamically updates the hybrid augmented knowledge base.

4. The master-slave satellite task allocation method according to claim 3, characterized in that, The formula for the probability of selecting the adjustment action is: in, Indicates the state Select action The probability, This indicates the state predicted by the DQN algorithm. Next action The value of the action, This indicates the weight assigned by brain-like cognitive mechanisms; in, Represents the feature vector of the current state. This represents historical task data stored in the hybrid augmented knowledge base; This represents the brain-like weight generation function.

5. The master-slave satellite task allocation method according to claim 4, characterized in that, The priority assigned to the experience samples in the DQN algorithm experience replay module is as follows: in, Indicates sample The probability of being selected. Indicates the adjustment parameter. Indicates sample Priority; Defined as the absolute value of the time difference error (TD-error): in, Indicates the discount factor. This represents the immediate reward in the current state. This indicates the maximum value of the action.

6. The master-slave satellite task allocation method according to claim 5, characterized in that, The update formula for the hybrid augmented knowledge base is: in, This indicates the updated knowledge base content. This represents existing knowledge base content. This represents the experience data for the current task.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images