Master-slave satellite task allocation method based on double-layer backtracking mechanism

Abstract

The invention discloses a master-slave satellite task allocation method based on a double-layer backtracking mechanism, and the method comprises the steps: constructing a master-slave satellite cooperative task planning model which takes the maximization of the efficiency generated by the execution of a task by a master-slave satellite as a target function according to the obtained master-slave satellite task planning demands and constraint conditions; task planning conversion is carried out based on a Markov decision process MDP, and a state space, an action space and a reward function are defined; solving the master-slave satellite collaborative task planning model by using a deep reinforcement learning DQN algorithm, and performing task planning optimization through a double-layer backtracking mechanism; and obtaining a task allocation strategy of the master and slave satellites based on the solved master and slave satellite collaborative task planning model. A double-layer backtracking mechanism is designed, and a high-cost and low-income planning decision is dynamically adjusted in a task planning process through a mode of combining global backtracking and local backtracking, so that resource waste and task conflicts are avoided, and the reasonability and robustness of task allocation are improved.

Description

Technical Field

[0001] The invention relates to satellite mission planning, and in particular to a master-slave satellite mission allocation method based on a double-layer backtracking mechanism. Background Art

[0002] Against the backdrop of increasingly complex multi-objective mission requirements, how to achieve efficient planning and intelligent coordination of tasks within satellite constellations has become an important research topic in the current field of satellite mission planning.

[0003] Traditional satellite mission planning methods usually rely on a centralized planning architecture, where the ground station uniformly schedules the task allocation and execution of all satellites in the constellation. However, this method has the limitations of concentrated computing load, delayed information update, and low resource utilization efficiency. In order to solve the above problems, distributed mission planning methods have gradually become a research hotspot in recent years. Among them, the collaborative mission planning model based on the master-slave structure is considered to be a more effective solution. In this model, the master satellite in the constellation is responsible for receiving task requests issued by the ground station and assigning tasks to the slave satellites; the slave satellites make independent decisions based on the task requirements and their own resource status and feedback to the master satellite. This distributed architecture can significantly reduce the computing load of a single node and improve the flexibility and robustness of mission planning.

[0004] The deep Q network (DQN) combines neural networks with reinforcement learning to efficiently solve the task allocation problem in high-dimensional state space. In the master-slave satellite collaborative task planning, DQN can dynamically evaluate the task cost and benefit and generate the optimal allocation strategy. Combined with the master-slave structure, DQN alleviates the bottleneck of centralized planning, realizes the intelligent and distributed optimization of task planning, and improves the operating efficiency of the low-orbit remote sensing satellite constellation system. However, the existing task planning still has problems such as resource waste, task conflict and low efficiency. Summary of the invention

[0005] Purpose of the invention: In view of the above shortcomings, the present invention provides a master-slave satellite task allocation method based on a double-layer backtracking mechanism to improve the rationality and robustness of task allocation.

[0006] Technical solution: To solve the above problems, the present invention adopts a master-slave satellite task allocation method based on a double-layer backtracking mechanism, comprising the following steps:

[0007] (1) Obtaining the master and slave satellite mission planning requirements;

[0008] (2) According to the planning requirements and constraints of the master-slave satellite missions, a master-slave satellite collaborative mission planning model is constructed with the objective function of maximizing the efficiency of the master-slave satellite missions. The efficiency of the master-slave satellite missions is the weighted sum of the cost required for the master-slave satellite to execute the planned mission and the benefits generated by the master-slave satellite to execute the planned mission.

[0009] (3) Based on the Markov decision process MDP, the task planning transformation is performed to define the state space, action space and reward function; the deep reinforcement learning DQN algorithm is used to solve the master-slave satellite collaborative task planning model, and the task planning is optimized through a double-layer backtracking mechanism. The double-layer backtracking mechanism includes local backtracking and global backtracking. The local backtracking is: for each step of the task planning process, if the backtracking condition is met, the state space is rolled back to the previous state to reselect the action and update the reward function; the global backtracking is: regularly evaluate the global benefits of the task allocation strategy. If the global benefits are significantly lower than the expected threshold, the task planning strategy is re-optimized;

[0010] (4) Based on the solved master-slave satellite collaborative task planning model, the task allocation strategy of the master and slave satellites is obtained.

[0011] Furthermore, the master-slave satellite collaborative mission planning model is:

[0012]

[0013] Where F represents the objective function, f1(s,a) represents the cost required for the master and slave satellites to perform the planned tasks, f2(s,a) represents the urgency benefit of the master and slave satellites, f3(s,a) represents the load balancing benefit of the master and slave satellites, λ1 represents the weight of the sub-goal resource cost, λ2 represents the weight of the sub-goal task benefit, λ3 represents the weight of the sub-goal load balancing, T represents the total duration of the task planning, k represents the time step in the task planning, and e represents the time step of the task planning. j Indicates the end time of the idle time window, o j Indicates the start time of the idle time window, d task Indicates the task working time, o task Indicates the time when the task starts executing, c i represents the remaining storage capacity of satellite i, c task represents the storage required by the task, g i represents the number of executions of satellite i in the planning cycle, g max Indicates the maximum number of times the satellite is turned on and off within a certain period.

[0014] Furthermore, the cost required for the master and slave satellites to perform the planned tasks includes the cost of the slave satellite mission working time and the storage cost required for the slave satellite mission, specifically:

[0015]

[0016] Among them, d task (t i ) represents the task working time, c task (t i ) represents the storage capacity required by the task, dtask_max Indicates the maximum working time of the task, c task_max represents the maximum storage capacity of the task; α1 represents the weight of the task execution time, and α2 represents the weight of the storage capacity required by the task;

[0017] The benefits generated by master-slave satellite mission planning include urgency benefits and load balancing benefits;

[0018] The urgency benefit of the master and slave satellites is calculated based on the mission urgency and the slave satellite decision variables, specifically:

[0019]

[0020] Among them, p task (t i ) represents the urgency of the task, p task_max Indicates the maximum urgency of the task, x i represents the decision variable from the star;

[0021] The load balancing benefit of the master and slave satellites is calculated based on the number of satellite missions and storage capacity, specifically:

[0022]

[0023] Among them, g max (w i ) represents the satellite w in the mission planning period i The maximum number of task executions, g(w i ) represents satellite w i The number of times the task has been executed from the beginning to now, c(w i ) represents satellite w i Remaining storage capacity, c max (w i ) represents satellite w i Maximum storage capacity.

[0024] Furthermore, the local backtracking condition is to satisfy any one of the following:

[0025] λ1f1(s,a)>C max

[0026] λ2f2(s,a)+λ3f3(s,a)<R min

[0027] Among them, C max represents the expected cost threshold, R min represents the expected return threshold;

[0028] The update formula of the reward function is:

[0029] R adjusted=R(s t ,a t )+δ·ΔK(s t ,a t )

[0030] Among them, δ represents the knowledge base correction coefficient, ΔK(s t ,a t ) represents the incremental change in action return after backtracking, R(s t ,a t ) represents the reward function before updating.

[0031] Furthermore, the DQN algorithm in step (3) is optimized, the DQN algorithm is combined with the brain-like cognitive mechanism to establish a hybrid enhanced knowledge base, and the hybrid enhanced knowledge base is used to solve the master-slave satellite collaborative task planning model. The DQN algorithm combined with the brain-like cognitive mechanism includes:

[0032] The brain-like cognitive mechanism introduces the cognitive model to dynamically adjust the action selection strategy, and adjusts the action selection probability by combining the Q value of the DQN algorithm with the brain-like weight;

[0033] In the brain-like framework, the experience samples in the experience replay module of the DQN algorithm are given priority based on the task value and learning importance.

[0034] The brain-like mechanism analyzes the similarities of new tasks, associates the current allocation strategy with historical experience, and dynamically updates the hybrid enhanced knowledge base.

[0035] Furthermore, the formula for adjusting the action selection probability is:

[0036]

[0037] Among them, π(a|s) represents the probability of selecting action a in state s, Q(s,a) represents the action value of action a in state s predicted by the DQN algorithm, and W(s,a) represents the weight assigned by the brain-like cognitive mechanism;

[0038] W(s,a)=f(φ(s),H)

[0039] Among them, φ(s) represents the current state feature vector, H represents the historical task data stored in the hybrid enhanced knowledge base; f represents the brain-like weight generation function.

[0040] Furthermore, the priority given to the experience samples in the DQN algorithm experience playback module is:

[0041]

[0042] Among them, P(i) represents the probability of sample i being selected, β represents the adjustment parameter, and p irepresents the priority of sample i;

[0043] p i Defined as the absolute value of the time difference error TD-error:

[0044]

[0045] Among them, γ represents the discount factor, R(s,a) represents the immediate reward in the current state, Represents the maximum action value.

[0046] Furthermore, the update formula of the hybrid enhanced knowledge base is:

[0047] H new =H old ∪{(s,a,r,s′) current}

[0048] Among them, H new represents the updated knowledge base content, H old Represents the existing knowledge base content,

[0049] (s,a,r,s′) current Represents the experience data of the current task.

[0050] The present invention also adopts a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0051] The present invention also adopts a computer-readable storage medium on which a computer program is stored, and the computer program implements the steps of the above method when executed by a processor.

[0052] Beneficial effects: Compared with the prior art, the significant advantage of the present invention is that it is based on a distributed collaborative architecture of master-slave satellites, adopts a distributed task planning and optimization strategy, significantly reduces the computational load of centralized planning, improves the real-time performance of task allocation and the efficiency of satellite resource utilization, and provides a new solution for complex task requirements. A two-layer backtracking mechanism is designed, which dynamically adjusts high-cost, low-benefit planning decisions during the task planning process by combining global backtracking with local backtracking, avoids resource waste and task conflicts, and improves the rationality and robustness of task allocation. A hybrid enhanced knowledge base is established by combining brain-like cognitive mechanisms with deep reinforcement learning (DQN) algorithms to achieve optimal task planning, avoiding the inefficiency problems caused by random exploration in traditional methods, and effectively improving the rationality of task planning and system intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1It is a principle structure block diagram of the master-slave satellite task allocation method in the present invention.

[0054] Figure 2 It is a specific flow chart of the master-slave satellite task allocation method in the present invention. DETAILED DESCRIPTION

[0055] like Figure 1 and Figure 2 As shown, a master-slave satellite task allocation method based on a double-layer backtracking mechanism in this embodiment includes the following steps:

[0056] Step 1: Obtain the requirements for master-slave satellite mission planning. According to the requirements and constraints, construct a master-slave satellite collaborative mission planning model with the objective function of maximizing the efficiency generated by the master-slave satellites performing their missions. The efficiency generated by the master-slave satellites performing their missions is the weighted sum of the cost required for the master-slave satellites to perform their planned missions and the benefits generated by the master-slave satellites performing their planned missions.

[0057] Step 1.1: Determine the cost required for master-slave satellite mission planning: The cost required for master-slave satellite mission planning includes the cost of slave satellite mission working time and the storage cost required for slave satellite mission, specifically:

[0058]

[0059] Where, d task (t i ) represents the task working time, c task (t i ) represents the storage required by the task, d task_max Indicates the maximum working time of the task, c task_max represents the maximum storage capacity of the task; α1 represents the weight of the task execution time, and α2 represents the weight of the storage capacity required by the task;

[0060] Step 1.2: Determine the benefits generated by the master-slave satellite mission planning. The benefits generated by the master-slave satellite mission planning include urgency benefits and load balancing benefits;

[0061] (1) The urgency benefit of the master and slave satellites is calculated based on the mission urgency and the slave satellite decision variables, specifically:

[0062]

[0063] In the formula, p task (t i ) represents the urgency of the task, p task_max Indicates the maximum urgency of the task, x i represents the decision variable from the star;

[0064] (2) The load balancing benefit of the master and slave satellites is calculated based on the number of satellite missions and storage capacity, specifically:

[0065]

[0066] In the formula, g max (w i ) represents the satellite w in the mission planning period i The maximum number of task executions, g(w i ) represents satellite w i The number of times the task has been executed from the beginning to now, c(w i ) represents satellite w i Remaining storage capacity, c max (w i ) represents satellite w i Maximum storage capacity;

[0067] Step 1.3: Determine the objective function of multi-master-slave satellite mission planning, specifically:

[0068]

[0069] In the formula, λ1 represents the weight of the sub-goal resource cost, λ2 represents the weight of the sub-goal task benefit, λ3 represents the weight of the sub-goal load balancing, and T represents the total duration of the task planning;

[0070] Step 1.4: Determine the constraints, only considering the star constraints, specifically:

[0071] e j -o j ≥d task (5)

[0072] o j ≤o task (6)

[0073] c i ≥c task (7)

[0074] g i ≤g max (8) In the formula, e j Indicates the end time of the idle time window, o j Indicates the start time of the idle time window, d task Indicates the task working time, o task Indicates the time when the task starts executing, c i represents the remaining storage capacity of satellite i, c task represents the storage required by the task, g i represents the number of executions of satellite i in the planning cycle, g maxIndicates the maximum number of times the satellite is turned on and off within a certain period;

[0075] The master-slave satellite mission planning model is:

[0076]

[0077] Step 2: Based on the Markov decision process MDP, the task planning is transformed to define the state space, action space and reward function; the deep reinforcement learning DQN algorithm is used to solve the master-slave satellite collaborative task planning model, and the task planning is optimized through a double-layer backtracking mechanism. The double-layer backtracking mechanism includes local backtracking and global backtracking. The local backtracking is: for each step of the task planning process, if the backtracking condition is met, the state space is rolled back to the previous state to reselect the action and update the reward function; the global backtracking is: regularly evaluate the global benefit of the task allocation strategy. If the global benefit is significantly lower than the expected threshold, the task planning strategy is re-optimized.

[0078] A two-layer backtracking mechanism is introduced to optimize task planning. By combining global backtracking with local backtracking, unreasonable decisions in the task planning process are dynamically adjusted. On the one hand, local backtracking quickly corrects high-cost and low-benefit decisions in single-step task allocation; on the other hand, global backtracking comprehensively analyzes the overall planning effect to avoid task planning falling into local optimality and ensure the global optimality and robustness of task allocation.

[0079] (1) For local backtracking, if any of the following conditions is met, the backtracking mechanism is triggered locally and the state is rolled back to the previous state s t-1 , and reselect the action. The trigger conditions are:

[0080] λ1f1(s,a)>C max (9)

[0081] λ2f2(s,a)+λ3f3(s,a)<R min (10)

[0082] In the formula, C max represents the expected cost threshold, R min represents the expected return threshold.

[0083] When backtracking is triggered, the update formula of the reward function is:

[0084] R adjusted =R(s t ,a t )+δ·ΔK(s t ,a t ) (11)

[0085] In the formula, δ represents the knowledge base correction coefficient, ΔK(st ,a t ) represents the incremental change in the payoff of the action after the backtest.

[0086] (2) Global backtracking mainly targets the overall strategy failure problem that may occur during the task planning process. It ensures the global optimality of task allocation by re-evaluating the global benefits of the overall planning. It includes the following steps:

[0087] a. Global benefit monitoring: At different stages of task planning, the comprehensive benefits of all task allocation schemes are regularly evaluated and compared with the optimal threshold set by the current system.

[0088] b. Triggering backtracking conditions: When the global revenue is significantly lower than the expected threshold, the global backtracking mechanism is triggered; for example, when multiple satellite missions conflict or uneven resource utilization leads to a decrease in revenue.

[0089] c. Overall strategy reconstruction: Combine the hybrid enhanced intelligence framework, use the experience of brain-like knowledge base and the predictive ability of deep reinforcement learning to re-optimize the task planning strategy.

[0090] d. Global feedback update: The feedback information collected during the backtracking process is stored in the hybrid enhanced knowledge base to provide guidance for subsequent task planning and improve the system's adaptive ability.

[0091] The deep reinforcement learning (DQN) algorithm is optimized, and a hybrid reinforcement knowledge base is established by combining the brain-like cognitive mechanism with the deep reinforcement learning (DQN) algorithm to achieve optimal task planning.

[0092] Based on the DQN algorithm, a hybrid enhanced knowledge base is introduced to combine the logical reasoning ability of brain-like mechanisms with the efficient learning ability of machine intelligence to optimize action selection and experience storage strategies. The hybrid enhanced knowledge base integrates human decision-making experience and machine learning models to perform efficient knowledge updates and intelligent reasoning on task allocation.

[0093] The hybrid enhanced knowledge base optimizes the action selection strategy of DQN by integrating the characteristics of brain-like cognitive mechanisms and machine intelligent automatic learning, avoiding the randomness and inefficiency of exploration behavior in traditional algorithms.

[0094] The DQN algorithm combines brain-like cognitive mechanisms including:

[0095] The brain-like mechanism makes exploration more intelligent by introducing cognitive models and dynamically adjusting action selection strategies. The Q value is combined with the brain-like weight to adjust the action probability:

[0096]

[0097] Where π(a|s) represents the probability of selecting action a in state s, Q(s,a) represents the action value predicted by DQN, and W(s,a) represents the weight assigned by the brain-like cognitive mechanism;

[0098] The brain-like weights are generated by analyzing the current state feature vector φ(s) and the historical task data H stored in the knowledge base:

[0099] W(s,a)=f(φ(s),H) (13)

[0100] Where f represents the brain-like weight generation function.

[0101] The hybrid enhanced knowledge base stores efficient experience samples in task allocation in real time, and improves training efficiency through the DQN priority experience replay module. Under the brain-like framework, it combines task value and learning importance, assigns priority weights to experience samples, and optimizes storage and sampling strategies:

[0102]

[0103] In the formula, P(i) represents the probability of sample i being selected, β represents the adjustment parameter, and p i Indicates the priority of sample i, which is defined as the absolute value of the time difference error TD-error:

[0104]

[0105] In the formula, γ represents the discount factor, and R(s,a) represents the immediate reward in the current state.

[0106] The brain-like mechanism analyzes the similarities of new tasks, associates the current allocation strategy with historical experience, and dynamically updates the hybrid enhanced knowledge base.

[0107] The update formula of the knowledge base is:

[0108] H new =H old ∪{(s,a,r,s′) current} (16)

[0109] In the formula, H new represents the updated knowledge base content, H old Represents the existing knowledge base content, (s,a,r,s′) current Represents the experience data of the current task.

[0110] Step 3: Based on the solved master-slave satellite collaborative mission planning model, obtain the task allocation strategy of the master and slave satellites.

Claims

1. A master-slave satellite task allocation method based on a double-layer backtracking mechanism, characterized in that: The following steps are involved: (1) Obtaining the master and slave satellite mission planning requirements; (2) According to the planning requirements and constraints of the master-slave satellite missions, a master-slave satellite collaborative mission planning model is constructed with the objective function of maximizing the efficiency of the master-slave satellite missions. The efficiency of the master-slave satellite missions is the weighted sum of the cost required for the master-slave satellite to execute the planned mission and the benefits generated by the master-slave satellite to execute the planned mission. (3) Based on the Markov decision process MDP, the task planning transformation is performed to define the state space, action space and reward function; the deep reinforcement learning DQN algorithm is used to solve the master-slave satellite collaborative task planning model, and the task planning is optimized through a double-layer backtracking mechanism. The double-layer backtracking mechanism includes local backtracking and global backtracking. The local backtracking is: for each step of the task planning process, if the backtracking condition is met, the state space is rolled back to the previous state to reselect the action and update the reward function; the global backtracking is: regularly evaluate the global benefits of the task allocation strategy. If the global benefits are significantly lower than the expected threshold, the task planning strategy is re-optimized; (4) Based on the solved master-slave satellite collaborative task planning model, the task allocation strategy of the master and slave satellites is obtained.

2. The master-slave satellite task allocation method according to claim 1, characterized in that: The master-slave satellite collaborative mission planning model is: Where F represents the objective function, f1(s,a) represents the cost required for the master and slave satellites to perform the planned tasks, f2(s,a) represents the urgency benefit of the master and slave satellites, f3(s,a) represents the load balancing benefit of the master and slave satellites, λ1 represents the weight of the sub-goal resource cost, λ2 represents the weight of the sub-goal task benefit, λ3 represents the weight of the sub-goal load balancing, T represents the total duration of the task planning, k represents the time step in the task planning, and e represents the time step of the task planning. j Indicates the end time of the idle time window, o j Indicates the start time of the idle time window, d task Indicates the task working time, o task Indicates the time when the task starts executing, c i represents the remaining storage capacity of satellite i, c task represents the storage required by the task, g i represents the number of executions of satellite i in the planning cycle, g max Indicates the maximum number of times the satellite is turned on and off within a certain period.

3. The master-slave satellite task allocation method according to claim 2, characterized in that: The cost required for the master and slave satellites to perform the planned tasks includes the cost of the slave satellite mission working time and the storage cost required for the slave satellite mission, specifically: Among them, d task (t i ) represents the task working time, c task (t i ) represents the storage capacity required by the task, d task_max Indicates the maximum working time of the task, c task_max represents the maximum storage capacity of the task; α1 represents the weight of the task execution time, and α2 represents the weight of the storage capacity required by the task; The benefits generated by master-slave satellite mission planning include urgency benefits and load balancing benefits; The urgency benefit of the master and slave satellites is calculated based on the mission urgency and the slave satellite decision variables, specifically: Among them, p task (t i ) represents the urgency of the task, p task_max Indicates the maximum urgency of the task, x i represents the decision variable from the star; The load balancing benefit of the master and slave satellites is calculated based on the number of satellite missions and storage capacity, specifically: Among them, g max (w i ) represents the satellite w in the mission planning cycle i The maximum number of task executions, g(w i ) represents satellite w i The number of times the task has been executed from the beginning to now, c(w i ) represents satellite w i Remaining storage capacity, c max (w i ) represents satellite w i Maximum storage capacity.

4. The master-slave satellite task allocation method according to claim 3, characterized in that: The local backtracking condition is to satisfy any of the following conditions: λ1f1(s,a)>C max λ2f2(s,a)+λ3f3(s,a)<R min Among them, C max represents the expected cost threshold, R min represents the expected return threshold; The update formula of the reward function is: R adjusted =R(s t ,a t )+δ·ΔK(s t ,a t ) Among them, δ represents the knowledge base correction coefficient, ΔK(s t ,a t ) represents the incremental change in action return after backtracking, R(s t ,a t ) represents the reward function before updating.

5. The master-slave satellite task allocation method according to claim 4, characterized in that: The DQN algorithm in step (3) is optimized. The DQN algorithm is combined with the brain-like cognitive mechanism to establish a hybrid enhanced knowledge base, and the hybrid enhanced knowledge base is used to solve the master-slave satellite collaborative task planning model. The DQN algorithm combined with the brain-like cognitive mechanism includes: The brain-like cognitive mechanism introduces the cognitive model to dynamically adjust the action selection strategy, and adjusts the action selection probability by combining the Q value of the DQN algorithm with the brain-like weight; In the brain-like framework, the experience samples in the experience replay module of the DQN algorithm are given priority based on the task value and learning importance. The brain-like mechanism analyzes the similarities of new tasks, associates the current allocation strategy with historical experience, and dynamically updates the hybrid enhanced knowledge base.

6. The master-slave satellite task allocation method according to claim 5, characterized in that: The formula for adjusting the action selection probability is: Among them, π(a|s) represents the probability of selecting action a in state s, Q(s,a) represents the action value of action a in state s predicted by the DQN algorithm, and W(s,a) represents the weight assigned by the brain-like cognitive mechanism; W(s,a)=f(φ(s),H) Among them, φ(s) represents the current state feature vector, H represents the historical task data stored in the hybrid enhanced knowledge base; f represents the brain-like weight generation function.

7. The master-slave satellite task allocation method according to claim 6, characterized in that: The priority of the experience samples in the DQN algorithm experience replay module is: Among them, P(i) represents the probability of sample i being selected, β represents the adjustment parameter, and p i represents the priority of sample i; p i Defined as the absolute value of the time difference error TD-error: Among them, γ represents the discount factor, R(s,a) represents the immediate reward in the current state, Represents the maximum action value.

8. The master-slave satellite task allocation method according to claim 7, characterized in that: The update formula of the hybrid enhanced knowledge base is: H new =H old ∪{(s,a,r,s′) current } Among them, H new represents the updated knowledge base content, H old Represents the existing knowledge base content, (s,a,r,s′) current Represents the experience data of the current task.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images