Low earth orbit satellite network task scheduling method based on hybrid reinforcement learning

Abstract

According to the low earth orbit satellite network task scheduling method based on hybrid reinforcement learning, in combination with an earliest deadline priority strategy, deep reinforcement learning and a self-adaptive time slot reservation mechanism, tasks are subjected to priority ranking through a multi-factor dynamic weighted ranking strategy; and constructing a multi-dimensional feature vector containing a node resource state, task characteristics and historical performance, and realizing scheduling decision optimization by adopting an A2C reinforcement learning architecture. According to the method, the system can intelligently adjust the scheduling gravity center according to the real-time load in a dynamically changing satellite network environment, the task success rate is effectively improved, particularly, high-priority tasks are more likely to be guaranteed when the load is high, the average task delay is reduced, and the satellite energy consumption is optimized.

Description

Technical Field

[0001] This invention relates to the fields of satellite communication technology, distributed task scheduling and reinforcement learning technology, and in particular to an efficient real-time task scheduling and resource optimization allocation method for LEO satellite networks. Background Technology

[0002] As low-Earth orbit (LEO) satellite networks rapidly expand towards large-scale, high-density deployment, the task scheduling problem between satellites and ground stations is becoming increasingly complex and challenging. As key nodes in the space network, on-orbit satellites have relatively limited computing resources and rely primarily on solar power, resulting in highly constrained energy reserves. This necessitates balancing energy efficiency optimization with the rational allocation of computing load during mission execution. Simultaneously, the communication links established between satellites and ground stations exhibit significant dynamic and time-varying characteristics: influenced by factors such as high-speed satellite motion, weather conditions, and obstruction effects, link bandwidth and transmission latency constantly change, further increasing the uncertainty of the scheduling process. Furthermore, the increasing diversity of user task types, coupled with varying priorities among different tasks, necessitates scheduling strategies capable of balancing multiple objectives.

[0003] Faced with complex and dynamic environments, traditional scheduling methods are gradually revealing their limitations. Static scheduling methods rely on pre-set fixed rules to allocate computing resources, lacking the ability to respond to real-time changes in network status, and are prone to uneven resource utilization or task backlog. Greedy scheduling methods can make quick decisions based on the current system state, but because they only consider the current state for local optimization, they are difficult to maximize the efficiency of global resource allocation and lack global planning capabilities. Although simple reinforcement learning methods have the potential to learn dynamic strategies through interaction, they often suffer from slow convergence and poor initial performance in actual deployment. Summary of the Invention

[0004] The technical problem solved by this invention is to overcome the shortcomings of the prior art and propose a low-orbit satellite network task scheduling method based on hybrid reinforcement learning, which effectively improves the task success rate, especially under high load, it tends to guarantee high-priority tasks, reduces average task latency and optimizes satellite energy consumption.

[0005] The technical solution of this invention is: a low-Earth orbit satellite network task scheduling method based on hybrid reinforcement learning, comprising: 1) Define the attributes of satellite and ground station nodes, and initialize the communication link parameters and reinforcement learning parameters; 2) Generate scheduling tasks based on the Poisson process, which include attributes such as computational requirements, data volume, arrival time, deadline, and priority. Then, normalize the attributes of each dimension of the scheduling tasks. 3) A multi-factor dynamic weighting strategy is adopted to prioritize the tasks to be scheduled in the waiting queue. The weighting weight is determined by dynamically adjusting the proportion of each factor in real time according to the load of the task scheduling system, so as to ensure that high-priority tasks are prioritized under high load. 4) For each candidate node, construct a multi-dimensional feature vector containing node resource status, task characteristics, and historical performance for the task to be scheduled. Calculate the comprehensive score for each candidate node and sort them. 5) Based on the candidate node scoring ranking results, check the resource availability and energy sufficiency of each candidate node in consecutive time slots from high to low, make time slot reservations and update the node status. 6) After the time slot reservation is completed, the scheduling system issues a task execution instruction to the target candidate node and transmits task data and time slot allocation information synchronously. The node starts task calculation within the reserved time slot, enters the task information into the node's running task list, and deducts the node energy required for task execution and updates the node's remaining energy. 7) After the scheduled task is completed, calculate the reward value based on the execution result, and update the value function and advantage function parameters using the time-series difference algorithm; 8) After the parameters are updated, the scheduling system returns the monitoring status of the task waiting queue. If there is a new task, the scheduling process of steps 3) to 7) is repeated. If there is no new task to be scheduled, the system enters the low power monitoring mode and scans the task queue every preset time until a new task is connected.

[0006] In step 3), the weighting weights w i The calculation formula is: w i =(0.5 0.2× L )× D +(0.2 0.1× L )× S +(0.3+0.3× L )× P Where L is the system load factor, with a value range of [0, 1]; D is the deadline pressure factor; S is the task size factor; and P is the priority factor.

[0007] The multidimensional feature vector φ It includes 10 dimensions, namely: =[ R , 1 / ( 1 + Tt ), Slack , En ,Isat , Ts , Hs , Fa , P , Qn ] The dimensions represent, in order: R The percentage of available resources, 1 / ( 1 + Tt () represents the reciprocal of the transmission time. Tt For task data transmission time, Slack For time relaxation, En For normalized energy values, Isat For node type identification, Ts To normalize the task size, Hs Based on historical success rate, Fa For future resource availability, P As a task priority, Qn This is the normalized queue length.

[0008] The calculation of the comprehensive score for each candidate node employs an A2C reinforcement learning architecture, combining the basic heuristic score with the learned score to arrive at the node's final comprehensive score. Score for: Score = Score base + ×( v + a ) in Score base Based on the score, To enhance the learning weight coefficients, v and a These are the parameter vectors for the value function and the advantage function, respectively. The value function... V = v .

[0009] When calculating the reward value based on the execution result, factors such as task success status, delay, energy consumption, deadline fulfillment, task priority, and queue length are taken into account.

[0010] The reward value Reward Specifically: Reward = Sc × × P 1.8×(1 Sc )× P 0.012× Ecost × Isat 2.5× Sc ×(1 Dmet )× P 0.05× Qlen; in Sc Indicates whether the task was successful. Indicates the total delay in task execution. Ecost Indicates the energy consumption for task execution. Dmet Indicates whether the deadline has been met. Qlen This indicates the current length of the task queue.

[0011] The step of updating the value function and advantage function parameters using a time-difference algorithm includes: = Reward + × Vnext Vcurrent ’ v ← v + × ×

[0012] ’ a ← a + × ×

[0013] in For timing difference error, As a discount factor, The learning rate; ’ v For the updated value function parameters, ’ a For the updated advantage function parameters, Vcurrent The value function representing the current scheduling state. Vnext The value function represents the value of the next scheduling state.

[0014] The step of checking the resource availability and energy sufficiency of each candidate node in multiple consecutive time slots from high to low according to the candidate node score ranking results, reserving time slots and updating node status includes: Starting with a high-scoring candidate node, check whether the node's consecutive time slots within the task arrival time to deadline meet the task's computation and communication resource requirements, and verify whether the node's remaining energy is greater than the task's execution energy consumption. If the constraints are met, bind the time slots of that period to the task and complete resource reservation, and update the node's remaining resources and energy level in real time. If not, check the next scoring candidate node in sequence until a node that meets the conditions is found or all candidate nodes are traversed.

[0015] The advantages of this invention compared to the prior art are: (1) By integrating the immediate decision-making ability of heuristic strategies with the long-term optimization ability of reinforcement learning, the basic performance of the system in the early stage of training is guaranteed, and the continuous performance improvement is achieved with the accumulation of experience. This reflects the advantages of the hybrid architecture and overcomes the problems of poor performance in the early stage of pure reinforcement learning and the limited vision of pure heuristic methods. (2) Adaptive task scheduling optimization is performed by using the introduced multi-factor dynamic weighted sorting strategy, which enables the system to intelligently adjust the scheduling focus according to the real-time load, and tends to guarantee high-priority tasks under high load, thereby improving the robustness of the system. (3) The constructed multidimensional feature vector integrates information on node static resources, dynamic load, historical performance and future availability, providing a comprehensive and accurate environmental state perception for reinforcement learning intelligent models, thereby making better scheduling decisions; (4) The designed reward function simultaneously weighs multiple key indicators such as task success rate, latency, energy consumption, and deadline fulfillment rate, guiding the intelligent model to learn a balanced and efficient scheduling strategy to achieve balanced multi-objective optimization. Attached Figure Description

[0016] Figure 1 This is a flowchart of the task scheduling method of the present invention.

[0017] Figure 2 The curves show the cumulative success rate of the two methods over time.

[0018] Figure 3 The curves show the average task latency comparison between the two methods. Detailed Implementation

[0019] This invention proposes a task scheduling method for low-Earth orbit satellite networks based on hybrid reinforcement learning. By combining the immediate decision-making capability of heuristic strategies with the long-term optimization capability of reinforcement learning, a multi-factor dynamic weighted ranking strategy is introduced. This enables the system to intelligently adjust the scheduling focus according to the real-time load, giving preference to high-priority tasks under high load, reducing average task latency and optimizing satellite energy consumption.

[0020] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0021] like Figure 1 As shown, this invention provides a low-Earth orbit satellite network task scheduling method based on hybrid reinforcement learning. The steps are as follows: (1) Define the attributes of satellite and ground station nodes, including computing power, initial energy level, historical mission success rate, etc.; initialize communication link parameters, including bandwidth, transmission delay, etc.; set reinforcement learning parameters, including learning rate, exploration rate, initial parameters of value function and advantage function, etc. (2) Generate a task to be scheduled based on the Poisson process, which includes attributes such as computational requirements, data volume, arrival time, deadline, and priority, and normalize the attributes of each dimension of the task to be scheduled. (3) A multi-factor dynamic weighting strategy is adopted to prioritize the tasks to be scheduled in the waiting queue. The weighting is determined by dynamically adjusting the proportion of each factor in real time according to the load of the task scheduling system, so as to ensure that high-priority tasks are given priority under high load. The dynamic weighting calculation formula is as follows: w i =(0.5 0.2× L )× D +(0.2 0.1× L )× S +(0.3+0.3× L )× P in, L This is the system load factor, with a value range of [0, 1]. D As the deadline pressure factor, S As a task size factor, P Priority factor; (4) For the task to be scheduled, construct a multi-dimensional feature vector for each candidate node, which includes the node resource status, task characteristics and historical performance. Specifically, it includes 10 dimensions: =[ R , 1 / ( 1 + Tt ), Slack ,En , Isat , Ts , Hs , Fa , P , Qn ] The dimensions represent, in order: R For resource availability ratio, 1 / ( (1 + Tt) The reciprocal of the transmission time ( Tt (for task data transmission time) Slack For time relaxation, En For normalized energy value, Isat Used as a node type identifier (1 for satellite, 0 for ground station). Ts To normalize task size, Hs For historical success rate, Fa For future resource availability, P For task priority, Qn This is the normalized queue length.

[0022] The A2C reinforcement learning architecture is used to calculate the final comprehensive score of candidate nodes and rank them. Score = Score base + ×( v + a ) in Score base Based on the score, To enhance the learning weight coefficients, v and a These are the parameter vectors for the value function and the advantage function, respectively. The value function... V = v ; (5) Based on the candidate node scoring ranking results, check the resource availability and energy sufficiency of each candidate node in consecutive time slots from high to low, make time slot reservations and update the node status, specifically as follows: Starting with a high-scoring candidate node, check whether the node's consecutive time slots within the task arrival time to deadline meet the task's computation and communication resource requirements, and verify whether the node's remaining energy is greater than the task's execution energy consumption. For high-priority tasks, the energy verification threshold can be relaxed to 50% of the normal threshold. If the constraints are met, bind the time slots of that period to the task and complete resource reservation, updating the node's remaining resources and energy level in real time. If not, check the next candidate node in the scoring process until a node that meets the conditions is found or all candidate nodes are traversed.

[0023] (6) After the time slot reservation is completed, the scheduling system issues a task execution instruction to the target candidate node and transmits task data and time slot allocation information synchronously. The node starts task calculation within the reserved time slot, enters the task information into the node's running task list, and deducts the node energy required for task execution and updates the node's remaining energy. (7) After the scheduled task is completed, the reward value is calculated based on the execution result. The reward function comprehensively considers the task success status, delay, energy consumption, deadline fulfillment, task priority, and queue length: Reward = Sc × × P 1.8×(1 Sc )× P 0.012× Ecost × Isat

[0024] 2.5× Sc ×(1 Dmet )× P 0.05× Qlen in Sc Indicates whether the task was successful. Indicates the total delay in task execution. Ecost Indicates the energy consumption for task execution. Dmet Indicates whether the deadline has been met. Qlen This indicates the current length of the task queue.

[0025] Subsequently, the parameters of the value function and advantage function of reinforcement learning are updated using a temporal difference algorithm: = Reward + × Vnext Vcurrent ’ v ← v + × ×

[0026] ’ a ← a + × ×

[0027] in For timing difference error, As a discount factor, The learning rate; ’ v For the updated value function parameters, ’ a For the updated advantage function parameters, Vcurrent The value function representing the current scheduling state. Vnext The value function represents the next scheduling state. The key parameter settings for the simulation experiment are as follows: Network scale: Simulation duration 2000 time slots; 20 satellite nodes and 4 ground station nodes.

[0028] Node resources: The satellite's initial energy is 1.5 × 10⁻⁶. 5 The unit is 1.0 × 10⁻⁶ for ground stations. 7 unit.

[0029] Task flow to be scheduled: Task arrival parameters The distribution follows a Poisson distribution with a value of 3.5; the computational requirements are within the range of 4 × 10⁻⁶. 8 Up to 9×10 9 Instructions; deadline range is 1 to 5 time slots; high priority tasks account for 20%.

[0030] Learning parameter: learning rate =0.015, initial exploration rate =0.3 (based on exponential decay), discount factor =0.9.

[0031] Simulation Result Analysis: Table 1 shows a performance comparison between the hybrid and static methods of this invention: Table 1 Performance Comparison of Two Methods

[0032] Simulation results show that, compared with the Static scheduling method, this invention improves the overall task success rate from 85.7% to 94.7% (a relative improvement of about 10.5%), improves the success rate of high-priority tasks from 86.5% to 95.0% (a relative improvement of about 9.8%), reduces the average task latency from 1.446 time slots to 1.237 time slots (a reduction of about 14.5%), and significantly optimizes the energy consumption of satellite nodes.

[0033] The cumulative success rates of the two methods over time are compared as follows: Figure 2 As shown in the figure, the average task latency comparison curve is as follows: Figure 3 As shown.

[0034] The results show that the present invention has better overall performance, especially in terms of task success rate and latency, which is superior to the static method, demonstrating its global optimization capability.

[0035] This invention aims to combine the immediate decision-making capabilities of heuristic strategies with the long-term optimization capabilities of reinforcement learning, integrating adaptive time slot reservation with a multi-objective reward function to reduce latency and energy consumption while ensuring the success rate of high-priority tasks. It has practical value in low-Earth orbit internet satellite systems and shows promise for applications in other communication systems such as 5G and 6G.

[0036] The contents not described in detail in this specification are common knowledge to those skilled in the art.

Claims

1. A low-Earth orbit satellite network task scheduling method based on hybrid reinforcement learning, characterized in that, include: 1) Define the attributes of satellite and ground station nodes, and initialize the communication link parameters and reinforcement learning parameters; 2) Generate scheduling tasks based on the Poisson process, which include attributes such as computational requirements, data volume, arrival time, deadline, and priority. Then, normalize the attributes of each dimension of the scheduling tasks. 3) A multi-factor dynamic weighting strategy is adopted to prioritize the tasks to be scheduled in the waiting queue. The weighting weight is determined by dynamically adjusting the proportion of each factor in real time according to the load of the task scheduling system, so as to ensure that high-priority tasks are prioritized under high load. 4) For each candidate node, construct a multi-dimensional feature vector containing node resource status, task characteristics, and historical performance for the task to be scheduled. Calculate the comprehensive score for each candidate node and sort them. 5) Based on the candidate node scoring ranking results, check the resource availability and energy sufficiency of each candidate node in consecutive time slots from high to low, make time slot reservations and update the node status; 6) After the time slot reservation is completed, the scheduling system issues task execution instructions to the target candidate nodes and transmits task data and time slot allocation information synchronously. The node starts task calculation within the scheduled time slot, enters the task information into the node's running task list, deducts the node energy required for task execution, and updates the node's remaining energy. 7) After the scheduled task is completed, calculate the reward value based on the execution result, and update the parameters of the value function and advantage function using the time-series difference algorithm; 8) After the parameters are updated, the scheduling system returns the monitoring status of the task waiting queue. If there is a new task, the scheduling process of steps 3) to 7) is repeated. If there is no new task to be scheduled, the system enters the low power monitoring mode and scans the task queue every preset time until a new task is connected.

2. The method according to claim 1, characterized in that, The weighting weight w i The calculation formula is: w i =(0.5 0.2× L )× D +(0.2 0.1× L )× S +(0.3+0.3× L )× P Where L is the system load factor, with a value range of [0, 1]; D is the deadline pressure factor; S is the task size factor; and P is the priority factor.

3. The method according to claim 2, characterized in that, The multidimensional feature vector φ It includes 10 dimensions, namely: =[ R , 1 / ( 1+Tt ), Slack , En , Isat , Ts , Hs , Fa , P , Qn ] The dimensions represent, in order: R The percentage of available resources, 1 / ( 1+Tt () represents the reciprocal of the transmission time. Tt For task data transmission time, Slack For time relaxation, En For normalized energy values, Isat For node type identification, Ts To normalize the task size, Hs Based on historical success rate, Fa For future resource availability, P As a task priority, Qn This represents the normalized queue length.

4. The method according to claim 3, characterized in that, The calculation of the comprehensive score for each candidate node employs an A2C reinforcement learning architecture, combining the basic heuristic score with the learned score to arrive at the node's final comprehensive score. Score for: Score = Score base + ×( v + a ) in Score base Based on the score, To enhance the learning weight coefficients, v and a These are the parameter vectors for the value function and the advantage function, respectively. The value function... V = v .

5. The method according to claim 3, characterized in that, When calculating the reward value based on the execution result, factors such as task success status, delay, energy consumption, deadline fulfillment, task priority, and queue length are taken into account.

6. The method according to claim 5, characterized in that, The reward value Reward Specifically: Reward = Sc × × P 1.8×(1 Sc )× P 0.012× Ecost × Isat 2.5× Sc ×(1 Dmet )× P 0.05× Qlen; in Sc Indicates whether the task was successful. Indicates the total delay in task execution. Ecost Indicates the energy consumption for task execution. Dmet Indicates whether the deadline has been met. Qlen This indicates the current length of the task queue.

7. The method according to claim 1, characterized in that, The step of updating the value function and advantage function parameters using a time-difference algorithm includes: = Reward + × Vnext Vcurrent ’ v ← v + × × ’ a ← a + × × in For timing difference error, As a discount factor, The learning rate; ’ v For the updated value function parameters, ’ a For the updated advantage function parameters, Vcurrent The value function representing the current scheduling state. Vnext The value function represents the value of the next scheduling state.

8. The method according to claim 1, characterized in that, Step 5) Based on the candidate node scoring ranking results, the resource availability and energy sufficiency of each candidate node in consecutive time slots are checked sequentially from high to low, time slot reservations are performed, and node status is updated, including: Starting with a high-scoring candidate node, check whether the node's consecutive time slots within the task arrival time to deadline meet the task's computation and communication resource requirements, and verify whether the node's remaining energy is greater than the task's execution energy consumption. If the constraints are met, bind the time slots of that period to the task and complete resource reservation, and update the node's remaining resources and energy level in real time. If not, check the next scoring candidate node in sequence until a node that meets the conditions is found or all candidate nodes are traversed.

Images