Resource scheduling optimization methods for mega-constellations

Abstract

This paper presents a resource scheduling optimization method for mega-constellations, focusing on satellite resource allocation in this field. It addresses the challenge of existing satellite resource scheduling methods in achieving global optimization while balancing time and energy consumption, aiming to improve the execution efficiency and reduce energy consumption of multi-satellite collaborative tasks. By simulating and processing real satellite scheduling tasks, satellite and task data are obtained. A reinforcement learning-guided genetic algorithm is used to optimize task sequences. These optimized sequences are then used for satellite resource allocation, and the time and energy consumption of each task sequence are calculated and normalized to obtain a comprehensive index. This method yields the task execution sequence with the minimum combined time and energy consumption index, thus providing the optimal scheduling scheme to meet the satellite scheduling requirements of mega-constellations.

Description

Technical Field

[0001] This invention relates to the field of satellite resource scheduling and allocation and computer science algorithm technology, specifically to a resource scheduling optimization method for mega-constellations. Background Technology

[0002] Unlike traditional small-scale satellite formations, mega-constellations are characterized by a large number of satellites and complex inter-satellite links, resulting in an exponentially increasing solution space for the task scheduling problem. Traditional greedy algorithms and heuristic rules, lacking global optimization capabilities, are prone to getting trapped in local optima when handling large-scale collaborative scheduling. This leads to lengthy task completion times and excessive energy consumption, severely restricting the operational efficiency and economic benefits of mega-constellations. Therefore, there is an urgent need for an intelligent scheduling method that can adapt to the characteristics of mega-constellations, dynamically adjusting optimization strategies according to different scheduling scenarios, improving the execution efficiency of multi-satellite collaborative tasks in mega-constellations while reducing energy consumption, while ensuring solution quality. Summary of the Invention

[0003] To address the challenge of achieving global optimization of both time and energy consumption in traditional satellite resource scheduling methods, this invention provides a resource scheduling optimization method for mega-constellations.

[0004] A resource scheduling optimization method for giant constellations, which is implemented by the following steps:

[0005] Step 1: Obtain task data based on the image transmission task, process the task data to obtain an initial population N, and calculate the fitness of the initial population N.

[0006] Step 2: Initialize the Q-table, including the crossover probability table. And the table of mutation probability Define the action interval set A and the state interval set S; based on the current population state value Find the corresponding reinforcement learning state interval, where the index value of the interval represents the current state of the population mapping; and calculate the crossover probability. and mutation probability ;

[0007] Step 3: Generate a new population and update fitness; allocate resources to each individual in the new population, record the time spent by each individual, update the time set, i.e. update fitness; then execute steps 4 and 6 in parallel;

[0008] Step 4: Calculate the cross-reward value and mutation reward value And calculate the timing difference error TD value td, and determine the update strategy;

[0009] Step 5: If the current stage is the SARSA policy phase of reinforcement learning and td is less than the adaptive threshold... If the current strategy is not met, switch to the Q-Learning strategy, update the Q table, and proceed to step nine; otherwise, maintain the current strategy stage and proceed to step nine.

[0010] Step 6: Select satellites based on the cluster and calculate the time and energy consumption in the link;

[0011] Step 7: Determine if this is the last task for the current individual. If yes, proceed to step 8; otherwise, proceed to step 6.

[0012] Step 8: Determine if this is the last individual in the current population. If so, calculate the comprehensive index and proceed to Step 9. Otherwise, initialize the satellite timestamp, select an individual from the population, and return to Step 6.

[0013] Step 9: When gen equals max gen, end the iteration and obtain the task sequence of the optimal satellite resource scheduling scheme; otherwise, increment gen by 1 and return to step 3.

[0014] The beneficial effects of this invention are:

[0015] This invention decouples the satellite resource scheduling problem into two sub-problems: simulated resource allocation and satellite scheduling. This effectively reduces the combinatorial complexity and search space dimension of the original problem. Simulated resource allocation provides a globally optimized task sequence, offering a high-quality initial solution for subsequent scheduling. Step-by-step solution reduces computational redundancy and improves the algorithm's convergence speed.

[0016] This invention is applicable to the real-time satellite scheduling needs of mega-constellations, maintaining stable computing performance even as the mission scale expands, and has significant practical value and scalability. Attached Figure Description

[0017] Figure 1 This is a process diagram of the resource scheduling optimization method for giant constellations described in this invention;

[0018] Figure 2 This is a flowchart of the resource scheduling optimization method for giant constellations described in this invention;

[0019] Figure 3 A schematic diagram of satellite resource scheduling for a mission sequence;

[0020] Figure 4 A schematic diagram of satellite resource scheduling for another mission sequence;

[0021] Figure 5 This is a schematic diagram of satellite resource scheduling for the third mission sequence. Detailed Implementation

[0022] Combination Figures 1 to 5 This embodiment describes a resource scheduling optimization method for mega-constellations. This method combines reinforcement learning and a genetic algorithm. Reinforcement learning selects states based on the fitness of the population, then selects actions based on the states and a Q-table. Finally, the genetic algorithm calculates the crossover and mutation probabilities based on the actions. The genetic algorithm continuously optimizes all individuals in the population through the crossover and mutation probabilities, i.e., updating the population, and provides updated fitness to reinforcement learning by calculating the time of resource allocation. Reinforcement learning calculates the reward values ​​for crossover and mutation based on the updated fitness. Then, it calculates the error based on the crossover and mutation reward values ​​and selects a strategy to update the Q-table, finally updating the Q-table. Specifically, when selecting actions based on the states and the Q-table, reinforcement learning uses an ε-softmax strategy to select actions based on the Q-value of the action corresponding to that state recorded in the Q-table. Each time an action is selected based on the state, the Q-table must be updated with the Q-value of the action corresponding to the selected state (because the population of the genetic algorithm is updated with each iteration, and the population fitness changes accordingly. Reinforcement learning judges the state and calculates the reward value through fitness. Therefore, when the same state and the same action are selected in different generations of the iteration, the corresponding Q-values ​​are different. So the Q-value of the action corresponding to the state in the Q-table must be updated in each iteration).

[0023] In this implementation, the Q-table is updated using a combination of SARSA and Q-Learning strategies. First, SARSA is used to update the Q-table, and then Q-Learning is used to update the Q-table based on a threshold. In each iteration, the genetic algorithm adjusts each individual in the entire population based on the crossover and mutation probabilities provided by reinforcement learning, aiming to optimize all individuals in the population. The optimized population is then the new population, and the reinforcement learning and genetic algorithm operations are repeated on this new population to update the population again.

[0024] This implementation iteratively updates the population through reinforcement learning and genetic algorithms to obtain the optimal individual. In reinforcement learning, the Q-table represents states in rows and actions in columns; in the genetic algorithm, individuals are task sequences, which are the order in which all tasks are arranged, with each individual having a different task order. The set of all individuals constitutes the population, and the population fitness is the time set for resource allocation across all task sequences.

[0025] like Figure 1As shown, satellites are first virtualized into a resource pool. Then, an individual is selected from the population, and resources are allocated in parallel according to the order of tasks S1, S2, S3, S4, and S5 on that individual. After the tasks are completed, the resources are returned to the resource pool, and the time spent on resource allocation for each individual is recorded. This process is repeated for all individuals in the current population. Each individual is then optimized based on its recorded time. The optimized individuals are then further allocated resources to obtain their optimized times. This process of repeatedly allocating resources and recording individual times continuously optimizes the task execution order of individuals. Simultaneously, satellite resource scheduling is performed on all optimized individuals each time, obtaining the time and energy consumption for individual satellite resource scheduling. Based on the time and energy consumption of satellite resource scheduling, the task sequence with the optimal satellite resource scheduling scheme is selected.

[0026] like Figure 2 As shown, the resource scheduling optimization method for giant constellations described in this embodiment specifically includes the following steps:

[0027] Step 1: Use the AGI System Tool Kit (STK) simulation software to build the Starlink constellation model.

[0028] Step 2: Utilize STK and Matlab for interoperability to calculate the distance and visibility relationships between Starlink constellation satellites, obtaining the distance matrix and visibility matrix; simultaneously, compile task data based on similar image transmission tasks from small satellite companies and communication tasks from some constellations.

[0029] Step 3: Generate a task execution sequence according to the CMO priority rule (number of satellite resources multiplied by execution time) in descending order of weight; then randomly swap the task positions of the base sequence to obtain a task sequence that is 3 times the initial population size N; then randomly select N from these task sequences to obtain N individuals, and obtain the initial population N of the genetic algorithm.

[0030] Step 4: Simulate the satellite as a resource and establish a dynamic resource pool. Allocate resources to all individuals in the population in parallel according to the individual. After the mission is completed, release the resources and return them to the resource pool. Record the set of time spent by each individual in execution. The set of time is the initial fitness of the initial population.

[0031] Step 5: Initialize the Q table and define actions and states;

[0032] Initialize two Q-tables for reinforcement learning, one for cross-probability tables. And the table of mutation probability The initial values ​​are all 1, and the table is... and table Used to calculate crossover probabilities and mutation probability Define an action set A and a state set S, where both actions and states are discrete intervals; set the initial value of the iteration number gen to 1.

[0033] Step 6: Map population states;

[0034] Calculate the average fitness, population diversity, and optimal fitness of the population based on the current population fitness. Normalize these three scalars to obtain the population state values. ,according to The value is used to find the corresponding reinforcement learning state interval, and the index value of this state interval is the selected state.

[0035] Step 7: Calculate the crossover probability and mutation probability ;exist In the corresponding state, reinforcement learning selects the crossover action interval according to the ε-softmax policy. In the action range Crossover probability is obtained by uniform sampling within the inner area. ;exist In the corresponding state, reinforcement learning selects the mutation action range according to the ε-softmax policy. In the action range Uniform sampling within the inner region yields the probability of variation. When random numbers Less than the greed rate At that time, randomly select an action interval from the action set A; when the random number Greed rate greater than or equal to At that time, the softmax strategy is used based on the crossover probability table. And the table of mutation probability Select the action range.

[0036] ;

[0037] ;

[0038] ;

[0039] ;

[0040] in, From 0 to 1, The initial temperature. The attenuation coefficient is... Cross probability table And the table of mutation probability In state The set of actions to be recorded below Cross probability table And the table of mutation probability The number of columns, For state Next action set corresponding A set of values For state Next action set corresponding A set of values In the state The following corresponding action set The probability distribution, From In a probability distribution, the index value is obtained by randomly sampling the index value corresponding to each probability according to the probability value of each probability itself. This index value is the selected action, and the action interval corresponding to this action is the action interval to be selected.

[0041] Step 8: Generate a new population and update fitness, while executing steps 9 and 11 in parallel;

[0042] A new population is generated using a three-player tournament based on the current fitness, following the crossover probabilities provided by reinforcement learning. Decide whether to perform crossover; if so, randomly select individuals in the population for pairwise crossover using either partial matching crossover or position-based crossover; and proceed according to the mutation probabilities provided by reinforcement learning. Use crossover mutation on each individual in the population; based on the time of resource allocation, retain a number of elite individuals to replace the worst individuals to obtain a new population, then allocate resources to each individual in the new population, record the time spent by each individual, and update the fitness;

[0043] Step 9: Calculate the cross-reward value for reinforcement learning based on the updated population fitness. and mutation reward value ;

[0044] Step 10: Based on the cross-reward value obtained in Step 9 and mutation reward value Calculate the time difference error (TD) and determine the update strategy;

[0045] First, calculate the action corresponding to the current state in the reinforcement learning cross probability table. And the table of mutation probability TD error of the value and Then calculate the total TD error. The TD error is smoothed using the exponential moving average (EMA) to obtain the smoothed error value td. The initial value of td is 0. A window is maintained to record the td values ​​of the most recent several generations to calculate the mean. and variance Reconstruct an adaptive threshold If the current stage is the SARSA policy phase of reinforcement learning and td is less than 1 / 2, then the current stage is SARSA policy phase of reinforcement learning. If the current policy is not met, switch the Q-Learning strategy and proceed to step 17; otherwise, maintain the current policy and proceed to step 17.

[0046] ;

[0047] ;

[0048] ;

[0049] ;

[0050] ;

[0051] in, As a discount factor, and Adaptive threshold coefficient For smoothing coefficients, In the state Next, select an action. hour The value, In state Next, select an action. hour The value of . For the next selection state, In the state Next, select an action. hour The expected value, In state Next, select an action. hour Expected value

[0052] In this implementation, if the current stage is SARSA, then the cross-probability table Q is updated using SARSA. pc And the mutation probability table Q pm Otherwise, update the cross-probability table Q using Q-Learning. pc And the mutation probability table Q pm ;

[0053] ;

[0054] ;

[0055] ;

[0056] ;

[0057] in, In the state Next, select an action. hour The highest expected value; In the state Next, select an action. hour The highest expected value; For learning rate, Update for improvement learning rate, Update for improvement learning rate, A learning coefficient greater than 1 The learning coefficient is less than 1.

[0058] Step 11: Initialize satellite timestamps;

[0059] Use a timestamp to mark the earliest available time of each satellite, initially set to 0; select an individual from the current population;

[0060] Step 12: Select satellites based on clusters;

[0061] The available satellites are divided into several clusters using agglomerative hierarchical clustering; starting from the current mission position of the individual, satellites are assigned to the mission in parallel; it is determined whether a suitable satellite has been selected. If so, step 13 is executed; otherwise, it waits for the next satellite release time, updates the timestamp, and continues to execute step 12.

[0062] In this implementation, the condition for determining the appropriate satellite first is: if there exists a cluster whose required resources num are greater than or equal to the number of cluster members, then the distance-greedy algorithm is used to select num satellites that are closest in distance within that cluster to make the appropriate satellite.

[0063] Step 13: Determine the visibility relationships of satellites based on the satellite visibility matrix. Use BFS (Breadth-First Search) to check if the selected satellites can form a connected graph. If so, proceed to step 14; otherwise, return to step 12.

[0064] Step 14: Calculate energy consumption;

[0065] The maximum spanning tree (MST) of the connected graph is calculated based on the satellite distance matrix. The longest edge in the spanning tree is selected as the critical communication link to calculate the propagation time and update the timestamp. The edges of the connected graph are considered as communication links to calculate the total edge distance of the connected graph. Calculate the propagation time in the link Then calculate the energy consumption in the link. and the energy consumption of the current task Total energy consumption for updating the task sequence , The initial value is 0. If the current task is the last task of the current individual, proceed to step 15; otherwise, return to step 12. The energy consumption in the link... and the energy consumption of the current task and total energy consumption This can be expressed as follows:

[0066] ;

[0067] in, and These represent the satellite's transmit power and receive power during link propagation, respectively. Energy consumption for satellites to perform missions.

[0068] Step 15: All individuals have been assessed;

[0069] Once all tasks are completed, update the timestamp and availability status of each satellite. The individual's completion time makespan is the maximum value among all satellite timestamps and is recorded in the current generation's completion time set. All individuals in the population have been fully evaluated; proceed to step 16. Otherwise, proceed to step 11.

[0070] Step 16: Normalize the completion time and energy consumption of all individuals in this generation to a weighted range of 0 to 1, and sum them to obtain a comprehensive index. After each iteration, update the comprehensive index and the corresponding task sequence to a better one; if there is no better one, do not update. As shown in the following formula:

[0071] ;

[0072] in, This represents the maximum time among all individual satellite resource scheduling completion times in the first generation. This represents the maximum energy consumption for resource scheduling among all individual satellites in the first generation. This is a weighted number.

[0073] Step 17: When the number of iterations gen equals the maximum number of iterations max gen, the loop ends, and the task sequence corresponding to the last updated comprehensive index is the optimal satellite resource scheduling scheme; otherwise, increment the number of iterations gen by 1 and return to step 6.

[0074] Combination Figures 3 to 5 This implementation method is described below. Figures 3-5All are satellite time Gantt charts. In this implementation, satellite resource scheduling is performed by marking satellites with timestamps. Figure 3 , Figure 4 and Figure 5 It is the satellite resource scheduling process for three task sequences (individuals) within a population. During satellite resource scheduling, Figure 3 The sequence of task execution is as follows , , , , , , , Because the task sequence is executed in parallel, tasks can be executed in parallel when satellite resources are sufficient; however, when satellite resources are insufficient, the current task and subsequent tasks in the sequence must wait and cannot be interrupted. Therefore... Figure 3 In the task sequence , , , The mission can be executed in parallel with sufficient satellite resources. Insufficient satellite resources were available, therefore subsequent missions in the mission sequence... , , , Entering the waiting state. End the release of satellite resources Sufficient resources are available for execution; Release resources implement; Release resources, , It can simultaneously satisfy the requirement of parallel execution. Therefore Figure 3 The completion time of the mission sequence is the timestamp recorded by the satellite Sat5.

[0075] Figure 4 Sat10 and Sat11 initially satisfy However, since tasks need to be executed sequentially, exist Front, Unable to obtain resources to execute, therefore We can only wait. It is the last task to be executed, but the completion time of the task sequence can only be the timestamp recorded by Sat0. Although Sat11 participated in the execution... However, Sat11 timestamps are short.

[0076] Figure 5 Although It was the last task to be executed, but The execution time is longer, so Figure 5 The completion time of the mission sequence is the timestamp recorded by the Sat7 satellite. According to... Figure 4 and Figure 5 The task sequence completion time is not determined by selecting the timestamp of the last satellite that finished its work, nor by selecting the timestamp of the satellite that executed the last task, but by selecting the maximum value of the timestamps recorded across all satellites. Figure 3 , Figure 4 and Figure 5 In practice, different task sequences result in different completion times and energy consumption. The essence of this invention is to find the optimal task sequence in terms of both time and energy consumption based on these different task sequences.

[0077] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0078] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method for resource scheduling optimization for mega-constellation, characterized in that: The method is realized by the following steps: Step one, according to the task data obtained from the image transmission task, the task data is processed, the initial population N is obtained, and the fitness of the initial population N is calculated; Step two, initialize Q table, including cross probability table and mutation probability table , define action interval set A and state interval set S; according to the current population state value find the corresponding reinforcement learning state interval, the index value of the state interval is the current population mapping state; and calculate the cross probability and mutation probability ; Step three, generate new population and update fitness; allocate resources to each individual of the new population, record the time spent by each individual, update the time set, that is, update the fitness; then execute step four and step six in parallel; Step four, calculate the cross reward value and the mutation reward value and calculate the time difference error TD error value td, and judge the update strategy; Step 5: If the current stage is the SARSA policy phase of reinforcement learning and td is less than the adaptive threshold... If the current strategy is not met, switch to the Q-Learning strategy, update the Q table, and proceed to step nine; otherwise, maintain the current strategy stage and proceed to step nine. Step six, select a satellite according to the cluster, and calculate the time and energy consumption in the link; Step seven, judge whether it is the last task of the current individual, if yes, execute step eight, otherwise execute step six; Step eight, judge whether it is the last individual of the current population, if yes, calculate the comprehensive index; execute step nine; Otherwise, initialize the satellite timestamp, select an individual from the population, and return to execute step six; Step nine, when gen is equal to max gen, end the iteration, and obtain the task sequence of the optimal satellite resource scheduling scheme; otherwise, let gen+1, and return to execute step three. 2.The method for resource scheduling optimization for mega constellation according to claim 1, wherein: In step one, the process of obtaining the initial population N is as follows: a task execution sequence is generated according to the weight from large to small for the task data in step one, the task positions in the task execution sequence are randomly exchanged, and a task sequence of 3 times the initial population number N is obtained; then N individuals are randomly extracted from the task sequence to obtain the initial population N. 3.The method for resource scheduling optimization for mega constellation according to claim 1, wherein: In step one, the satellite is simulated as a resource and a dynamic resource pool is established, all individuals in the initial population are allocated resources in parallel according to the task order of the individual, the resources are released back to the resource pool after the task is completed, and the time set spent by each individual in executing the task is recorded, which is the fitness of the initial population N.

4. The method for resource scheduling optimization for giant constellations according to claim 1, characterized in that: In step two, in the state of mapping The cross action interval is selected and the mutation action interval , respectively, to obtain the cross probability and the mutation probability ; the formula of the cross action interval and the mutation action interval is as follows: ； ； ； ； wherein is a random number between 0 and 1, is a greed rate, is a table and a table is a set of actions recorded under a state , is a table and a table is a column number of the table is an initial temperature, is a decay coefficient, is a set of actions under a state , is a set of values corresponding to the set of actions under a state , is a set of actions under a state , is a set of values corresponding to the set of actions under a state , is a probability distribution of a set of actions corresponding to a state is an index value corresponding to a probability value in the probability distribution , which is randomly sampled according to a probability, and the index value is an action interval corresponding to the probability value.

5. The resource scheduling optimization method for a mega constellation according to claim 1, characterized in that: In step four, the TD error of and is calculated and , and the total TD error is calculated again; the error value td of the TD is calculated by using the EMA smoothing algorithm, the initial value of td is 0, and the mean and variance are calculated, and the adaptive threshold is constructed; The TD error , TD error , total TD error , error value td and adaptive threshold is expressed by the following equation: ； ； ； ； ； wherein and is an adaptive threshold coefficient is a smoothing coefficient, is a discount factor, is a value of the selected action at state when is a value of the selected action at state when is a value of the selected action at state is the next selected state, is an expected value of the selected action at state when is an expected value of the selected action at state when is an expected value of the selected action at state 6. The resource scheduling optimization method for a mega constellation according to claim 1, characterized in that: In step five, if the current belongs to the SARSA policy phase, then use the SARSA policy update and , otherwise, use the Q-Learning policy update and ; ； ； wherein and The formula is as follows: ； ； In the formula, In the state Next, select an action. hour The highest expected value; In the state Next, select an action. hour The highest expected value; For learning rate, Update for improvement learning rate, For improvement learning rate, A learning coefficient greater than 1 The learning coefficient is less than 1.

7. The method for resource scheduling optimization for giant constellations according to claim 1, characterized in that: The specific process of step six is as follows: judge whether a suitable satellite is selected and a connected graph is formed; if yes, calculate the time and energy consumption in the link; otherwise, return to execute step six.

8. The method for resource scheduling optimization for giant constellations according to claim 7, characterized in that: In step six, the suitable satellite is obtained as follows: use condensed hierarchical clustering to divide the available satellites into several clusters; start from the task sequence to be executed by the individual currently, and allocate satellites to the tasks in parallel; if there is a cluster whose number of required resources num is greater than or equal to the number of cluster members, select the num satellites closest in distance within the cluster as the suitable satellites by distance greed. 9.The method for resource scheduling optimization for giant constellations according to claim 7, characterized in that: In step six, the maximum spanning tree MST of the connected graph is calculated, the longest edge in the tree is selected as the key communication link, the propagation time is calculated, and the timestamp is updated; Treating the edges of a connected graph as communication links, calculate the total edge distance of the connected graph, and then calculate the propagation time in the communication links based on the total edge distance. And calculate the energy consumption in the communication link. Total energy consumption of the current task and total energy consumption It can be expressed as follows: ； ； ； wherein and respectively the transmission power and the reception power of the satellite at the time of link propagation, the energy consumption of the satellite to perform the mission.

10. The method for resource scheduling optimization for giant constellations according to claim 1, characterized in that: In step eight, the comprehensive index is calculated as follows: the completion time and total energy consumption of all individuals in the current generation are normalized to 0 to 1 and weighted to obtain the comprehensive index, which is represented by the following formula: ； wherein is the maximum value of the completion time of all individuals of the first generation, is the maximum value of the energy consumption of all individuals of the first generation, is the weighting number; the minimum is updated and the corresponding individual.

Images