Optimization method for distributed task scheduling of giant constellation

Abstract

The application provides a kind of giant constellation distributed task scheduling optimization method, comprising: initialization network parameter, define the mapping function of low-dimensional continuous latent variable mapping to discrete satellite number and initialize particle set;Collect state information containing task characteristics, satellite resource state characteristics and timing context characteristics;The fusion feature is obtained by fusion processing to state information, and the decision distribution for guiding particle set optimization is generated according to strategy network;The particle position is updated based on the decision distribution through particle optimization algorithm, and it is mapped to the corresponding satellite number as scheduling action through mapping function;The scheduling action is executed, and the transition tuple is stored to sample set;Sample from sample set to update the parameters of evaluation network, strategy network and target network;Iterative training until meeting the stopping condition, and output strategy network for actual scheduling. In this way, the application can effectively process large-scale discrete action space and achieve balanced optimization among multiple scheduling targets.

Description

Technical Field

[0001] This invention relates to the field of multi-satellite planning and scheduling technology, and in particular to a method, system, computer-readable storage medium, and computer device for optimizing distributed task scheduling of a giant constellation. Background Technology

[0002] With the continuous growth of global communication demands, traditional terrestrial networks are struggling to meet the increasingly complex application scenarios, especially the network coverage requirements in remote areas, at sea, and in the air. Satellite internet technology has rapidly emerged as an important supplement to global network connectivity. Mega-hybrid constellations, networks composed of low-Earth orbit (LEO), medium-Earth orbit (MEO), and high-Earth orbit (GEO) satellites, are becoming an important direction for future communication development due to their wide coverage, low latency, and high reliability.

[0003] In mega-constellation systems, task scheduling and resource management are critical issues. Due to the widespread distribution of satellite nodes and the limited and dynamic nature of resources, traditional scheduling algorithms face numerous challenges, failing to effectively meet the system's real-time performance and resource utilization efficiency requirements. Existing task scheduling methods often employ centralized architectures, such as classic heuristic algorithms (genetic algorithms, ant colony algorithms, etc.) or centralized solutions based on integer programming. These methods perform well in small-scale or relatively low-complexity systems, but their shortcomings become apparent when applied to mega-constellations. As the number of satellites and the workload increase, centralized methods require comprehensive consideration of global information on all satellites and resources, leading to an exponential increase in computational load, resulting in excessive computational overhead and difficulty in meeting real-time performance requirements. Furthermore, centralized scheduling decisions rely on the aggregation and feedback of large amounts of information, easily causing excessively high task response latency. During expansion, centralized architectures also struggle to transition smoothly; if the central node fails or becomes overloaded, the overall system scheduling faces significant risks, lacking sufficient fault tolerance and scalability.

[0004] In recent years, deep reinforcement learning (DRL) has gradually become a powerful tool for solving complex scheduling problems due to its adaptive learning and decision optimization capabilities. However, existing DRL models still suffer from low training efficiency and insufficient generalization ability when facing complex satellite network environments, making them difficult to directly apply to distributed task scheduling of mega-constellations.

[0005] Therefore, there is an urgent need for a new method that can effectively address the challenges of scheduling tasks for giant constellations, in order to improve the efficiency of task scheduling and optimize resource utilization. Summary of the Invention

[0006] To address the problems of existing technologies, this invention proposes a method, apparatus, computer-readable storage medium, and computer device for optimizing distributed task scheduling in giant constellations. Through innovative deep reinforcement learning algorithms and efficient distributed architecture design, it achieves efficient, intelligent, and reliable scheduling of giant constellation tasks.

[0007] To achieve the aforementioned technical effects, firstly, the present invention provides a method for optimizing distributed task scheduling in a giant constellation, comprising: Step S1: Initialize the parameters of the evaluation network, policy network, and target network, define a mapping function to map low-dimensional continuous latent variables to discrete satellite numbers, and initialize a particle set consisting of multiple particles, where each particle represents the position of a scheduling decision in continuous space; Step S2: Collect the status information of the current scheduling environment. The status information includes the task characteristics of the current task, the real-time satellite resource status characteristics of the satellite constellation, and the temporal context characteristics reflecting the historical scheduling situation. Step S3: The task features, satellite resource status features, and temporal context features are fused to obtain fused features. Based on the fused features and the policy network, a decision distribution is generated to guide the optimization of the particle set. Step S4: Based on the decision distribution, update the position of each particle in the particle set using the particle optimization algorithm, and map the updated particle position to the corresponding satellite number using the mapping function, as a task scheduling action; Step S5: Execute the task scheduling action, observe the scheduling result and reward generated after execution, and store the transition tuple containing the current state, the executed action, the reward and the next state into the sample set; Step S6: Randomly select the transition tuples from the sample set, calculate the target value of each scheduling decision based on the target network, and update and adjust the parameters of the evaluation network according to the target value; update the parameters of the policy network according to the advantages of each scheduling decision and the diversity constraints of the policy; and integrate the parameters of the evaluation network into the parameters of the target network according to a preset ratio, so that the target network gradually approaches the evaluation network. Step S7: Determine whether the preset iteration stopping condition is met. If so, output the trained policy network; otherwise, repeat steps S2 to S6 until the preset iteration stopping condition is met, and output the trained policy network for scheduling actual satellite missions.

[0008] Further, in step S1, the defined mapping function is:

[0009] in, For continuous latent variables, their dimensions are... d satisfy , N Total number of satellites; A learnable projection matrix, used to project the continuous latent variables. x Mapped to satellite feature space, This is a bias term.

[0010] Furthermore, the status information is composed of the task characteristics, satellite resource status characteristics, and temporal context characteristics. The task characteristics are composed of a priority vector representing task priority, a normalized resource demand vector, and a type embedding vector. The satellite resource status characteristics are composed of a resource utilization vector reflecting the average utilization rate of the cluster, a load balancing vector reflecting the load difference between nodes, and a communication window vector identifying the real-time communication capabilities of each satellite. The temporal context characteristics are composed of a scheduling history sequence recording historical scheduling actions, a success rate sequence recording historical scheduling results, and a resource temporal window sequence recording historical resource utilization trends.

[0011] Furthermore, the fusion processing of the task features, the satellite resource status features, and the temporal context features to obtain fused features specifically includes: The temporal context features are encoded by the temporal feature extraction module to obtain a temporal feature vector that captures long-range dependencies; The static feature gating module generates a gating vector for controlling information flow based on the mission features and the satellite resource status features; the gating vector is calculated based on the following formula: in, For activation function, For learnable parameter matrix, The task features are as described above. The satellite resource status characteristics; The dynamic weighted fusion module performs weighted fusion of the temporal feature vector and the static feature composed of the task feature and the satellite resource status feature based on the gate vector to obtain the fused feature; wherein the fused feature is calculated based on the following formula: ;in, This represents element-wise multiplication, where M is a multilayer perceptron. The time-series feature vector is described above.

[0012] Furthermore, the decision distribution is a piecewise continuous distribution constructed by combining a base distribution and the output of the policy network, and is expressed as:

[0013] in, For the base distribution, This is the output of the policy network.

[0014] Further, in step S4, the calculation formula of the particle optimization algorithm is: ; in, For learning rate, It is a differentiable substitution distribution. For a positive definite kernel function used to maintain particle diversity, Importance weights used to correct distribution bias. For gradient calculation, Let be the position vector of the i-th particle in the particle set in continuous space.

[0015] Furthermore, the reward observed in step S5 is calculated using a multi-objective reward function, which is expressed as: ; in, The weighting coefficients for each sub-reward. The task scheduling reward is calculated based on task priority and scheduling success rate. The resource optimization reward is calculated based on satellite resource utilization rate and resource fragmentation rate. The load balancing reward is calculated based on the standard deviation of resource utilization. This is a real-time latency penalty calculated based on task waiting time and failure scenarios.

[0016] Furthermore, the task scheduling reward is calculated based on the following formula: ;in, N For the total number of tasks, Indicates task An indicator function for whether something is scheduled. This is a task priority vector. and These are the priority and the reward coefficient for successful scheduling, respectively. Provides a function to indicate task success; The resource optimization reward is calculated based on the following formula: ;in, For satellite Resource comprehensive utilization rate The resource fragmentation penalty coefficient, For satellite Resource fragmentation rate; The load balancing reward is calculated based on the following formula: ;in, For resource type The standard deviation of cluster utilization For smoothing terms; The real-time latency penalty is calculated based on the following formula: ;in, For the task The waiting time For task execution time, This is the waiting time penalty coefficient. As the baseline coefficient for failure penalty, The exponentially penalized growth coefficient, This is a function to indicate task failure.

[0017] Further, in step S6, updating and adjusting the parameters of the evaluation network according to the target value includes: The evaluation network employs an independently parameterized dual-Q network structure, using the mean square error of the time difference error as the loss function. Parameters are updated by minimizing the sum of the losses of the dual-Q networks. The target value used to calculate the mean square error is derived by the target network, and its calculation method is as follows: ; in, For instant rewards, As a discount factor, The output of the target network, For the next state, For the scheduling action in the next state; The parameters of the policy network are updated based on the advantages of each scheduling decision and the diversity of policies, including: The policy network updates its parameters by minimizing a combined loss function, which is a weighted sum of the policy gradient term and the entropy regularization term, expressed as: ; in, Let z represent the probability of taking action z in state s. This is the advantage function, used to measure the relative advantage of taking action z in state s. Entropy represents the strategy. These are the weighting coefficients used for smooth exploration and exploitation.

[0018] Further, in step S7, the iteration stopping condition is: continuous N The average reward change in each iteration is less than the preset convergence threshold; or The current iteration round has reached the preset maximum iteration round.

[0019] Secondly, the present invention also provides a giant constellation distributed task scheduling optimization system, comprising: The initialization module is used to initialize the parameters of the evaluation network, policy network, and target network, define a mapping function to map low-dimensional continuous latent variables to discrete satellite numbers, and initialize a particle set consisting of multiple particles, where each particle represents the position of a scheduling decision in continuous space. The acquisition module is used to collect the status information of the current scheduling environment. The status information includes the task characteristics of the current task, the real-time satellite resource status characteristics of the satellite constellation, and the temporal context characteristics reflecting the historical scheduling situation. The fusion generation module is used to fuse the task features, the satellite resource status features, and the temporal context features to obtain fused features, and to generate a decision distribution to guide the optimization of the particle set based on the fused features and the policy network. The update and mapping module is used to update the position of each particle in the particle set based on the decision distribution using a particle optimization algorithm, and to map the updated particle position to the corresponding satellite number using the mapping function, as a task scheduling action. The execution module is used to execute the task scheduling action, observe the scheduling result and reward generated after execution, and store the transition tuple containing the current state, the executed action, the reward and the next state into the sample set; The training and parameter update module is used to randomly extract the transition tuples from the sample set, calculate the target value of each scheduling decision based on the target network, and update and adjust the parameters of the evaluation network according to the target value; update the parameters of the policy network according to the advantages of each scheduling decision and the diversity constraints of the policy; and integrate the parameters of the evaluation network into the parameters of the target network according to a preset ratio, so that the target network gradually approaches the evaluation network. The output module is used to determine whether the preset iteration stopping condition is met. If so, it outputs the trained policy network; otherwise, it returns to the acquisition module to repeat the process until the preset iteration stopping condition is met, and outputs the trained policy network for scheduling actual satellite missions.

[0020] Thirdly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0021] Fourthly, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described in the first aspect.

[0022] Compared with the prior art, the present invention has at least the following beneficial effects: This invention, by introducing a highly efficient temporal feature encoder and an advanced policy exploration mechanism, can accurately capture historical state information and dynamic environmental changes. This capability enables the algorithm to quickly and adaptively adjust to the complex and ever-changing satellite network environment, thereby significantly improving its adaptability to dynamic environments. This efficient adaptability is particularly important for highly dynamic systems like satellite networks, as satellite orbital positions, communication link states, and mission requirements are constantly changing.

[0023] This invention designs a multi-objective reward function that comprehensively considers task success rate, resource utilization, load balancing, and real-time latency. This multi-objective reward function design enables the algorithm to achieve a good balance among multiple key performance indicators. In practical applications, task success rate directly affects the service quality of satellite networks, resource utilization relates to the overall system efficiency, load balancing ensures stable system operation, and real-time latency affects user experience. Through this multi-objective optimization, the algorithm can simultaneously optimize these key indicators within a comprehensive framework, thereby achieving a better task scheduling strategy.

[0024] This invention further enhances the robustness and stability of the algorithm by adjusting the strategy through a dynamic reward function. In real-world satellite network environments, various uncertainties and interferences may be encountered, such as communication link interruptions and satellite malfunctions. The dynamic reward function allows the algorithm to simulate these uncertainties during training, thereby improving its anti-interference capability in actual operation. Simultaneously, the dynamic reward function can adjust the reward signal in real time according to environmental changes, enabling the algorithm to better adapt to different scenarios and further optimize the scheduling strategy. Attached Figure Description

[0025] Figure 1 A flowchart illustrating the steps of the giant constellation distributed task scheduling optimization method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the giant constellation distributed task scheduling optimization system provided in another embodiment of the present invention; Figure 3 A schematic diagram of the structure of the computer device provided by the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0027] It should be noted that references to "an embodiment," "embodiment," "example embodiment," etc., in this specification refer to the described embodiment including specific features, structures, or characteristics, but not every embodiment must include these specific features, structures, or characteristics. Furthermore, such expressions do not refer to the same embodiment. Moreover, when describing specific features, structures, or characteristics in conjunction with embodiments, whether or not explicitly described, it is indicated that incorporating such features, structures, or characteristics into other embodiments is within the knowledge of those skilled in the art.

[0028] Furthermore, certain terms are used in the specification and subsequent claims to refer to specific components or parts. Those skilled in the art will understand that manufacturers may use different names or terms to refer to the same component or part. This specification and subsequent claims do not distinguish components or parts by differences in name, but rather by differences in function. The terms "comprising" and "including" used throughout the specification and subsequent claims are open-ended and should be interpreted as "including but not limited to." Additionally, the term "connection" here includes any direct and indirect electrical connection means. Indirect electrical connection means include connections made through other means.

[0029] To address the problem of excessively large discrete action space caused by the sheer number of satellites in mega-constellations, this invention provides a method that models the discrete action space as a segmented continuous distribution. It establishes a bridge between the continuous space and discrete satellite numbers using low-dimensional continuous latent variables and mapping functions, and employs a particle optimization algorithm to efficiently search for optimal scheduling decisions within the continuous space. Simultaneously, it deeply fuses multi-dimensional state information through a temporal feature extraction module, a static feature gating module, and a dynamic weighted fusion module. Combined with a multi-objective reward function and a policy gradient update mechanism based on a dominance function, it achieves efficient and intelligent scheduling of satellite missions in complex dynamic environments.

[0030] To enable those skilled in the art to reproduce and implement the present invention, the various steps of the giant constellation distributed task scheduling optimization method will be described in detail below with reference to the accompanying drawings and specific embodiments. These embodiments are specific illustrations of the present invention, and their descriptions are helpful in understanding the present invention, but should not be construed as limiting the present invention.

[0031] Figure 1This invention illustrates a method for optimizing distributed task scheduling in a giant constellation, comprising the following steps: S1: Initialize the parameters of the evaluation network, policy network, and target network, define a mapping function to map low-dimensional continuous latent variables to discrete satellite numbers, and initialize a particle set consisting of multiple particles, where each particle represents the position of a scheduling decision in continuous space.

[0032] S2: Collect the status information of the current scheduling environment, including the task characteristics of the current task, the real-time satellite resource status characteristics of the satellite constellation, and the temporal context characteristics reflecting historical scheduling.

[0033] S3: The task features, satellite resource status features, and temporal context features are fused to obtain fused features. Based on the fused features and the policy network, a decision distribution is generated to guide the optimization of the particle set.

[0034] S4: Based on the decision distribution, update the position of each particle in the particle set using a particle optimization algorithm, and map the updated particle position to the corresponding satellite number using a mapping function, as a task scheduling action.

[0035] S5: Execute the task scheduling action, observe the scheduling result and reward generated after execution, and store the transition tuple containing the current state, the executed action, the reward and the next state into the sample set.

[0036] S6: Randomly extract the transition tuples from the sample set, calculate the target value of each scheduling decision based on the target network, and update and adjust the parameters of the evaluation network according to the target value; update the parameters of the policy network according to the advantages of each scheduling decision and the diversity constraints of the policy; and integrate the parameters of the evaluation network into the parameters of the target network according to a preset ratio, so that the target network gradually approaches the evaluation network.

[0037] S7: Determine whether the preset iteration stopping condition is met. If so, output the trained policy network; otherwise, repeat steps S2 to S6 until the preset iteration stopping condition is met, and output the trained policy network for scheduling actual satellite missions.

[0038] In satellite mission scheduling, the objective of the action space is to select one satellite from N satellites for scheduling. This problem is modeled as a discrete action space, where each action corresponds to a satellite index. In hybrid constellation mission scheduling, the number of satellites can be very large (e.g., Directly processing such a large-scale discrete action space would pose challenges in terms of computational and sampling efficiency. Therefore, the discrete action space is modeled as a piecewise continuous distribution.

[0039] Specifically, step S1 includes: S11: Define the mapping function ; in, It is a continuous latent variable, and its dimension satisfies , To create a learnable projection matrix, latent variables are... Mapped to satellite feature space; The bias term is initialized to a uniform distribution. To avoid insufficient feature representation due to excessively low dimensionality and excessively high dimensionality leading to increased kernel matrix computation, the implementation example adopts a dimensionality of... Balancing computational cost and performance.

[0040] S12: Initialize the particle set ,in, It is a basic distribution, defined as 3D standard Gaussian distribution: , yes The identity matrix.

[0041] The status information in step S2 is composed of the task characteristics, satellite resource status characteristics, and temporal context characteristics. The task characteristics are composed of a priority vector representing task priority, a normalized resource demand vector, and a type embedding vector. The satellite resource status characteristics are composed of a resource utilization vector reflecting the average utilization rate of the cluster, a load balancing vector reflecting the load difference between nodes, and a communication window vector identifying the real-time communication capabilities of each satellite. The temporal context characteristics are composed of a scheduling history sequence recording historical scheduling actions, a success rate sequence recording historical scheduling results, and a resource temporal window sequence recording historical resource utilization trends.

[0042] Specifically, step S2 includes: S21: Define task characteristics This mainly includes task-related priority, resource requirements, and type information.

[0043] The priority vector is shown below: ; This vector uses one-hot encoding to represent the priority of the task, where Indicates an urgent priority task. Indicates a high-priority task. This indicates a normal priority task.

[0044] The resource demand vector is shown below: ; This vector represents the CPU and memory requirements of the task, and after normalization, the values ​​are limited to the range [0,1].

[0045] The type embedding vector is shown below: ; Task type features are mapped to a vector through the embedding layer, including CPU-intensive, memory-intensive, and bandwidth-intensive tasks.

[0046] Finally, the task features are encoded as follows: .

[0047] S22: Define satellite resource status characteristics The real-time operating status of the hybrid constellation is accurately quantified through multi-dimensional vectors, mainly including three types: resource utilization vector, load balancing vector, and communication window vector.

[0048] The resource utilization rate vector is represented as follows: ; in, This represents the average CPU utilization of the cluster. Similarly, calculate, and These represent the maximum possible resource values ​​in the cluster.

[0049] The load balancing degree vector is represented as follows: ; This feature calculates the resource load balancing degree among satellite nodes. The standard deviation of CPU utilization across all nodes in the cluster. The same calculation applies.

[0050] The communication window vector is represented as follows: ; in, For visibility indication function, in this embodiment, N=130 indicates that there are 130 heterogeneous satellite nodes in the corresponding algorithm environment, including three types of satellites: 6 low Earth orbit (LEO) satellites, 24 medium Earth orbit (MEO) satellites, and 100 high Earth orbit (GEO) satellites. Each satellite occupies a 1-dimensional binary identifier to identify its communicability.

[0051] Finally, the resource status code is as follows: .

[0052] S23: Define temporal context features The modeling of the dynamic evolution of historical scheduling trajectories mainly includes scheduling history sequences, success rate sequences, and resource time sequence windows.

[0053] The definition of the scheduling history sequence is as follows: ; in, For the pace of history, For the embedding dimension. This feature records the previous... The task status at each scheduling moment includes the task type, priority, and resource requirements. Dimensional features, total dimension; and The value can be adjusted according to requirements.

[0054] In this embodiment, the historical sequence is scheduled. By using bidirectional LSTM network encoding, its hidden state captures long-term dependencies in task scheduling.

[0055] LSTM (Long Short-Term Memory) is a special type of recurrent neural network (RNN) architecture primarily used to address the vanishing or exploding gradient problems that traditional RNNs often encounter when processing long sequences of data. This network employs gating mechanisms, including forget gates, input gates, and output gates, to selectively retain or discard information, effectively capturing long-range dependencies. LSTM performs exceptionally well in natural language processing, time series prediction, and other fields, and is widely used in tasks such as text generation, sentiment analysis, and stock prediction.

[0056] The definition of a success rate sequence is as follows: ; This feature is related to the scheduling history sequence. Related, record the most recent The scheduling result is 1, indicating successful scheduling, and 0, indicating failed scheduling.

[0057] The definition of the resource timing window is as follows: ; This feature statistics in the past Global resource utilization at each time step and the current moment (CPU / memory each in one dimension), total Dimension; the length of the time series window. It can also be customized and adjusted.

[0058] Finally, the timing context is encoded as follows: .

[0059] S24: Define the state space It consists of three parts: mission characteristics, satellite resource status, and timing context, defined as follows: ; Total Dimensions of State Features ,in, , , These represent the task feature dimension, resource status dimension, and temporal context dimension, respectively.

[0060] Furthermore, the mission characteristics, satellite resource status characteristics, and temporal context characteristics are fused to obtain fused features, specifically including: The temporal context features are encoded by the temporal feature extraction module to obtain a temporal feature vector that captures long-range dependencies; a static feature gating module generates a gating vector for controlling information flow based on the task features and satellite resource status features; the gating vector is calculated based on the following formula: ;in, For activation function, For learnable parameter matrix, As a task characteristic, The satellite resource status characteristics are used as follows: A dynamic weighted fusion module, based on the gate vector, weights and fuses the time-series feature vector with the static features composed of the mission characteristics and the satellite resource status characteristics to obtain fused features; wherein, the fused features are calculated based on the following formula: ;in, This represents element-wise multiplication, where M is a multilayer perceptron. The time-series feature vector is described above.

[0061] Specifically, step S3 includes: S31: The feature fusion stage achieves feature fusion through a three-stage process, which corresponds to three independent modules: the temporal feature extraction module, the static feature gating module, and the dynamic weighted fusion module.

[0062] The temporal feature extraction module utilizes the encoder The historical trajectory is modeled, and after being processed by the encoder, a low-dimensional feature vector is output. The dimensions can be customized. Through the encoder's self-attention mechanism, It can effectively capture long-term dependencies, thus providing more accurate historical context information for subsequent task scheduling. Mathematically, the temporal feature extraction module can be represented as... ; in, This is the output of the temporal feature extraction module. It is a temporal context feature.

[0063] In this embodiment, the encoder The Transformer encoder is used; Transformer is a deep learning architecture based on self-attention mechanism. It abandons the traditional recurrent neural network structure and processes sequence data in parallel through multi-head self-attention mechanism, which greatly improves training efficiency and performance.

[0064] The goal of the static feature gating module is to generate static weight vectors to control the contribution of static features to the final decision, using a learnable parameter matrix. and activation function Generate gate vector Mathematically, a static feature gating module can be represented as: ; Among them, the gate vector By controlling the contribution of static features to the final decision, static features that are more relevant to the current task are effectively selected. In this embodiment, the activation function... use function.

[0065] The goal of the dynamic weighted fusion module is to dynamically weight and fuse two heterogeneous features, temporal features and static features, to obtain fused features. Using element-wise multiplication and multilayer perceptron M By weighted fusion of temporal and static features, we obtain: .

[0066] In this embodiment, a multilayer perceptron M It is a two-layer fully connected network (hidden layer dimension 64, ReLU activation); S32: Generate decision distributions, specifically including: By combining the base distribution and the policy network output, a sampleable piecewise continuous distribution can be constructed. It is represented as: ; in, It is a basic distribution, defined as a d-dimensional standard Gaussian distribution: , yes The identity matrix, It is the output of the policy network; It retains the characteristics of the base distribution while incorporating the probabilistic information of the policy network for satellite selection.

[0067] By constructing the target distribution The algorithm can transform a discrete action space into a continuous distribution, thereby optimizing the sampling process using efficient sampling methods. This approach is particularly suitable for scenarios with large-scale action spaces, significantly improving sampling efficiency and algorithm stability.

[0068] In step S4, the calculation formula of the particle optimization algorithm is as follows: ; in: It's the learning rate, which controls the update step size. It is a differentiable substitution distribution. It is a positive definite kernel function used to maintain particle diversity. The gradient of a differentiable substitution distribution guides particles to move towards high-probability regions. These are importance weights used to correct for biases introduced by the substitution distribution; Let be the position vector of the i-th particle in the particle set in continuous space.

[0069] In each iteration, the position of the particles is adjusted according to the update rules, gradually approaching the target distribution.

[0070] Finally, discrete actions are calculated and samples are output through a mapping function. The optimized particle positions are converted into discrete satellite numbers, which are used as the final action selection. Mathematically, the output samples... It can be represented as: Among them, the output It is an approximate discrete distribution of the target. The sample indicates the available satellite numbers. The set number of particles.

[0071] Specifically, the selected action Will be applied to the current state To perform task scheduling; in a satellite task scheduling scenario, this means assigning a task to a specific satellite; after performing the action, observe the feedback from the environment, including new states. and rewards The reward is calculated based on a predefined reward function, reflecting the effectiveness of the action. The observed results are stored as a transition tuple. .

[0072] The reward function It is composed of a weighted combination of multiple sub-reward functions, specifically in the following form: ; in, These are the weighting coefficients for each sub-reward, used to balance the importance of different objectives; The task scheduling reward is calculated based on task priority and scheduling success rate. The resource optimization reward is calculated based on satellite resource utilization rate and resource fragmentation rate. The load balancing reward is calculated based on the standard deviation of resource utilization. This is a real-time latency penalty calculated based on task waiting time and failure scenarios.

[0073] In this embodiment, the reward function achieves dynamic perception through the following mechanism: First, the environment state awareness module monitors key indicators in real time, such as task backlog(t), resource utilization rate, cluster resource standard deviation, and maximum task latency, dynamically assessing the difficulty of task scheduling and environmental changes. Second, the adaptive weight adjustment mechanism dynamically adjusts the weights of each sub-reward based on the current system state, ensuring flexible optimization of task objectives under different environments. Finally, a multi-objective balancing strategy is adopted, using weight parameters... Through coordinated adjustments, a balance is achieved between task completion rate, resource utilization rate and system stability.

[0074] The dynamic adjustment strategy for each reward weight is shown in Table 1 below: Table 1:

[0075] Task scheduling rewards It is a key indicator for measuring the direct effectiveness of scheduling decisions, aiming to incentivize the model to schedule tasks more effectively through a comprehensive evaluation of task priority and scheduling success rate. Its mathematical expression is: Where N is the total number of tasks; It is a task The scheduled indicator function, if the task If scheduled, the value is 1; otherwise, it is 0. It is a task The priority vector; It is the reward coefficient for task priority; It is the reward coefficient for successful task scheduling; It is a task The indicator function for successful scheduling, if the task If the scheduling is successful, the value is 1; otherwise, it is 0.

[0076] Resource optimization rewards It aims to encourage high resource utilization and curb resource fragmentation. It is defined as: ;in, It is a satellite CPU / memory overall utilization rate; It is the resource fragmentation penalty coefficient; It is a satellite The resource fragmentation rate is calculated using the following formula: ;in, and They are satellites Currently available CPU and memory resources It is a small smoothing term that prevents division by zero.

[0077] Load balancing rewards The aim is to promote a more balanced distribution of resources within the cluster. The calculation method is as follows: ;in, It is the standard deviation of cluster utilization of resource Z (CPU or memory); It is a small smoothing term to prevent numerical overflow.

[0078] Real-time delay penalty It aims to balance latency sensitivity and completion rate during task scheduling. Its specific form is: ;in, It is a task The waiting time from submission to the start of scheduling; It is the penalty coefficient for waiting time. It is a task The duration from start to finish (success or failure); It is the baseline coefficient for failure penalty; It is the exponentially penalized growth coefficient; It is a task The failure indicator function, if the task If it fails, the value is 1; otherwise, it is 0.

[0079] Further, in step S6, updating and adjusting the parameters of the evaluation network according to the target value includes: the evaluation network adopts an independently parameterized dual-Q network structure, using the mean square error of the time difference error as the loss function, and updating the parameters by minimizing the sum of the losses of the dual-Q network, wherein the target value used to calculate the mean square error is calculated by the target network, and its calculation method is as follows: ; in, For instant rewards, As a discount factor, The output of the target network, For the next state, For scheduling actions in the next state; The parameters of the policy network are updated based on the advantages of each scheduling decision and the diversity constraints of the policies, including: the policy network updates its parameters by minimizing a combined loss function, which is a weighted sum of policy gradient terms and entropy regularization terms, expressed as: ;in, This represents the probability of taking action z in state s. This is the advantage function, used to measure the relative advantage of taking action z in state s. Entropy represents the policy and is used to measure the randomness or uncertainty of the policy. These are the weighting coefficients used for smooth exploration and exploitation.

[0080] Specifically, step S6 includes: S61: Randomly sample a transition tuple from the recorded results. .

[0081] S62: Adjust the parameters of the evaluation network, including: Evaluation network input It consists of two parts: an enhanced heterogeneous feature vector and the next action. After feature fusion processing, an enhanced heterogeneous feature vector is generated. Mathematically, the input to the evaluation network is represented as... ,in This indicates a splicing operation.

[0082] The evaluation network employs an independently parameterized dual-Q network, with the loss function being the mean squared error of the time difference error. This network is then used in conjunction with the target network for stabilization training. The loss function for both Q networks is defined as follows.

[0083] ; in, and These are the outputs of the two Q-networks, and These are the parameters of two Q-networks and two target values. Calculated through a target network with delayed updates; Discount factor; The target network parameters have the same structure as the main network but are updated with a delay, using a soft update method. The overall loss of the network can be expressed as: .

[0084] Q-networks are neural network models used in reinforcement learning to approximate the action-value function (Q-function). They help agents make optimal decisions in their environment by learning the value relationship between states and actions. The core of Q-networks is estimating Q-values ​​through neural network parameterization, thereby achieving efficient processing of complex state and action spaces.

[0085] In this embodiment, the Q-value calculation process is as follows: ; ; In the network parameter settings, the hidden layer dimension is 256. It is used to balance gradient propagation efficiency and nonlinear expression capability.

[0086] Finally, the parameters of the evaluation network are updated by minimizing the overall loss of the evaluation network.

[0087] S63: Adjust the parameters of the policy network, including: The policy network achieves probabilistic transformation from a continuous feature space to a discrete action space through end-to-end differentiable mappings. First, the fused features... Convert to a discrete probability distribution to obtain the probability of each action (satellite selection). , where represents the probability of taking action z in state s.

[0088] The policy optimization objective, by incorporating multi-objective constraints, aims to balance the policy's exploration capability and robustness while maximizing cumulative reward. Specifically, the objective function includes policy gradient guidance and entropy regularization constraints. These components work together to ensure the algorithm's efficiency and stability in complex dynamic environments, as shown below: .

[0089] In this embodiment, the entropy calculation formula for the strategy is as follows: Where N is the number of satellites, Used to mitigate the numerical risks of zero-probability logarithmic operations; entropy coefficient Employing an exponential decay strategy: Initial stage Enhanced exploration, decay rate increases with training steps t make Gradually reduce the level to achieve a smooth transition from exploration to utilization.

[0090] Finally, the parameters of the policy network are updated by minimizing the overall loss of the policy network.

[0091] Specifically, step S7 includes: S71: Determine whether the change in average reward over N consecutive iterations is less than a certain threshold. If it is greater than the threshold If the algorithm converges, proceed to the next step; otherwise, the algorithm is considered to have converged and terminates.

[0092] S72: Update Iteration Rounds Determine if the iteration is complete. ,make If the algorithm fails, return to step S2; otherwise, the algorithm terminates.

[0093] Finally, the optimized policy network is used for actual task scheduling. Corresponding actions are taken based on the current state of the satellite communication system to achieve rational resource allocation.

[0094] In summary, the method provided in this embodiment has the following beneficial effects: First, by combining an efficient temporal feature encoder and a policy exploration mechanism, this method can effectively capture historical state information and dynamic environmental changes, thereby significantly improving the algorithm's adaptability to complex dynamic environments. Second, the multi-objective reward function design, which comprehensively considers task success rate, resource utilization, load balancing, and real-time latency, enables the algorithm to achieve a good balance among multiple key performance indicators, further optimizing the task scheduling strategy. Finally, by adjusting the strategy through a dynamic reward function, the robustness and stability of the algorithm are enhanced, allowing it to maintain efficient scheduling performance in the face of uncertainty and interference, providing an efficient, intelligent, and reliable solution for scheduling giant constellation tasks.

[0095] Figure 2 This invention illustrates a giant constellation distributed task scheduling optimization system 100 according to another embodiment of the present invention. The device 100 includes an initialization module 10, an acquisition module 20, a fusion generation module 30, an update and mapping module 40, an execution module 50, a training and parameter update module 60, and an output module 70, wherein: Initialization module 10 initializes the parameters of the evaluation network, policy network, and target network, defines a mapping function to map low-dimensional continuous latent variables to discrete satellite numbers, and initializes a particle set consisting of multiple particles, where each particle represents the position of a scheduling decision in continuous space. Acquisition module 20 collects the state information of the current scheduling environment, including the task characteristics of the current task, the real-time satellite resource status characteristics of the satellite constellation, and the temporal context characteristics reflecting historical scheduling. Fusion generation module 30 fuses the task characteristics, satellite resource status characteristics, and temporal context characteristics to obtain fused features, and generates a decision distribution to guide the optimization of the particle set based on the fused features and the policy network. Update and mapping module 40 updates the positions of each particle in the particle set based on the decision distribution using a particle optimization algorithm, and maps the updated particle positions to the corresponding satellites using the mapping function. The task scheduling action is assigned a number; the execution module 50 executes the task scheduling action, observes the scheduling result and reward generated after execution, and stores the transition tuple containing the current state, the executed action, the reward, and the next state into the sample set; the training and parameter update module 60 randomly selects the transition tuple from the sample set, calculates the target value of each scheduling decision based on the target network, and updates and adjusts the parameters of the evaluation network according to the target value; updates the parameters of the policy network according to the advantages of each scheduling decision and the diversity constraints of the strategy; and integrates the parameters of the evaluation network into the parameters of the target network according to a preset ratio, so that the target network gradually approaches the evaluation network; the output module 70 determines whether a preset iteration stopping condition is met, and if so, outputs the trained policy network; otherwise, it returns to the acquisition module 20 to repeat the execution until the preset iteration stopping condition is met, and outputs the trained policy network for scheduling actual satellite missions.

[0096] The specific implementation and operation process of each module in the system 100 described in this embodiment can be understood by referring to the relevant descriptions in the foregoing method embodiments, and all of them can achieve the functions and effects achieved by the corresponding method steps. For the sake of simplicity and to avoid repetition, they will not be described again here.

[0097] Based on the same inventive concept as other embodiments, this invention also provides a computer-readable storage medium for storing a computer program for executing the aforementioned giant constellation distributed task scheduling optimization method. For example, computer program instructions, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. The program instructions invoking the methods of this application may be stored in a fixed or removable storage medium, and / or transmitted via data streams in broadcast or other signal carrying media, and / or stored in the storage medium of a computer device operating according to the program instructions. Here, in one embodiment according to this application, it includes a... Figure 3 The computer device 400 shown preferably includes a memory 200 for storing computer programs and a processor 300 for executing computer programs. When the computer program is executed by the processor 300, the computer device 400 is triggered to execute the methods and / or technical solutions based on the foregoing embodiments.

[0098] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0099] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0100] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A method for optimizing distributed task scheduling in a giant constellation, characterized in that, Including: Step S1: Initialize the parameters of the evaluation network, policy network, and target network, define a mapping function to map low-dimensional continuous latent variables to discrete satellite numbers, and initialize a particle set consisting of multiple particles, where each particle represents the position of a scheduling decision in continuous space; Step S2: Collect the status information of the current scheduling environment. The status information includes the task characteristics of the current task, the real-time satellite resource status characteristics of the satellite constellation, and the temporal context characteristics reflecting the historical scheduling situation. Step S3: The task features, satellite resource status features, and temporal context features are fused to obtain fused features. Based on the fused features and the policy network, a decision distribution is generated to guide the optimization of the particle set. Step S4: Based on the decision distribution, update the position of each particle in the particle set using the particle optimization algorithm, and map the updated particle position to the corresponding satellite number using the mapping function, as a task scheduling action; Step S5: Execute the task scheduling action, observe the scheduling result and reward generated after execution, and store the transition tuple containing the current state, the executed action, the reward and the next state into the sample set; Step S6: Randomly select the transition tuples from the sample set, calculate the target value of each scheduling decision based on the target network, and update and adjust the parameters of the evaluation network according to the target value; update the parameters of the policy network according to the advantages of each scheduling decision and the diversity constraints of the policy; and integrate the parameters of the evaluation network into the parameters of the target network according to a preset ratio, so that the target network gradually approaches the evaluation network. Step S7: Determine whether the preset iteration stopping condition is met. If so, output the trained policy network; otherwise, repeat steps S2 to S6 until the preset iteration stopping condition is met, and output the trained policy network for scheduling actual satellite missions.

2. The method according to claim 1, characterized in that, In step S1, the defined mapping function is: in, For continuous latent variables, their dimensions are... d satisfy , N Total number of satellites; A learnable projection matrix, used to project the continuous latent variables. x Mapped to satellite feature space, This is a bias term.

3. The method according to claim 1, characterized in that, The status information is composed of the task characteristics, satellite resource status characteristics, and temporal context characteristics. The task characteristics are composed of a priority vector representing task priority, a normalized resource demand vector, and a type embedding vector. The satellite resource status characteristics are composed of a resource utilization vector reflecting the average utilization rate of the cluster, a load balancing vector reflecting the load difference between nodes, and a communication window vector identifying the real-time communication capabilities of each satellite. The temporal context characteristics are composed of a scheduling history sequence recording historical scheduling actions, a success rate sequence recording historical scheduling results, and a resource temporal window sequence recording historical resource utilization trends.

4. The method according to claim 1, characterized in that, The process of fusing the mission features, the satellite resource status features, and the temporal context features to obtain fused features specifically includes: The temporal context features are encoded by the temporal feature extraction module to obtain a temporal feature vector that captures long-range dependencies; The static feature gating module generates a gating vector for controlling information flow based on the mission features and the satellite resource status features; the gating vector is calculated based on the following formula: in, For activation function, For learnable parameter matrix, The task features are as described above. The satellite resource status characteristics; The dynamic weighted fusion module performs weighted fusion of the temporal feature vector and the static feature composed of the task feature and the satellite resource status feature based on the gate vector to obtain the fused feature; wherein the fused feature is calculated based on the following formula: ;in, This represents element-wise multiplication, where M is a multilayer perceptron. The time-series feature vector is described above.

5. The method according to claim 1, characterized in that, The decision distribution is a piecewise continuous distribution constructed by combining a base distribution and the output of the policy network, and is expressed as follows: in, For the base distribution, This is the output of the policy network.

6. The method according to claim 1, characterized in that, In step S4, the calculation formula of the particle optimization algorithm is as follows: ； in, For learning rate, It is a differentiable substitution distribution. For a positive definite kernel function used to maintain particle diversity, Importance weights used to correct distribution bias. For gradient calculation, Let be the position vector of the i-th particle in the particle set in continuous space.

7. The method according to claim 1, characterized in that, The reward observed in step S5 is calculated by a multi-objective reward function, which is expressed as: ； in, The weighting coefficients for each sub-reward. The task scheduling reward is calculated based on task priority and scheduling success rate. The resource optimization reward is calculated based on satellite resource utilization rate and resource fragmentation rate. The load balancing reward is calculated based on the standard deviation of resource utilization. This is a real-time latency penalty calculated based on task waiting time and failure scenarios.

8. The method according to claim 7, characterized in that, The task scheduling reward is calculated based on the following formula: ;in, N For the total number of tasks, Indicates task An indicator function for whether something is scheduled. This is a task priority vector. and These are the priority and the reward coefficient for successful scheduling, respectively. Provides a function to indicate task success; The resource optimization reward is calculated based on the following formula: ;in, For satellite Resource comprehensive utilization rate The resource fragmentation penalty coefficient, For satellite Resource fragmentation rate; The load balancing reward is calculated based on the following formula: ;in, For resource type The standard deviation of cluster utilization For smoothing terms; The real-time latency penalty is calculated based on the following formula: ;in, For the task The waiting time For task execution time, This is the waiting time penalty coefficient. As the baseline coefficient for failure penalty, The exponentially penalized growth coefficient, This is a function to indicate task failure.

9. The method according to claim 1, characterized in that, In step S6, updating and adjusting the parameters of the evaluation network according to the target value includes: The evaluation network employs an independently parameterized dual-Q network structure, using the mean square error of the time difference error as the loss function. Parameters are updated by minimizing the sum of the losses of the dual-Q networks. The target value used to calculate the mean square error is derived by the target network, and its calculation method is as follows: ; in, For instant rewards, As a discount factor, The output of the target network, For the next state, For the scheduling action in the next state; The parameters of the policy network are updated based on the advantages of each scheduling decision and the diversity of policies, including: The policy network updates its parameters by minimizing a combined loss function, which is a weighted sum of the policy gradient term and the entropy regularization term, expressed as: ; in, Let z represent the probability of taking action z in state s. This is the advantage function, used to measure the relative advantage of taking action z in state s. Entropy represents the strategy. These are the weighting coefficients used for smooth exploration and exploitation.

10. The method according to claim 1, characterized in that, In step S7, the iteration stopping condition is: continuous N The average reward change in each iteration is less than the preset convergence threshold; or The current iteration round has reached the preset maximum iteration round.

Images