A method, system, electronic equipment and medium for integrated space-ground resource scheduling

Abstract

This application discloses a method, system, electronic device, and medium for integrated space-ground resource scheduling. The method trains a policy parameter prediction model for all agents based on a first loss function and a second loss function until a new policy parameter set is output after training. This new policy parameter set is used as the current policy parameter set in the transfer learning phase to calculate the updated policy parameter set and construct a third loss function. Based on the updated and current policy parameter sets, a stability constraint term is constructed and added to the third loss function to obtain the overall objective function. The policy parameter prediction model and the transfer learning phase are trained iteratively until the overall objective function converges to a preset threshold, yielding the target policy parameter set output by the transfer learning phase. The target policy parameter set is then deployed to the integrated space-ground network. This application enables real-time resource scheduling and improves the reliability of resource scheduling.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a method, system, electronic device and medium for integrated space-ground resource scheduling. Background Technology

[0002] The integrated space-ground network comprises satellite systems, airborne platforms, ground base stations, and edge computing nodes, aiming to achieve collaborative work and data exchange among space, air, and ground resources. These systems are characterized by widely distributed nodes, heterogeneous resource types (such as computing, storage, and bandwidth), dynamically changing link states, and diverse mission types (such as communication, remote sensing, and observation). Due to the large scale of the system and the time-varying topology, resource scheduling requires optimal decision-making under multiple constraints (such as latency and energy consumption), making integrated space-ground resource scheduling a high-dimensional, highly dynamic, and complex optimization problem.

[0003] There are two main types of existing integrated space-ground resource scheduling technologies: traditional centralized optimization scheduling methods and intelligent adaptive scheduling methods based on reinforcement learning. However, these existing methods still suffer from poor scalability, low collaborative efficiency, and a lack of dynamic adaptability in complex and dynamic integrated space-ground environments, and cannot yet meet the requirements for resource scheduling with high real-time performance and high reliability. Summary of the Invention

[0004] This application aims to propose a method, system, electronic device, and medium for integrated space-ground resource scheduling, which can solve the problems of poor scalability, low collaborative efficiency, and lack of dynamic adaptability of existing methods in complex and dynamic integrated space-ground environments. It can achieve real-time resource scheduling and improve the reliability of resource scheduling.

[0005] In a first aspect, embodiments of this application provide a method for integrated space-ground resource scheduling, the method comprising:

[0006] By assigning each satellite, ground station, and edge node in the integrated space-ground network to an intelligent agent, multiple intelligent agents are obtained.

[0007] The instant global reward is determined based on the task completion rate, average latency, energy consumption, and resource allocation balance corresponding to the current time step. The instant global reward is updated after the multiple agents perform resource scheduling operations according to the new set of policy parameters.

[0008] Each agent is assigned a policy parameter prediction model that includes a policy network and a value network, and a first loss function for the value network and a second loss function for the policy network are constructed based on the instantaneous global reward.

[0009] Based on the first loss function and the second loss function, a training sample set is randomly drawn from the experience pool to train the policy parameter prediction model of all agents until the training is completed and a new set of policy parameters is output.

[0010] The new policy parameter set is used as the current policy parameter set in the transfer learning stage to calculate the policy parameter set updated by transfer and construct the third loss function of the transfer learning stage. The policy parameter set updated by transfer is used for training the policy parameter prediction model in the next round.

[0011] Based on the updated policy parameter set and the current policy parameter set, a stability constraint term is constructed, and the stability constraint term is added to the third loss function to obtain the total objective function;

[0012] The policy parameter prediction model and the transfer learning stage are trained iteratively until the total objective function converges to a preset threshold, and the set of target policy parameters output by the transfer learning stage is obtained.

[0013] The target strategy parameter set is deployed to the integrated space-ground network so that integrated space-ground resource scheduling can be performed according to the target strategy parameter set.

[0014] Compared with the prior art, the first aspect of this application has the following beneficial effects:

[0015] This method determines the real-time global reward by comprehensively considering task completion rate, average latency, energy consumption, and resource allocation balance. It can adaptively adjust and optimize the real-time global reward based on the real-time operating status of the integrated space-ground network, thereby improving the flexibility, long-term efficiency, and adaptability of the scheduling strategy in complex environments. By training a policy parameter prediction model for all agents using a random sample set drawn from the experience pool based on a first and second loss function, a new set of policy parameters is output after training is complete. This allows for dynamic adjustment of the joint value relationship between agents based on task load and network status, significantly improving the collaborative efficiency and decision-making stability of multiple nodes in the integrated space-ground network. Based on the updated policy parameter set and the current policy parameter set, a stability constraint term is constructed and added to the third loss function to obtain the overall objective function. The policy parameter prediction model and the transfer learning stage are trained iteratively until the overall objective function converges to a preset threshold, resulting in the target policy parameter set output by the transfer learning stage. The target policy parameter set is then deployed to the integrated space-ground network to perform integrated space-ground resource scheduling based on the target policy parameter set. Through transfer learning and the stability constraint term, policy oscillation and forgetting can be prevented, enabling rapid policy transfer and smooth convergence when the environment changes. This achieves rapid transfer and long-term stability of the model in dynamic environments, as well as real-time resource scheduling and improved reliability.

[0016] Secondly, embodiments of this application also provide an integrated space-ground resource scheduling system, the system comprising:

[0017] The intelligent agent construction unit is used to assign an intelligent agent to each satellite, each ground station, and each edge node in the integrated space-ground network, resulting in multiple intelligent agents.

[0018] The global reward determination unit is used to determine the instant global reward based on the task completion rate, average latency, energy consumption, and resource allocation balance corresponding to the current time step. The instant global reward is updated after the multiple agents perform resource scheduling operations according to the new set of policy parameters.

[0019] The first construction unit is used to assign a policy parameter prediction model containing a policy network and a value network to each agent, and to construct a first loss function of the value network and a second loss function of the policy network based on the instantaneous global reward.

[0020] The model training unit is used to train the policy parameter prediction model of all agents by randomly sampling a training sample set from the experience pool based on the first loss function and the second loss function, until a new set of policy parameters is output after training is completed.

[0021] The second construction unit is used to take the new policy parameter set as the current policy parameter set in the transfer learning stage, to calculate the policy parameter set after transfer and construct the third loss function of the transfer learning stage, and the policy parameter set after transfer is used for training the policy parameter prediction model in the next round.

[0022] The third construction unit is used to construct a stability constraint term based on the migration-updated policy parameter set and the current policy parameter set, and add the stability constraint term to the third loss function to obtain the total objective function;

[0023] The iterative training unit is used to iteratively train the policy parameter prediction model and the transfer learning stage until the total objective function converges to a preset threshold, thereby obtaining the target policy parameter set output by the transfer learning stage.

[0024] The resource scheduling unit is used to deploy the target policy parameter set to the integrated space-ground network so as to perform integrated space-ground resource scheduling according to the target policy parameter set.

[0025] Thirdly, embodiments of this application also provide an electronic device, including at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, the instructions being executed by the at least one control processor to enable the at least one control processor to execute a space-ground integrated resource scheduling method as described above.

[0026] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the integrated space-ground resource scheduling method described above.

[0027] It is understood that the beneficial effects of the second to fourth aspects compared with the related technologies are the same as the beneficial effects of the first aspect compared with the related technologies. Please refer to the relevant description in the first aspect above, which will not be repeated here. Attached Figure Description

[0028] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0029] Figure 1 This is a flowchart illustrating an embodiment of the integrated space-ground resource scheduling method provided in this application;

[0030] Figure 2This is a schematic diagram of the centralized training and distributed execution architecture in the best embodiment of the integrated space-ground resource scheduling method provided in this application;

[0031] Figure 3 This is a schematic diagram of the adaptive reward mechanism structure in the best embodiment of the integrated space-ground resource scheduling method provided in this application;

[0032] Figure 4 This is a schematic diagram of the online migration and continuous learning mechanism structure in the best embodiment of the integrated space-ground resource scheduling method provided in this application;

[0033] Figure 5 This is a schematic diagram of the structure of an embodiment of the integrated space-ground resource scheduling system provided in this application;

[0034] Figure 6 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application. Detailed Implementation

[0035] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0036] In the description of this application, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of the technical features indicated.

[0037] In the description of this application, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0038] In the description of this application, it should be noted that, unless otherwise explicitly defined, terms such as "setup," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this application in conjunction with the specific content of the technical solution.

[0039] First, let's analyze some of the terms used in this application:

[0040] Space-ground integrated network: refers to a multi-layered heterogeneous network structure composed of satellites, air platforms (such as drones and high-altitude balloons) and ground communication systems. It realizes the integration and interoperability of resources in the air, space and ground through unified scheduling and collaborative communication, and provides support for mission scheduling, data transmission and edge computing.

[0041] Multi-agent reinforcement learning: a distributed decision-making framework based on reinforcement learning, in which multiple agents interact, cooperate or compete with each other in a shared environment, and optimize the overall performance of the system through joint learning of states, actions and rewards.

[0042] Centralized Training and Decentralized Execution (CTDE): A typical architecture in multi-agent systems. During the training phase, global state information is centrally acquired to optimize the overall policy, while during the execution phase, each agent makes independent decisions based only on local observations, thus balancing system optimization with real-time requirements.

[0043] Value Decomposition Network (VDN) is a multi-agent reinforcement learning framework that achieves collaborative optimization by decomposing the global value function into the sum of local value functions of multiple agents. The VDN structure enables efficient centralized training and distributed execution.

[0044] Differential transfer update: a lightweight transfer learning strategy that only updates local parameters in the model that are significantly affected by environmental changes, reducing computational overhead and accelerating the policy adaptation process.

[0045] Stability constraint: A regularization term introduced in policy updates or transfer training to limit the rate of parameter change and prevent the model from oscillating or degrading in rapid environment switching.

[0046] Differential incremental transfer: During online learning, parameters are updated incrementally. The learning rate and update direction are dynamically adjusted by comparing the differences between the current task and historical tasks, ensuring the continuity and stability of the model transfer process.

[0047] Catastrophic forgetting refers to the phenomenon where, during continuous learning, a model forgets knowledge from previous tasks while learning new ones. Introducing stability constraints and differential incremental transfer mechanisms can effectively mitigate this problem and maintain the model's long-term memory capabilities.

[0048] Existing integrated space-ground resource scheduling methods still suffer from poor scalability, low collaborative efficiency, and lack of dynamic adaptability in complex and dynamic integrated space-ground environments, and cannot yet meet the resource scheduling requirements of high real-time performance and high reliability.

[0049] To address the problems existing in the prior art, this application proposes an integrated space-ground resource scheduling method, system, electronic device, and medium.

[0050] Reference Figure 1 This application provides a schematic flowchart of an integrated space-ground resource scheduling method. This integrated space-ground resource scheduling method is applied to electronic devices, such as servers or mobile terminals. Figure 1 As shown, the integrated space-ground resource scheduling method may include the following steps:

[0051] Step S101: Assign each satellite, ground station, and edge node in the integrated space-ground network to an intelligent agent, resulting in multiple intelligent agents;

[0052] Step S102: Determine the real-time global reward based on the task completion rate, average latency, energy consumption, and resource allocation balance corresponding to the current time step. The real-time global reward is updated after multiple agents perform resource scheduling operations according to the new set of policy parameters.

[0053] Step S103: Assign a policy parameter prediction model containing a policy network and a value network to each agent, and construct the first loss function of the value network and the second loss function of the policy network based on the immediate global reward.

[0054] Step S104: Based on the first loss function and the second loss function, randomly select a training sample set from the experience pool to train the policy parameter prediction model of all agents until the training is completed and a new policy parameter set is output.

[0055] Step S105: Use the new policy parameter set as the current policy parameter set in the transfer learning stage to calculate the policy parameter set after transfer and construct the third loss function in the transfer learning stage. The policy parameter set after transfer is used for training the policy parameter prediction model in the next round.

[0056] Step S106: Based on the updated policy parameter set and the current policy parameter set, construct a stability constraint term, and add the stability constraint term to the third loss function to obtain the total objective function;

[0057] Step S107: Iterate through the policy parameter prediction model and the transfer learning stage until the overall objective function converges to a preset threshold, and obtain the target policy parameter set output by the transfer learning stage.

[0058] Step S108: Deploy the target strategy parameter set to the integrated space-ground network so as to perform integrated space-ground resource scheduling according to the target strategy parameter set.

[0059] In this embodiment, multiple agents are obtained by assigning an agent to each satellite, ground station, and edge node in the integrated space-ground network. The real-time global reward is determined based on the task completion rate, average latency, energy consumption, and resource allocation balance at the current time step. This real-time global reward is updated after the multiple agents perform resource scheduling operations according to the new policy parameter set. By comprehensively considering task completion rate, average latency, energy consumption, and resource allocation balance, the real-time global reward can be adaptively adjusted and optimized according to the real-time operating status of the integrated space-ground network, thereby improving the flexibility, long-term efficiency, and adaptability of the scheduling strategy in complex environments. A policy parameter prediction model containing a policy network and a value network is assigned to each agent. Based on the real-time global reward, a first loss function for the value network and a second loss function for the policy network are constructed. Based on the first and second loss functions, a training sample set is randomly drawn from the experience pool to train the policy parameter prediction models of all agents until a new policy parameter set is output after training. By jointly training the policy parameter prediction models of all agents, the joint value relationship between agents can be dynamically adjusted according to task load and network status, thereby significantly improving the collaborative efficiency and decision-making stability of multiple nodes in the integrated space-ground network. The new policy parameter set is used as the current policy parameter set in the transfer learning stage to calculate the updated policy parameter set and construct the third loss function for the transfer learning stage. The updated policy parameter set is used for training the policy parameter prediction model in the next round. Based on the updated policy parameter set and the current policy parameter set, a stability constraint term is constructed and added to the third loss function to obtain the overall objective function. The policy parameter prediction model and the transfer learning stage are trained iteratively until the overall objective function converges to a preset threshold, resulting in the target policy parameter set output by the transfer learning stage. The target policy parameter set is deployed to the integrated space-ground network to perform integrated space-ground resource scheduling based on the target policy parameter set. Through transfer learning and stability constraints, policy oscillation and forgetting can be prevented, enabling rapid policy transfer and smooth convergence when the environment changes. This achieves rapid transfer and long-term stability of the model in dynamic environments, as well as real-time resource scheduling and improved reliability.

[0060] The aforementioned experience pool can be formed by performing several rounds of initial interaction before model training begins. A randomly initialized policy network outputs an action probability distribution and performs sampling decisions. The environment returns the corresponding state transitions and rewards, generating initial four-tuple samples (including the current state, action, reward, and next state) to fill the experience pool. Once the number of samples in the experience pool reaches a set threshold, the model training phase begins.

[0061] The aforementioned transfer learning stage can combine real-time collected data (i.e., collected environmental state data, including information such as task load, link bandwidth, and node energy consumption) with historical high-value samples stored in the experience memory buffer to form a hybrid training sample set. This hybrid training sample set is used to train the transfer learning stage to quickly update the strategy.

[0062] The above method of adding the stability constraint term to the third loss function to obtain the total objective function can be achieved by weighting the stability constraint term and then adding it to the third loss function to obtain the total objective function.

[0063] The aforementioned preset threshold can be set manually and can be changed according to actual conditions; this embodiment does not impose specific limitations.

[0064] In some implementations, the immediate global reward is determined based on the task completion rate, average latency, energy consumption, and resource allocation balance corresponding to the current time step, including:

[0065] The task completion rate is calculated based on the ratio of the number of tasks successfully completed within the current time step to the total number of tasks.

[0066] The average latency is calculated based on the difference between the start time and completion time of the task within the current time step.

[0067] Energy consumption is calculated based on the power of the node and the task execution duration within the current time step. The node can be a satellite, ground station, or edge node in the integrated space-ground network.

[0068] The resource allocation balance is calculated by using the standard deviation of the resource utilization rate of each node within the current time step.

[0069] The task completion rate, average latency, energy consumption, and resource allocation balance are weighted and summed to obtain the immediate global reward.

[0070] In this embodiment, by comprehensively considering task completion rate, average latency, energy consumption, and resource allocation balance, the real-time global reward is determined. The real-time global reward can be adaptively adjusted and optimized according to the real-time operating status of the integrated space-ground network, thereby improving the flexibility, long-term efficiency, and adaptability of the scheduling strategy in complex environments.

[0071] The number of tasks successfully completed within the time step and the total number of tasks, the start time and completion time difference of tasks within the time step, the power of nodes and task execution time within the time step, and the resource utilization rate of each node are all data that can be directly obtained by the system, and will not be described in detail in this embodiment.

[0072] In some implementations, a first loss function for the value network and a second loss function for the policy network are constructed based on immediate global rewards, including:

[0073] Calculate the local value function for each agent based on the value function of the value network.

[0074] The global value function is obtained by summing the local value functions corresponding to each agent.

[0075] Construct the first loss function of the value network based on the global value function and the immediate global reward;

[0076] A second loss function for the policy network is constructed based on the square norm of the difference between the policies of every two agents and the immediate global reward.

[0077] In this embodiment, a local value function is calculated for each agent based on the value function of the value network; the local value functions of each agent are summed to obtain the global value function; a first loss function of the value network is constructed based on the global value function and the immediate global reward; and a second loss function of the policy network is constructed based on the squared norm of the difference between the policies of every two agents and the immediate global reward. This provides a solid data foundation for the subsequent joint training of policy parameter prediction models for all agents, thereby improving the collaborative efficiency and decision-making stability of multiple nodes in the integrated space-ground network.

[0078] In some implementations, when training a policy parameter prediction model for all agents by randomly drawing training samples from an experience pool based on a first loss function and a second loss function, the method further includes:

[0079] State value estimation is obtained through value networks;

[0080] Construct a policy gradient update function based on the policy network's value function, state value estimation, and immediate global reward;

[0081] Gradient updates are performed on the second loss function based on the policy gradient update function to train the policy parameter prediction model.

[0082] In this embodiment, a policy gradient update function is constructed based on the value function of the policy network, the state value estimate, and the immediate global reward. The second loss function is then updated using the policy gradient update function, which better guides the parameter update direction of the agent's policy network and lays a good data foundation for improving the reliability of resource scheduling in the future.

[0083] In some implementations, the new policy parameter set is used as the current policy parameter set in the transfer learning phase to calculate the updated policy parameter set and construct a third loss function for the transfer learning phase, including:

[0084] The updated set of policy parameters after migration is calculated as follows:

[0085] ;

[0086] ;

[0087] in, This represents the set of policy parameters after the migration update. Indicates the dynamic learning rate. This represents the loss function calculated based on the current reward feedback. Represents the policy parameters at the current time. gradient operator, This represents the stability constraint coefficient. This represents the baseline policy parameters from the previous convergence phase. This indicates that in the empirical sample set, for Perform expectation calculation. Indicates the current time Instant global rewards Indicates the discount factor. Indicates the next moment The observation status, Indicates the next moment Candidate actions, This represents the set of policy parameters for the target network. Indicates the current time The observation status, Indicates the current time The action.

[0088] In this embodiment, the above calculation formula ensures that the update range of the policy parameters is limited to a stable range, thereby avoiding scheduling anomalies caused by large policy fluctuations. Through this "differential incremental migration," the model can quickly correct its adaptation bias to the new environment with extremely low computational overhead.

[0089] In some implementations, the new policy parameter set is used as the current policy parameter set in the transfer learning phase to calculate the updated policy parameter set and construct a third loss function for the transfer learning phase, including:

[0090] The third loss function for the transfer learning stage is constructed as follows:

[0091] ;

[0092] in, This represents the third loss function. Indicates from the experience memory buffer and the current interactive dataset And the expectation of the concentrated sampling of experience samples, This represents the immediate global reward corresponding to the current sample. Indicates the discount factor. Indicates the observation state at the next moment. Indicates the candidate action for the next moment. Represents the current set of policy parameters. This indicates the current observation state. It indicates the action at the current moment.

[0093] In this embodiment, by jointly optimizing the old and new samples, the model can adapt to the new environmental characteristics while retaining the original task scheduling rules, thereby maintaining long-term performance stability in dynamic environments.

[0094] In some implementations, stability constraints are constructed based on the updated set of policy parameters and the current set of policy parameters, including:

[0095] The difference between all parameters in the updated strategy parameter set and all parameters in the current strategy parameter set is calculated to obtain the difference result;

[0096] Solve for the L2 norm square of the difference results to construct stability constraint terms.

[0097] In this embodiment, during continuous updates, frequent parameter adjustments may cause policy oscillations or even learning divergence. Therefore, this embodiment introduces a stability constraint term into the loss function to limit the magnitude of policy parameter changes between adjacent updates, ensuring that the policy parameter update path exhibits smooth convergence characteristics. This guarantees that in complex, time-varying network environments, scheduling performance maintains a gradual improvement over continuous time periods rather than experiencing drastic fluctuations.

[0098] To facilitate understanding by those skilled in the art, a set of preferred embodiments is provided below:

[0099] In engineering and research practice, there are two main existing technologies for integrated space-ground resource scheduling: traditional centralized optimization scheduling methods and intelligent adaptive scheduling methods based on reinforcement learning. However, existing methods have the following drawbacks:

[0100] (1) The centralized scheduling structure lacks system scalability and real-time performance.

[0101] Existing methods generally employ centralized control or single-agent reinforcement learning architectures, where ground or central nodes collect system states and calculate scheduling strategies. This centralized structure can achieve relatively good global decision-making when the network size is small; however, in integrated space-ground environments with a large number of satellites and frequent link changes, the centralized node needs to handle massive amounts of state information and task requests, increasing computational complexity and leading to greater scheduling latency, making it difficult to meet real-time requirements. Furthermore, if the central node experiences a communication interruption, it will affect the stability and reliability of the entire system.

[0102] To address the aforementioned issues, this embodiment designs a multi-agent reinforcement learning architecture based on "Centralized Training and Distributed Execution (CTDE)". During the centralized training phase, the central node aggregates global information and obtains the globally optimal policy through centralized optimization. During the distributed execution phase, each satellite, ground station, and edge node acts as an independent agent, making autonomous decisions based solely on its local state. This design retains the optimization capabilities provided by global information while avoiding the bottlenecks of real-time communication and centralized computation. Through this distributed parallel decision-making mechanism, this embodiment enables the scheduling system to maintain high response speed and system stability even in complex dynamic environments, solving the problems of insufficient scalability and real-time performance in centralized scheduling.

[0103] (2) The lack of a collaborative mechanism between agents leads to an imbalance in resource allocation.

[0104] Current reinforcement learning scheduling methods mostly employ single-agent decision-making or independent learning by each node, lacking effective collaboration and policy sharing among agents. Due to significant differences in task load, bandwidth, and computing resources among different nodes, independent decision-making easily leads to resource allocation biases, resulting in some nodes being overloaded and others idle, thus reducing overall resource utilization. In long-term operation, this lack of collaboration leads to reduced system scheduling efficiency, making it impossible to achieve global equilibrium and optimal allocation.

[0105] To address the aforementioned issues, this embodiment introduces a multi-agent cooperation mechanism, enabling each node to consider not only its own gains but also the overall system benefits during the learning process. In the centralized training phase, agents achieve collaborative optimization through parameter sharing and joint value decomposition (such as VDN or QMIX), ensuring that local strategies align with global objectives. In the distributed execution phase, nodes periodically exchange lightweight load information for dynamically balancing task and resource allocation. The core idea of ​​this mechanism is to achieve global resource balancing through "shared learning + local negotiation," ensuring the system maintains high throughput and stable resource utilization efficiency even under varying loads.

[0106] (3) The reward function is fixed and cannot dynamically reflect changes in task and network status.

[0107] Most existing reinforcement learning methods employ statically or empirically designed reward functions, pre-setting fixed weights during the training phase to balance task completion rate, latency, and energy consumption. However, the task types, priorities, and link states in integrated space-ground networks change significantly over time, and fixed reward functions struggle to reflect these changes. When high-priority tasks occur or network congestion arises, the agent executes decisions based on outdated objectives, leading to slow task response, delayed resource scheduling, and impacting overall system performance.

[0108] To address the aforementioned issues, this embodiment proposes an adaptive reward mechanism. This mechanism dynamically adjusts reward weights by monitoring the system's operational status in real time. When a high-priority task is detected, the system automatically increases the weight of the task completion rate in the reward function; when there is link congestion or excessive energy consumption, the weight of the penalty term is strengthened to guide the agent to reduce latency or energy consumption. Through this dynamic reward adjustment method, the system can adaptively optimize the scheduling strategy based on the urgency of tasks and resource status, enabling the reinforcement learning model to maintain flexibility and efficiency in complex environments.

[0109] (4) The scheduling strategy lacks environmental adaptation and migration capabilities.

[0110] Existing reinforcement learning models are mostly trained offline in fixed simulation environments, assuming stable task distribution and link conditions. However, in actual deployments, satellite motion, channel fluctuations, and task arrival rates all exhibit strong randomness, resulting in significant differences between the training and real-world environments. After deployment, the model cannot adjust parameters based on real-time feedback, leading to degraded policy performance and difficulty in generalization. Some algorithms may also experience training oscillations and policy degradation in non-stationary environments, affecting the long-term stability and reliability of the system.

[0111] To address the aforementioned issues, this embodiment employs an online transfer and continuous learning mechanism. During operation, the system continuously collects data such as node status and task execution feedback, and uses this data to perform lightweight online fine-tuning of the model. When a change in environmental feature distribution is detected, the system quickly retrains using existing model parameters as initial conditions, thereby shortening the adaptation time. Through this design, the scheduling system can maintain long-term learning capabilities and high-performance operation under different network topologies, task types, or channel conditions.

[0112] The purpose of this embodiment is to address the problems existing in the current integrated space-ground network, such as poor scalability of the centralized scheduling structure, weak agent collaboration capability, static and inflexible reward function, and insufficient policy transfer capability in dynamic environments. It proposes an integrated space-ground resource scheduling method that integrates adaptive reward and transfer learning.

[0113] This embodiment aims to enable different nodes to autonomously perceive resource status and make collaborative decisions in a distributed environment through a multi-agent cooperative reinforcement learning framework. By designing a dynamic adaptive reward function, it guides the system to continuously optimize scheduling strategies under complex tasks and time-varying networks. Furthermore, through a centralized training and distributed execution architecture, and an online transfer learning mechanism, it improves the generalization ability of the scheduling strategy. The overall method of this embodiment is based on a multi-agent reinforcement learning framework. Through the synergistic effect of a "centralized training, distributed execution" structure (i.e., CTDE structure), an adaptive reward adjustment mechanism, and an online transfer learning mechanism, it forms a self-perceiving and self-feedback intelligent scheduling system. This enables the system to achieve efficient, stable, and adaptive resource scheduling in a complex and dynamic integrated space-ground environment, realizing global collaboration and dynamic adaptation at the system level.

[0114] The technical solution of this embodiment specifically includes the following contents:

[0115] (1) The architecture of centralized training and decentralized execution (CTDE framework).

[0116] In integrated space-ground network resource scheduling scenarios, satellites, ground stations, and edge nodes have strongly coupled communication and task collaboration relationships, resulting in a large and time-varying system state space. Traditional centralized scheduling structures rely on a central node to collect state information and generate decisions. While this can achieve global optimization, it leads to computational bottlenecks and communication delays as the number of nodes increases, severely degrading scheduling real-time performance. To address this issue, this embodiment proposes a CTDE architecture that unifies global collaborative decision-making with distributed independent scheduling. Figure 2 As shown, the CTDE architecture in this embodiment includes two layers: an upper layer, the centralized training layer, where a central server or master ground station is responsible for training the global policy and optimizing parameters; and a lower layer, the distributed execution layer, where each agent node independently executes decisions and performs local interactions. During the centralized training phase, each node uploads its state data to the centralized training module (i.e., the policy parameter prediction model) via a secure communication link to complete joint optimization. During the distributed execution phase, nodes use the downloaded policy parameters to quickly make decisions locally, achieving self-organizing scheduling and autonomous operation of the system.

[0117] In this system, the system consists of It consists of several intelligent agents, each of which can correspond to a satellite, ground station, or edge computing node, and each has a locally observable state. ,action and local strategies The joint state and joint action of the system are defined as follows:

[0118] ;

[0119] ;

[0120] During the intensive training phase, the system adopts a centralized training mode to uniformly construct the joint value function. By integrating the state and behavioral information of each node, a global strategy that can be collaboratively optimized is learned. For each agent, this embodiment adopts a monotonically decomposable function structure:

[0121] ;

[0122] in, Represents a hybrid network The parameters, Indicates the first Local value function of an agent Indicates the first Local observable states of an agent Indicates the first The actions of an intelligent agent This represents a learnable hybrid network that combines local values ​​into a global value. It should be noted that the hybrid network in this embodiment can be a network known to those skilled in the art, or it can be a hybrid network including a value network and a policy network (i.e., a policy parameter prediction model).

[0123] Unlike the existing QMIX, this embodiment introduces a hybrid parameter structure based on reward signal weighting into the decomposition function, enabling the value decomposition process to dynamically adjust according to the network state, thereby improving the consistency of global value assessment. Satisfying monotonicity constraints ,in, Indicates the first The local value of an intelligent agent Representing the global value, the monotonicity constraint ensures that improving the value of each agent contributes to the overall increase in revenue. In this way, the system can jointly optimize the global value during the intensive training phase while maintaining structural decomposability.

[0124] During intensive training, a joint temporal difference update criterion is employed to minimize the global error:

[0125] ;

[0126] in, Represents the global loss function. This indicates the expectation for the sample batch. Indicates the joint state at the next moment. Indicates the joint action to be taken in the next moment. Indicates global instant rewards. Indicates the discount factor. This represents the target network parameters. The optimization objective is to balance short-term task gains with long-term resource utilization during the network's learning process.

[0127] During the distributed execution phase, the system switches to a distributed autonomous mode. Each node no longer depends on a central controller, but instead operates based on its locally observable state. Optimal action for independent decision-making:

[0128] ;

[0129] This enables real-time task allocation and link scheduling without requiring global synchronization. Nodes can maintain policy consistency through lightweight message exchange, while reducing network bandwidth consumption.

[0130] This "centralized training-decentralized execution" mechanism achieves global optimality in the centralized training phase and distributed autonomy in the decentralized execution phase. On the one hand, centralized training ensures that the global collaborative relationships between multiple nodes are modeled, avoiding local optima in individual work; on the other hand, the decentralized execution mechanism significantly reduces system latency and communication overhead during operation, exhibiting good scalability and robustness. When the network topology changes dynamically or some nodes become disconnected, other nodes can still independently complete resource scheduling under local observation, ensuring the continuity and stability of system tasks.

[0131] (2) Multi-agent collaborative reinforcement learning mechanism.

[0132] In integrated space-ground networks, the task relationships between satellites, ground stations, and edge nodes exhibit significant spatial coupling and resource competition. Single-agent reinforcement learning methods often update policies solely based on individual payoffs, failing to achieve globally optimal resource coordination and task scheduling. Therefore, this embodiment proposes a multi-agent cooperative reinforcement learning mechanism (MARL). Through joint value modeling and local information interaction among agents, it enables collaborative decision-making and resource sharing among nodes, improving the overall scheduling performance and stability of the system. In this mechanism, the integrated space-ground network is abstracted as a multi-agent Markov game process, consisting of a set of agents. State space Action space set State transition function and the set of reward functions Composed of. Each intelligent agent At any moment Observing its own local state Select Action The environment transitions to the next global state based on the joint actions of all agents. and return reward signals to each agent. During training and execution, each agent makes sampling decisions based on the action probability distribution output by the policy network, thereby preserving necessary exploratory behavior while maintaining optimal action tendencies. The core of this embodiment lies in achieving collaborative optimization of multi-agent policies through a centralized training and distributed execution (CTDE) structure. This allows each agent to share global information during the training phase and make decisions based solely on local observations during the execution phase, reducing communication overhead and latency.

[0133] During the intensive training phase, to ensure cooperation and global optimality among multiple agents, this embodiment employs a joint value decomposition approach. The system constructs a global value function. (From each intelligent agent locally) The network output is generated through hybrid network parameterization to achieve a collaborative fusion of local and global values, in conjunction with centralized computation within the CTDE framework. (These agents, though sharing the same goal of global optimization, follow different computational paths) to characterize the overall benefit of the multi-agent strategy, and decompose it into the local value functions of each agent. Taking Value Decomposition Network (VDN) as an example, the global value function can be expressed in a linearly additive form:

[0134] ;

[0135] in, Indicates the first Local policy parameters of an agent Indicates a joint state. Indicates joint action, This represents the global policy parameters. For more complex collaborative scenarios, this embodiment can further employ a nonlinear additive structure (such as the QMIX network) to achieve nonlinear combination through a learnable hybrid network, thereby more flexibly modeling the dependencies between agents. This hybrid network is constrained by the hypernetwork parameterization to ensure... This refers to the monotonicity of the global value function with respect to each local value function, thereby ensuring that improvements in individual strategies can promote the improvement of global returns.

[0136] During centralized training, each agent updates its gradient based on the global joint experience pool. The loss function (i.e., the first loss function) is defined as follows:

[0137] ;

[0138] in, This indicates that the intelligent agent is performing a joint action. The immediate global reward returned afterward (calculated through a reward function, i.e.) equal (Results obtained from reward function calculation) Indicates the environment receiving joint action The joint state at the next moment after the transition. Indicates the system in the next state The following joint actions are selected based on the current strategy. Indicates the empirical sample Expectation operation, For the target network parameters, The discount factor is used. This design achieves unified optimization of the global optimum, enabling multi-agent systems to learn cooperative and stable scheduling strategies during the training phase.

[0139] During the distributed execution phase, this embodiment allows each agent to base its actions on only local observations. and trained local policies Independent decision-making is achieved, enabling real-time, low-latency task scheduling and response. Simultaneously, to mitigate the performance loss caused by information asymmetry in distributed systems, this embodiment introduces a lightweight communication mechanism, allowing adjacent nodes to communicate via message encoding vectors. To conduct necessary information exchange, among which, Indicates the first The hidden state vector of each agent reflects the historical information retained by the agent in the previous time step. Message coding network A sequence structure based on gated cyclic units (GRUs) can be used to dynamically fuse the state information of neighboring nodes, thereby achieving coordinated adjustment at the policy level without significantly increasing the communication burden.

[0140] Furthermore, considering the large number of nodes and the changing communication links in the integrated space-ground environment, to prevent instability during system training, this embodiment introduces a time-weighted factor and a joint policy regularization term in policy updates. The following optimization objective function suppresses training oscillations caused by excessive differences between policies, ensuring the system's convergence robustness in dynamic environments. Specifically:

[0141] ;

[0142] in, Indicates information about global policy parameters The objective function (i.e., the second loss function). Indicates the trajectory Expected value calculation, trajectory The sampling distribution follows the policy parameter as joint strategy The generated distribution, Indicates the discount factor. Indicates at time step The immediate global reward (calculated through a reward function). This represents the weight of the joint policy regularization term, used to control the strength of the policy difference penalty term's influence on the overall objective. Indicates the first and the The square norm of the difference between the policies of two agents is used to quantify the degree of deviation between the two policy functions in terms of their parameters. The objective function is used to train the policy network to obtain the final local policy. It is used to make independent decisions during the decentralized execution phase.

[0143] (3) Adaptive reward mechanism design.

[0144] In integrated space-ground network scheduling tasks, task priorities, link status, and resource scarcity all change continuously over time. If a fixed reward function is still used, the optimization objectives of the reinforcement learning agent will not be able to reflect environmental changes in a timely manner, leading to a mismatch between policy learning direction and system performance, resulting in problems such as slow response and uneven energy consumption. To solve this problem, this embodiment introduces an Adaptive Reward Mechanism (ARM) into the multi-agent reinforcement learning framework. By dynamically sensing network status and task characteristics, the reward function weights are adjusted in real time, thereby achieving a balance between latency, task completion rate, and energy consumption in multi-objective optimization. The core idea of ​​this mechanism is to establish a state-driven dynamic reward function, forming a closed-loop mapping relationship between system state characteristics, performance feedback, and reward weights, enabling the agent to automatically correct its optimization direction under different operating conditions, thereby achieving dynamic policy adaptation and global performance improvement. Figure 3 As shown, the adaptive reward mechanism forms a self-regulating loop through four functional modules: "state awareness, weight generation, reward fusion, and performance feedback," which can maintain a stable direction of scheduling strategy optimization under different task scenarios.

[0145] The reward function in this embodiment applies to each decision cycle. (i.e., a time step) is defined as:

[0146] ;

[0147] in, This indicates a reward based on the task completion rate. This indicates a delay penalty. This indicates an energy consumption penalty item. This indicates a reward for the balanced distribution of resources. , , and This represents the dynamic weights of the four reward items. Unlike traditional fixed-weight designs, the weight parameters in this embodiment are not preset manually, but are automatically generated by the adaptive control module based on environmental conditions and system performance feedback.

[0148] At each time step, the system extracts state feature vectors from the agent's local observations. ,in, Indicates the length of the task queue. Indicates the channel bandwidth status. Indicates the node's computational load. This indicates the remaining energy percentage. Simultaneously, the system calculates and summarizes the global performance feedback vector from the previous cycle based on the operational status of each node. Record the average task throughput of the system in the previous period. Average end-to-end task latency and average energy consumption per unit task The adaptive weight control module takes this as input and processes it via a parameterized function. Output weight vector:

[0149] ;

[0150] in, For lightweight neural networks or attention mapping functions, their parameters It was learned through backpropagation during the intensive training phase. The operation ensures that the sum of the four weights is 1, thereby automatically balancing between different optimization objectives.

[0151] When the system is experiencing a surge in tasks or network congestion, the model automatically increases the weight of latency-related terms by sensing the backlog of tasks and bandwidth utilization. Enhance the rapid response of scheduling strategies; when system energy consumption approaches the threshold, Increase the energy consumption, and prioritize optimizing energy efficiency (i.e., energy consumption); when node resources are unevenly utilized. Increasing the priority guides the learning process towards a more balanced distribution; conversely, when the system detects the arrival of high-priority tasks or a significant decrease in task completion rate, it may intervene. The weight of each factor increases accordingly, and the system prioritizes task success rate and completion efficiency as its primary optimization objectives, scheduling critical business links and high-priority task nodes. Through this dynamic feedback mechanism, the direction of the reward function always remains consistent with the system's operational goals, enabling adaptive adjustment of the scheduling strategy in complex environments.

[0152] During intensive training, the agent performs policy gradient updates based on the aforementioned dynamic reward signals:

[0153] ;

[0154] in, Indicates policy parameters The gradient is used to guide the direction of parameter updates in the agent's policy network. Indicates the time step status and actions Mathematical expectation of the joint distribution, Represented as the gradient operator, it computes the function with respect to parameters. The partial derivatives, For the policy function, This is a state value estimate (obtained through value network estimation). Because... The dynamic changes directly reflect the state of the environment and system, allowing the model to continuously adjust the policy direction and avoiding the problem of learning objective lag in traditional fixed-reward mechanisms. This policy gradient update... Used to guide the objective function Perform gradient updates.

[0155] During system deployment, each agent stores a simplified version of the reward adjustment network locally, relying solely on local observation data for lightweight inference to quickly calculate dynamic weights, thereby achieving adaptive optimization without increasing communication load. When tasks arrive at the distribution or link conditions change, the agent can quickly adjust its strategy without retraining, enabling the system to maintain stable scheduling performance in dynamic environments.

[0156] The core innovation of this mechanism lies in extending the reward function from a static constant mapping to a dynamic function of the environment state, realizing target adjustment and environment adaptation. It breaks through the limitations of traditional reinforcement learning scheduling algorithms in dynamic network scenarios, such as rigid rewards and a single learning direction. In actual deployment, it can be implemented through a lightweight inference module of the edge node, and has an engineering-friendly implementation path.

[0157] (4) Online migration and continuous learning mechanism.

[0158] During the long-term operation of the integrated space-ground network, the system environment exhibits significant non-stationarity due to the periodic drift of satellite orbits, channel fading, and dynamic fluctuations in mission arrival rates. If statically trained reinforcement learning models continue to be used, their policies will gradually degrade as the environment distribution changes, leading to resource allocation imbalances, increased mission latency, and even system instability. Therefore, this embodiment introduces an Online Transfer and Continual Learning (OTCL) mechanism into the multi-agent reinforcement learning framework to achieve real-time response to environmental changes and adaptive policy updates. The core idea of ​​this mechanism is that when the system detects significant changes in the network environment or mission distribution, it can perform lightweight transfer corrections to the policy model while preserving historical optimal decision-making patterns, thereby maintaining long-term stability of scheduling performance without relying on centralized retraining.

[0159] like Figure 4 As shown, OTCL mainly consists of four core components: an environment change detection module, a migration update module, an experience memory buffer, and a stability constraint module, forming an adaptive policy update closed loop. The system first collects environmental states from each agent, including task load, link bandwidth, and node energy consumption, and then uses the environment change detection module to determine if the current state distribution has changed significantly. When a change is detected, the system automatically triggers the migration update process. During the migration update phase, the system calls the differential migration update module to perform lightweight local fine-tuning of the existing model parameters. It also combines real-time collected data with historical high-value samples stored in the experience memory buffer to form a mixed training sample set for rapid policy updates, maintaining the model's adaptability and stability in the new environment. Subsequently, the stability constraint module smoothly corrects the updated policy to prevent decision oscillations or over-adjustment caused by frequent changes. Finally, the policy parameters are corrected by stability constraints. The policy output module publishes the policy to each agent and simultaneously feeds back the state-action-reward data generated during the interaction process to the environment input module to complete the closed-loop update, ensuring that the parameters of the policy output module always match the dynamic environment.

[0160] In each scheduling cycle (i.e., time step) Each agent obtains the environmental state vector from its local perception module. ,in, Indicates the length of the task queue. Indicates the channel bandwidth status. Indicates the node's computational load. This indicates the remaining energy percentage. The system also maintains a moving average state. By calculating the state difference To sense changes in the environment. When the difference exceeds a threshold... hour, This threshold indicates a significant shift in the environmental distribution, triggering the online migration phase. This ensures that the model adjusts parameters only when a genuine environmental change is detected, avoiding oscillations caused by frequent updates.

[0161] During the transfer phase, the model does not undergo full retraining. Instead, it uses a differential parameter update strategy to fine-tune the affected local weights. Let the current policy parameter set of the agent be... Its update rule is defined as follows:

[0162] ;

[0163] ;

[0164] in, This represents the set of policy parameters after the migration update. For dynamic learning rate, The loss function is calculated based on the current reward feedback. Represents the loss function Regarding the current strategy parameters gradient, The stability constraint coefficient is... These are the baseline policy parameters from the previous convergence phase. The action value function represents the action of the intelligent agent. As a discount factor, Derived from the adaptive reward module, This indicates that the samples in the empirical sample set (i.e., the samples in the empirical memory buffer) are... Perform expectation calculation. Indicates that the agent performs an action. The environment then shifts to the observation state at the next moment. Indicates the next time step In the observed values Any of the following candidate actions can be taken. This represents the set of policy parameters for the target network. This indicates that the agent is at time step The observation status, This indicates that the agent is at time step The chosen action This represents the agent's policy network parameters at the current moment. This formula ensures that the parameter update magnitude is limited to a stable range, thus avoiding scheduling anomalies caused by large policy fluctuations. Through this "differential incremental transfer," the model can quickly correct adaptation biases to new environments with extremely low computational overhead.

[0165] To prevent "catastrophic forgetting" during multiple migrations and updates, this embodiment designs a small experience memory buffer. Used to preserve several key historical samples This represents past efficient policy behaviors. During each online update, the system retrieves data from the current interaction dataset. With memory buffer Simultaneous sampling is performed in a hybrid manner for incremental training, and its loss function (i.e., the third loss function) is:

[0166] ;

[0167] in, This represents the objective function loss during incremental training. Indicates from memory buffer and the current interactive dataset And the expectation of the concentrated sampling of experience samples, This represents the instantaneous global reward signal corresponding to the current sample (also calculated using the reward function). Indicates the observation state at the next moment. This indicates the current observation state. Indicates the next observation state Any candidate action that the agent can take. Indicates the action at the current moment. As a discount factor, This represents the current set of policy parameters. This mechanism, through joint optimization of new and old samples, enables the model to adapt to new environmental characteristics while retaining the original task scheduling patterns, thus maintaining long-term performance stability in dynamic environments.

[0168] During continuous updates, frequent parameter adjustments may cause policy oscillations or even learning divergence. Therefore, this embodiment introduces a stability constraint term into the loss function to limit the magnitude of policy parameter changes between adjacent updates:

[0169] ;

[0170] By measuring the parameter changes in adjacent update steps, stability constraints are added to the overall objective function:

[0171] ;

[0172] in, For stability weighting coefficients, This represents the square of the L2 norm. It is used to adjust the trade-off between the main task loss and the stability constraint. When the system detects a change in the environmental distribution, the policy transfer module triggers a new gradient update. At this time, the stability constraint is added as a penalty factor to the backpropagation process, dynamically limiting the parameter step size, thereby preventing the model from falling into an unstable oscillation range. After being constrained by this mechanism, the policy parameter update path exhibits smooth convergence characteristics, ensuring that in complex, time-varying network environments, scheduling performance maintains a gradual improvement rather than drastic fluctuations over continuous time periods.

[0173] The online transfer and continuous learning mechanism constructs a self-healing learning loop through four stages: "drift detection—differential transfer—memory retention—stability constraints." This enables the multi-agent scheduling system to continuously optimize and operate in a steady state when faced with changes in the topology of the integrated space-ground network, link fluctuations, and dynamic task arrivals. Compared with traditional static reinforcement learning, this mechanism significantly improves the generalization and lifecycle robustness of the scheduling strategy, ensuring the system's adaptive evolutionary capability during long-term deployment.

[0174] In summary, this embodiment achieves hierarchical collaboration between centralized training and distributed execution through the CTDE framework, realizes global optimization of cross-node resources through multi-agent collaborative reinforcement learning, dynamically guides the optimization direction of the strategy with an adaptive reward mechanism, and combines online transfer and continuous learning mechanisms to achieve adaptive model updates when the environment changes, thereby constructing an intelligent scheduling system with continuous learning and dynamic evolution capabilities. Compared with existing independent modular reinforcement learning methods, this embodiment achieves organic synergy between training, reward, and transfer at the system level, enabling the scheduling process to maintain high efficiency, stability, and long-term optimization performance even in complex and non-stationary integrated space-ground network environments.

[0175] The specific steps of the resource scheduling method in this embodiment are as follows:

[0176] Step 1: Environmental Data Acquisition and Initialization. Collect satellite operating parameters to form a system status dataset. Input the data into the simulation environment module to complete network topology loading and node registration, initialize the system operating environment, and provide status input for the training process.

[0177] Step 2, Model and Parameter Initialization. Allocate independent training modules containing policy networks and value networks to each node agent. The policy network generates action decisions based on local observable states, used for behavior selection during the execution phase. The value network evaluates global value based on joint state-action pairs, used to calculate loss and guide policy network updates during the intensive training phase. Both the policy network and value network are trained during the intensive training phase. Set hyperparameters such as learning rate, discount factor, and target update frequency. Establish the intensive training module and configure the joint optimizer, initializing all model parameters. Output the initial training state.

[0178] Step 3: Interactive Execution and Data Recording. The system initiates an interactive loop. The agent samples and selects actions based on the action probability distribution output by the current policy network, including operations such as resource allocation and task migration. The environment returns the next state and immediate reward based on the action. The state, action, reward, and next state are stored in the experience pool as a four-tuple for subsequent training.

[0179] Step four: Centralized training and policy update. Training samples are randomly drawn from the experience pool and input into the training module in batches. The global value function is optimized using the joint value decomposition algorithm. Gradient descent is used to update the parameters. The updated policy network is obtained, and a new set of policy parameters is output after training. .

[0180] Step 5: Strategy Distribution and Decentralized Execution. The strategy parameters obtained in Step 4 are distributed to each satellite and ground node. Each node performs strategy inference in real time based on local observations, independently calculates scheduling actions, and implements resource allocation operations to form a scheduling decision. Execution data is simultaneously fed back to the training module for performance evaluation.

[0181] Step Six: Adaptive Reward Calculation. After each round of scheduling in Step Five, the immediate reward value is calculated based on the acquired task completion rate, average latency, energy consumption, and resource allocation balance. The system dynamically adjusts the weights of each indicator according to the real-time load, enabling the reward function to adapt to changes in network status. The weighted reward result is output as the input for the next round of training.

[0182] Step 7, Online Migration and Update. When a change in status such as task queue length or channel bandwidth is detected, the system triggers the online migration mechanism. The strategy parameter set obtained in Step 4 is then updated. As the initial policy parameter during the migration phase (i.e., the current policy parameter during the migration phase) The newly acquired data is used to perform local retraining, updating only some key layer parameters to achieve rapid environmental adaptation. The updated parameters are then synchronized to all nodes.

[0183] Step 8, Iterative Looping and Convergence Deployment. Repeat steps 3 through 7 until the overall objective function is achieved. The convergence reaches the set threshold. The final stable set of policy parameters (i.e., the target policy parameter set) is output and deployed to the actual integrated space-ground network system to achieve online optimization and continuous updating of resource scheduling.

[0184] To better illustrate this, this embodiment also provides an embodiment tailored to a specific application scenario:

[0185] In this embodiment, a multi-agent cooperative reinforcement learning framework and adaptive reward mechanism, combined with the CTDE architecture, are used to optimize task scheduling and resource allocation in an integrated space-ground network. This embodiment uses a small integrated space-ground network consisting of six low-Earth orbit satellites, two ground base stations, and three edge servers as a practical application scenario. The specific implementation steps are as follows:

[0186] Step 1: Collect operational parameters such as satellite link signal-to-noise ratio, node bandwidth, and remaining node power consumption to form a system state dataset. Input the data into the simulation environment module for standardization and smoothing, load the network topology and node registration information, and initialize the task queue and system state matrix. This generates initial state input, providing the environmental foundation for subsequent training.

[0187] Step 2, Model and Parameter Initialization. Establish independent policy and value networks for each node agent. The policy network uses a two-layer fully connected structure, and the value network uses a two-layer fully connected or lightweight convolutional structure. Set hyperparameters such as learning rate, discount factor, target update frequency, and experience pool size. The centralized training module uses the QMIX framework and configures the Adam optimizer. All model parameters are initialized to... Generate and save the initial training state.

[0188] Step 3: Interactive Execution and Data Recording. The interactive loop is initiated. Each agent samples and selects actions based on the action probability distribution output by the current policy network, including resource allocation and link selection. The environment module returns the next state and immediate reward based on the execution results, storing the state, action, reward, and next state as a four-tuple. Store in the experience pool.

[0189] Step 4: Centralized Training and Policy Update. Samples are randomly selected from the experience pool and input into the centralized training module (i.e., the policy parameter prediction model). Using a joint value decomposition algorithm based on satellite nodes, the QMIX network structure is employed in the master control ground station training module to perform nonlinear combination optimization of the local Q-values ​​of each node, generating cross-node collaborative scheduling policy parameters. Gradient descent is used to minimize temporal difference errors, and the target network parameters are periodically synchronized. Training terminates when the global average reward stabilizes within the sliding window or the training steps reach their maximum limit. After training, a new set of policy parameters is output. And performance records.

[0190] Step 5, Strategy Distribution and Decentralized Execution. The strategy parameters obtained in Step 4... The data is distributed to each satellite and ground node. During the distributed execution phase, each node makes real-time inferences based on local observation inputs, independently decides, and implements task scheduling and resource allocation operations. The scheduling execution results (including task completion rate, average latency, and energy consumption) are fed back to the training module (i.e., the policy parameter prediction model) for performance evaluation.

[0191] Step Six: Adaptive Reward Calculation. After each round of scheduling, the system calculates an immediate reward based on task completion rate, average latency, energy consumption, and resource allocation balance. The reward function uses a weighted combination approach, with weights dynamically adjusted according to network load and task priority. For example, if the proportion of high-priority tasks increases, the weight of task completion rate is increased; if link congestion increases, the latency penalty term is strengthened. The adjusted reward value serves as input feedback for the next round of training.

[0192] Step 7, Online Migration and Update. When changes in status parameters such as task queue length or channel bandwidth exceed a threshold, the system automatically triggers the online migration mechanism. The strategy parameter set obtained in Step 4 is then updated. As the initial policy parameter during the migration phase (i.e., the current policy parameter during the migration phase) Based on the newly acquired data, local retraining is performed, updating only the output layer parameters of the Actor network to reduce computational complexity. After transfer training, changes in rewards are detected in the validation environment. If performance improves, the new parameters are synchronized to all nodes; otherwise, the original strategy is retained.

[0193] Step 8, Iterative Looping and Convergent Deployment. Repeat steps 3 through 7 until the comprehensive loss function converges to the set threshold. After satisfying the convergence conditions of the average return variance and parameter change norm within the sliding window, output the final set of strategy parameters and deploy it to the integrated space-ground system, entering the stable operation and lightweight online update phase, achieving adaptive optimization of resource scheduling.

[0194] Compared with the prior art, the technical solution of this embodiment has the following advantages:

[0195] 1. This embodiment makes structural improvements to the existing CTDE framework by introducing a dynamic value decomposition mechanism based on reward signal weighting during the centralized training phase. This enables agents to achieve global collaborative optimization while maintaining local autonomy. Unlike traditional CTDE, which only shares parameters during the centralized phase and makes independent decisions during the execution phase, the technical solution in this embodiment can dynamically adjust the joint value relationship between agents according to the task load and network state, thereby significantly improving the collaborative efficiency and decision-making stability of multiple nodes in the integrated space-ground network.

[0196] 2. Existing reinforcement learning methods generally employ static or empirically set reward functions, which cannot adapt to the heterogeneous nodes, dynamic tasks, and time-varying channels characteristic of integrated space-ground networks. This embodiment applies a dynamic reward weight adjustment mechanism based on state awareness and performance feedback. Utilizing state characteristics such as node task queue length and energy consumption levels, and combining global feedback signals such as system task completion rate and average latency, a lightweight neural network generates dynamic weights for four reward items: task completion rate, average latency, energy consumption, and resource balance in real time. This mechanism enables the reinforcement learning model to adaptively adjust its optimization objectives based on the real-time operating status of the integrated space-ground network, improving the flexibility, long-term efficiency, and adaptability of the scheduling strategy in complex environments (i.e., heterogeneous and dynamic integrated space-ground network environments).

[0197] 3. Existing reinforcement learning models typically require retraining when the environment changes, resulting in slow adaptation speed and high computational cost. This embodiment proposes a differential online transfer learning mechanism. When a change in task distribution or channel state is detected, only the affected key parameters in the model are updated using lightweight differential methods. A stability constraint term is introduced to limit the magnitude of parameter changes between adjacent updates. Simultaneously, historical high-value samples are used for hybrid optimization to prevent policy oscillation and forgetting, achieving rapid policy transfer and smooth convergence in the face of environmental changes. This mechanism significantly reduces computational overhead while achieving rapid model transfer and long-term stability in dynamic environments.

[0198] Reference Figure 5 This application also provides an integrated space-ground resource scheduling system, which includes an intelligent agent construction unit 501, a global reward determination unit 502, a first construction unit 503, a model training unit 504, a second construction unit 505, a third construction unit 506, a cyclic training unit 507, and a resource scheduling unit 508, wherein:

[0199] The intelligent agent construction unit 501 is used to assign each satellite, each ground station and each edge node in the space-ground integrated network to an intelligent agent, thereby obtaining multiple intelligent agents;

[0200] The global reward determination unit 502 is used to determine the real-time global reward based on the task completion rate, average latency, energy consumption and resource allocation balance corresponding to the current time step. The real-time global reward is updated after multiple agents perform resource scheduling operations according to the new set of policy parameters.

[0201] The first building unit 503 is used to assign a policy parameter prediction model containing a policy network and a value network to each agent, and to build a first loss function for the value network and a second loss function for the policy network based on the immediate global reward.

[0202] The model training unit 504 is used to train the policy parameter prediction model of all agents by randomly sampling the training sample set from the experience pool based on the first loss function and the second loss function, until the training is completed and a new set of policy parameters is output.

[0203] The second building unit 505 is used to take the new policy parameter set as the current policy parameter set in the transfer learning stage, to calculate the policy parameter set after transfer and to build the third loss function in the transfer learning stage. The policy parameter set after transfer is used for training the policy parameter prediction model in the next round.

[0204] The third construction unit 506 is used to construct a stability constraint term based on the updated policy parameter set and the current policy parameter set, and add the stability constraint term to the third loss function to obtain the total objective function;

[0205] The loop training unit 507 is used to loop train the policy parameter prediction model and the transfer learning stage until the total objective function converges to a preset threshold, and the target policy parameter set output by the transfer learning stage is obtained.

[0206] Resource scheduling unit 508 is used to deploy the target policy parameter set to the integrated space-ground network so as to perform integrated space-ground resource scheduling according to the target policy parameter set.

[0207] It should be noted that since the integrated space-ground resource scheduling system in this embodiment is based on the same inventive concept as the integrated space-ground resource scheduling method described above, the corresponding content in the method embodiment is also applicable to this system embodiment, and will not be described in detail here.

[0208] Reference Figure 6 This application also provides an electronic device, which includes:

[0209] At least one memory;

[0210] At least one processor;

[0211] At least one program;

[0212] The program is stored in memory, and the processor executes at least one program to implement the above-described integrated space-ground resource scheduling method of this disclosure.

[0213] This electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0214] The electronic devices according to embodiments of this application will now be described in detail.

[0215] The processor 1600 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.

[0216] The memory 1700 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1700 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called and executed by the processor 1600 to implement the integrated space-ground resource scheduling method of the embodiments of this disclosure.

[0217] The input / output interface 1800 is used to implement information input and output.

[0218] The communication interface 1900 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0219] Bus 2000 transmits information between various components of the device (e.g., processor 1600, memory 1700, input / output interface 1800, and communication interface 1900);

[0220] The processor 1600, memory 1700, input / output interface 1800 and communication interface 1900 are connected to each other within the device via bus 2000.

[0221] This disclosure also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the above-described integrated space-ground resource scheduling method.

[0222] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0223] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.

[0224] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0225] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0226] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0227] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0228] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0229] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0230] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0231] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0232] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. The embodiments of this application have been described in detail above with reference to the accompanying drawings, but this application is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of this application.

[0233] The embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of this application.

Claims

1. A space-ground integrated resource scheduling method, characterized in that, The method comprises: Each satellite, each ground station and each edge node in the integrated network corresponds to an agent, obtaining a plurality of agents; According to the task completion rate, average time delay, energy consumption and resource allocation balance degree corresponding to the current time step, determine the instant global reward, the instant global reward is updated after the plurality of agents execute resource scheduling operation according to the new strategy parameter set; Assign a strategy parameter prediction model comprising a policy network and a value network to each agent, and based on the instant global reward, construct a first loss function of the value network and a second loss function of the policy network; Based on the first loss function and the second loss function, randomly extract a training sample set from the experience pool to train the strategy parameter prediction model of all agents until the training is completed and the new strategy parameter set is output; The new strategy parameter set is used as the current strategy parameter set of the transfer learning stage to calculate the transfer updated strategy parameter set and construct the third loss function of the transfer learning stage, and the transfer updated strategy parameter set is used for the training of the next round of strategy parameter prediction model; According to the transfer updated strategy parameter set and the current strategy parameter set, construct a stability constraint term, and add the stability constraint term to the third loss function to obtain a total objective function; Cycle training the strategy parameter prediction model and the transfer learning stage until the total objective function converges to a preset threshold, obtaining the target strategy parameter set output by the transfer learning stage; Deploy the target strategy parameter set to the integrated network to execute integrated resource scheduling according to the target strategy parameter set.

2. The method of claim 1, wherein, According to the task completion rate, average time delay, energy consumption and resource allocation balance degree corresponding to the current time step, determine the instant global reward, comprising: According to the proportion of the number of successfully completed tasks to the total number of tasks in the current time step, the task completion rate is calculated; According to the difference between the start time and the completion time of the task in the current time step, the average time delay is calculated; According to the power of the node and the task execution time in the current time step, the energy consumption is calculated, the node is the satellite, ground station or edge node in the integrated network; The resource allocation balance degree is calculated by the standard deviation of the resource occupation rate of each node in the current time step; The task completion rate, the average time delay, the energy consumption and the resource allocation balance degree are weighted and summed to obtain the instant global reward.

3. The method of claim 1, wherein, Based on the instant global reward, the first loss function of the value network and the second loss function of the policy network are constructed, comprising: According to the value function of the value network, the local value function corresponding to each agent is calculated; The local value function corresponding to each agent is summed to obtain the global value function; According to the global value function and the instant global reward, the first loss function of the value network is constructed; According to the square norm of the difference between the strategies of each two agents and the instant global reward, the second loss function of the policy network is constructed.

4. The method of claim 1, wherein, The method further comprises: obtaining a state value estimation through the value network; constructing a policy gradient update function according to the value function of the policy network, the state value estimation, and the instant global reward; performing gradient update on the second loss function according to the policy gradient update function to train the policy parameter prediction model.

5. The method of claim 1, wherein, The method further comprises: calculating the transfer updated policy parameter set by the following method: ； ； in, This represents the set of policy parameters after the migration update. Indicates the dynamic learning rate. This represents the loss function calculated based on the current reward feedback. Represents the policy parameters with respect to the current time. gradient operator, This represents the stability constraint coefficient. This represents the baseline policy parameters from the previous convergence phase. This indicates that in the empirical sample set, for Perform expectation calculation. Indicates the current time Instant global rewards Indicates the discount factor. Indicates the next moment The observation status, Indicates the next moment Candidate actions, This represents the set of policy parameters for the target network. Indicates the current time The observation status, Indicates the current time The action, Take a positive integer.

6. The method of claim 1, wherein, The method further comprises: constructing the third loss function of the transfer learning stage by the following method: ； wherein, represents a third loss function, represents a sampling experience sample from an experience memory buffer and a current interaction dataset an expectation of, represents an immediate global reward corresponding to a current sample, represents a discount factor, represents an observation state at a next time, represents a candidate action at a next time, represents a current set of policy parameters, represents an observation state at a current time, represents an action at a current time.

7. The method of claim 1, wherein, The method further comprises: calculating the difference between all parameters in the transfer updated policy parameter set and all parameters in the current policy parameter set to obtain a difference result; performing L2 norm square solving on the difference result to construct a stability constraint term.

8. A space-ground integrated resource scheduling system, characterized in that, The system comprises: An agent construction unit configured to obtain a plurality of agents by respectively corresponding each satellite, each ground station, and each edge node in the integrated space-ground network to an agent; A global reward determination unit configured to determine an instant global reward according to a task completion rate, an average time delay, energy consumption, and resource allocation balance degree corresponding to a current time step, the instant global reward being updated after a resource scheduling operation is performed by the plurality of agents according to a new policy parameter set; A first construction unit configured to assign a policy parameter prediction model comprising a policy network and a value network to each agent, and construct a first loss function of the value network and a second loss function of the policy network based on the instant global reward; A model training unit configured to train the policy parameter prediction model of all agents by randomly extracting a training sample set from an experience pool based on the first loss function and the second loss function, until a new policy parameter set is output after training is completed; A second construction unit configured to take the new policy parameter set as a current policy parameter set of a transfer learning stage, to calculate a transfer updated policy parameter set and construct a third loss function of the transfer learning stage, the transfer updated policy parameter set being used for training of a policy parameter prediction model in a next round; A third construction unit configured to construct a stability constraint term according to the transfer updated policy parameter set and the current policy parameter set, and add the stability constraint term to the third loss function to obtain a total objective function; a cycle training unit, configured to cyclically train the policy parameter prediction model and the transfer learning stage until the total target function converges to a preset threshold, to obtain a target policy parameter set output by the transfer learning stage; a resource scheduling unit, configured to deploy the target policy parameter set to the integrated space-ground network, so as to perform integrated space-ground resource scheduling according to the target policy parameter set.

9. An electronic device, comprising: comprising at least one control processor and a memory connected in communication with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to perform the integrated space-ground resource scheduling method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions for causing a computer to perform the integrated space-ground resource scheduling method according to any one of claims 1 to 7.

Images