A method for dynamic collaborative optimization of off-site computing power based on multi-agent reinforcement learning

Abstract

The application discloses a kind of off-site computing power dynamic collaborative optimization methods based on multi-agent reinforcement learning, including the following steps: constructing resource topology diagram;Based on resource topology diagram, get node-level state feature vector and system-level state feature vector;Multi-agent environment is constructed;Local observation, global information and agent action set are input into improved CTDE model, output policy network parameters and value network parameters and construct training batch;Based on training batch, get converged policy network parameters and converged value network parameters;Get execution result;Obtain dynamically updated policy network parameters and dynamically updated value network parameters, realize the dynamic collaborative optimization of task acceptance, resource allocation, task migration, replica start-stop and bandwidth matching.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and distributed computing, and in particular to a method for dynamic collaborative optimization of remote computing power based on multi-agent reinforcement learning. Background Technology

[0002] With the rapid development of cloud computing, edge computing, and high-performance computing, the demand for computing resources from various social applications has exploded. Especially in tasks such as big data analytics, artificial intelligence training, scientific computing, and video rendering, computing resources within a single region or data center often struggle to meet the demands of high concurrency, low latency, and high reliability. Therefore, cross-regional computing power collaborative optimization has become a research hotspot. This involves unified scheduling and management of multiple computing nodes, storage nodes, and network links distributed across different regions to achieve cross-domain collaboration and efficient utilization of computing resources.

[0003] In existing technologies, the following are the main approaches to addressing the problem of collaborative computing power across different locations: The first type is resource allocation methods based on centralized scheduling. These methods collect the operational status information of each node through a central controller and then make unified decisions and allocations. Their advantage is relative simplicity, but they suffer from a single point of failure; if the central controller fails, the entire system scheduling will fail. Furthermore, centralized methods struggle to handle large-scale dynamic tasks and complex network topologies, and their scalability is insufficient. The second type is optimization methods based on heuristics or rules, such as scheduling based on shortest paths, minimum task completion times, and load balancing. These methods are effective in small-scale and static scenarios, but they are prone to getting trapped in local optima due to a lack of adaptability to complex dynamic environments. The third type is computing power scheduling methods based on reinforcement learning, which have emerged in recent years. These methods can learn scheduling strategies through interaction with the environment, exhibiting a certain degree of adaptability and generalization. However, most existing reinforcement learning-based research remains within a single-agent framework, failing to fully consider the collaborative decision-making problem of multiple nodes in a distributed computing environment, and still has significant shortcomings in state modeling and training methods. Summary of the Invention

[0004] One objective of this invention is to propose a method for dynamic collaborative optimization of computing power across different locations based on multi-agent reinforcement learning. This invention makes full use of the multi-agent reinforcement learning framework and the improved CTDE model. Through resource topology modeling, node-level and system-level state feature extraction, agent environment construction, and the introduction of neighborhood aggregation and link feature splicing, it describes in detail the intelligent optimization process of dynamic task scheduling and resource allocation among computing power resources in multiple regions.

[0005] A method for dynamic collaborative optimization of remote computing power based on multi-agent reinforcement learning according to an embodiment of the present invention includes the following steps:

[0006] Resource topology modeling is performed on computing nodes, storage nodes, and network links distributed in different regions to form a resource topology map;

[0007] Based on the resource topology map, data is collected on the real-time operating status of computing nodes, storage nodes and network links. The collected data is preprocessed to obtain node-level state feature vectors and system-level state feature vectors.

[0008] The node-level state feature vector is defined as local observation information, the system-level state feature vector is defined as global information, and the action set of each agent is set to construct a multi-agent environment.

[0009] Based on the agent environment, local observation information, global information and agent action set are input into the improved CTDE model for centralized training and parallel environment interaction, outputting policy network parameters and value network parameters, and constructing training batches by using time window bucket sampling.

[0010] Iterative optimization of the policy network parameters and value network parameters is performed based on the training batches until the convergence criterion is met, resulting in converged policy network parameters and converged value network parameters.

[0011] The converged policy network parameters and converged value network parameters are deployed to each computing node and run in the distributed execution channel of the improved CTDE model to generate execution results;

[0012] The performance metrics are calculated based on the execution results and compared with a preset threshold. When the results are less than the preset threshold, incremental online training is triggered in the improved CTDE model to obtain dynamically updated policy network parameters and dynamically updated value network parameters.

[0013] Optionally, the formation of the resource topology map specifically includes:

[0014] Obtain computing nodes, storage nodes, and network links distributed across different regions, and establish sets of computing nodes, storage nodes, and network links;

[0015] For each computing node in the computing node set, the computing power, memory capacity, and power consumption parameters are collected to form a computing node attribute set. For each storage node in the storage node set, the storage capacity, read / write throughput, and power consumption parameters are collected to form a storage node attribute set.

[0016] For each network link in the network link set, bandwidth and round-trip time are collected to form a set of network link attributes;

[0017] Establish a mapping relationship between the node attribute set, the storage node attribute set, and the network link attribute set. The mapping relationship is to identify the connection relationship between computing nodes and storage nodes through the network link attribute set, and to associate the bandwidth and round-trip time of each network link with the corresponding node connection to form a correspondence.

[0018] Resource topology modeling is completed based on the set of computing nodes, the set of storage nodes, the set of network links, the set of node attributes, the set of storage nodes, and the set of network link attributes, forming a resource topology graph.

[0019] Optionally, obtaining the node-level state feature vector and the system-level state feature vector specifically includes:

[0020] At a preset time step, running status data is collected for each computing node in the computing node set in the resource topology graph. The running status data includes processor utilization, video memory usage, memory usage, task queue length, remaining task time, and data location information.

[0021] At a preset time step, the running status data of each storage node in the storage node set in the resource topology map is collected. The running status data includes storage capacity utilization, read / write throughput utilization and power consumption parameters.

[0022] At a preset time step, operational status data is collected for each network link in the network link set in the resource topology diagram. The operational status data includes available bandwidth and round-trip latency.

[0023] The running status data of the computing node set and the network link set are processed by missing value filling, noise removal and normalization to obtain the preprocessed running status sequence;

[0024] The preprocessed running state sequence is sliced ​​according to time windows to generate node-level state feature vectors and system-level state feature vectors. The node-level state feature vectors include the resource utilization and task execution status of a single computing node and a single storage node within the time window. The system-level state feature vectors include global task arrival rate, global task completion rate, global resource utilization rate, available network link bandwidth, network link round-trip time, system power consumption, and cross-regional communication overhead.

[0025] Optionally, the construction of the multi-agent environment specifically includes:

[0026] The node-level state feature vector is defined as local observation information. A corresponding agent is configured for each computing node in the computing node set. The definition process is to establish an input mapping relationship between the node-level state feature vector and the corresponding agent, and associate it with the corresponding agent.

[0027] The system-level state feature vector is defined as global information. The process of defining it is to establish a shared input mapping relationship for the system-level state feature vector and input it as global information to all agents for sharing.

[0028] For each agent in the agent set, an agent action set is set, which includes task acceptance, task assignment, task migration, computing power quota adjustment, replica start / stop, and bandwidth allocation;

[0029] A multi-agent environment is constructed based on local observation information, global information, and the set of agent actions.

[0030] Optionally, the construction of the training batch specifically includes:

[0031] The improved CTDE model is input into the local observation information, global information, and agent action set in a multi-agent environment for centralized training and parallel environment interaction. The improved CTDE model includes a state and observation processing module, a policy network module, a value network module, and a training and execution module. The state and observation processing module enhances the local and global information through neighborhood aggregation and link feature concatenation. The policy network module receives the local observation information and outputs the agent action distribution. The value network module receives the global information and the actions of all agents, evaluates the state-action pairs, and outputs a value estimate. The training and execution module optimizes the policy network parameters and value network parameters during the centralized training phase and deploys the optimized parameters for independent operation by the agents during the decentralized execution phase.

[0032] In the state and observation processing module, local observation information is input into the corresponding agent, and global information is input into all agents. An improved method of neighborhood aggregation and link feature splicing is introduced to enhance the local observation information and global information. The neighborhood aggregation and link feature splicing refers to weighting the running states of neighboring nodes using a mind-force mechanism and splicing the attributes of the links with the running state features of the aggregated neighboring nodes.

[0033] In the policy network module, enhanced local observation information is used as input to output the distribution of agent actions. The distribution of actions covers operations such as task acceptance, task assignment, task migration, computing power quota adjustment, replica start and stop, and bandwidth allocation.

[0034] In the value network module, the state-action pairs are evaluated for value using the enhanced global information and the actions of all agents as input, and the global value estimate is output.

[0035] During the intensive training phase of the training and execution module, local observation information, global information, agent action distribution, and global value estimation are jointly input into the training module, and policy network parameters and value network parameters are output.

[0036] Sample scheduling is performed on the policy network parameters and value network parameters. Training batches are constructed using time window binning sampling. Time window binning sampling refers to storing the data of policy network parameters and value network parameters in segments according to time order, with each segment serving as a sampling bucket. During training, samples are evenly drawn from buckets of different time periods.

[0037] Optionally, obtaining the converged policy network parameters and the converged value network parameters specifically includes:

[0038] Iterative optimization is performed on the policy network parameters and value network parameters based on training batches. The iterative optimization includes advantage estimation, policy parameter update, target network soft update and gradient pruning.

[0039] In the advantage estimation phase, the target value is calculated, which is equal to the reward plus a discount factor multiplied by the value estimate of the next time step, and the advantage value is calculated, which is equal to the target value minus the value estimate of the current time step.

[0040] During the policy network parameter update phase, a policy loss function is constructed based on the agent's action distribution and advantage value. An entropy regularization term is introduced to enhance exploration. The gradient of the policy loss function is used to update the policy network parameters, and gradient pruning is performed during the update process to control the update magnitude.

[0041] During the target network soft update phase, the value network parameters are updated by minimizing the mean square error between the global value estimate and the target value. The policy network parameters and value network parameters are then synchronized to the target network using a soft update method until the convergence criterion is met, resulting in converged policy network parameters and converged value network parameters.

[0042] Optionally, the generation of the execution result specifically includes:

[0043] The converged policy network parameters and converged value network parameters are deployed to each computing node in the computing node set and run in the distributed execution channel of the improved CTDE model;

[0044] At each dynamic runtime time step, the action vector is output using the converged policy network parameters.

[0045] The feasible region projection is performed on the output action vector to obtain actions that are constrained to be feasible. The feasible region projection is the process of correcting and mapping the action vector output by the agent according to system capacity constraints, time delay constraints, bandwidth constraints and power consumption constraints.

[0046] For actions that are feasible under constraints, conflict resolution is performed to obtain the final action instruction. The conflict resolution refers to the process of eliminating contradictions between actions of agents when multiple agents' actions all satisfy a single constraint but there is resource competition at the system level. This is achieved by prioritizing allocation, bandwidth sharing, and task replanning.

[0047] Dynamic collaborative optimization scheduling is completed based on the final action instruction, and execution results are generated. The dynamic collaborative optimization scheduling includes task acceptance, resource allocation, task migration, replica start-up and shutdown, and bandwidth allocation. The execution results include the actual task completion time, resource consumption, network overhead, migration events, and default events.

[0048] Optionally, obtaining the dynamically updated policy network parameters and the dynamically updated value network parameters specifically includes:

[0049] Performance metrics are calculated based on the execution results. The calculation process involves converting the actual task completion time, resource consumption, network overhead, migration events, and default events in the execution results into average completion time, resource utilization, bandwidth utilization, migration overhead, and default rate.

[0050] The performance metrics are compared with a preset threshold. When the performance metrics are less than the preset threshold, incremental online training is triggered in the improved CTDE model. The incremental online training refers to the improved CTDE model updating parameters gradually and incrementally based on the latest data collected in real time during the running phase.

[0051] During incremental online training, new local observation information, global information, and execution results are written into the replay buffer, and new training batches are constructed using time window bucket sampling.

[0052] Incremental optimization is performed on the policy network parameters and value network parameters based on the new training batches until the convergence criterion is met, generating dynamically updated policy network parameters and dynamically updated value network parameters.

[0053] The beneficial effects of this invention are:

[0054] This invention introduces a multi-agent reinforcement learning method to achieve dynamic collaborative optimization of computing nodes, storage nodes, and network links distributed across different regions. It effectively addresses the shortcomings of traditional centralized scheduling in complex, heterogeneous computing environments, such as insufficient scalability, poor adaptability of rule-based optimization methods, and the lack of global coordination capabilities in single-agent reinforcement learning. By comprehensively modeling the sets of computing nodes, storage nodes, and network links to form a resource topology map, this invention can more accurately reflect the global state and operational characteristics of computing resources. During state feature extraction, not only are node-level local states collected, but system-level global information is also integrated. This allows the agent to consider both local resource utilization and overall system performance during decision-making, thereby ensuring global optimality in resource scheduling and task execution.

[0055] Regarding the training mechanism, this invention employs an improved CTDE model, combining neighborhood aggregation and link feature concatenation methods to enhance local and global information, thereby improving state representation capabilities and the level of cooperation among multiple agents. Simultaneously, this invention utilizes a time-window bucket sampling mechanism to construct training batches, effectively improving sample utilization and accelerating policy convergence, thus addressing the slow convergence issue of existing reinforcement learning methods in large-scale distributed scenarios. Through this training method, each agent can output feasible actions that meet capacity, latency, bandwidth, and power consumption constraints during the execution phase, and obtain the final scheduling instruction after conflict resolution, thereby significantly reducing the uncertainty caused by resource contention and link conflicts.

[0056] Furthermore, by comparing performance indicators with preset thresholds during execution, this invention can trigger incremental online training based on environmental changes, ensuring continuous updates and dynamic optimization of policy network parameters and value network parameters, thus giving the entire system adaptability and robustness. Therefore, this invention not only improves the utilization rate and task completion rate of remote computing resources but also effectively reduces cross-regional communication overhead and system power consumption, significantly improving the flexibility and stability of computing power scheduling, ultimately achieving efficient collaboration and dynamic optimization of remote computing power, demonstrating promising application prospects and widespread application value. Attached Figure Description

[0057] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0058] Figure 1 This is an overall flowchart of a multi-agent reinforcement learning-based method for dynamic collaborative optimization of computing power in different locations, as proposed in this invention.

[0059] Figure 2This is a schematic diagram of the module structure of an improved CTDE model for a multi-agent reinforcement learning-based dynamic collaborative optimization method for remote computing power proposed in this invention. Detailed Implementation

[0060] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0061] refer to Figure 1-2 A method for dynamic collaborative optimization of remote computing power based on multi-agent reinforcement learning includes the following steps:

[0062] Resource topology modeling is performed on computing nodes, storage nodes, and network links distributed in different regions to form a resource topology map;

[0063] Based on the resource topology map, data is collected on the real-time operating status of computing nodes, storage nodes and network links. The collected data is preprocessed to obtain node-level state feature vectors and system-level state feature vectors.

[0064] The node-level state feature vector is defined as local observation information, the system-level state feature vector is defined as global information, and the action set of each agent is set to construct a multi-agent environment.

[0065] Based on the agent environment, local observation information, global information and agent action set are input into the improved CTDE model for centralized training and parallel environment interaction, outputting policy network parameters and value network parameters, and constructing training batches by using time window bucket sampling.

[0066] Iterative optimization of the policy network parameters and value network parameters is performed based on the training batches until the convergence criterion is met, resulting in converged policy network parameters and converged value network parameters.

[0067] The converged policy network parameters and converged value network parameters are deployed to each computing node and run in the distributed execution channel of the improved CTDE model to generate execution results;

[0068] The performance metrics are calculated based on the execution results and compared with a preset threshold. When the results are less than the preset threshold, incremental online training is triggered in the improved CTDE model to obtain dynamically updated policy network parameters and dynamically updated value network parameters.

[0069] In this embodiment, the formation of the resource topology map specifically includes:

[0070] Obtain computing nodes, storage nodes, and network links distributed across different regions, and establish sets of computing nodes, storage nodes, and network links;

[0071] For each computing node in the computing node set, the computing power, memory capacity, and power consumption parameters are collected to form a computing node attribute set. For each storage node in the storage node set, the storage capacity, read / write throughput, and power consumption parameters are collected to form a storage node attribute set.

[0072] For each network link in the network link set, bandwidth and round-trip time are collected to form a set of network link attributes;

[0073] Establish a mapping relationship between the node attribute set, the storage node attribute set, and the network link attribute set. The mapping relationship is to identify the connection relationship between computing nodes and storage nodes through the network link attribute set, and to associate the bandwidth and round-trip time of each network link with the corresponding node connection to form a correspondence.

[0074] Resource topology modeling is completed based on the set of computing nodes, the set of storage nodes, the set of network links, the set of node attributes, the set of storage nodes, and the set of network link attributes, forming a resource topology graph.

[0075] In this embodiment, obtaining the node-level state feature vector and the system-level state feature vector specifically includes:

[0076] At a preset time step, running status data is collected for each computing node in the computing node set in the resource topology graph. The running status data includes processor utilization, video memory usage, memory usage, task queue length, remaining task time, and data location information.

[0077] At a preset time step, the running status data of each storage node in the storage node set in the resource topology map is collected. The running status data includes storage capacity utilization, read / write throughput utilization and power consumption parameters.

[0078] At a preset time step, operational status data is collected for each network link in the network link set in the resource topology diagram. The operational status data includes available bandwidth and round-trip latency.

[0079] The running status data of the computing node set and the network link set are processed by missing value filling, noise removal and normalization to obtain the preprocessed running status sequence;

[0080] The preprocessed running state sequence is sliced ​​according to time windows to generate node-level state feature vectors and system-level state feature vectors. The node-level state feature vectors include the resource utilization and task execution status of a single computing node and a single storage node within the time window. The system-level state feature vectors include global task arrival rate, global task completion rate, global resource utilization rate, available network link bandwidth, network link round-trip time, system power consumption, and cross-regional communication overhead.

[0081] In this embodiment, the construction of the multi-agent environment specifically includes:

[0082] The node-level state feature vector is defined as local observation information. A corresponding agent is configured for each computing node in the computing node set. The definition process is to establish an input mapping relationship between the node-level state feature vector and the corresponding agent, and associate it with the corresponding agent.

[0083] The system-level state feature vector is defined as global information. The process of defining it is to establish a shared input mapping relationship for the system-level state feature vector and input it as global information to all agents for sharing.

[0084] For each agent in the agent set, an agent action set is set, which includes task acceptance, task assignment, task migration, computing power quota adjustment, replica start / stop, and bandwidth allocation;

[0085] A multi-agent environment is constructed based on local observation information, global information, and the set of agent actions.

[0086] In this embodiment, the construction of the training batch specifically includes:

[0087] The improved CTDE model is input into the local observation information, global information, and agent action set in a multi-agent environment for centralized training and parallel environment interaction. The improved CTDE model includes a state and observation processing module, a policy network module, a value network module, and a training and execution module. The state and observation processing module enhances the local and global information through neighborhood aggregation and link feature concatenation. The policy network module receives the local observation information and outputs the agent action distribution. The value network module receives the global information and the actions of all agents, evaluates the state-action pairs, and outputs a value estimate. The training and execution module optimizes the policy network parameters and value network parameters during the centralized training phase and deploys the optimized parameters for independent operation by the agents during the decentralized execution phase.

[0088] In the state and observation processing module, local observation information is input into the corresponding agent, and global information is input into all agents. An improved method of neighborhood aggregation and link feature splicing is introduced to enhance the local observation information and global information. The neighborhood aggregation and link feature splicing refers to weighting the running states of neighboring nodes using a mind-force mechanism and splicing the attributes of the links with the running state features of the aggregated neighboring nodes.

[0089] In the policy network module, enhanced local observation information is used as input to output the distribution of agent actions. The distribution of actions covers operations such as task acceptance, task assignment, task migration, computing power quota adjustment, replica start and stop, and bandwidth allocation.

[0090] In the value network module, the state-action pairs are evaluated for value using the enhanced global information and the actions of all agents as input, and the global value estimate is output.

[0091] During the intensive training phase of the training and execution module, local observation information, global information, agent action distribution, and global value estimation are jointly input into the training module, and policy network parameters and value network parameters are output.

[0092] Sample scheduling is performed on the policy network parameters and value network parameters. Training batches are constructed using time window binning sampling. Time window binning sampling refers to storing the data of policy network parameters and value network parameters in segments according to time order, with each segment serving as a sampling bucket. During training, samples are evenly drawn from buckets of different time periods.

[0093] In this embodiment, obtaining the converged policy network parameters and the converged value network parameters specifically includes:

[0094] Iterative optimization is performed on the policy network parameters and value network parameters based on training batches. The iterative optimization includes advantage estimation, policy parameter update, target network soft update and gradient pruning.

[0095] In the advantage estimation phase, the target value is calculated, which is equal to the reward plus a discount factor multiplied by the value estimate of the next time step, and the advantage value is calculated, which is equal to the target value minus the value estimate of the current time step.

[0096] During the policy network parameter update phase, a policy loss function is constructed based on the agent's action distribution and advantage value. An entropy regularization term is introduced to enhance exploration. The gradient of the policy loss function is used to update the policy network parameters, and gradient pruning is performed during the update process to control the update magnitude.

[0097] During the target network soft update phase, the value network parameters are updated by minimizing the mean square error between the global value estimate and the target value. The policy network parameters and value network parameters are then synchronized to the target network using a soft update method until the convergence criterion is met, resulting in converged policy network parameters and converged value network parameters.

[0098] In this embodiment, the generation of the execution result specifically includes:

[0099] The converged policy network parameters and converged value network parameters are deployed to each computing node in the computing node set and run in the distributed execution channel of the improved CTDE model;

[0100] At each dynamic runtime time step, the action vector is output using the converged policy network parameters.

[0101] The feasible region projection is performed on the output action vector to obtain actions that are constrained to be feasible. The feasible region projection is the process of correcting and mapping the action vector output by the agent according to system capacity constraints, time delay constraints, bandwidth constraints and power consumption constraints.

[0102] For actions that are feasible under constraints, conflict resolution is performed to obtain the final action instruction. The conflict resolution refers to the process of eliminating contradictions between actions of agents when multiple agents' actions all satisfy a single constraint but there is resource competition at the system level. This is achieved by prioritizing allocation, bandwidth sharing, and task replanning.

[0103] Dynamic collaborative optimization scheduling is completed based on the final action instruction, and execution results are generated. The dynamic collaborative optimization scheduling includes task acceptance, resource allocation, task migration, replica start-up and shutdown, and bandwidth allocation. The execution results include the actual task completion time, resource consumption, network overhead, migration events, and default events.

[0104] In this embodiment, obtaining the dynamically updated policy network parameters and the dynamically updated value network parameters specifically includes:

[0105] Performance metrics are calculated based on the execution results. The calculation process involves converting the actual task completion time, resource consumption, network overhead, migration events, and default events in the execution results into average completion time, resource utilization, bandwidth utilization, migration overhead, and default rate.

[0106] The performance metrics are compared with a preset threshold. When the performance metrics are less than the preset threshold, incremental online training is triggered in the improved CTDE model. The incremental online training refers to the improved CTDE model updating parameters gradually and incrementally based on the latest data collected in real time during the running phase.

[0107] During incremental online training, new local observation information, global information, and execution results are written into the replay buffer, and new training batches are constructed using time window bucket sampling.

[0108] Incremental optimization is performed on the policy network parameters and value network parameters based on the new training batches until the convergence criterion is met, generating dynamically updated policy network parameters and dynamically updated value network parameters.

[0109] Example 1:

[0110] In the production environment of a smart manufacturing enterprise, the data center in the eastern region is mainly responsible for large-scale image recognition and detection tasks, the data center in the southern region mainly handles production scheduling and real-time control tasks, and the data center in the western region undertakes massive storage and historical data analysis. Traditional centralized scheduling methods have significant shortcomings in this type of scenario: when the eastern data center is overloaded, although the southern and western regions still have a lot of spare computing power and bandwidth, the lack of an effective cross-regional collaboration mechanism often leads to some tasks queuing or even timeouts, ultimately affecting the overall task completion rate and user experience.

[0111] To verify the effectiveness of the method of this invention, topology modeling was performed using data centers in three different regions as sets of computing and storage nodes. Simultaneously, attributes such as computing power, storage capacity, bandwidth, and latency were collected to construct a resource topology map. During operation, the system collects data in real time on each node's processor utilization, task queue length, memory and GPU memory usage, as well as the read / write throughput and bandwidth utilization of storage nodes, forming node-level and system-level feature inputs. This invention constructs a multi-agent environment, where each computing node corresponds to an agent, and training and optimization are performed under an improved centralized training-distributed execution model. The agents not only make decisions based on their own node states but also utilize global information to achieve cross-node collaboration.

[0112] During the experiment, three methods were compared: the traditional centralized scheduling method, the rule-driven minimum task completion time method, and the collaborative optimization method based on multi-agent reinforcement learning proposed in this invention. The experimental tasks mainly consisted of large-scale image recognition and data retrieval tasks, with a task arrival rate set at 150 requests per second and cross-regional network round-trip latency fluctuating between 20ms and 80ms. The experiment lasted for 24 hours, and performance indicators included average task completion time, task completion rate, cross-regional communication overhead, resource utilization, and system power consumption.

[0113] Experimental results show that the method of this invention can significantly improve the collaborative efficiency of cross-regional computing power. During peak task periods, when the computing load of the eastern data center exceeds 90%, this invention can dynamically trigger task migration and computing power quota adjustment, distributing some tasks to data centers in the south and west, thereby effectively reducing task queuing time. Simultaneously, when storage resources are scarce, the agent can perform replica start / stop operations based on global information to balance storage read / write pressure. The final results show that the method of this invention significantly outperforms existing methods in terms of average task completion time, completion rate, and energy consumption, verifying the beneficial effects of this invention.

[0114] Table 1 Comparison of experimental data on remote computing power collaborative optimization based on multi-agent reinforcement learning.

[0115] Indicator Categories Traditional centralized dispatch Rule-driven scheduling Method of the present invention Average task completion time (seconds) 3.85 3.42 2.67 Task completion rate (%) 91.2 93.6 98.4 Cross-regional communication overhead (GB) 128 117 95 Resource utilization rate (%) 76.5 81.3 89.7 System power consumption (kWh) 2140 2075 1890 Peak-hour task queuing rate (%) 18.7 14.3 6.8 Number of node conflict events (times) 53 37 15 Number of dynamic migration events (times) 26 41 32 Number of default events (times) 12 9 2 Number of times online training was triggered. - - 7

[0116] As shown in Table 1, the proposed multi-agent reinforcement learning-based dynamic collaborative optimization method for remote computing power outperforms traditional methods in all metrics. Firstly, in terms of average task completion time, the proposed method significantly reduces it to 2.67 seconds, a reduction of approximately 30.6% compared to traditional centralized scheduling, thus significantly improving the system's response speed. Secondly, in terms of task completion rate, the proposed method achieves 98.4%, nearly 5 percentage points higher than rule-driven methods, demonstrating its ability to effectively avoid task timeouts and loss in dynamic environments.

[0117] Regarding cross-regional communication overhead, the method of this invention reduces communication traffic from 128GB to 95GB through intelligent task migration and bandwidth allocation strategies, reducing cross-regional traffic by approximately 25.8% and effectively alleviating link pressure. Simultaneously, the overall system resource utilization rate increases to 89.7%, indicating that this invention can more fully exploit the computing potential of remote nodes and avoid resource idleness. In terms of energy consumption, the system power consumption of this invention is controlled at 1890kWh, which is approximately 11.7% lower than traditional centralized methods, demonstrating its green and energy-saving advantages.

[0118] Furthermore, the task queuing rate of this invention during peak periods is only 6.8%, far lower than the 18.7% of traditional methods, verifying its effectiveness in handling sudden loads. The number of node conflict events also decreased significantly, indicating that the multi-agent collaborative mechanism has a good effect on conflict resolution. It is worth noting that this invention can trigger incremental online training through performance indicators during execution, dynamically optimizing policy network parameters, and ultimately achieving the ability to continuously adapt to environmental changes, which is also not available in existing methods.

[0119] In summary, this invention not only improves system performance in multi-regional computing power scheduling scenarios, but also reduces network and energy costs, fully demonstrating the technical advantages of intelligence, adaptability, and efficiency.

Claims

1. A method for dynamic collaborative optimization of remote computing power based on multi-agent reinforcement learning, characterized in that, Includes the following steps: Resource topology modeling is performed on computing nodes, storage nodes, and network links distributed in different regions to form a resource topology map; Based on the resource topology map, data is collected on the real-time operating status of computing nodes, storage nodes and network links. The collected data is preprocessed to obtain node-level state feature vectors and system-level state feature vectors. The node-level state feature vector is defined as local observation information, the system-level state feature vector is defined as global information, and the action set of each agent is set to construct a multi-agent environment. Based on the agent environment, local observation information, global information and agent action set are input into the improved CTDE model for centralized training and parallel environment interaction, outputting policy network parameters and value network parameters, and constructing training batches by using time window bucket sampling. Iterative optimization of the policy network parameters and value network parameters is performed based on the training batches until the convergence criterion is met, resulting in converged policy network parameters and converged value network parameters. The converged policy network parameters and converged value network parameters are deployed to each computing node and run in the distributed execution channel of the improved CTDE model to generate execution results; The performance index is calculated based on the execution result and compared with a preset threshold. When it is less than the preset threshold, incremental online training is triggered in the improved CTDE model to obtain the dynamically updated policy network parameters and the dynamically updated value network parameters. The specific methods for obtaining the node-level state feature vector and the system-level state feature vector include: At a preset time step, running status data is collected for each computing node in the computing node set in the resource topology graph. The running status data includes processor utilization, video memory usage, memory usage, task queue length, remaining task time, and data location information. At a preset time step, the running status data of each storage node in the storage node set in the resource topology map is collected. The running status data includes storage capacity utilization, read / write throughput utilization and power consumption parameters. At a preset time step, operational status data is collected for each network link in the network link set in the resource topology diagram. The operational status data includes available bandwidth and round-trip latency. The running status data of the computing node set and the network link set are processed by missing value filling, noise removal and normalization to obtain the preprocessed running status sequence; The preprocessed running state sequence is sliced ​​according to time windows to generate node-level state feature vectors and system-level state feature vectors. The node-level state feature vectors include the resource utilization and task execution status of a single computing node and a single storage node within the time window. The system-level state feature vectors include global task arrival rate, global task completion rate, global resource utilization rate, available network link bandwidth, network link round-trip time, system power consumption, and cross-regional communication overhead. The construction of the training batch specifically includes: The improved CTDE model is input into the local observation information, global information, and agent action set in a multi-agent environment for centralized training and parallel environment interaction. The improved CTDE model includes a state and observation processing module, a policy network module, a value network module, and a training and execution module. The state and observation processing module enhances the local and global information through neighborhood aggregation and link feature concatenation. The policy network module receives the local observation information and outputs the agent action distribution. The value network module receives the global information and the actions of all agents, evaluates the state-action pairs, and outputs a value estimate. The training and execution module optimizes the policy network parameters and value network parameters during the centralized training phase and deploys the optimized parameters for independent operation by the agents during the decentralized execution phase. In the state and observation processing module, local observation information is input into the corresponding agent, and global information is input into all agents. An improved method of neighborhood aggregation and link feature concatenation is introduced to enhance the local observation information and global information. The neighborhood aggregation and link feature concatenation refers to weighting and aggregating the running states of neighboring nodes through an attention mechanism, and concatenating the attributes of the links with the running state features of the aggregated neighboring nodes. In the policy network module, enhanced local observation information is used as input to output the distribution of agent actions. The distribution of actions covers task acceptance, task assignment, task migration, computing power quota adjustment, replica start and stop, and bandwidth allocation operations. In the value network module, the state-action pairs are evaluated for value using the enhanced global information and the actions of all agents as input, and the global value estimate is output. During the intensive training phase of the training and execution module, local observation information, global information, agent action distribution, and global value estimation are jointly input into the training module, and policy network parameters and value network parameters are output. Sample scheduling is performed on the policy network parameters and value network parameters. Training batches are constructed using time window binning sampling. Time window binning sampling refers to storing the data of policy network parameters and value network parameters in segments according to time order, with each segment serving as a sampling bucket. During training, samples are evenly drawn from buckets of different time periods.

2. The method for dynamic collaborative optimization of remote computing power based on multi-agent reinforcement learning according to claim 1, characterized in that, The formation of the resource topology map specifically includes: Obtain computing nodes, storage nodes, and network links distributed across different regions, and establish sets of computing nodes, storage nodes, and network links; For each computing node in the computing node set, the computing power, memory capacity, and power consumption parameters are collected to form a computing node attribute set. For each storage node in the storage node set, the storage capacity, read / write throughput, and power consumption parameters are collected to form a storage node attribute set. For each network link in the network link set, bandwidth and round-trip time are collected to form a set of network link attributes; Establish a mapping relationship between the node attribute set, the storage node attribute set, and the network link attribute set. The mapping relationship is to identify the connection relationship between computing nodes and storage nodes through the network link attribute set, and to associate the bandwidth and round-trip time of each network link with the corresponding node connection to form a correspondence. Resource topology modeling is completed based on the set of computing nodes, the set of storage nodes, the set of network links, the set of node attributes, the set of storage nodes, and the set of network link attributes, forming a resource topology graph.

3. The method for dynamic collaborative optimization of remote computing power based on multi-agent reinforcement learning according to claim 1, characterized in that, The construction of the multi-agent environment specifically includes: The node-level state feature vector is defined as local observation information. A corresponding agent is configured for each computing node in the computing node set. The definition process is to establish an input mapping relationship between the node-level state feature vector and the corresponding agent, and associate it with the corresponding agent. The system-level state feature vector is defined as global information. The process of defining it is to establish a shared input mapping relationship for the system-level state feature vector and input it as global information to all agents for sharing. For each agent in the agent set, an agent action set is set, which includes task acceptance, task assignment, task migration, computing power quota adjustment, replica start / stop, and bandwidth allocation; A multi-agent environment is constructed based on local observation information, global information, and the set of agent actions.

4. The method for dynamic collaborative optimization of remote computing power based on multi-agent reinforcement learning according to claim 1, characterized in that, The specific steps for obtaining the converged policy network parameters and the converged value network parameters include: Iterative optimization is performed on the policy network parameters and value network parameters based on training batches. The iterative optimization includes advantage estimation, policy parameter update, target network soft update and gradient pruning. In the advantage estimation phase, the target value is calculated, which is equal to the reward plus a discount factor multiplied by the value estimate of the next time step, and the advantage value is calculated, which is equal to the target value minus the value estimate of the current time step. During the policy network parameter update phase, a policy loss function is constructed based on the agent's action distribution and advantage value. An entropy regularization term is introduced to enhance exploration. The gradient of the policy loss function is used to update the policy network parameters, and gradient pruning is performed during the update process to control the update magnitude. During the target network soft update phase, the value network parameters are updated by minimizing the mean square error between the global value estimate and the target value. The policy network parameters and value network parameters are then synchronized to the target network using a soft update method until the convergence criterion is met, resulting in converged policy network parameters and converged value network parameters.

5. The method for dynamic collaborative optimization of remote computing power based on multi-agent reinforcement learning according to claim 1, characterized in that, The generation of the execution result specifically includes: The converged policy network parameters and converged value network parameters are deployed to each computing node in the computing node set and run in the distributed execution channel of the improved CTDE model; At each dynamic runtime time step, the action vector is output using the converged policy network parameters. The feasible region projection is performed on the output action vector to obtain actions that are constrained to be feasible. The feasible region projection is the process of correcting and mapping the action vector output by the agent according to system capacity constraints, time delay constraints, bandwidth constraints and power consumption constraints. For actions that are feasible under constraints, conflict resolution is performed to obtain the final action instruction. The conflict resolution refers to the process of eliminating contradictions between actions of agents when multiple agents' actions all satisfy a single constraint but there is resource competition at the system level. This is achieved by prioritizing allocation, bandwidth sharing, and task replanning. Dynamic collaborative optimization scheduling is completed based on the final action instruction, and execution results are generated. The dynamic collaborative optimization scheduling includes task acceptance, resource allocation, task migration, replica start-up and shutdown, and bandwidth allocation. The execution results include the actual task completion time, resource consumption, network overhead, migration events, and default events.

6. The method for dynamic collaborative optimization of remote computing power based on multi-agent reinforcement learning according to claim 1, characterized in that, The acquisition of the dynamically updated policy network parameters and the dynamically updated value network parameters specifically includes: Performance metrics are calculated based on the execution results. The calculation process involves converting the actual task completion time, resource consumption, network overhead, migration events, and default events in the execution results into average completion time, resource utilization, bandwidth utilization, migration overhead, and default rate. The performance metrics are compared with a preset threshold. When the performance metrics are less than the preset threshold, incremental online training is triggered in the improved CTDE model. The incremental online training refers to the improved CTDE model updating parameters gradually and incrementally based on the latest data collected in real time during the running phase. During incremental online training, new local observation information, global information, and execution results are written into the replay buffer, and new training batches are constructed using time window bucket sampling. Incremental optimization is performed on the policy network parameters and value network parameters based on the new training batches until the convergence criterion is met, generating dynamically updated policy network parameters and dynamically updated value network parameters.

Images