A distributed resource scheduling optimization system based on multi-agent reinforcement learning

Abstract

The application discloses a kind of distributed resource scheduling optimization systems based on multi-agent reinforcement learning, including bottom agent and upper agent, bottom agent includes bottom server level agent and middle layer rack level agent, and upper agent includes upper data center level agent;Bottom server level agent is responsible for CPU, memory, storage resource allocation of single server;Middle layer rack level agent is responsible for task scheduling and load balancing of server in rack;Upper data center level agent is responsible for global resource coordination and task routing across data center.The application designs layered cooperation architecture, communication mechanism, competition balance strategy and dynamic adaptation algorithm, to realize the collaborative optimization scheduling of distributed resources.System uses MADDPG, QMIX and MAPPO and other multi-agent reinforcement learning algorithms, so that each agent makes decisions under local information, achieves global optimization through cooperation, supports load balancing, energy optimization, quality of service guarantee and other multi-objective optimization.

Description

Technical Field

[0001] This invention relates to the field of distributed computing and intelligent optimization technology, specifically to a distributed resource scheduling optimization system based on multi-agent reinforcement learning (MARL). It is applicable to various distributed resource management scenarios such as cloud computing data centers, edge computing networks, Internet of Things systems, 5G network slicing, and smart grids, and can effectively solve the limitations of traditional centralized scheduling methods in terms of scalability, dynamic adaptability, and multi-objective optimization. Background Technology

[0002] With the rapid development of cloud computing, edge computing, and the Internet of Things (IoT) technologies, the scale and complexity of distributed systems have increased dramatically, and resource scheduling has become a key bottleneck restricting system performance. Traditional resource scheduling methods mainly adopt a centralized control architecture, where a central scheduler collects global state information and makes scheduling decisions. However, in large-scale distributed systems, centralized scheduling methods face the following serious problems:

[0003] 1. Scalability Bottleneck: As the system scales up, the state space that the central scheduler needs to process grows exponentially, and the computational complexity increases dramatically, making it difficult to meet the real-time scheduling requirements of large-scale systems. When the system contains thousands of nodes, the decision latency of the centralized scheduler can reach several seconds or even tens of seconds, seriously affecting the system's response performance.

[0004] 2. Single point of failure risk: The centralized architecture has obvious single point of failure problem. Once the central scheduler fails, the resource scheduling function of the entire system will be completely invalidated, and the availability and reliability of the system will be seriously threatened.

[0005] 3. Weak adaptability to dynamic environments: Distributed systems operate in highly dynamic environments, including load fluctuations, node failures, and changes in network latency. Centralized schedulers need to frequently collect global information and recalculate, resulting in slow response times and difficulty in quickly adapting to environmental changes, leading to decreased resource utilization and deterioration of service quality.

[0006] 4. Difficulty in multi-objective optimization: In practical applications, it is necessary to optimize multiple conflicting objectives simultaneously, such as minimizing task latency, minimizing energy consumption, maximizing resource utilization, and ensuring service quality. Traditional methods usually use weighted summation to transform multiple objectives into a single objective, but the weight setting lacks theoretical basis and it is difficult to find a true Pareto optimal solution.

[0007] 5. Huge communication overhead: Centralized scheduling requires all nodes to report status information to the central scheduler, which will generate huge communication overhead in large systems, occupy a lot of network bandwidth, and affect normal business data transmission.

[0008] 6. Privacy and security issues: Centralized architecture requires each node to report detailed resource status and task information to the central scheduler. In cross-organizational and cross-data center scenarios, there may be risks of sensitive data leakage.

[0009] In recent years, reinforcement learning technology has shown great potential in the field of resource scheduling, but existing reinforcement learning-based scheduling methods still have the following shortcomings:

[0010] 1. Limitations of single-agent methods: Existing methods mostly use single-agent reinforcement learning, treating the entire system as a single decision-making agent, which cannot fully utilize the advantages of parallel decision-making in distributed systems, resulting in low decision-making efficiency.

[0011] 2. Lack of agent collaboration mechanism: In multi-agent scenarios, independent decision-making by each agent may lead to resource competition and conflict. The lack of an effective collaboration mechanism to coordinate agent behavior makes it difficult to achieve global optimization.

[0012] 3. Difficulty in balancing competition and cooperation: In distributed resource scheduling, agents both compete for scarce resources and need to cooperate to jointly optimize the global goal. Existing methods struggle to effectively balance competition and cooperation, easily leading to a "prisoner's dilemma" and resulting in decreased system performance.

[0013] 4. Insufficient hierarchical scheduling architecture: Large-scale distributed systems typically have a hierarchical structure (such as data center-rack-server), but existing methods lack a hierarchical multi-agent architecture design, and cannot make full use of the system hierarchy for efficient scheduling.

[0014] 5. Weak transfer learning capability: Resource scheduling problems in different application scenarios are similar, but existing methods lack effective transfer learning mechanisms, making it impossible to quickly adapt trained models to new scenarios, resulting in high training costs.

[0015] Therefore, there is an urgent need for a distributed resource scheduling optimization system that can fully utilize the characteristics of distributed systems, achieve effective cooperation among agents, balance competition and cooperation, support multi-objective optimization, and dynamic environmental adaptation. This invention addresses these technical challenges by designing a hierarchical multi-agent reinforcement learning architecture, a communication and cooperation mechanism, a competition balancing strategy, and a dynamic adaptation algorithm, providing an efficient and reliable technical solution for distributed resource scheduling. Summary of the Invention

[0016] To address the aforementioned problems, this invention provides a distributed resource scheduling optimization system based on multi-agent reinforcement learning. By modeling computing nodes, storage nodes, and network nodes as independent agents, a hierarchical collaborative architecture, communication mechanism, competition balancing strategy, and dynamic adaptation algorithm are designed to achieve collaborative optimization scheduling of distributed resources. The system employs multi-agent reinforcement learning algorithms such as MADDPG (Multi-Agent Deep Deterministic Policy Gradient), QMIX (Q-value Mixing), and MAPPO (Multi-Agent Proximal Policy Optimization), enabling each agent to make decisions based on local information and achieve global optimization through cooperation. This supports multi-objective optimization such as load balancing, energy consumption optimization, and quality of service assurance.

[0017] The specific plan is as follows:

[0018] A distributed resource scheduling and optimization system based on multi-agent reinforcement learning includes bottom-layer agents and upper-layer agents. The bottom-layer agents include bottom-layer server-level agents and mid-layer rack-level agents, responsible for local resource scheduling. The upper-layer agents include upper-layer data center-level agents, responsible for global resource coordination. Specifically, the bottom-layer server-level agents are responsible for allocating CPU, memory, and storage resources for a single server; the mid-layer rack-level agents are responsible for task scheduling and load balancing of servers within a rack; and the upper-layer data center-level agents are responsible for global resource coordination and task routing across data centers.

[0019] Furthermore, the design method of the system is as follows:

[0020] S1. Design a layered multi-agent architecture, dividing the distributed system into two layers: a lower-level agent and an upper-level agent, to achieve scalable distributed scheduling. This architecture is the foundation of the system, and subsequent modules such as communication mechanisms, competition and cooperation balancing, and dynamic adaptation are all built on this architecture.

[0021] S2. After completing the hierarchical multi-agent architecture design, each agent needs to make resource scheduling decisions based on local information. However, relying solely on local information may lead to suboptimal decisions because agents cannot understand the global system state and the intentions of other agents. To improve the quality of global decision-making, this invention designs a communication protocol and information sharing mechanism between agents, enabling them to exchange information such as local state, resource requirements, and load predictions. This allows them to obtain a more comprehensive system view based on local information and make better scheduling decisions.

[0022] S3. After extracting the communication and collaboration features of the agents, the agents need to make resource scheduling decisions based on this information. However, in distributed resource scheduling, agents both compete (for scarce resources) and cooperate (to jointly optimize the global goal). If only local interests are considered, agents may overcompete, leading to a decline in system performance; if only global interests are considered, agents may lack the motivation to optimize their own performance. Therefore, this invention designs a multi-objective reward function and a game mechanism to balance competition and cooperation, guiding agents to optimize their own performance while promoting the optimization of the global goal.

[0023] S4. After completing the reward function design and competitive-cooperative balance, the agent learns the optimal policy through reinforcement learning algorithms. However, the operating environment of distributed systems is highly dynamic, including load fluctuations, node failures, and changes in network latency. If the system cannot quickly adapt to environmental changes, the learned policy may become ineffective, leading to a decline in system performance. Therefore, this invention designs a dynamic adaptation mechanism to enable the system to quickly detect environmental changes and adjust its policy, maintaining efficient system operation.

[0024] S5. After completing the design of the dynamic environment adaptation mechanism, the system possesses basic resource scheduling capabilities. However, in practical applications, it is necessary to simultaneously optimize multiple conflicting objectives, such as task latency, system energy consumption, and resource costs. Traditional single-objective optimization methods cannot simultaneously meet these requirements, while simple weighted summation methods struggle to find truly Pareto optimal solutions. Therefore, this invention utilizes a multi-objective reinforcement learning method to generate a set of Pareto optimal solutions, supporting decision-makers in selecting the optimal strategy based on current preferences.

[0025] Further, step S1 includes:

[0026] S11, Underlying Intelligent Agent Design

[0027] The underlying intelligent agents are responsible for local resource scheduling, with one underlying intelligent agent corresponding to each computing node, storage node, or network node; assuming there are a total of The first underlying intelligent agent, the first The local state space of each agent is: The action space is The reward function is ,in , representing intelligent agents The current status (including local resource utilization, task queue length, energy consumption, etc.). Represents intelligent agents The chosen actions (such as resource allocation ratios, task acceptance / rejection decisions, etc.) Indicates the next state after the action is performed;

[0028] The underlying intelligent agents employ deterministic policy networks. Output continuous actions, where The parameters are for the policy network; to learn the optimal policy, the MADDPG algorithm is used, with each agent maintaining an Actor network (policy network) and a Critic network (value function network); the Critic network... Assessment in status Next action And other intelligent agents perform actions The expected cumulative reward at that time, of which These are the parameters for the Critic network.

[0029] The policy gradient update formula for the underlying agent is:

[0030]

[0031] in For intelligent agents Expected cumulative reward For state distribution, The gradient of the policy network with respect to the parameters. Let be the gradient of the Critic network with respect to actions; this formula optimizes the agent's policy through the policy gradient method, enabling it to learn to make optimal decisions considering the behavior of other agents.

[0032] S12, Upper-layer intelligent agent design

[0033] The upper-layer intelligent agent is responsible for global resource coordination, such as load balancing across data centers and global task routing; assuming the system has a total of There are three upper-level intelligent agents, each managing a group of lower-level intelligent agents; The state space of each upper-level intelligent agent It contains aggregated state information of all underlying agents it manages, and its action space. Includes global coordination actions (such as task migration instructions, resource reservation strategies, etc.);

[0034] The upper-layer intelligent agent is trained using the QMIX algorithm, which achieves centralized training and distributed execution through value function decomposition. The core idea of ​​QMIX is to decompose the global Q-value function. Decomposed into a mixture of the local Q-value functions of each agent:

[0035]

[0036] in This is the global state. For the set of actions of all upper-level agents, For the first The local Q-value function of each upper-level agent For hybrid networks, ensure that monotonicity constraints are met: For all This constraint is valid. It ensures that an increase in the local Q-value of each agent will not lead to a decrease in the global Q-value, thereby promoting cooperation among agents.

[0037] Further, step S2 includes:

[0038] S21. Communication Topology Construction

[0039] First, a graph neural network (GNN) is used to model the communication topology between agents; this topology reflects the physical connections (like on the same rack, in the same data center) and logical collaboration relationships between agents; let the agent communication graph be defined. ,in A set of nodes (containing all) A low-level intelligent agent and (a higher-level intelligent agent) Let edge be the set of edges, representing the communication relationships between agents (e.g., ... Represents intelligent agents and (They can communicate directly); each agent Maintain an initial feature vector This indicates its initial state information (such as resource utilization, task queue length, etc.). For feature dimensions;

[0040] The feature representation of the agent is iteratively updated through multi-layer graph convolution operations, enabling each agent to aggregate information from its neighboring agents:

[0041]

[0042] in For the first layer( Intelligent agents The feature representation vector, The number of convolutional layers for the graph (usually set to 2-3 layers). For intelligent agents The neighborhood group, For intelligent agents The number of neighbors, For intelligent agents The number of neighbors, and The first The learnable weight matrix and bias vector of the layer, The activation function is usually ReLU. The coefficients are symmetric normalization coefficients (using the standard normalization method for graph convolutional networks to ensure the stability of feature updates and avoid excessively large or small feature values). This formula aggregates information from neighboring agents, enabling each agent to obtain a global view of its local region, thereby making better scheduling decisions. After layer graph convolution, the agent Features Includes its Information about skipping neighbors (i.e., passing through) After this information transmission, the agent is able to obtain the distance to its (Information on all neighbors within the jump range);

[0043] S22, Attention Mechanism Collaboration

[0044] After completing the graph convolution feature extraction, in order to further improve the collaboration efficiency between agents, this invention adopts a multi-head attention mechanism to learn the collaboration pattern between agents; this mechanism enables agents to adaptively pay attention to other agents with close collaboration relationships, thereby making better collaboration decisions.

[0045] For intelligent agents Calculate its relationship with other intelligent agents Attention weights:

[0046]

[0047] in and They were respectively through Intelligent agents after layer graph convolution and The feature representation vector, and These are the projection matrices for the query and key, respectively (projecting features into the query space and key space). This is the attention weight vector (used to calculate the similarity between the query and the key). This represents a vector concatenation operation. For intelligent agents The neighborhood group, For feature dimension, To sum the index variables (avoiding indices from the upper-level agent's index) Conflict); Attention weight Represents intelligent agents For intelligent agents The degree of attention is indicated by the value of the weight. The larger the value, the closer the cooperation between the two. This weight is normalized by Softmax (i.e., the summation operation of the denominator) to ensure that the sum of the attention weights of all neighbors is 1, so that the attention weights form a probability distribution.

[0048] intelligent agent The collaborative features are obtained through weighted aggregation:

[0049]

[0050] in The projection matrix of the value, For the process Intelligent agents after layer graph convolution The feature representation vector, For intelligent agents The collaborative feature vector is derived by incorporating relevant information from neighboring agents. This collaborative feature serves as input to the agent's decision-making process, enabling the agent to consider the states and intentions of neighboring agents when making resource scheduling decisions, thereby improving collaborative efficiency. The collaborative feature is then convolved with the original feature obtained from the graph. They can be further integrated (such as by splicing or weighted summation) to form the final decision feature representation.

[0051] Further, step S3 includes:

[0052] S31. Design of Multi-Objective Reward Function

[0053] Each agent's reward function consists of two parts: a local reward and a global reward. A balance between competition and cooperation is achieved through a weighted combination.

[0054]

[0055] in For intelligent agents Total reward value, For intelligent agents Local rewards reflect its own performance (such as resource utilization and task completion rate). It serves as a global reward, reflecting the overall performance of the system (such as global load balancing and total energy consumption). The balancing coefficient controls the weights of local and global optimization; when When the agent is small, it focuses more on optimizing its own performance (primarily competition); when... When the system is large, the agent focuses more on global performance optimization (mainly cooperation); this design enables the agent to dynamically adjust the balance between competition and cooperation according to the system state.

[0056] The local reward design is a weighted combination of multiple objectives:

[0057]

[0058] in For intelligent agents The overall resource utilization rate (average of CPU, memory, and storage utilization). For intelligent agents Normalized energy consumption (current energy consumption divided by maximum power consumption). For intelligent agents The normalized task queue length (current queue length divided by maximum queue length). Let be the weighting coefficient, satisfying Through positive rewards ( ) Encourage agents to improve resource utilization through negative penalties ( Encourage agents to reduce energy consumption and task latency to achieve local multi-objective optimization;

[0059] The global reward is designed with multi-objective weighted penalties:

[0060]

[0061] in The normalized variance of resource utilization of all intelligent agents (variance divided by the maximum possible variance) reflects the degree of system load balance. The smaller the value, the more balanced the load. This is the normalized value of the total system energy consumption (total energy consumption divided by the maximum total power consumption of the system). This is the normalized value of the maximum task queue length among all agents. Let be the weighting coefficient, satisfying This design encourages the system to achieve load balancing (minimizing variance), reduce total energy consumption, and minimize maximum latency through a negative penalty mechanism, thereby promoting cooperation among agents. The global reward is the same for all agents, enabling them to optimize their own performance while collectively optimizing the global objective.

[0062] S32. Solving Nash Equilibrium

[0063] By designing a reward function, the agent is guided to learn a Nash equilibrium policy. Under Nash equilibrium, no agent can obtain a higher reward by unilaterally changing its policy, thus achieving a stable cooperative state. This invention uses experience replay and policy gradient methods to solve for Nash equilibrium. The specific implementation process is as follows:

[0064] First, each agent collects experience data by interacting with the environment. The data is stored in the experience replay buffer. The experience replay buffer adopts a first-in-first-out (FIFO) strategy, maintains a fixed size, and deletes the oldest experience when the buffer is full. During training, a batch of experience data is randomly sampled from the buffer to break the temporal correlation of the data and improve training stability.

[0065] Then, the agent's policy is updated using the policy gradient method; for the agent... Its policy gradient update formula is:

[0066]

[0067] in For the distribution of experience in the experience playback buffer, The empirical data sampled from the experience replay buffer contains the agent. status ,action Actions with other intelligent agents , The gradient of the policy network with respect to the parameters. Let be the gradient of the Critic network for actions; through iterative updates, the policies of each agent gradually converge to a Nash equilibrium policy; under Nash equilibrium, for each agent... Its current strategy satisfy:

[0068]

[0069] That is, any agent that deviates from the Nash equilibrium policy alone cannot obtain a higher reward, thus achieving a stable cooperative state; through multiple rounds of iterative training, the agent can learn a policy close to the Nash equilibrium, achieving an effective balance between competition and cooperation.

[0070] Further, step S4 includes:

[0071] S41. Online Learning and Strategy Updates

[0072] When a significant change in the environment is detected (such as load fluctuations exceeding a preset threshold) Node failure rate exceeds threshold When the system triggers an online learning mechanism to quickly update the agent's strategy, it uses an incremental learning algorithm to fine-tune the strategy network parameters using new environmental data while maintaining the original knowledge, thus avoiding knowledge loss caused by training from scratch.

[0073] The specific implementation process is as follows: First, the system continuously monitors environmental change indicators, including load volatility. and node failure rate ,in, and The average loads for the new and old environments are respectively. The number of faulty nodes. The total number of nodes; when or At that time, online learning is triggered; then, the system collects experience data in the new environment. ,in For the amount of experience in the new environment; adaptive loss function Defined as a weighted combination of policy loss and value function loss:

[0074]

[0075] in For strategic losses, The advantage function measures the advantage of an action relative to the average action. Calculation, where It is a state-value function; For value function loss, For the target Q value, use a deterministic strategy. Rather than the maximum value, because MADDPG deals with a continuous action space. and Weighting coefficients (usually set to) , ).

[0076] The formula for updating the strategy parameters is:

[0077]

[0078] in For intelligent agents In the The policy network parameter vector at the next update For parameter dimensions, To adapt the learning rate (usually set to the initial training learning rate). arrive (to ensure stability) For parameters The gradient operator enables the system to quickly adapt to environmental changes while maintaining the stability of the original strategy, avoiding a sharp decline in performance. Online learning employs a sliding window strategy, using only the nearest gradient operator. Use empirical data to avoid overfitting new data and forgetting old knowledge;

[0079] S42, Rapid Adaptation Through Transfer Learning

[0080] When the system is deployed to a new scenario, transfer learning technology is used to transfer the knowledge of the trained model to the new scenario; the specific implementation process is as follows:

[0081] First, in the source scenario (the already trained scenario), the model has learned general resource scheduling strategies, including how to balance load, optimize energy consumption, and coordinate operations. This knowledge is mainly stored in the feature extraction layer of the policy network. During transfer learning, the policy network trained in the source scenario is transferred... As an initial model, the parameters of the feature extraction layers (the first two layers, i.e., from the input layer to the first hidden layer and from the first hidden layer to the second hidden layer) are directly transferred to the new scene because these layers learn general feature representations (such as resource utilization patterns, load distribution features, etc.), which are similar across different scenes.

[0082] Then, in the new scenario, a fine-tuning strategy is used to update the model parameters; specifically, the policy network is divided into two parts: a feature extraction layer... and decision-making level The parameters of the feature extraction layer are fine-tuned using a small learning rate (e.g., While maintaining general features, it adapts to the specificities of new scenarios; the decision layer parameters are trained using a normal learning rate (e.g., ), quickly learn decision-making strategies for new scenarios;

[0083] The loss function for the fine-tuning process is:

[0084]

[0085] in For the strategy loss in the new scenario, The regularization coefficient is . This is a regularization term to prevent the parameters of the feature extraction layer from deviating too far from the parameters of the source scene, thus maintaining general features. Through feature extraction and fine-tuning, the model can quickly adapt to the resource scheduling requirements of new scenes, significantly reducing training costs and time. Experiments show that transfer learning reduces training time in new scenes from 1 million steps to 100,000 steps, and reduces training costs by 90%.

[0086] Further, step S5 includes: using a multi-objective value function decomposition method to transform the multi-objective optimization problem into multiple single-objective sub-problems, each sub-problem corresponding to an optimization objective. The specific implementation process is as follows:

[0087] First, construct an independent Q-value function network for each optimization objective. Assume the system has... One optimization objective, construct An independent Critic network Each network corresponds to an optimization objective, where For the first The parameters of the network; the multi-objective Q-value vector is defined as:

[0088]

[0089] in For a multi-objective Q-value vector, The target quantity (usually 3-5). For the first The Q-value function of an optimization objective (e.g.) Corresponding task delay, Corresponding system energy consumption, (corresponding to resource utilization variance, etc.) For system status, For action;

[0090] Then, the Pareto front is generated using the weight vector method; for different weight combinations ;satisfy and Define the weighted Q-value function:

[0091]

[0092] The weighted Q-value function is optimized using reinforcement learning algorithms (such as MADDPG, Actor-Critic, etc.; in this invention, the MADDPG algorithm is used) to obtain the optimal policy under the corresponding weight combination. Multiple solutions on the Pareto front are generated by enumerating or sampling different weight combinations (e.g., using uniform grid sampling, random sampling, or adaptive sampling); each solution corresponds to an optimal policy under a certain weight combination. These policies are on the Pareto front, meaning that no other policy is better than this policy on all objectives (the definition of Pareto optimality: for a given policy...). There are no other strategies. To have in all objectives And at least in one objective, it is strictly greater than);

[0093] Decision-makers select a weight combination based on current business needs and preferences, and the system automatically switches to the corresponding optimal strategy, achieving flexible multi-objective optimization. For example, during peak business periods, a "low latency priority" weight combination can be selected (…). During off-peak business hours, select the "low energy consumption priority" weighting combination. ).

[0094] Furthermore, after completing the design of all the above modules, the system needs to be deployed and run in a real-world environment. The system adopts a distributed architecture, with each agent running independently on its corresponding node and communicating via message passing. An experience replay buffer stores historical experience, and a target network is used for stable training. The system supports pre-training in simulation environments and fine-tuning in real-world environments, improving system robustness and practicality. The modules work collaboratively: a layered architecture provides the system's organizational foundation, a communication mechanism enables information sharing, a competition-cooperation balancing mechanism guides policy learning, a dynamic adaptation mechanism ensures system robustness, and multi-objective optimization meets practical needs, ultimately forming a complete, efficient, and scalable distributed resource scheduling and optimization system.

[0095] Through the above technical solution, this invention constructs an efficient, scalable, and adaptive distributed resource scheduling and optimization system, which breaks through the limitations of traditional centralized scheduling and realizes a paradigm shift from "centralized control" to "intelligent collaboration", significantly improving resource utilization and system performance.

[0096] The beneficial effects of this invention are as follows:

[0097] 1. By employing a hierarchical multi-agent architecture and a distributed decision-making mechanism, this invention overcomes the scalability bottleneck of traditional centralized scheduling. Experiments show that when the system scales to 4800 servers, the scheduling latency increases by only 15%, while the traditional centralized method increases by 120%. The system supports smooth scaling from hundreds to tens of thousands of servers, demonstrating excellent scalability and meeting the scheduling needs of large-scale distributed systems.

[0098] 2. The distributed architecture eliminates the risk of single points of failure. When some agents or nodes fail, the system can automatically adjust while other agents continue to function normally. Experiments show that when simulating a 10% node failure, the system automatically adjusts within 5 minutes, with a performance degradation of less than 5%. This feature gives the system good fault tolerance, improving its reliability and availability.

[0099] 3. Through online learning and transfer learning mechanisms, the system can quickly adapt to dynamic environments such as load fluctuations, node failures, and network changes. Experiments show that the system can adapt to load fluctuations within 5 minutes and node failures within 10 minutes. Transfer learning reduces the training cost for new scenarios by 90%. This capability enables the system to operate stably and efficiently in complex and ever-changing real-world environments.

[0100] 4. Through a multi-objective Pareto optimization method, the system can simultaneously optimize multiple conflicting objectives, generating a Pareto optimal solution set, supporting decision-makers in selecting solutions based on preferences. Experiments show that the system achieves significant improvements in resource utilization, load balancing, and energy consumption optimization: resource utilization reaches 85.3% (an improvement of 18.3%), load balancing reaches 0.92 (an improvement of 21.1%), total energy consumption is reduced by 18.7%, and average task latency is reduced by 23.5%. This capability enables the system to meet the diverse needs of different application scenarios.

[0101] 5. Compared to centralized scheduling, which requires all nodes to report their status to the central scheduler, the distributed architecture of this invention only requires communication between neighboring agents, significantly reducing communication overhead. Experiments show that the system communication overhead is only 15% of the traditional centralized method, effectively reducing network bandwidth usage and improving system efficiency.

[0102] 6. The distributed architecture allows each organization's data center to operate agents independently without reporting detailed resource status to a central scheduler, thus protecting data privacy and security. This feature enables the system to be applied across organizations and data centers, expanding its application scope.

[0103] 7. Through multi-agent reinforcement learning algorithms such as MADDPG, QMIX, and MAPPO, the system can learn the optimal scheduling strategy from experience without the need for manual design of complex scheduling rules. The agents continuously optimize their strategies by interacting with the environment, adapting to different workloads and system configurations, demonstrating strong learning capabilities and adaptability.

[0104] 8. By designing a multi-objective reward function and a game theory mechanism, the system effectively balances the competition and cooperation among agents, avoiding the "prisoner's dilemma" and guiding agents to learn a Nash equilibrium strategy. This mechanism enables the system to remain stable and efficient in an environment of intense resource competition, achieving global optimality.

[0105] 9. The technical framework of this invention has good scalability, can easily integrate more types of resources (such as GPUs, network bandwidth, etc.), can be extended to more levels (such as regional level, national level), and can be adapted to different multi-agent algorithms, providing a solid foundation for subsequent technology iteration and functional enhancement.

[0106] 10. This invention can be applied to multiple fields such as cloud computing data centers, edge computing networks, IoT systems, 5G network slicing, smart grids, logistics and distribution, smart manufacturing, and game servers, and has broad market application prospects and commercial value. Especially in scenarios requiring high scalability, strong robustness, and multi-objective optimization, this invention has irreplaceable technical advantages. Attached Figure Description

[0107] Figure 1 This is a flowchart of the present invention.

[0108] Figure 2 This is an architectural diagram of the present invention. Detailed Implementation

[0109] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0110] As shown in the figure, this embodiment provides a distributed resource scheduling optimization system based on multi-agent reinforcement learning. Taking cloud computing data centers as the application scenario, it includes multiple data centers, each data center contains multiple racks, and each rack contains multiple servers.

[0111] I. System Architecture and Deployment Configuration

[0112] This embodiment uses a cloud computing data center as an application scenario to describe the specific implementation process of the system in detail. A three-layer distributed resource scheduling system is constructed: the bottom layer consists of server-level agents, responsible for the allocation of CPU, memory, and storage resources for a single server; the middle layer consists of rack-level agents, responsible for task scheduling and load balancing of servers within the rack; and the upper layer consists of data center-level agents, responsible for global resource coordination and task routing across data centers. This three-layer architecture is a specific implementation of the layered multi-agent architecture design in the invention, where the bottom-layer server-level agents and the middle-layer rack-level agents correspond to the bottom-layer agents in the invention (responsible for local resource scheduling), and the upper-layer data center-level agents correspond to the upper-layer agents in the invention (responsible for global resource coordination).

[0113] System configuration: Number of data centers: 3, Number of racks in each data center: 20, Number of servers in each rack: 40, Total number of servers: 2400, Number of bottom-layer intelligent agents: 2400 (one per server), Number of middle-layer intelligent agents: 60 (one per rack), Number of upper-layer intelligent agents: 3 (one per data center).

[0114] The system adopts a distributed deployment, with each agent running on a corresponding node and communicating asynchronously through a message queue (such as RabbitMQ), with communication latency controlled within 10ms.

[0115] II. Implementation of a Layered Multi-Agent Architecture

[0116] Based on the system architecture configuration, this embodiment describes in detail the specific implementation of each layer of intelligent agents. The layered architecture design enables the bottom-layer intelligent agents to be responsible for fine-grained resource allocation, the middle-layer intelligent agents to be responsible for local coordination, and the top-layer intelligent agents to be responsible for global optimization. Through information transmission and cooperation between layers, efficient distributed resource scheduling is achieved.

[0117] (1) Implementation of the underlying intelligent agent

[0118] Each server agent maintains local state information, including: CPU utilization. Memory utilization Storage utilization Task queue length (range of values) Current energy consumption (Unit: Watt, range of values) State vector Defined as:

[0119]

[0120] The task queue length is determined by... Normalization to Range, energy consumption Normalization to The range ensures that all state features are on the same order of magnitude, which is beneficial for neural network training.

[0121] Action space includes continuous actions: CPU allocation ratio (Indicates the proportion of CPU resources allocated to the current task), memory allocation proportion (Indicates the proportion of memory resources allocated to the current task), whether to accept new tasks. (0 indicates rejection, 1 indicates acceptance). Action vector Defined as:

[0122]

[0123] The underlying agents are trained using the MADDPG algorithm, which is suitable for multi-agent reinforcement learning problems with continuous action spaces. The Actor network (policy network) is a 3-layer fully connected network: a 5-dimensional input layer (corresponding to the state vector dimension), 128-dimensional and 64-dimensional hidden layers, and a 3-dimensional output layer (corresponding to the action vector dimension). It uses the ReLU activation function and a Tanh output layer (which restricts actions to a specific range). Range, then mapped to The Critic network (value function network) is a 4-layer fully connected network: input layer... Dimensions (including 5 dimensions of self-state, 3 dimensions of self-action, and others) Actions of an agent The Critic network requires global information for centralized training, thus needing input from the action information of all agents. It has 256-dimensional and 128-dimensional hidden layers, and a 1-dimensional output layer (Q-value), using the ReLU activation function. In actual deployment, the Critic network is only used during the training phase; only the Actor network is needed during the inference phase, therefore, action information from other agents is not required during inference. The policy gradient update uses the formula:

[0124]

[0125] in For intelligent agents Expected cumulative reward For state distribution, The gradient matrix of the policy network with respect to the parameters. Here is the gradient vector of the Critic network for the action. For policy network parameters. Training hyperparameters are set as follows: learning rate. Using the Adam optimizer ( The experience replay buffer size is Batch size is 64, discount factor .

[0126] (2) Implementation of mid-level intelligent agents

[0127] Each rack agent manages 40 servers, responsible for task scheduling and load balancing within the rack. Its state space contains aggregated information: average CPU utilization within the rack. Average memory utilization Total number of tasks (Assuming a maximum queue length of 10 per server), total rack power consumption (Unit: Watts, assuming a maximum power consumption of 1000W per server), Task distribution variance (Reflects the degree of load balancing). State vector Defined as:

[0128]

[0129] All features are normalized to Scope. The action space includes: task migration instructions (specifying which servers receive the task, represented by a 40-dimensional binary vector), and resource reservation ratio. Load balancing strategy parameters The QMIX algorithm is used for training. This algorithm achieves centralized training and distributed execution through value function decomposition, making it suitable for collaborative scenarios involving mid-level agents. The global Q-value function is decomposed as follows:

[0130]

[0131] in The global Q value, The set of states of all mid-level agents. For the set of actions of all mid-level agents, For the first The local Q-value function of a mid-level agent Hybrid networks. The network consists of 3 fully connected layers (61-dimensional input layer: 60 local Q-values ​​+ 1 global state feature, 128-dimensional and 64-dimensional hidden layers, and 1-dimensional output layer), ensuring that the monotonicity constraint is satisfied. For all This constraint is established by using a nonnegative weight matrix and an absolute value activation function. This constraint ensures that an increase in the local Q-value of each agent does not lead to a decrease in the global Q-value, thereby promoting cooperation among agents.

[0132] (3) Implementation of upper-level intelligent agents

[0133] Each data center agent manages 20 racks and is responsible for global coordination across data centers. The MAPPO algorithm is employed, with learning strategies optimized through near-end policy improvements. The reward function of the upper-layer agent focuses on global load balancing and cross-data center task routing efficiency.

[0134] III. Implementation of Communication Mechanism

[0135] After implementing the hierarchical multi-agent architecture, each agent needs to establish a communication mechanism to exchange information and collaborate on decision-making. This embodiment describes in detail the specific implementation of the communication mechanism, corresponding to the agent communication mechanism and information sharing module in the invention.

[0136] (1) Communication topology construction

[0137] Based on the system architecture, a three-layer communication topology is constructed: bottom-layer agents communicate with other agents within the same rack (40 agents per rack forming a complete graph); middle-layer agents communicate with other middle-layer agents within the same data center (20 agents per data center forming a complete graph); and top-layer agents communicate fully interconnectedly (3 top-layer agents forming a complete graph). This communication topology reflects the physical hierarchy of the system, enabling agents to exchange information with relevant nodes while controlling communication overhead.

[0138] Using graph neural networks to model communication relationships, for the underlying intelligent agents its neighbor set It includes 39 other agents within the same rack. Each agent's initial feature vector. From its state vector Obtained through linear projection.

[0139] Graph convolution operation iteratively updates feature representation:

[0140]

[0141] in For the first layer( Intelligent agents The feature representation vector, The number of convolutional layers in the graph. For intelligent agents The set of neighbors (containing 39 agents in the same rack). For intelligent agents The number of neighbors, For intelligent agents The number of neighbors, and The first The learnable weight matrix and bias vector of the layer, It is the ReLU activation function. These are the symmetric normalization coefficients (using the standard normalization method for graph convolutional networks to ensure the stability of feature updates). After... After layer graph convolution, the agent Features It includes information about its two-hop neighbors, that is, information about all agents in the entire rack, enabling each agent to obtain a global view of the rack and thus make better resource allocation decisions.

[0142] (2) Attention mechanism collaboration

[0143] After completing graph convolution feature extraction, a 4-head attention mechanism is used to calculate the collaboration weights between agents to further improve the collaboration efficiency. For each agent... Calculate its relationship with neighboring intelligent agents Attention weights:

[0144]

[0145] in For the process Intelligent agents after layer graph convolution Feature representation, and These are the projection matrices for the query and the key, respectively. For attention weight vectors, This represents a vector concatenation operation. For intelligent agents The set of neighbors (the other 39 agents in the same rack). To sum the index variable (and avoid conflicts with other variables), the LeakyReLU negative slope is set to 0.2. Attention weights. Represents intelligent agents For intelligent agents The level of attention given to it.

[0146] Collaborative features are obtained through weighted aggregation:

[0147]

[0148] in The projection matrix of the value, For intelligent agents The collaborative feature vector. This mechanism enables agents to adaptively focus on other agents with similar loads or complementary resources, improving collaborative efficiency. For example, when agents... When the load is high, it will pay more attention to agents with lower loads in order to migrate tasks and thus automatically achieve load balancing.

[0149] IV. Implementation of a Balanced Approach to Competition and Cooperation

[0150] After the communication mechanism is implemented, the agents need to make resource scheduling decisions based on the information obtained from the communication. However, there are both competitive relationships (competing for scarce resources) and cooperative relationships (jointly optimizing the global goal) among the agents. This embodiment describes in detail the specific implementation of the competition and cooperation balancing mechanism, corresponding to the competition and cooperation balancing mechanism module in the invention.

[0151] (1) Design of multi-objective reward function

[0152] Based on the cooperative features obtained through the communication mechanism, the underlying agent needs to design a reward function to guide policy learning. The reward function of the underlying agent is designed as a weighted combination of local and global rewards:

[0153]

[0154] in For intelligent agents Total reward value, This is the overall resource utilization rate (the average of CPU, memory, and storage utilization rates). Normalized energy consumption (current energy consumption divided by maximum power consumption of 1000W). The normalized task queue length is calculated by dividing the current queue length by the maximum queue length of 100. For the global reward, weight coefficients of 0.4, 0.3, 0.2, and 0.1 correspond to the weights of resource utilization, energy consumption, queue length, and global reward, respectively, satisfying the following conditions: .

[0155] The global reward is designed with multi-objective weighted penalties:

[0156]

[0157] in The normalized variance of resource utilization for all 2400 agents (variance divided by the maximum possible variance of 0.25) reflects the degree of system load balancing. Total system energy consumption (unit: watts, maximum value is 2400 servers × 1000W). The maximum task queue length among all agents is represented by the weight coefficients 0.4, 0.4, and 0.2, corresponding to the weights for load balancing, total energy consumption, and maximum latency, respectively. This design incentivizes agents to improve their own performance through local rewards (the first three items) and promotes cooperation among agents through a global reward (the last item), achieving load balancing and energy consumption optimization. The global reward is the same for all agents, enabling them to optimize their own performance while collectively optimizing the global objective.

[0158] (2) Training process

[0159] The agent is trained using experience replay and policy gradient methods, and a balance between competition and cooperation is achieved through Nash equilibrium solution. The training process consists of three stages:

[0160] Phase 1: Independent Training (100,000 steps)

[0161] Each agent learns independently, without considering the behavior of other agents, and learns basic resource scheduling strategies. During this stage, each agent only optimizes its own reward. It does not introduce global rewards or communication mechanisms. Training uses the MADDPG algorithm, but at this stage, the Critic network only considers its own state and actions, without considering the actions of other agents. Therefore, the policy gradient update formula simplifies to:

[0162]

[0163] in For the distribution of experience in the experience playback buffer, For empirical data sampled from the experience replay buffer (containing only agent data) status and actions ), The gradient of the policy network with respect to the parameters. The gradient of the Critic network with respect to the action (at this stage, the Critic network) (Only its own state and actions are input, without inputting actions from other agents). This stage enables the agent to acquire preliminary scheduling capabilities, allowing it to make basic resource allocation decisions based on its own state.

[0164] Phase Two: Collaborative Training (500,000 steps)

[0165] By introducing a global reward and communication mechanism, the agent learns to optimize the global objective through collaboration. The reward function at this stage comprises two parts: a local reward and a global reward.

[0166]

[0167] Where the balance coefficient The value is gradually increased from 0.05 to 0.1, enabling the agent to gradually learn to balance its own interests with the overall interests. Simultaneously, a graph neural network communication mechanism is introduced, allowing the agent to obtain information about its neighbors and make better collaborative decisions. At this stage, the agent gradually learns to optimize its own performance while simultaneously promoting the optimization of the global objective.

[0168] Phase 3: Competitive Balance Training (400,000 steps)

[0169] Training is conducted in resource-scarce scenarios, where agents learn to cooperate in a competitive environment to reach a Nash equilibrium policy. This stage achieves Nash equilibrium through experience replay and policy gradient methods. Each agent samples experience data from the experience replay buffer and updates its policy using the policy gradient method. Through multiple iterations, the policy converges to Nash equilibrium. At Nash equilibrium, no agent can obtain a higher reward by unilaterally changing its policy, thus achieving a stable cooperative state. This stage enables the system to remain stable and efficient under high load.

[0170] After training, the system achieved a resource utilization rate of 85.3% on the test set, a load balance (1-coefficient of variation) of 0.92, a total energy consumption reduction of 18.7%, and a reduction in average task latency of 23.5%.

[0171] V. Implementation of Dynamic Environmental Adaptation

[0172] After implementing the competition and cooperation balancing mechanism and training the strategy, the system possesses basic resource scheduling capabilities. However, the distributed system operating environment is highly dynamic and needs to adapt quickly to environmental changes. This embodiment describes in detail the specific implementation of the dynamic environment adaptation mechanism, corresponding to the dynamic environment adaptation mechanism module in the invention.

[0173] (1) Online learning mechanism

[0174] After system deployment, continuously monitor environmental change indicators, including load volatility. and node failure rate When load fluctuations exceed a threshold are detected. Or the node failure rate exceeds the threshold When the time comes, an online learning mechanism is triggered. An incremental learning algorithm is used to quickly update the strategy, adapting to the new environment while maintaining existing knowledge.

[0175]

[0176] in For intelligent agents In the The policy network parameter vector at the next update To adapt the learning rate (usually set to the initial training learning rate) of (To ensure stability and avoid performance degradation due to excessive updates) To adapt to the loss function, calculations are performed based on empirical data from the new environment. Strategy loss, For parameters The gradient operator is used. Online learning uses a sliding window to store the most recent 10,000 empirical data points, updating the policy parameters every 1,000 steps to avoid overfitting to new data and forgetting old knowledge. Experiments show that the system can adapt to load fluctuations within 5 minutes and node failures within 10 minutes, with a performance degradation of less than 5%.

[0177] (2) Application of transfer learning

[0178] When the system is deployed to a new data center, transfer learning technology is used for rapid adaptation. The specific implementation process is as follows:

[0179] First, in the source data center (the already trained scenario), the model has already learned a general resource scheduling strategy. During transfer learning, the policy network from the source data center is then applied. As an initial model, the parameters of the feature extraction layers (the first two layers, i.e., from the input layer to the first hidden layer and from the first hidden layer to the second hidden layer) are directly transferred to the new data center because these layers learn general feature representations (such as resource utilization patterns, load distribution characteristics, etc.) that are similar across different data centers.

[0180] Then, a small amount of data (approximately 10,000 experience points) was collected in the real environment of the new data center. The model parameters are updated using a fine-tuning strategy. The policy network is divided into feature extraction layers. and decision-making level The parameters of the feature extraction layer are fine-tuned using a small learning rate. While maintaining general characteristics, it adapts to the specificities of the new data center; the decision layer parameters are trained using a normal learning rate. ( ), quickly learn the decision-making strategies of new data centers.

[0181] The loss function for the fine-tuning process is:

[0182]

[0183] in The strategic loss due to the new data center The regularization coefficient is used to prevent the parameters of the feature extraction layer from deviating too far from the parameters of the source data center. Through feature extraction and fine-tuning, the model can quickly adapt to the resource scheduling requirements of the new data center. Experiments show that transfer learning reduces the training time in the new scenario from 1 million steps to 100,000 steps, reduces the training cost by 90%, and achieves a performance loss of less than 5%.

[0184] VI. Multi-objective optimization implementation

[0185] After implementing the dynamic environment adaptation mechanism, the system possesses basic resource scheduling and adaptability capabilities. However, in practical applications, it is necessary to simultaneously optimize multiple conflicting objectives, and a single strategy cannot meet all requirements. This embodiment details the specific implementation of multi-objective Pareto optimization, corresponding to the multi-objective Pareto optimization module in the invention. The system simultaneously optimizes three objectives: task delay... (Unit: milliseconds), System energy consumption (Unit: Watt), Variance of resource utilization rate (Reflects the degree of load balancing).

[0186] The specific implementation process is as follows: First, an independent Q-value function network is constructed for each optimization objective. Three independent Critic networks are constructed: Corresponding to the task delay target, Corresponding to the system energy consumption target, Corresponding to the load balancing target, where , , These are the parameters of each network. The multi-objective Q-value vector is defined as:

[0187]

[0188] in For a multi-objective Q-value vector, The Q-value function represents the target delay for the task (a larger value indicates a smaller delay). Q is the Q-value function of the system's energy consumption target (the larger the value, the smaller the energy consumption). The Q-value function is used to determine the load balancing objective (a larger value indicates a more balanced load). For system status, For action.

[0189] Then, the Pareto front is generated using the weight vector method. For different weight combinations... (satisfy , Define the weighted Q-value function:

[0190]

[0191] The weighted Q-value function is optimized using the MADDPG algorithm to obtain the optimal strategy under the corresponding weight combination. Using uniform grid sampling, 15 different weight combinations were generated (e.g., , , , Each weight combination trains an optimal policy, generating 15 solutions on the Pareto front. Each solution corresponds to an optimal policy under a certain weight combination, which is on the Pareto front, meaning that no other policy is superior to this policy on all objectives.

[0192] The system provides an interactive interface, allowing administrators to select weight combinations based on current business needs and preferences, and the system automatically switches to the corresponding optimal strategy. The experiment generated 15 Pareto optimal solutions, covering options ranging from "low latency first" (…). From "low energy consumption priority" to "low energy consumption priority" Various strategies were employed to achieve flexible multi-objective optimization.

[0193] VII. System Performance Evaluation

[0194] A system was deployed in a cloud computing data center with 2400 servers and run for 30 days to collect performance data. The experimental results show that:

[0195] Resource utilization rate: The average resource utilization rate reached 85.3%, which is 18.3% higher than the traditional centralized scheduling method (72.1%).

[0196] Load balancing: The load balancing degree (1 - coefficient of variation) reached 0.92, which is 21.1% higher than the traditional method (0.76).

[0197] Energy consumption optimization: The total energy consumption of the system was reduced by 18.7%, mainly due to the agent learning a better resource allocation strategy, which reduced resource waste.

[0198] Task latency: Average task latency decreased by 23.5%, P99 latency decreased by 31.2%, and service quality was significantly improved.

[0199] Scalability: When the system is scaled up to 4,800 servers, the scheduling latency increases by only 15%, while the traditional centralized approach increases by 120%, demonstrating good scalability.

[0200] Robustness: When simulating a 10% node failure, the system automatically adjusts within 5 minutes, with a performance degradation of less than 5%, demonstrating good fault tolerance.

[0201] In summary, this invention innovatively designs a three-layer, multi-agent architecture (bottom layer server-level, middle layer rack-level, and top layer data center-level), decomposing the complex global resource scheduling problem into multiple layers of local optimization problems. The bottom-layer agent is responsible for fine-grained resource allocation, the middle-layer agent for local coordination, and the top-layer agent for global optimization. This layered collaboration achieves scalable distributed scheduling. This architecture fully leverages the hierarchical structure of distributed systems, avoiding the limitations of a single-layer architecture.

[0202] This invention employs different multi-agent reinforcement learning algorithms for agents at different levels: the bottom layer uses MADDPG to learn continuous action policies, the middle layer uses QMIX to achieve value function decomposition and centralized training with distributed execution, and the top layer uses MAPPO for policy optimization. By fusing these algorithms, the advantages of each are fully utilized to achieve efficient multi-agent collaborative learning.

[0203] This invention employs graph neural networks to model the communication topology between agents. Through multi-layer graph convolution operations, each agent can aggregate information from its neighbors to obtain a global view of the local region. This mechanism enables agents to make near-globally optimal decisions with only local information, effectively solving the problem of incomplete information in distributed systems.

[0204] This invention designs a multi-head attention mechanism that enables agents to adaptively learn collaborative patterns with other agents. Attention weights reflect the closeness of collaboration between agents, allowing agents to focus on other agents with whom they have close collaborative relationships, thus improving collaboration efficiency. This mechanism enables the system to dynamically adapt to different collaborative scenarios.

[0205] This invention balances competition (competition for scarce resources) and cooperation (joint optimization of the global objective) among agents by designing a multi-objective reward function that includes local and global rewards. Local rewards encourage agents to optimize their own performance, while global rewards promote cooperation among agents. By adjusting the balance coefficient, a dynamic balance between competition and cooperation is achieved, avoiding the "prisoner's dilemma" and guiding agents to learn a Nash equilibrium strategy.

[0206] This invention designs online learning and transfer learning mechanisms, enabling the system to quickly adapt to environmental changes. Online learning, through incremental updates of policy parameters, rapidly adapts to load fluctuations and node failures while maintaining stability. Transfer learning, through feature extraction and fine-tuning, allows the model to quickly adapt to new scenarios, significantly reducing training costs. This mechanism endows the system with good robustness and adaptability.

[0207] This invention employs multi-objective value function decomposition, transforming a multi-objective optimization problem into multiple single-objective sub-problems. Pareto optimal solutions are generated by optimizing different weight combinations. This method allows decision-makers to select strategies based on current preferences, achieving flexible multi-objective optimization and meeting the needs of various application scenarios.

[0208] This invention employs a distributed architecture, with each agent operating independently and communicating asynchronously via message queues. It utilizes experience replay and a stable training process for the target network. It supports pre-training in simulation environments and fine-tuning in real-world environments, enhancing the system's practicality and robustness. This implementation technique enables the system to operate efficiently in large-scale distributed environments.

[0209] Through the detailed description of this embodiment, those skilled in the art can clearly understand the technical solution of the present invention, and can implement the present invention in actual application scenarios according to the described method and system configuration, so as to achieve efficient and scalable distributed resource scheduling optimization.

[0210] The above embodiments are merely typical illustrative methods of the present invention, and the scope of protection of the present invention is not limited thereto. All equivalent substitutions and improvements made under the concept of the present invention should fall within the scope of protection. It should be emphasized that any modifications or minor adjustments made by those skilled in the art without departing from the basic principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A distributed resource scheduling optimization system based on multi-agent reinforcement learning, characterized in that, It includes bottom-level intelligent agents and top-level intelligent agents. The bottom-level intelligent agents include bottom-level server-level intelligent agents and mid-level rack-level intelligent agents, which are responsible for local resource scheduling. The top-level intelligent agents include top-level data center-level intelligent agents, which are responsible for global resource coordination. Specifically, the bottom-level server-level intelligent agents are responsible for allocating CPU, memory, and storage resources for a single server; the mid-level rack-level intelligent agents are responsible for task scheduling and load balancing of servers within a rack; and the top-level data center-level intelligent agents are responsible for global resource coordination and task routing across data centers.

2. The distributed resource scheduling and optimization system based on multi-agent reinforcement learning according to claim 1, characterized in that, The system design methodology is as follows: S1. Design a layered multi-agent architecture, dividing the distributed system into two layers: a lower-level agent and an upper-level agent, to achieve scalable distributed scheduling. S2. Design communication protocols and information sharing mechanisms between intelligent agents to enable them to exchange information and thus obtain a more comprehensive system view. S3. After completing the extraction of agent communication and cooperation features, by designing a multi-objective reward function and game mechanism, the competition and cooperation are balanced, and the agent is guided to promote the optimization of the global goal while optimizing its own performance. S4. After completing the design of the reward function and the balance between competition and cooperation, the agent learns the optimal policy through reinforcement learning algorithm; a dynamic adaptation mechanism is designed so that the system can quickly detect environmental changes and adjust its policy to maintain the efficient operation of the system. S5. Through multi-objective reinforcement learning methods, a Pareto optimal solution set is generated to support decision-makers in selecting the optimal strategy based on current preferences.

3. The distributed resource scheduling and optimization system based on multi-agent reinforcement learning according to claim 2, characterized in that, Step S1 includes: S11, Underlying Intelligent Agent Design The underlying intelligent agents are responsible for local resource scheduling, with one underlying intelligent agent corresponding to each computing node, storage node, or network node; assuming there are a total of The first underlying intelligent agent, the first The local state space of each agent is: The action space is The reward function is ,in , representing intelligent agents The current state, Represents intelligent agents The chosen action Indicates the next state after the action is performed; The underlying intelligent agents employ deterministic policy networks. Output continuous actions, where The parameters are the policy network parameters; to learn the optimal policy, the MADDPG algorithm is used, with each agent maintaining an Actor network and a Critic network; the Critic network... Assessment in status Next action And other intelligent agents perform actions The expected cumulative reward at that time, of which These are the parameters for the Critic network. The policy gradient update formula for the underlying agent is: in For intelligent agents Expected cumulative reward For state distribution, The gradient of the policy network with respect to the parameters. The gradient of the Critic network for the action; S12, Upper-layer intelligent agent design The upper-level intelligent agent is responsible for global resource coordination. Assume the system has a total of... There are three upper-level intelligent agents, each managing a group of lower-level intelligent agents; The state space of each upper-level intelligent agent It contains aggregated state information of all underlying agents it manages, and its action space. Includes globally coordinated actions; The upper-layer intelligent agent is trained using the QMIX algorithm, which achieves centralized training and distributed execution through value function decomposition. The core idea of ​​QMIX is to decompose the global Q-value function. Decomposed into a mixture of the local Q-value functions of each agent: in This is the global state. For the set of actions of all upper-level agents, For the first The local Q-value function of each upper-level agent For hybrid networks, ensure that monotonicity constraints are met: For all Established.

4. A distributed resource scheduling and optimization system based on multi-agent reinforcement learning according to claim 2, characterized in that, Step S2 includes: S21. Communication Topology Construction First, a graph neural network is used to model the communication topology between agents; let the agent communication graph be... ,in A set of nodes, containing all A low-level intelligent agent and A higher-level intelligent agent, Let edge be the set of edges, representing the communication relationships between agents; each agent... Maintain an initial feature vector This indicates its initial state information. For feature dimensions; The feature representation of the agent is iteratively updated through multi-layer graph convolution operations, enabling each agent to aggregate information from its neighboring agents: in For the first In-layer intelligent agents The feature representation vector, The number of convolutional layers in the graph. For intelligent agents The neighborhood group, For intelligent agents The number of neighbors, For intelligent agents The number of neighbors, and The first The learnable weight matrix and bias vector of the layer, For activation function, The coefficients are symmetric normalized; after... After layer graph convolution, the agent Features Includes its Jump to neighbor information; S22, Attention Mechanism Collaboration After completing the graph convolution feature extraction, a multi-head attention mechanism is used to learn the collaborative patterns among agents; For intelligent agents Calculate its relationship with other intelligent agents Attention weights: in and They were respectively through Intelligent agents after layer graph convolution and The feature representation vector, and These are the projection matrices for the query and the key, respectively. For attention weight vectors, This represents a vector concatenation operation. For intelligent agents The neighborhood group, For feature dimension, For summation index variables; attention weights Represents intelligent agents For intelligent agents The degree of attention is indicated by the value of the weight. The larger the value, the closer the cooperation between the two. This weight is normalized by Softmax to ensure that the sum of the attention weights of all neighbors is 1, so that the attention weights form a probability distribution. intelligent agent The collaborative features are obtained through weighted aggregation: in The projection matrix of the value, For the process Intelligent agents after layer graph convolution The feature representation vector, For intelligent agents The collaborative feature vectors are derived by incorporating relevant information from neighboring agents.

5. A distributed resource scheduling optimization system based on multi-agent reinforcement learning according to claim 2, characterized in that, Step S3 includes: S31. Design of Multi-Objective Reward Function Each agent's reward function consists of two parts: a local reward and a global reward. A balance between competition and cooperation is achieved through a weighted combination. in For intelligent agents Total reward value, For intelligent agents The local reward reflects its own performance. This is a global reward that reflects the overall performance of the system. The balancing coefficient controls the weights of local and global optimization; when When the agent is small, it focuses more on optimizing its own performance; when... When the size is large, the agent focuses more on global performance optimization; The local reward design is a weighted combination of multiple objectives: in For intelligent agents Comprehensive resource utilization rate For intelligent agents Normalized energy consumption For intelligent agents The normalized task queue length, Let be the weighting coefficient, satisfying Through positive rewards ( ) Encourage agents to improve resource utilization through negative penalties ( Encourage agents to reduce energy consumption and task latency to achieve local multi-objective optimization; The global reward is designed with multi-objective weighted penalties: in The normalized variance of resource utilization for all agents reflects the degree of system load balancing; the smaller the value, the more balanced the load. This is the normalized value of the system's total energy consumption. This is the normalized value of the maximum task queue length among all agents. Let be the weighting coefficient, satisfying ;; S32. Solving Nash Equilibrium By designing a reward function, the agent is guided to learn a Nash equilibrium policy. The Nash equilibrium is solved using experience replay and policy gradient methods. The specific implementation process is as follows: First, each agent collects experience data by interacting with the environment. The data is stored in the experience replay buffer. The experience replay buffer adopts a first-in-first-out strategy and maintains a fixed size. When the buffer is full, the oldest experience is deleted. During training, a batch of experience data is randomly sampled from the buffer to break the temporal correlation of the data and improve training stability. Then, the agent's policy is updated using the policy gradient method; for the agent... Its policy gradient update formula is: in For the distribution of experience in the experience playback buffer, The empirical data sampled from the experience replay buffer contains the agent. status ,action Actions with other intelligent agents , The gradient of the policy network with respect to the parameters. Let be the gradient of the Critic network for actions; through iterative updates, the policies of each agent gradually converge to a Nash equilibrium policy; under Nash equilibrium, for each agent... Its current strategy satisfy: That is, any agent that deviates from the Nash equilibrium policy alone cannot obtain a higher reward, thus achieving a stable cooperative state; through multiple rounds of iterative training, the agent can learn a policy close to the Nash equilibrium, achieving an effective balance between competition and cooperation.

6. A distributed resource scheduling and optimization system based on multi-agent reinforcement learning according to claim 2, characterized in that, Step S4 includes: S41. Online Learning and Strategy Updates When a significant change in the environment is detected, the system triggers an online learning mechanism to quickly update the agent's strategy. An incremental learning algorithm is used to fine-tune the strategy network parameters using new environmental data while maintaining the original knowledge, thus avoiding knowledge loss caused by training from scratch. The specific implementation process is as follows: First, the system continuously monitors environmental change indicators, including load volatility. and node failure rate ,in, and The average loads for the new and old environments are respectively. The number of faulty nodes. The total number of nodes; when or At that time, online learning is triggered; then, the system collects experience data in the new environment. ,in For the amount of experience in the new environment; adaptive loss function Defined as a weighted combination of policy loss and value function loss: in For strategic losses, The advantage function measures the advantage of an action relative to the average action. Calculation, where It is a state-value function; For value function loss, For the target Q value, use a deterministic strategy. Rather than the maximum value, because MADDPG deals with a continuous action space. and These are the weighting coefficients; The formula for updating the strategy parameters is: in For intelligent agents In the The policy network parameter vector at the next update For parameter dimensions, To adapt to the learning rate, For parameters The gradient operator; S42, Rapid Adaptation Through Transfer Learning When the system is deployed to a new scenario, transfer learning technology is used to transfer the knowledge of the trained model to the new scenario; the specific implementation process is as follows: First, in the source scenario, the model has already learned a general resource scheduling strategy. During transfer learning, the policy network trained in the source scenario is used... As an initial model, the parameters of the feature extraction layer are directly transferred to the new scene; Then, in the new scenario, a fine-tuning strategy is used to update the model parameters; specifically, the policy network is divided into two parts: a feature extraction layer... and decision-making level The parameters of the feature extraction layer are fine-tuned using a small learning rate to maintain general features while adapting to the specificities of new scenarios; the parameters of the decision layer are trained using a normal learning rate to quickly learn decision-making strategies for new scenarios. The loss function for the fine-tuning process is: in For the strategy loss in the new scenario, The regularization coefficient is . This is a regularization term.

7. A distributed resource scheduling optimization system based on multi-agent reinforcement learning according to claim 2, characterized in that, Step S5 includes: using a multi-objective value function decomposition method to transform the multi-objective optimization problem into multiple single-objective sub-problems, each sub-problem corresponding to an optimization objective. The specific implementation process is as follows: First, construct an independent Q-value function network for each optimization objective. Assume the system has... One optimization objective, construct An independent Critic network Each network corresponds to an optimization objective, where For the first The parameters of the network; the multi-objective Q-value vector is defined as: in For a multi-objective Q-value vector, For the target quantity, For the first The Q-value function of an optimization objective. For system status, For action; Then, the Pareto front is generated using the weight vector method; for different weight combinations ;satisfy and Define the weighted Q-value function: The weighted Q-value function is optimized using a reinforcement learning algorithm to obtain the optimal policy under the corresponding weight combination. By enumerating or sampling different weight combinations, multiple solutions on the Pareto front are generated; each solution corresponds to the optimal policy under a certain weight combination, and these policies are on the Pareto front, that is, no other policy is better than this policy on all objectives. Decision-makers select weight combinations based on current business needs and preferences, and the system automatically switches to the corresponding optimal strategy to achieve flexible multi-objective optimization.

Images