Distributed resource scheduling optimization system based on multi-agent reinforcement learning

Abstract

The present application relates to the technical field of distributed computing resource scheduling and artificial intelligence, and particularly relates to a distributed resource scheduling optimization system based on multi-agent reinforcement learning.The present application comprises a state perception module for collecting multi-dimensional physical state information of a distributed system, a digital twin module for constructing a digital twin graph structure using a graph neural network and multi-dimensional physical state information, a competition prediction module for predicting resource demand prediction sequences of each resource node, an early warning module for determining resource competition intensity and generating early warning information, a hierarchical decision module for determining a scheduling strategy of multi-agents, a weight distribution module for adjusting decision weights of each agent in joint decision-making, a simulation execution module for simulation execution by the digital twin module, a scheduling execution module for determining a maximum node deviation value, and a processing analysis module for determining a node feature aggregation layer update rate of the graph neural network and a competition intensity threshold value based on the maximum node deviation value.The present application improves the efficiency of resource scheduling.

Description

Technical Field

[0001] This invention relates to the field of distributed computing resource scheduling and artificial intelligence, and in particular to a distributed resource scheduling optimization system based on multi-agent reinforcement learning. Background Technology

[0002] With the rapid development of cloud computing, edge computing, and big data processing technologies, resource scheduling problems in distributed systems are becoming increasingly complex. Existing technologies mainly suffer from the following shortcomings: First, traditional centralized scheduling methods suffer from single points of failure and scalability bottlenecks. Existing centralized scheduling methods collect operational status information from each node through a central controller and then make unified decisions, which faces the risk of single points of failure and is difficult to handle large-scale dynamic tasks and complex network topologies.

[0003] Second, heuristic or rule-based methods lack dynamic adaptability. Methods based on heuristic rules such as shortest path, minimum task completion time, and load balancing perform well in small-scale static scenarios, but are prone to getting trapped in local optima in complex dynamic environments.

[0004] Third, existing reinforcement learning methods have shortcomings in multi-agent collaboration. Recent scheduling methods based on multi-agent reinforcement learning have begun to adopt a hierarchical collaborative architecture, but existing technologies generally suffer from the following problems: agents lack effective predictive collaboration mechanisms, and each agent can only respond passively after a conflict occurs; resource competition between heterogeneous nodes is difficult to identify in advance and proactively avoid; and the role division in multi-agent systems is fixed and rigid, making it impossible to dynamically adjust the decision weights and collaboration modes of each agent based on real-time load conditions.

[0005] Fourth, the existing system lacks the ability to proactively warn and pre-schedule resource contention risks. Adjustments are made only after resource conflicts actually occur, rather than anticipating and mitigating them beforehand. This reactive scheduling model leads to system lag when facing sudden workloads, and scheduling efficiency fails to meet increasingly stringent quality-of-service requirements.

[0006] Fifth, the application of digital twin technology in the field of resource scheduling is still in its early stages. Although digital twin technology has been applied in fields such as industrial manufacturing and urban planning, there is still a lack of effective technical solutions in the field of distributed resource scheduling for constructing a digital twin environment that can map the physical resource status in real time and be used for reinforcement learning training and policy pre-validation.

[0007] Chinese Patent Publication No. CN114745317A discloses a computing task scheduling method and related equipment for computing power networks. In the computing power network scenario, a task scheduling method based on the Multi-agent Proximal Proxy Optimization (MAPPO) algorithm is proposed. The network computing model includes a computing resource agent and a network resource agent. The two agents share the Critic network. Through iterative optimization of the network computing model, the final output is the destination computing node and forwarding path of the computing task.

[0008] It is evident that existing technologies have the following problems: existing distributed resource scheduling systems lack the ability to predict resource competition risks in multi-agent collaborative decision-making and cannot achieve collaborative scheduling in dynamic heterogeneous environments, resulting in low efficiency in resource scheduling. Summary of the Invention

[0009] To address this, the present invention provides a distributed resource scheduling optimization system based on multi-agent reinforcement learning, which overcomes the problem that existing distributed resource scheduling systems lack the ability to predict resource competition risks in multi-agent collaborative decision-making, and cannot achieve collaborative scheduling in dynamic heterogeneous environments, thus resulting in low resource scheduling efficiency.

[0010] To achieve the above objectives, this invention provides a distributed resource scheduling optimization system based on multi-agent reinforcement learning, comprising: The state awareness module is deployed on each resource node to collect multi-dimensional physical state information of the distributed system. A digital twin module is used to construct a digital twin graph structure based on a graph neural network and the multidimensional physical state information. The graph neural network includes a node feature aggregation layer, and the node feature aggregation layer has a dynamically adjustable parameter update rate. The competition prediction module is used to predict the resource demand prediction sequence of each resource node within a preset time window based on the digital twin graph structure and historical state sequence. The early warning module is used to determine the resource competition intensity at a preset time step based on the resource demand prediction sequence, and to generate early warning information based on the resource competition intensity and the competition intensity threshold. A hierarchical decision-making module is used to determine the scheduling strategy of the hierarchical decision-making link of the multi-agent based on the digital twin graph structure and the early warning information; The weight allocation module is used to adjust the decision weight of each agent in joint decision-making based on the role allocation result. The role allocation result is determined based on the adaptability of each agent to several predefined role types calculated from the current system state and the warning information. The simulation execution module transmits the weighted scheduling strategy to the digital twin module so that the digital twin module can perform simulation execution. The weighted scheduling strategy is determined based on feeding back the decision weights to the hierarchical decision module. The scheduling execution module is used to determine the maximum node deviation value by comparing the state evolution trajectory generated after executing the scheduling strategy based on the digital twin graph structure with the actual execution result generated by distributing the schedulable strategy to the corresponding resource nodes. The schedulable strategy is determined based on the state evolution trajectory. The processing and analysis module is used to determine the node feature aggregation layer update rate and competition intensity threshold of the graph neural network based on the maximum node deviation value.

[0011] Furthermore, the processing and analysis module includes: a state analysis submodule, which is used to determine the scheduling state of the distributed resources based on the maximum node deviation value; and an adjustment submodule, which is used to determine whether to determine the node feature aggregation layer update rate of the graph neural network based on the normalized deviation amplitude based on the deviation attribution coefficient, or to determine the competition intensity threshold based on the demand forecast deviation fluctuation coefficient.

[0012] Furthermore, the adjustment submodule includes: a deviation amplitude calculation unit, which is used to determine the normalized deviation amplitude based on the maximum node deviation value; and a determination unit, which is used to increase the node feature aggregation layer update rate based on the normalized deviation amplitude, wherein the increase in the node feature aggregation layer update rate is proportional to the normalized deviation amplitude.

[0013] Furthermore, the adjustment submodule is also used to reduce the update rate of the node feature aggregation layer based on the response lag time when the response lag time in the multidimensional physical state information is less than or equal to a preset lag threshold, and the reduction in the update rate of the node feature aggregation layer is inversely proportional to the response lag time.

[0014] Furthermore, the adjustment submodule is also used to determine the scheduling state after re-determining the node feature aggregation layer update rate, and to determine whether to determine the competition intensity threshold based on the demand prediction deviation fluctuation coefficient based on the scheduling state.

[0015] Furthermore, the competition prediction module includes: a deviation determination submodule, which is used to determine the prediction deviation between the predicted demand value and the observed value of each resource node in each resource dimension; a mean determination submodule, which is used to determine the mean of the prediction deviation based on the prediction deviation of each resource node in each resource dimension; and a fluctuation coefficient determination submodule, which is used to determine the fluctuation coefficient of the demand prediction deviation based on the mean of the prediction deviation corresponding to each resource node.

[0016] Furthermore, the adjustment submodule is also used to increase the competition intensity threshold based on the demand forecast deviation fluctuation coefficient, and the increase in the competition intensity threshold is proportional to the demand forecast deviation fluctuation coefficient.

[0017] Furthermore, the adjustment submodule is also used to determine whether to issue a dual-path failure alarm notification for excessive scheduling system deviation based on the scheduling state after redetermining the competition intensity threshold.

[0018] Furthermore, the early warning module includes: a competition determination submodule, which is used to determine the resource competition coefficient between any two resource nodes based on the resource type, resource capacity, and task load characteristics of each resource node in the digital twin graph structure; a graph construction submodule, which is used to construct a competition relationship graph based on each of the resource competition coefficients; and an intensity prediction submodule, which is used to input the resource demand prediction sequence into the competition relationship graph to predict the resource competition intensity at a preset time step.

[0019] Furthermore, the weight allocation module includes: a role definition submodule, which is used to predefine several agent role types; a state encoding submodule, which is used to determine the state feature vector of the system state based on the digital twin graph structure and the early warning information; a fitness calculation submodule, which is used to calculate the fitness of each agent and each role type based on the state feature vector; and a role negotiation submodule, which is used to exchange information based on the fitness to determine the role allocation result.

[0020] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention includes a state perception module for collecting multi-dimensional physical state information of a distributed system; a digital twin module for constructing a digital twin graph structure using graph neural networks and multi-dimensional physical state information; a competition prediction module for predicting the resource demand prediction sequence of each resource node; an early warning module for determining the resource competition intensity and generating early warning information; a hierarchical decision-making module for determining the scheduling strategy of multiple agents; a weight allocation module for adjusting the decision weights of each agent in joint decision-making; a simulation execution module for transmitting the weighted scheduling strategy to the digital twin module for simulation execution; a scheduling execution module for determining the maximum node deviation value; and a processing and analysis module for determining the node feature aggregation layer update rate and competition intensity threshold of the graph neural network based on the maximum node deviation value. This invention improves the efficiency of resource scheduling.

[0021] Furthermore, this invention accurately attributes the source of deviation to prediction error or model representation error based on the deviation attribution coefficient, and makes differentiated and targeted adjustments to the node feature aggregation layer update rate and competition intensity threshold, thereby avoiding ineffective iterations caused by blindly adjusting a single parameter, and further improving the efficiency of distributed resource scheduling optimization.

[0022] Furthermore, this invention increases the node feature aggregation layer update rate proportionally to the normalized deviation amplitude, which can more effectively drive the model calibration speed when the deviation is determined to be attributable to the representation error of the digital twin model. This enables the digital twin module to adaptively adjust the learning response speed under different deviation severity levels, thereby further improving the efficiency of resource scheduling.

[0023] Furthermore, in the process of increasing the update rate of the node feature aggregation layer, which enables the digital twin model to gradually approach the physical reality, the present invention uses the response lag time as a quantitative indicator of whether the model tracking speed meets the standard, and timely callback update rate, thereby effectively preventing model overfitting and parameter oscillation, and further improving the efficiency of resource scheduling.

[0024] Furthermore, this invention aggregates the prediction deviations of each resource node across each resource dimension into a node-level prediction deviation mean and a system-level demand prediction deviation fluctuation coefficient, enabling more effective adjustments in the future and thereby further improving the efficiency of resource scheduling.

[0025] Furthermore, this invention determines the competition intensity threshold based on the demand forecast deviation fluctuation coefficient, which can raise the threshold to increase the early warning threshold and filter out false early warnings caused by noise, thereby making the early warning information received by the hierarchical decision-making module more accurate.

[0026] Furthermore, this invention calculates static competition coefficients and constructs a competition relationship graph based on the resource type, resource capacity, and task load characteristics of each node in the digital twin graph structure. Then, the dynamic resource demand prediction sequence is input into the graph to predict the future competition intensity at a preset time step, thereby making the competition topology relationship and future demand trend more accurate, and further improving the efficiency of resource scheduling. Attached Figure Description

[0027] Figure 1 This is a structural block diagram of a distributed resource scheduling optimization system based on multi-agent reinforcement learning, according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating the steps of the distributed resource scheduling optimization method based on multi-agent reinforcement learning in an embodiment of the present invention. Figure 3 This is a flowchart illustrating the steps of determining the maximum node deviation value according to an embodiment of the present invention. Figure 4 This is a flowchart illustrating the steps of determining the competition intensity threshold based on the demand forecast deviation fluctuation coefficient in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0029] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0030] It should be noted that in the description of this invention, the terms "upper", "lower", "left", "right", "inner", "outer", etc., which indicate directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and is not intended to indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.

[0031] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0032] Please see Figure 1 The diagram shown is a structural block diagram of a distributed resource scheduling optimization system based on multi-agent reinforcement learning according to an embodiment of the present invention. The embodiment of the present invention includes: The state awareness module is deployed on each resource node to collect multi-dimensional physical state information of the distributed system. Specifically, multidimensional physical state information includes, but is not limited to: CPU utilization and frequency of each node, memory usage and access latency, storage IOPS and read / write latency, network bandwidth utilization and transmission latency, task queue length, task arrival rate and task type distribution, node temperature and power consumption. A digital twin module, connected to the state perception module, is used to construct a digital twin graph structure based on a graph neural network and the multidimensional physical state information. The graph neural network includes a node feature aggregation layer, and the node feature aggregation layer has a dynamically adjustable parameter update rate. Specifically, the digital twin module includes: The construction submodule is used to construct the initial system topology graph structure by treating each resource node as a graph node and the physical connection relationship between resource nodes as the graph edge. The vector generation submodule is used to encode the multidimensional physical state information of each resource node in the initial system topology graph structure using a graph neural network to generate the initial feature vector of each node. The update submodule is used to input the initial feature vector into the node feature aggregation layer. The node feature aggregation layer performs weighted aggregation of the features of adjacent nodes with a parameter update rate to update the aggregated features of each node. The structure building submodule is used to construct a digital twin graph structure based on the updated aggregated features of each node.

[0033] The competition prediction module is connected to the state perception module and the digital twin module respectively, and is used to predict the resource demand prediction sequence of each resource node within a preset time window based on the digital twin graph structure and historical state sequence. It is understood that this embodiment uses a time-series prediction network to predict resource demand prediction sequences. This embodiment does not impose specific limitations. Those skilled in the art can adaptively select and replace the time-series prediction network according to the temporal characteristics of actual resource demand changes, the length of the prediction time window, and the configuration of system computing resources. It is sufficient that it can predict the demand of each resource node in each resource dimension within a future preset time window based on the digital twin graph structure and historical state sequence.

[0034] The early warning module, which is connected to the competition prediction module, is used to determine the resource competition intensity at a preset time step based on the resource demand prediction sequence, and to generate early warning information based on the resource competition intensity and the competition intensity threshold. Specifically, the early warning module includes: The competition determination submodule is used to determine the resource competition coefficient between any two resource nodes based on the resource type, resource capacity and task load characteristics of each resource node in the digital twin graph structure. The resource competition coefficient is determined by multiplying the resource type overlap of the two resource nodes by the sum of the products of the load occupancy rates of the two nodes. The resource competition coefficient is positively correlated with both the type matching degree and the load pressure. The graph construction submodule is used to construct a competition relationship graph based on the competition coefficients of each resource. The intensity prediction submodule is used to input the resource demand prediction sequence into the competition relationship graph to predict the resource competition intensity at a preset time step. Understandably, the intensity prediction submodule maps the predicted demand increment of each node at each preset time step to each edge of the competition relationship graph. The combined value of the predicted demand increment of the two nodes connected to each edge at that time step is used as the dynamic weight. The inherent resource competition coefficient of each edge is weighted to obtain the dynamic competition weight of each edge at each preset time step. Then, the dynamic competition weights of several edges connected to the target node at the same time step in the future are summed to obtain the resource competition intensity of the target node at that time step. After traversing all nodes and all time steps, the resource competition intensity of each resource node in the future preset time step is obtained. This embodiment does not impose specific limitations. Those skilled in the art can adaptively select and replace the weight coefficients of each resource dimension, the edge weight aggregation method, and the normalization method of the competition intensity according to the node scale, the number of resource dimensions, and the early warning sensitivity requirements of the actual distributed system. It is only necessary to be able to quantify the degree of resource competition between any two resource nodes due to changes in resource demand in the future preset time step. The early warning submodule is used to generate early warning information when the resource competition intensity exceeds the competition intensity threshold.

[0035] A hierarchical decision-making module, which is connected to the digital twin module and the early warning module respectively, is used to determine the scheduling strategy of the hierarchical decision-making link of the multi-agent based on the digital twin graph structure and the early warning information; In this embodiment of the invention, the hierarchical decision-making module adopts a hierarchical collaborative architecture, including a bottom-level resource agent layer, a middle-level coordination agent layer, and an upper-level global agent layer, wherein... The underlying resource agent layer consists of local agents deployed on each resource node. Each local agent generates resource allocation actions for the corresponding node based on the local state information of the node in the digital twin graph structure using a first deep reinforcement learning network. The decision-making process of the local agent also receives pre-scheduling suggestions for the node in the early warning signal as a decision reference. The middle-layer coordination agent layer consists of coordination agents deployed at rack-level or region-level aggregation nodes. Each coordination agent receives local action information and local state information from the lower-level resource agents within its management domain, as well as domain-wide early warning signals. It uses a second deep reinforcement learning network to generate domain-wide coordination actions, including domain-wide task load balancing strategies, domain-wide resource reservation ratios, and domain-wide task priority reordering strategies. The middle-layer coordination agent layer also aggregates the received local action information and local state information from the lower-level resource agents within its management domain to generate a domain aggregation state, which is then uploaded to the upper-layer global agent layer. The upper global agent layer consists of global agents deployed at the central scheduling node. The global agents receive domain aggregation states and domain coordination actions uploaded by several mid-level coordinating agents, as well as global early warning signals, and use a third deep reinforcement learning network to generate global scheduling strategies. In this embodiment of the invention, a hierarchical decision-making link is formed between the bottom-level resource agent layer, the middle-level coordination agent layer, and the upper-level global agent layer. The cross-domain task routing decision in the global scheduling strategy generated by the upper-level global agent layer limits the total task load limit within each domain, serving as the decision boundary for the middle-level coordination agent layer to perform intra-domain resource reservation and load balancing. The intra-domain resource reservation ratio in the intra-domain coordination action generated by the middle-level coordination agent layer limits the upper limit of the resource quota that the bottom-level resource agent layer can allocate, and the intra-domain task priority reordering strategy limits the task execution order of the bottom-level resource agent layer. The actual execution results of the local resource allocation actions of the bottom-level resource agent layer are aggregated by the middle-level coordination agent layer and fed back to the upper-level global agent layer, generating a closed-loop hierarchical decision-making link with top-down decision constraints and bottom-up execution feedback. In this embodiment of the invention, the first deep reinforcement learning network, the second deep reinforcement learning network, and the third deep reinforcement learning network undergo a centralized training phase before deployment. With the optimization objective of maximizing the global reward value, they are pre-trained offline in the digital twin environment constructed by the digital twin module. The global reward value is a weighted comprehensive function of load balancing, energy efficiency, and resource utilization. After training, each network fixes its network parameters and directly generates the corresponding output action based on the current input state through forward propagation. It is understood that, in a preferred embodiment of the present invention, the first deep reinforcement learning network adopts a duel-style dual-deep Q-network structure, the second deep reinforcement learning network adopts a multi-agent proximal policy optimization structure, and the third deep reinforcement learning network adopts a multi-agent deep deterministic policy gradient structure. Both the second and third deep reinforcement learning networks are built based on the Actor-Critic framework and transmit information vertically through a hierarchical decision link. However, this embodiment does not impose specific restrictions on the specific deep reinforcement learning network structure used. Those skilled in the art can adaptively select and replace the deep reinforcement learning network structure used by each layer of agents according to the node scale of the actual distributed system, the continuous or discrete characteristics of the action space, the response latency requirements of each layer's decision task, and the available computing resources. It is sufficient that the corresponding layer can generate corresponding resource allocation actions, intra-domain coordination actions, or global scheduling strategies under given local state information and early warning information input conditions.

[0036] A weight allocation module, which is connected to the early warning module, is used to adjust the decision weight of each agent in joint decision-making based on the role allocation result. The role allocation result is determined based on the current system state and the adaptability of each agent to several predefined role types calculated from the early warning information. Specifically, the weight allocation module includes: The role definition submodule is used to predefine several agent role types, including load balancing priority, energy consumption optimization priority, and resource utilization maximization. A state coding submodule is used to determine the state feature vector of the system state based on the digital twin graph structure and the early warning information; The adaptation calculation submodule is used to calculate the adaptation of each agent and each role type based on the state feature vector. The role negotiation submodule is used to use a distributed negotiation protocol to exchange information based on the suitability to determine the role allocation result. The negotiation protocol adopts a multi-round iterative algorithm based on a consensus mechanism. In each round of iteration, each agent adjusts its own role selection according to the current suitability and the role preferences of other agents until the role selections of all agents converge. Specifically, the weight allocation module further includes an allocation submodule, which is used to calculate the decision weight of each agent in joint decision-making based on the role type assigned to each agent in the role allocation result and the current priority strategy of the system. In this embodiment of the invention, when the current priority strategy of the system is load balancing, the decision weight of the agent assigned to the load balancing priority role is increased; when the current priority strategy of the system is energy consumption optimization, the decision weight of the agent assigned to the energy consumption optimization priority role is increased.

[0037] The simulation execution module is connected to the weight allocation module and the hierarchical decision module respectively, and is used to transmit the weighted scheduling strategy to the digital twin module so that the digital twin module can perform simulation execution. The weighted scheduling strategy is determined based on feeding back each decision weight to the hierarchical decision module. The scheduling execution module, which is connected to the simulation execution module, is used to determine the maximum node deviation value by comparing the state evolution trajectory generated after executing the scheduling strategy based on the digital twin graph structure with the actual execution result generated by distributing the schedulable strategy to the corresponding resource nodes. The schedulable strategy is determined based on the state evolution trajectory. Specifically, the scheduling execution module compares the actual operating status of each resource node with the digital twin simulation status across multiple resource dimensions. Based on the state deviation corresponding to each resource dimension, it determines the degree of deviation between the actual operating status of each resource node and the digital twin simulation status. It then identifies the resource node with the largest deviation and uses the state deviation result corresponding to the resource node with the largest deviation as the scheduling execution deviation result. This is used to determine the degree of deviation between the simulation accuracy of the digital twin module for the physical system state and the real physical environment. The lower the simulation accuracy of the digital twin module, the weaker its decision support capability for scheduling strategies.

[0038] In this embodiment of the invention, the schedulable strategy is determined based on a safety constraint check and screening of the state evolution trajectory; The processing and analysis module, which is connected to the scheduling and execution module, is used to determine the node feature aggregation layer update rate and competition intensity threshold of the graph neural network based on the maximum node deviation value.

[0039] Please see Figure 2 The diagram shows a flowchart of the steps in the distributed resource scheduling optimization method based on multi-agent reinforcement learning according to an embodiment of the present invention, including: S1 collects multi-dimensional physical state information of the distributed system through state awareness modules deployed on each resource node; S2, a digital twin graph structure is constructed based on the graph neural network and the multidimensional physical state information through the digital twin module. The graph neural network includes a node feature aggregation layer, and the node feature aggregation layer has a dynamically adjustable parameter update rate. S3, the competition prediction module predicts the resource demand prediction sequence of each resource node within a preset time window based on the digital twin graph structure and historical state sequence; S4, the early warning module determines the resource competition intensity at a preset time step based on the resource demand prediction sequence, and generates early warning information based on the resource competition intensity and the competition intensity threshold; S5, the hierarchical decision-making module determines the scheduling strategy of the hierarchical decision-making link of the multi-agent based on the digital twin graph structure and the early warning information; S6, the decision weights of each agent in the joint decision are adjusted by the weight allocation module based on the role allocation results. The role allocation results are determined based on the adaptability of each agent to several predefined role types calculated from the current system state and the warning information. S7, the weighted scheduling strategy is transmitted to the digital twin module through the simulation execution module so that the digital twin module can perform simulation execution, wherein the weighted scheduling strategy is determined based on feeding back each decision weight to the hierarchical decision module; S8, by using the state evolution trajectory generated after the scheduling execution module executes the scheduling strategy based on the digital twin graph structure and the actual execution result generated by distributing the schedulable strategy to the corresponding resource node, the maximum node deviation value is determined, wherein the schedulable strategy is determined based on the state evolution trajectory; S9, the processing and analysis module determines the node feature aggregation layer update rate and competition intensity threshold of the graph neural network based on the maximum node deviation value.

[0040] Please see Figure 3 The diagram shown is a flowchart illustrating the steps of determining the maximum node deviation value according to an embodiment of the present invention, including: The processing and analysis module determines the scheduling state of distributed resources based on the maximum node deviation value, including a first scheduling state when the maximum node deviation value is less than or equal to a preset deviation value and a second scheduling state when the maximum node deviation value is greater than the preset deviation value. Specifically, the preset deviation value is set based on the historical statistical distribution of the maximum node deviation value under normal system operation; When the scheduling state is the second scheduling state, the processing and analysis module obtains the comprehensive state deviation value and demand prediction deviation value of each node. The demand prediction deviation value of each node is determined based on the comprehensive value of the node's observed values ​​and demand prediction values ​​across multiple resource dimensions. The actual operating state is the actual value collected by the state awareness module at the corresponding time step after scheduling execution. The digital twin simulation state is the simulated value of the digital twin module in the state evolution trajectory at the corresponding time step. Preferably, the Euclidean distance of the demand deviations at each resource dimension is used as the demand prediction deviation value of the node, and the Euclidean distance between the actual operating state and the digital twin simulation state at each resource dimension is used as the comprehensive state deviation value of the node. The processing and analysis module determines the state deviation vector and the demand prediction deviation vector based on the comprehensive state deviation value and demand prediction deviation value of each node. The processing and analysis module determines the deviation attribution coefficient based on the covariance of the state deviation vector and the standard deviation of the demand prediction deviation vector, as well as the standard deviation of the state deviation vector and the demand prediction deviation vector. This coefficient is used to determine the degree of correlation between the source of the deviation of the maximum node deviation value in the distributed resource scheduling system and the representation error of the digital twin model or the prediction error of the competition prediction module. The larger the value is than zero, the less accurate the prediction of future resource demand trends by the competition prediction module is; the smaller the value is than zero, the less accurate the encoding and aggregation of node features by the graph neural network in the digital twin module is, and the less capable the model representation is. The processing and analysis module generates corresponding processing methods based on the deviation attribution coefficient, including determining the node feature aggregation layer update rate of the graph neural network based on the normalized deviation amplitude when the deviation attribution coefficient is less than or equal to the second correlation coefficient threshold; determining the competition intensity threshold based on the demand prediction deviation fluctuation coefficient when the deviation attribution coefficient is greater than or equal to the first correlation coefficient threshold; and issuing a manual verification notification when the deviation attribution coefficient is less than the first correlation coefficient threshold but greater than the second correlation coefficient threshold, wherein the first correlation coefficient threshold is greater than zero and the second correlation coefficient threshold is less than zero. In this embodiment of the invention, the correlation coefficient threshold is preset based on the statistical value of the distribution of deviation attribution coefficients in the historical operating data of the system.

[0041] Specifically, the process by which the processing and analysis module determines the update rate of the node feature aggregation layer of the graph neural network based on the normalized bias magnitude includes: The normalized deviation amplitude is determined based on the maximum node deviation value and the preset deviation value, which is used to determine the severity of the deviation of the current simulation accuracy of the digital twin module from the upper limit of the system's allowable deviation. The larger the value, the more severe the degree of deviation of the digital twin module from the physical system's simulation from the allowable range, and the lower the representation accuracy of the graph neural network node feature aggregation layer. The node feature aggregation layer update rate is increased based on the normalized deviation magnitude. The greater the increase in the node feature aggregation layer update rate, the larger the step size of the digital twin module in correcting the weights of the graph neural network aggregation layer during each backpropagation. The faster the model responds to recent changes in the physical environment, and the more accurate the mapping of the digital twin graph structure to the physical system. It is understandable that the first, second, and third deep reinforcement learning networks keep their parameters fixed after deployment, while the graph neural network in the digital twin module dynamically adjusts the parameter update rate of the node feature aggregation layer based on the maximum node deviation value, and updates the parameters when the update conditions are met. The processing and analysis module obtains the response lag time from the multidimensional physical state information to determine the degree of lag between the digital twin module's tracking response speed to changes in the physical system state and the system's required response speed. The smaller this value, the faster the node feature aggregation layer of the digital twin module tracks changes in the physical environment state. When the response lag time in the multidimensional physical state information is less than or equal to a preset lag threshold, the processing and analysis module reduces the node feature aggregation layer update rate based on the response lag time. The greater the reduction in the node feature aggregation layer update rate, the slower the digital twin module responds to recent changes in the physical environment, the more stable and convergent the parameter updates become, and the less parameter oscillations occur in the model due to over-updates in the converged state. The processing and analysis module determines the scheduling status after redetermining the update rate of the node feature aggregation layer. When the scheduling state is the second scheduling state, the processing and analysis module determines the competition intensity threshold based on the demand forecast deviation fluctuation coefficient.

[0042] Please see Figure 4 The diagram shows a flowchart illustrating the steps of determining the competition intensity threshold based on the demand forecast deviation fluctuation coefficient according to an embodiment of the present invention, including: The competition prediction module determines the prediction deviation between the predicted and observed values ​​of the demand for each resource node in each resource dimension. The predicted value of the demand is obtained based on the resource demand prediction sequence, and the observed value is obtained based on multi-dimensional physical state information. The competition prediction module determines the average prediction deviation based on the prediction deviation of the resource node in each resource dimension; The competition prediction module determines the demand prediction deviation fluctuation coefficient based on the mean prediction deviation of each resource node and the average of the mean prediction deviations. This coefficient is used to determine the dispersion of the prediction deviations among the resource nodes. The larger the value, the more inconsistent the prediction deviation directions of each resource node are, the stronger the randomness of the prediction error, the higher the proportion of false warnings in the early warning information generated based on the prediction result, and the lower the overall credibility of the early warning signal. The processing and analysis module increases the competition intensity threshold based on the demand forecast deviation fluctuation coefficient. The greater the increase in the competition intensity threshold, the higher the warning threshold, the stronger the effect of filtering false warnings, and the higher the overall credibility of the warning signal. The processing and analysis module determines the scheduling status after redetermining the competition intensity threshold; When the processing and analysis module is in the second scheduling state, it issues a dual-path failure alarm notification indicating that the deviation of the scheduling system exceeds the limit. The dual-path failure means that the deviation of both the digital twin simulation channel and the actual execution channel exceeds the limit.

[0043] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

[0044] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A distributed resource scheduling optimization system based on multi-agent reinforcement learning, characterized in that, include: The state awareness module is deployed on each resource node to collect multi-dimensional physical state information of the distributed system. A digital twin module is used to construct a digital twin graph structure based on a graph neural network and the multidimensional physical state information. The graph neural network includes a node feature aggregation layer, and the node feature aggregation layer has a dynamically adjustable parameter update rate. The competition prediction module is used to predict the resource demand prediction sequence of each resource node within a preset time window based on the digital twin graph structure and historical state sequence. The early warning module is used to determine the resource competition intensity at a preset time step based on the resource demand prediction sequence, and to generate early warning information based on the resource competition intensity and the competition intensity threshold. A hierarchical decision-making module is used to determine the scheduling strategy of the hierarchical decision-making link of the multi-agent based on the digital twin graph structure and the early warning information; The weight allocation module is used to adjust the decision weight of each agent in joint decision-making based on the role allocation result. The role allocation result is determined based on the adaptability of each agent to several predefined role types calculated from the current system state and the warning information. The simulation execution module transmits the weighted scheduling strategy to the digital twin module so that the digital twin module can perform simulation execution. The weighted scheduling strategy is determined based on feeding back the decision weights to the hierarchical decision module. The scheduling execution module is used to determine the maximum node deviation value by comparing the state evolution trajectory generated after executing the scheduling strategy based on the digital twin graph structure with the actual execution result generated by distributing the schedulable strategy to the corresponding resource nodes. The schedulable strategy is determined based on the state evolution trajectory. The processing and analysis module is used to determine the node feature aggregation layer update rate and competition intensity threshold of the graph neural network based on the maximum node deviation value.

2. The distributed resource scheduling and optimization system based on multi-agent reinforcement learning according to claim 1, characterized in that, The processing and analysis module includes: The status analysis submodule is used to determine the scheduling status of distributed resources based on the maximum node deviation value; The adjustment submodule is used to determine whether to determine the node feature aggregation layer update rate of the graph neural network based on the normalized deviation amplitude, or to determine the competition intensity threshold based on the demand forecast deviation fluctuation coefficient.

3. The distributed resource scheduling and optimization system based on multi-agent reinforcement learning according to claim 2, characterized in that, The adjustment submodule includes: A deviation magnitude calculation unit is used to determine the normalized deviation magnitude based on the maximum node deviation value; A determining unit is used to increase the update rate of the node feature aggregation layer based on the normalized deviation amplitude, and the increase in the update rate of the node feature aggregation layer is proportional to the normalized deviation amplitude.

4. The distributed resource scheduling and optimization system based on multi-agent reinforcement learning according to claim 3, characterized in that, The adjustment submodule is also used to reduce the update rate of the node feature aggregation layer based on the response lag time when the response lag time in the multidimensional physical state information is less than or equal to a preset lag threshold, and the reduction in the update rate of the node feature aggregation layer is inversely proportional to the response lag time.

5. The distributed resource scheduling and optimization system based on multi-agent reinforcement learning according to claim 4, characterized in that, The adjustment submodule is also used to determine the scheduling status after re-determining the node feature aggregation layer update rate, and to determine whether to determine the competition intensity threshold based on the demand prediction deviation fluctuation coefficient based on the scheduling status.

6. The distributed resource scheduling and optimization system based on multi-agent reinforcement learning according to claim 5, characterized in that, The competition prediction module includes: The deviation determination submodule is used to determine the prediction deviation between the predicted and observed values ​​of the demand for each resource node in each resource dimension. The mean determination submodule is used to determine the mean of prediction deviation based on the prediction deviation of the resource node in each resource dimension. The volatility coefficient determination submodule is used to determine the demand forecast deviation volatility coefficient based on the average forecast deviation corresponding to each of the resource nodes.

7. The distributed resource scheduling and optimization system based on multi-agent reinforcement learning according to claim 6, characterized in that, The adjustment submodule is also used to increase the competition intensity threshold based on the demand forecast deviation fluctuation coefficient, and the increase in the competition intensity threshold is proportional to the demand forecast deviation fluctuation coefficient.

8. The distributed resource scheduling optimization system based on multi-agent reinforcement learning according to claim 7, characterized in that, The adjustment submodule is also used to determine whether to issue a dual-path failure alarm notification for excessive scheduling system deviation based on the scheduling status after the competition intensity threshold has been redefined.

9. The distributed resource scheduling and optimization system based on multi-agent reinforcement learning according to claim 1, characterized in that, The early warning module includes: The competition determination submodule is used to determine the resource competition coefficient between any two resource nodes based on the resource type, resource capacity and task load characteristics of each resource node in the digital twin graph structure. The graph construction submodule is used to construct a competition relationship graph based on the competition coefficients of each resource. The intensity prediction submodule is used to input the resource demand prediction sequence into the competition relationship graph to predict the resource competition intensity at a preset time step.

10. The distributed resource scheduling optimization system based on multi-agent reinforcement learning according to claim 1, characterized in that, The weight allocation module includes: The role definition submodule is used to predefine several agent role types; A state coding submodule is used to determine the state feature vector of the system state based on the digital twin graph structure and the early warning information; The adaptation calculation submodule is used to calculate the adaptation of each agent and each role type based on the state feature vector. The role negotiation submodule is used to exchange information based on the fit to determine the role allocation result.

Images