Data task scheduling system based on multi-agent reinforcement learning

Abstract

The embodiment of the invention provides a data task scheduling system based on multi-agent reinforcement learning, and relates to the technical field of data task scheduling technologies. The system comprises a state acquisition module used for acquiring a local state vector of an execution node; the hierarchical aggregation module is used for performing hierarchical aggregation on the local state vector based on a preset hierarchical topological structure to generate a partition state vector; performing second aggregation processing on the partition state vector to generate a global state vector; the decision module is used for determining a scheduling action of the data task based on the local state vector and at least one of the partition state vector or the global state vector; the strategy optimization module is used for determining a global reward based on the global state vector and the joint scheduling action of the plurality of data tasks; and generating gradient information for updating the scheduling strategy according to the global reward. By means of the task scheduling method and device, the problem of communication delay is solved, and then the effect of improving the task scheduling efficiency is achieved.

Description

Technical Field

[0001] This invention relates to the field of data task scheduling technology, and more specifically, to a data task scheduling system based on multi-agent reinforcement learning. Background Technology

[0002] In modern large-scale data centers, how to efficiently schedule massive computing tasks to maximize resource utilization and system throughput is a core technical challenge.

[0003] Multi-agent reinforcement learning (MARL) offers a promising paradigm for addressing this issue. This paradigm abstracts each computational task to be scheduled as an independent agent, with each agent making autonomous decisions based on its local environment and coordinating policy optimization through a global evaluation network.

[0004] However, existing MARL scheduling methods face technical problems in engineering practice, such as high communication overhead leading to communication delays, which limits their application in ultra-large-scale clusters. Summary of the Invention

[0005] This invention provides a data task scheduling system based on multi-agent reinforcement learning, which at least solves the problem of high communication latency in related technologies.

[0006] According to an embodiment of the present invention, a data task scheduling system based on multi-agent reinforcement learning is provided, comprising: The status acquisition module is used to acquire the local status vector of the execution node, wherein the execution node is associated with the data task to be scheduled; The hierarchical aggregation module is used to perform hierarchical aggregation on the local state vector based on a preset hierarchical topology to generate a partitioned state vector; and to perform a second aggregation process on the partitioned state vector to generate a global state vector. The decision module is used to determine the scheduling action of the data task based on at least one of the local state vector and the partition state vector or the global state vector. The strategy optimization module is used to determine the global reward based on the global state vector and the joint scheduling actions of multiple data tasks; and to generate gradient information for updating the scheduling strategy based on the global reward.

[0007] In one exemplary embodiment, obtaining the local state vector of the execution node includes: Collect the hardware status of the execution node and the task status associated with the data task; The hardware state and the task state are normalized to generate the local state vector.

[0008] In one exemplary embodiment, the hierarchical aggregation module includes: The partition aggregation unit is used to calculate a cooperative state potential value based on multiple local state vectors within the same partition, the cooperative state potential value being used to characterize the cooperative state risk of the partition; and to generate the partition state vector based on the cooperative state potential value and the multiple local state vectors.

[0009] In one exemplary embodiment, the policy optimization module includes: A global value network update unit is used to update the global value network based on the global reward. The advantage function calculation unit is used to calculate the advantage function based on the updated global value network; A gradient generation unit is used to generate the gradient information based on the dominance function.

[0010] In one exemplary embodiment, the system further includes: The predictive intervention module is used to predict the predicted state vector at future times based on the time series of the local state vector; compare the predicted state vector with a preset resource contention threshold to determine whether there is a near-term resource contention event; and perform a first operation when it is determined that there is a near-term resource contention event.

[0011] According to another embodiment of the present invention, a data task scheduling method based on multi-agent reinforcement learning is provided, comprising: Obtain the local state vector of the execution node, which is associated with the data task to be scheduled; Based on a preset hierarchical topology, the local state vector is hierarchically aggregated to generate a partitioned state vector; and the partitioned state vector is then subjected to a second aggregation process to generate a global state vector. The scheduling action of the data task is determined based on at least one of the local state vector and the partition state vector or the global state vector. Based on the global state vector and the joint scheduling actions of multiple data tasks, a global reward is determined; and gradient information for updating the scheduling strategy is generated according to the global reward.

[0012] In one exemplary embodiment, obtaining the local state vector of the execution node includes: Collect the hardware status of the execution node and the task status associated with the data task; The hardware state and the task state are normalized to generate the local state vector.

[0013] In an exemplary embodiment, the hierarchical aggregation of the local state vector to generate a partitioned state vector includes: Based on multiple local state vectors within the same partition, a cooperative state potential value is calculated, which is used to characterize the cooperative state risk of the partition. The partition state vector is generated based on the cooperative state potential value and the multiple local state vectors.

[0014] According to yet another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0015] According to yet another embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0016] This invention reconstructs a flat information interaction pattern into an efficient tree-like hierarchical structure through a hierarchical state aggregation bus and a partitioned value evaluation network, thereby reducing communication load and avoiding network broadcast storms. At the same time, through partitioned and globally collaborative value evaluation and strategy optimization, the computational pressure on the central node is distributed. Therefore, it can solve the problem of high communication latency in related technologies and improve the system response speed. Attached Figure Description

[0017] Figure 1 This is a flowchart of a data task scheduling method based on multi-agent reinforcement learning according to an embodiment of the present invention; Figure 2 This is a structural block diagram of a data task scheduling system based on multi-agent reinforcement learning according to an embodiment of the present invention; Figure 3 This is a flowchart according to Embodiment 3 of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0019] In the following description, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0020] Furthermore, in this application, directional terms such as "upper," "lower," "left," and "right" may be defined relative to the orientation of the components shown in the accompanying drawings. It should be understood that these directional terms can be relative concepts, used for relative description and clarification, and may change accordingly depending on the orientation of the components in the accompanying drawings.

[0021] In this application, unless otherwise expressly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral part; it can be a direct connection or an indirect connection through an intermediate medium. Furthermore, the term "coupled" can refer to an electrical connection that enables signal transmission.

[0022] As used herein, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from the given value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the given quantity (i.e., the limitations of the measurement system).

[0023] Reference Figure 1 and Figure 2 This application provides a data task scheduling method based on multi-agent reinforcement learning. In one embodiment, the method can be applied to a computing cluster consisting of multiple execution nodes, first-level aggregation nodes, and second-level (global) aggregation nodes. Each data task to be scheduled is controlled by an actor agent deployed on an execution node. The method specifically includes the following steps: S100: Obtain the local state vector of the execution node associated with the data task to be scheduled.

[0024] In this embodiment, each basic decision-making unit of the system (i.e., the actor agent bound one-to-one with the data task) needs to perceive its direct operating environment. Specifically, it transforms the original heterogeneous physical and logical state information into a structured local state vector, which is used for the machine learning model.

[0025] In one specific embodiment, the following sub-steps are included: S101, collect the hardware status of the execution node and the task status related to the data task; S102 normalizes the hardware state and task state.

[0026] The actor agent module integrates a state collector, which interacts with the operating system kernel or virtualization management layer of the execution node to periodically query a series of performance counters and state parameters (e.g., the time interval can be set between 50 milliseconds and 200 milliseconds, exemplarily 100 milliseconds). Hardware state is a direct measure of the physical resource load of the execution node, and its specific indicators may include the current utilization of the central processing unit (CPU), the used and available space of physical memory, the input / output bandwidth of the network interface, and the disk read / write IOPS (read / write operations per second). Task state describes the running status of the data task itself, and its specific indicators may include the length of the current task in the waiting queue, the task's own priority setting, the running time, and the estimated remaining running time, etc., which are not limited here.

[0027] For example, consider a micro-cluster consisting of four execution nodes (N1, N2, N3, N4), all belonging to a rack partition R1; each node runs a data task controlled by an actor agent. The state vector contains four dimensions: [CPU utilization (%), memory usage (GB), network outgoing bandwidth (Mbps), task queue length (number of tasks)]. [In the timestamp...] At that time, the state collector acquired the raw data of node N1: CPU utilization was 80.5% (dimensionless), memory usage was 12.3 gigabytes (GB), network outgoing bandwidth was 350.2 megabits per second (Mbps), and the number of tasks waiting in the task queue was 5. To eliminate the differences in dimensions and numerical ranges between different state dimensions, the acquired raw data needs to be normalized. Here, minimum-maximum scaling is used for normalization, which linearly maps the raw data to the [0,1] interval, meaning that for any state value... Its normalized value The calculation formula is:

[0028] in, and These are the minimum and maximum values ​​of that state dimension in historical data or preset based on physical constraints; for example, CPU utilization. =0, It is 100; memory usage It can be set to 0. The total memory capacity of the node can be set, such as 64GB; the network outgoing bandwidth ranges from [0, 1000] Mbps; the task queue length ranges from [0, 50]; for node N1 in The process of normalizing the original state at that time is as follows: CPU utilization normalized value: ; Normalized memory usage: ; Normalized network outgoing bandwidth: ; Normalized value of task queue length: .

[0029] After normalization, these values ​​together constitute the local state vector:

[0030] All components of this vector are in the interval [0,1] and are dimensionless floating-point numbers, which can be directly used as input for subsequent calculations; similarly, assume the state vectors of other nodes are as follows: , , .

[0031] S200: Based on a preset hierarchical topology, hierarchically aggregate the local state vector to generate a partitioned state vector; and perform a second aggregation process on the partitioned state vector to generate a global state vector.

[0032] In this embodiment, a cooperative state potential model is constructed to quantify the systemic risk generated by the combination of states of multiple nodes within a partition.

[0033] S210: Generate partitioned state vectors and cooperative state potential values.

[0034] The state aggregation module of a primary aggregation node (such as the management node of rack R1) collects all local state vectors within its managed partition. Then, perform the following operations: First, calculate the cooperative state potential energy value. This value is used to capture the nonlinear synergistic effect between different state dimensions; for example, when an extreme state in one dimension occurs simultaneously with an extreme state in another dimension, the resulting system risk is far greater than the sum of the risks when the two exist independently.

[0035] In one specific embodiment, the potential energy function It can be a polynomial containing linear terms and cross terms (second or higher order); for example, for a four-dimensional state vector. Potential energy of a single node It can be defined as:

[0036] in, (Unit: dimensionless) is the coefficient of the linear term, representing the baseline risk contribution of each dimension of the state; (Unit: dimensionless) is the coefficient of the cross term, used to capture synergistic effects. For example, cross term... This reflects the following judgment: high CPU utilization is one state, long task queue is another state, but the risk of congestion and collapse indicated by the simultaneous occurrence of both (CPU computing power is about to be exhausted, while tasks are still piling up) is the product of the two, which is much greater than their linear superposition.

[0037] Cooperative state potential value of the entire partition It is the sum of the potential energy of all nodes:

[0038] For example, suppose the potential energy function parameters are: ; Subsequently, the state vector of the potential energy of node N3 is calculated as follows: : Linear term part =

[0039] Intersection part =

[0040] Potential energy of node N3 .

[0041] It's easy to understand why this value is significantly higher than other nodes, as it's close to saturation in all three dimensions: CPU, memory, and queue length. The synergy term intuitively demonstrates the enormous risk value.

[0042] Assuming the calculated potential energy of other nodes is The cooperative state potential energy of the entire partition R1 is then... This scalar value (unit: dimensionless) is a quantitative judgment of the rationality of the current state of the partition. The system can then set a potential energy threshold, for example... ,when If the partition is in an unreasonable or high-risk state, an early warning or a more aggressive scheduling strategy can be triggered, and so on.

[0043] Secondly, generate the partition state vector. This vector is an enhancement vector, which concatenates statistical information about the states within the partition with the calculated potential energy value, for example:

[0044] in, It is the mean vector of all local state vectors. It is a vector of element-wise maximum values. It is an element-wise standard deviation vector. The resulting partition state vector not only contains the average state of the partition, but also reflects the dispersion of the state (standard deviation) and extreme cases (maximum value), and adds a collaborative judgment on the overall risk (potential value).

[0045] S230: Global state vector generation.

[0046] The first-level aggregation node will generate the partition state vector The state vectors are reported to the global node, where the state aggregation module aggregates all partition state vectors in a similar manner to generate the final global state vector. Simultaneously, a global cooperative state potential value will also be calculated. To assess the macroscopic stability of the entire system.

[0047] S300: Determine the scheduling action of the data task based on at least one of the local state vector and the partition state vector or the global state vector.

[0048] In this embodiment, each actor agent module stores its own local state vector. It receives the upper-level aggregation state vector (e.g., the partition state vector issued by the first-level aggregation node). The states are concatenated to form a higher-dimensional mixed state vector. This mixed state vector is input into the policy network of the actor agent. This policy network is typically a deep neural network (e.g., a multilayer perceptron (MLP)) that receives... As input, the output is an action vector. This action vector is used to indicate the specific scheduling operation for the current data task, such as adjusting resource requests, changing task priorities, or execution timing.

[0049] For example, assume the action space is two-dimensional, representing the "CPU core count adjustment ratio" and the "task priority adjustment value", both ranging from [-1, 1]; the actor agent of node N1 receives its local state. and partition status (A value containing the mean, maximum, standard deviation, and potential energy) (high-dimensional vector), because Exceeding the threshold indicates resource scarcity in the partition; after learning, the policy network tends to output a conservative action, for example, the output action vector might be... This action vector is interpreted as: reducing the number of CPU cores allocated to the current task by 30% to alleviate CPU pressure, while increasing the task priority by 50% to ensure it can be completed first under resource constraints. The action executor within the actor agent module translates this logical action vector into specific API calls to the underlying resource manager (such as Kubernetes' kubelet or YARN's ResourceManager).

[0050] S400: Determines the global reward based on the global state vector and the joint scheduling actions of multiple data tasks; and generates gradient information for updating the scheduling policy based on the global reward.

[0051] This implementation includes the following sub-steps: S410: Global reward calculation.

[0052] The global comment module is based on the global state vector. Calculate the global reward by combining actions with those of all agents. In this embodiment, the reward function penalizes high-risk states and consists of two parts: one part is a positive incentive for system performance, and the other part is a negative penalty for cooperative state potential energy.

[0053] in, This could be metrics such as total system throughput or average task completion time. This is the potential energy value of the coordinated state. and It is a weighting coefficient (dimensionless). This design guides the intelligent agent cluster not only to pursue high performance, but also to learn to actively avoid and resolve entering high-risk "unreasonable" states.

[0054] S420: Partition-Global Collaborative Value Network Update.

[0055] First, a partition commentator module is deployed on each first-level aggregation node. This module maintains a partition value network, with the partition state vector as its input. The output is the assessed value of this partition. This valuation reflects the cumulative rewards the partition may receive in the future.

[0056] Secondly, a global comment module is deployed on the global nodes. This module maintains a global value network, but its input is not a simple global state vector. Instead, it uses a gated fusion network to represent the global state. Compared with the zone value assessment values ​​collected from each zone Dynamic fusion is performed, and the fusion mechanism is as follows:

[0057]

[0058] in, This is the first The gated vectors calculated for each partition It is the Sigmoid function. It is an element-wise product, a mechanism that allows global comments to selectively focus on the value assessment of those more important partitions based on the current global state.

[0059] Therefore, the update target (TD-Target) of the global value network is:

[0060] in It represents the fusion value of the next state.

[0061] S430: Gradient information generation and backpropagation.

[0062] After calculating a reliable global value (or advantage function) After that, it is necessary to solve the credit allocation problem of reasonably distributing this global signal to each actor agent.

[0063] To address this, this application employs a contribution-based gradient modulation mechanism for signal allocation. The global evaluation module not only calculates the global dominance function... It will also estimate the intelligence of each actor's agent. action global cooperative state potential Contribution .

[0064] Ultimately used for updating actors The individual advantage function of the policy network is modulated as follows:

[0065] in, It is the penalty coefficient; this means that if an agent's action causes a significant increase in the system's potential energy, it will receive a penalized and lower advantage signal, thereby inhibiting such behavior.

[0066] This individualized gradient signal is backpropagated via a hierarchical control bus protocol (e.g., an asynchronous streaming service based on gRPC), and then the global nodes will compute the calculated gradient signal. The data is sent to the corresponding primary aggregation node, which then distributes it to the individual agent agents within its jurisdiction. Simultaneously, each agent agent uses the received individual advantage function. This updates the policy network parameters. Through this loop, the entire system learns to maintain a reasonable overall operating state while pursuing performance.

[0067] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0068] This embodiment also provides a data task scheduling system based on multi-agent reinforcement learning, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0069] Reference Figure 2 The system may include: The status acquisition module 710 is used to acquire the local status vector of the execution node, wherein the execution node is associated with the data task to be scheduled; The hierarchical aggregation module 720 is used to perform hierarchical aggregation on the local state vector based on a preset hierarchical topology to generate a partitioned state vector; and to perform a second aggregation process on the partitioned state vector to generate a global state vector. The decision module 730 is used to determine the scheduling action of the data task based on at least one of the local state vector and the partition state vector or the global state vector. The strategy optimization module 740 is used to determine the global reward based on the global state vector and the joint scheduling actions of multiple data tasks; and to generate gradient information for updating the scheduling strategy based on the global reward. These modules can be implemented through computer program instructions and integrated into the cluster management software of the data center, functioning as an intelligent scheduling layer.

[0070] The process of obtaining the local state vector of the execution node includes: Collect the hardware status of the execution node and the task status associated with the data task; The hardware state and the task state are normalized to generate the local state vector.

[0071] In an optional example, the hierarchical aggregation module 720 includes: The partition aggregation unit is used to calculate a cooperative state potential value based on multiple local state vectors within the same partition, the cooperative state potential value being used to characterize the cooperative state risk of the partition; and to generate the partition state vector based on the cooperative state potential value and the multiple local state vectors.

[0072] In an optional example, the policy optimization module 740 includes: A global value network update unit is used to update the global value network based on the global reward. The advantage function calculation unit is used to calculate the advantage function based on the updated global value network; A gradient generation unit is used to generate the gradient information based on the dominance function.

[0073] Example 2 For some instantaneous, millisecond-level resource contention (e.g., system turbulence caused by sudden memory requests), the latency of responsive decision-making may be too high. In response, this embodiment, based on the technical solution of embodiment one, further introduces the Illuminated Resource Contention Prediction and Intervention Mechanism (ICLIM). This mechanism solves the problem of excessive latency through advance prediction and low-latency intervention, thereby greatly improving the instantaneous response speed and stability of the system.

[0074] The ICLIM mechanism can be implemented by a predictive intervention module 750 deployed on each execution node, which works in parallel with the actor agent module (decision module 730). The specific method flow is as follows: S500: Based on the time series of local state vectors, predict the state vector for future moments.

[0075] The predictive intervention module internally maintains a lightweight time series prediction model, such as a gated recurrent unit (GRU), which continuously receives local state vectors generated by the state acquisition module 710. .

[0076] In one embodiment, the GRU model uses past A sequence of local state vectors at each time step As input to predict the future The state at any given time, i.e., the predicted state vector :

[0077] in, These are the parameters of the GRU model, which can be trained offline or updated online along with the main MARL framework. The value is typically a short time window (e.g., 500 milliseconds) to capture upcoming resource change trends.

[0078] For example, assuming a time step of 100 milliseconds, the model uses the past 5 time steps ( Using data to predict five time steps later ( The status of the timestamp; The predictive intervention module collected sequences The normalized value sequence for memory usage is [0.75, 0.78, 0.81, 0.85, 0.88]. This sequence shows a clear rapid growth trend; based on this sequence, the GRU model may predict... At that time, the value of the memory usage dimension will reach This predicted state vector This forms the basis for the next step of the judgment.

[0079] S600: Compare the predicted state vector with the preset resource contention threshold to determine whether there is a near-term resource contention event.

[0080] The system pre-sets a set of resource contention threshold vectors This vector has the same dimension as the state vector and defines the safety upper limit for each resource dimension.

[0081] The predictive intervention module 750 predicts the state vector element by element. and Compare, if The value of any one dimension exceeds If the threshold of the corresponding dimension is met, the system determines that a near-terminal resource contention event is about to occur:

[0082] For example, suppose the threshold of the memory usage dimension in the resource contention threshold vector. Set as (This represents a physical memory utilization rate of 95%); in the previous example, the predicted memory usage value was... ,because The system will trigger a "pre-memory memory resource contention" event.

[0083] S700: When a preemptive micro-operation is performed upon confirmation of a resource contention event.

[0084] Once a situational event is triggered, the system does not wait for the main MARL framework to complete its relatively slow "perception-decision-execution" loop. Instead, it immediately executes a first action through a separate, low-latency bypass intervention channel to alleviate the impending resource pressure. This first action is associated with a specific type of situational event; for example: 1) For the "Present Memory Resource Contention" event, the first possible action includes: Triggering Lightweight Garbage Collection (LGC) in the target memory region: For the runtime environment of the currently running task (such as JVM, V8), force a generational garbage collection or a memory cleanup of a specific area to quickly release memory objects that are no longer in use.

[0085] Refresh file system page cache: Executing kernel commands such as `sync; echo 1 > / proc / sys / vm / drop_caches` will release memory areas that the operating system is using as caches, but which are not dirty pages, for application use.

[0086] 2) For the "Present CPU Resource Contention" event, the first possible action includes: Reduce the CPU scheduling priority of non-core tasks: Use commands such as renice to temporarily reduce the CPU priority of background tasks such as log collection and monitoring on the same node to ensure the computing power of core business tasks.

[0087] For example, upon triggering a "presence memory contention" event, the anticipatory intervention module immediately sends instructions to the operating system kernel or the task's runtime environment via a bypass intervention channel to perform a micro-operation of "refreshing the file system page cache." This operation can be completed within milliseconds, rapidly releasing, for example, hundreds of MB of cache memory. Therefore, when the anticipated peak in memory requests actually arrives 500 milliseconds later, there is already sufficient available memory on the node, thus avoiding performance fluctuations or task failures due to insufficient memory.

[0088] By introducing the ICLIM mechanism, this embodiment constructs a dual-modal, dual-rate intelligent scheduling system. The main MARL framework (Embodiment 1) serves as the strategic decision-making layer, performing global collaborative macro-level resource allocation at a low frequency (seconds). Meanwhile, the ICLIM mechanism (Embodiment 2) performs single-point prediction and preemptive micro-level intervention at an extremely high frequency (milliseconds). The two complement each other, enabling the system to possess both macro-level stability and micro-level instantaneous response and risk avoidance capabilities. Its overall performance and robustness far exceed those of a single-mode scheduling system. Correspondingly, in one exemplary embodiment, the aforementioned system further includes: The predictive intervention module 750 is used to predict the predicted state vector at future times based on the time series of the local state vector; compare the predicted state vector with a preset resource contention threshold to determine whether there is a near-term resource contention event; and perform the first operation when it is determined that there is a near-term resource contention event.

[0089] Example 3 The cooperative state potential model in Implementation 1 assesses risk at a macroscopic level, while the ICLIM mechanism in Implementation 2 performs preemptive intervention at a microscopic level. However, the decision-making basis of both mechanisms remains "correlation," i.e., state patterns learned from historical data. When unseen changes occur in the cluster environment or task patterns (e.g., the introduction of new types of tasks, mutations in other agents' strategies), correlation-based models may fail. Unlike Implementations 1 and 2, this implementation introduces an adaptive scheduling mechanism, CIC, based on causal inference and counterfactual communication. This mechanism empowers each agent to build and utilize causal models, enabling their decisions to improve the system's adaptability and robustness.

[0090] The CIC mechanism can be implemented jointly by a causal inference module and a decision module 730 deployed on each execution node; for example... Figure 3 As shown, the specific method and process are as follows: S800: Constructing a Local Causal Belief Map The causal inference module maintains a dynamic Local Causal Belief Graph (LCBG) for each actor agent. This graph is a directed acyclic graph (DAG) in which the nodes represent observable or actionable variables of the agent, and the directed edges represent direct causal relationships between the variables.

[0091] In this embodiment, the node set of LCBG may include: Self-state node: Local state vector Each dimension, such as .

[0092] Self-action node: Actions that can be executed by itself, such as .

[0093] Neighboring agent action nodes: summaries of actions of physically or logically neighboring agents perceived through listening or low-frequency broadcasting, such as... .

[0094] Shared resource node: Partition state vector The shared resource status reflected in, such as .

[0095] The causal inference module uses historical data (states, action sequences) to continuously learn the graph structure and edge parameters (i.e. causal strength) of LCBG through causal discovery algorithms (such as PC algorithm, FCI algorithm or gradient-based structure learning methods).

[0096] For example, suppose that the causal inference module of agent i learns that neighboring agent j performs a high network bandwidth task ( This will significantly increase the load on the shared network switch, thereby causing network latency for its own tasks. As it rises; within its LCBG, a line will form from... point to The directed edges, along with a quantified causal effect parameter, for example, .

[0097] S900: Generate counterfactual communication messages Traditional MARL communication mechanisms typically broadcast raw states or actions, resulting in low information density and a susceptibility to communication storms. To address this, this embodiment proposes a counterfactual communication paradigm, where a counterfactual communication message... It consists of a single tuple:

[0098] In the formula: The actual action that the agent currently intends to perform.

[0099] Based on LCBG, execute Then, the key results of the forecast (e.g., the estimated occupancy of shared resources).

[0100] : An alternative action that an agent considers beneficial to the collective.

[0101] :implement Then, the key results of the prediction.

[0102] For example, after analyzing its LCBG, the decision module of agent i determines that the optimal action is to immediately initiate a high-IO task. It calculated through causal graph intervention that this action would cause shared disk I / O to reach saturation at 98%. At the same time, it also evaluated another alternative action: delaying the start of the task by 30 seconds. The cause-effect graph predicts that this alternative action will only cause shared disk I / O to rise to 45%. Therefore, agent i broadcasts a counterfactual channel message to its neighbors: "I plan to immediately initiate a high-IO task (estimated shared IO of 98%); but if I delay by 30 seconds, the estimated shared IO will only be 45%." Thus, when a neighboring agent receives this message, it can understand the impact of agent i's behavior on shared resources and potential cooperation space without knowing agent i's complete state or policy; at the same time, the agent does not need to expose its internal policy network, but only communicates high-level intentions and consequences; moreover, the strategic information carried by a counterfactual message far exceeds the original state broadcast over several time steps.

[0103] S1000: Decision-making based on causal intervention and counterfactual assessment The decision-making process of Decision Module 730 integrates its own causal model and received counterfactual information, specifically including: Local causal assessment: For each candidate action, the decision module 730 uses LCBG to perform causal intervention simulation to assess its causal effects on its own goal and local environment, forming a preliminary action preference ranking.

[0104] Counterfactual negotiation: Decision module 730 analyzes counterfactual messages received from neighboring agents; if it discovers its own factual actions ( ) will be with a neighbor This can lead to severe resource conflicts (e.g., both causing shared resource overload), and one of the alternative actions ( If this conflict can be avoided, then the agent can proactively choose its alternative actions, or decide who makes concessions through a simple negotiation agreement.

[0105] Final decision: The decision-making module integrates the results of local causal assessment and counterfactual negotiation, and selects a final scheduling action that can both meet its own needs and maximize the local collective interests to execute, and so on.

[0106] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0107] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0108] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0109] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0110] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0111] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0113] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0115] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0116] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data task scheduling method based on multi-agent reinforcement learning, characterized in that, include: The status acquisition module is used to acquire the local status vector of the execution node, wherein the execution node is associated with the data task to be scheduled; The hierarchical aggregation module is used to perform hierarchical aggregation on the local state vector based on a preset hierarchical topology to generate a partitioned state vector; and to perform a second aggregation process on the partitioned state vector to generate a global state vector. The decision module is used to determine the scheduling action of the data task based on at least one of the local state vector and the partition state vector or the global state vector. The strategy optimization module is used to determine the global reward based on the global state vector and the joint scheduling actions of multiple data tasks; and to generate gradient information for updating the scheduling strategy based on the global reward.

2. The system according to claim 1, characterized in that, The process of obtaining the local state vector of the execution node includes: Collect the hardware status of the execution node and the task status associated with the data task; The hardware state and the task state are normalized to generate the local state vector.

3. The system according to claim 1, characterized in that, The hierarchical aggregation module includes: The partition aggregation unit is used to calculate a cooperative state potential value based on multiple local state vectors within the same partition, the cooperative state potential value being used to characterize the cooperative state risk of the partition; and to generate the partition state vector based on the cooperative state potential value and the multiple local state vectors.

4. The system according to claim 1, characterized in that, The strategy optimization module includes: A global value network update unit is used to update the global value network based on the global reward. The advantage function calculation unit is used to calculate the advantage function based on the updated global value network; A gradient generation unit is used to generate the gradient information based on the dominance function.

5. The system according to claim 1, characterized in that, The system also includes: The predictive intervention module is used to predict the predicted state vector at future times based on the time series of the local state vector; compare the predicted state vector with a preset resource contention threshold to determine whether there is a near-term resource contention event; and perform a first operation when it is determined that there is a near-term resource contention event.

6. A data task scheduling method based on multi-agent reinforcement learning, characterized in that, include: Obtain the local state vector of the execution node, which is associated with the data task to be scheduled; Based on a preset hierarchical topology, the local state vector is hierarchically aggregated to generate a partitioned state vector; and the partitioned state vector is then subjected to a second aggregation process to generate a global state vector. The scheduling action of the data task is determined based on at least one of the local state vector and the partition state vector or the global state vector. Based on the global state vector and the joint scheduling actions of multiple data tasks, a global reward is determined; and gradient information for updating the scheduling strategy is generated according to the global reward.

7. The method according to claim 6, characterized in that, The process of obtaining the local state vector of the execution node includes: Collect the hardware status of the execution node and the task status associated with the data task; The hardware state and the task state are normalized to generate the local state vector.

8. The method according to claim 6, characterized in that, The hierarchical aggregation of the local state vector to generate a partitioned state vector includes: Based on multiple local state vectors within the same partition, a cooperative state potential value is calculated, which is used to characterize the cooperative state risk of the partition. The partition state vector is generated based on the cooperative state potential value and the multiple local state vectors.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is configured to perform the method described in any one of claims 1 to 5 when executed.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method as described in any one of claims 1 to 5.

Images