System and method for centralized self-adaptive control scheduling of computing power center resources

Abstract

The invention provides a computing power center resource centralized self-adaptive control scheduling system and method, and the method comprises the steps: S1, a user submits a computing power training task through an API, and sends the task to a DAG task composer, and forms a task list to be scheduled; s2, the intelligent monitoring analysis module is used for continuously collecting node data and calculating real-time CS and HS scores of each node; s3, analyzing the dependency relationship type data by a task composer, and calling a dynamic programming algorithm in an adaptive scheduler to make a decision; s4, the adaptive scheduler performs dynamic task distribution according to the task list of the DAG task orchestrator and the intelligent monitoring analysis module, and S5, the distributed runtime and environment management module is used for uploading the prepared Docker mirror image to the distributed storage; and S6, through the unified communication adaptation layer, the mirror image is pulled from the distributed storage on the target node and the container is started, the user does not need to concern the difference of the operating system in the whole process, and efficient and intelligent adaptive scheduling is realized.

Description

Technical Field

[0001] This invention relates to the field of computing power allocation, and in particular to a system and method for centralized adaptive control and scheduling of computing power center resources. Background Technology

[0002] With the rapid development of fields such as artificial intelligence and distributed computing, the demand for computing power is increasing daily. Many computing centers consist of heterogeneous nodes, such as Linux and Windows systems, and use different communication protocols, such as SSH and HTTP. Existing cluster management tools have the following shortcomings:

[0003] Limited protocol support: Typically, there is insufficient native support for Windows systems, and a lack of a unified communication abstraction layer.

[0004] Superficial monitoring: The monitoring indicators are mostly static data such as CPU and memory usage, lacking in-depth analysis and modeling of the real-time computing capabilities and health of nodes, and thus failing to provide intelligent decision-making basis for scheduling.

[0005] Rigid scheduling strategies: They often employ static strategies, such as priority scheduling, round-robin, or simple real-time load-based scheduling, which cannot make forward-looking optimal decisions based on the global task topology and future resource change trends.

[0006] Poor environmental consistency: The task execution environment depends on physical machine or virtual machine environment, making it difficult to achieve fast migration and accurate reproduction between computing power nodes with different architectures.

[0007] For example, CN114745317A discloses a computing task scheduling method and related equipment for computing power networks, a task scheduling method for multi-agent near-end policy optimization MAPPO algorithm, the network computing model includes computing resource agents and network resource agents, the two agents share the Critic network, and through iterative optimization of the network computing model, the final output of the target computing node and forwarding path of the computing task is output. It cannot form a unified communication adaptation and performs distributed management of computing power scheduling.

[0008] Therefore, there is an urgent need for a system that can uniformly manage heterogeneous resources, perform intelligent monitoring and analysis, and achieve dynamic adaptive scheduling. Summary of the Invention

[0009] This invention aims to at least solve the technical problems existing in the prior art, and in particular, it innovatively proposes a system and method for centralized adaptive control and scheduling of computing center resources.

[0010] To achieve the above-mentioned objectives of this invention, this invention provides a method for centralized adaptive control scheduling of computing center resources, characterized by comprising:

[0011] S1, the user submits a computing power training task via API, which includes a DAG task topology definition file that specifies the dependencies between data preprocessing, training, and evaluation steps; the above task is sent to the DAG task orchestrator to form a list of tasks ready for scheduling;

[0012] S2, the intelligent monitoring and analysis module, is used to continuously collect node data and calculate the real-time CS and HS scores of each node; it also collects the raw indicators of different computing node groups in the controlled computing power resource pool in real time.

[0013] S3, the DAG task orchestrator parses dependency-based data, determines the first batch of executable data preprocessing tasks based on the task list provided in S1, and calls a dynamic programming algorithm in the adaptive scheduler to make decisions based on the real-time resource performance profile formed in S2.

[0014] S4, the adaptive scheduler dynamically dispatches tasks based on the task list of the DAG task orchestrator and the intelligent monitoring and analysis module, scheduling data preprocessing tasks to nodes with sufficient CPU resources; and scheduling critical GPU training tasks to the more stable N2 nodes.

[0015] S5, the distributed runtime and environment management module, is used to upload the prepared Docker image containing Python, PyTorch, and CUDA to distributed storage.

[0016] S6, through a unified communication adaptation layer, pulls images from distributed storage on the target node and starts containers to execute tasks; the entire process does not require the user to care about differences in the operating system, achieving efficient and intelligent adaptive scheduling.

[0017] In a preferred embodiment of the above technical solution, S1 includes:

[0018] Construct a Directed Acyclic Graph (DAG); define nodes and edges to identify all nodes in the graph, each node representing an entity or task; define directed edges between nodes to represent dependencies or execution order; ensure no circular dependencies exist, i.e., it is impossible to start from a node and return to that node along a directed edge; choose data structures such as: adjacency list: each node maintains a list pointing to its successor nodes, saving space and easy to traverse; adjacency matrix: uses a two-dimensional array to represent the connection relationships between nodes, suitable for dense graphs; in-degree list: records the in-degree of each node for topological sorting.

[0019] In a preferred embodiment of the above technical solution, step S1 further includes:

[0020] Initialize an empty graph structure, add nodes one by one, perform cycle detection when adding directed edges, and provide API or DSL for users to define task flow; the DAG task orchestrator is responsible for managing the lifecycle of tasks according to the topological order of the DAG;

[0021] The user sets tasks A, B, C, and D; and task A has no dependencies, task B depends on task A, task C depends on tasks A and B, and task D depends on tasks A, B, and C; construct an adjacency list based on the dependencies: {A: [], B: [A], C: [A, B], D: [B,C]};

[0022] Create a 4×4 two-dimensional array and fill it with dependencies according to the adjacency list; traverse the adjacency list and count the number of times each task is depended on by other tasks to get [0, 1, 2, 2].

[0023] Initialize the executable task queue, scan the in-degree table, and add all tasks with an in-degree of 0 to the queue; initially, only task A has an in-degree of 0, so the queue is [A]; dynamically handle task dependencies; remove task A from the queue, mark it as "in execution", and remove all its outgoing edges from the adjacency list and adjacency matrix, i.e., delete the successor relationship of A, and set the matrices [B][A] and [C][A] to False; traverse the successor tasks B and C of A, and update them by decrementing their in-degree by 1;

[0024] Add task B, which has an in-degree of 0 after the update, to the queue; the queue is now [B]. Take out task B, execute it, and remove its outgoing edges. In each step, if there are multiple tasks with an in-degree of 0 in the queue, execute these tasks in parallel.

[0025] In a preferred embodiment of the above technical solution, step S2 includes:

[0026] The performance data of each computing node in the intelligent monitoring and analysis module is normalized.

[0027] Each raw metric is converted into a score S between 0 and 1.0. value

[0028] S cpu = 1.0 - (Current CPU utilization / 100%);

[0029] S gpu = 1.0 - (current GPU utilization / 100%);

[0030] S mem = 1.0 - (current memory utilization / 100%);

[0031] S ssd= 1.0 - (Current SSD utilization / 100%);

[0032] ;

[0033] in, This represents the base weight of CPU resources in the overall computing power calculation, used to adjust the initial contribution ratio of CPU performance to the system's computing power. This represents the weighted nonlinear contribution of CPU resources. α is the "master switch" for this term, controlling the basic proportion of CPU in the overall computing power. The CPU resource index adjusts the weight of CPU resources in the overall computing power, reflecting the non-linear impact of CPU performance improvement on system computing power. This is a GPU resource index that adjusts the weighting of GPU resources' contribution to overall computing power, capturing the non-linear amplification effect of GPU parallel computing capabilities. It is a memory resource index that adjusts the weight of memory resources in the overall computing power and reflects the non-linear constraint of memory capacity on data throughput. The synergistic interaction index adjusts the strength of the synergistic effect among CPU, GPU, and memory, capturing the multiplier effect of nonlinear interactions between resources. When ε>0, resource synergy improves overall computing power, such as efficient matching between GPU and memory; when ε<0, resource competition leads to a decrease in computing power, such as CPU and GPU competing for memory. β represents the CPU resources raised to the power of β, indicating the non-linear contribution of CPU performance to the system's computing power. γ represents the power of GPU resources, indicating the nonlinear amplification effect of GPU parallel computing capabilities; δ represents the memory resources raised to the power of δ, indicating the non-linear constraint of memory capacity on data throughput. ε represents the product of CPU, GPU, and memory resources, indicating the nonlinear multiplier of the synergistic effect among the three. The sum of CPU and GPU resources minus the hyperbolic tangent of memory resources represents the soft constraint of resource balance.

[0034] In a preferred embodiment of the above technical solution, step S2 includes:

[0035] Build a dynamic weight generator based on task type.

[0036]

[0037] σ is the sigmoid function, TaskFeature contains task type features, and Priority is the real-time priority coefficient; where the subscript j is the dimension index of the target feature vector, and W... ij Let b be the weight coefficient of the i-th task for the j-th target feature.i This is the baseline bias term for the i-th task; This serves as a task priority adjustment factor. Weights are generated in real-time from task features via a neural network, enabling precise matching of tasks and resources.

[0038] Obtain data on CPU and GPU temperature, power consumption, error rate, and fan speed.

[0039] ; ; ; ; ; ; ; ;

[0040] Current overall health score

[0041] ;

[0042] ΔX k To monitor the difference between the value and the threshold in real time, η is the exponential decay coefficient, κ is the error rate penalty factor, and k is the value in the indicator data; the geometric mean form ensures that exceeding the limit in any dimension will cause a sharp drop in HS, and the exponential term introduces the historical healthy memory of the error rate;

[0043] Integrating an LSTM time series prediction model incorporates the predicted values ​​into the current evaluation.

[0044] The sliding window captures the changing trends of health indicators, and the weight ω dynamically adjusts the ratio of historical and predicted values ​​in the i-th task state.

[0045] In a preferred embodiment of the above technical solution, the dynamic programming algorithm in S3 includes:

[0046] State S is defined as the resource state of the entire cluster, represented by the CS and HS score vectors of each node, and the task completion state when scheduling the i-th task. In each decision and state transition phase, decision D(i) is to select a target node N(j) for the current task T(i), which will lead to state transition S(i) -> S(i+1); that is, to consume the resources of the node and update its CS and HS.

[0047] Define a value function V(S, i) to represent the shortest estimated time required to schedule all remaining tasks starting from state S; V(S, i) = time spent on currently completed tasks + shortest estimated time spent on remaining tasks.

[0048] The adaptive scheduler uses a dynamic programming algorithm to find a decision sequence {D(1), D(2), ..., D(n)} to minimize V(S, 0), which is the total time taken from the initial state.

[0049] For each task i = 0 to n-1, each state S, and each available target node j; calculate the immediate cost: cost = the estimated execution time of task T(i) on target node N(j);

[0050] Calculate the new state S':

[0051] S' = Update resource state(S, j, T(i));

[0052] Update the value function; V(S', i+1) = min{ V(S', i+1), V(S, i) + cost}.

[0053] In a preferred embodiment of the above technical solution, step S6 includes:

[0054] It encapsulates the HTTP and SSH protocols, and defines basic operations on the target server through the abstract NetService interface; the HTTP protocol also encapsulates an HTTP-AGENT that needs to be deployed on the target server to realize HTTP communication, and the HTTP-AGETN completes the execution of the above instructions.

[0055] In a preferred embodiment of the above technical solution, S6 on the Linux node includes:

[0056] Pull the image from distributed storage by executing the `docker pull` command via SSH; start the container by calling `docker run` and inject environment variables via the `Exec` command in SSH; capture the container output in real time via SSH log stream, and automatically execute `docker restart` or roll back to the previous version of the image if an error is detected.

[0057] In a preferred embodiment of the above technical solution, S6 on the Windows node includes:

[0058] After downloading the image via HTTP, the Windows Docker API is called to parse the image and create a container; PowerShell scripts are executed using SSH to configure the container network, compensating for the shortcomings of the HTTP API in complex network scenarios; the container status is monitored through both the Windows Event Viewer and the HTTP / api / v1 / containers / {id} / logs interface.

[0059] The present invention also discloses a computer system, comprising:

[0060] processor;

[0061] Memory used to store processor-executable instructions;

[0062] The processor is configured to implement the method for centralized adaptive control scheduling of computing center resources as described in any one of claims 1 to 8 when executing the executable instructions.

[0063] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0064] Through a unified communication adaptation layer, it effectively manages heterogeneous computing nodes such as Linux and Windows; by utilizing time-series prediction and comprehensive scoring models, it transforms monitoring data into predictive "resource performance profiles," providing high-quality input for scheduling; through distributed storage and containerization technologies, it achieves second-level deployment and accurate restoration of the operating environment; by combining dynamic programming algorithms with real-time monitoring data, it makes scheduling decisions no longer locally optimal or immediately greedy, but rather globally near-optimal decisions based on future resource trends, significantly reducing the total task completion time and improving the overall cluster throughput.

[0065] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0066] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0067] Figure 1 This is a schematic diagram of the overall invention. Detailed Implementation

[0068] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0069] like Figure 1 As shown, this invention discloses a system and method for centralized adaptive control scheduling of computing center resources, including the following:

[0070] The specific implementation example of the training task is as follows:

[0071] S1. The user submits a computing power training task via API, which includes a DAG task topology definition file, such as YAML, specifying the dependencies between data preprocessing, training, and evaluation steps; the above task is sent to the DAG task orchestrator to form a list of tasks ready for scheduling.

[0072] S2, the intelligent monitoring and analysis module, is used to continuously collect node data and calculate the real-time CS and HS scores of each node. For example, if node N1, i.e., the GPU server, has a high CS score, it indicates that the GPU is idle, but its load is predicted to increase in 5 minutes; node N2 has a moderate but stable CS score, thus forming a real-time resource performance profile. At the same time, the intelligent monitoring and analysis module collects the raw indicators of different computing power node groups in the controlled computing power resource pool in real time. These different computing power node groups include Linux computing power node groups, Windows computing power node groups, and computing power node groups of other systems.

[0073] S3, the DAG task orchestrator parses dependency-based data, determines the first batch of executable data preprocessing tasks based on the task list provided in S1, and calls a dynamic programming algorithm in the adaptive scheduler to make decisions based on the real-time resource performance profile formed in S2.

[0074] S4, the adaptive scheduler dynamically dispatches tasks based on the task list of the DAG task orchestrator and the intelligent monitoring and analysis module. It determines that immediately assigning GPU training tasks to N1, while currently fast, will slow down after 5 minutes due to increased load; assigning them to N2, while initially slower, offers a better overall completion time. Data preprocessing tasks are scheduled to nodes with sufficient CPU resources; critical GPU training tasks are scheduled to the more stable N2 node.

[0075] S5, the distributed runtime and environment management module, is used to upload prepared Docker images containing Python, PyTorch, and CUDA to distributed storage.

[0076] S6, through a unified communication adaptation layer, pulls images from distributed storage on the target node and starts containers to execute tasks; the entire process does not require the user to care about differences in the operating system, achieving efficient and intelligent adaptive scheduling.

[0077] N1 is a GPU server node that executes non-persistent, interruptible, or short-term tasks, while N2 is a node that performs stable and sustainable tasks.

[0078] Preferably, the DAG task orchestrator is used to parse complex computational tasks submitted by users, decompose them into multiple sub-tasks, identify the dependencies between sub-tasks, and construct a directed acyclic graph (DAG).

[0079] The definition of nodes and edges determines all nodes in the graph, with each node representing an entity or task.

[0080] Define directed edges between nodes to represent dependencies or execution order; ensure no circular dependencies exist, meaning no node can start from a directed edge and eventually return to that node; the chosen data structures include: adjacency list, adjacency matrix, and in-degree list; where, adjacency list: each node maintains a list pointing to its successor nodes, saving space and easy to traverse; adjacency matrix: uses a two-dimensional array to represent the connection relationships between nodes, suitable for dense graphs; in-degree list: records the in-degree of each node, i.e., the number of edges pointing to that node, used for topological sorting;

[0081] When stored as a dictionary, the key is the task ID, and the value is a list of all direct successor task IDs of that task; for example, if task A depends on tasks B and C, then the list corresponding to A in the adjacency list is [B, C]. A two-dimensional boolean array is used to represent the dependencies between tasks; the rows and columns of the array correspond to the task IDs respectively. If task i depends on task j, then matrix [i][j] = True, otherwise False. A one-dimensional integer array records the in-degree of each task, that is, the number of predecessor tasks pointing to that task. For example, if task D has 8 predecessor tasks, then the in-degree list [D] = 8.

[0082] Initialize an empty graph structure, add nodes one by one, perform cycle detection when adding directed edges to prevent the formation of loops, and verify the integrity and correctness of the graph; provide API or DSL for users to define task flow; the orchestrator is responsible for managing the life cycle of tasks according to the topological order of the DAG, including ready, execution, blocking, and completion.

[0083] Users define task flows through a domain-specific language DSL or a visual interface, setting tasks A, B, C, and D; and task A has no dependencies, task B depends on task A, task C depends on tasks A and B, and task D depends on tasks A, B, and C.

[0084] Construct an adjacency list based on dependencies: {A: [], B: [A], C: [A, B], D: [B, C]};

[0085] Create a 4×4 two-dimensional array. Assume the task IDs are A=0, B=1, C=2, D=3. Fill the dependency relationships according to the adjacency list. Traverse the adjacency list and count the number of times each task is depended on by other tasks to get [0, 1, 2, 2], that is, A has an in-degree of 0, B has an in-degree of 1, C has an in-degree of 2, and D has an in-degree of 2.

[0086] Initialize the executable task queue, scan the in-degree table, and add all tasks with an in-degree of 0 (i.e., no predecessor task) to the queue. For example, initially only task A has an in-degree of 0, so the queue is [A]. Dynamically handle task dependencies; remove task A from the queue, mark it as "in execution", and remove all its outgoing edges from the adjacency list and adjacency matrix, i.e., delete the successor relationship of A, and set matrices [B][A] and [C][A] to False; traverse the successor tasks B and C of A, and update them by decrementing their in-degree by 1.

[0087] Add task B, whose in-degree is 0 after the update, to the queue. The queue is now [B]. Take out task B, execute it, and then remove its outgoing edges; for example, if D depends on B.

[0088] In each step, if there are multiple tasks with an in-degree of 0 in the queue, and in subsequent steps B and C both have an in-degree of 0, then these tasks are executed in parallel.

[0089] Synchronize subsequent dependent tasks using shared variables or semaphores. For example, task D needs to wait for both B and C to complete before it can execute, so it is necessary to check the completion status of both.

[0090] The DAG task orchestrator must ensure that task dependencies are acyclic, otherwise deadlock will occur; after each update of the in-degree table, it checks whether there are any cases where the in-degree of a task has not been reduced to 0 and the queue is empty.

[0091] If a circular dependency exists, such as A→B→A, an exception will be thrown and scheduling will be terminated, prompting the user to correct the task dependency relationship.

[0092] The status needs to be tracked in real time during task execution. If a task fails, such as due to insufficient resources, it should be marked as FAILED and the in-degree of the affected task should be recalculated.

[0093] For example, if task C fails, the in-degree of task D, which depends on C, remains non-zero. The execution of D needs to be paused until C retryes successfully or the user intervenes.

[0094] The unified communication adaptation layer is used to encapsulate various communication protocols, and to perform data transmission and execution operations for task dispatch instructions and environment storage and distribution instructions.

[0095] It encapsulates the HTTP and SSH protocols, defining basic operations on the target server through an abstract NetService interface, such as Ping, Empty, Exist, Mkdir, Del, Find, Exec, Run, Stop, and Untell. HTTP and SSH respectively implement these capabilities. The HTTP protocol also encapsulates an HTTP-AGENT that needs to be deployed on the target server to enable HTTP communication, while the HTTP-AGENT executes the commands.

[0096] For Linux nodes, commands are executed and files are transferred via the SSH protocol; for Windows nodes, the same functionality is achieved by calling their built-in REST API, i.e., the HTTP protocol. This layer normalizes operations in heterogeneous environments; using the definitions in NetService, it obtains the corresponding communication method through the actual communication method at runtime and calls the target capability; the specific implementation of NetService then communicates through the corresponding communication method and completes the execution of the action.

[0097] Verify node reachability by calling the NetService.Ping() method. For example, send an SSH heartbeat to a Linux node, or send an HTTP GET request to a Windows node's management port, such as port 5985 of WinRM.

[0098] If a node does not respond, the adaptation layer triggers a retry mechanism and records the faulty node to the blacklist.

[0099] If monitoring detects an anomaly in the SSH service of a Linux node, the adaptation layer automatically switches to a backup HTTP channel, such as through a deployed lightweight HTTP proxy, to ensure data continuity.

[0100] For mixed node groups, such as clusters containing both Linux and Windows, the adaptation layer uses both protocols to collect data in parallel and merges the results by aligning them with timestamps.

[0101] For Linux nodes, the task command `python preprocess.py` is encapsulated into an SSH `Exec` command, and dependent files, such as input data, are uploaded via the `UnTar` operation.

[0102] For Windows nodes, the command is converted into an HTTP POST request to its task management interface, and the file is uploaded via a Multipart form.

[0103] If the distributed storage is NFS, which is common in Linux environments, the adapter layer mounts the storage via SSH's Mount operation and directly copies the image file.

[0104] If the storage is a Windows SMB share, it is uploaded via the HTTP / api / v1 / storage / upload interface, and the network drive is mapped by executing the net use command on the target node using SSH.

[0105] After the upload is complete, the adapter layer calculates the SHA256 hash of the image on the Linux node via SSH, and verifies it on the Windows node using PowerShell's Get-FileHash command to ensure cross-platform data consistency.

[0106] When the target node pulls the image and starts the container, the protocol interaction of the adaptation layer is as follows:

[0107] Linux node:

[0108] Pull an image from distributed storage by executing the `docker pull` command via SSH.

[0109] The container is started by calling `docker run`, and environment variables such as `CUDA_VISIBLE_DEVICES` are injected via the `Exec` command in SSH.

[0110] Capture container output in real time via SSH log stream. If an error is detected, such as OOM, automatically execute dockerrestart or roll back to the previous version of the image.

[0111] Windows node:

[0112] After downloading the image via HTTP, the Windows Docker API is called, such as POST / containers / create, to parse the image and create a container.

[0113] Use SSH to execute PowerShell scripts to configure container networks, such as docker network connect, to compensate for the shortcomings of the HTTP API in complex network scenarios.

[0114] The container status can be monitored using both the Windows Event Viewer and the HTTP / api / v1 / containers / {id} / logs interface.

[0115] When a Linux node reports a container die event via SSH, the adaptation layer parses the container ID in the event; it then queries the status of the corresponding container on the Windows node via HTTP or SSH; if the Windows container has not terminated, it automatically triggers a forced stop command, prioritizing the HTTP command DELETE / containers / {id}, and if that fails, it falls back to the SSH command dockerstop.

[0116] It is a crucial link, through which the basic communication capabilities of the entire system are completed, giving the system's communication capabilities scalability and ease of use.

[0117] It truly achieves "write once, run anywhere," meaning users don't need to care whether the underlying technology is Linux's Docker commands or Windows' REST API. The adaptation layer automatically completes all conversions, automatically selecting the optimal protocol based on real-time network conditions, node load, and operation type to avoid performance bottlenecks caused by fixed protocols. When a protocol fails, the adaptation layer can seamlessly switch to a backup protocol to ensure that tasks are not interrupted, such as automatically restarting the container with SSH after an HTTP timeout.

[0118] The intelligent monitoring and analysis module is used to collect performance data of each computing node in real time and perform in-depth analysis through a lightweight time-series prediction model, which goes beyond simple instantaneous value monitoring.

[0119] Create real-time performance profiles: not only record current values, but also predict resource performance trends in the near future based on historical time-series data, and predict the trend of CPU load in the next 2 minutes.

[0120] A comprehensive health score is calculated, and a multi-indicator weighted evaluation model is constructed to calculate a real-time comprehensive health score (HS) and an available computing power score (Capacity Score, CS) for each node.

[0121] The module's output is no longer fragmented metrics, but rather predictive data that has undergone intelligent analysis and quantification, providing the dynamic programming scheduler with high-quality and forward-looking decision-making support.

[0122] S cpu = 1.0 - (Current CPU utilization / 100%)

[0123] S gpu = 1.0 - (Current GPU utilization / 100%)

[0124] S mem = 1.0 - (Current memory utilization / 100%)

[0125] S ssd = 1.0 - (Current SSD utilization / 100%)

[0126] ;

[0127] in, This represents the base weight of CPU resources in the overall computing power calculation, used to adjust the initial contribution ratio of CPU performance to the system's computing power. This represents the weighted nonlinear contribution of CPU resources. α is the "master switch" for this term, having high priority and controlling the basic proportion of CPU in the overall computing power. The CPU resource index adjusts the weight of CPU resources in the overall computing power, reflecting the non-linear impact of CPU performance improvement on system computing power. This is a GPU resource index that adjusts the weighting of GPU resources' contribution to overall computing power, capturing the non-linear amplification effect of GPU parallel computing capabilities. It is a memory resource index that adjusts the weight of memory resources in the overall computing power and reflects the non-linear constraint of memory capacity on data throughput. The synergistic interaction index adjusts the strength of the synergistic effect among CPU, GPU, and memory, capturing the multiplier effect of nonlinear interactions between resources. When ε>0, resource synergy improves overall computing power, such as efficient matching between GPU and memory; when ε<0, resource competition leads to a decrease in computing power, such as CPU and GPU competing for memory. β represents the CPU resources raised to the power of β, indicating the non-linear contribution of CPU performance to the system's computing power. γ represents the power of GPU resources, indicating the nonlinear amplification effect of GPU parallel computing capabilities; δ represents the memory resources raised to the power of δ, indicating the non-linear constraint of memory capacity on data throughput. ε represents the product of CPU, GPU, and memory resources, indicating the nonlinear multiplier of the synergistic effect among the three. The sum of CPU and GPU resources minus the hyperbolic tangent of memory resources represents the soft constraint of resource balance. By dynamically adjusting the resource contribution through exponential parameters β, γ, δ, and ε, the limitations of the linear weighted model are overcome. The model can calculate the nonlinear contribution of each resource in real time, providing the scheduler with accurate resource allocation basis and improving the overall computing power utilization.

[0128] The following content explains the cross-term mechanism of β as a CPU resource index, in the second term. In this context, β does not directly regulate GPU resources, but rather serves as a system-level coordination coefficient, reflecting the coupling strength between the CPU and GPU. For example, in GPU-intensive computational tasks, the CPU's scheduling efficiency β significantly impacts the actual utilization rate S of GPU resources. gpu Therefore, β acts as a moderating factor here, rather than a simple exponent. This design aligns with the physical constraints of "CPU-GPU collaborative bandwidth" in real-world systems. When the CPU cannot supply data in a timely manner, the GPU's computing power cannot be fully utilized, hence β needs to participate in the correction of the GPU term. Although the parameter symbol β appears in cross-term usage between the CPU and GPU terms, β is always associated with CPU-related effects through its physical meaning and the mathematical constraint β>0, ensuring parameter identifiability. The influence weights of β on the CPU and GPU can be independently solved, avoiding estimation biases caused by parameter coupling.

[0129] γ is the GPU resource index. In memory tuning, γ essentially acts as a GPU-memory interaction weight. Memory bandwidth S mem The impact on GPU computing is non-linear: when video memory is insufficient, the GPU needs to frequently interact with main memory, leading to performance degradation; and γ, as a GPU resource index, can quantify this interaction loss. For example, in deep learning training, video memory usage is related to the batch size by a power of γ. The larger the value of γ, the more significant the constraint of memory on the GPU. Therefore, γ acts as a non-linear correction term here.

[0130] δ is the memory resource index. The δ-cooperative interaction directly represents the strength of the synergistic effect among the three resources. This interaction term is designed based on system-level bottleneck theory: when the CPU, GPU, and memory are all under high load simultaneously, they compete for bus bandwidth and power consumption, resulting in actual computing power being lower than the sum of their individual components. δ, as a synergy coefficient, quantifies this "1+1+1<3" competitive effect. For example, in HPC scenarios, insufficient memory bandwidth directly limits the data transfer efficiency between the CPU and GPU; the larger the δ value, the more significant the synergistic loss.

[0131] In the traditional linear model, the computational power fraction CS = α·S can be used. cpu +β·S gpu +γ·S mem It cannot capture non-linear coupling between resources. For example, when S cpu =100%, S gpu When the value is 100%, simple linear superposition will overestimate the actual computing power, while interaction terms... By introducing the ε power and δ coefficient, this "saturation effect" is accurately characterized. Actual test data shows that when all three resources exceed their load simultaneously, the contribution ratio of the interaction term increases, significantly outperforming the linear model.

[0132] Nonlinear saturation constraints are introduced to prevent the distortion of computing power scores caused by the infinite growth of total resources. The asymptotic property of the tanh function is used, where the output approaches 1 as the input approaches infinity, simulating the physical law that "resource utilization has an upper limit" in real-world systems.

[0133] In AI training tasks, as CPU beta scheduling latency increases, GPU utilization S... gpu It will drop, at this time This item can capture the effect of CPU slowing down GPU performance. In database query scenarios, memory bandwidth S mem Insufficient processing time will cause the GPU to wait during computation. The effect of memory constraints on GPUs can be quantified.

[0134] In deep learning training computational tasks, the CPU is responsible for task scheduling and data preprocessing, the GPU handles core computation, and memory stores intermediate data and model parameters. These three components together form a closed loop of computation-storage-scheduling, and their utilization directly affects the efficiency of computational output. Solid-state drives (SSDs), on the other hand, primarily handle persistent data storage and loading, falling under the data input / output stage. Their direct impact on computational power is indirectly reflected through mechanisms such as memory caching and preloading; they do not directly participate in the calculation of computational power scores. While SSD read / write performance affects task startup speed, once data is in memory, subsequent computations mainly rely on CPU / GPU / memory resources, making CS calculations more accurate.

[0135] Build a dynamic weight generator based on task type.

[0136]

[0137] σ is the sigmoid function, TaskFeature includes task type features such as computational cost, memory requirements, and I / O bandwidth, and Priority is the real-time priority coefficient; where the subscript j is the dimension index of the target feature vector, and W... ij Let b be the weight coefficient of the i-th task for the j-th target feature. i This is the baseline bias term for the i-th task; This serves as a task priority adjustment factor. Weights are generated in real-time from task features via a neural network, enabling precise matching of tasks and resources.

[0138] TaskFeature includes multi-dimensional features such as computational cost (FLOPS) and memory requirement (GB). Standardization is needed to eliminate dimensional differences: for each feature dimension j, its mean and standard deviation are calculated. The original values ​​are then substituted into the normalization formula, mapping the feature values ​​to the [0,1] interval. Linear combinations can be performed for features with computational cost = 1000 FLOPS and memory requirement = 4GB. These multi-dimensional features are then standardized and weighted summed into a single weight value. This calculation comprehensively reflects the intensity of the task's demand for various resources.

[0139] When λ>1, high-priority tasks receive more resources; when λ<1, the impact of priority on resource allocation weakens, resulting in a more balanced allocation.

[0140] Obtain data on CPU and GPU temperature, power consumption, error rate, and fan speed.

[0141] ; ; ; ; ; ; ; ;

[0142] Component temperatures are collected in real time by CPU and GPU temperature sensors and monitored by the power management module or current / voltage sensors to reflect the CPU and GPU power consumption status. System logs and error detection modules record the number or frequency of errors during device operation, such as disk I / O errors and program crashes. Fan speed indicators are provided by the fan controller or speed sensor.

[0143] The data acquisition process involves the data acquisition module entering the database, performing standardized preprocessing, and storing the data in the database to ensure its real-time performance and accuracy.

[0144] Current overall health score

[0145] ;

[0146] ΔX k To monitor the difference between the real-time value and the threshold, η is the exponential decay coefficient, κ is the error rate penalty factor, and k is the value in the indicator data. The geometric mean form ensures that exceeding the limit in any dimension will cause a sharp drop in HS, and the exponential term introduces the historical health memory of the error rate.

[0147] ΔX k It directly reflects the deviation of the current state from the normal range, such as ΔX when the temperature exceeds the threshold. k A positive value indicates a risk of overheating; when power consumption is below the threshold, ΔX... k A negative value indicates that the equipment is operating inefficiently.

[0148] To prevent different indicators ΔX k To address the issue of different dimensions between the threshold value (thresholdk) and other metrics, we need to normalize the various indicators and map them to the [0,1] interval. ErrorRate is the global error rate and does not distinguish the error contribution of different devices. Therefore, we need to define an independent error rate (ErrorRate) for each device and adjust the penalty factor accordingly.

[0149] An LSTM time-series forecasting model is integrated to incorporate forecasts for the next two minutes into the current evaluation.

[0150] The sliding window captures the changing trend of health indicators, and the weight ω dynamically adjusts the ratio of historical and predicted values ​​in the i-th task state.

[0151] Tasks are prioritized for allocation to nodes with high CS and HS scores; a deterioration in either metric will cause a sharp drop in the overall score. The higher the HS score, the more stable and reliable the node. A new computing power scheduling paradigm has been constructed, moving from "instantaneous monitoring" to "continuous health management" and from "passive response" to "proactive prediction," which can significantly improve the operational efficiency and reliability of computing centers.

[0152] The distributed runtime and environment management module is used to ensure the consistency and reproducibility of the computing task environment.

[0153] Python also supports using Docker containers or pre-configured virtual machine images to directly encapsulate the runtime environment using intermediate environment images such as .venv and Conda.

[0154] All image files are stored in a distributed file system, such as Ceph or MinIO.

[0155] When the scheduler issues a task, it uses a unified communication layer to instruct nodes to pull the required environment image from distributed storage and start it quickly, thereby ensuring that the task can obtain a consistent running environment on any node.

[0156] The environment is stored on the internal OSS object storage, which enables the environment to be retrieved on the target machine. The internal OSS uses MINIO for support.

[0157] The adaptive scheduler is used to perform optimal task allocation based on DAG information, real-time resource profiles from the monitoring and analysis module, and task resource requirements.

[0158] Dynamic programming algorithm: This modeles the task scheduling process as a multi-stage decision optimization process. The goal is to find a scheduling scheme that minimizes the total task completion time (Makespan) or maximizes global resource utilization. The state S is defined as the resource state of the entire cluster when scheduling the i-th task, represented by the CS and HS score vectors of each node and the task completion state. In each stage, decision D(i) is to select a target node N(j) for the current task T(i); this decision leads to a state transition S(i) -> S(i+1), i.e., consuming the node's resources and updating its CS and HS.

[0159] Define a value function V(S, i) to represent the shortest estimated time required to schedule all remaining tasks starting from state S; V(S, i) = time spent on currently completed tasks + shortest estimated time spent on remaining tasks.

[0160] The adaptive scheduler uses a dynamic programming algorithm to find a decision sequence {D(1), D(2), ..., D(n)} to minimize V(S0, 0), which is the total time taken from the initial state.

[0161] For each task i = 0 to n-1; each state S; each available target node j; calculate the immediate cost: cost = the estimated execution time of task T(i) on node N(j);

[0162] Calculate the new state S':

[0163] S' = Update resource state(S, j, T(i));

[0164] Update the value function; V(S', i+1) = min{ V(S', i+1), V(S, i) + cost}.

[0165] The estimated time for each task is calculated on each node, and the node with the best time is selected for execution. Since precise DP calculation is very large, this invention adopts rolling time-domain optimization: DP optimization calculation is only performed on the tasks to be scheduled within a short time window in the future, and then the scheduling is re-planned based on the new monitoring data, thereby realizing real-time adaptive scheduling.

[0166] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for centralized adaptive control and scheduling of computing center resources, characterized in that, include: S1, the user submits a computing power training task via API, which includes a DAG task topology definition file that specifies the dependencies between data preprocessing, training, and evaluation steps; the above task is sent to the DAG task orchestrator to form a list of tasks ready for scheduling; S2, the intelligent monitoring and analysis module, is used to continuously collect node data and calculate the real-time available computing power score CS and comprehensive health score HS of each node; it also collects the original indicators of different computing power node groups in the controlled computing power resource pool in real time. S3, the DAG task orchestrator parses dependency-based data, determines the first batch of executable data preprocessing tasks based on the task list provided in S1, and calls a dynamic programming algorithm in the adaptive scheduler to make decisions based on the real-time resource performance profile formed in S2. S4, the adaptive scheduler dynamically dispatches tasks based on the task list of the DAG task orchestrator and the intelligent monitoring and analysis module, scheduling data preprocessing tasks to nodes with sufficient CPU resources; and scheduling critical GPU training tasks to the more stable N2 nodes. S5, the distributed runtime and environment management module, is used to upload the prepared Docker image containing Python, PyTorch, and CUDA to distributed storage. S6, through a unified communication adaptation layer, pulls images from distributed storage on the target node and starts containers to execute tasks.

2. The method for centralized adaptive control and scheduling of computing center resources according to claim 1, characterized in that, S1 includes: Construct a Directed Acyclic Graph (DAG); define nodes and edges to determine all nodes in the graph, each node representing an entity or task; define directed edges between nodes to represent dependencies or execution order; ensure no circular dependencies exist, i.e., it is impossible to start from a node and return to that node along a directed edge; the chosen data structures include: adjacency list, adjacency matrix, and in-degree list; where, adjacency list: each node maintains a list pointing to its successor nodes, saving space and easy to traverse; adjacency matrix: uses a two-dimensional array to represent the connection relationship between nodes, suitable for dense graphs; in-degree list: records the in-degree of each node for topological sorting.

3. The method for centralized adaptive control and scheduling of computing center resources according to claim 1, characterized in that, S1 further includes: Initialize an empty graph structure, add nodes one by one, perform cycle detection when adding directed edges, and provide API or DSL for users to define task flow; the DAG task orchestrator is responsible for managing the lifecycle of tasks according to the topological order of the DAG; The user sets tasks A, B, C, and D; and task A has no dependencies, task B depends on task A, task C depends on tasks A and B, and task D depends on tasks A, B, and C; construct an adjacency list based on the dependencies: {A: [], B: [A], C: [A, B], D: [B, C]}; Create a 4×4 two-dimensional array and fill it with dependencies according to the adjacency list; traverse the adjacency list and count the number of times each task is depended on by other tasks to get [0, 1, 2, 2]. Initialize the executable task queue, scan the in-degree table, and add all tasks with an in-degree of 0 to the queue; initially, only task A has an in-degree of 0, so the queue is [A]; dynamically handle task dependencies; remove task A from the queue, mark it as "in execution", and remove all its outgoing edges from the adjacency list and adjacency matrix, i.e., delete the successor relationship of A, and set matrices [B][A] and [C][A] to False; traverse the successor tasks B and C of A, and update them by decrementing their in-degree by 1; Add task B, which has an in-degree of 0 after the update, to the queue; the queue is now [B]. Take out task B, execute it, and remove its outgoing edges. In each step, if there are multiple tasks with an in-degree of 0 in the queue, execute these tasks in parallel.

4. The method for centralized adaptive control and scheduling of computing center resources according to claim 1, characterized in that, S2 includes: The performance data of each computing node in the intelligent monitoring and analysis module is normalized. Each raw metric is converted into a score S between 0 and 1.

0. value ; S cpu = 1.0 - (Current CPU utilization / 100%); S gpu = 1.0 - (current GPU utilization / 100%); S mem = 1.0 - (current memory utilization / 100%); S ssd = 1.0 - (Current SSD utilization / 100%); ； in, This represents the base weight of CPU resources in the overall computing power calculation, used to adjust the initial contribution ratio of CPU performance to the system's computing power. This represents the weighted nonlinear contribution of CPU resources. α is the master switch for this term, controlling the basic proportion of CPU in the overall computing power. The CPU resource index adjusts the weight of CPU resources in the overall computing power, reflecting the non-linear impact of CPU performance improvement on system computing power. This is a GPU resource index that adjusts the weighting of GPU resources' contribution to overall computing power, capturing the non-linear amplification effect of GPU parallel computing capabilities. It is a memory resource index that adjusts the contribution weight of memory resources to the overall computing power, reflecting the non-linear constraint of memory capacity on data throughput. The synergistic interaction index adjusts the strength of the synergistic effect among CPU, GPU, and memory, capturing the multiplier effect of nonlinear interactions between resources. When ε>0, resource synergy improves overall computing power; when ε<0, resource competition leads to a decrease in computing power. β represents the CPU resources raised to the power of β, indicating the non-linear contribution of CPU performance to the system's computing power. γ represents the nonlinear amplification effect of GPU parallel computing capabilities, which is the power of GPU resources. δ represents the memory resources raised to the power of δ, indicating the non-linear constraint of memory capacity on data throughput. ε is the power of the product of CPU, GPU, and memory resources, representing the nonlinear multiplier of the synergistic effect among the three. The sum of CPU and GPU resources minus the hyperbolic tangent of memory resources represents the soft constraint of resource balance.

5. The method for centralized adaptive control and scheduling of computing center resources according to claim 1, characterized in that, S2 includes: Build a dynamic weight generator based on task type. ； σ is the sigmoid function, TaskFeature contains task type features, and Priority is the real-time priority coefficient; where the subscript j is the dimension index of the target feature vector, and W... ij Let b be the weight coefficient of the i-th task to the j-th target node. i This is the baseline bias term for the i-th task; It serves as a task priority adjustment factor; the weights are generated in real time by the task features through a neural network to achieve precise "task-resource" matching. Obtain data on CPU and GPU temperature, power consumption, error rate, and fan speed. ； ； ； ； ； ； ； ； Current overall health score ； ΔX k To monitor the difference between the value and the threshold in real time, η is the exponential decay coefficient, κ is the error rate penalty factor, and k is the value in the indicator data; the geometric mean form ensures that exceeding the limit in any dimension will cause a sharp drop in HS, and the exponential term introduces the historical healthy memory of the error rate; Integrating an LSTM time series prediction model incorporates the predicted values ​​into the current evaluation. The sliding window captures the changing trends of health indicators, and the weight ω dynamically adjusts the ratio of historical and predicted values ​​in the i-th task state.

6. The method for centralized adaptive control and scheduling of computing center resources according to claim 1, characterized in that, The dynamic programming algorithm in S3 includes: State S is defined as the resource state of the entire cluster, represented by the CS and HS score vectors of each node, and the task completion state when scheduling the i-th task. In each decision and state transition phase, decision D(i) is to select a target node N(j) for the current task T(i), which will lead to state transition S(i) -> S(i+1); that is, to consume the resources of the node and update its CS and HS. Define a value function V(S, i) to represent the shortest estimated time required to schedule all remaining tasks starting from state S; V(S, i) = time spent on currently completed tasks + shortest estimated time spent on remaining tasks. The adaptive scheduler uses a dynamic programming algorithm to find a decision sequence {D(1), D(2), ..., D(n)} to minimize V(S, 0), which is the total time taken from the initial state. For each task i = 0 to n-1, each state S, and each available target node j; calculate the immediate cost: cost = the estimated execution time of task T(i) on target node N(j); Calculate the new state S': S' = Update resource state(S, j, T(i)); Update the value function; V(S', i+1) = min{ V(S', i+1), V(S, i) + cost}.

7. The method for centralized adaptive control and scheduling of computing center resources according to claim 1, characterized in that, S6 includes: It encapsulates the HTTP and SSH protocols and defines basic operations on the target server through an abstract NetService interface; the HTTP protocol also encapsulates an HTTP-AGENT; the HTTP-AGENT needs to be deployed on the target server to implement HTTP communication, and the HTTP-AGENT executes the above instructions.

8. The method for centralized adaptive control and scheduling of computing center resources according to claim 1, characterized in that, The S6 on the Linux node includes: Pull the image from distributed storage by executing the `docker pull` command via SSH; start the container by calling `docker run` and inject environment variables via the `Exec` command in SSH; capture the container output in real time via SSH log stream, and automatically execute `docker restart` or roll back to the previous version of the image if an error is detected.

9. The method for centralized adaptive control and scheduling of computing center resources according to claim 1, characterized in that, The S6 on the Windows node includes: After downloading the image via HTTP, the Windows Docker API is called to parse the image and create a container; PowerShell scripts are executed using SSH to configure the container network, compensating for the shortcomings of the HTTP API in complex network scenarios; the container status is monitored through both the Windows Event Viewer and the HTTP / api / v1 / containers / {id} / logs interface.

10. A computer system, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method for centralized adaptive control scheduling of computing center resources as described in any one of claims 1 to 9 when executing the executable instructions.

Images