A distributed computing power intelligent scheduling system and method

Abstract

This invention proposes a distributed intelligent computing power scheduling system and method, relating to the field of computing power scheduling technology. It constructs a computing power resource pool and assigns an identifier to each resource node; generates a comprehensive score for each resource node; quickly locates resource nodes with abnormal states using their identifiers; adjusts the comprehensive score of resource nodes with abnormal states using state adjustment parameters; receives task requirements submitted by users, extracts task features, and integrates the task features, task requirements, and estimated output data size into a task vector; sorts the task vector set in descending order; performs preliminary screening of the task vector set based on task requirements; and selects the optimal resource node for the selected task vectors through iterative matching using a scheduling model based on the task features of the selected task vectors.

Description

Technical Field

[0001] This invention proposes a distributed intelligent computing power scheduling system and method, which relates to the field of computing power scheduling technology. Background Technology

[0002] With the rapid development of fields such as artificial intelligence, metaverse, and high-performance computing, the global scale of intelligent computing power is growing rapidly. The proportion of unstructured data (such as text and images) is surging, and traditional computing architectures are struggling to meet real-time and elastic requirements. Distributed scheduling is needed to optimize resource utilization, making heterogeneous computing power (CPU, GPU, FPGA, ASIC, etc.) the mainstream, with significant differences in type, performance, and network conditions. While GPUs excel at parallel computing and FPGAs are suitable for customized acceleration, existing systems lack a unified solution for managing heterogeneous resources, making resource pooling difficult. Intelligent scheduling is urgently needed to achieve computing-network convergence.

[0003] Traditional centralized scheduling systems struggle to perceive the dynamic status of heterogeneous computing power in real time, leading to inaccurate task-resource matching. Scheduling based on routing tables or fixed rules cannot adapt to sudden load changes, resulting in poor resource elasticity and scalability. Existing scheduling systems do not fully coordinate network status with computing resources, causing network bottlenecks to be ignored during task allocation, exacerbating congestion and packet loss. The centralized controller has limited processing power and cannot support the rapid expansion of large-scale distributed nodes, easily becoming a performance bottleneck. It also has insufficient response to node failures or timeouts, resulting in low efficiency in task rescheduling and poor system fault tolerance. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a distributed computing power intelligent scheduling method, comprising the following steps:

[0005] Step S1: Construct a computing resource pool and assign an identifier to each resource node;

[0006] Step S2: Generate a comprehensive score for each resource node. For resource nodes with abnormal status, quickly locate them using identifiers and adjust the comprehensive score of the resource nodes with abnormal status using status adjustment parameters.

[0007] Step S3: Receive the task requirements submitted by the user, extract the task features, estimate the size of the output data after the task is completed, and integrate the task features, task requirements and estimated size of the output data into a task vector.

[0008] Step S4: Sort the task vector set in descending order, perform preliminary screening of the task vector set based on task requirements, and select the optimal resource node for the task characteristics of the screened task vectors through iterative matching of the scheduling model.

[0009] In a preferred embodiment, step S3 includes the following steps:

[0010] Step 3.1: Receive user task requirements, extract task features, analyze task input data, and calculate the theoretical time expected to execute the task.

[0011] Step 3.2: Analyze the dependencies between the task input data and estimate the size of the output data after the task is completed based on the theoretical time expected to be executed.

[0012] Step 3.3: Integrate the task characteristics, task requirements, and estimated output data size into a multi-dimensional vector to obtain the task vector.

[0013] In a preferred embodiment, in step S3.2, the estimated size Y of the output data after the task is completed is:

[0014]

[0015] N task requirements are sorted in descending order of their expected execution time, forming a task chain S. n The order; the coefficient of the nth data in the input data is L. n When task S n Data coefficients L need to be adjusted. n To compress, let L n =0.5, Task S n+1 Based on task S n The output needs to be processed by the data coefficients L. n+1 To expand the data, let L n+1 =2, Task S n+2 Task S needs to be re-evaluated. n+1 If the output of L is filtered, then let L n+2 =0.3.

[0016] In a preferred embodiment, in step S4, T = {t1, t2, ... t} k …t Q Let} represent the set consisting of all task vectors, where t k Represents the k-th task vector, and each task vector t k It has the following attributes:

[0017] G k Estimate the standardized value of the output data size for the k-th task vector;

[0018] r kJ This represents the resource requirement vector of the Jth dimension of the kth task vector;

[0019] f kJ This represents the task feature vector of the Jth dimension of the kth task vector;

[0020] The task vector set T is standardized to the value G based on the estimated output data size. k Sort the data in descending order to obtain a new task vector sequence T' = {t'1, t'2, ..., t'}. k …t' Q}, where G1≥G2…≥G k ≥…≥G Q ;

[0021] Initialize the set of scheduled tasks Remaining resource demand vector R J =[R1,R2,...,R A A represents the initial total available resources; iterate through each task t' in the sorted task sequence T'. k Check if r is satisfied. kJ ≤R J If for all r kJ All have r kJ ≤R J Then the task t' k Add to the set of scheduled tasks S, S = S∪{t' k}, and simultaneously update the remaining resource vector, then execute R' J =R J -R kJ ,Deduct task t' k Resource R used kJ If the check does not meet the condition r kJ ≤R J Skip this task and proceed to check the next one.

[0022] In a preferred embodiment, in step S4, an optimization function h is used to optimize the scheduling model.

[0023]

[0024] Among them, T ex MaxU represents the theoretical execution time of the task; MaxU represents the maximum utilization rate of the resource nodes; MaxU T The preset utilization threshold is defined by Co; the task feature is defined by Co; and the weighting coefficients are α, β, and γ.

[0025] Input the optimization function h into the following scheduling model:

[0026] Z(s,a)←(1-δ)Z(s,a)+δ[h+μmax a′ Z(s′,a′)]

[0027] Where: Z(s,a) is the function of taking scheduling operation a in state s, and its output value is the Z value; δ is the learning rate, μ is the learning factor, and s' is the new state transitioned to after taking scheduling operation a; max a′ Z(s′,a′) is the maximum scheduling value for all actions that take scheduling operation a′ under the new state s′.

[0028] State s consists of the set of scheduled tasks S and the vector of remaining resources R. J Together they form s = (S, R) J );

[0029] After multiple iterations, calculate Z(s,a) for all scheduling operations a, select the scheduling operation a with the largest Z(s,a) output value, and the associated resource node is the optimal resource node that matches the task scheduling target.

[0030] In a preferred embodiment, in step S3.2, the total computational load (FLOP) of the task is calculated. 总 :

[0031] FLOP 总 =FLOP 单 ×(Total data volume / Batch Size)×Epoch;

[0032] Among them, FLOP 单 The computational cost per batch; Batch Size is the number of samples computed per batch; Epoch is the number of training epochs.

[0033] The effective computing power that computing resources can provide:

[0034] Effective computing power = Single-point computing power × Number of points × Utilization rate;

[0035] The theoretical time to complete the task is:

[0036] Theoretical time = FLOP 总 Effective computing power.

[0037] In a preferred embodiment, in step S3.3, the task vector t is assumed to consist of the following dimensions:

[0038] t = [f1, f2, ..., f J ,…f M ,r1,r2,…,r J ,…r B [,G];

[0039] G is the standardized value of the reference value for the estimated output data size of the task vector; r J f represents the J-th resource demand dimension vector among B resource demand dimension vectors; JLet J represent the J-th task feature dimension vector among M task feature dimension vectors.

[0040] This invention also proposes a distributed computing power intelligent scheduling system to implement the above-mentioned distributed computing power intelligent scheduling method, characterized in that it includes: a resource management module, a resource evaluation module, a task parsing and feature extraction module, a task vectorization module, and a scheduling algorithm module;

[0041] The resource management module is used to build a distributed computing power resource pool and assign a unique identifier to each node.

[0042] The resource assessment module is used to generate a comprehensive score for each resource node. For resource nodes with abnormal status, the module can quickly locate them using an identifier and adjust the comprehensive score of the resource nodes with abnormal status using status adjustment parameters.

[0043] The task parsing and feature extraction module is used to receive task requirements submitted by users, calculate the theoretical time expected to be executed, and estimate the size of the output data after the task is completed.

[0044] The task vectorization module is used to integrate task features, task requirements, and the estimated size of the output data into a multi-dimensional vector to obtain the task vector.

[0045] The scheduling algorithm module is used to sort the task vector set in descending order, perform preliminary screening of the task vector set based on task requirements, and select the optimal resource node for the task characteristics of the screened task vectors through iterative matching of the scheduling model.

[0046] Compared with the prior art, the present invention has the following beneficial technical effects:

[0047] By constructing a computing resource pool, centralized management of physical / virtual resources is achieved. Combined with a comprehensive scoring system, node performance (such as computing power, storage, network, etc.) is dynamically evaluated, and high-scoring nodes are prioritized to maximize resource utilization.

[0048] The comprehensive scoring and priority ranking mechanism can detect the node load status in real time, avoid concentrated load testing on high-load nodes, achieve global load balancing, and improve system stability.

[0049] Task vectors quantify task characteristics (such as computational complexity, real-time requirements, and data volume) into structured parameters. Combined with node priorities, scheduling algorithms enable precise matching of tasks and resources, reducing ineffective resource allocation.

[0050] Priority sorting and rapid location mechanisms shorten task waiting time, and high-priority tasks are allocated high-quality resources first, significantly improving the execution efficiency of critical tasks.

[0051] This invention achieves multi-dimensional optimization in terms of resource utilization, task execution efficiency, system reliability, and scalability through the synergistic effect of resource pooling management, dynamic scoring, task feature quantification, and intelligent scheduling algorithms. It is suitable for the high-efficiency computing power scheduling needs in distributed scenarios such as cloud computing and edge computing. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart of the distributed computing power intelligent scheduling method of the present invention;

[0054] Figure 2 This is a topology diagram of the resource nodes in this invention;

[0055] Figure 3 This is a flowchart illustrating how the present invention integrates task characteristics and requirements into a task vector;

[0056] Figure 4 This is a structural diagram of the distributed computing power intelligent scheduling system of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0058] In the accompanying drawings of specific embodiments of the present invention, in order to better and more clearly describe the working principle of each component in the system and show the connection relationship of each part in the device, only the relative positional relationship between each component is clearly distinguished. It does not constitute a limitation on the signal transmission direction, connection sequence, or size, dimension, and shape of each part within the component or structure.

[0059] Example 1

[0060] like Figure 1 The diagram shows a flowchart of the distributed computing power intelligent scheduling method of the present invention, which includes the following steps:

[0061] S1. Construct a computing resource pool and assign an identifier to each resource node.

[0062] A comprehensive inventory of computing resources (such as CPU, GPU, memory, etc.), network resources (bandwidth, latency, etc.), and storage resources (capacity, I / O performance, etc.) in a distributed system is conducted, and the location, performance indicators, and availability status of each resource node are marked.

[0063] All inventoried resources are integrated into a global computing resource pool, forming a topology diagram of resource nodes, such as... Figure 2 As shown.

[0064] In the resource pool, each resource node is assigned a unique identifier, and its resource type, performance parameters, and current load status are recorded.

[0065] S2. Generate a comprehensive score for each resource node. For resource nodes with abnormal status, quickly locate them using identifiers and adjust the comprehensive score of the resource nodes with abnormal status using status adjustment parameters.

[0066] After assigning a unique identifier to each resource node in step S1, these identifiers play a crucial role in association and indexing in this step.

[0067] Identifiers (such as UUIDs or IP addresses) serve as primary keys, binding real-time monitoring data (CPU utilization, memory usage, etc.) of resource nodes with static information (number of CPU cores, total memory). Nodes can be quickly located and their real-time states updated using these identifiers.

[0068] Step 2.1: Monitor the status of resource nodes in real time.

[0069] On resource nodes such as servers, virtual machines, and containers, monitoring agents (such as Prometheus Exporter and Zabbix Agent) can be deployed automatically via scripts, adapting to different operating systems.

[0070] Linux systems: Install the monitoring agent via package manager (apt / yum) and configure it to start automatically on boot;

[0071] Windows system: Download the installation package and register it as a system service.

[0072] Configure the data collection task, set the frequency to periodically collect data representing the status of resource nodes, and the monitoring agent sends the data to the monitoring server via a protocol. The monitoring server uses a time-series database or distributed storage and supports high-concurrency writes.

[0073] Step 2.2: Define evaluation metrics and metric weights, and generate a comprehensive score for each resource node.

[0074] Evaluation metrics include: computing power metrics, network metrics, and storage metrics;

[0075] Computing power metrics: CPU utilization, memory usage, GPU utilization, etc.

[0076] Network metrics: network bandwidth utilization, latency, packet loss rate, etc.

[0077] Storage metrics: remaining storage capacity, I / O performance, etc.

[0078] Assign weights to different metrics. For real-time computing tasks, network latency and CPU utilization are given higher weights.

[0079] Each evaluation indicator is normalized to obtain its normalized value:

[0080]

[0081] Based on the normalized value of each evaluation indicator i and the weight of each indicator, a comprehensive score is generated for each resource node, where n is the total number of evaluation indicators.

[0082]

[0083] The overall score is used for subsequent scheduling decisions, prioritizing resource nodes with higher scores.

[0084] Table 1. Weighted scores for each evaluation indicator.

[0085]

[0086] In a preferred embodiment, the weighted scores of various evaluation indicators are dynamically adjusted based on the real-time status changes of resource nodes to ensure the real-time nature and accuracy of scheduling decisions. For resource nodes with abnormal status, their scores are lowered to avoid assigning high-priority tasks.

[0087] The adjusted overall score is calculated as follows:

[0088] Overall score = ×(1 - state adjustment parameter).

[0089] The status adjustment parameter represents the penalty for a resource node's abnormal status, with a value range of 0 ≤ status adjustment parameter < 1. The more severe the abnormality (such as hardware failure or network interruption), the higher the coefficient (e.g., 0.3 for severe abnormalities and 0.1 for minor abnormalities). For example, if the original overall score is 80 and the status adjustment parameter is 0.2, the adjusted score will be 80 × 0.8 = 64.

[0090] The status adjustment parameters can be dynamically adjusted according to the node status (e.g., updated every 10 seconds) to ensure that the scheduling decision is synchronized with the current status; different penalty coefficients can be set for different anomaly types (e.g., the penalty coefficient for hardware failure is higher than that for network fluctuation); by using multiplication instead of subtraction, the problem of extreme values ​​caused by a sudden drop in score is avoided (e.g., when the overall score is 5, subtraction may result in a negative score).

[0091] S3. Receive the task requirements submitted by the user, extract the task features, estimate the size of the output data after the task is completed, and integrate the task features, task requirements and estimated size of the output data into a task vector.

[0092] Step 3.1: Receive user task requirements, extract task features, analyze task input data, and calculate the theoretical time expected to execute the task.

[0093] It receives computing task requests submitted by users, including task type (such as data processing, machine learning training, real-time analysis, etc.), expected performance requirements (such as computing time, latency limits), and resource requirements (such as CPU / GPU core count, memory size, etc.); it calculates the total size of the input data based on the file metadata or the number of bytes in the data stream, identifies the format of the input data (such as text, image, video, binary, etc.), and estimates the processing complexity based on the format characteristics; it determines whether the data needs to be loaded from remote storage and the network overhead of loading.

[0094] The algorithm used in the analysis task is polynomial or exponential in complexity. The computational cost increases non-linearly with factors such as model parameters, number of layers, and batch size. The computational cost is estimated based on the input data size, and the execution time of the task is predicted by combining the algorithm complexity and the performance of the target resources.

[0095] Specifically,

[0096] (1): The total computational load of the task is FLOPs 总

[0097] FLOP 总 =FLOP 单 ×(Total data volume / Batch Size)×Epoch;

[0098] FLOP 单 The computational load per batch is 100; Batch Size is the number of samples processed per batch. For example, if there are 1000 samples and the Batch Size is 100, then 10 batches need to be processed.

[0099] Epoch refers to the number of training rounds, i.e., the number of times the data is used for training.

[0100] (2): Calculate the effective computing power that the resources can actually provide by using hardware parameters and actual usage efficiency. Effective computing power: Effective computing power = single-point computing power × number of points × utilization rate;

[0101] Among them, single-point computing power is the theoretical peak computing power of a single computing hardware (such as a GPU) (unit: FLOP / s), the number of points is the number of hardware for parallel computing, and the utilization rate is the theoretical computing power used for correction in actual computing.

[0102] (3): Divide the total computational load by the effective computing power per unit time to obtain the theoretical time required to complete the task, i.e., the theoretical time.

[0103] Theoretical time = FLOP 总 Effective computing power.

[0104] Step 3.2: Analyze the dependencies between task input data and between task requirements, and estimate the size of the output data after the task is completed.

[0105] The estimated size Y of the output data after the task is completed is:

[0106]

[0107] The theoretically expected execution time of N task requirements determines the task chain S. n The tasks are ordered in descending order of their expected execution time, forming task chain S. n The order in which subsequent tasks are passed is based on the output of the preceding task; the nth data coefficient in the input data is L. n The formula is essentially a mathematical expression of the process of "input data being processed step by step in a demand-driven task chain". The final output data size is the result of the combined effect of input dependence and demand dependence.

[0108] As the starting point of the task chain, it is the original input of the entire linear dependency chain. For example, in the task chain S1→S2→S3, the input of the subsequent task S2 depends on the output of S1, and the input of S3 depends on the output of S2, forming a chain dependency of task propagation.

[0109] The data coefficient L for each data point n The transformation rules reflect the input data. For example, task S1 needs to compress the data coefficient L1, so L1 = 0.5. Task S2, based on the output of task S1, needs to expand the data coefficient L2, so L2 = 2. Task S3 needs to filter the output of task S2, so L3 = 0.3. The product of these coefficients, L1×S1 + L2×S2 + L3×S3, is the cumulative effect of the input data being processed layer by layer in the chain dependency, ultimately determining the size of the output data.

[0110] Step 3.3: Integrate the task characteristics, task requirements, and estimated output data size into a multi-dimensional vector to obtain the task vector.

[0111] r J This represents the J-th resource requirement dimension vector in a set of B resource requirement dimension vectors, including: number of CPU cores, memory size, storage capacity, network bandwidth, etc. For example, an 8-core CPU (total cores 32, r...) cpu =0.25), 16GB of RAM (total RAM 64GB, r memory =0.25), 50GB storage (total storage 1TB, T storage =0.05), requires GPU (r gpu =1)

[0112] f J : Represents the J-th task feature dimension vector among M task feature dimension vectors, including: maximum latency, minimum throughput, expected completion time, etc.

[0113] G is the standardized value of the estimated output data size reference value;

[0114] G = Estimated output data size / Maximum output baseline value.

[0115] Specifically, suppose the task vector t consists of the following dimensions: t = [f1, f2, ..., f J ,…f M ,r1,r2,…,r J ,…r B ,G].

[0116] S4. Sort the task vector set in descending order, perform preliminary screening of the task vector set based on task requirements, and select the optimal resource node for the task characteristics of the screened task vectors through iterative matching of the scheduling model.

[0117] Let T = {t1, t2, ... t} k …t Q Let} represent the set consisting of Q task vectors, where t k This represents the k-th task vector. Each task vector t... k It has the following attributes:

[0118] G k Estimate the standardized value of the output data size for the k-th task vector;

[0119] r kJ : Represents the resource requirement vector of the Jth dimension of the kth task vector;

[0120] f kJ: Represents the task feature vector of the Jth dimension of the k-th task vector;

[0121] The task vector set T is standardized to the value G based on the estimated output data size. k Sort the data in descending order to obtain a new task vector sequence T' = {t'1, t'2, ..., t'}. k …t' Q}, where G1≥G2…≥G k ≥…≥G Q .

[0122] Initialize the set of scheduled tasks Remaining resource demand vector R J =[R1,R2,...,R A A represents the total amount of resources initially available.

[0123] Iterate through each task t' in the sorted task sequence T' k Check if r is satisfied. kJ ≤R J , i.e., task t' k Resource requirements in each dimension do not exceed the remaining resources.

[0124] If for all r kJ All have r kJ ≤R J Then the task t' k Add to the set of scheduled tasks S, S = S∪{t' k}, and simultaneously update the remaining resource vector, then execute R' J =R J -R kJ ,Deduct task t' k Resource R used kJ .

[0125] If the check does not meet the condition r kJ ≤R J Skip this task and proceed to check the next one.

[0126] The scheduling model is optimized using an optimization function, h, which is:

[0127]

[0128] Among them, T ex MaxU represents the theoretical execution time of the task; MaxU represents the maximum utilization rate of the resource nodes; MaxU T α is the preset utilization threshold; Co is the task feature; α, β, γ are weighting coefficients.

[0129] Input the optimization function h into the scheduling model:

[0130] Z(s,a)←(1-δ)Z(s,a)+δ[h+μmax a′ Z(s′,a′)]

[0131] Where: Z(s,a) is the Z value of scheduling operation a in state s; δ is the learning rate, ranging from (0,1]; μ is the learning factor, ranging from [0,1]; s' is the new state transitioned to after scheduling operation a; max a′ Z(s′,a′) is the maximum scheduling value for all operations that take scheduling operation a′ under the new state s′.

[0132] State s consists of the set of scheduled tasks S and the vector of remaining resources R. J Together they form, s=(S,R) J ).

[0133] The scheduling operation 'a' defines the decision-making behavior in the scheduling process, specifically: selecting a task vector from the unscheduled tasks and adding it to the set of scheduled tasks 'S'. This scheduling operation directly changes the 'S' part of the state 's', driving the task scheduling process forward.

[0134] This formula updates the Z value iteratively, enabling the system to gradually learn the optimal strategy during task scheduling, balancing immediate gains with long-term planning.

[0135] Through repeated interactive training, Z(s,a) gradually learns the comprehensive value that includes objectives such as minimizing task completion time and reducing resource waste.

[0136] During the resource node matching phase, for the current state s, calculate Z(s,a) for all possible scheduling operations a (i.e., all resource nodes). Select the scheduling operation a with the largest Z(s,a) value; its corresponding resource node is the optimal resource node that satisfies the task scheduling objective after combining with the optimization function R.

[0137] After multiple iterations, calculate Z(s,a) for all scheduling operations, select the scheduling operation with the largest Z(s,a), and the associated resource node is the optimal resource node that matches the task scheduling target.

[0138] Through this process, the optimization function h is directly integrated into the scheduling model, enabling the system to learn the optimal scheduling strategy that balances task completion time, resource utilization, and cost savings during iteration.

[0139] In a specific embodiment, the operation steps are as follows:

[0140] To distribute a batch of computing tasks across three resource nodes (A, B, C), the degree of task splitting (high / medium / low) and the load balancing threshold (e.g., 70%, 80%) need to be dynamically adjusted.

[0141] Scheduling objective: Minimize task completion time and reduce resource waste.

[0142] Define the state, scheduling operation, and initial parameters:

[0143] Status s: The current average utilization rate of resource nodes is 40%, and the backlog of tasks is 100.

[0144] Scheduling operation a: Select the task splitting level (high / medium / low) and load threshold (e.g., 70%, 80%).

[0145] Perform scheduling operations and compute the optimization function h:

[0146] Perform scheduling operation: Select equal splitting, threshold 70%; after execution, calculate h.

[0147] Substitute h into the scheduling model, update the Z value, and after multiple iterations, select the node corresponding to the scheduling operation with the largest Z value, which is the optimal node for matching the task scheduling target.

[0148] Example 2

[0149] This invention also proposes a distributed computing power intelligent scheduling system, such as... Figure 2 As shown, the distributed computing power intelligent scheduling system includes: a resource management module, a resource evaluation module, a task parsing and feature extraction module, a task vectorization module, and a scheduling algorithm module.

[0150] Resource Management Module: Constructs a distributed computing resource pool and assigns a unique identifier, such as IP+UUID, to each node.

[0151] Record node hardware configuration (CPU / GPU / memory / storage), network status, and geographical location; monitor node load, available resources, and health status in real time. The resource node metadata database in the resource management module is used to store node identifiers, configuration parameters, and real-time status.

[0152] Resource assessment module: Generates a comprehensive score for each resource node. For resource nodes with abnormal status, it can quickly locate them using an identifier and adjust the comprehensive score of the resource nodes with abnormal status using status adjustment parameters.

[0153] Define evaluation metrics and their weights to generate a comprehensive score for each resource node. Evaluation metrics include: computing power metrics, network metrics, and storage metrics.

[0154] Computing power metrics: CPU utilization, memory usage, GPU utilization, etc.

[0155] Network metrics: network bandwidth utilization, latency, packet loss rate, etc.

[0156] Storage metrics: remaining storage capacity, I / O performance, etc.

[0157] Weights are assigned to different metrics based on scheduling objectives (such as performance priority or cost priority). For real-time computing tasks, network latency and CPU utilization are given higher weights.

[0158] Each evaluation indicator is normalized to obtain its normalized value:

[0159]

[0160] Based on the normalized value of each evaluation indicator i and the weight of each indicator, a comprehensive score is generated for each resource node, where n is the total number of evaluation indicators.

[0161]

[0162] The overall score is used for subsequent scheduling decisions, prioritizing resource nodes with higher scores.

[0163] Task parsing and feature extraction module: Receives task requirements submitted by users, calculates the theoretical time expected to be executed, and estimates the size of the output data after the task is completed.

[0164] Receive user task requirements, analyze task input data, extract task features, calculate the theoretical time expected to be executed; analyze the dependencies between task input data, and based on the theoretical time expected to be executed, estimate the size of the output data after the task is completed.

[0165] Task vectorization module: integrates task features, task requirements, and the estimated size of the output data into a multi-dimensional vector to obtain the task vector.

[0166] The scheduling algorithm module sorts the task vector set in descending order, performs preliminary screening of the task vector set based on task requirements, and selects the optimal resource node through iterative matching of the task characteristics of the screened task vectors.

[0167] An optimization function is used to optimize the scheduling model. By iteratively updating the Z value, the system gradually learns the optimal strategy in task scheduling, balancing immediate benefits and long-term planning. Through multiple interactive training sessions, Z(s,a) for all scheduling operations is calculated. The scheduling operation with the largest Z(s,a) is selected, and its associated resource node is the optimal resource node matching the task scheduling target.

[0168] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0169] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0170] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0171] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. The databases involved in the embodiments provided in this application can include at least one of relational and non-relational databases. Non-relational databases can include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application can be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0172] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0173] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A distributed computing power intelligent scheduling method, characterized in that, Includes the following steps: Step S1: Construct a computing resource pool and assign an identifier to each resource node; Step S2: Generate a comprehensive score for each resource node. For resource nodes with abnormal status, quickly locate them using identifiers and adjust the comprehensive score of the resource nodes with abnormal status using status adjustment parameters. Step S3: Receive the task requirements submitted by the user, extract the task features, estimate the size of the output data after the task is completed, and integrate the task features, task requirements and estimated size of the output data into a task vector. Step S4: Sort the task vector set formed by multiple task vectors in descending order according to the estimated output data size value in each task vector, perform preliminary screening of the task vector set, and select the optimal resource node by iterative matching of the task characteristics of the screened task vectors through the scheduling model. In step S4, T = {t1, t2, ... t} k …t Q Let} represent the set consisting of Q task vectors, where t k Represents the k-th task vector, each task vector t k It has the following attributes: G k Estimate the standardized output data size r for the k-th task vector. kJ f represents the resource requirement vector of the J-th dimension of the k-th task vector. kJ This represents the task feature vector of the Jth dimension of the kth task vector; For the task vector set T, the standardized value G is based on the estimated output data size. k Sort the data in descending order to obtain a new sequence of task vectors. Its G1≥G2…≥G k ≥…≥G Q ; Initialize the set of scheduled tasks S=Æ, and the remaining resource vector R J =[R1,R2,..., R A ]; Traverse each task vector t' in the sorted task vector sequence T' k Check if r is satisfied. kJ ≤R J If for all r kJ Both have r kJ ≤R J Then the task vector t' k Add to the set of scheduled tasks S, S = S∪{t' k }, and simultaneously update the remaining resource vector, then execute R' J =R J -R kJ , deducting task vector t' k Resource R used kJ If the check does not meet the condition r kJ ≤R J Skip this task and proceed to check the next one; The scheduling model is optimized using an optimization function, h, which is: Among them, T ex MaxU represents the theoretical execution time of the task; MaxU represents the maximum utilization rate of the resource nodes; MaxU T The preset utilization threshold is represented by Co; Co represents the task characteristic. , These are the weighting coefficients; Input the optimization function h into the following scheduling model: Where Z(s,a) is a function that takes scheduling operation a in state s, and its output value is the value Z. It's the learning rate. It is the learning factor, and s' is the new state transitioned to after the scheduling operation a is performed; It is the maximum scheduling value for all scheduled operations a' taken under the new state s'; State s consists of the set of scheduled tasks S and the vector of remaining resources R. J Together they form s=(S,R) J ); After multiple iterations, calculate Z(s,a) for all scheduling operations a, select the scheduling operation a with the largest Z(s,a) output value, and the associated resource node is the optimal resource node that matches the task scheduling target.

2. The distributed computing power intelligent scheduling method according to claim 1, characterized in that, Step S3 includes the following steps: Step 3.1: Receive user task requirements, extract task features, analyze task input data, and calculate the theoretical time expected to execute the task. Step 3.2: Analyze the dependencies between the input data of the task, and based on the theoretical time expected to be executed by the task, estimate the size of the output data after the task is completed. Step 3.3: Integrate the task characteristics, task requirements, and estimated output data size into a multi-dimensional vector.

3. The distributed computing power intelligent scheduling method according to claim 2, characterized in that, In step 3.2, the total computational load of the task is calculated in FLOPs. 总 : FLOP 总 =FLOP 单 ×(Total data volume / Batch Size)×Epoch; Among them, FLOP 单 The computational cost per batch; Batch Size is the number of samples computed per batch; Epoch is the number of training epochs. The effective computing power that computing resources can provide: Effective computing power = single-point computing power × number of points × utilization rate; The theoretical time to complete the task is: Theoretical time = FLOP 总 Effective computing power.

4. The distributed computing power intelligent scheduling method according to claim 3, characterized in that, In step 3.3, the task vector t is assumed to consist of the following dimensions: t=[f1,f2,…,f J ,…f M ,r1,r2,…,r J ,…r B ,G]; G is the standardized value of the reference value for the estimated output data size of the task vector; r J f represents the J-th resource demand dimension vector among B resource demand dimension vectors; J Let J represent the J-th task feature dimension vector among M task feature dimension vectors.

5. A distributed computing power intelligent scheduling system, used to implement the distributed computing power intelligent scheduling method according to any one of claims 1-4, characterized in that, It includes: a resource management module, a resource evaluation module, a task parsing and feature extraction module, a task vectorization module, and a scheduling algorithm module; The resource management module is used to build a distributed computing power resource pool and assign a unique identifier to each node. The resource assessment module is used to generate a comprehensive score for each resource node. For resource nodes with abnormal status, the module can quickly locate them using an identifier and adjust the comprehensive score of the resource nodes with abnormal status using status adjustment parameters. The task parsing and feature extraction module is used to receive task requirements submitted by users, calculate the theoretical time expected to be executed, and estimate the size of the output data after the task is completed. The task vectorization module is used to integrate task features, task requirements, and the estimated size of the output data into a multi-dimensional vector to obtain the task vector. The scheduling algorithm module is used to sort the task vector set in descending order, perform preliminary screening of the task vector set based on task requirements, and select the optimal resource node for the task characteristics of the screened task vectors through iterative matching of the scheduling model.

Images