Distributed computing power intelligent scheduling system and method

Abstract

The invention provides a distributed computing power intelligent scheduling system and method, and relates to the technical field of computing power scheduling. A computing power resource pool is constructed, and an identifier is allocated to each resource node; generating a comprehensive score value for each resource node, quickly positioning the resource node in an abnormal state through an identifier, and adjusting the comprehensive score of the resource node in the abnormal state through a state adjustment parameter; receiving a task demand submitted by a user, extracting task features, and integrating the task features, the task demand and the estimated size scale value of the output data into a task vector; and performing descending sort on the task vector set, performing preliminary screening on the task vector set based on task requirements, and selecting an optimal resource node for task features of the screened task vectors through iterative matching of a scheduling model.

Description

Technical Field

[0001] The present invention proposes a distributed computing power intelligent scheduling system and method, which relate to the technical field of computing power scheduling. Background Art

[0002] With the rapid development of fields such as artificial intelligence, the metaverse, and high-performance computing, the scale of global intelligent computing power is growing rapidly. The proportion of unstructured data (such as text and images) has increased dramatically, making traditional computing architectures unable to meet real-time and elastic requirements. Distributed scheduling is needed to optimize resource utilization. Heterogeneous computing power (CPU, GPU, FPGA, ASIC, etc.) has become mainstream, with significant differences in type, performance, and network conditions. While GPUs excel at parallel computing and FPGAs are suitable for customized acceleration, existing systems lack a unified solution for managing heterogeneous resources, making resource pooling difficult. Intelligent scheduling is urgently needed to achieve computing-network integration.

[0003] Traditional centralized scheduling systems find it difficult to perceive the dynamic status of heterogeneous computing power in real time, resulting in inaccurate matching of tasks and resources. Scheduling based on routing tables or fixed rules cannot adapt to sudden load changes and has poor resource elasticity and scalability. The network status of existing scheduling systems and computing resources are not fully coordinated, resulting in ignoring network bottlenecks when allocating tasks, exacerbating congestion and packet loss. The centralized controller has limited processing power and cannot support the rapid expansion of large-scale distributed nodes. It can easily become a performance bottleneck, with insufficient response to node failures or timeouts, low task rescheduling efficiency, and poor system fault tolerance. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention proposes a distributed computing power intelligent scheduling method, comprising the following steps:

[0005] Step S1: Build a computing resource pool and assign an identifier to each resource node;

[0006] Step S2: Generate a comprehensive score for each resource node, quickly locate resource nodes with abnormal status using identifiers, and adjust the comprehensive score of the resource nodes with abnormal status using status adjustment parameters;

[0007] Step S3: Receive the task requirements submitted by the user, extract the task features, estimate the size of the output data after the task is completed, and integrate the task features, task requirements and the estimated size of the output data into a task vector;

[0008] Step S4: Arrange the task vector set in descending order, preliminarily screen the task vector set based on task requirements, and select the optimal resource node through iterative matching of the scheduling model for the task features of the screened task vectors.

[0009] In a preferred embodiment, step S3 includes the following steps:

[0010] Step 3.1: Receive user task requirements, extract task features, analyze task input data, and calculate the expected theoretical execution time of the task;

[0011] Step 3.2: Analyze the dependencies between task input data and, based on the expected theoretical execution time of the task, estimate the size of the output data after the task is completed.

[0012] Step 3.3: Integrate the task characteristics, task requirements, and estimated output data size into a multidimensional vector to obtain a task vector.

[0013] In a preferred embodiment, in step S3.2, the size value Y of the output data after the task is estimated to be:

[0014]

[0015] N task requirements are sorted from largest to smallest according to the expected theoretical execution time to form a task chain S n The order of the input data; the nth data coefficient is L n ;;When task S n The data coefficient L n To compress, let L n =0.5, Task S n+1 Based on the task S n The output of the data coefficient L n+1 To perform data expansion, let L n+1 =2, Task S n+2 Need to re-task S n+1 Filter the output of n+2 =0.3.

[0016] In a preferred embodiment, in step S4, T={t1, t2, ... t k …t Q} to represent the set of all task vectors, where t k Represents the kth task vector, each task vector t k Has the following properties:

[0017] G k Estimate the normalized value of the output data size for the kth task vector;

[0018] r kJ represents the resource requirement vector of the Jth dimension of the kth task vector;

[0019] f kJ Represents the task feature vector of the Jth dimension of the kth task vector;

[0020] The value G after normalizing the task vector set T according to the estimated output data size k Arrange in descending order to obtain a new task vector sequence T'={t'1,t'2,…t' k …t' Q}, where G1≥G2…≥G k ≥…≥G Q ;

[0021] Initialize the scheduled task set Remaining resource demand vector R J =[R1,R2,...,R A ], A represents the total amount of resources initially available; traverse each task t' in the sorted task sequence T' k , check whether r is satisfied kJ ≤R J ; If for all r kJ Both have r kJ ≤R J , then the task t' k Add the scheduled task set S, S=S∪{t' k}, and update the remaining resource vectors at the same time, executing R' J =R J -R kJ , deduct task t' k Occupied resources R kJ If the check does not meet the condition r kJ ≤R J , skip this task and continue checking the next task.

[0022] In a preferred embodiment, in step S4, an optimization function is used to optimize the scheduling model, and the optimization function h is:

[0023]

[0024] Among them, T ex is the theoretical time for task execution; MaxU is the maximum utilization of resource nodes; MaxU T is the preset utilization threshold; Co is the task feature; α, β, γ are weight coefficients;

[0025] Input the optimization function h into the following scheduling model:

[0026] Z(s,a)←(1-δ)Z(s,a)+δ[h+μmax a′ Z(s′,a′)]

[0027] Where: Z(s,a) is the function that takes scheduling action a in state s, and the output value is Z value; δ is the learning rate, μ is the learning factor, s' is the new state transferred to after taking scheduling action a; max a′ Z(s′,a′) is the maximum scheduling value of all scheduling actions a′ taken in the new state s′;

[0028] The state s is composed of the set of scheduled tasks S and the remaining resource vector R J Together, s=(S,R J );

[0029] After multiple iterations, Z(s,a) corresponding to all scheduling operations a is calculated, and the scheduling operation a with the largest Z(s,a) output value is selected. Its associated resource node is the optimal resource node that matches the task scheduling target.

[0030] In a preferred embodiment, in step S3.2, the total amount of FLOPs of the calculation task is calculated. 总 :

[0031] FLOP 总 =FLOP 单 ×(total data volume / Batch Size)×Epoch;

[0032] Among them, FLOP 单 is the amount of computation for a single batch; Batch Size is the number of samples calculated for each batch; Epoch is the number of training rounds;

[0033] The effective computing power that computing resources can provide:

[0034] Effective computing power = single-point computing power × number of points × utilization rate;

[0035] Get the theoretical time to complete the task:

[0036] Theoretical time = FLOP 总 / Effective computing power.

[0037] In a preferred embodiment, in step S3.3, the task vector t is assumed to be composed of the following dimensions:

[0038] t=[f1,f2,…,f J ,…f M ,r1,r2,…,r J ,…r B ,G];

[0039] G is the normalized value of the estimated output data size reference value of the task vector; r J represents the Jth resource demand dimension vector among B resource demand dimension vectors; f JRepresents the Jth task feature dimension vector among the M task feature dimension vectors.

[0040] The present invention also proposes a distributed computing power intelligent scheduling system for implementing the above-mentioned distributed computing power intelligent scheduling method, which is characterized by including: a resource management module, a resource evaluation module, a task parsing and feature extraction module, a task vectorization module and a scheduling algorithm module;

[0041] The resource management module is used to build a distributed computing resource pool and assign a unique identifier to each node;

[0042] The resource evaluation module is used to generate a comprehensive score value for each resource node, quickly locate resource nodes with abnormal status through identifiers, and adjust the comprehensive score of resource nodes with abnormal status through status adjustment parameters;

[0043] The task analysis and feature extraction module is used to receive the task requirements submitted by the user, calculate the theoretical time expected to execute the task, and estimate the size of the output data after the task is completed;

[0044] The task vectorization module is used to integrate task characteristics, task requirements and estimated output data size values into a multi-dimensional vector to obtain a task vector;

[0045] The scheduling algorithm module is used to sort the task vector set in descending order, preliminarily screen the task vector set based on task requirements, and select the optimal resource node for the task features of the screened task vectors through iterative matching of the scheduling model.

[0046] Compared with the prior art, the present invention has the following beneficial technical effects:

[0047] By building a computing resource pool, centralized management of physical / virtual resources is achieved. Node performance (such as computing power, storage, network, etc.) is dynamically evaluated based on comprehensive scores, and high-scoring nodes are called first to maximize resource utilization.

[0048] The comprehensive scoring and priority sorting mechanism can perceive the node load status in real time, avoid concentrated stress testing of high-load nodes, achieve global load balancing, and improve system stability.

[0049] The task vector quantifies task characteristics (such as computational complexity, real-time requirements, data volume, etc.) into structured parameters, combines them with node priorities, and uses scheduling algorithms to achieve precise matching of tasks and resources, reducing ineffective resource allocation.

[0050] Priority sorting and rapid positioning mechanisms shorten task waiting time, prioritize the allocation of high-quality resources to high-priority tasks, and significantly improve the execution efficiency of critical tasks.

[0051] Through the synergistic effect of resource pooling management, dynamic scoring, task feature quantification and intelligent scheduling algorithms, the present invention achieves multi-dimensional optimization in terms of resource utilization, task execution efficiency, system reliability and scalability, and is suitable for the efficient computing power scheduling needs in distributed scenarios such as cloud computing and edge computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 This is a flow chart of the distributed computing power intelligent scheduling method of the present invention;

[0054] Figure 2 A topological structure diagram of resource nodes of the present invention;

[0055] Figure 3 A flowchart of integrating task characteristics and requirements into a task vector according to the present invention;

[0056] Figure 4 This is a structural diagram of the distributed computing power intelligent scheduling system of the present invention. DETAILED DESCRIPTION

[0057] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0058] In the drawings of the specific embodiments of the present invention, in order to better and more clearly describe the working principles of the various components in the system, the connection relationship of the various parts in the device is shown, which only clearly distinguishes the relative position relationship between the various components, and does not constitute a limitation on the signal transmission direction, connection sequence and size, dimension and shape of the components or structures.

[0059] Example 1

[0060] like Figure 1 FIG. 1 is a flow chart of the distributed computing power intelligent scheduling method of the present invention, which includes the following steps:

[0061] S1. Build a computing resource pool and assign an identifier to each resource node.

[0062] Conduct a comprehensive inventory of computing resources (such as CPU, GPU, memory, etc.), network resources (bandwidth, latency, etc.), and storage resources (capacity, I / O performance, etc.) in the distributed system, and mark the location, performance indicators, and availability status of each resource node.

[0063] Integrate all inventoried resources into a global computing resource pool to form a topological structure diagram of resource nodes, such as Figure 2 shown.

[0064] In the resource pool, each resource node is assigned a unique identifier, and its resource type, performance parameters, and current load status are recorded.

[0065] S2. Generate a comprehensive score for each resource node, quickly locate resource nodes with abnormal status through identifiers, and adjust the comprehensive score of the resource nodes with abnormal status through status adjustment parameters.

[0066] After each resource node is assigned a unique identifier in step S1, these identifiers play a key role in association and indexing in this step.

[0067] Identifiers (such as UUID or IP address) are used as primary keys to bind the real-time monitoring data (CPU utilization, memory usage, etc.) of resource nodes with static information (number of CPU cores, total memory). Use identifiers to quickly locate nodes and update their real-time status.

[0068] Step 2.1: Monitor resource node status in real time.

[0069] On resource nodes such as servers, virtual machines, and containers, monitoring agents (such as Prometheus Exporter and Zabbix Agent) are automatically deployed through scripts, adapting deployment to different operating systems:

[0070] Linux system: Install the monitoring agent through the package manager (apt / yum) and configure it to start automatically at boot;

[0071] Windows system: Download the installation package and register it as a system service.

[0072] Configure the collection task and set the frequency to regularly collect data representing the status of resource nodes. The monitoring agent sends the data to the monitoring server through the protocol. The monitoring server uses a time series database or distributed storage to support high-concurrency writes.

[0073] Step 2.2: Define evaluation metrics and metric weights to generate a comprehensive score for each resource node.

[0074] Evaluation indicators include: computing power indicators, network indicators, and storage indicators;

[0075] Computing power indicators: CPU utilization, memory usage, GPU utilization, etc.

[0076] Network indicators: network bandwidth utilization, latency, packet loss rate, etc.

[0077] Storage indicators: remaining storage capacity, I / O performance, etc.

[0078] Assign weights to different metrics. For real-time computing tasks, network latency and CPU utilization are given higher weights.

[0079] Normalize each evaluation indicator to obtain the normalized value of the indicator:

[0080]

[0081] Based on the normalized value of each evaluation indicator i and the weight of each indicator, a comprehensive score is generated for each resource node, where n is the total number of evaluation indicators.

[0082]

[0083] The comprehensive score is used for subsequent scheduling decisions, and resource nodes with higher scores are given priority.

[0084] Table 1 Table 1 Weighted scores of various evaluation indicators

[0085]

[0086] In a preferred embodiment, the weighted scores of various evaluation indicators are dynamically adjusted based on the real-time status changes of resource nodes to ensure the real-time and accuracy of scheduling decisions. For resource nodes with abnormal status, their scores are reduced to avoid assigning high-priority tasks.

[0087] The adjusted composite score is calculated as follows:

[0088] Comprehensive score = × (1-state adjustment parameter).

[0089] The state adjustment parameter indicates the severity of the penalty imposed on the resource node score when the resource node's status is abnormal. The value range is 0 ≤ state adjustment parameter < 1. The more severe the abnormality (such as hardware failure or network interruption), the higher the coefficient (for example, 0.3 for severe abnormality and 0.1 for minor abnormality). For example, if the original comprehensive score is 80 and the state adjustment parameter is 0.2, the adjusted score is 80 × 0.8 = 64.

[0090] State adjustment parameters can be dynamically adjusted according to the node status (e.g., updated every 10 seconds) to ensure that scheduling decisions are synchronized with the current status; different penalty coefficients can be set for different types of anomalies (e.g., the penalty coefficient for hardware failure is higher than that for network fluctuations); and extreme value problems caused by a sudden drop in the score are avoided by multiplication rather than subtraction (e.g., when the comprehensive score is 5, subtraction may result in a negative score).

[0091] S3. Receive the task requirements submitted by the user, extract the task features, estimate the size of the output data after the task is completed, and integrate the task features, task requirements and the estimated size of the output data into a task vector.

[0092] Step 3.1: Receive user task requirements, extract task features, analyze task input data, and calculate the expected theoretical execution time of the task.

[0093] Receive computing task requests submitted by users, including task type (such as data processing, machine learning training, real-time analysis, etc.), expected performance requirements (such as computing time, latency limit) and resource requirements (such as number of CPU / GPU cores, memory size, etc.); calculate the total size of input data through file metadata or the number of bytes of data stream, identify the format of input data (such as text, image, video, binary, etc.), and estimate the processing complexity based on the format characteristics; determine whether data needs to be loaded from remote storage, and the network overhead of loading.

[0094] Analyze the algorithm used in the computing task, such as polynomial or exponential complexity, polynomial or exponential complexity, and the nonlinear growth of the computational amount with factors such as model parameters, number of layers, and batch size. Estimate the computational amount based on the input data scale, and estimate the execution time of the task based on the algorithm complexity and target resource performance.

[0095] Specifically,

[0096] (1): Total FLOPs of computing tasks 总

[0097] FLOP 总 =FLOP 单 ×(total data volume / Batch Size)×Epoch;

[0098] FLOP 单 is the amount of computation for a single batch; Batch Size is the number of samples calculated for each batch; for example, if there are 1000 samples and Batch Size is 100, 10 batches need to be processed.

[0099] Epoch is the number of training rounds, that is, the number of times the data is fully trained.

[0100] (2): Calculate the effective computing power that resources can actually provide through hardware parameters and actual usage efficiency: Effective computing power = single-point computing power × number of points × utilization rate;

[0101] Among them, the single-point computing power is the theoretical peak computing power of a single computing hardware (such as GPU) (unit: FLOP / s), the number of points is the number of hardware for parallel computing, and the utilization rate is the theoretical computing power used for correction in actual calculations.

[0102] (3): The total amount of computation divided by the effective computing power per unit time gives the theoretical time required to complete the task, i.e., the theoretical time.

[0103] Theoretical time = FLOP 总 / Effective computing power.

[0104] Step 3.2: Analyze the dependencies between task input data and task requirements, and estimate the size of the output data after the task is completed.

[0105] The estimated output data size Y after the task is completed is:

[0106]

[0107] The expected execution time of N tasks determines the theoretical time of the task chain S. n The order of tasks is sorted from largest to smallest according to the expected theoretical execution time to form a task chain S n The order of subsequent tasks is based on the output of the previous task; the nth data coefficient in the data coefficient of the input data is L n The essence of the formula is a mathematical expression of the process of "input data being processed step by step in a demand-driven task chain". The final output data size is the result of the combined effect of input dependency and demand dependency.

[0108] As the starting point of a task chain, it is the original input of the entire linear dependency chain. For example, in the task chain S1→S2→S3, the input of subsequent task S2 depends on the output of S1, and the input of S3 depends on the output of S2, forming a chain dependency of task transfer.

[0109] The data coefficient L for each data n This reflects the transformation rules for the input data. For example, task S1 compresses the data coefficient L1, resulting in L1 = 0.5. Task S2, based on the output of task S1, expands the data coefficient L2, resulting in L2 = 2. Task S3 further filters the output of task S2, resulting in L3 = 0.3. The product of these coefficients, L1 × S1 + L2 × S2 + L3 × S3, is the cumulative effect of the input data being processed layer by layer in the chain dependency, ultimately determining the output data size.

[0110] Step 3.3: Integrate the task characteristics, task requirements, and estimated output data size into a multidimensional vector to obtain a task vector.

[0111] r J : represents the Jth resource requirement dimension vector among B resource requirement dimension vectors, including: number of CPU cores, memory size, storage capacity, network bandwidth, etc. For example, CPU 8 cores (total number of cores 32, r cpu =0.25), memory 16GB (total memory 64GB, r memory =0.25), storage 50GB (total storage 1TB, T storage =0.05), GPU required (r gpu =1)

[0112] f J : represents the Jth task feature dimension vector among the M task feature dimension vectors, including: maximum delay, minimum throughput, expected completion time, etc.

[0113] G is the normalized value of the estimated output data size reference value;

[0114] G = estimated output data size / maximum output reference value.

[0115] Specifically, let the task vector t be composed of the following dimensions: t=[f1,f2,…,f J ,…f M ,r1,r2,…,r J ,…r B ,G].

[0116] S4. Arrange the task vector set in descending order, preliminarily screen the task vector set based on task requirements, and select the optimal resource node for the task features of the screened task vectors through iterative matching of the scheduling model.

[0117] Use T={t1,t2,…t k …t Q} to represent the set of Q task vectors, where t k Represents the kth task vector. Each task vector t k Has the following properties:

[0118] G k Estimate the normalized value of the output data size for the kth task vector;

[0119] r kJ : represents the resource requirement vector of the Jth dimension of the kth task vector;

[0120] f kJ: represents the task feature vector of the Jth dimension of the kth task vector;

[0121] The value G after normalizing the task vector set T according to the estimated output data size k Arrange in descending order to obtain a new task vector sequence T'={t'1,t'2,…t' k …t' Q}, where G1≥G2…≥G k ≥…≥G Q .

[0122] Initialize the scheduled task set Remaining resource demand vector R J =[R1,R2,...,R A ], A represents the total amount of resources initially available.

[0123] Traverse each task t' in the sorted task sequence T' k , check whether r is satisfied kJ ≤R J , that is, task t' k The resource requirements in each dimension do not exceed the remaining resources.

[0124] If for all r kJ Both have r kJ ≤R J , then the task t' k Add the scheduled task set S, S=S∪{t' k}, and update the remaining resource vectors at the same time, executing R' J =R J -R kJ , deduct task t' k Occupied resources R kJ .

[0125] If the check does not meet the condition r kJ ≤R J , skip this task and continue checking the next task.

[0126] The optimization function is used to optimize the scheduling model. The optimization function h is:

[0127]

[0128] Among them, T ex is the theoretical time for task execution; MaxU is the maximum utilization of resource nodes; MaxU T is the preset utilization threshold; Co is the task feature; α, β, γ are weight coefficients.

[0129] Input the optimization function h into the scheduling model:

[0130] Z(s,a)←(1-δ)Z(s,a)+δ[h+μmax a′ Z(s′,a′)]

[0131] Where: Z(s,a) is the Z value of taking scheduling action a in state s; δ is the learning rate, which ranges from (0,1]; μ is the learning factor, which ranges from [0,1]; s' is the new state transferred to after taking scheduling action a; max a′ Z(s′,a′) is the maximum scheduling value of all scheduling actions a′ taken in the new state s′.

[0132] The state s is composed of the set of scheduled tasks S and the remaining resource vector R J Together, s=(S,R J ).

[0133] The scheduling operation a defines the decision-making behavior in the scheduling process, specifically: selecting a task vector from the unscheduled tasks and adding it to the scheduled task set S. This scheduling operation directly changes the S part of the state s, promoting the task scheduling process.

[0134] This formula iteratively updates the Z value, allowing the system to gradually learn the optimal strategy in task scheduling, balancing immediate benefits and long-term planning.

[0135] Through multiple interactive trainings, Z(s,a) gradually learns the comprehensive value of goals such as minimizing task completion time and reducing resource waste.

[0136] In the resource node matching phase, for the current state s, the corresponding Z(s,a) values for all optional scheduling operations a (i.e., all resource nodes) are calculated. The scheduling operation a with the largest Z(s,a) value is selected, and the corresponding resource node is the optimal resource node that meets the task scheduling goal after combining the optimization function R.

[0137] After multiple iterations, Z(s,a) corresponding to all scheduling operations is calculated, and the scheduling operation with the largest Z(s,a) is selected. Its associated resource node is the optimal resource node that matches the task scheduling target.

[0138] Through this process, the optimization function h is directly integrated into the scheduling model, so that the system can learn the optimal scheduling strategy that takes into account task completion time, resource utilization, and cost savings during iteration.

[0139] In a specific embodiment, the operation steps are as follows:

[0140] To allocate a batch of computing tasks to three resource nodes (A, B, and C), it is necessary to dynamically adjust the task splitting degree (high / medium / low) and the load balancing threshold (such as 70% and 80%).

[0141] Scheduling goal: minimize task completion time and reduce resource waste.

[0142] Define states, schedule operations, and initial parameters:

[0143] Status s: The current average utilization of resource nodes is 40%, and the task backlog is 100.

[0144] Scheduling operation a: Select the task splitting degree (high / medium / low) and load threshold (such as 70%, 80%).

[0145] Perform the scheduling operation and calculate the optimization function h:

[0146] Execute the scheduling operation: select equal split and threshold 70%; after execution, calculate h.

[0147] Substitute h into the scheduling model, update the Z value, and after multiple iterations, select the node corresponding to the scheduling operation with the largest Z, which is the optimal node that matches the task scheduling target.

[0148] Example 2

[0149] The present invention also proposes a distributed computing power intelligent scheduling system, such as Figure 2 As shown in the figure, the distributed computing power intelligent scheduling system includes: resource management module, resource evaluation module, task parsing and feature extraction module, task vectorization module and scheduling algorithm module.

[0150] Resource management module: Build a distributed computing resource pool and assign a unique identifier to each node, such as IP+UUID.

[0151] Record node hardware configuration (CPU / GPU / memory / storage), network status and geographic location; monitor node load, available resources and health status in real time. The resource node metadata database in the resource management module is used to store node identifiers, configuration parameters and real-time status.

[0152] Resource evaluation module: Generates a comprehensive score for each resource node, quickly locates resource nodes with abnormal status through identifiers, and adjusts the comprehensive score of resource nodes with abnormal status through status adjustment parameters.

[0153] Define evaluation indicators and indicator weights to generate a comprehensive score for each resource node. Evaluation indicators include: computing power indicators, network indicators, and storage indicators;

[0154] Computing power indicators: CPU utilization, memory usage, GPU utilization, etc.

[0155] Network indicators: network bandwidth utilization, latency, packet loss rate, etc.

[0156] Storage indicators: remaining storage capacity, I / O performance, etc.

[0157] Weights are assigned to different metrics based on scheduling goals (such as performance priority or cost priority). For real-time computing tasks, network latency and CPU utilization are given higher weights.

[0158] Normalize each evaluation indicator to obtain the normalized value of the indicator:

[0159]

[0160] Based on the normalized value of each evaluation indicator i and the weight of each indicator, a comprehensive score is generated for each resource node, where n is the total number of evaluation indicators.

[0161]

[0162] The comprehensive score is used for subsequent scheduling decisions, and resource nodes with higher scores are given priority.

[0163] Task parsing and feature extraction module: receives task requirements submitted by users, calculates the expected theoretical execution time of the task, and estimates the size of the output data after the task is completed.

[0164] Receive user task requirements, analyze task input data, extract task features, and calculate the expected theoretical time for task execution; analyze the dependencies between task input data, and based on the expected theoretical time for task execution, estimate the size of the output data after the task is completed.

[0165] Task vectorization module: Integrates task characteristics, task requirements, and estimated output data size into a multidimensional vector to obtain a task vector.

[0166] Scheduling algorithm module: sorts the task vector set in descending order, preliminarily screens the task vector set based on task requirements, and selects the optimal resource node through iterative matching of the scheduling model based on the task characteristics of the screened task vectors.

[0167] The scheduling model is optimized using an optimization function. By iteratively updating the Z value, the system gradually learns the optimal strategy for task scheduling, balancing immediate benefits with long-term planning. Through multiple interactive training sessions, the Z(s, a) corresponding to all scheduling operations is calculated. The scheduling operation with the largest Z(s, a) is selected, and its associated resource node is the optimal resource node that matches the task scheduling goal.

[0168] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0169] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0170] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of each of the above-described method embodiments.

[0171] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. The database involved in the embodiments provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited to this. The processor involved in the embodiments provided in this application may be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, etc., but is not limited to this.

[0172] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0173] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A distributed computing power intelligent scheduling method, characterized in that: The steps include: Step S1: Build a computing resource pool and assign an identifier to each resource node; Step S2: Generate a comprehensive score for each resource node, quickly locate resource nodes with abnormal status using identifiers, and adjust the comprehensive score of the resource nodes with abnormal status using status adjustment parameters; Step S3: Receive the task requirements submitted by the user, extract the task features, estimate the size of the output data after the task is completed, and integrate the task features, task requirements and the estimated size of the output data into a task vector; Step S4: Arrange the task vector set formed by the multiple task vectors in descending order, preliminarily screen the task vector set based on task requirements, and select the optimal resource node for the task features of the screened task vectors through iterative matching of the scheduling model.

2. The distributed computing power intelligent scheduling method according to claim 1 is characterized in that: The step S3 comprises the following steps: Step 3.1: Receive user task requirements, extract task features, analyze task input data, and calculate the expected theoretical execution time of the task; Step 3.2: Analyze the dependencies between the task's input data and, based on the expected theoretical execution time of the task, estimate the size of the output data after the task is completed. Step 3.3: Integrate the task characteristics, task requirements, and estimated output data size into a multidimensional vector.

3. The distributed computing power intelligent scheduling method according to claim 2, characterized in that: In step S3.2, it is estimated that the size value Y of the output data after the task is completed is: N task requirements are sorted from largest to smallest according to the expected theoretical execution time to form the order of the task chain, S n is the nth task in the task chain; the number of input data corresponding to the nth task is L n ; When task S n The data coefficient L n When compression is performed, let L n =0.5, Task S n+1 Based on the task S n The output of the data coefficient L n+1 When data expansion is performed, let L n+1 =2, Task S n+2 Need to re-task S n+1 When filtering the output of n+2 =0.

3.

4. The distributed computing power intelligent scheduling method according to claim 1, characterized in that: In step S4, T={t1, t2, ... t k …t Q } to represent the set of Q task vectors, where t k Represents the kth task vector, each task vector t k Has the following properties: G k Estimate the normalized value of the output data size for the kth task vector; r kJ represents the resource requirement vector of the Jth dimension of the kth task vector; f kJ Represents the task feature vector of the Jth dimension of the kth task vector; The value G after normalizing the task vector set T according to the estimated output data size k Arrange in descending order to obtain a new task vector sequence T'={t'1,t'2,…t' k …t' Q }, where G1≥G2…≥G k ≥…≥G Q ; Initialize the scheduled task set Remaining resource demand vector R J =[R1,R2,...,R A ], A represents the total amount of resources initially available; traverse each task vector t' in the sorted task sequence T' k , check whether r is satisfied kJ ≤R J ; If for all r kJ , both have r kJ ≤R J , then the task vector t' k Add the scheduled task set S, S=S∪{t' k }, and update the remaining resource vectors at the same time, executing R' J =R J -R kJ , deduct the task vector t' k Occupied resources R kJ If the check does not meet the condition r kJ ≤R J , skip this task and continue checking the next task.

5. The distributed computing power intelligent scheduling method according to claim 4 is characterized in that: In step S4, the scheduling model is optimized using an optimization function, and the optimization function h is: Among them, T ex is the theoretical time for task execution; MaxU is the maximum utilization of resource nodes; MaxU T is the preset utilization threshold; Co is the task feature; α, β, γ are weight coefficients; Input the optimization function h into the following scheduling model: Z(s,a)←(1-δ)Z(s,a)+δ[h+μmax a′ Z(s′,a′)] Where: Z(s,a) is the function that takes scheduling action a in state s, and the output value is Z value; δ is the learning rate, μ is the learning factor, s' is the new state transferred to after taking scheduling action a; max a′ Z(s′,a′) is the maximum scheduling value of all scheduling actions a′ taken in the new state s′; The state s is composed of the set of scheduled tasks S and the remaining resource vector R J Together, s=(S,R J ); After multiple iterations, Z(s,a) corresponding to all scheduling operations a is calculated, and the scheduling operation a with the largest Z(s,a) output value is selected. Its associated resource node is the optimal resource node that matches the task scheduling target.

6. The distributed computing power intelligent scheduling method according to claim 2, characterized in that: In step S3.2, the total amount of FLOPs of the calculation task 总 : FLOP 总 =FLOP 单 ×(total data volume / Batch Size)×Epoch; Among them, FLOP 单 is the amount of computation for a single batch; Batch Size is the number of samples calculated for each batch; Epoch is the number of training rounds; The effective computing power that computing resources can provide: Effective computing power = single-point computing power × number of points × utilization rate; Get the theoretical time to complete the task: Theoretical time = FLOP 总 / Effective computing power.

7. The distributed computing power intelligent scheduling method according to claim 6, characterized in that: In step S3.3, it is assumed that the task vector t is composed of the following dimensions: t=[f1,f2,…,f J ,…f M ,r1,r2,…,r J ,…r B ,G]; G is the normalized value of the estimated output data size reference value of the task vector; r J represents the Jth resource demand dimension vector among B resource demand dimension vectors; f J Represents the Jth task feature dimension vector among the M task feature dimension vectors.

8. A distributed computing power intelligent scheduling system, used to implement the distributed computing power intelligent scheduling method according to any one of claims 1 to 7, characterized in that: Includes: resource management module, resource evaluation module, task analysis and feature extraction module, task vectorization module and scheduling algorithm module; The resource management module is used to build a distributed computing resource pool and assign a unique identifier to each node; The resource evaluation module is used to generate a comprehensive score value for each resource node, quickly locate resource nodes with abnormal status through identifiers, and adjust the comprehensive score of resource nodes with abnormal status through status adjustment parameters; The task analysis and feature extraction module is used to receive the task requirements submitted by the user, calculate the theoretical time expected to execute the task, and estimate the size of the output data after the task is completed; The task vectorization module is used to integrate task characteristics, task requirements and estimated output data size values into a multi-dimensional vector to obtain a task vector; The scheduling algorithm module is used to sort the task vector set in descending order, preliminarily screen the task vector set based on task requirements, and select the optimal resource node for the task features of the screened task vectors through iterative matching of the scheduling model.

Images