Task scheduling method and system based on distributed computing

Abstract

The invention relates to the technical field of task scheduling, in particular to a task scheduling method and system based on distributed computing, and the method comprises the following steps: based on a task metadata set, extracting a computing density value of a task, counting a data input and output ratio, detecting a task dependency number, and combining the computing density value and the data ratio to carry out joint computing. According to the method, through joint calculation of the density value and the data ratio, task classification is refined, the distribution accuracy is improved, resource mismatching is avoided, the resource utilization rate is optimized in combination with task dependence and node performance matching distribution, performance bottleneck is prevented, load distribution is dynamically adjusted, node resource occupation is balanced, and overload and idle phenomena are reduced; according to the method, the dependency state is detected in real time, the uncompleted tasks are redistributed, task delay caused by dependency conflicts is avoided, the task completion time and the time sequence data of the resource utilization rate are integrated, a quantitative basis is provided for optimization, and the efficiency and the adaptive capacity of a distributed system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of task scheduling, and in particular to a task scheduling method and system based on distributed computing. Background Art

[0002] The field of task scheduling technology involves a series of methods and tools in computer science and software engineering, which are aimed at efficiently managing and distributing computing tasks to multiple processing units. The core problem in this field is how to distribute tasks among multiple computing nodes to optimize resource utilization, reduce computing time and handle complex dependencies. Task scheduling strategies can be static or dynamic, depending on the characteristics of the tasks, the state of the system and the predetermined priority. Efficient task scheduling can significantly improve the performance of distributed systems, especially in big data and cloud computing environments.

[0003] Among them, the task scheduling method based on distributed computing refers to a technical means for automatically allocating and managing computing tasks on multiple computing nodes. The purpose of this method is to maximize the work efficiency of the entire system by reasonably allocating resources, ensuring that each task can be completed in the shortest time while maintaining a balanced system load. This method is widely used in cloud computing, big data processing, and high-performance computing to cope with large-scale data processing needs.

[0004] Existing technologies lack detailed analysis of task characteristics, and task allocation often relies on coarse-grained static strategies, which makes it difficult to accurately adapt task distribution to node performance and reduces resource utilization. The lack of a dynamic load adjustment mechanism makes it easy for load differences to occur between nodes, affecting overall computing efficiency. There is a lack of real-time detection and update mechanisms in dependency management, and tasks are easily interrupted during execution due to unmet dependencies, which prolongs the task completion time. The lack of quantitative analysis methods for task completion time and resource utilization makes it difficult to accurately evaluate system performance, resulting in insufficient optimization basis. These deficiencies limit the adaptability of existing technologies in dynamic and complex scenarios, affecting the stability and efficiency of distributed systems, especially in large-scale computing scenarios. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a task scheduling method and system based on distributed computing.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: a task scheduling method based on distributed computing, comprising the following steps:

[0007] S1: Based on the task metadata set, extract the calculation density value of the task, calculate the input-output ratio of the data, detect the number of task dependencies, perform joint calculations based on the calculation density value and the data ratio, compare and classify the results with the dependency number, and generate high and low entropy task classification results;

[0008] S2: Based on the high-entropy and low-entropy task classification results, extract the dependency quantity and computational density value of the high-entropy task, match the computational density value with the node performance parameter item by item, assign the high-entropy task to the node that meets the computing capacity, and assign the low-entropy task to the remaining nodes, and generate an initial task assignment result;

[0009] S3: Based on the initial task allocation result, the total resource occupancy of each node task is calculated, the resource occupancy is compared with the available resource capacity of the node to obtain a load difference value, nodes whose load difference values ​​exceed a specified range are screened, some of their tasks are matched with the capacity of nodes with sufficient resources, and then the task allocation is adjusted to generate a node load adjustment result;

[0010] S4: Based on the node load adjustment result, detect the dependent target status of the migration task, mark the unfinished dependent tasks and their nodes, determine the status information of the dependent target nodes and unsatisfied tasks, reallocate the unfinished tasks to the corresponding dependent target nodes, update the task status, and generate a dynamic task allocation status;

[0011] S5: Based on the dynamic task allocation status, count the task completion time and resource utilization of all nodes, integrate the completion time and resource utilization according to the time series, record the time series data of node task completion and resource utilization sequence, and establish a task scheduling performance data set.

[0012] The high- and low-entropy task classification results include high-entropy task classification and low-entropy task classification. The initial task allocation results include the matching results of high-entropy tasks and nodes and the allocation results of low-entropy tasks and nodes. The node load adjustment results include a load difference node list, task migration matching results, and task allocation results after resource adjustment. The dynamic task allocation status includes the status of unfinished dependent tasks, the status of dependent target nodes, and the status of reallocated tasks. The task scheduling performance data set includes the time series data of node task completion, resource utilization series data, and performance statistics.

[0013] As a further solution of the present invention, the steps for obtaining the high and low entropy task classification results are specifically as follows:

[0014] S111: extracting a computing density value of each task based on the task metadata set, and generating a computing density value result by collecting the processing time and computing resource occupancy rate of the task and the ratio of the processing time to the resource occupancy rate;

[0015] S112: calling the calculation result of the density value, analyzing the data input and output of the task, calculating the input-output ratio by counting the sum of the input data volume and the output data volume, and generating a data ratio result;

[0016] S113: Based on the calculated density value result and the data ratio result, the number of dependencies of the task is counted, and the number of associated nodes and the hierarchical depth of the task are summarized, using the formula:

[0017]

[0018] Calculate task classification indicators and generate joint classification results;

[0019] Among them, T is the classification index, d i is the data ratio, c i To calculate the density value, D is the dependent quantity, |D represents the absolute value of the dependent quantity, represents the square root of the number of dependencies, and n represents the total number of tasks;

[0020] S114: Using the joint classification result, by comparing the classification index value with the high and low entropy classification thresholds, the tasks are classified into high and low entropy categories according to the comparison results to generate high and low entropy task classification results.

[0021] As a further solution of the present invention, the step of obtaining the initial task allocation result is specifically as follows:

[0022] S211: Based on the high-entropy and low-entropy task classification results, high-entropy tasks are screened, the dependency quantity and computational density value of each high-entropy task are recorded, and dependency and computational density analysis results are generated by analyzing the resource usage and dependency properties of the tasks;

[0023] S212: starting from the dependency and computation density analysis results, matching the computation density value with the node performance parameter, capturing the most matching node for each task, and generating a matching result by calculating the matching degree between the performance requirements of multiple tasks and the node capabilities;

[0024] S213: According to the matching result, the task allocation strategy is optimized, and high entropy tasks are preferentially allocated to nodes with corresponding computing capabilities, and low entropy tasks are allocated according to the remaining resources, using the formula:

[0025]

[0026] Calculate the task allocation index of each node and generate the initial task allocation result;

[0027] Among them, A i Assign an index to the task of node i, c i is the computational density of the task, pi is the performance parameter of node i, represents the sum of all node performance parameters, ∈ to prevent division by zero error, n represents the total number of nodes, p k Represents the performance parameter of the kth node.

[0028] As a further solution of the present invention, the step of obtaining the node load adjustment result is specifically:

[0029] S311: Based on the initial task allocation result, calculate the total resource usage of all tasks on each node, and generate the total resource usage of each node by accumulating the GPU, CPU, memory and storage resources required by the tasks on each node;

[0030] S312: Compare the total resource occupation of each node with its total available resource capacity, calculate the resource utilization rate of each node, and obtain the load difference value through the difference between the resource utilization rate and the ideal utilization rate;

[0031] S313: Filter the nodes whose load difference values ​​exceed the specified range, and adjust the matching of tasks and resources with nodes with sufficient resources, using the formula:

[0032]

[0033] Calculate the node load difference percentage to reflect the imbalanced resource allocation between nodes and generate the node load adjustment result;

[0034] Among them, Δ i is the load difference percentage of node i, u i is the actual resource utilization of node i, U is the target resource utilization, ∈ is a small constant to prevent division by zero errors, and v i Represents the number of tasks on node i.

[0035] As a further solution of the present invention, the step of acquiring the dynamic task allocation state is specifically as follows:

[0036] S411: Based on the node load adjustment result, detecting the dependent target state of the migration task, and generating a task dependency state analysis result by analyzing the dependency relationship of each migration task and the completion state of the target node;

[0037] S412: calling the task dependency status analysis result, marking all unfinished dependent tasks and their nodes, combining the dependent target nodes and the status information of the unsatisfied tasks, and generating an unfinished task status analysis result through the dependency status matching logic;

[0038] S413: Using the unfinished task status analysis result, reallocate the unfinished task to the corresponding node where the dependent target is located, using the formula:

[0039]

[0040] Calculate task status update indicators and generate dynamic task allocation status;

[0041] Among them, S i represents the status update index of task i, which is used to measure the adaptability of the task to the target node after reallocation, w j represents the dependency weight of task j, d ij represents the dependency strength between task i and task j, c i It represents the available resources of the target node and is used to evaluate the resource matching of the assigned tasks. ∈ is a constant to avoid division by zero errors.

[0042] As a further solution of the present invention, the step of acquiring the task scheduling performance data set is specifically as follows:

[0043] S511: Based on the dynamic task allocation state, the task completion time of each node is counted, and the completion time statistics of each node task are generated by recording the start and end time points of each task and comprehensively referring to the parallelism and sequence of multiple tasks;

[0044] S512: synchronously calling the completion time statistics of each node task, counting the resource utilization of each node, and integrating and obtaining the resource utilization time series data of each node by real-time monitoring the GPU, CPU, memory and storage usage of multiple nodes using the weighted average method to generate resource utilization statistics;

[0045] S513: Integrate the completion time statistics of each node task and the resource utilization statistics, integrate the task completion time and resource utilization in time series, record the time series data of node task completion and resource utilization series, and use the formula:

[0046]

[0047] Calculate the task scheduling performance index of each node and generate a task scheduling performance data set;

[0048] Among them, P i represents the comprehensive performance index of node i, referring to the deviation between the task completion time and the preset standard, as well as the task weight, w j is the weight of task j, indicating the criticality or urgency of the task, t ij is the actual completion time of task j on node i, T jis the scheduled completion time of task j, providing a benchmark to compare the actual execution time, ∈ is to avoid the small amount of division by zero error introduced, and m represents the total number of tasks on node i, which is used to determine the scope of tasks in the performance indicator calculation.

[0049] A task scheduling system based on distributed computing, the task scheduling system based on distributed computing is used to execute the above-mentioned task scheduling method based on distributed computing, the system comprises:

[0050] The task classification module extracts the calculation density value of the task based on the task metadata set, calculates the input-output ratio of the data, performs joint calculations on the calculation density value and ratio, compares the calculation results with the number of task dependencies, and generates high and low entropy task classification results;

[0051] The node matching module extracts the dependency quantity and computational density value of the high entropy task based on the high and low entropy task classification results, matches the computational density value of the high entropy task with the node performance parameter, assigns the high entropy task to the corresponding node, and the low entropy task to the remaining nodes, thereby generating an initial task assignment result;

[0052] The load balancing module counts the total resource usage of multi-node tasks based on the initial task allocation result, compares the total resource usage with the available resource capacity of the node, calculates the load difference value, selects the nodes with the difference value exceeding the range, compares and adjusts the allocation of the tasks of the nodes exceeding the range with the nodes with sufficient resources, and generates the node load adjustment result;

[0053] Based on the node load adjustment result, the dependency adjustment module extracts the migration task dependency target status, marks the unfinished dependent tasks and their nodes, extracts the dependent target nodes and task status, reallocates the unfinished tasks to the dependent target nodes, and establishes a dynamic task allocation status.

[0054] Compared with the prior art, the advantages and positive effects of the present invention are:

[0055] In the present invention, by calculating the density value and the data ratio jointly, the task classification is refined, the allocation accuracy is improved, and resource mismatch is avoided. The matching allocation of task dependency and node performance is combined to optimize resource utilization and prevent performance bottlenecks. The load distribution is dynamically adjusted to balance node resource occupancy and reduce overload and idle phenomena. The dependency status is detected in real time and unfinished tasks are reallocated to avoid task delays due to dependency conflicts. The time series data of task completion time and resource utilization are integrated to provide a quantitative basis for optimization and improve the efficiency and adaptability of distributed systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of the workflow of the present invention;

[0057] Figure 2A flowchart of the steps for obtaining the classification results of high and low entropy tasks of the present invention;

[0058] Figure 3 A flowchart of the steps for obtaining the initial task allocation result of the present invention;

[0059] Figure 4 A flowchart of the steps for obtaining the node load adjustment result of the present invention;

[0060] Figure 5 A flowchart of the steps for obtaining the dynamic task allocation state of the present invention;

[0061] Figure 6 The flowchart is a step chart of obtaining a task scheduling performance data set according to the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0063] In the description of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, in the description of the present invention, "multiple" means two or more, unless otherwise clearly and specifically defined.

[0064] Embodiment 1

[0065] See also Figure 1 The present invention provides a technical solution: a task scheduling method based on distributed computing, comprising the following steps:

[0066] S1: Based on the task metadata set, extract the calculation density value of the task, calculate the input-output ratio of the data, detect the number of task dependencies, perform joint calculations based on the calculation density value and the data ratio, compare and classify the results with the dependency number, and generate high and low entropy task classification results;

[0067] S2: Based on the classification results of high and low entropy tasks, extract the dependency quantity and computational density value of high entropy tasks, match the computational density value with the node performance parameters item by item, assign high entropy tasks to nodes that meet the computing capacity, and assign low entropy tasks to the remaining nodes to generate the initial task assignment results;

[0068] S3: Based on the initial task allocation results, the total resource usage of each node task is calculated, and the resource usage is compared with the available resource capacity of the node to obtain the load difference value. The nodes whose load difference values ​​exceed the specified range are screened, and some of their tasks are matched with the capacity of nodes with sufficient resources, and then the task allocation is adjusted to generate the node load adjustment result;

[0069] S4: Based on the node load adjustment result, detect the dependent target status of the migration task, mark the unfinished dependent tasks and their nodes, determine the status information of the dependent target nodes and unsatisfied tasks, reallocate the unfinished tasks to the corresponding dependent target nodes, update the task status, and generate the dynamic task allocation status;

[0070] S5: Based on the dynamic task allocation status, count the task completion time and resource utilization of all nodes, integrate the completion time and resource utilization according to the time series, record the time series data of node task completion and resource utilization sequence, and establish a task scheduling performance data set.

[0071] The high and low entropy task classification results include high entropy task classification and low entropy task classification. The initial task allocation results include the matching results of high entropy tasks and nodes, and the allocation results of low entropy tasks and nodes. The node load adjustment results include the load difference node list, task migration matching results, and task allocation results after resource adjustment. The dynamic task allocation status includes the status of unfinished dependent tasks, the status of dependent target nodes, and the status of reallocated tasks. The task scheduling performance data set includes the time series data of node task completion, resource utilization series data, and performance statistics.

[0072] See also Figure 2 ,The specific steps for obtaining the classification results of high and low entropy tasks are:

[0073] S111: extracting a computing density value of each task based on the task metadata set, and generating a computing density value result by collecting the processing time and computing resource occupancy rate of the task and the ratio of the processing time to the resource occupancy rate;

[0074] Based on the collection and analysis of task metadata sets, the computing density value of each task is obtained by specifically measuring the processing time and resource occupancy of each task, combined with the data monitoring system of the actual operating environment. This process involves detailed records of the time and resource consumption of task execution to ensure the accuracy and reliability of the data, so as to accurately calculate the ratio of computing resource occupancy to processing time of each task; through this method, a detailed computing density value result can be obtained, which can be directly used for the next step of data ratio analysis.

[0075] S112: calling the density value calculation result, analyzing the data input and output of the task, calculating the input-output ratio by counting the sum of the input data volume and the output data volume, and generating a data ratio result;

[0076] The density value calculation result is called, and the input and output of the task data are analyzed in detail, especially in a high-performance computing environment, and the input and output data are accurately measured and their ratio is calculated. This process includes the processing and analysis of a large number of data points to ensure the accuracy and consistency of the data input and output statistics of each task, and then obtain accurate data ratio results, which will serve as an important basis for task-dependent quantity statistics and subsequent classification.

[0077] S113: Based on the calculation results of density value and data ratio, the number of dependencies of the task is counted, and the number of associated nodes and the depth of the hierarchy of the task are summarized using the formula:

[0078]

[0079] Calculate task classification indicators and generate joint classification results;

[0080] Among them, T is the classification index, d i is the data ratio, c i To calculate the density value, D is the dependent quantity, |D represents the absolute value of the dependent quantity, represents the square root of the number of dependencies, and n represents the total number of tasks;

[0081] formula:

[0082]

[0083] The benefit of the formula is that by weighting the product of the data ratio and the computational density value, and then considering the square root and absolute value of the number of task dependencies, the overall complexity and resource requirements of the task can be evaluated more carefully, thereby effectively classifying the tasks.

[0084] Detailed explanation of the formula and the process of formula calculation and derivation:

[0085] Setting i =0.5,c i =1.2, D=4, the calculation formula is:

[0086]

[0087] The result shows that the task classification index is 1.0, which indicates that these tasks have medium dependencies and computational complexity, and the tasks can be appropriately classified into high and low entropy according to this index.

[0088] S114: Using the joint classification result, by comparing the classification index value with the high and low entropy classification thresholds, the tasks are classified into high and low entropy categories according to the comparison results to generate high and low entropy task classification results.

[0089] Using the joint classification results, the classification indicators are compared in detail with the set high and low entropy classification thresholds. This process includes detailed analysis and evaluation of the classification indicators to ensure that each task can be accurately classified according to its resource requirements and dependencies. Through comparative analysis, the entropy classification of each task can be clarified, providing a basis for resource allocation and task optimization, and generating high and low entropy task classification results, which helps to optimize task scheduling strategies and resource allocation efficiency.

[0090] See also Figure 3 , the specific steps for obtaining the initial task allocation results are:

[0091] S211: Based on the classification results of high and low entropy tasks, high entropy tasks are screened, the dependency quantity and computational density value of each high entropy task are recorded, and dependency and computational density analysis results are generated by analyzing the resource usage and dependency properties of the tasks;

[0092] During the screening process of high-entropy tasks, the system analyzes the resource usage and dependency properties of the tasks, records the dependency quantity and computational density value of each task, and determines which tasks have higher resource requirements and complex dependency chains by calculating the statistical data of resource usage and task dependencies. These data are obtained through real-time monitoring and historical data analysis to ensure the accuracy and real-time nature of the data. The system generates dependency and computational density analysis results based on these data. This process involves a large amount of data collection, processing and analysis to ensure the accuracy of task screening and the rationality of operations.

[0093] S212: starting from the dependency and computation density analysis results, matching the computation density value with the node performance parameter, capturing the most matching node for each task, and generating a matching result by calculating the matching degree between the performance requirements of multiple tasks and the node capabilities;

[0094] By calling the dependency and computing density analysis results, the system matches the computing density value with the node performance parameters item by item, and makes a detailed comparison between the computing requirements of each task and the processing power of each node. During this comparison process, multiple performance indicators such as the node's GPU floating-point computing performance, CPU speed, memory size, and I / O capabilities are considered. These performance data are directly extracted from the node's hardware configuration information and verified through performance test results to ensure the accuracy and practicality of the data. The system optimizes the task allocation strategy based on the comparison results and finds the most suitable node for each task, ensuring maximum resource utilization and improved task execution efficiency. The generated matching results provide an accurate execution guide for actual task allocation.

[0095] S213: According to the matching result, the task allocation strategy is optimized, and high entropy tasks are preferentially allocated to nodes with corresponding computing capabilities, and low entropy tasks are allocated according to the remaining resources, using the formula:

[0096]

[0097] Calculate the task allocation index of each node and generate the initial task allocation result;

[0098] Among them, A i Assign an index to the task of node i, c i is the computational density of the task, p i is the performance parameter of node i, represents the sum of all node performance parameters, ∈ to prevent division by zero error, n represents the total number of nodes, p k Represents the performance parameter of the kth node.

[0099] formula:

[0100]

[0101] The benefit of the formula is that it takes into account the performance parameters of each node and the computational density of the task, and improves the stability and accuracy of the calculation by adding a small constant ∈ in the denominator to avoid division by zero errors.

[0102] Detailed explanation of the formula and the process of formula calculation and derivation:

[0103] Assuming a simplified example, there are three nodes with performance parameters p1 = 10, p2 = 15, p3 = 20, the computational density of the task is c1 = 30, and the small constant ∈ = 0.1, the formula becomes:

[0104]

[0105] The result A1=6.62 indicates that for node 1, the allocation index of task 1 is 6.62, which reflects the efficiency and suitability of node 1 relative to other nodes in processing task 1.

[0106] This result shows that node 1 has higher fitness and efficiency in processing task 1, which helps to optimize the overall task allocation strategy and improve the operating efficiency of the system.

[0107] See also Figure 4 , the specific steps for obtaining the node load adjustment result are:

[0108] S311: Based on the initial task allocation result, the total resource usage of all tasks on each node is calculated, and the total resource usage of each node is generated by accumulating the GPU, CPU, memory and storage resources required by the tasks on each node;

[0109] Based on the initial task allocation results, the total resource usage of all tasks on each node is obtained by accumulating the GPU, CPU, memory and storage resources required for each task. This calculation process not only involves the calculation of the total resources, but also requires detailed analysis and recording of the usage of each resource. First, the resource requirements of each task are collected, and then added one by one to the total node amount to ensure the correct aggregation of each type of resource. In this process, the data integrity and accuracy of each resource type must also be verified to avoid errors in subsequent resource allocation, ensuring efficient and accurate system resource management, thereby providing accurate basic data for node load balancing.

[0110] S312: Compare the total resource usage of each node with its total available resource capacity, calculate the resource utilization rate of each node, and obtain the load difference value through the difference between the resource utilization rate and the ideal utilization rate;

[0111] Data is obtained from the total resource usage. By comparing the resource usage of each node with its total available resource capacity, the resource utilization rate of each node is calculated. The calculation of resource utilization rate requires accurate measurement of the current resource usage status and total resource capacity of each node. This process involves comparative operations of resource usage and total resource capacity to ensure that each data is processed correctly. The load difference value is obtained by calculating the difference between the resource usage rate and the ideal usage rate. This load difference value is a key indicator for evaluating the operating efficiency of the node, helping administrators understand the resource usage of each node and providing decision support for further resource optimization.

[0112] S313: Filter nodes whose load difference values ​​exceed the specified range, and adjust the matching of tasks and resources with nodes with sufficient resources, using the formula:

[0113]

[0114] Calculate the node load difference percentage to reflect the imbalanced resource allocation between nodes and generate the node load adjustment result;

[0115] Among them, Δ i is the load difference percentage of node i, u i is the actual resource utilization of node i, U is the target resource utilization, ∈ is a small constant to prevent division by zero errors, and v i Represents the number of tasks on node i.

[0116] formula:

[0117]

[0118] The benefit of the formula is that by introducing the inverse square root operation of the number of node tasks, the sensitivity of the calculation model to the difference in the number of tasks is enhanced, making the load difference assessment more accurate. Especially when the number of tasks changes significantly, the calculation of the load difference percentage can be effectively adjusted to make it closer to the actual situation.

[0119] Detailed explanation of the formula and the process of formula calculation and derivation:

[0120] Assume that the actual resource usage u of node i i is 75%, the target utilization U is 60%, and the parameter ∈ is 0.1 to avoid the denominator being zero. The number of tasks v on node i i For 10:

[0121]

[0122] The result shows that the load difference percentage of node i is 113.31%, which means that the actual load of node i is much higher than the target load. Task reallocation is needed to reduce the load difference and ensure the efficiency and stability of system operation.

[0123] See also Figure 5 ,The specific steps for obtaining the dynamic task allocation status are:

[0124] S411: Based on the node load adjustment result, detect the dependent target state of the migration task, and generate a task dependency state analysis result by analyzing the dependency relationship of each migration task and the completion state of the target node;

[0125] Based on the node load adjustment results, we start to analyze the dependency status of each task and mark the unfinished dependent tasks. This process involves a detailed review of the existing task status and node load data. First, we extract the current status of each task and the list of dependent tasks from the database, and then compare the actual running status of the node. For those dependent tasks that are not completed, we mark these tasks and their current nodes. This process ensures that the tasks that need to be rescheduled can be accurately identified, so as to prepare for the next step of task reallocation. The whole process not only involves data comparison and screening, but also requires comprehensive consideration of the dependencies between tasks and the capacity of nodes to ensure that each step is based on the latest and most comprehensive data information to achieve optimal resource utilization and task execution efficiency.

[0126] S412: calling the task dependency status analysis result, marking all unfinished dependent tasks and their nodes, combining the dependent target nodes and the status information of the unsatisfied tasks, and generating the unfinished task status analysis result through the dependency status matching logic;

[0127] After completing the analysis of the task dependency status, the specific status information of the dependent target nodes and unmet tasks is further confirmed. This step is a key link in the task reallocation process. It not only involves a detailed review of the current status of the task, but also includes a performance evaluation of the node where the task is located. A comprehensive evaluation model is used to determine the priority and required resources of each task. The model takes into account the urgency of the task, resource consumption, and complexity of the dependency relationship, and outputs a scheduling plan with a high degree of matching between tasks and nodes. This plan will serve as the basis for reallocating tasks to ensure maximum utilization of system resources and efficient task execution. This process ensures that all tasks can be executed on the most suitable nodes to achieve optimal resource utilization.

[0128] S413: Using the unfinished task status analysis results, reallocate the unfinished tasks to the corresponding nodes where the dependent targets are located, using the formula:

[0129]

[0130] Calculate task status update indicators and generate dynamic task allocation status;

[0131] Among them, S i represents the status update index of task i, which is used to measure the adaptability of the task to the target node after reallocation, w j represents the dependency weight of task j, d ij represents the dependency strength between task i and task j, c i It represents the available resources of the target node and is used to evaluate the resource matching of the assigned tasks. ∈ is a constant to avoid division by zero errors.

[0132] formula:

[0133]

[0134] The benefit of the formula is that it allows the system to dynamically adjust task allocation based on task dependencies and the current resource status of the nodes by increasing the weight parameter w j and the dependence strength d ij , the formula can more accurately measure the mutual impact between tasks and the demand for resources, thereby optimizing the load and performance of the entire system.

[0135] Detailed explanation of the formula and the process of formula calculation and derivation:

[0136] Assume that there are three tasks on a node, the dependency weight of task 1 is 0.5, the dependency weight of task 2 is 0.3, and the dependency weight of task 3 is 0.2. The available resources of the node are 10 units. Considering that the anti-zero factor ∈ of the formula is 0.1, the dependency strengths of task 1 to task 3 are 0.8, 0.6, and 0.4 respectively. The contribution of each task is calculated as follows:

[0137] Contribution Task1 =0.5×0.8=0.4

[0138] Contribution Task2 =0.3×0.6=0.18

[0139] Contribution Task3 =0.2×0.4=0.08

[0140] The total contribution is

[0141] 0.4+0.18+0.08=0.66

[0142] Then the node's task status update index S is:

[0143]

[0144] The result shows that the node's task status update index is 0.208, which means that the node's resource utilization efficiency is being optimized based on the existing task dependency weights and strengths, which helps improve the node's ability to handle complex tasks, thereby improving the performance and response speed of the entire network.

[0145] See also Figure 6 ,The specific steps for obtaining the task scheduling performance data set are:

[0146] S511: Based on the dynamic task allocation state, the task completion time of each node is counted, and the completion time statistics of each node task are generated by recording the start and end time points of each task and comprehensively referring to the parallelism and sequence of multiple tasks;

[0147] Based on the statistics of dynamic task allocation status, we start to record the start and end time of tasks on each node in detail. In this way, we can not only observe the independent running time of each task, but also analyze the dependencies between tasks and their impact on completion time. Furthermore, this monitoring helps optimize the task scheduling algorithm to reduce waiting time and resource waste, and ensure that each node runs under the optimal load. With these data, detailed time logs can be created, which will be used to analyze the execution efficiency of tasks and the performance of nodes.

[0148] S512: synchronously calling the completion time statistics of each node task, counting the resource utilization of each node, and using the weighted average method to integrate and obtain the resource utilization time series data of each node by real-time monitoring the GPU, CPU, memory and storage usage of multiple nodes, and generating resource utilization statistics;

[0149] Based on the completion time statistics, the resource utilization of each node is monitored at the same time, and the usage of GPU, CPU, memory and storage resources is recorded in real time. This monitoring includes not only the peak utilization of resources, but also the average utilization of resources, which is crucial for analyzing node performance and determining future resource allocation strategies. For example, if the memory utilization of a node continues to be higher than the average level, it may be necessary to consider upgrading the hardware or reallocating some memory-intensive tasks to optimize the performance and efficiency of the overall system. Resource utilization data will be used to evaluate the health of the node and adjust maintenance plans.

[0150] S513: Integrate the completion time statistics and resource utilization statistics of each node task, integrate the task completion time and resource utilization in time series, record the time series data of node task completion and resource utilization series, and use the formula:

[0151]

[0152] Calculate the task scheduling performance index of each node and generate a task scheduling performance data set;

[0153] Among them, P i represents the comprehensive performance index of node i, referring to the deviation between the task completion time and the preset standard, as well as the task weight, w j is the weight of task j, indicating the criticality or urgency of the task, t ij is the actual completion time of task j on node i, T j is the scheduled completion time of task j, providing a benchmark to compare the actual execution time, ∈ is to avoid the small amount of division by zero error introduced, and m represents the total number of tasks on node i, which is used to determine the scope of tasks in the performance indicator calculation.

[0154] formula:

[0155]

[0156] The benefit of the formula is that it not only takes into account the completion time of each task, but also can evaluate the performance of nodes more fairly by introducing the weight and scheduled time of tasks, which is applicable to scenarios with multiple task importance and urgency.

[0157] Detailed explanation of the formula and the process of formula calculation and derivation:

[0158] Set task weight w j = 2, the actual completion time t of task j on node i ij = 3 hours, the scheduled completion time of the task is T j = 2.5 hours, ∈ = 0.1 to prevent division by zero error:

[0159]

[0160] The result shows that node i performs well in the given evaluation period and completes the task more efficiently than the predetermined target, which indicates that the task scheduling and resource allocation strategies are relatively reasonable.

[0161] A task scheduling system based on distributed computing, the task scheduling system based on distributed computing is used to execute the above-mentioned task scheduling method based on distributed computing, the system includes:

[0162] The task classification module extracts the calculation density value of the task based on the task metadata set, calculates the input-output ratio of the data, performs joint calculations on the calculation density value and ratio, compares the calculation results with the number of task dependencies, and generates high and low entropy task classification results;

[0163] Based on the classification results of high and low entropy tasks, the node matching module extracts the dependency quantity and computational density value of high entropy tasks, matches the computational density value of high entropy tasks with node performance parameters, assigns high entropy tasks to corresponding nodes, and assigns low entropy tasks to remaining nodes, generating the initial task assignment results.

[0164] Based on the initial task allocation results, the load balancing module counts the total resource usage of multi-node tasks, compares the total resource usage with the available resource capacity of the nodes, calculates the load difference value, selects nodes with out-of-range differences, compares and adjusts the allocation of out-of-range node tasks with nodes with sufficient resources, and generates node load adjustment results;

[0165] Based on the node load adjustment results, the dependency adjustment module extracts the dependency target status of the migration task, marks the unfinished dependent tasks and their nodes, extracts the dependency target nodes and task status, reallocates the unfinished tasks to the nodes where the dependent targets are located, and establishes a dynamic task allocation status.

[0166] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in other forms. Any technician familiar with the profession may use the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A task scheduling method based on distributed computing, characterized in that: The following steps are involved: Based on the task metadata set, extract the calculation density value of the task, calculate the input-output ratio of the data, detect the number of task dependencies, combine the calculation density value with the data ratio for joint calculation, compare and classify the results with the dependency number, and generate high and low entropy task classification results; Based on the high-entropy and low-entropy task classification results, the dependency quantity and computational density value of the high-entropy task are extracted, the computational density value is matched with the node performance parameter item by item, the high-entropy task is assigned to the node that meets the computing capacity, and the low-entropy task is assigned to the remaining nodes, so as to generate an initial task assignment result; Based on the initial task allocation result, the total resource occupancy of each node task is calculated, the resource occupancy is compared with the available resource capacity of the node to obtain a load difference value, nodes whose load difference values ​​exceed a specified range are screened, and some of their tasks are matched with the capacity of nodes with abundant resources, and then the task allocation is adjusted to generate a node load adjustment result; Based on the node load adjustment result, detect the dependent target status of the migration task, mark the unfinished dependent tasks and their nodes, determine the status information of the dependent target nodes and unsatisfied tasks, reallocate the unfinished tasks to the nodes where the corresponding dependent targets are located, and update the task status to generate a dynamic task allocation status; Based on the dynamic task allocation status, the task completion time and resource utilization of all nodes are counted, the completion time and resource utilization are integrated according to the time series, the time series data of the node task completion and the resource utilization sequence are recorded, and the task scheduling performance data set is established.

2. The task scheduling method based on distributed computing according to claim 1, characterized in that: The high- and low-entropy task classification results include high-entropy task classification and low-entropy task classification. The initial task allocation results include the matching results of high-entropy tasks and nodes and the allocation results of low-entropy tasks and nodes. The node load adjustment results include a load difference node list, task migration matching results, and task allocation results after resource adjustment. The dynamic task allocation status includes the status of unfinished dependent tasks, the status of dependent target nodes, and the status of reallocated tasks. The task scheduling performance data set includes the time series data of node task completion, resource utilization series data, and performance statistics.

3. The task scheduling method based on distributed computing according to claim 2 is characterized in that: The steps for obtaining the high and low entropy task classification results are specifically as follows: Based on the task metadata set, the computing density value of each task is extracted. By collecting the task processing time and computing resource occupancy rate, and the ratio of processing time to resource occupancy rate, the computing density value result is generated. The density value calculation result is called to analyze the data input and output of the task, and the input-output ratio is calculated by counting the sum of the input data volume and the output data volume to generate a data ratio result; Based on the calculated density value result and the data ratio result, the number of dependencies of the task is counted, and the number of associated nodes and the hierarchical depth of the task are summarized, using the formula: Calculate task classification indicators and generate joint classification results; Among them, T is the classification index, d i is the data ratio, c i To calculate the density value, D is the dependent quantity, |D represents the absolute value of the dependent quantity, represents the square root of the number of dependencies, and n represents the total number of tasks; By using the joint classification result, by comparing the classification index value with the high and low entropy classification thresholds, the tasks are classified into high and low entropy categories according to the comparison results to generate high and low entropy task classification results.

4. The task scheduling method based on distributed computing according to claim 3 is characterized in that: The steps for obtaining the initial task allocation result are specifically as follows: Based on the high and low entropy task classification results, high entropy tasks are screened, the dependency quantity and computational density value of each high entropy task are recorded, and dependency and computational density analysis results are generated by analyzing the resource usage and dependency properties of the tasks; Based on the dependency and computational density analysis results, the computational density value is matched with the node performance parameter, the best matching node is captured for each task, and the matching degree result is generated by calculating the matching degree between the performance requirements of multiple tasks and the node capabilities; According to the matching results, the task allocation strategy is optimized to prioritize high entropy tasks to nodes with corresponding computing capabilities, and low entropy tasks are allocated according to the remaining resources using the formula: Calculate the task allocation index of each node and generate the initial task allocation result; Among them, A i Assign an index to the task of node i, c i is the computational density of the task, p i is the performance parameter of node i, represents the sum of all node performance parameters, ∈ to prevent division by zero error, n represents the total number of nodes, p k Represents the performance parameter of the kth node.

5. The task scheduling method based on distributed computing according to claim 4 is characterized in that: The steps for obtaining the node load adjustment result are specifically as follows: Based on the initial task allocation result, the total resource usage of all tasks on each node is calculated, and the total resource usage of each node is generated by accumulating the GPU, CPU, memory and storage resources required by the tasks on each node; Compare the total resource usage of each node with its total available resource capacity, calculate the resource utilization rate of each node, and obtain the load difference value by the difference between the resource utilization rate and the ideal utilization rate; Nodes whose load difference values ​​exceed the specified range are selected, and matching and adjusting tasks and resources with nodes with sufficient resources are performed using the formula: Calculate the node load difference percentage to reflect the imbalanced resource allocation between nodes and generate the node load adjustment result; Among them, Δ i is the load difference percentage of node i, u i is the actual resource utilization of node i, U is the target resource utilization, ∈ is a small constant to prevent division by zero errors, and v i Represents the number of tasks on node i.

6. The task scheduling method based on distributed computing according to claim 5, characterized in that: The steps for obtaining the dynamic task allocation status are specifically as follows: Based on the node load adjustment result, detecting the dependent target state of the migration task, and generating a task dependency state analysis result by analyzing the dependency relationship of each migration task and the completion state of the target node; Call the task dependency status analysis result, mark all unfinished dependent tasks and their nodes, combine the dependent target nodes and the status information of unsatisfied tasks, and generate the unfinished task status analysis result through the dependency status matching logic; Using the unfinished task status analysis results, the unfinished tasks are reallocated to the corresponding nodes where the dependent targets are located, using the formula: Calculate task status update indicators and generate dynamic task allocation status; Among them, S i represents the status update index of task i, which is used to measure the adaptability of the task to the target node after reallocation, w j represents the dependency weight of task j, d ij represents the dependency strength between task i and task j, c i It represents the available resources of the target node and is used to evaluate the resource matching of the assigned tasks. ∈ is a constant to avoid division by zero errors.

7. The task scheduling method based on distributed computing according to claim 6 is characterized in that: The steps for obtaining the task scheduling performance data set are specifically as follows: Based on the dynamic task allocation state, the task completion time of each node is counted, and by recording the start and end time points of each task, the completion time statistics of each node task are generated by comprehensively referring to the parallelism and sequence of multiple tasks; Synchronously call the completion time statistics of each node task, count the resource utilization of each node, monitor the GPU, CPU, memory and storage usage of multiple nodes in real time, use the weighted average method to integrate and obtain the resource utilization time series data of each node, and generate resource utilization statistics; Integrate the completion time statistics of each node task and the resource utilization statistics, integrate the task completion time and resource utilization in time series, record the time series data of node task completion and resource utilization series, and use the formula: Calculate the task scheduling performance index of each node and generate a task scheduling performance data set; Among them, P i represents the comprehensive performance index of node i, referring to the deviation between the task completion time and the preset standard, as well as the task weight, w j is the weight of task j, indicating the criticality or urgency of the task, t ij is the actual completion time of task j on node i, T j is the scheduled completion time of task j, providing a benchmark to compare the actual execution time, ∈ is to avoid the small amount of division by zero error introduced, and m represents the total number of tasks on node i, which is used to determine the scope of tasks in the performance indicator calculation.

8. A task scheduling system based on distributed computing, characterized in that: According to the distributed computing-based task scheduling method according to any one of claims 1 to 7, the system comprises: The task classification module extracts the calculation density value of the task based on the task metadata set, calculates the input-output ratio of the data, performs joint calculation on the calculation density value and ratio, compares the calculation result with the task dependency quantity, and generates high and low entropy task classification results; The node matching module extracts the dependency quantity and computational density value of the high entropy task based on the high and low entropy task classification results, matches the computational density value of the high entropy task with the node performance parameter, assigns the high entropy task to the corresponding node, and the low entropy task to the remaining nodes, thereby generating an initial task assignment result; The load balancing module counts the total resource usage of multi-node tasks based on the initial task allocation result, compares the total resource usage with the available resource capacity of the nodes, calculates the load difference value, selects nodes with out-of-range difference values, compares and adjusts the allocation of out-of-range node tasks with nodes with sufficient resources, and generates a node load adjustment result; Based on the node load adjustment result, the dependency adjustment module extracts the migration task dependency target status, marks the unfinished dependent tasks and their nodes, extracts the dependent target nodes and task status, reallocates the unfinished tasks to the dependent target nodes, and establishes a dynamic task allocation status.

Images