A task scheduling method and system based on distributed computing

Abstract

The present invention relates to the field of task scheduling technology, specifically a task scheduling method and system based on distributed computing, comprising the following steps: based on a task metadata set, extracting a computational density value of the task, statistically analyzing input and output ratios of the data, detecting the number of task dependencies, and performing joint calculations in combination with the computational density value and the data ratio. In the present invention, by jointly calculating the computational density value and the data ratio, task classification is refined, allocation accuracy is improved, resource mismatch is avoided, resource utilization is optimized by combining task dependency with node performance matching allocation, performance bottlenecks are prevented, load distribution is dynamically adjusted, node resource occupancy is balanced, overload and idle phenomena are reduced, dependency status is detected in real time and unfinished tasks are reallocated, task delays due to dependency conflicts are avoided, time series data of task completion time and resource utilization are integrated, a quantitative basis is provided for optimization, and the efficiency and adaptability of distributed systems are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of task scheduling, and in particular to a task scheduling method and system based on distributed computing. Background Art

[0002] The field of task scheduling encompasses a range of methods and tools in computer science and software engineering aimed at efficiently managing and distributing computing tasks across multiple processing units. A central question in this area is how to allocate tasks across multiple computing nodes to optimize resource utilization, reduce computation time, and handle complex dependencies. Task scheduling strategies can be static or dynamic, depending on the characteristics of the tasks, the system state, and predetermined priorities. Efficient task scheduling can significantly improve the performance of distributed systems, particularly in big data and cloud computing environments.

[0003] Distributed computing-based task scheduling refers to a technical approach used to automatically allocate and manage computing tasks across multiple computing nodes. This approach aims to maximize overall system efficiency by rationally allocating resources, ensuring that tasks are completed in the shortest possible time while maintaining a balanced system load. This approach is widely used in cloud computing, big data processing, and high-performance computing to address large-scale data processing needs.

[0004] Existing technologies lack detailed analysis of task characteristics, and task allocation often relies on coarse-grained static strategies, which makes it difficult to accurately adapt task distribution to node performance and reduces resource utilization. The lack of a dynamic load adjustment mechanism makes it easy for load differences to occur between nodes, affecting overall computing efficiency. The lack of a real-time detection and update mechanism in dependency management means that tasks are easily interrupted during execution due to unmet dependencies, extending task completion time. The lack of quantitative analysis methods for task completion time and resource utilization makes it difficult to accurately evaluate system performance, resulting in insufficient basis for optimization. These deficiencies limit the adaptability of existing technologies in dynamic and complex scenarios, affecting the stability and efficiency of distributed systems, especially in large-scale computing scenarios. Summary of the Invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a task scheduling method and system based on distributed computing.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: a task scheduling method based on distributed computing, comprising the following steps:

[0007] S1: Based on the task metadata set, extract the computational density value of the task, calculate the input-output ratio of the data, detect the number of task dependencies, perform joint calculations based on the computational density value and the data ratio, compare and classify the results with the dependency number, and generate high-entropy and low-entropy task classification results;

[0008] S2: Based on the high-entropy and low-entropy task classification results, extract the dependency quantity and computational density value of the high-entropy task, match the computational density value with the node performance parameters item by item, assign the high-entropy task to the node with the required computing capacity, and assign the low-entropy task to the remaining nodes, thereby generating an initial task assignment result;

[0009] S3: Based on the initial task allocation result, the total resource usage of each node task is calculated, and the resource usage is compared with the available resource capacity of the node to obtain a load difference value. Nodes with load difference values exceeding a specified range are screened, and some of their tasks are matched with the capacity of nodes with sufficient resources. After that, the task allocation is adjusted to generate a node load adjustment result.

[0010] S4: Based on the node load adjustment result, detect the dependent target status of the migration task, mark the unfinished dependent tasks and their nodes, determine the status information of the dependent target nodes and unsatisfied tasks, reallocate the unfinished tasks to the nodes where the corresponding dependent targets are located, and update the task status to generate a dynamic task allocation status;

[0011] S5: Based on the dynamic task allocation status, count the task completion time and resource utilization of all nodes, integrate the completion time and resource utilization according to the time series, record the time series data of node task completion and resource utilization sequence, and establish a task scheduling performance data set.

[0012] The high- and low-entropy task classification results include high-entropy task classification and low-entropy task classification. The initial task allocation results include the matching results of high-entropy tasks and nodes and the allocation results of low-entropy tasks and nodes. The node load adjustment results include a list of load difference nodes, task migration matching results, and task allocation results after resource adjustment. The dynamic task allocation status includes the status of unfinished dependent tasks, the status of dependent target nodes, and the status of reallocated tasks. The task scheduling performance data set includes time series data of node task completion, resource utilization series data, and performance statistics.

[0013] As a further solution of the present invention, the steps for obtaining the high and low entropy task classification results are specifically as follows:

[0014] S111: extracting the computing density value of each task based on the task metadata set, and generating a computing density value result by collecting the task processing time and computing resource utilization rate, and the ratio of the processing time to the resource utilization rate;

[0015] S112: calling the density value calculation result, analyzing the data input and output of the task, calculating the input-output ratio by counting the sum of the input data volume and the output data volume, and generating a data ratio result;

[0016] S113: Based on the calculated density value result and the data ratio result, the number of dependencies of the task is counted, and the number of associated nodes and the hierarchical depth of the task are summarized using the formula:

[0017]

[0018] Calculate the task classification index and generate the joint classification result; among them, is a classification indicator, is the data ratio, To calculate the density value, is the dependent quantity, represents the absolute value of the dependent quantity, represents the square root of the dependent quantity, Represents the total number of tasks;

[0019] S114: Using the joint classification result, by comparing the classification index value with the high and low entropy classification thresholds, the tasks are classified into high and low entropy according to the comparison result to generate a high and low entropy task classification result.

[0020] As a further solution of the present invention, the steps of obtaining the initial task allocation result are specifically as follows:

[0021] S211: Based on the high-entropy and low-entropy task classification results, high-entropy tasks are screened, the dependency quantity and computational density value of each high-entropy task are recorded, and dependency and computational density analysis results are generated by analyzing the resource usage and dependency properties of the tasks;

[0022] S212: Based on the dependency and computational density analysis results, the computational density values are matched with node performance parameters, the most suitable node is captured for each task, and a matching result is generated by calculating the matching degree between the performance requirements of multiple tasks and the node capabilities;

[0023] S213: Based on the matching result, the task allocation strategy is optimized, high entropy tasks are preferentially allocated to nodes with corresponding computing capabilities, and low entropy tasks are allocated based on the remaining resources, using the formula:

[0024]

[0025] Calculate the task allocation index of each node and generate the initial task allocation result;

[0026] in, Assign indicators to the tasks of node i, is the computational density of the task, is the performance parameter of node i, represents the sum of all node performance parameters, To prevent division by zero errors, Represents the total number of nodes, Representative performance parameters of each node.

[0027] As a further solution of the present invention, the step of obtaining the node load adjustment result is specifically as follows:

[0028] S311: Based on the initial task allocation result, calculate the total resource usage of all tasks on each node, and generate the total resource usage of each node by accumulating the GPU, CPU, memory, and storage resources required by the tasks on each node;

[0029] S312: Compare the total resource usage of each node with its total available resource capacity, calculate the resource utilization rate of each node, and obtain a load difference value by the difference between the resource utilization rate and the ideal utilization rate;

[0030] S313: Filter the nodes whose load difference value exceeds the specified range, and adjust the matching of tasks and resources with nodes with sufficient resources, using the formula:

[0031]

[0032] Calculate the node load difference percentage to reflect the imbalance of resource allocation between nodes and generate the node load adjustment result;

[0033] in, For nodes The percentage of load difference, For nodes The actual resource usage rate, is the target resource utilization rate, is a small constant to prevent division by zero errors, Representative Node The number of tasks on .

[0034] As a further solution of the present invention, the step of acquiring the dynamic task allocation state is specifically as follows:

[0035] S411: Based on the node load adjustment result, detecting the dependent target status of the migration task, and generating a task dependency status analysis result by analyzing the dependency relationship of each migration task and the completion status of the target node;

[0036] S412: calling the task dependency status analysis result, marking all unfinished dependent tasks and their nodes, combining the dependent target nodes and the status information of the unsatisfied tasks, and generating an unfinished task status analysis result through dependency status matching logic;

[0037] S413: Using the unfinished task status analysis results, reallocate the unfinished tasks to the corresponding dependent target nodes using the formula:

[0038]

[0039] Calculate task status update indicators and generate dynamic task allocation status;

[0040] in, Indicates a task The state update indicator is used to measure the adaptability of the task to the target node after reallocation. Indicates a task The dependency weight of Indicates a task With the task The strength of dependence, Indicates the available resources of the target node, which is used to evaluate the resource matching of the assigned tasks. A constant to avoid division by zero errors.

[0041] As a further solution of the present invention, the steps of obtaining the task scheduling performance data set are specifically as follows:

[0042] S511: Based on the dynamic task allocation state, the task completion time of each node is counted, and the completion time statistics of each node task are generated by recording the start and end time points of each task and comprehensively referring to the parallelism and sequentiality of multiple tasks;

[0043] S512: Synchronously call the completion time statistics of each node task, calculate the resource utilization of each node, and use the weighted average method to integrate and obtain the resource utilization time series data of each node by real-time monitoring the GPU, CPU, memory, and storage usage of multiple nodes to generate resource utilization statistics.

[0044] S513: Integrate the completion time statistics of each node task and the resource utilization statistics, integrate the task completion time and resource utilization in time series, record the time series data of node task completion and resource utilization series, and use the formula:

[0045]

[0046] Calculate the task scheduling performance indicators of each node and generate a task scheduling performance data set;

[0047] in, Representative Node The comprehensive performance index refers to the deviation between the task completion time and the preset standard, as well as the task weight. It's a task The weight of the task indicates the criticality or urgency of the task. It's a task At the node The actual completion time, It's a task The scheduled completion time provides a benchmark for comparing actual execution time. is to avoid the small amount of zero error introduced, Representative Node The total number of tasks above, used to determine the scope of tasks in performance metric calculations.

[0048] A task scheduling system based on distributed computing, wherein the task scheduling system based on distributed computing is used to execute the above-mentioned task scheduling method based on distributed computing, and the system comprises:

[0049] The task classification module extracts the computational density value of the task based on the task metadata set, calculates the input-output ratio of the data, performs joint operations on the computational density value and ratio, compares the computational results with the number of task dependencies, and generates classification results for high and low entropy tasks.

[0050] The node matching module extracts the dependency quantity and computational density value of the high-entropy task based on the high-entropy and low-entropy task classification results, matches the computational density value of the high-entropy task with the node performance parameters, assigns the high-entropy task to the corresponding node, and the low-entropy task to the remaining nodes, thereby generating an initial task assignment result;

[0051] The load balancing module calculates the total resource usage of multiple node tasks based on the initial task allocation result, compares the total resource usage with the available resource capacity of the nodes, calculates the load difference value, filters out the nodes with the difference value exceeding the range, compares and adjusts the allocation of the tasks of the nodes exceeding the range with the nodes with sufficient resources, and generates a node load adjustment result;

[0052] Based on the node load adjustment result, the dependency adjustment module extracts the migration task dependency target status, marks the unfinished dependent tasks and the nodes where they are located, extracts the dependent target nodes and task status, reallocates the unfinished tasks to the dependent target nodes, and establishes a dynamic task allocation status.

[0053] Compared with the prior art, the advantages and positive effects of the present invention are:

[0054] In this paper, by combining density values with data ratios, we refine task classification, improve allocation accuracy, and avoid resource mismatches. We optimize resource utilization by matching task dependencies with node performance, preventing performance bottlenecks. We dynamically adjust load distribution to balance node resource usage, reducing overload and idleness. We detect dependency status in real time and reallocate unfinished tasks to avoid task delays caused by dependency conflicts. We integrate time series data on task completion times and resource utilization to provide a quantitative basis for optimization, improving the efficiency and adaptability of distributed systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a schematic diagram of the workflow of the present invention;

[0056] Figure 2 Flowchart of the steps for obtaining the classification results of high and low entropy tasks of the present invention;

[0057] Figure 3 A flow chart of the steps for obtaining the initial task assignment result of the present invention;

[0058] Figure 4 Flowchart of the steps for obtaining the node load adjustment result of the present invention;

[0059] Figure 5 A flow chart of the steps for obtaining the dynamic task allocation status of the present invention;

[0060] Figure 6 This is a flow chart of the steps for obtaining a task scheduling performance data set according to the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0062] In the description of the present invention, it should be understood that the terms "length," "width," "up," "down," "front," "back," "left," "right," "vertical," "horizontal," "top," "bottom," "inside," "outside," and the like, indicating positions or relationships, are based on the positions or relationships shown in the accompanying drawings and are intended only to facilitate the description of the present invention and simplify the description. They do not indicate or imply that the devices or elements referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limiting the present invention. Furthermore, in the description of the present invention, "plurality" means two or more, unless otherwise expressly and specifically defined. Example 1

[0063] See also Figure 1The present invention provides a technical solution: a task scheduling method based on distributed computing, comprising the following steps:

[0064] S1: Based on the task metadata set, extract the computational density value of the task, calculate the input-output ratio of the data, detect the number of task dependencies, perform joint calculations based on the computational density value and the data ratio, compare and classify the results with the dependency number, and generate high-entropy and low-entropy task classification results;

[0065] S2: Based on the high-entropy and low-entropy task classification results, the dependency number and computational density value of the high-entropy task are extracted. The computational density value is matched with the node performance parameters item by item. The high-entropy task is assigned to the node with the required computing capacity, and the low-entropy task is assigned to the remaining nodes to generate the initial task assignment result.

[0066] S3: Based on the initial task allocation results, the total resource usage of each node's tasks is calculated. The resource usage is compared with the node's available resource capacity to obtain a load difference value. Nodes with load difference values exceeding a specified range are screened out, and some of their tasks are matched with the capacity of nodes with sufficient resources. Task allocation is then adjusted to generate a node load adjustment result.

[0067] S4: Based on the node load adjustment results, detect the dependent target status of the migration task, mark the unfinished dependent tasks and their nodes, determine the status information of the dependent target nodes and unsatisfied tasks, reallocate the unfinished tasks to the corresponding dependent target nodes, update the task status, and generate a dynamic task allocation status;

[0068] S5: Based on the dynamic task allocation status, count the task completion time and resource utilization of all nodes, integrate the completion time and resource utilization according to the time series, record the time series data of node task completion and resource utilization sequence, and establish a task scheduling performance data set.

[0069] The high-entropy and low-entropy task classification results include high-entropy task classification and low-entropy task classification. The initial task allocation results include the matching results of high-entropy tasks and nodes and the allocation results of low-entropy tasks and nodes. The node load adjustment results include the load difference node list, task migration matching results, and task allocation results after resource adjustment. The dynamic task allocation status includes the status of unfinished dependent tasks, the status of dependent target nodes, and the status of reallocated tasks. The task scheduling performance data set includes the time series data of node task completion, resource utilization series data, and performance statistics.

[0070] See also Figure 2 ,The specific steps for obtaining the classification results of high and low entropy tasks are:

[0071] S111: extracting the computing density value of each task based on the task metadata set, and generating a computing density value result by collecting the task processing time and computing resource utilization rate, and the ratio of the processing time to the resource utilization rate;

[0072] Based on the collection and analysis of task metadata sets, by specifically measuring the processing time and resource utilization of each task, combined with the data monitoring system of the actual operating environment, the computational density value of each task is obtained. This process involves detailed recording of the time and resource consumption of task execution to ensure the accuracy and reliability of the data, thereby accurately calculating the ratio of the computing resource utilization to the processing time of each task; this method can obtain a detailed computational density value result, which can be directly used for the next step of data ratio analysis.

[0073] S112: Call the density calculation result, analyze the data input and output of the task, calculate the input-output ratio by counting the sum of the input data volume and the output data volume, and generate a data ratio result;

[0074] The density value calculation results are called up, and through detailed analysis of the input and output of the task data, especially in a high-performance computing environment, the input and output data are accurately measured and their ratio is calculated. This process includes the processing and analysis of a large number of data points to ensure the accuracy and consistency of the data input and output statistics of each task, and then obtain accurate data ratio results, which will serve as an important basis for task-dependent quantity statistics and subsequent classification.

[0075] S113: Based on the calculated density value and data ratio results, the number of dependencies of the tasks is counted. By summarizing the number of associated nodes and the depth of the hierarchy of the tasks, the formula is used:

[0076]

[0077] Calculate the task classification index and generate the joint classification result; among them, is a classification indicator, is the data ratio, To calculate the density value, is the dependent quantity, represents the absolute value of the dependent quantity, represents the square root of the dependent quantity, Represents the total number of tasks;

[0078] formula: The benefit of the formula is that by weighting the product of the data ratio and the computational density value, and then considering the square root and absolute value of the number of task dependencies, the overall complexity and resource requirements of the task can be evaluated more carefully, thereby effectively classifying the tasks.

[0079] Detailed explanation of the formula and the process of formula calculation and derivation:

[0080] set up , , , the calculation formula is:

[0081]

[0082]

[0083]

[0084] The results show that the task classification index is 1.0, which indicates that these tasks have medium dependencies and computational complexity, and the tasks can be appropriately classified into high and low entropy based on this index.

[0085] S114: Using the joint classification result, by comparing the classification index value with the high and low entropy classification thresholds, the task is classified into high and low entropy according to the comparison result to generate a high and low entropy task classification result.

[0086] Using the joint classification results, the classification indicators are compared in detail with the set high and low entropy classification thresholds. This process includes detailed analysis and evaluation of the classification indicators to ensure that each task can be accurately classified according to its resource requirements and dependencies. Through comparative analysis, the entropy classification of each task can be clarified, providing a basis for resource allocation and task optimization, and generating high and low entropy task classification results, which helps to optimize task scheduling strategies and resource allocation efficiency.

[0087] See also Figure 3 ,The specific steps for obtaining the initial task allocation results are:

[0088] S211: Based on the high-entropy and low-entropy task classification results, high-entropy tasks are screened, the number of dependencies and computational density values of each high-entropy task are recorded, and dependency and computational density analysis results are generated by analyzing the resource usage and dependency properties of the tasks;

[0089] During the screening process of high-entropy tasks, the system analyzes the resource usage and dependency properties of the tasks, records the dependency quantity and computational density value of each task, and determines which tasks have high resource requirements and complex dependency chains by calculating the statistical data of resource usage and task dependencies. These data are obtained through real-time monitoring and historical data analysis, ensuring the accuracy and real-time nature of the data. The system generates dependency and computational density analysis results based on these data. This process involves a large amount of data collection, processing and analysis, ensuring the accuracy of task screening and the rationality of operations.

[0090] S212: Based on the dependency and computational density analysis results, the computational density values are matched with node performance parameters to capture the most suitable node for each task. The matching degree between the performance requirements of multiple tasks and the node capabilities is calculated to generate a matching degree result.

[0091] By calling the dependency and computing density analysis results, the system matches the computing density value with the node performance parameters item by item, and makes a detailed comparison between the computing requirements of each task and the processing power of each node. During this comparison process, multiple performance indicators such as the node's GPU floating-point operation performance, CPU speed, memory size and I / O capabilities are taken into account. These performance data are directly extracted from the node's hardware configuration information and verified through performance test results to ensure the accuracy and practicality of the data. The system optimizes the task allocation strategy based on the comparison results and finds the most suitable node for each task, ensuring maximum resource utilization and improved task execution efficiency. The generated matching results provide an accurate execution guide for actual task allocation.

[0092] S213: Based on the matching results, the task allocation strategy is optimized, high-entropy tasks are preferentially allocated to nodes with corresponding computing capabilities, and low-entropy tasks are allocated based on the remaining resources, using the formula:

[0093]

[0094] Calculate the task allocation index of each node and generate the initial task allocation result;

[0095] in, Assign indicators to the tasks of node i, is the computational density of the task, is the performance parameter of node i, represents the sum of all node performance parameters, To prevent division by zero errors, Represents the total number of nodes, Representative performance parameters of each node.

[0096] formula: The benefit of the formula is that it takes into account the performance parameters of each node and the computational density of the task, by adding a small constant in the denominator Avoid division by zero errors and improve the stability and accuracy of calculations.

[0097] Detailed explanation of the formula and the process of formula calculation and derivation:

[0098] Assume a simplified example where there are three nodes with performance parameters , , , the computational density of the task is , a small constant , the formula becomes:

[0099]

[0100] This result It shows that for node 1, the allocation index of task 1 is 6.62, which reflects the efficiency and fitness of node 1 relative to other nodes in processing task 1.

[0101] The results show that node 1 has higher fitness and efficiency in processing task 1, which helps to optimize the overall task allocation strategy and improve the operating efficiency of the system.

[0102] See also Figure 4 ,The specific steps for obtaining the node load adjustment results are:

[0103] S311: Based on the initial task allocation result, the total resource usage of all tasks on each node is calculated. The total resource usage of each node is generated by accumulating the GPU, CPU, memory, and storage resources required by the tasks on each node.

[0104] Based on the initial task allocation results, the total resource usage of all tasks on each node is obtained by accumulating the GPU, CPU, memory and storage resources required for each task. This calculation process not only involves the total calculation of resources, but also requires detailed analysis and recording of the usage of each resource. First, the resource requirements of each task are collected, and then added one by one to the total amount of the node to ensure the correct aggregation of each type of resource. During this process, the data integrity and accuracy of each resource type must be verified to avoid errors in subsequent resource allocation, ensuring efficient and accurate system resource management, thereby providing accurate basic data for node load balancing.

[0105] S312: Compare the total resource usage of each node with its total available resource capacity, calculate the resource utilization rate of each node, and obtain the load difference value by the difference between the resource utilization rate and the ideal utilization rate;

[0106] Data is obtained from the total resource usage. By comparing the resource usage of each node with its total available resource capacity, the resource utilization rate of each node is calculated. The calculation of resource utilization rate requires accurate measurement of the current resource usage status and total resource capacity of each node. This process involves comparative operations of resource usage and total resource capacity to ensure that each data is processed correctly. The load difference value is obtained by calculating the difference between the resource usage rate and the ideal usage rate. This load difference value is a key indicator for evaluating the operating efficiency of the node, helping administrators understand the resource usage of each node and providing decision support for further resource optimization.

[0107] S313: Filter nodes whose load difference exceeds the specified range and adjust the matching of tasks and resources with nodes with sufficient resources using the formula:

[0108]

[0109] Calculate the node load difference percentage to reflect the imbalance of resource allocation between nodes and generate the node load adjustment result;

[0110] in, For nodes The percentage of load difference, For nodes The actual resource usage rate, is the target resource utilization rate, is a small constant to prevent division by zero errors, Representative Node The number of tasks on .

[0111] formula:

[0112]

[0113] The benefit of the formula is that by introducing the inverse square root operation of the number of node tasks, the calculation model is more sensitive to differences in the number of tasks, making the load difference assessment more accurate. Especially when the number of tasks changes significantly, the calculation of the load difference percentage can be effectively adjusted to make it closer to the actual situation.

[0114] Detailed explanation of the formula and the process of formula calculation and derivation:

[0115] Assume that the actual resource usage of node i is 75%, target utilization rate is 60%, assuming the parameter is 0.1 to avoid the denominator being zero. The number of tasks on node i For 10:

[0116]

[0117]

[0118] The result shows that the load difference percentage of node i is 113.31%, which means that the actual load of node i is much higher than the target load. Task reallocation is needed to reduce the load difference and ensure system operation efficiency and stability.

[0119] See also Figure 5 ,The specific steps for obtaining the dynamic task allocation status are:

[0120] S411: Based on the node load adjustment result, detecting the dependent target status of the migration task, and generating a task dependency status analysis result by analyzing the dependency relationship of each migration task and the completion status of the target node;

[0121] Based on the results of node load adjustment, we start to analyze the dependency status of each task and mark the unfinished dependent tasks. This process involves a detailed review of the existing task status and node load data. First, we extract the current status of each task and the list of dependent tasks from the database, and then compare them with the actual running status of the node. For those dependent tasks that are not completed, we mark these tasks and their current nodes. This process ensures that the tasks that need to be rescheduled can be accurately identified, thereby preparing for the next step of task reallocation. The whole process involves not only data comparison and screening, but also comprehensive consideration of the dependencies between tasks and the capacity of the nodes to ensure that each step is based on the latest and most comprehensive data information to achieve optimal resource utilization and task execution efficiency.

[0122] S412: Call the task dependency status analysis result, mark all unfinished dependent tasks and their nodes, combine the dependent target nodes and the status information of the unsatisfied tasks, and generate the unfinished task status analysis result through the dependency status matching logic;

[0123] After completing the analysis of the task dependency status, the specific status information of the dependent target nodes and unmet tasks was further confirmed. This step is a key link in the task reallocation process. It not only involves a detailed review of the current status of the task, but also includes a performance evaluation of the node where the task is located. A comprehensive evaluation model is used to determine the priority and required resources of each task. The model takes into account the urgency of the task, resource consumption, and the complexity of the dependency relationship, and outputs a scheduling plan with a high degree of matching between tasks and nodes. This plan will serve as the basis for reallocating tasks to ensure maximum utilization of system resources and efficient task execution. This process ensures that all tasks can be executed on the most suitable nodes to achieve optimal resource utilization.

[0124] S413: Using the unfinished task status analysis results, reallocate the unfinished tasks to the corresponding dependent target nodes using the formula:

[0125]

[0126] in, Indicates a task The state update indicator is used to measure the adaptability of the task to the target node after reallocation. Indicates a task The dependency weight of Indicates a task With the task The strength of dependence, Indicates the available resources of the target node, which is used to evaluate the resource matching of the assigned tasks. A constant to avoid division by zero errors.

[0127] formula:

[0128]

[0129] The benefit of the formula is that it allows the system to dynamically adjust task allocation based on task dependencies and the current resource status of the nodes by adding weight parameters and dependency strength ,The formula can more accurately measure the mutual impact between tasks and the ,resource requirements, thereby optimizing the load and performance of the ,whole system.

[0130] Detailed explanation of the formula and the process of formula calculation and derivation:

[0131] Assume that there are three tasks on a node, the dependency weight of task 1 is 0.5, the dependency weight of task 2 is 0.3, and the dependency weight of task 3 is 0.2. The available resources of the node are 10 units. Considering the anti-zero factor of the formula is 0.1, the dependency strengths of tasks 1 to 3 are 0.8, 0.6, and 0.4 respectively. The contribution of each task is calculated as follows:

[0132]

[0133]

[0134]

[0135] The total contribution is

[0136]

[0137] Then the node's task status update indicator for:

[0138]

[0139] The result shows that the node's task status update index is 0.208, which means that the node's resource utilization efficiency is being optimized based on the existing task dependency weights and strengths, which helps improve the node's ability to handle complex tasks, thereby improving the performance and response speed of the entire network.

[0140] See also Figure 6 ,The steps for obtaining the task scheduling performance dataset are as follows:

[0141] S511: Based on the dynamic task allocation status, the task completion time of each node is counted. By recording the start and end time points of each task and comprehensively referring to the parallelism and sequentiality of multiple tasks, the completion time statistics of each node task are generated;

[0142] Based on the statistics of dynamic task allocation status, we begin to record the start and end time of tasks on each node in detail. In this way, we can not only observe the independent running time of each task, but also analyze the dependencies between tasks and their impact on completion time. Furthermore, this monitoring helps optimize the task scheduling algorithm to reduce waiting time and resource waste, ensuring that each node runs under the optimal load. With this data, we can create detailed time logs, which will be used to analyze the execution efficiency of tasks and the performance of nodes.

[0143] S512: Synchronously call the completion time statistics of each node task, calculate the resource utilization of each node, and use the weighted average method to integrate and obtain the resource utilization time series data of each node by real-time monitoring the GPU, CPU, memory, and storage usage of multiple nodes to generate resource utilization statistics.

[0144] Based on the completion time statistics, the resource utilization of each node is monitored simultaneously, and the usage of GPU, CPU, memory and storage resources is recorded in real time. This monitoring includes not only the peak utilization of resources, but also the average utilization of resources. This is crucial for analyzing node performance and determining future resource allocation strategies. For example, if a node's memory utilization is consistently higher than the average level, it may be necessary to consider upgrading the hardware or reallocating some memory-intensive tasks to optimize the performance and efficiency of the overall system. Resource utilization data will be used to assess the health of the node and adjust maintenance plans.

[0145] S513: Integrate the completion time statistics and resource utilization statistics of each node task, integrate the task completion time and resource utilization in time series, record the time series data of node task completion and resource utilization series, and use the formula:

[0146]

[0147] Calculate the task scheduling performance indicators of each node and generate a task scheduling performance data set;

[0148] in, Representative Node The comprehensive performance index refers to the deviation between the task completion time and the preset standard, as well as the task weight. It's a task The weight of the task indicates the criticality or urgency of the task. It's a task At the node The actual completion time, It's a task The scheduled completion time provides a benchmark for comparing actual execution time. is to avoid the small amount of zero error introduced, Representative Node The total number of tasks above, used to determine the scope of tasks in performance metric calculations.

[0149] formula:

[0150]

[0151] The benefit of the formula is that it not only takes into account the completion time of each task, but also can more fairly evaluate the performance of nodes by introducing the weight and scheduled time of tasks, which is applicable to scenarios with a variety of task importance and urgency.

[0152] Detailed explanation of the formula and the process of formula calculation and derivation:

[0153] Setting task weights ,Task At the node Actual completion time Hours, the scheduled completion time of the task Hour, To prevent division by zero errors:

[0154]

[0155] This result shows that the node The performance was good within the given evaluation period and the efficiency of completing the task was higher than the predetermined target, which shows that the task scheduling and resource allocation strategies were relatively reasonable.

[0156] A task scheduling system based on distributed computing, which is used to execute the above-mentioned task scheduling method based on distributed computing, includes:

[0157] The task classification module extracts the computational density value of the task based on the task metadata set, calculates the input-output ratio of the data, performs joint operations on the computational density value and ratio, compares the computational results with the number of task dependencies, and generates classification results for high and low entropy tasks.

[0158] Based on the classification results of high and low entropy tasks, the node matching module extracts the dependency quantity and computational density value of high entropy tasks, matches the computational density value of high entropy tasks with node performance parameters, assigns high entropy tasks to corresponding nodes, and assigns low entropy tasks to remaining nodes, thus generating the initial task assignment results.

[0159] Based on the initial task allocation results, the load balancing module calculates the total resource usage of tasks on multiple nodes, compares the total resource usage with the available resource capacity of the nodes, calculates the load difference value, filters out the nodes with the difference value exceeding the range, and adjusts the allocation of tasks on the nodes exceeding the range with nodes with sufficient resources to generate the node load adjustment result;

[0160] Based on the node load adjustment results, the dependency adjustment module extracts the dependency target status of the migration task, marks the unfinished dependent tasks and the nodes where they are located, extracts the dependency target nodes and task status, reallocates the unfinished tasks to the nodes where the dependency targets are located, and establishes a dynamic task allocation status.

[0161] The above are merely preferred embodiments of the present invention and do not limit the present invention in any other form. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A task scheduling method based on distributed computing, characterized in that: The following steps are involved: Based on the task metadata set, the computational density value of the task is extracted, the input-output ratio of the data is calculated, the number of task dependencies is detected, and the computational density value and the data ratio are combined for joint calculation. The results are compared and classified with the dependency number to generate high and low entropy task classification results; Based on the high-entropy and low-entropy task classification results, the dependency quantity and computational density value of the high-entropy task are extracted, the computational density value is matched with the node performance parameters item by item, the high-entropy task is assigned to the node with the required computing capacity, and the low-entropy task is assigned to the remaining nodes, thereby generating an initial task assignment result; Based on the initial task allocation result, the total resource usage of each node task is calculated, the resource usage is compared with the available resource capacity of the node to obtain a load difference value, nodes with load difference values exceeding a specified range are screened, some of their tasks are matched with the capacity of nodes with sufficient resources, and then the task allocation is adjusted to generate a node load adjustment result; Based on the node load adjustment result, the dependent target status of the migration task is detected, the unfinished dependent tasks and their nodes are marked, the status information of the dependent target nodes and the unsatisfied tasks is determined, the unfinished tasks are reallocated to the nodes where the corresponding dependent targets are located, and the task status is updated to generate a dynamic task allocation status; Based on the dynamic task allocation state, statistics are collected on the completion time and resource utilization of all node tasks, the completion time and resource utilization are integrated in a time series, the time series data of the node task completion and the resource utilization sequence are recorded, and a task scheduling performance data set is established; The high- and low-entropy task classification results include high-entropy task classification and low-entropy task classification. The initial task allocation results include the matching results of high-entropy tasks and nodes and the allocation results of low-entropy tasks and nodes. The node load adjustment results include a list of load difference nodes, task migration matching results, and task allocation results after resource adjustment. The dynamic task allocation status includes the status of unfinished dependent tasks, the status of dependent target nodes, and the status of reallocated tasks. The task scheduling performance data set includes time series data of node task completion, resource utilization series data, and performance statistics.

2. The task scheduling method based on distributed computing according to claim 1, characterized in that: The steps for obtaining the high and low entropy task classification results are specifically as follows: Based on the task metadata set, the computational density value of each task is extracted. By collecting the task's processing time and computing resource utilization rate, and the ratio of processing time to resource utilization rate, the computational density value result is generated. Calling the calculated density value result, analyzing the data input and output of the task, calculating the input-output ratio by counting the sum of the input data volume and the output data volume, and generating a data ratio result; Based on the calculated density value result and the data ratio result, the number of dependencies of the task is counted, and the number of associated nodes and the hierarchical depth of the task are summarized using the formula: Calculate the task classification index and generate the joint classification result; among them, is a classification indicator, is the data ratio, To calculate the density value, is the dependent quantity, represents the absolute value of the dependent quantity, represents the square root of the dependent quantity, Represents the total number of tasks; By using the joint classification result, by comparing the classification index value with the high and low entropy classification thresholds, the tasks are classified into high and low entropy according to the comparison result to generate high and low entropy task classification results.

3. The task scheduling method based on distributed computing according to claim 2, characterized in that: The steps for obtaining the initial task allocation result are specifically as follows: Based on the high-entropy and low-entropy task classification results, high-entropy tasks are screened, the dependency quantity and computational density value of each high-entropy task are recorded, and dependency and computational density analysis results are generated by analyzing the resource usage and dependency properties of the tasks; Based on the dependency and computational density analysis results, the computational density values are matched with node performance parameters to capture the most suitable node for each task. The matching degree between the performance requirements of multiple tasks and the node capabilities is calculated to generate a matching degree result. According to the matching results, the task allocation strategy is optimized, high-entropy tasks are preferentially allocated to nodes with corresponding computing capabilities, and low-entropy tasks are allocated according to the remaining resources. The task allocation index of each node is calculated to generate the initial task allocation result.

4. The task scheduling method based on distributed computing according to claim 3, characterized in that: The steps for obtaining the node load adjustment result are specifically as follows: Based on the initial task allocation result, the total resource usage of all tasks on each node is calculated, and the total resource usage of each node is generated by accumulating the GPU, CPU, memory, and storage resources required by the tasks on each node; Comparing the total resource usage of each node with its total available resource capacity, calculating the resource utilization rate of each node, and obtaining a load difference value by the difference between the resource utilization rate and the ideal utilization rate; Nodes whose load difference values exceed the specified range are screened, and matching adjustments of tasks and resources are made with nodes with abundant resources. The node load difference percentage is calculated to reflect the imbalanced resource allocation between nodes, and a node load adjustment result is generated.

5. The task scheduling method based on distributed computing according to claim 4, characterized in that: The steps for obtaining the dynamic task allocation status are specifically as follows: Based on the node load adjustment result, detecting the dependent target status of the migration task, and generating a task dependency status analysis result by analyzing the dependency relationship of each migration task and the completion status of the target node; Call the task dependency status analysis result, mark all unfinished dependent tasks and their nodes, combine the dependent target nodes and the status information of the unsatisfied tasks, and generate the unfinished task status analysis result through the dependency status matching logic; By using the unfinished task status analysis result, the unfinished tasks are reallocated to the corresponding dependent target nodes, the task status update index is calculated, and the dynamic task allocation status is generated.

6. The task scheduling method based on distributed computing according to claim 5, characterized in that: The steps for obtaining the task scheduling performance data set are specifically as follows: Based on the dynamic task allocation state, the task completion time of each node is counted, and by recording the start and end time points of each task, the completion time statistics of each node task are generated with comprehensive reference to the parallelism and sequentiality of multiple tasks; Synchronously call the completion time statistics of each node task, calculate the resource utilization of each node, and use the weighted average method to integrate and obtain the resource utilization time series data of each node by real-time monitoring of the GPU, CPU, memory and storage usage of multiple nodes to generate resource utilization statistics; Integrate the completion time statistics of each node task and the resource utilization statistics, integrate the task completion time and resource utilization in time series, record the time series data of node task completion and resource utilization series, calculate the task scheduling performance index of each node, and generate a task scheduling performance data set.

7. A task scheduling system based on distributed computing, characterized in that: The distributed computing-based task scheduling method according to any one of claims 1 to 6, wherein the system comprises: The task classification module extracts the computational density value of the task based on the task metadata set, calculates the input-output ratio of the data, performs joint operations on the computational density value and ratio, compares the computational results with the number of task dependencies, and generates classification results for high and low entropy tasks. The node matching module extracts the dependency quantity and computational density value of the high-entropy task based on the high-entropy and low-entropy task classification results, matches the computational density value of the high-entropy task with the node performance parameters, assigns the high-entropy task to the corresponding node, and the low-entropy task to the remaining nodes, thereby generating an initial task assignment result; The load balancing module calculates the total resource usage of multiple node tasks based on the initial task allocation result, compares the total resource usage with the available resource capacity of the nodes, calculates the load difference value, filters out the nodes with the difference value exceeding the range, compares and adjusts the allocation of the tasks of the nodes exceeding the range with the nodes with sufficient resources, and generates a node load adjustment result; Based on the node load adjustment result, the dependency adjustment module extracts the migration task dependency target status, marks the unfinished dependent tasks and the nodes where they are located, extracts the dependent target nodes and task status, reallocates the unfinished tasks to the dependent target nodes, and establishes a dynamic task allocation status.

Images