Intelligent scheduling method and device for scheduling platform

Abstract

The invention provides an intelligent scheduling method and device for a scheduling platform. The method comprises the following steps: evaluating the task value of each task by using a value evaluation model; evaluating resource consumption during task execution through a resource evaluation model on the basis of resource requirements during task execution, evaluating the total amount of all resources, and collecting uncompleted tasks; building a backpack problem model based on the task value, the resource consumption, the total amount of all resources and the uncompleted tasks of each task; and solving the knapsack problem model in combination with the scheduling score card to obtain an optimal task combination, and allocating the tasks to corresponding resources or execution units for execution according to the optimal task combination. The dependency, the weight and the resource consumption sequence among the tasks are optimized by automatically analyzing the task values of the tasks; and calculating an optimal task combination in real time by adopting a knapsack algorithm model according to the scheduling progress and the server resource remaining condition. And dynamic changes of task characteristics and resource consumption are fully considered, so that the efficiency and the stability of the system are improved.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to an intelligent scheduling method and device for a scheduling platform. Background Art

[0002] With the continuous advancement of information technology, scheduling platforms play a vital role in modern computing environments. Their core mission is to rationally allocate computing resources and ensure the efficient completion of tasks. However, existing scheduling systems suffer from significant flaws that limit their ability to efficiently allocate resources and manage tasks.

[0003] Currently, many scheduling platforms use traditional scheduling algorithms, such as First-Come, First-Served (FCFS), Shortest Job First (SJF), and high-priority scheduling. However, these algorithms often ignore task execution duration, resource consumption, and the relationship between tasks, resulting in poor scheduling flexibility and adaptability. Traditional scheduling algorithms are often unable to effectively respond to external load fluctuations or peak resource consumption periods, resulting in some tasks not being processed in a timely manner, which in turn affects the efficiency and stability of the entire system.

[0004] Based on this, traditional scheduling algorithms fail to fully consider the dynamic changes of task characteristics and resource consumption, ignore task execution time, resource consumption and task lineage, and lack flexibility in the face of load fluctuations and peak periods, thus affecting system efficiency and stability. Summary of the Invention

[0005] In view of this, an embodiment of the present invention provides an intelligent scheduling method and device for a scheduling platform to solve the problem of low task scheduling efficiency of the scheduling platform.

[0006] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0007] A first aspect of the present invention discloses an intelligent scheduling method for a scheduling platform, the method comprising:

[0008] For each task, using a value assessment model to assess the task value of the task; the value assessment model is pre-trained based on historical task data;

[0009] Evaluating resource consumption during task execution based on resource requirements during task execution using a resource evaluation model; the resource evaluation model is pre-trained based on the historical task data;

[0010] Evaluate the total amount of all resources through the resource evaluation model and collect unfinished tasks from the scheduling platform;

[0011] Constructing a knapsack problem model based on the task value of each task, the resource consumption of each task, the total amount of all resources and the unfinished tasks;

[0012] The knapsack problem model is solved in conjunction with a scheduling scorecard to obtain an optimal task combination, and tasks are assigned to corresponding resources or execution units for execution based on the optimal task combination. The scheduling scorecard is generated in advance.

[0013] Preferably, for each task, evaluating the task value of the task using a value evaluation model includes:

[0014] For each task, use the value assessment model to obtain the key factors of the task;

[0015] Convert the key factors into quantitative indicators, scale each key factor using a pairwise comparison matrix, and obtain the relative importance of each key factor;

[0016] The relative importance of each of the key factors is standardized and normalized to obtain the task value of the task.

[0017] Preferably, evaluating resource consumption during task execution based on resource requirements during task execution using a resource evaluation model includes:

[0018] Extracting characteristic data related to resource consumption during task execution and performing preprocessing to obtain resource demand data during task execution;

[0019] estimating the execution time of the task based on the resource requirement data;

[0020] Obtain the resource utilization of tasks from the core scheduling service module of the scheduling platform and the software and hardware computing service modules of the resource cluster;

[0021] Obtain the historical response time and historical success rate of tasks from the historical log module of the scheduling platform;

[0022] The resource consumption during the execution of the task is evaluated based on the resource demand data, the execution time, the resource utilization rate, the historical response time and the historical success rate through a resource evaluation model.

[0023] Preferably, the process of generating the scheduling resource scorecard includes:

[0024] Get the historical batch logs of the task;

[0025] Analyzing the impact factor metric and weighted metric value of each task based on the historical batch run log and the resource requirement type of the task through the resource evaluation model;

[0026] Calculate the impact factor weight of each task and the number of all tasks;

[0027] Evaluate the total amount of all resources using the resource evaluation model;

[0028] A scheduling scorecard is generated based on the impact factor metrics, the impact factor weights, the weighted metric values, the total amount of all resources, and the number of all tasks.

[0029] Preferably, the evaluating the total amount of all resources by the resource evaluation model includes:

[0030] The resource evaluation model is used to obtain the total amount of CPU resources, memory resources, storage resources and network bandwidth resources in the scheduling platform to obtain the total amount of all resources.

[0031] Preferably, the knapsack problem model is constructed based on the task value of each task, the resource consumption of each task, the total amount of all resources and the unfinished tasks, including:

[0032] Defining parameters of a knapsack problem model based on the task value of each task, the resource consumption of each task, the total amount of all resources, and the unfinished tasks;

[0033] Define decision variables based on all tasks;

[0034] determining resource utilization and task completion time based on the task value of each task, the resource consumption of each task, and the total amount of all resources to establish an objective function;

[0035] Determining constraints based on the total amount of all resources;

[0036] A knapsack problem model is constructed based on the parameters, the decision variables, the objective function and the constraint conditions.

[0037] Preferably, after solving the knapsack problem model in combination with the scheduling scorecard to obtain the optimal task combination, and allocating the tasks to corresponding resources or execution units for execution according to the optimal task combination, the method further includes:

[0038] Obtain real-time status information of resource usage and task scheduling;

[0039] generating resource usage trend information based on the real-time status information of resource usage;

[0040] The real-time status information of the resource usage, the status information of the task scheduling and the usage trend information of the resource are displayed through a visual interface.

[0041] A second aspect of the present invention discloses an intelligent scheduling device for a scheduling platform, the device comprising:

[0042] A value evaluation unit, configured to evaluate the task value of each task using a value evaluation model; the value evaluation model is pre-trained based on historical task data;

[0043] a resource consumption evaluation unit, configured to evaluate resource consumption during task execution based on resource requirements during task execution using a resource evaluation model; the resource evaluation model is pre-trained based on the historical task data;

[0044] A collection unit, configured to evaluate the total amount of all resources using the resource evaluation model and collect unfinished tasks from the scheduling platform;

[0045] A model building unit is used to build a knapsack problem model based on the task value of each task, the resource consumption of each task, the total amount of all resources and the unfinished tasks;

[0046] A solving unit is used to solve the knapsack problem model in combination with a scheduling scorecard to obtain an optimal task combination, and to allocate tasks to corresponding resources or execution units for execution according to the optimal task combination. The scheduling scorecard is generated in advance.

[0047] Preferably, the value assessment unit includes:

[0048] The first acquisition module is used to obtain the key factors of each task using the value assessment model;

[0049] A scaling module is used to convert the key factors into quantitative indicators, scale each key factor through a pairwise comparison matrix, and obtain the relative importance of each key factor;

[0050] The processing module is used to standardize and normalize the relative importance of each of the key factors to obtain the task value of the task.

[0051] Preferably, the resource consumption assessment unit includes:

[0052] An extraction module is used to extract characteristic data related to resource consumption during the execution of the task and perform preprocessing to obtain resource demand data during the execution of the task;

[0053] an estimation module, configured to estimate the execution time of a task based on the resource requirement data;

[0054] The calculation module is used to obtain the resource utilization of tasks from the core scheduling service module of the scheduling platform and the software and hardware computing service modules of the resource cluster;

[0055] The second acquisition module is used to obtain the historical response time and historical success rate of the task from the historical log module of the scheduling platform;

[0056] An evaluation module is used to evaluate the resource consumption during the execution of the task based on the resource demand data, the execution time, the resource utilization rate, the historical response time and the historical success rate through a resource evaluation model.

[0057] Based on the above-mentioned embodiments of the present invention, an intelligent scheduling method and device for a scheduling platform are provided. This method aims to achieve intelligent management and optimized allocation of resources. By analyzing task lineage and execution duration, task priority determination and resource allocation can be more accurately performed. Simultaneously, based on real-time operational data and changes in external factors, the scheduling strategy is dynamically adjusted online using a knapsack algorithm, effectively avoiding excessive resource consumption or task scheduling delays, ensuring that tasks are executed on time and efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0059] Figure 1 A flowchart of an intelligent scheduling method for a scheduling platform provided in an embodiment of the present invention;

[0060] Figure 2 An example diagram of the level of factor metrics provided by an embodiment of the present invention;

[0061] Figure 3 A schematic diagram of a calculation process for calculating impact factor weights provided by an embodiment of the present invention;

[0062] Figure 4 This is a structural block diagram of an intelligent scheduling device for a scheduling platform provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0064] In this application, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0065] As we've seen from the background, traditional scheduling algorithms (such as FCFS, SJF, and high-priority scheduling) have limitations. These primarily ignore task execution duration, resource consumption, and inter-task dependencies, resulting in poor scheduling flexibility. Furthermore, these algorithms lack dynamic adjustment capabilities and are unable to effectively respond to external load fluctuations or resource consumption spikes, which in turn impacts system efficiency and stability, causing some tasks to be delayed.

[0066] Therefore, an embodiment of the present invention provides an intelligent scheduling method and device for a scheduling platform. For each task, a value evaluation model is used to evaluate the task value; a resource evaluation model is used to evaluate the resource consumption during task execution based on the resource requirements during task execution; the total amount of all resources is evaluated through the resource evaluation model, and unfinished tasks are collected from the scheduling platform; a knapsack problem model is constructed based on the task value of each task, the resource consumption of each task, the total amount of all resources, and the unfinished tasks; the knapsack problem model is solved in combination with a scheduling score card to obtain the optimal task combination, and tasks are assigned to corresponding resources or execution units for execution based on the optimal task combination. By automatically analyzing the blood relationship and execution time of tasks, the dependencies, weights, and resource consumption order between tasks are optimized; at the same time, the knapsack algorithm model is used to calculate the optimal task combination in real time based on the scheduling progress and the remaining server resources. This method fully considers the dynamic changes of task characteristics and resource consumption, thereby improving the efficiency and stability of the system.

[0067] See also Figure 1 , which shows a flow chart of an intelligent scheduling method for a scheduling platform provided by an embodiment of the present invention. The method is specifically an intelligent scheduling method based on blood relationship, including:

[0068] Step S101: For each task, the task value of the task is evaluated using a value evaluation model.

[0069] It is understandable that the primary core of intelligent scheduling is to establish a value assessment model. In the embodiment of the present application, the value assessment model is pre-trained based on historical task data.

[0070] It's important to note that the value assessment model is value-driven and is used to determine the relative priority of tasks in the scheduling queue. The model continuously monitors task changes and evaluates task value based on a variety of factors. Given limited resources, it ensures efficient and reasonable scheduling based on task value, prioritizing high-value, critical tasks.

[0071] In the specific implementation of step S101 , for each task in the scheduling platform, the task value of each task is evaluated using a value evaluation model.

[0072] Specifically, as shown in process A1 to process A3:

[0073] Process A1: For each task, use the value assessment model to obtain the key factors of the task.

[0074] It should be noted that the key factors in scheduling tasks include the current task itself and the factors that influence the current task. These two factors together form the core basis for scheduling decisions and ensure accurate assessment of task value.

[0075] It can be understood that the current task's own factors are specifically factors for calculating the task's own value, including five factors: importance, deadline type, running time, progress deviation, and failure probability.

[0076] These five factors are introduced below:

[0077] 1. Importance

[0078] The importance factor is pre-defined by the user as to the criticality of the task in the business process.

[0079] 2. Term Type

[0080] Term type factors include hard term and soft term.

[0081] It is understandable that scheduling tasks can be divided into hard-deadline tasks and soft-deadline tasks according to the deadline requirements and delay sensitivity of the tasks.

[0082] Hard deadlines have a clear and strict deadline, often determined by business requirements or the time constraints of critical business processes. Failure to meet these deadlines can have serious consequences. Therefore, hard deadlines are highly sensitive to latency and must be completed before the deadline, otherwise significant problems may arise.

[0083] While soft-deadline tasks also have completion time requirements, these deadlines aren't fixed; they're derived from analyzing historical operational data. Therefore, soft-deadline tasks offer more flexibility, allowing for some delay. While they should strive to be completed on time, they can be more relaxed than hard-deadline tasks.

[0084] 3. Runtime

[0085] The runtime factor is an important basis for predicting tasks based on historical logs.

[0086] It's understandable that within a set of dependent tasks, task runtimes help prioritize execution, preventing excessively long predecessor tasks from causing subsequent tasks to wait too long, or short successor tasks from wasting waiting time. By factoring task runtimes into scheduling, we can better balance bottlenecks within the workflow, optimizing overall process efficiency.

[0087] 4. Schedule deviation

[0088] The progress deviation factor refers to the difference between the estimated running time of the task and the remaining time (that is, the difference between the completion time and the current time), which reflects the situation of the task execution process being ahead of or behind the expected progress.

[0089] It should be noted that the progress deviation can be a positive value (indicating that the task progress is behind schedule), a negative value (indicating that the task progress is ahead of schedule), or zero (indicating that the task is executed according to schedule).

[0090] During the application process, if the remaining time is insufficient to complete the task, hard-deadline tasks should be prioritized to minimize delays; while soft-deadline tasks may need to have their priority lowered and postponed to the next appropriate time window.

[0091] 5. Failure probability

[0092] The failure probability factor refers to the possibility of failure during the execution of a task, as well as the negative impacts such as system interruption, data loss, and task delay that may be caused.

[0093] The influencing factors of a current task refer to the impact of each task's characteristics on all downstream tasks that are directly or indirectly dependent on it. These factors include the estimated total runtime of downstream tasks, the proportion of downstream tasks to the total number of tasks, the average value of downstream tasks, and the proportion of downstream tasks with hard deadlines.

[0094] These four factors are introduced below:

[0095] 1. Estimation of the total running time of downstream tasks

[0096] The total running time estimate of downstream branches refers to the total time required to complete all subsequent tasks (i.e., downstream branches) that are directly or indirectly dependent on the current scheduled task after it is completed.

[0097] Understandably, this estimate helps identify the critical path in the task chain—the path in the entire workflow that takes the longest to complete. The scheduler can then prioritize tasks on the critical path to shorten the overall completion time.

[0098] It's important to note that if the downstream branch of the current task has a longer estimated total duration than other branches, this may indicate a potential bottleneck in that branch. In this case, the scheduler can adjust the execution order of the current task to trigger the execution of the bottleneck branch in advance, thereby avoiding impacting the progress of the entire workflow.

[0099] 2. Proportion of Downstream Tasks

[0100] The proportion of downstream tasks to the total number refers to the number of subsequent tasks that the current task directly or indirectly depends on, as a proportion of the total number of global tasks, reflecting the potential impact of the task.

[0101] It's understandable that the higher the proportion of tasks with branching, the more likely that task will have a greater impact on the completion time and resource requirements of the entire workflow. Based on the number of branching tasks, you can set an appropriate priority for the current task, prioritizing those with many branching tasks and the greatest impact on the overall process.

[0102] By combining the proportion of branch tasks and the estimated runtime of each branch task, we can more accurately identify the critical path in the workflow and prioritize the tasks on the critical path, thereby helping to shorten the completion time of the entire process.

[0103] 3. Average value of downstream tasks

[0104] The average value of downstream tasks refers to the arithmetic mean of the values ​​of all subsequent tasks that the current task directly or indirectly depends on, reflecting the contribution of the task to the overall task chain.

[0105] It's important to note that if the average value is high, it means that completing the current task is crucial to achieving a large number of high-value subsequent tasks. Therefore, these high-value branch tasks should be prioritized to ensure that high-value work is completed as soon as possible, thereby maximizing organizational benefits. Conversely, if the average value is low, it may indicate that the current task has little impact on the overall task chain, and its priority can be appropriately lowered.

[0106] 4. Proportion of downstream hard-deadline tasks

[0107] The proportion of downstream hard-deadline tasks refers to the ratio of tasks with hard deadlines, among all subsequent tasks that the current task directly or indirectly depends on, to the total number of hard-deadline tasks globally. This ratio helps the scheduling system more accurately assess the importance of the current task.

[0108] It should be noted that if the ratio is high, it means that the completion of the current task is critical to meeting the deadlines of a large number of hard-deadline tasks, and therefore it should be given a higher priority and value. Conversely, if the ratio is low, the current task has little impact on the hard-deadline tasks, and its priority can be appropriately lowered.

[0109] Process A2: Convert key factors into quantitative indicators, scale each key factor through a pairwise comparison matrix, and obtain the relative importance of each key factor.

[0110] When implementing process A2, the current task's inherent factors and factors influencing the current task, namely importance, deadline type, runtime, schedule deviation, and failure probability, as well as the estimated total runtime of downstream tasks, the proportion of the total number of downstream tasks, the average value of downstream tasks, and the proportion of downstream hard-deadline tasks, are converted into computable quantitative indicators. All factors are then put together and compared pairwise using a pairwise comparison matrix to scale each key factor and determine its relative importance.

[0111] It should be noted that when converting all factors into quantifiable indicators, the measurement values ​​are in the range of 10 to 100, with larger numbers indicating greater importance. This is shown in Tables 1 to 3 below.

[0112] Table 1

[0113]

[0114] The self-factors are shown in Table 2.

[0115] Table 2

[0116]

[0117] The influencing factors are shown in Table 3.

[0118] Table 3

[0119]

[0120] It can be understood that after these non-quantitative factors are converted into calculable quantitative indicators, they are compared pairwise through a pairwise comparison matrix to scale each key factor and obtain the relative importance of each key factor.

[0121] It should be noted that the pairwise comparison matrix represents the relative importance of all factors at one level relative to a factor (such as a criterion or goal) at the previous level. The comparison results are quantified using the 1-9 scale proposed by Santy. See Table 4 for details.

[0122] Table 4

[0123]

[0124] Tables 5 to 7 below illustrate how to use the pairwise comparison matrix to compare two key factors, scale each key factor, and obtain the relative importance of each key factor.

[0125] Table 5

[0126]

[0127] Table 6

[0128]

[0129] Table 7

[0130]

[0131] Process A3: Standardize and normalize the relative importance of each key factor to obtain the task value.

[0132] When implementing process A3, the relative importance of each key factor (such as the comparison matrix listed in Tables 5 and 6 above) is normalized using a standardized general model, and the relative weights of each decision factor are adjusted in real time to reflect the relative importance of the task in the value assessment, ultimately deriving the total value of the current task.

[0133] The relative weights are shown in Table 8.

[0134] Table 8

[0135]

[0136] The task value examples of the tasks are shown in Table 9.

[0137] Table 9

[0138]

[0139] Step S102: Evaluate resource consumption during task execution based on resource requirements during task execution using a resource evaluation model.

[0140] In the embodiment of the present application, the resource assessment model is pre-trained based on historical task data. By analyzing the patterns in the historical task data, the machine learning method can predict the resource requirements during the execution of new tasks and thus assess their resource consumption.

[0141] In the specific implementation of step S102 , resource demand data during task execution is extracted through a resource evaluation model to clarify resource demand, thereby evaluating resource consumption during task execution.

[0142] Specifically, the evaluation process includes process B1 to process B7.

[0143] Process B1: Extract characteristic data related to resource consumption during task execution and perform preprocessing to obtain resource demand data during task execution.

[0144] When implementing process B1, characteristic data related to resource consumption during task execution is extracted, including but not limited to: the type of task and its parameters, the size of the task's input data, the task's execution history, the task's priority or service level agreement (SLA) requirements, the current state of the system (such as system load, time (peak or off-peak hours)), and the dependencies between tasks.

[0145] The extracted feature data is cleaned, missing values ​​are processed, and some continuous values ​​are normalized or feature encoded to obtain resource requirement data during task execution.

[0146] In some embodiments, to manage resources more precisely, the relationship between different types of tasks and resource usage can be analyzed, for example, identifying memory-intensive tasks or I / O-intensive tasks, and optimizing resource allocation based on these characteristics.

[0147] Process B2: Estimate the execution time of the task based on the resource requirement data.

[0148] When implementing process B2, the execution order and execution time of tasks are estimated based on the resource demand data and the task dependencies, which helps identify the tasks on the critical path.

[0149] Process B3: Obtain the resource utilization of the task from the core scheduling service module of the scheduling platform and the software and hardware computing service modules of the resource cluster.

[0150] It should be noted that the core scheduling service module is generally responsible for managing and scheduling task execution, while the scheduling platform generally tracks the resource allocation of tasks. It can collect and provide feedback on the system resources used by tasks during execution (such as CPU, memory, I / O bandwidth, etc.) as well as the execution status of the tasks.

[0151] The hardware and software computing service modules are usually responsible for the actual allocation and management of computing resources, including the resource usage of computing nodes (for example, CPU cores, memory, storage, etc.).

[0152] It is understandable that resource utilization includes but is not limited to the task's CPU utilization, memory utilization, network bandwidth utilization, etc. These data can reflect the task's demand for resources and resource utilization efficiency within a specific time.

[0153] In specific implementations, by comparing resource utilization in different time periods or under different tasks, the performance of the platform in different scenarios can be evaluated and potential optimization opportunities can be identified.

[0154] Process B4: Obtain the historical response time and historical success rate of the task from the historical log module of the scheduling platform.

[0155] In the specific implementation process B4, the performance of task scheduling can be evaluated by obtaining the historical response time and success rate of the task from the historical log module of the scheduling platform.

[0156] As one of the core functions of the platform, task scheduling efficiency directly impacts resource utilization. Analyzing task scheduling metrics such as response time, success rate, and execution time helps evaluate the platform's performance in task scheduling.

[0157] Process B5: Evaluate the resource consumption during task execution based on resource demand data, execution time, resource utilization, historical response time, and historical success rate through the resource evaluation model.

[0158] When implementing process B5, the resource demand data, execution time, resource utilization, historical response time, and historical success rate are analyzed using statistical methods through the resource evaluation model, such as calculating the average value and standard deviation, to evaluate the resource consumption during task execution.

[0159] Step S103: Evaluate the total amount of all resources through the resource evaluation model, and collect unfinished tasks from the scheduling platform.

[0160] In the specific implementation of step S103, the total amount of CPU resources, memory resources, storage resources and network bandwidth resources in the scheduling platform is obtained through the resource evaluation model to obtain the total amount of all resources; secondly, unfinished tasks are collected from the scheduling platform.

[0161] It's important to note that unfinished tasks are also in-progress tasks. These tasks may be urgently in need of reconsideration or rejected tasks. Urgently in need of reconsideration tasks require urgent reconsideration, perhaps due to new circumstances or issues. Rejected tasks are tasks that have been rejected or dismissed and are no longer being executed.

[0162] For unfinished tasks, it is important to pay attention to the number of errors that occurred during the execution of the task; and whether the task is on the critical path of the project, which is crucial to the project schedule and success.

[0163] As you can see, these statuses are used to monitor and manage the progress of scheduled tasks, ensuring they are completed on time and that issues are addressed promptly. This kind of status tracking is crucial for project management and resource scheduling, helping teams understand the current status of tasks and make adjustments and decisions accordingly.

[0164] Step S104: Construct a knapsack problem model based on the task value of each task, the resource consumption of each task, the total amount of all resources and unfinished tasks.

[0165] In the specific implementation of step S104, a knapsack problem model is constructed based on the task value of each task, the resource consumption of each task, the total amount of all resources, and unfinished tasks. The specific construction process is as follows (process C1 to process C5):

[0166] Process C1: Define the parameters of the knapsack problem model based on the task value of each task, the resource consumption of each task, the total amount of all resources, and the number of unfinished tasks.

[0167] In the specific implementation process C1, the problem parameters of the knapsack problem model are defined based on the task value of each task, the resource consumption of each task, the total amount of all resources and unfinished tasks, including the definition of resources and tasks, resource capacity and task resource occupancy and value.

[0168] It can be understood that 1. Resource definition: resources can be CPU, memory, storage, etc., and tasks are jobs or job processes that need to be executed. The number of tasks is n, that is, there are n tasks. 2. Resource capacity definition: the maximum available amount of scheduling resources, also known as the capacity of scheduling resources, denoted as C, which is equivalent to the capacity of a backpack. 3. Task resource occupancy and value definition: The demand and value of each task for resources, including CPU usage, memory consumption, runtime, urgency, critical path, and error rate, determine the weight of the task. The runtime and resource usage of a task can be regarded as the capacity of an item w. i (i=1, 2, ..., n), the weight of the task represents the value of the item v i (i=1, 2, ..., n).

[0169] With these definitions, the relationship between tasks and resources can be optimized and scheduled using the knapsack problem model, ensuring that resources are used rationally while meeting the priorities and requirements of tasks.

[0170] Process C2: Define decision variables based on all tasks.

[0171] When implementing process C2, for each task, a binary variable is used to indicate whether it is scheduled to a certain resource, which is similar to the decision variable in the knapsack problem.

[0172] For example, for the i-th task (item):

[0173] x i =1: Indicates that the task is scheduled for execution, that is, the task is "put into the backpack".

[0174] x i =0: Indicates that the task is in a waiting state, that is, the task is "not put into the backpack".

[0175] This approach can help determine which tasks should be executed and which tasks should wait during the scheduling process, thereby achieving optimal allocation of resources.

[0176] Process C3: Determine resource utilization and task completion time based on the task value of each task, the resource consumption of each task, and the total amount of all resources to establish an objective function.

[0177] It should be noted that the objective function is defined as follows: Generally, the goal is to maximize the overall efficiency or performance of the system, which can be achieved by considering indicators such as resource utilization and task completion time.

[0178] In this case, the objective function can be expressed as the sum of the task values ​​(as shown in Equation (1)):

[0179] (1)

[0180] In formula (1), x i is a decision variable, indicating whether the i-th task is scheduled (1 for scheduling, 0 for not scheduling); v i is the value or weight of the ith task. n is the number of tasks.

[0181] The goal of maximizing the total value is to maximize the total value brought by the scheduled tasks. As shown in formula (2):

[0182] (2)

[0183] In formula (2), the meanings of the parameters are the same as those in formula (1) and will not be repeated here.

[0184] Process C4: Determine the constraints based on the total amount of all resources.

[0185] It can be understood that the capacity constraint is as shown in formula (3).

[0186] (3)

[0187] In formula (3), C is the capacity of the scheduling resource. i (i=1, 2, ..., n) is the capacity of the item, that is, the running time and resource usage of the task. iis a decision variable indicating whether the i-th task is scheduled (1 for scheduled, 0 for not scheduled). n is the number of tasks.

[0188] Process C5: Construct a knapsack problem model based on parameters, decision variables, objective function and constraints.

[0189] When specifically implementing process C5, the parameters, decision variables, objective function and constraints are integrated into a 0-1 knapsack problem model.

[0190] Step S105: Solve the knapsack problem model to obtain the optimal task combination, and assign the tasks to corresponding resources or execution units for execution according to the optimal task combination.

[0191] During step S105, a knapsack problem-solving algorithm (such as integer programming solvers like CPLEX and Gurobi, or a dynamic programming algorithm) is used to determine how to maximize total value within resource constraints. This process incorporates the scoring mechanism and weights defined in the scheduling scorecard to ensure the optimal task combination is selected. Based on this optimal task combination, tasks are then assigned to the appropriate resources or execution units for execution.

[0192] It should be noted that the scheduling resource scorecard is generated in advance, and the specific generation process is as follows (process D1 to process D5).

[0193] Process D1: Get the historical batch log of the task.

[0194] When implementing process D1, the historical batch logs of the tasks are obtained from the scheduling platform.

[0195] Process D2: Analyze the impact factor metrics and weighted metrics of each task based on historical batch logs and the resource requirement type of the task through the resource evaluation model.

[0196] It should be noted that the resource requirement type of a task is specifically obtained by collecting and analyzing configuration information such as the association between the task and resources, runtime, input and output parameters, etc.

[0197] When implementing process D2, the evaluation model is used to analyze the impact factor measurement and impact factor weight of each task based on historical batch logs and the resource requirement type of the task.

[0198] It should be noted that the impact factor measurement of each task includes but is not limited to factor measurements such as computing resources CPU, computing resources IO, computing resources memory, network bandwidth, and local and network storage. The level of factor measurement is pre-set by the user and divided into 3 levels (such as low, medium and high). For example, in the embodiment of the present invention, Figure 2 As shown, different levels correspond to different numbers.

[0199] It can be understood that the weighted metric of each task (Weighted Metric) = Factor 1 Metric * Factor 1 Weight + Factor 2 Metric * Factor 2 Weight + Factor 3 Metric * Factor 3 Weight + Factor 4 Metric * Factor 4 Weight + Factor 5 Metric * Factor 5 Weight.

[0200] Process D3: Calculate the impact factor weight of each task and the number of all tasks.

[0201] When specifically implementing process D3, the impact factor weight is calculated based on the weighted measurement value of each task, and the number of all tasks is calculated.

[0202] In practical applications, the number of all valid and ready tasks during the current scheduling period is calculated, excluding tasks in the completed and executing states, to obtain the total number of tasks n.

[0203] Specifically, the calculation process of calculating the weight of the impact factor according to the weighted measurement value is as follows: Figure 3 The specific calculation formula is shown in formula (4).

[0204] (4)

[0205] In formula (4), W i represents the occupancy weight of task i, that is, the weighted measurement value; i represents the task, i=1, 2, 3…n; j represents all the influencing factors of task i, j=1, 2, 3…n.

[0206] Represents the weighted measurement value of all factors of task i.

[0207] Represents the sum of the weighted metrics for all tasks.

[0208] Represents the weighted measure of factor j for task i.

[0209] It can be understood that referring to Table 10, each impact factor metric and impact factor weight of each task is exemplified.

[0210] Table 10

[0211]

[0212] In some embodiments, for each task, its corresponding resource occupation weight is also calculated.

[0213] Specifically, the task resource occupancy weight = the weighted measurement values ​​corresponding to the task resource / the sum of the weighted measurement values ​​of the factors corresponding to the task resource.

[0214] The calculation results are shown in Table 11.

[0215] Table 11

[0216]

[0217] Process D4: Evaluate the total amount of all resources through the resource evaluation model.

[0218] In the specific implementation process D4, the total amount of all resources managed by the scheduling platform is obtained through the resource evaluation model.

[0219] It should be noted that for different types of resources (such as CPU, memory, storage, network bandwidth, etc.), it is necessary to query the resource specifications of all nodes in the cluster, including:

[0220] CPU: Get the number of cores (logical cores) and clock frequency of each node.

[0221] Memory: Gets the total memory size of each node.

[0222] Storage: Accumulates the storage space of each node.

[0223] Network: Get the network bandwidth within the cluster.

[0224] In actual application, the resource evaluation model can be used to calculate the overall remaining resources for scheduling based on the total amount of all resources.

[0225] It's important to note that the scheduling platform's resource utilization analysis covers all resources in idle, unsaturated, and saturated states, including CPU utilization, memory utilization, and network bandwidth utilization. By comparing resource utilization across different time periods or tasks, we can evaluate the platform's performance in different scenarios and identify potential areas for optimization. Resource utilization is categorized into three levels: high (81%-100%), medium (51%-80%), and low (0%-50%), using real-time resource usage data from each node as the metric.

[0226] According to the plank effect, the resource utilization of the scheduling platform is limited by the weakest resource, rather than the sum of all resource utilizations. Therefore, the calculation formula for the overall resource utilization is shown in formula (5):

[0227] Total resource utilization = MAX(factor 1 utilization, factor 2 utilization, factor 3 utilization, factor 4 utilization, factor 5 utilization) (5)

[0228] Based on this, the calculation formula for scheduling available resources is shown in formula (6):

[0229] Scheduling available resources = total scheduling resources * (1-overall resource utilization) (6)

[0230] It can be understood that the overall resource occupancy measurement value is as shown in Table 12.

[0231] Table 12

[0232]

[0233] As shown in the example in Table 12, during period 1, the system's total available resource capacity is set to support a maximum of 750 concurrent tasks, ensuring that maximum resource utilization does not exceed 85%. To improve system availability and reliability, 15% of redundant resources must be reserved for backup. Furthermore, if the system supports dynamic scaling, it can rapidly expand capacity as demand increases, thus tolerating higher resource utilization.

[0234] Process D5: Generate a scheduling scorecard based on the impact factor metrics, impact factor weights, weighted metrics, the total amount of all resources, and the number of all tasks.

[0235] It is understandable that in actual applications, new scheduling scorecards can be regenerated regularly or irregularly, and the knapsack problem model can be solved using the new scheduling scorecards to obtain a new optimal task combination. The optimized optimal task combination is then passed to the scheduling platform to continue executing tasks until the task completion rate reaches 100%.

[0236] It should be noted that the practice of generating new scheduling scorecards regularly or irregularly helps to continuously optimize the scheduling strategy through feedback loops, ensuring that tasks can be executed more efficiently.

[0237] It is understandable that the knapsack problem model can be flexibly adjusted and optimized according to the characteristics of the specific problem, such as selecting appropriate algorithm variants, adjusting parameters, or introducing constraints to better adapt to different application scenarios. The algorithm is easy to understand and implement. Through dynamic programming, the execution process and decision-making principle of each step can be clearly described, which is convenient for analysis and optimization. By calculating the maximum value of selecting the first i items under "different knapsack spaces", a large number of invalid searches are avoided, thereby reducing the complexity to N 2 .

[0238] In some preferred embodiments, real-time resource usage status information and task scheduling status information are obtained. Resource usage trend information is generated based on the real-time resource usage status information. The real-time resource usage status information, task scheduling status information, and resource usage trend information are displayed through a visual interface.

[0239] For example: Use the dashboard to display the usage of IO, CPU, memory and network in real time; use the Gantt chart to display the start time, end time and execution status of the task; use the curve chart or bar chart to display the historical trend and pattern of resource usage; and provide users with the function of performing scheduling operations, such as starting and stopping tasks, iterating tasks and reallocating resources.

[0240] Corresponding to the intelligent scheduling method of a scheduling platform proposed in the above embodiment of the present invention, see Figure 4 , shows a structural block diagram of an intelligent scheduling device of a scheduling platform proposed in an embodiment of the present invention.

[0241] The device includes: a value assessment unit 401 , a resource consumption assessment unit 402 , a collection unit 403 , a model building unit 404 and a solution unit 405 .

[0242] The value evaluation unit 401 is used to evaluate the task value of each task using a value evaluation model; the value evaluation model is pre-trained based on historical task data.

[0243] The resource consumption evaluation unit 402 is configured to evaluate resource consumption during task execution based on resource requirements during task execution using a resource evaluation model; the resource evaluation model is pre-trained based on historical task data.

[0244] The collecting unit 403 is configured to evaluate the total amount of all resources using a resource evaluation model and collect unfinished tasks from the scheduling platform.

[0245] The model building unit 404 is used to build a knapsack problem model based on the task value of each task, the resource consumption of each task, the total amount of all resources and unfinished tasks.

[0246] The solving unit 405 is used to solve the knapsack problem model in combination with the scheduling scorecard to obtain the optimal task combination, and allocate tasks to corresponding resources or execution units for execution according to the optimal task combination. The scheduling scorecard is generated in advance.

[0247] During actual application, the performance of the intelligent scheduling device is dynamically evaluated.

[0248] It should be noted that performance evaluation covers multiple aspects, including response time, scalability, resource utilization, failure recovery time, average completion time, waiting time and fairness.

[0249] Response time: This refers to the time required to obtain scheduling task and resource information, generate the scheduling scorecard, solve the knapsack problem, and submit it to the scheduling system. Shorter response times improve user satisfaction and timely resource utilization.

[0250] Scalability: Evaluates whether the scheduler can maintain or improve performance by expanding resources (e.g., bypassing resource bottlenecks) without increasing resources, especially in the context of vertical scaling (increasing the capacity of existing nodes).

[0251] Resource utilization: Monitor the usage of CPU, memory, storage, and network bandwidth. Efficient scheduling should maximize resource utilization and maintain balanced distribution.

[0252] Failure recovery time: The time required for the scheduling device to recover and reschedule when a failure occurs (such as a scheduling node crashes).

[0253] Average completion time: The average time it takes for all tasks to complete from start to finish. Reducing this time means optimizing system performance.

[0254] Waiting time: The average time a task waits in the queue to be scheduled. Long waiting times may indicate a bottleneck in the scheduling system or resources.

[0255] Fairness: The scheduling algorithm should ensure that all users or task groups receive fair resource allocation and processing time.

[0256] In this embodiment of the present invention, inter-task dependencies, weights, and resource consumption order are optimized by automatically analyzing task relationships and execution durations. This non-explicit automatic evaluation method automatically generates task weights, dependencies, and a topology diagram. Users only need to focus on the prerequisite tasks, and the system automatically analyzes and calculates weight scores and completes automated iterative updates, simplifying the operational process and reducing operational complexity.

[0257] Combine Figure 4 The content shown, the value assessment unit 401, includes: a first acquisition module, a scaling module and a processing module.

[0258] The first acquisition module is used to acquire the key factors of each task using a value assessment model.

[0259] The scaling module is used to convert key factors into quantitative indicators, scale each key factor through a pairwise comparison matrix, and obtain the relative importance of each key factor.

[0260] The processing module is used to standardize and normalize the relative importance of each key factor to obtain the task value of the task.

[0261] Combine Figure 4 As shown, the resource consumption evaluation unit 402 includes: an extraction module, an estimation module, a calculation module, a second acquisition module and an evaluation module.

[0262] The extraction module is used to extract feature data related to resource consumption during task execution and perform preprocessing to obtain resource demand data during task execution.

[0263] The estimation module is used to estimate the execution time of the task based on the resource requirement data.

[0264] The computing module is used to obtain the resource utilization of tasks from the core scheduling service module of the scheduling platform and the software and hardware computing service modules of the resource cluster.

[0265] The second acquisition module is used to obtain the historical response time and historical success rate of the task from the historical log module of the scheduling platform.

[0266] The evaluation module is used to evaluate the resource consumption during task execution based on resource demand data, execution time, resource utilization, historical response time and historical success rate through a resource evaluation model.

[0267] Combine Figure 4 As shown, the device further includes: a log acquisition unit, an analysis unit, a calculation unit, an evaluation unit and a generation unit.

[0268] The log acquisition unit is used to obtain the historical batch logs of the task.

[0269] The analysis unit is used to analyze the impact factor measurement and weighted measurement value of each task based on the historical batch logs and the resource requirement type of the task through the resource evaluation model.

[0270] The calculation unit is used to calculate the impact factor weight of each task and the number of all tasks.

[0271] The evaluation unit is used to evaluate the total amount of all resources through the resource evaluation model.

[0272] The generation unit is used to generate a scheduling scorecard according to the impact factor measurement, the impact factor weight, the weighted measurement value, the total amount of all resources and the number of all tasks.

[0273] Combine Figure 4 As shown in the content, the collecting unit 403 includes: a third obtaining module, which is used to obtain the total amount of CPU resources, total amount of memory resources, total amount of storage resources and total amount of network bandwidth resources in the scheduling platform through a resource evaluation model to obtain the total amount of all resources.

[0274] Combine Figure 4 The content shown, the model building unit 404, includes: a first definition module, a second definition module, a first determination module, a second determination module and a construction module.

[0275] The first definition module is used to define parameters of the knapsack problem model based on the task value of each task, the resource consumption of each task, the total amount of all resources and unfinished tasks.

[0276] The second definition module is used to define decision variables based on all tasks.

[0277] The first determination module is used to determine the resource utilization rate and the task completion time based on the task value of each task, the resource consumption of each task, and the total amount of all resources to establish an objective function.

[0278] The second determining module is used to determine the constraint conditions according to the total amount of all resources.

[0279] The building module is used to construct a knapsack problem model based on parameters, decision variables, objective functions and constraints.

[0280] Combine Figure 4 As shown in the content, the device also includes: a status information acquisition unit, a usage trend information generation unit and a display unit.

[0281] The status information acquisition unit is used to obtain real-time status information of resource usage and status information of task scheduling.

[0282] The usage trend information generating unit is used to generate resource usage trend information according to the real-time status information of resource usage.

[0283] The display unit is used to display real-time status information of resource usage, task scheduling status information and resource usage trend information through a visual interface.

[0284] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0285] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0286] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An intelligent scheduling method for a scheduling platform, characterized in that: The method comprises: For each task, using a value assessment model to assess the task value of the task; the value assessment model is pre-trained based on historical task data; Evaluating resource consumption during task execution based on resource requirements during task execution using a resource evaluation model; the resource evaluation model is pre-trained based on the historical task data; Evaluate the total amount of all resources through the resource evaluation model and collect unfinished tasks from the scheduling platform; Constructing a knapsack problem model based on the task value of each task, the resource consumption of each task, the total amount of all resources and the unfinished tasks; The knapsack problem model is solved in conjunction with a scheduling scorecard to obtain an optimal task combination, and tasks are assigned to corresponding resources or execution units for execution based on the optimal task combination. The scheduling scorecard is generated in advance.

2. The method according to claim 1, characterized in that For each task, the task value of the task is evaluated using a value evaluation model, including: For each task, use the value assessment model to obtain the key factors of the task; Convert the key factors into quantitative indicators, scale each key factor using a pairwise comparison matrix, and obtain the relative importance of each key factor; The relative importance of each of the key factors is standardized and normalized to obtain the task value of the task.

3. The method according to claim 1, characterized in that The evaluating the resource consumption during the execution of the task based on the resource requirements during the execution of the task using a resource evaluation model includes: Extracting characteristic data related to resource consumption during task execution and performing preprocessing to obtain resource demand data during task execution; estimating the execution time of the task based on the resource requirement data; Obtain the resource utilization of tasks from the core scheduling service module of the scheduling platform and the software and hardware computing service modules of the resource cluster; Obtain the historical response time and historical success rate of tasks from the historical log module of the scheduling platform; The resource consumption during the execution of the task is evaluated based on the resource demand data, the execution time, the resource utilization rate, the historical response time and the historical success rate through a resource evaluation model.

4. The method according to claim 3, characterized in that The process of generating the scheduling resource scorecard includes: Get the historical batch logs of the task; Analyzing the impact factor metric and weighted metric value of each task based on the historical batch run log and the resource requirement type of the task through the resource evaluation model; Calculate the impact factor weight of each task and the number of all tasks; Evaluate the total amount of all resources using the resource evaluation model; A scheduling scorecard is generated based on the impact factor metrics, the impact factor weights, the weighted metric values, the total amount of all resources, and the number of all tasks.

5. The method according to claim 1, wherein The resource evaluation model is used to evaluate the total amount of all resources, including: The resource evaluation model is used to obtain the total amount of CPU resources, memory resources, storage resources and network bandwidth resources in the scheduling platform to obtain the total amount of all resources.

6. The method according to claim 1, characterized in that The knapsack problem model is constructed based on the task value of each task, the resource consumption of each task, the total amount of all resources and the unfinished tasks, including: Defining parameters of a knapsack problem model based on the task value of each task, the resource consumption of each task, the total amount of all resources, and the unfinished tasks; Define decision variables based on all tasks; determining resource utilization and task completion time based on the task value of each task, the resource consumption of each task, and the total amount of all resources to establish an objective function; Determining constraints based on the total amount of all resources; A knapsack problem model is constructed based on the parameters, the decision variables, the objective function and the constraint conditions.

7. The method according to claim 1, characterized in that After solving the knapsack problem model in combination with the scheduling scorecard to obtain the optimal task combination and allocating the tasks to corresponding resources or execution units for execution according to the optimal task combination, the following steps are also included: Obtain real-time status information of resource usage and task scheduling; generating resource usage trend information based on the real-time status information of resource usage; The real-time status information of the resource usage, the status information of the task scheduling and the usage trend information of the resource are displayed through a visual interface.

8. An intelligent scheduling device for a scheduling platform, characterized in that: The device comprises: A value evaluation unit, configured to evaluate the task value of each task using a value evaluation model; the value evaluation model is pre-trained based on historical task data; a resource consumption evaluation unit, configured to evaluate resource consumption during task execution based on resource requirements during task execution using a resource evaluation model; the resource evaluation model is pre-trained based on the historical task data; A collection unit, configured to evaluate the total amount of all resources using the resource evaluation model and collect unfinished tasks from the scheduling platform; A model building unit is used to build a knapsack problem model based on the task value of each task, the resource consumption of each task, the total amount of all resources and the unfinished tasks; A solving unit is used to solve the knapsack problem model in combination with a scheduling scorecard to obtain an optimal task combination, and to allocate tasks to corresponding resources or execution units for execution according to the optimal task combination. The scheduling scorecard is generated in advance.

9. The device according to claim 8, characterized in that The value assessment unit includes: The first acquisition module is used to obtain the key factors of each task using the value assessment model; A scaling module is used to convert the key factors into quantitative indicators, scale each key factor through a pairwise comparison matrix, and obtain the relative importance of each key factor; The processing module is used to standardize and normalize the relative importance of each of the key factors to obtain the task value of the task.

10. The device according to claim 8, characterized in that The resource consumption evaluation unit includes: An extraction module is used to extract characteristic data related to resource consumption during the execution of the task and perform preprocessing to obtain resource demand data during the execution of the task; an estimation module, configured to estimate the execution time of a task based on the resource requirement data; The calculation module is used to obtain the resource utilization of tasks from the core scheduling service module of the scheduling platform and the software and hardware computing service modules of the resource cluster; The second acquisition module is used to obtain the historical response time and historical success rate of the task from the historical log module of the scheduling platform; An evaluation module is used to evaluate the resource consumption during the execution of the task based on the resource demand data, the execution time, the resource utilization rate, the historical response time and the historical success rate through a resource evaluation model.

Images