Cluster job scheduling method and system for task multi-copy execution

Abstract

The present invention provides a cluster job scheduling method and system for multi-copy execution of tasks, including: searching for lagging machines: using machine learning methods to find lagging machines currently running tasks; calculating the optimal number of copies: tasks on lagging machines The copy and all tasks are started at the same time, and an optimization model with the goal of minimizing task execution time and operating cost is established, and then the optimal number of starting copies is obtained by solving the optimization model using the alternating direction method. The invention eliminates the detection process and the execution time of the lagging task before it is discovered; it establishes the optimization goal of minimizing the process time of the job and the computing cost in the cluster at the same time, and the number of tasks executed in the cluster does not exceed the cluster The optimization model with the constraints that the number of available computing nodes and the number of copies of each task do not exceed a given threshold.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a machine learning-based cluster job scheduling method and system for multi-copy execution of tasks. Background technique [0002] Support Vector Machine (SVM) is a machine learning method based on statistical learning theory developed in the mid-1990s. It improves the generalization ability of learning machines by seeking the minimum structured risk, and minimizes the empirical risk and confidence range, so as to achieve In the case of a small statistical sample size, the purpose of good statistical regularity can also be obtained. It is a two-class classification model, and its basic model is defined as a linear classifier with the largest interval in the feature space, that is, the learning strategy of the support vector machine is to maximize the interval, which can finally be transformed into the solution of a convex quadratic programming problem. . [0003] Alte...

AI Technical Summary

Problems solved by technology

[0004] With the increasing size and complexity of clusters, how to ensure the measurability and predictability of cluster performance has become increasingly important. However, the prevalence of lagging phenomenon in clusters is a key factor affecting the predictability of cluster performance. The execution time of a task on a certain computing node in a cluster is greatly extended, which makes it impossible to estimate the task completion time, which greatly affects the cluster performance

The most basic way to deal with this lagging phenomenon is to run several copies of the lagging task on other machines. When any copy finishes executing the task first, the task execution is completed, and the other copies that are still running are terminated and cleared. And data, the most classic algorithm is the speculative execution method. According to the execution of each task, it is speculated which tasks will become the tasks that are slowing down, and then according to the results of the speculation, copies of the tasks that are slowing down are executed on other machines. The passive method of performing the copy does not perform well in some tasks that are particularly sensitive to latency

Images