A Method for Optimal Scheduling of Tasks

Abstract

The invention provides a task optimal scheduling method and can improve the task execution performance. The method comprises the following steps: S1, constructing a data dependency matrix between any data blocks in a DC (Data Center); S2, converting the data dependency matrixes into data clustering matrixes; S3, carrying out submatrix segmentation on the data clustering matrixes according to maximum available storage capacities of machine frames and of nodes in the machine frames, and placing the data blocks corresponding to the data clustering matrixes onto the nodes in the machine frames according to a segmentation result to obtain a current data placement scheme; S4, under the constraints of the current data placement scheme, with data migration cost minimization as a goal, determining a current optimal task scheduling scheme; and S5, when a data locality degree and a data migration cost in task execution process meet predetermined conditions, executing steps S1 to S4 repeatedly; or otherwise, outputting the current data placement scheme and the current task scheduling scheme as optimal solutions for task scheduling. The task optimal scheduling method provided by the invention is applied to the technical field of computers.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a task optimization scheduling method. Background technique [0002] In recent years, big data processing frameworks such as MapReduce have played an increasing role in processing data-intensive applications. It cannot be ignored that in practical applications, there is a correlation between the input data of some businesses. This correlation can be expressed as several data blocks may be processed by the same task, or as the input data of the next stage contains Intermediate results from the previous stage. Taking user profile information in social networks as an example, users who are in the same social circle or have common attributes in a certain aspect have correlations between their data information. When analyzing user data, if such user information is merged / aggregated, the size of task output results can be effectively reduced, thereby reducing data migration during ...

AI Technical Summary

Problems solved by technology

[0005] The technical problem to be solved by the present invention is to provide a task optimization scheduling method to solve the low utilization rate of resources in heterogeneous clusters in the prior art, and the scheduling scheme is not dynamically adjusted according to the current data locality constraints, resulting in The scheduling scheme is not the optimal solution

Images