Scheduling method, scheduler, storage medium and system

Abstract

The invention provides a scheduling method, a scheduler, a storage medium and a system, and belongs to the technical field of communication. The scheduling method comprises the steps: acquiring an incidence relation graph and data distribution information, wherein each node in the incidence relation graph refers to one piece of data, and each directed edge is used for representing an operation request for calculating according to the data referenced by a source node to obtain the data referenced by a target node; according to a preset node sorting strategy, sequentially positioning the operation requests corresponding to the traversed nodes to any server where the data indicated by the traversed nodes are located; and sequentially scheduling at least one job request represented by at leastone directed edge to the positioned server. For the scheduling method, the incidence relation graph is used for representing the incidence relation between the data and the corresponding relation between the operation requests and the data, and the data distribution information is used for representing the data distribution condition, so that cross-node data transmission can be reduced, the operation execution time of the whole operation request is shortened, and the calculation efficiency is improved, and the system performance is improved.

Description

technical field [0001] The present disclosure relates to the technical field of communication, and in particular to a scheduling method, a scheduler, a storage medium and a system. Background technique [0002] With the growth of data scale and the development trend of data diversity, data analysis and processing technology is also constantly improving. At present, a variety of data analysis and processing architectures have emerged, such as Hadoop / MapReduce (distributed computing / mapping and merging) architecture , Spark, a big data processing framework based on memory computing, etc. Big data applications developed based on these data analysis and processing architectures can improve the efficiency of big data analysis and processing, and meet the real-time requirements of big data analysis and processing to a certain extent. [0003] When implementing big data applications, it is necessary to build a computing cluster with a certain scale, and dynamically adjust the comp...

AI Technical Summary

Problems solved by technology

[0005] The calculation process will be affected by the distribution of data. For example, if the data required by a certain job request is distributed on different servers, when a server is performing calculations according to the job request, it needs to transmit all data from other servers. required data, which results in data transfer across nodes

However, the storage location of the data in the above scheduling process is fixed, and the scheduling effect can only depend on the initial data distribution. If the initial data distribution is not good, there will be a large amount of cross-node data transmission in the calculation process, and then The job execution time is too long and the calculation efficiency is low, which affects the performance of the cloud computing system

Images