Scheduling method, scheduler, storage medium and system

Abstract

The disclosure provides a scheduling method, a scheduler, a storage medium and a system, belonging to the technical field of communication. The method includes: acquiring association relationship diagram and data distribution information, each node in the association relationship diagram refers to a piece of data, and each directed edge is used to represent the data referred to by the destination node calculated according to the data referred to by the source node Job request; according to the preset node sorting strategy, sequentially locate the job request corresponding to the traversed node to any server where the data pointed to by the traversed node is located; at least one of the nodes represented by at least one directed edge Job requests are dispatched to the located servers in turn. By using the association relationship diagram to represent the relationship between data and the corresponding relationship between job requests and data, and using data distribution information to represent data distribution, data transmission across nodes can be reduced, and the job execution time of the overall job request can be shortened. Improve computing efficiency and improve system performance.

Description

technical field [0001] The present disclosure relates to the technical field of communication, and in particular to a scheduling method, a scheduler, a storage medium and a system. Background technique [0002] With the growth of data scale and the development trend of data diversity, data analysis and processing technology is also constantly improving. At present, a variety of data analysis and processing architectures have emerged, such as Hadoop / MapReduce (distributed computing / mapping and merging) architecture , Spark, a big data processing framework based on memory computing, etc. Big data applications developed based on these data analysis and processing architectures can improve the efficiency of big data analysis and processing, and meet the real-time requirements of big data analysis and processing to a certain extent. [0003] When implementing big data applications, it is necessary to build a computing cluster with a certain scale, and dynamically adjust the comp...

AI Technical Summary

Problems solved by technology

[0005] The calculation process will be affected by the distribution of data. For example, if the data required by a certain job request is distributed on different servers, when a server is performing calculations according to the job request, it needs to transmit all data from other servers. required data, which results in data transfer across nodes

However, the storage location of the data in the above scheduling process is fixed, and the scheduling effect can only depend on the initial data distribution. If the initial data distribution is not good, there will be a large amount of cross-node data transmission in the calculation process, and then The job execution time is too long and the calculation efficiency is low, which affects the performance of the cloud computing system

Images