Job scheduling method and device for streaming data

Abstract

The invention relates to a job scheduling method for streaming data. The job scheduling method for the streaming data comprises the following steps: a scheduling manager acquires jobs to be scheduled from a scheduling queue in real time, and utilizes a directed acyclic graph for generating a processing unit queue according to information of the jobs to be scheduled; according to the ratio of the non-local communication number of physical nodes to dominant resources, the scheduling manager selects one physical node for each processing unit in the processing unit queue and distributes all the processing units to the corresponding physical nodes respectively; when an actuator starts one processing unit, a linux container is established on the physical node of the processing unit, and then the processing unit is started inside the linux container. According to the job scheduling method for the streaming data, the processing units are scheduled to the physical nodes which are small in non-local communication number and low in load, the processing units which need communication frequently can be concentrated to the same physical node, and therefore network communication across the physical nodes is reduced.

Description

technical field [0001] The present invention relates to the field of computer parallel computing, in particular to a stream data-oriented job scheduling method and device. Background technique [0002] In recent years, with the continuous development of applications such as real-time search, advertisement recommendation, social networking, and log online analysis, a new data form - streaming data is emerging. Streaming data refers to a large, fast, uninterrupted sequence of events. In different scenarios, streaming data can be in various data forms such as real-time queries, user clicks, online logs, and streaming media. Streaming applications focus on real-time interaction, and high-latency responses will seriously affect their functions or user experience. Due to the importance and uniqueness of streaming data, a batch of streaming data processing systems have emerged, such as Yahoo! The S4 system. [0003] Events are the basic unit of streaming data in the form of key...

AI Technical Summary

Problems solved by technology

[0005] The existing streaming data processing system does not comprehensively consider the communication of processing units and the real-time load of physical nodes, but only ensures that the data volume of processing units is evenly distributed. However, because different processing units have different requirements for various resources, the balance in quantity does not mean Balancing resource usage and actual load

Moreover, processing units that communicate frequently with each other may be dispatched to remote environments, increasing their communication costs and delays

Existing scheduling methods cannot well meet the needs of streaming data processing, resulting in uneven loads on physical nodes and high communication delays, which affect the overall performance of streaming data processing

Images