Online and offline hybrid scheduling system for improving quality of online services and utilization rate of cluster resource

Abstract

The invention provides an online and offline hybrid scheduling system for improving quality of online services and utilization rate of a cluster resource. The system comprises an information acquisition module, an online AM module, an offline AM module, a client side module and a scheduler module; the information acquisition module collects a scheduling relation among the online services in the system and stores the scheduling relation in a Redis; when a user submits an online application, the online AM module parses a dependence relation of each component, and orderly deploys the components in a cluster according to the dependence relation; when the user submits an offline work, the offline AM module applies to an RM for the resource, and schedules the offline work to run in the cluster;the client side module parses and processes the work submitted by the user, converts the work into a corresponding request for applying to the RM for the resource, and takes charge of pulling up the online AM module to present application running conditions for clients; the scheduler module is located in the RM, the scheduler module periodically calculates criticality of each server node accordingto criticality of each container reported by the online AM module, and performs scheduling according to the calculated server criticality when scheduling the offline work.

Description

technical field [0001] The invention relates to the field of cluster resource task scheduling, and mainly relates to an online and offline hybrid scheduling system for improving online service quality and cluster resource utilization. Background technique [0002] With the vigorous development of cloud computing technology, more and more calculations and applications are hosted on public clouds. Through the cloud platform, users can quickly and easily expand simple applications into large-scale complex applications, and pay on demand, at each time node, they only pay costs comparable to their scale. Suppliers can gain economies of scale by building large-scale data centers and sharing resources with multiple tenants. [0003] However, cluster resource utilization in most cloud facilities is very low, which greatly reduces cost efficiency. The researchers conducted a resource utilization analysis for a production cluster of thousands of machines on Twitter. This cluster was...

AI Technical Summary

Problems solved by technology

However, the existing technology only supports scheduling offline jobs, which cannot meet the needs of modern data centers to simultaneously schedule online services and offline jobs; for multiple components deployed in the system, some are very busy, while others are very idle, lacking an accurate The method of locating the key components that have the greatest impact on user experience in the system; in addition, the system does not consider giving preferential treatment to these key components, and lacks a method to ensure the quality of online services

Images