Adaptive configuration method for short-time variable big data job cluster scheduling

A technology of cluster scheduling and configuration method, which is applied in multi-programming devices, neural learning methods, electrical digital data processing and other directions, and can solve problems such as manual adjustment and unified optimal scheduling.

Active Publication Date: 2020-01-31
BEIJING INSTITUTE OF TECHNOLOGYGY
View PDF4 Cites 20 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The scheduling of the cluster needs to be adjusted in time according to the changing jobs. Therefore, it is necessary for the operator o

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Adaptive configuration method for short-time variable big data job cluster scheduling
  • Adaptive configuration method for short-time variable big data job cluster scheduling
  • Adaptive configuration method for short-time variable big data job cluster scheduling

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0122] Embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. Such as figure 1 As shown, the method of the present invention can be divided into four parts, which are the history log-driven sample generator, the experience playback memory, the Agent module based on reinforcement learning and the Env module responsible for interacting with the environment. image 3 It is a flowchart of the method of the present invention; as Figure 1-5 Shown, method of the present invention comprises the following steps:

[0123] Step 1, initialize each module:

[0124] The Controller module is used to initialize the Env module and the Agent module and control the execution of the code in the pre-training phase, the formal operation phase, and the evaluation phase; the Controller module is divided into two subcategories according to the requirements of the system when it is running;

[0125] The Controller module is used in the ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses an adaptive configuration method for short-time variable big data job cluster scheduling. The method is a self-adaptive cluster scheduler configuration optimization method provided by aiming at the characteristics of cloud platform isomerism, cluster scheduler configuration optimization of dynamic load, isomerism of cloud platform load and short-time variability. Cloud platform loads can be divided into service applications and analysis applications, and different classifications are different in resource consumption and time requirements. According to the method, the configuration of the cluster scheduler is adjusted according to the state information of the job and the information of the cluster environment; the optimal scheduling configuration is always kept; therefore, the operation performance is improved, the operation delay is reduced, the heterogeneous load of the cloud platform can be better adapted, and the optimal configuration item corresponding to the current cluster state can be better found, so that the waiting time of the cluster operation is close to the minimum, the operation efficiency is improved, and the short-time variable big data operation is effectively scheduled.

Description

technical field [0001] The invention belongs to the technical field of cluster scheduling, and in particular relates to an adaptive configuration method for cluster scheduling of short-term variable big data operations. Background technique [0002] Currently, cluster scheduling is a necessary prerequisite for performance optimization and resource management of cloud computing systems. A good scheduler can effectively improve the utilization of the cluster and save the user's investment cost, so cluster scheduling has always been one of the hot research directions. Cluster scheduling for short-term big data jobs faces three major challenges: 1. The heterogeneity and dynamics of short-term jobs; 2. How to configure the scheduler, which will determine the performance of the job; 3. No one is suitable for all situations the optimal configuration. For cloud platforms, cluster jobs can be divided into two types: [0003] 1. Service applications: such as search engines (Search ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/50G06F9/48G06N3/08H04L29/08
CPCG06F9/5038G06F9/5072G06F9/4881G06N3/084H04L67/10G06F2209/5021G06F2209/484H04L67/60
Inventor 韩锐刘驰刘子峰李泽清
Owner BEIJING INSTITUTE OF TECHNOLOGYGY
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap