Cluster scheduling model construction method, scheduling model, scheduling method and system

A scheduling model and construction method technology, applied in the field of big data processing, can solve the problems of lack of scheduling decision-making and single scheduling basis of the cluster, and achieve the effect of ensuring stable and efficient operation and realizing elastic scaling

Pending Publication Date: 2022-05-06
SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0007] The technical task of the present invention is to address the above deficiencies and provide a cluster scheduling model construction method, sche

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Cluster scheduling model construction method, scheduling model, scheduling method and system
  • Cluster scheduling model construction method, scheduling model, scheduling method and system
  • Cluster scheduling model construction method, scheduling model, scheduling method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0059] The cluster scheduling model construction method of the present invention is used to realize the elastic scaling of large data cluster resources, including the following steps:

[0060] S100. Collect sample data of optimal adjustment points through actual set resource scheduling, where the sample data includes big data cluster specifications, cluster load, expected cluster adjusted load, and optimal number of adjustment nodes;

[0061] S200. Perform normalization processing on the sample data to obtain preprocessed sample data;

[0062] S300. Construct a prediction model based on the deep neural network of backpropagation. The prediction model takes the big data cluster specification, cluster load and expected cluster adjustment load as input, and outputs the optimal number of adjustment nodes;

[0063] S400. Train the prediction model based on the preprocessed sample data, and adjust the neuron weights of each layer of the prediction model based on the backpropagation ...

Embodiment 2

[0084] The cluster scheduling model of the present invention is a post-training prediction model obtained through the cluster scheduling model construction method disclosed in Embodiment 1. The post-training prediction model includes an input layer, a hidden layer, and an output layer, and is used for large data cluster specifications, The cluster load and the expected cluster adjusted load are input, and the optimal number of adjusted nodes is predicted and output.

[0085] The trained model takes big data cluster specifications, cluster load, and expected cluster adjusted load as input, predicts and outputs the optimal number of adjustment nodes, and uses the optimal number of adjustment nodes as a scheduling strategy to perform elastic scaling of big data cluster resources.

Embodiment 3

[0087] The cluster scheduling method of the present invention is used to realize the scheduling of large data cluster resource elastic scaling, including the following steps:

[0088] S100. Acquire big data cluster specifications, cluster load, and expected cluster adjusted load as input data, and normalize the input data to obtain preprocessed input data;

[0089] S200. Using the preprocessed input data as input, obtain the number of nodes that need to be adjusted by predicting through the big data cluster elastic scaling scheduling model disclosed in Embodiment 2.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a scheduling model construction method, a scheduling model, a scheduling method and a scheduling system of a cluster, belongs to the technical field of big data processing, and aims to solve the technical problems that the cluster lacks scheduling decisions and the scheduling basis is single. Comprising the following steps: collecting sample data of an optimal adjustment point number through actual set resource scheduling; performing normalization processing on the sample data to obtain preprocessed sample data; a prediction model is constructed based on the deep neural network of back propagation, and the prediction model takes the big data cluster specification, the cluster load and the expected cluster adjustment load input, and takes the optimal adjustment node number as the output; and training a prediction model based on the preprocessed sample data, and adjusting the weight of each layer of neurons of the prediction model based on a back propagation method according to an error between a predicted value output by the prediction model and an actual value of the sample data to obtain a trained prediction model.

Description

technical field [0001] The invention relates to the technical field of big data processing, in particular to a cluster scheduling model construction method, scheduling model, scheduling method and system. Background technique [0002] Big data cluster elastic scaling is resource scheduling for YARN. YARN is a new resource management system module introduced by Hadoop 2.x. It is mainly used to manage resources in the cluster (mainly various hardware resources of the server, such as memory, CPU, etc. ), it not only manages hardware resources, but also manages some running task information, etc. Elastic scaling of computing resources means that the resource scheduler dynamically adjusts the number of YARN nodes according to the current cluster load and the expected cluster load to ensure that the cluster resource usage is close to the expected cluster load. [0003] The current auto-scaling strategy is relatively simple, and the main problems are as follows: [0004] 1. Faili...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/48G06F9/50G06K9/62G06N3/04
CPCG06F9/4843G06F9/505G06N3/045G06F18/214
Inventor 张栋魏金雷胡清李国涛刘传涛周永进孙亮亮林森宋丽丽
Owner SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap