Method for dynamically configuring distributed cluster node label

A distributed cluster, node label technology, applied in the field of dynamically configuring distributed cluster node labels, can solve the problem that the server can only be idle

Inactive Publication Date: 2019-09-24
CHINA CITIC BANK
View PDF7 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0017] The disadvantage is that when there are no jobs in a tag group, and there are no jobs in queues with no tags configured, the server of the tag group can only be idle
Cannot be used by tasks in other tab groups

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method for dynamically configuring distributed cluster node label
  • Method for dynamically configuring distributed cluster node label
  • Method for dynamically configuring distributed cluster node label

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0044] [Example 1: Dynamic labeling cluster deployment method]

[0045] image 3 is a flowchart of the dynamic labeling cluster deployment method, such as image 3 shown, including:

[0046] Label setting step S1, YARN starts the node label function, adds the required labels in the cluster, and establishes a node pool;

[0047] The label activation step S1 also includes the mapping node mapping step S11, assigning corresponding labels to the cluster nodes that need to be physically isolated, and then configuring the mapping relationship between labels and queues in the resource queue of YARN.

[0048] As shown in 4, it is a schematic diagram of the mapping relationship between tags and queues in the tag startup step of the present invention. Figure 4 , the queue a (queue_a) contains two nodes (node_1 / node_2) to establish a mapping relationship with the corresponding label a (label_a), and the two nodes (node_3 / node_4) contained in the queue b (queue_b) also correspond to t...

Embodiment 2

[0064] Such as Figure 9 Shown is a schematic diagram of an embodiment of the present invention, such as Figure 9 As shown, the same set of clusters supports the different needs of multiple businesses, including data warehouse ETL jobs, machine learning, JAVA class engines, etc. Different jobs have very different resource requirements for clusters. All kinds of jobs (jobs) are classified by queue, submitted to the corresponding job (job), and then the queue is configured with a label, and this label is given to a specific node, so that the job can obtain the corresponding node resources.

[0065] This embodiment can avoid data synchronization among multiple clusters. For example, the current data mining needs to update data about 300G every day, and the synchronization speed is 30M / s, which takes 3h.

[0066] At the same time, this embodiment can improve resource utilization. For example, a business needs to apply for 4 nodes for daily data modeling, and the modeling time is...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a method for dynamically configuring distributed cluster node labels, which is realized on the basis of YARN NODE LABEL and comprises the following steps: a label setting step: adding required labels for cluster nodes in a cluster, taking nodes with the labels as label nodes, and the nodes with the same labels belonging to the same label group; a label monitoring step, monitoring the resource use conditions of the nodes of different label groups at regular intervals, and obtaining the resource utilization rates of the different label groups; and a label configuration step: adding or removing the labels of the nodes in the label group according to the resource utilization rate of the label group so as to realize label dynamic. According to the method, idle resources can be reduced to the maximum extent, the resource utilization rate is increased, and meanwhile operation and maintenance work caused by multiple clusters is reduced.

Description

technical field [0001] The invention relates to the field of computer networks, and relates to a distributed system architecture cluster deployment method with dynamic tags, more specifically, a method for dynamically configuring distributed cluster node tags. Background technique [0002] Such as figure 1 Shown is a schematic diagram of the framework based on HADOOP 2.0 big data platform. The YARN structure monitors cluster resources through the Resource Manager. This entity controls the entire cluster and manages the allocation of applications to basic computing resources. Resource Manager monitors cluster resources and carefully arranges various resource parts (computing, memory, bandwidth, etc.) to the basic Node Manager (YARN's per-node agent). Resource Manager monitors cluster resources and also allocates resources with Application Master (APP Mstr), and starts and monitors their underlying applications with Node Manager. In this structure, Application Master assume...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/08
CPCH04L67/10H04L67/1044
Inventor 王西斌叶依福
Owner CHINA CITIC BANK
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap