Cluster state supervision method and device

A state and cluster technology, applied in the field of big data processing, can solve the problems of Jobmanager hanging up and crashing state, and achieve the effect of reducing cost, reducing cost and task blocking time

Pending Publication Date: 2021-11-02
杭州玳数科技有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] In view of the above problems, the present invention provides a cluster state supervision method and device, which can effectively solve problems such as dead state and Jobmanager hang-up that are easy to occur in the existing yarn-session mode task submission process

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Cluster state supervision method and device
  • Cluster state supervision method and device
  • Cluster state supervision method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the specific implementation manners of the present invention will be described below with reference to the accompanying drawings. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention, and those skilled in the art can obtain other accompanying drawings based on these drawings and obtain other implementations.

[0058] The first embodiment of the present invention is a cluster state supervision method applied to the flink on yarn cluster mode, such as figure 1 As shown, the cluster state supervision method includes:

[0059] S10 obtains the Appid of the Flink Session on the current resource management platform through the yarn client;

[0060] S20 If the Appid of the Flink Session is not obtained, enter the step of starting the Flink Session, and determine whether the current engine node...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a cluster state supervision method and device, and the method comprises the steps: obtaining an Appid of a Flink Session on a current resource management platform through a yarn client; if the Appid of a Flink Session is not acquired, starting Flink Session, and judging whether the current engine node is the leader node or not; if so, judging whether the current engine node is in an engine automatic hosting state or not; if so, deploying a Flink Session; and starting a state monitor, and polling the health state of the Flink Session. According to the method, the Flink Session management component is created, the functions of starting, stopping and the like of the Flink Session are achieved, meanwhile, the Flink Session is periodically monitored through the state monitor, and stable and healthy operation of the Flink Session is guaranteed.

Description

technical field [0001] The invention relates to the technical field of big data processing, in particular to a cluster state supervision method and device. Background technique [0002] In the customer's production environment, most of them will use the yarn-session mode to submit tasks. During the running process, the cluster will be pre-started so that the job does not need to repeatedly apply for resources. Therefore, it is generally suitable for short-term jobs that are sensitive to startup delays, such as interactive queries. , synchronization tasks, etc. Although this mode can complete the submission of flink tasks, there are often two problems: 1) flink session running for a long time on yarn will appear in a dead state; 2) due to the sharing of resources between jobs, the task load is particularly large Under the circumstances, it will bring problems such as concurrency and resource competition, which will lead to the problem of Jobmanager hanging. Contents of the...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F11/30
CPCG06F11/3055G06F11/3006
Inventor 兰洋
Owner 杭州玳数科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap