Method for constructing and monitoring Hadoop cluster and alarm based on containerization technology under PK system

A technology of hadoop clustering and containerization technology, applied in hardware monitoring, instrumentation, program control design, etc., it can solve the problems of cumbersome deployment environment configuration, insufficient number of servers, and low timeliness, so as to improve resource utilization and facilitate expansion. , the effect of eliminating environmental differences

Pending Publication Date: 2020-12-15
湖南长城科技信息有限公司
View PDF0 Cites 7 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] Software developers often need to develop and debug multi-node Hadoop clusters during their work. One of the problems they often encounter is that the number of servers in the R&D environment is not enough
Similarly, testers and operation and maintenance perso

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method for constructing and monitoring Hadoop cluster and alarm based on containerization technology under PK system
  • Method for constructing and monitoring Hadoop cluster and alarm based on containerization technology under PK system
  • Method for constructing and monitoring Hadoop cluster and alarm based on containerization technology under PK system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0060] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be described in further detail below in conjunction with the accompanying drawings and specific embodiments. And the features in the embodiments can be combined with each other.

[0061] combine figure 1 and figure 2 As shown, the embodiment of the present invention provides a method for constructing and monitoring Hadoop clusters and alarms based on containerization technology under the PK system, including the following steps:

[0062] S1, build k8s (Kubernetes, container orchestration system) on the node server based on the PK system and deploy the mirror warehouse service of the application. The steps to build a k8s are as follows:

[0063] S11. Select nodes on the node server as management nodes, computing nodes and storage nodes. If the selected nodes are multi-management nodes, you need to configure proxy nodes on the no...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method for constructing and monitoring a Hadoop cluster and an alarm based on a containerization technology in a PK system. The method comprises the steps: S1, constructing k8s on a node server based on the PK system and deploying an applied mirror image warehouse service; S2, deploying Prometheus on the k8s, wherein deploying comprises the steps of creating a custom object Alertmanger, a Prometheus Rule, a Prometheus, a PodMonitor, a ServiceMonitor and other Prometheus components of the Prometheus, and configuring Grafana to carry out graphical interface display; S3,establishing a Hadoop cluster on the k8s, wherein the Hadoop cluster comprises an HDFS cluster and a YARN cluster for deploying the Hadoop cluster; S4, deploying a Hadoop-exporter on a node server, acquiring monitoring data of the Hadoop cluster, and providing a data interface according to a format specified by a container monitoring platform, wherein the container monitoring platform calls a Hadoop-exporter data interface to acquire the acquired monitoring data; and S5, creating an alarm rule through the PrometreusRule object, and enabling the Alertmanager to receive the alarm information ofthe PrometreusRule and notify a user of the alarm information. The whole deployment can be realized by one key, the deployment is rapid and concise, and the requirements of data analysts, testers andoperation and maintenance personnel in software development, application and management work are met.

Description

technical field [0001] The invention belongs to the technical field of Hadoop cluster management, and specifically relates to a method for building and monitoring Hadoop clusters and alarms based on container technology under a PK system. Background technique [0002] PK, P refers to the Phytium processor, which is a processor chip independently designed by China Electronics that is compatible with the ARMV8 instruction set, and K refers to the Kylin operating system, which supports cloud computing, virtualization, big data and other applications and is processed with Phytium Adapter depth adaptation. The PK system is a combination of basic and open architectures, and it is the core system of national network security, benchmarking against Windows and Intel. Hadoop (distributed system infrastructure) cluster is a software framework capable of distributed processing of large amounts of data. Data processing is performed in a reliable, efficient, and scalable manner. It is th...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/455G06F11/30
CPCG06F9/45558G06F11/301G06F11/3051G06F11/3093G06F2009/45591G06F2009/45562G06F2009/4557
Inventor 高晓飞杜变霞曹科宁聂芳华张航
Owner 湖南长城科技信息有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap