Distributed hadoop cluster fault automatic diagnosis and restoration system

A hadoop cluster and automatic diagnosis technology, applied in the transmission system, digital transmission system, electrical components, etc., can solve the problems of automatic analysis of monitoring data, monitoring data analysis, and establishment of predictive models, etc., to solve the problem of inability to perform intelligent analysis Effect

Active Publication Date: 2016-02-17
SHANGHAI SNC NET INFORMATION TECH CO LTD
View PDF3 Cites 90 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] The main disadvantages of the existing technology are as follows: 1. Real-time fault alarm cannot be realized, and each configured monitoring indicator requires maintenance personnel to log in to the web platform to check the problematic nodes; 2. Cluster monitoring can only view the current monitoring data without storing and historical query function, it is impossible to analyze the monitoring data and establish a predictive model; 3. It is impossible to automatically analyze the monitoring data and perform automatic repairs based on the analysis results. It is necessary to manually log in to the problem node to analyze the error log and solve the fault based on experience

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed hadoop cluster fault automatic diagnosis and restoration system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0023] figure 1 It is a schematic diagram of the architecture of the distributed hadoop cluster fault automatic diagnosis and repair system of the present invention.

[0024] See figure 1 , the distributed hadoop cluster fault automatic diagnosis and repair system provided by the present invention adopts the cluster monitoring module to monitor the cluster file system, job tasks and physical nodes respectively, and the database and the data analysis module constitute a data storage analysis processing module, thereby forming a cluster file System monitoring module, job task monitoring module, data storage analysis and processing module, and automatic repair module are five modules in total. These five small modules are realized and displayed and managed on the web, thus forming a hadoop automatic monitoring operation and maintenance platform, which can ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a distributed hadoop cluster fault automatic diagnosis and restoration system which comprises a cluster file system monitoring module for collecting and obtaining cluster node information and a database file; a work and task monitoring module for collecting information of work and tasks; a physical node monitoring module for monitoring resource consumption information of each physical node; a data storage and analysis and processing module for storing monitoring data to a database, setting monitoring alarm rules and configuring alarm ID, level and reasons in advance; and an automatic restoration module for defining and configuring various common alarm faults in advance and making a preprocessing script for each alarm fault, matching the fault happened at present with the alarm faults defined and configured in advance when monitoring a fault, and calling the corresponding preprocessing script to finish automatic restoration of the fault. The method can diagnose and restore system fault automatically to allow maintenance to become easier, and performance data and node state to be clearer and more obvious.

Description

technical field [0001] The invention relates to a cluster fault automatic diagnosis and repair system, in particular to a distributed hadoop cluster fault automatic diagnosis and repair system. Background technique [0002] There is no solution in the industry to automatically analyze and solve problems found in hadoop cluster monitoring. Currently, the solution to hadoop cluster failures is to pre-configure key operation and maintenance monitoring indicators, check the health of hadoop clusters and related projects, and perform job and task execution at the same time. Analysis, the monitoring information is exposed, and the maintenance personnel log in to the web platform to check the nodes with problems and their performance, log in to the nodes to analyze the logs, and repair the cluster. [0003] The main disadvantages of the existing technology are as follows: 1. Real-time alarm of faults cannot be realized, and each configured monitoring indicator requires maintenance ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L12/24
CPCH04L41/0654
Inventor 程永新胡永李京龙
Owner SHANGHAI SNC NET INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap