Emergency guarantee method and system based on Hadoop double clusters

A clustering and fault technology, applied in the direction of response error generation, redundancy in hardware for data error detection, etc., can solve problems such as slow Hive task running, impact on business timeliness assessment, cluster instability, etc., and achieve easy maintenance. , to ensure consistency, the process is simple and convenient

Pending Publication Date: 2022-07-05
北京思特奇信息技术股份有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] Although the Hadoop cluster HA function can solve the problem of machine crashes or machine upgrades and maintenance, the NameNode can quickly switch the NameNode to another machine through hot standby, but other unknown factors will be encountered during actual business operation, such as: cluster Instability, non-HA sudden failures, slow running of Hive tasks, etc., affect business timeliness assessment

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Emergency guarantee method and system based on Hadoop double clusters
  • Emergency guarantee method and system based on Hadoop double clusters
  • Emergency guarantee method and system based on Hadoop double clusters

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0048] In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present invention.

[0049] Below in conjunction with accompanying drawing, the present invention is described in further detail:

[0050] like figure 1 As shown, a Hadoop dual-cluster-based emergency guarantee method provided according to the present invention includes:

[0051] Configure a Hadoop standby cluster for the Hadoop production cluster;

[0052] P...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses an emergency guarantee method and system based on Hadoop double clusters. The method comprises the steps that a Hadoop standby cluster is configured for a Hadoop production cluster; pre-configuring a task state and an execution state of each task, and scheduling according to the task state and the execution state of the task; when the production cluster breaks down, the current task breakpoint state is obtained, and task intervention is executed; operation and data synchronization of the production cluster are stopped, and the standby cluster continues to execute the current task; when emergency recovery is carried out on the repaired production cluster, data in the standby cluster is back-prepared to the production cluster; operation of the standby cluster is stopped, the production cluster continues to execute the task, and meanwhile the data synchronization process is started. By means of the technical scheme, operation of the scheduling task can be flexibly controlled, it is guaranteed that the task can be operated on the standby cluster when the production cluster breaks down, the consistency of the task state and task data is guaranteed, the process is simple and convenient, maintenance is easy, and stability is high.

Description

technical field [0001] The invention relates to the technical field of communication data processing, in particular to an emergency support method based on Hadoop dual clusters and an emergency support system based on Hadoop dual clusters. Background technique [0002] Hadoop is a general processing platform for big data, which provides distributed file storage and distributed offline parallel computing. Due to the high scalability of Hadoop, Hadoop is usually run as a cluster. Process petabytes of data. [0003] The HA function of the Hadoop cluster HDFS implements the hot backup of the NameNode in the cluster by configuring two NameNodes Active / Standby to solve the failure of the NameNode machine (such as downtime or software or hardware upgrade). [0004] Although the HA function of a Hadoop cluster can solve machine crashes or machines that need to be upgraded and maintained, NameNode quickly switches the NameNode to another machine through hot backup, but other unknown...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F11/20G06F11/16
CPCG06F11/203G06F11/2028G06F11/1687
Inventor 胡俊锋
Owner 北京思特奇信息技术股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap