Method and system for realizing NiFi high-availability cluster mode

An implementation method and cluster technology, applied in the field of clusters, can solve the problems of insufficient function of designated running nodes, loss of component status data, affecting cluster stability, etc., to increase the node failure task recovery function, improve high availability, and increase task load. The effect of the equalization function

Pending Publication Date: 2022-07-01
SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In this case, the task creation operation of other non-executing task nodes will cause a waste of resources
At the same time, the function of the existing designated running node is not perfect, and only supports the operation of the designated master node, which will cause excessive pressure on the master node, and t

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and system for realizing NiFi high-availability cluster mode
  • Method and system for realizing NiFi high-availability cluster mode
  • Method and system for realizing NiFi high-availability cluster mode

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0054] Example 1:

[0055] as attached figure 2 As shown, this embodiment provides a method for implementing NiFi high-availability cluster mode. The method is to add Master nodes and Worker nodes to the original NiFi cluster mode to construct an optimized NiFi cluster mode, and store the cluster state in the NiFi cluster mode. In the external Redis cluster, it is ensured that each node of the task status data can be accessed according to the task identifier, so as to ensure that the abnormally recovered task can normally obtain the persistent status data; in this way, the more worker nodes, the higher the cluster stability.

[0056] Among them, the Master node is used for backup and recovery of configuration data, load balancing of tasks, and rapid recovery of failed nodes and tasks running on them;

[0057] Worker nodes are used for task execution.

[0058] In the optimized NiFi cluster mode in this embodiment, after the worker node fails, in the scenario where the newly ...

Example Embodiment

[0077] Example 2:

[0078] This embodiment provides a NiFi high-availability cluster mode implementation system, the system includes:

[0079] The cluster optimization module is used to add Master nodes and Worker nodes to the original NiFi cluster mode to build an optimized NiFi cluster mode;

[0080] The configuration module is used to back up the configuration data on the master node's own node to the backup database, and restore the configuration data after the master node recovers abnormally;

[0081] The balancing module is used to create user tasks on different Worker nodes, and the execution of tasks in the Worker nodes is scheduled by the Master node to achieve task load balancing;

[0082] The recovery module is used to restore the task to the Worker node according to the task configuration information on the Master node after the Worker node fails, and continue to schedule task execution to quickly restore the failed node and the tasks running on it.

[0083] In t...

Example Embodiment

[0086] Example 3:

[0087] This embodiment also provides an electronic device, including: a memory and a processor;

[0088] wherein, the memory stores computer-executed instructions;

[0089] The processor executes the computer-executed instructions stored in the memory, so that the processor executes the method for implementing the NiFi high-availability cluster mode of any embodiment.

[0090] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs) or other programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or the processor may be any conventional processor or the like.

[0091] The memory can be used to store computer programs and / or modules, and the processor implements various functions of the electronic device by ru...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method and a system for realizing a NiFi high-availability cluster mode, belongs to the technical field of clusters, and aims to solve the technical problem of how to improve the automation degree of NiFi cluster exception recovery so as to improve the high availability of NiFi. According to the technical scheme, a Master node and a Worker node are added in an original NiFi cluster mode to construct an optimized NiFi cluster mode, and a cluster state is stored in an external Redis cluster, so that each node of task state data can be accessed according to a task identifier; therefore, it is ensured that persistent state data can be normally acquired by an exception recovery task; wherein the Master node is used for configuring backup and recovery of data, load balancing of tasks and rapid recovery of failure nodes and tasks running on the failure nodes; and the Worker node is used for task operation. The system comprises a cluster optimization module, a configuration module, an equalization module and a recovery module.

Description

technical field [0001] The invention relates to the technical field of clusters, in particular to a method and system for realizing a NiFi high-availability cluster mode. Background technique [0002] Today's society is a fast-developing society, with advanced technology, information flow, closer communication between people, and more convenient life. Big data is the product of this high-tech era. Big data is so important that its data collection, storage, search, sharing, analysis, and even visual presentation have become important research topics at present, and data integration and governance are the beginning and premise of all big data applications. During the development and implementation of many projects, customers and R&D personnel tend to pay more attention to data mining and analysis applications, but only later found that data collection and data governance are the biggest difficulties and the most time-consuming and labor-intensive work. Being able to extract e...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): H04L67/10H04L67/1001
CPCH04L67/10
Inventor 翟建峰胡清
Owner SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap