A data processing flow design method based on NIFI

A technology of data processing and design method, applied in the direction of electrical digital data processing, program control design, transaction processing, etc., can solve problems such as single point bottlenecks in the system, and achieve the effect of ensuring processing quality, avoiding single point bottlenecks, and small data volume

Pending Publication Date: 2019-03-29
INSPUR TIANYUAN COMM INFORMATION SYST CO LTD
View PDF1 Cites 7 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0013] Aiming at the problems in the prior art, the present invention provides a data processing flow design method based on NIFI, which solves the problem that the system is likely to form a single-point bottleneck when the amount of collected and analyzed data increases, and simplifies the complexity and complexity of the original processing flow. Strong coupling with business, to achieve the purpose of flexible and simple file collection, analysis and configuration, and loose coupling of data processing process and business combination

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A data processing flow design method based on NIFI
  • A data processing flow design method based on NIFI
  • A data processing flow design method based on NIFI

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0053] The present invention provides a kind of data processing process design method based on NIFI:

[0054] The Collect processor in the master node operation mode in the NIFI cluster collects the file list of files in the directory and passes it to the PublishKafka processor in the cluster operation mode.

[0055] The PublishKafka processor sends the file list to the Kafka Topic, and the ConsumeKafka processor in the cluster operation mode reads the Kafka Topic file list and passes it to the FetchFiles processor in the cluster operation mode.

[0056] The FetchFiles processor downloads the corresponding files from the directory according to the file list, and transfers the downloaded corresponding files to the Parse processor in the cluster operation mode. The Parse processor performs adaptive analysis on the corresponding files according to the parsing rules and file types of the corresponding files.

[0057] The application of the method of the present invention is furthe...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data processing flow design method based on NIFI, which relates to the data processing field. The method includes giving full play to the cluster performance to get the processors that collect the list of file names; Distributing the list of files to processors on each node of the nifi cluster through kafka; Processor capable of multi-node and multi-thread parallel downloading of remote computer files; A processor that parses different acquisition files according to different file parsing rules, solves the single-point bottleneck problem of data processing, and can solve the problem of re-customizing the code when the file format changes by flexibly configuring collection instances, concurrency, scheduling policies, and file parsing rules.

Description

technical field [0001] The invention discloses a method for designing a data processing flow, relates to the field of data processing, and specifically relates to a method for designing a data processing flow based on NIFI. Background technique [0002] The traditional network management data collection is through the development of special business applications, by writing scripts, and then using crontab for timing scheduling, and then realize the timing collection of network management data. The collection directory of network management data is obtained by reading configuration files. The process is cumbersome and prone to problems. [0003] The analysis of the collected files is also implemented by adopting a customized code program to develop corresponding codes for a type of collected files and format, which is strongly related to the type of collected files. The advantage of this file parsing method is that the code logic is simple and easy to implement. The disadva...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/46G06F9/54
CPCG06F9/466G06F9/543
Inventor 杨凯杰郑国生
Owner INSPUR TIANYUAN COMM INFORMATION SYST CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap