Distributed monitoring system and method for NIFI task

Abstract

The invention discloses a distributed monitoring system and method for an NIFI task, belongs to the field of task monitoring, and aims to solve the technical problem of how to perform log statistics and monitoring on the NIFI from two aspects of tasks and steps. The system comprises a kafka distributed cluster, a relational database, wherein a log table is configured in the relational database; alog processor which is a component and comprises a task log processor and a step log processor; and a monitoring platform, used for reading the log sheet and displaying the log sheet through a monitoring interface. The method comprises the following steps: constructing a distributed monitoring system for an NIFI task, and configuring a log processor based on an NIFI task chain; performing log statistics on the NIFI task from two aspects of the task and the step through a log processor; and respectively displaying the task log table and the step log table through a monitoring interface to monitor tasks and steps.

Description

technical field [0001] The invention relates to the field of task monitoring, in particular to a distributed monitoring system and method for NIFI tasks. Background technique [0002] In the tax industry, with the development and use of various information systems such as the "Golden Tax Phase III", VAT invoice management system, and personal tax management system, as well as the deep integration of people's living consumption and Internet technology, the internal taxation system has Data, third-party data from other government departments, and Internet tax-related data have exploded. In order to effectively store, manage and apply these big tax data and improve tax governance, tax systems across the country have introduced various big data component technologies such as Hive, Hbase, and Kudu. [0003] Due to the variety of systems, platforms and fields involved, current tax-related data is characterized by a large amount, multiple sources, and heterogeneity, which brings c...

AI Technical Summary

Problems solved by technology

[0006] Problem 1: The NIFI canvas only displays error messages at the processor level, and cannot perform unified monitoring from the perspective of tasks, and needs to manually locate the processor position. When deploying multiple tasks on a NIFI node, it is not convenient to run monitoring

[0007] Problem 2: NIFI adopts a streaming data processing method, the data does not land during the processing, and the problem data cannot be viewed afterwards

[0008] Question 3: The error message displayed on the canvas is only kept for a certain period of time, and the log cannot be saved persistently

However, due to the NIFI streaming data processing method, the same batch of data will flow through each processor on the branch and obtain a success or failure status at each node, so the overall execution of the task cannot be compared with each processor's Simple addition of statistical results

Images