A method for recording data points based on NIFI for data cleaning and conversion

Abstract

The present invention provides a method for recording data cleaning and conversion based on NIFI, which belongs to the field of big data in Internet applications. The present invention realizes the execution log and problem data recording of each data cleaning and conversion processor by configuring the original data cleaning and conversion processing flow with a tracking device. By defining a tracking processor to receive the result data of the cleaning and conversion processor for targeted analysis and recording, the analysis and monitoring of the execution results of different cleaning and conversion processors in the entire data cleaning and conversion task can be realized. Relevant problem data and data record log information of each cleaning and conversion can be recorded, which greatly reduces the time and labor cost of problem troubleshooting.

Description

Technical Field

[0001] The present invention relates to the fields of big data, data processing and computer technology in Internet applications, and in particular to a point-of-care recording method for data cleaning and conversion based on NIFI. Background Art

[0002] NiFi: is an easy-to-use, powerful and reliable streaming data processing and distribution system. NiFi is designed for data flow, supports dynamic data pulling from multiple data sources, and is based on a WEB graphical interface. Through dragging, connecting, and configuring, process-based programming is completed to achieve data collection, processing and other functions. Its code is completely open source.

[0003] Data cleaning: Clean up errors, inconsistencies, and incomplete parts of the data to ensure the validity and accuracy of the data. This process includes identifying and correcting erroneous data, removing redundant data (i.e., duplicate data), processing missing data, and converting data into a unified standard format for subsequent processing and analysis. The main goal of data cleaning is to eliminate noise and inaccuracies in the data so that the data is suitable for subsequent analysis and use.

[0004] Data conversion: converting data from one format or data model to another format or data model to support data sharing and interaction between different systems. Data conversion includes steps such as data integration, data reconstruction, and data calculation. Its main function is to improve database performance and application effects, ensure data quality and consistency, and meet the requirements of data warehouse models.

[0005] Processors: To create efficient data flow processing, NiFi provides about 300 ready-made processors. These processors provide the ability to extract data from different systems, route, transform, process, split and aggregate data, and distribute data to multiple systems. If this does not meet your needs, you can also customize the processor.

[0006] Connectio: You can connect different Processors together to create an automatic data processing flow.

[0007] Relationship: Each processor has zero or more relationships. These relationships indicate how to process the FlowFile: After the processor processes the FlowFile, it routes (transfers) the FlowFile to one of the relationships. The DFM can connect each relationship to other components to specify where the FlowFile should be processed next.

[0008] FlowFile: A FlowFile represents a single piece of data in NiFi. A FlowFile consists of attributes and content. The content is the data represented by the FlowFile, and the attributes consist of key-value pairs that provide information about the data or contextual characteristics.

[0009] Process Group: When data flows become complex, it is useful to manage data flows at a higher, more abstract level. NiFi allows multiple components (such as processors) to be combined into a process group. Components in the group can be viewed and manipulated through the interface.

[0010] At the current stage, NIFI-based data cleaning or data conversion operations lack clear problem data records and processor log record information. If the final data obtained after the data cleaning and conversion process cannot meet the usage requirements, it is necessary to gradually analyze the reasons for the formation of the problem data.

[0011] In the process of data cleaning or data conversion based on NIFI, a large amount of abnormal data that does not meet the standards or has conversion problems is often generated. However, in the entire cleaning and conversion process, it is difficult for data engineers to find out how this data is generated or what specific cleaning or conversion operations caused it. Summary of the invention

[0012] In order to solve the above technical problems, the present invention provides a point-of-care recording method for data cleaning and conversion based on NIFI. By recording the log records of each step of the logic, that is, the cleaning and conversion process and the problem data records, data development engineers can clearly eliminate the causes of the problem data, which is convenient for data developers to locate related problems and reduce the time for data development engineers to troubleshoot and analyze problem data when using NIFI for data cleaning and conversion.

[0013] The technical solution of the present invention is:

[0014] A method for tracking and recording data cleaning and conversion based on NIFI, which realizes the execution log and problem data recording of each data cleaning and conversion processor by configuring the tracking device for the original data cleaning and conversion processing flow; by defining the tracking processor to receive the result data of the cleaning and conversion processor for targeted analysis and recording, it realizes the analysis and monitoring of the execution results of different cleaning and conversion processors in the entire data cleaning and conversion task.

[0015] Further including,

[0016] 01. Create a task group (Process Group) and declare the current task group unique identifier (Group_Id). Create the cleaning conversion processor (Processer) information and the processing relationship (Relationship) information after the file flow (FlowFile) cleaning conversion logic code is successfully executed or the cleaning conversion logic code fails in the group according to the process information. The successful processing relationship needs to be connected to the next step of data cleaning conversion processor by default.

[0017] 102. During the configuration of the cleaning conversion processor, it is necessary to declare the identifier of the current processor, i.e. Processer_Id, and configure the processing operations that the current processor needs to perform, such as filtering the empty value of a field in the data record, or converting the time string to the corresponding date format, etc., and create a new data identifier for each data record in the data stream. This identifier is used to declare whether the current data record is defined as problem data due to logic verification failure or processing error during the data cleaning or data conversion process.

[0018] 103. Develop a tracking point processor. In order not to affect the original processing logic and configuration process, define and declare a new processor. This processor is used to receive the data flow information of the cleaning conversion processor and perform analysis and recording operations. By judging whether the access relationship (Relationship) of the connected upper-level cleaning conversion processor is successful or failed, it is judged whether the current access node is successfully accessed or failed to access. By judging the problem data identifier of each data in the data flow (FlowFile), it is judged whether the current data is problem data, and the data sum of the data flow (FlowFile) connected by the current tracking point processor and the sum of the problem data are calculated and stored.

[0019] 104. Connect to the tracking processor. In step 102, each cleaning conversion processor that succeeds or fails needs to be connected to a tracking processor, and connected to the cleaning conversion processor through a connection and passed in the relevant failed or successful data stream. If some data records cause NIFI execution to throw an exception, they are also considered to be problem data. Such data will not be identified as separate problem data for each data record. Data connected through a failure relationship is defined as problem data.

[0020] 105. Define the tracking processor, access the successful tracking processor and the failed tracking processor according to the relationship defined in steps 101-104. The configuration information includes the information of the cleaning and conversion processor (Processer_Id, etc.) accessed by the current tracking processor, the information of the current task group (Group_Id, etc.), the configuration information of the cleaning and conversion processor accessed by the upper layer, the failure or success relationship (FlowFile) information, the abnormal data storage data source information, etc. The successful tracking processor needs to filter the data, and store the data that failed to be cleaned or converted due to failure to meet the processing rules, as well as the data source information and cleaning and conversion information in the abnormal data storage data source, that is, the data record defined as the problem data in 102. The failed tracking processor needs to store all the received data information, data source and cleaning and conversion information in the abnormal data storage data source. The failed tracking processor is used to receive data records that cause the system to throw an exception due to conversion or cleaning triggering exception capture. In order to ensure that different problem data can be distinguished from different task execution batches, each task defines and declares a batch number (Batch_Id) for each cleaning and conversion processor when executing.

[0021] 106. Two steps required to declare

[0022] 1061. Save the log batch information executed by the current cleaning conversion processor, that is, the flag information (Processer_Id) of the current cleaning conversion processor, the task group information (Group_Id), the currently executed batch information (Batch_Id), the total amount of data accessed, the amount of problem data accessed, the execution time and other information. According to the number of data flows (FlowFiles) divided by the total amount of data accessed by this data cleaning conversion, the log batch information of each cleaning conversion processor may generate multiple records.

[0023] 1062. Save the data that failed to be cleaned or converted by the current cleaning conversion processor, i.e., the problem data, including the source information of the current data, data-related cleaning conversion information, log batch information, and the data itself information.

[0024] Going further,

[0025] This tracking point processor can record the problem data that causes NIFI execution abnormality or the problem data that does not meet the cleaning conversion requirements.

[0026] Each data business processor that performs cleaning and conversion based on NIFI needs to be connected to two tracking point processors.

[0027] Each tracking processor can record the total amount of data processed by different processors in different batches when using NIFI for data cleaning and conversion, the amount of problem data identified, and the corresponding problem data.

[0028] The beneficial effects of the present invention are

[0029] 1. Record the log information of each step of data cleaning and conversion operation: The log record of data cleaning and conversion can be defined at the component level, that is, the amount of problem data generated by each data cleaning and conversion operation can be clearly recorded.

[0030] 2. Problem data analysis: You can clearly view the problem data generated by each cleaning and conversion step, that is, the reasons for the problem data, to provide better decision-making for data developers. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a workflow configuration diagram of the present invention;

[0032] Figure 2 It is a flow chart of the buried point information recording of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0034] The present invention provides a method for recording data points for data cleaning and conversion based on NIFI. After data developers deploy and configure relevant parameters according to the requirements of this device, they can record relevant problem data and data record log information for each cleaning and conversion, greatly reducing the time and labor cost of problem troubleshooting.

[0035] The present invention realizes the execution log and problem data recording of each data cleaning conversion processor by configuring the embedding device for the original data cleaning conversion process. By defining the embedding processor to receive the result data of the cleaning conversion processor for targeted analysis and recording, the analysis and monitoring of the execution results of different cleaning conversion processors in the entire data cleaning conversion task can be realized.

[0036] The specific steps are as follows

[0037] Step 1. First, define the process for the cleaning and conversion tasks that require data cleaning or data conversion, that is, design the task process, clarify the cleaning and conversion operations that need to be done on the data, and define the relevant execution logic and sequence.

[0038] Step 2. Configure the NIFI task in the NIFI canvas or call the NIFIRest API according to the process design information in step 1, create a task group (Process Group), and declare the current task group unique identifier (Group_Id). Create the cleaning conversion processor (Processer) information and the processing relationship (Relationship) information after the file flow (FlowFile) cleaning conversion logic code in the processor is successfully executed or the cleaning conversion logic code fails to execute according to the process information in the group. The successful processing relationship needs to be connected to the data cleaning conversion processor to be performed in the next step by default.

[0039] Step 3. During the configuration of the cleaning conversion processor in step 2, it is necessary to declare the identifier of the current processor, i.e., Processer_Id, and configure the processing operations that the current processor needs to perform, such as filtering the null value of a field in the data record, or converting the time string to the corresponding date format, etc., and create a new data identifier for each data record in the data stream. This identifier is used to declare whether the current data record is defined as problem data during the data cleaning or data conversion process.

[0040] Step 4. After steps 1-3 are configured, the relevant configuration content in the task group needs to be stored, including the processor identification information (Processer_Id), cleaning conversion information, and processor configuration process sequence information.

[0041] Step 5. Develop a tracking point processor. In order not to affect the original processing logic and configuration process, define and declare a new processor. This processor is used to receive the data flow information of the cleaning conversion processor and perform analysis and recording operations. By judging whether the access relationship (Relationship) of the connected upper-level cleaning conversion processor is successful or failed, it is judged whether the current access node is successfully accessed or failed to access. By judging the problem data identifier of each data in the data flow (FlowFile), it is judged whether the current data is problem data, and the data sum of the data flow (FlowFile) connected by the current tracking point processor and the sum of the problem data are calculated and stored.

[0042] Step 6. Connect the tracking processor. In step 2, each cleaning conversion processor that succeeds or fails needs to be connected to a tracking processor, and connected to the cleaning conversion processor through a connection and passed in the relevant failed or successful data stream. If some data records cause NIFI execution to throw an exception, they are also considered to be problem data. Such data will not be identified as separate problem data for each data record. Data connected through a failure relationship is defined as problem data.

[0043] Step 7. Define the tracking processor, and connect the successful tracking processor and the failed tracking processor according to the relationship defined in steps 2-6. The configuration information includes the information of the cleaning and conversion processor connected to the current tracking processor (Processer_Id, etc.), the information of the current task group (Group_Id, etc.), the configuration information of the cleaning and conversion processor connected to the upper layer, the failure or success relationship (FlowFile) information, the abnormal data storage data source information, etc. The successful tracking processor needs to filter the data, and store the data that failed to be cleaned or converted due to failure to meet the processing rules, as well as the data source information and cleaning and conversion information in the abnormal data storage data source, that is, the data record defined as the problem data in step 3. The failed tracking processor needs to store all the received data information, data source and cleaning and conversion information in the abnormal data storage data source. The failed tracking processor is used to receive data records that cause the system to throw an exception due to conversion or cleaning triggering exception capture. In order to ensure that different problem data can be distinguished from different task execution batches, each task defines and declares a batch number (Batch_Id) for each cleaning and conversion processor when executing.

[0044] Step 8. The buried point storage is divided into two steps when performing operations.

[0045] Link 1 saves the log batch information executed by the current cleaning conversion processor, that is, the flag information (Processer_Id) of the current cleaning conversion processor, the task group information (Group_Id), the currently executed batch information (Batch_Id), the total amount of data accessed, the amount of problem data accessed, the execution time, etc. According to the number of data flows (FlowFiles) divided by the total amount of data accessed by this data cleaning conversion, the log batch information of each cleaning conversion processor may generate multiple records.

[0046] Step 2. Save the data that failed to be cleaned or converted by the current cleaning and conversion processor, i.e., the problem data, including the source information of the current data, data-related cleaning and conversion information, log batch information, and the data itself information.

[0047] Step 9. After the task configuration is completed, start the NIFI task to start execution. NIFI will perform data cleaning and conversion operations based on the task configuration information. The configured tracking processor will record the log information and problem data information separately according to the configuration information.

[0048] Step 10. Data developers can check data by viewing logs or problem data information, check abnormal data generated during cleaning and conversion operations, and analyze the causes of abnormal data.

[0049] The above description is only a preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, and is not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for recording data points based on NIFI for data cleaning and conversion, characterized in that: By configuring the tracking device for the original data cleaning and conversion process, the execution log and problem data record of each data cleaning and conversion processor can be realized; by defining the tracking processor to receive the result data of the cleaning and conversion processor for targeted analysis and recording, the analysis and monitoring of the execution results of different cleaning and conversion processors in the entire data cleaning and conversion task can be realized; The specific steps are as follows Step (1): First, define the process for the cleaning and conversion tasks that require data cleaning or data conversion, that is, design the task process, clarify the cleaning and conversion operations that need to be performed on the data, and define the relevant execution logic and sequence; Step (2): Based on the process design information in step (1), configure the NIFI task in the NIFI canvas or call the NIFIRest API, create a task group Process Group, and declare the current task group unique identifier Group_Id. In the group, create the cleaning conversion processor Processer information and the processing relationship Relationship information after the file stream FlowFile cleaning conversion logic code is successfully executed or the cleaning conversion logic code fails in the processor. The successful processing relationship needs to be connected to the data cleaning conversion processor to be performed in the next step by default; Step (3): In the process of configuring the cleaning conversion processor in step (2), the identifier of the current processor, i.e., Processer_Id, is declared, and the processing operation to be performed by the current processor is configured; Step (4): After steps (1) to (3) are configured, the relevant configuration content in the task group is stored, and the stored content is the processor identification information Processer_Id, the cleaning conversion information, and the processor configuration process sequence information; Step (5): Develop a tracking point processor, define and declare a new processor, which is used to receive the data flow information of the cleaning conversion processor and perform analysis and recording operations. By judging whether the access relationship of the upper-level cleaning conversion processor is successful or failed, the current access node is judged to be successfully accessed or failed to access. By judging the problem data identifier of each data in the data flow FlowFile, it is judged whether the current data is problem data, and the data sum of the data flow FlowFile connected by the current tracking point processor and the sum of the problem data are calculated and stored; Step (6): Connect to the tracking processor. Each cleaning conversion processor that succeeds or fails in step (2) needs to be connected to a tracking processor, and connected to the cleaning conversion processor through Connection and pass in the relevant failure or success data stream. Step (7): Define the tracking point processor, and connect the successful tracking point processor and the failed tracking point processor according to the relationship defined in steps (2)-(6); Step (8): When the buried point storage performs an operation, it is divided into two steps: Step 1: Save the log batch information executed by the current cleaning conversion processor, that is, the flag information Processer_Id of the current cleaning conversion processor, the task group information Group_Id, the currently executed batch information Batch_Id, the total amount of data accessed, the amount of data of the accessed problem data, and the execution time; according to the number of data flow FlowFiles divided by the total amount of data accessed by this data cleaning conversion, the log batch information of each cleaning conversion processor may generate several records; Step 2: Save the data that failed to be cleaned or converted by the current cleaning and conversion processor, i.e., the problematic data, including the source information of the current data, data-related cleaning and conversion information, log batch information, and data information itself; Step (9): After the task configuration is completed, the NIFI task is started and executed. NIFI performs data cleaning and conversion operations based on the task configuration information. The configured tracking processor will record the log information and problem data information separately according to the configuration information; Step (10) can check the data by viewing the log or problem data information, check the abnormal data generated in the cleaning and conversion operation, and analyze the reasons for the abnormal data; The processing operation in step (3) includes filtering the null value of a field in the data record, or converting the time string to the corresponding date format, and creating a new data identifier for each data record in the data stream. This identifier is used to declare whether the current data record is defined as problematic data during the data cleaning or data conversion process; In step (7), the configuration information includes the information of the cleaning conversion processor connected to the current tracking point processor, the information of the current task group, the configuration information related to the cleaning conversion processor connected to the upper layer, the failure or success relationship FlowFile information, and the abnormal data storage data source information.

2. The method according to claim 1, characterized in that The tracking processor records the problem data that causes NIFI execution exception or does not meet the cleaning conversion requirements.

3. The method according to claim 1, characterized in that: In step (6), if some data records cause NIFI to throw an exception, they are also considered to be problematic data. Such data is not identified as a separate problematic data for each data record. All data accessed through the Failure Relationship is defined as problematic data.

4. The method according to claim 3, characterized in that The successful tracking processor needs to filter the data and store the data that fails to be cleaned or converted because it does not meet the processing rules, as well as the data source information and cleaning and conversion information, in the abnormal data storage data source, that is, the data record defined as the problem data in step (3).

5. The method according to claim 3, characterized in that: The failure tracking processor needs to store all the received data information, data source and cleaning conversion information into the exception data storage data source. The failure tracking processor is used to receive data records caused by the system throwing exceptions due to conversion or cleaning triggering exception capture; each task defines and declares a batch number Batch_Id for each cleaning conversion processor when it is executed.

6. The method according to claim 1, characterized in that Each tracking processor records the total amount of data processed by different processors in different batches when using NIFI for data cleaning and conversion, the amount of problem data identified, and the corresponding problem data.

Images