NiFi-based CSV type data processing method and system
The CSV file is completed and cleaned through the data processing method of the NiFi platform, which solves the problem of irregularities in CSV data, improves data quality and processing efficiency, and is suitable for a variety of data analysis scenarios.
Patent Information
- Application Number
- CN202411196552.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-12-10
AI Technical Summary
There are often irregular, wrong or redundant data in CSV files, which affects subsequent data analysis and application effects.
Using NiFi-based CSV data processing method, data verification is performed by obtaining original data, custom field verification, using Javascript or Groovy scripts, and writing qualified data to the qualified library, failing data to the problem library, and using NiFi's Monitoring and management tools for error handling and logging.
It realizes effective completion and cleaning of CSV data, improves the efficiency and accuracy of data processing, has good scalability and customization, and is suitable for scenarios such as big data analysis and data mining.
Smart Images
Figure CN119106027A_ABST
Abstract
Description
Technical Field
[0001] The invention discloses a CSV type data processing method and system based on NiFi, and relates to the technical field of data processing. Background Art
[0002] CSV (Comma-Separated Values) files are a common data exchange format, widely used in data storage and data exchange. However, due to the diversity of data sources and data entry, CSV files often contain a large amount of non-standard, erroneous or redundant data, which will seriously affect the subsequent data analysis and application effects. Summary of the invention
[0003] In view of the problems of the prior art, the present invention provides a CSV type data processing method and system based on NiFi, which completes and cleans CSV format data, and is suitable for various scenarios that require preprocessing and analysis of CSV data, such as big data analysis, data mining, etc.
[0004] The specific scheme proposed by the present invention is:
[0005] The present invention also provides a CSV type data processing method based on NiFi, comprising:
[0006] Step 1: Use NiFi to obtain the original CSV type data from the corresponding library table according to the configured database connection information.
[0007] Step 2: Query the fields and field values required for data verification based on the original CSV type data, and add the fields and field values required for verification to FlowFile Content in the form of key-value, or obtain the response data through the WEB request according to the configured interface information, and customize the field name, and add the customized field name and response data to FlowFile Content in the form of key-value.
[0008] Step 3: Verify the data in FlowFile Content according to the custom Javascript script or Groovy format script.
[0009] Step 4: Write the data that has passed the verification into the qualified database, and write the data that has failed the verification into the problem database for subsequent processing.
[0010] Furthermore, in step 1 of the CSV type data processing method based on NiFi, NiFi is used to obtain original CSV type data from the corresponding library table according to the configured database connection information, including: for the identity verification scenario, the GetFile processor is used to read the original CSV type data from the corresponding library table according to the configured database connection information, and the original CSV type data includes name, certificate information, and identity status information.
[0011] Furthermore, in step 2 of the CSV type data processing method based on NiFi, response data is obtained through a WEB request according to the configured interface information, including: a custom NiFi processor obtains response data through a WEB request, wherein the WEB request method of the custom NiFi processor includes GET and POST; when establishing a remote URL connection, the request method is selected as GET, and the request parameters are spliced after the URL address.
[0012] Furthermore, in step 3 of the NiFi-based CSV type data processing method, the NiFi processor is customized to write a customized verification script, set the script type and script content, and define the script format in the script content.
[0013] Furthermore, the CSV type data processing method based on NiFi also includes step 5, using the monitoring and management tools provided by NiFi to track the data processing process, and perform error handling and logging, wherein the HandleHttpRequest or HandleException processor is used to capture errors that may occur in the data processing process, and the LogAttribute processor is used to record the log information of key steps.
[0014] The present invention also provides a CSV type data processing system based on NiFi, including: an original data acquisition module, an acquisition completion module, a custom script verification module and a data writing module.
[0015] NiFi uses the raw data acquisition module to obtain raw CSV type data from the corresponding library table according to the configured database connection information.
[0016] The collection and completion module queries the fields and field values required for data verification based on the original CSV type data, and adds the fields and field values required for verification to FlowFile Content in the form of key-value, or obtains the response data through WEB request according to the configured interface information, and customizes the field name, and adds the customized field name and response data to FlowFile Content in the form of key-value.
[0017] The custom script verification module uses custom Javascript scripts or Groovy format scripts to verify the data in FlowFile Content.
[0018] The data writing module writes the data that has passed the verification into the qualified database, and the data that has failed the verification into the problem database for subsequent processing.
[0019] Furthermore, the raw data acquisition module of the NiFi-based CSV type data processing system obtains the original CSV type data from the corresponding library table according to the configured database connection information, including: for the identity verification scenario, using the GetFile processor to read the original CSV type data from the corresponding library table according to the configured database connection information, the original CSV type data including name, certificate information, and identity status information.
[0020] Furthermore, the collection and completion module of the CSV type data processing system based on NiFi obtains response data through a WEB request according to the configured interface information, including: a custom NiFi processor obtains response data through a WEB request, wherein the WEB request method of the custom NiFi processor includes GET and POST; when establishing a remote URL connection, the request method is selected as GET, and the request parameters are spliced after the URL address.
[0021] Furthermore, the custom script verification module of the NiFi-based CSV type data processing system is deployed in a custom NiFi processor, which is used to write a custom verification script, set the script type and script content, and define the script format in the script content.
[0022] Furthermore, the CSV type data processing system based on NiFi also includes a supervision module, which uses the monitoring and management tools provided by NiFi to track the data processing process, and performs error handling and log recording, wherein the supervision module is deployed in the HandleHttpRequest or HandleException processor to capture errors that may occur in the data processing process, and the LogAttribute processor uses the supervision module to record log information of key steps.
[0023] The benefits of the present invention are:
[0024] By leveraging the power and flexibility of the NiFi platform, further data completion of CSV data through the interface is achieved, and after data completion, flexible processing is performed through custom scripts. A series of operations such as data reception, preprocessing, parsing, structuring, complex structure processing, post-processing, and output can be completed automatically, greatly improving the efficiency and accuracy of data processing. At the same time, this method has good scalability and customizability, and can easily adapt to different data processing needs and scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 It is a schematic flow chart of the method of the present invention.
[0026] Figure 2 This is a schematic diagram of the interface of the custom NiFi processor AddValueFromWebToCSVRecord involved in the present invention.
[0027] Figure 3 This is a schematic diagram of the interface of the custom NiFi processor UserDefineValidate involved in the present invention. DETAILED DESCRIPTION
[0028] NiFi is a powerful data stream processing tool with flexible configuration and rich processors, which can realize the functions of real-time data collection, cleaning, conversion and transmission. In combination with these features of NiFi, this paper proposes an efficient CSV data cleaning and conversion method, aiming to solve the pain points in CSV data processing and improve data quality and processing efficiency.
[0029] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.
[0030] The present invention also provides a CSV type data processing method based on NiFi, comprising:
[0031] Step 1: Use NiFi to obtain the original CSV type data from the corresponding library table according to the configured database connection information.
[0032] Wherein, using NiFi to obtain the original CSV type data from the corresponding library table according to the configured database connection information may further include: for the identity verification scenario, using the GetFile processor to read the original CSV type data from the corresponding library table according to the configured database connection information, the original CSV type data including name, certificate information, and identity status information.
[0033] Step 2: Query the fields and field values required for data verification based on the original CSV type data, and add the fields and field values required for verification to FlowFile Content in the form of key-value, or obtain the response data through a WEB request according to the configured interface information, and customize the field name, and add the customized field name and response data to FlowFile Content in the form of key-value.
[0034] The response data is obtained through a WEB request according to the configured interface information, including: the custom NiFi processor AddValueFromWebToCSVRecord obtains the response data through a WEB request, wherein the WEB request methods of the custom NiFi processor include GET and POST; when establishing a remote URL connection, the request method is selected as GET, and the request parameters are concatenated after the URL address.
[0035] For example, if the parameter value is a constant, you can directly concatenate it.
[0036] If the parameter value needs to be obtained dynamically from the upstream queue data, the parameter value needs to be replaced with a placeholder and then concatenated after the URL address. Configure the [Parameter Column] and [Source Column] to specify the value that the placeholder needs to be replaced with.
[0037] Ensure that the placeholder is unique in the URL and cannot have the same symbols. For example, if the placeholder is "SFZ", then "SFZ" must not appear in the URL address, and it is case-sensitive. You can use the placeholder "SFZ" to replace the specific certificate number, and "NAME" to replace the specific name. After splicing them into the URL address, configure the parameter column, that is, the correspondence between the placeholder and the source column, that is, which column in the upstream queue data should be used to replace each placeholder.
[0038] Step 3: Verify the data in FlowFile Content according to the custom Javascript script or Groovy format script. The custom NiFi processor UserDefineValidate in step 3 is used to write a custom validation script, set the script type and script content, and define the script format in the script content.
[0039] Step 4: Write the data that has passed the verification into the qualified database, and write the data that has failed the verification into the problem database for subsequent processing.
[0040] Step 5: Use the monitoring and management tools provided by NiFi to track the data processing process and perform error handling and logging. Use the HandleHttpRequest or HandleException processor to capture errors that may occur in the data processing process, and use the LogAttribute processor to record log information for key steps.
[0041] Example 2
[0042] The present invention also provides a CSV type data processing system based on NiFi, including: an original data acquisition module, an acquisition completion module, a custom script verification module and a data writing module.
[0043] NiFi uses the raw data acquisition module to obtain raw CSV type data from the corresponding library table according to the configured database connection information.
[0044] The collection and completion module queries the fields and field values required for data verification based on the original CSV type data, and adds the fields and field values required for verification to FlowFile Content in the form of key-value, or obtains the response data through WEB request according to the configured interface information, and customizes the field name, and adds the customized field name and response data to FlowFile Content in the form of key-value.
[0045] The custom script verification module uses custom Javascript scripts or Groovy format scripts to verify the data in FlowFile Content.
[0046] The data writing module writes the data that has passed the verification into the qualified database, and the data that has failed the verification into the problem database for subsequent processing.
[0047] The information interaction, execution process, etc. of the processor in the above system are based on the same concept as the embodiment of the method of the present invention. For specific contents, please refer to the description in the embodiment of the method of the present invention, and no further description is given here.
[0048] Similarly, the system of the present invention realizes further data completion of CSV data through the interface by utilizing the powerful functions and flexibility of the NiFi platform, and after the data completion, it is flexibly processed by custom scripts. A series of operations such as data reception, preprocessing, parsing, structuring, complex structure processing, post-processing and output can be completed automatically, greatly improving the efficiency and accuracy of data processing. At the same time, the system of the present invention has good scalability and customizability, and can easily adapt to different data processing requirements and scenarios.
[0049] It should be noted that not all steps and modules in the above-mentioned processes and system structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.
[0050] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.
Claims
1. A CSV type data processing method based on NiFi, characterized by include: Step 1: Use NiFi to obtain the original CSV type data from the corresponding library table according to the configured database connection information. Step 2: Query the fields and field values required for data verification based on the original CSV type data, and add the fields and field values required for verification to FlowFile Content in the form of key-value, or obtain the response data through the WEB request according to the configured interface information, and customize the field name, and add the customized field name and response data to FlowFile Content in the form of key-value. Step 3: Verify the data in FlowFile Content according to the custom Javascript script or Groovy format script. Step 4: Write the data that has passed the verification into the qualified database, and write the data that has failed the verification into the problem database for subsequent processing.
2. According to a method for processing CSV type data based on NiFi according to claim 1, it is characterized in that In step 1, NiFi is used to obtain the original CSV type data from the corresponding library table according to the configured database connection information, including: for the identity verification scenario, the GetFile processor is used to read the original CSV type data from the corresponding library table according to the configured database connection information. The original CSV type data includes name, certificate information, and identity status information.
3. A CSV type data processing method based on NiFi according to claim 1, characterized in that In step 2, the response data is obtained through a WEB request according to the configured interface information, including: the custom NiFi processor obtains the response data through a WEB request, wherein the WEB request methods of the custom NiFi processor include GET and POST; when establishing a remote URL connection, the request method is selected as GET, and the request parameters are concatenated after the URL address.
4. A CSV type data processing method based on NiFi according to claim 1, characterized in that In step 3, the custom NiFi processor is used to write a custom verification script, set the script type and script content, and define the script format in the script content.
5. A CSV type data processing method based on NiFi according to claim 1, characterized in that It also includes step 5, using the monitoring and management tools provided by NiFi to track the data processing process, and perform error handling and logging. The HandleHttpRequest or HandleException processor is used to capture errors that may occur in the data processing process, and the LogAttribute processor is used to record log information of key steps.
6. A CSV type data processing system based on NiFi, characterized by include: Original data collection module, collection completion module, custom script verification module and data writing module, NiFi uses the raw data acquisition module to obtain raw CSV type data from the corresponding library table according to the configured database connection information. The collection and completion module queries the fields and field values required for data verification based on the original CSV type data, and adds the fields and field values required for verification to FlowFile Content in the form of key-value, or obtains the response data through WEB request according to the configured interface information, and customizes the field name, and adds the customized field name and response data to FlowFile Content in the form of key-value. The custom script verification module uses custom Javascript scripts or Groovy format scripts to verify the data in FlowFileContent. The data writing module writes the data that has passed the verification into the qualified database, and the data that has failed the verification into the problem database for subsequent processing.
7. A CSV type data processing system based on NiFi according to claim 6, characterized in that The original data acquisition module obtains the original CSV type data from the corresponding library table according to the configured database connection information, including: for the identity verification scenario, using the GetFile processor to read the original CSV type data from the corresponding library table according to the configured database connection information. The original CSV type data includes name, certificate information, and identity status information.
8. A CSV type data processing system based on NiFi according to claim 6, characterized in that The collection and completion module obtains response data through a WEB request according to the configured interface information, including: the custom NiFi processor obtains response data through a WEB request, where the WEB request methods of the custom NiFi processor include GET and POST; when establishing a remote URL connection, select the request method as GET, and splice the request parameters after the URL address.
9. A CSV type data processing system based on NiFi according to claim 6, characterized in that a custom The script verification module is deployed in the custom NiFi processor and is used to write custom verification scripts, set the script type and script content, and define the script format in the script content.
10. A CSV type data processing system based on NiFi according to claim 6, characterized in that It also includes a supervision module, which uses the monitoring and management tools provided by NiFi to track the data processing process and perform error handling and logging. The supervision module is deployed in the HandleHttpRequest or HandleException processor to capture errors that may occur in the data processing process, and the LogAttribute processor uses the supervision module to record log information of key steps.