NiFi-based method for efficiently processing complex JSON data

By pre-processing, parsing and structured processing of JSON data on the NiFi platform, using recursive algorithms and NiFi processors, the problem of inefficient JSON data processing in the existing technology is solved, and efficient and stable complex JSON data processing is achieved. , adapt to the needs of different scenarios.

CN119025583APending Publication Date: 2024-11-26SHANGHAI INSPUR CLOUD COMPUTING SERVICE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411088347.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing JSON data processing methods are inefficient, difficult to cope with large-scale and complex data processing needs, and are prone to errors.

Method used

Using NiFi-based method, JSON data is received through the input port, pre-processing, parsing, structured processing and post-processing, and using recursive algorithms and processors provided by NiFi, such as EvaluateJSONPath, UpdateRecord, SplitRecord, MergeRecord and RouteOnAttribute, to realize the data Automate processing and conversion.

Benefits of technology

It improves the efficiency and stability of JSON data processing, realizes efficient extraction and conversion of complex structure data, enhances the accuracy and flexibility of data processing, and adapts to different data processing needs and scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025583A_ABST
    Figure CN119025583A_ABST
Patent Text Reader

Abstract

The invention discloses a method for efficiently processing complex JSON data based on NiFi, and relates to the technical field of data processing. Comprising the steps of 1, receiving JSON data through an input port of NiFi; step 2, the received JSON data is preprocessed; the method comprises the steps of 1, preprocessing JSON data, 2, analyzing the preprocessed JSON data by using a JSON analyzer in NiFi, 3, analyzing the preprocessed JSON data by using a JSON analyzer in NiFi, and 4, carrying out structured processing on the analyzed JSON data: traversing the JSON data with a complex structure by using a recursive algorithm, extracting required information according to a preset rule, and carrying out structured processing on the analyzed JSON data. Carrying out data conversion processing by utilizing a conversion function according to the extracted required information to convert a non-standard data format into a unified data format, and storing the converted data into structured data; and 5, post-processing the structured data, and outputting the processed data to a specified target position.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention discloses a method, which relates to the technical field of data processing, and in particular to a method for efficiently processing complex JSON data based on NiFi. Background Art

[0002] With the rapid development of information technology, the diversification of data exchange formats has become an important feature in the field of data processing. Among them, the JSON (JavaScript Object Notation) data format is widely used in Internet data exchange due to its lightweight, self-descriptive, easy to read and write characteristics. However, with the continuous expansion of data scale and the increasing complexity of data structure, existing JSON data processing methods usually rely on specific programming languages ​​and tools, and require writing a lot of code to complete data parsing, extraction and conversion operations. This method is not only inefficient, but also prone to errors, and it is difficult to cope with large-scale and complex JSON data processing needs. Summary of the invention

[0003] In view of the problems of the prior art, the present invention provides a method for efficiently processing complex JSON data based on NiFi, thereby improving the processing efficiency and stability of complex JSON.

[0004] The specific scheme proposed by the present invention is:

[0005] The present invention provides a method for efficiently processing complex JSON data based on NiFi, comprising:

[0006] Step 1: Receive JSON data through NiFi's input port;

[0007] Step 2: Preprocess the received JSON data;

[0008] Step 3: Use the JSON parser in NiFi to parse the preprocessed JSON data.

[0009] Step 4: Structural processing of the parsed JSON data: For JSON data with complex structures, a recursive algorithm is used to traverse, and the required information is extracted according to the preset rules. According to the extracted required information, the conversion function is used to convert the non-standard data format into a unified data format, and the converted data is stored in the structured data;

[0010] Step 5: Post-process the structured data and output the processed data to the specified target location.

[0011] Furthermore, the JSON data received in step 1 of the method for efficiently processing complex JSON data based on NiFi includes real-time data streams and historical data stored in files or databases.

[0012] Furthermore, in step 2 of the method for efficiently processing complex JSON data based on NiFi, preprocessing is performed, including: data cleaning, format conversion and verification, removing irrelevant characters and correcting grammatical errors through data cleaning; converting the data into a unified format through format conversion to facilitate subsequent processing; and checking the integrity and accuracy of the data through verification.

[0013] Furthermore, in step 3 of the method for efficiently processing complex JSON data based on NiFi, the preprocessed JSON data is parsed, including: configuring a JSONPath expression using the EvaluateJSONPath parser provided by NiFi, extracting required information from the JSON data according to the JSONPath expression,

[0014] The properties of EvaluateJSONPath are configured to configure the JSONPath expression, including:

[0015] Where Destination: select flowfile-attribute, and set the extracted value as the attribute of FlowFile.

[0016] Return Type: Select scalar,

[0017] Path Not Found Behavior: Set to warn to issue a warning if the path cannot be found.

[0018] Add dynamic properties to specify the JSON fields and JsonPath expressions to extract.

[0019] Furthermore, in step 5 of the method for efficiently processing complex JSON data based on NiFi, the structured data is post-processed, including: using the UpdateRecord processor provided by NiFi to convert the structured data or add new fields; using the SplitRecord processor and the MergeRecord processor to split and merge the structured data; using the RouteOnAttribute processor to route the structured data to different output ports according to its attributes.

[0020] The present invention also provides a device for efficiently processing complex JSON data based on NiFi, including a receiving module, a preprocessing module, a parsing module, a structured processing module and a post-processing module.

[0021] The receiving module receives JSON data through NiFi's input port;

[0022] The preprocessing module preprocesses the received JSON data;

[0023] The parsing module uses the JSON parser in NiFi to parse the preprocessed JSON data.

[0024] The structured processing module performs structured processing on the parsed JSON data: for JSON data with complex structures, a recursive algorithm is used to traverse it, and the required information is extracted according to the preset rules. According to the extracted required information, the conversion function is used to perform data conversion processing to convert the non-standard data format into a unified data format, and the converted data is stored in the structured data;

[0025] The post-processing module performs post-processing on the structured data and outputs the processed data to the specified target location.

[0026] Furthermore, the JSON data received by the receiving module of the device for efficiently processing complex JSON data based on NiFi includes real-time data streams and historical data stored in files or databases.

[0027] Furthermore, the preprocessing module of the device for efficiently processing complex JSON data based on NiFi performs preprocessing, including: data cleaning, format conversion and verification, removing irrelevant characters and correcting grammatical errors through data cleaning; converting data into a unified format through format conversion to facilitate subsequent processing; and checking the integrity and accuracy of the data through verification.

[0028] Furthermore, the parsing module of the device for efficiently processing complex JSON data based on NiFi parses the preprocessed JSON data, including: configuring a JSONPath expression using the EvaluateJSONPath parser provided by NiFi, extracting required information from the JSON data according to the JSONPath expression,

[0029] The properties of EvaluateJSONPath are configured to configure the JSONPath expression, including:

[0030] Where Destination: select flowfile-attribute, and set the extracted value as the attribute of FlowFile.

[0031] Return Type: Select scalar,

[0032] Path Not Found Behavior: Set to warn to issue a warning if the path cannot be found.

[0033] Add dynamic properties to specify the JSON fields and JsonPath expressions to extract.

[0034] Furthermore, the post-processing module of the device for efficiently processing complex JSON data based on NiFi performs post-processing on the structured data, including: using the UpdateRecord processor provided by NiFi to convert the structured data or add new fields; using the SplitRecord processor and the MergeRecord processor to split and merge the structured data; using the RouteOnAttribute processor to route the structured data to different output ports according to its attributes.

[0035] The benefits of the present invention are:

[0036] The present invention realizes efficient processing of complex JSON data by utilizing the powerful functions and flexibility of the NiFi platform. It can automatically complete a series of operations such as data reception, preprocessing, parsing, structuring, complex structure processing, post-processing and output, greatly improving the efficiency and accuracy of data processing. At the same time, the present invention has good scalability and customizability, and can easily adapt to different data processing requirements and scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0038] As a powerful and easy-to-use data processing and distribution system, NiFi is highly scalable and flexible and suitable for a variety of data processing scenarios.

[0039] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0040] Example 1

[0041] The specific scheme proposed by the present invention is:

[0042] The present invention provides a method for efficiently processing complex JSON data based on NiFi, comprising:

[0043] Step 1: Receive JSON data through NiFi's input port. This can be achieved by configuring NiFi's GetFile, ConsumeKafka and other processors to adapt to different data sources. The received JSON data can include real-time data streams and historical data stored in files or databases. This data can come from a variety of sources, such as sensors, databases, APIs, etc.

[0044] Step 2: Preprocess the received JSON data, including data cleaning, format conversion, and verification. Data cleaning mainly involves removing irrelevant characters and correcting grammatical errors; format conversion converts data into a unified format for subsequent processing; and verification checks the integrity and accuracy of the data.

[0045] Step 3: Use the JSON parser in NiFi to parse the preprocessed JSON data.

[0046] You can further use the EvaluateJSONPath processor provided by NiFi to complete this step. By configuring the JSONPath expression of the processor, you can flexibly extract the required information from the JSON data and convert it into structured data. These structured data are stored in the form of key-value pairs, which is convenient for subsequent processing and analysis. The relevant configuration examples are as follows:

[0047] Configure the properties of EvaluateJSONPath:

[0048] Destination: Select flowfile-attribute to set the extracted value as an attribute of the FlowFile.

[0049] Return Type: Select scalar, since a single value is extracted.

[0050] Path Not Found Behavior: Can be set to warn to issue a warning if the path cannot be found.

[0051] Add dynamic properties to specify the JSON fields and JsonPath expressions to extract. For example:

[0052] name->$.name

[0053] age->$.age

[0054] address_street->$.address.street

[0055] first_phone_number->$.phoneNumbers[0].number

[0056] Step 4: Structuring the parsed JSON data: For JSON data with complex structures, a recursive algorithm is used to traverse it, and the required information, such as date, time, etc., is extracted according to the preset rules. According to the extracted required information, the conversion function is used to convert the non-standard data format into a unified data format, and the converted data is stored in the structured data;

[0057] Step 5: Post-process the structured data and output the processed data to the specified target location.

[0058] The structured data can be further post-processed, including: using the UpdateRecord processor provided by NiFi to transform the structured data or add new fields; using the SplitRecord processor and the MergeRecord processor to split and merge the structured data; using the RouteOnAttribute processor to route the structured data to different output ports according to its attributes. By combining these processors, various complex data processing tasks can be flexibly completed. Finally, the processed data is output to the specified target location, such as a database, file, message queue, etc. This data can be used for subsequent analysis, visualization, or machine learning tasks.

[0059] Example 2

[0060] The present invention also provides a device for efficiently processing complex JSON data based on NiFi, including a receiving module, a preprocessing module, a parsing module, a structured processing module and a post-processing module.

[0061] The receiving module receives JSON data through NiFi's input port;

[0062] The preprocessing module preprocesses the received JSON data;

[0063] The parsing module uses the JSON parser in NiFi to parse the preprocessed JSON data.

[0064] The structured processing module performs structured processing on the parsed JSON data: for JSON data with complex structures, a recursive algorithm is used to traverse it, and the required information is extracted according to the preset rules. According to the extracted required information, the conversion function is used to perform data conversion processing to convert the non-standard data format into a unified data format, and the converted data is stored in the structured data;

[0065] The post-processing module performs post-processing on the structured data and outputs the processed data to the specified target location.

[0066] As the information interaction and execution process between the modules in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.

[0067] Similarly, the device of the present invention realizes efficient processing of complex JSON data by utilizing the powerful functions and flexibility of the NiFi platform. It can automatically complete a series of operations such as data reception, preprocessing, parsing, structuring, complex structure processing, post-processing and output, greatly improving the efficiency and accuracy of data processing. At the same time, the device of the present invention has good scalability and customizability, and can easily adapt to different data processing requirements and scenarios.

[0068] It should be noted that not all steps and modules in the above-mentioned processes and device structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0069] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A method for efficiently processing complex JSON data based on NiFi, characterized by include: Step 1: Receive JSON data through NiFi's input port; Step 2: Preprocess the received JSON data; Step 3: Use the JSON parser in NiFi to parse the preprocessed JSON data. Step 4: Structural processing of the parsed JSON data: For JSON data with complex structures, a recursive algorithm is used to traverse, and the required information is extracted according to the preset rules. According to the extracted required information, the conversion function is used to convert the non-standard data format into a unified data format, and the converted data is stored in the structured data; Step 5: Post-process the structured data and output the processed data to the specified target location.

2. According to claim 1, a method for efficiently processing complex JSON data based on NiFi is characterized in that The JSON data received in step 1 includes real-time data streams and historical data stored in files or databases.

3. According to claim 1, a method for efficiently processing complex JSON data based on NiFi is characterized in that Preprocessing is performed in step 2, including: data cleaning, format conversion and verification. Data cleaning removes irrelevant characters and corrects grammatical errors; format conversion converts data into a unified format for subsequent processing; and verification checks the integrity and accuracy of the data.

4. According to claim 1, a method for efficiently processing complex JSON data based on NiFi is characterized in that In step 3, the preprocessed JSON data is parsed, including: using the EvaluateJSONPath parser provided by NiFi to configure the JSONPath expression, extracting the required information from the JSON data according to the JSONPath expression, The properties of EvaluateJSONPath are configured to configure the JSONPath expression, including: Where Destination: select flowfile-attribute, and set the extracted value as the attribute of FlowFile. Return Type: Select scalar, Path Not Found Behavior: Set to warn to issue a warning if the path cannot be found. Add dynamic properties to specify the JSON fields and JsonPath expressions to extract.

5. A method for efficiently processing complex JSON data based on NiFi according to claim 1, characterized in that In step 5, the structured data is post-processed, including: using the UpdateRecord processor provided by NiFi to convert the structured data or add new fields; using the SplitRecord processor and the MergeRecord processor to split and merge the structured data; using the RouteOnAttribute processor to route the structured data to different output ports according to its attributes.

6. A device for efficiently processing complex JSON data based on NiFi, characterized by It includes receiving module, pre-processing module, parsing module, structured processing module and post-processing module. The receiving module receives JSON data through NiFi's input port; The preprocessing module preprocesses the received JSON data; The parsing module uses the JSON parser in NiFi to parse the preprocessed JSON data. The structured processing module performs structured processing on the parsed JSON data: for JSON data with complex structures, a recursive algorithm is used to traverse it, and the required information is extracted according to the preset rules. According to the extracted required information, the conversion function is used to perform data conversion processing to convert the non-standard data format into a unified data format, and the converted data is stored in the structured data; The post-processing module performs post-processing on the structured data and outputs the processed data to the specified target location.

7. The device for efficiently processing complex JSON data based on NiFi according to claim 6 is characterized in that The JSON data received by the receiving module includes real-time data streams and historical data stored in files or databases.

8. The device for efficiently processing complex JSON data based on NiFi according to claim 6 is characterized in that The preprocessing module performs preprocessing, including: data cleaning, format conversion and verification. Through data cleaning, irrelevant characters are removed and grammatical errors are corrected; through format conversion, data is converted into a unified format for subsequent processing; through verification, the integrity and accuracy of the data are checked.

9. The device for efficiently processing complex JSON data based on NiFi according to claim 6, characterized in that The parsing module parses the preprocessed JSON data, including: using the EvaluateJSONPath parser provided by NiFi to configure the JSONPath expression, extracting the required information from the JSON data according to the JSONPath expression, The properties of EvaluateJSONPath are configured to configure the JSONPath expression, including: Where Destination: select flowfile-attribute, and set the extracted value as the attribute of FlowFile. Return Type: Select scalar, Path Not Found Behavior: Set to warn to issue a warning if the path cannot be found. Add dynamic properties to specify the JSON fields and JsonPath expressions to extract.

10. The device for efficiently processing complex JSON data based on NiFi according to claim 6, characterized in that The post-processing module performs post-processing on structured data, including: using the UpdateRecord processor provided by NiFi to convert structured data or add new fields; using the SplitRecord processor and MergeRecord processor to split and merge structured data; using the RouteOnAttribute processor to route the structured data to different output ports according to its attributes.