JSON (JavaScript Object Notation) data processing method and device based on NiFi and large model, and medium

Abstract

The invention discloses a JSON (JavaScript Object Notation) data processing method and equipment based on NiFi and a large model and a medium, belongs to the technical field of crossing of data security and artificial intelligence, and aims to solve the technical problems that a traditional data processing process lacks complex semantic understanding ability, the large model is high in real-time reasoning delay and poor in dynamic data adaptability, and the data processing efficiency is high. According to the technical scheme, the method comprises the following steps: reading structured data through a NiFi processor; processing to-be-processed fields in the read structured data through a NiFi processor, assembling the to-be-processed fields into a JSON file, and storing the JSON file on a server; transmitting the generated JSON file to the large model, processing the JSON file through the large model, returning the processed file in the same format, and writing the processed file back to the server; and reading the processed JSON file through the NiFi processor, reading contents in the JSON file, and selecting to replace the original data or re-store the original data according to service requirements.

Description

Technical Field

[0001] The present invention relates to the technical field of intersection of data security and artificial intelligence, and specifically to a JSON data processing method, device, and medium based on NiFi and a large model. Background Art

[0002] Large models, such as language and vision models based on the Transformer architecture, have achieved remarkable results in fields such as natural language processing and image recognition, demonstrating powerful semantic understanding and pattern recognition capabilities. However, directly processing data streams with these large models results in high latency (over seconds), failing to meet real-time requirements (such as millisecond-level processing of financial transaction logs). Feeding these large models with the full data set wastes computing power and poses the risk of privacy breaches.

[0003] JSON (JavaScript Object Notation), designed based on a subset of ECMAScript, is an open-standard file format and data exchange format. It is easy for humans to read and write, and easy for machines to parse and generate. JSON is also a common data format in data management and can be converted between multiple data formats.

[0004] Traditional data processing frameworks, such as Apache NiFi, while efficient and reliable in data processing, lack deep semantic understanding capabilities. When handling complex pattern recognition tasks, such as real-time fraud detection in financial transactions and failure prediction in industrial equipment, traditional approaches are limited to simple data rule matching and statistical analysis, making it difficult to uncover the deeper semantic information behind the data, resulting in low recognition accuracy and efficiency. Summary of the Invention

[0005] The technical task of the present invention is to provide a JSON data processing method, device and medium based on NiFi and large models to solve the problems of lack of complex semantic understanding ability, high real-time reasoning delay of large models and poor adaptability to dynamic data in traditional data processing.

[0006] The technical task of the present invention is achieved in the following way: a JSON data processing method based on NiFi and a large model, the method is as follows:

[0007] Read structured data through NiFi processors;

[0008] The NiFi processor processes the fields to be processed in the read structured data, assembles them into JSON files, and saves them on the server;

[0009] The generated JSON file is passed to the large model, which processes the JSON file and returns the processed file in the same format and writes it back to the server;

[0010] Read the processed JSON file through the NiFi processor, read the content in the JSON file, and choose to replace the original data or re-save it according to business needs.

[0011] Preferably, the structured data includes library table data, JSON data and Excel data; wherein, the library table data and Excel data can be converted into JSON data.

[0012] Preferably, the NiFi processor includes a data extraction component QueryDatabaseTable, a field processing component, a large model interaction component, and a streaming data generation component;

[0013] Among them, the data extraction component is used to read library table data, JSON data and Excel data; among them, the data extraction component includes the QueryDatabaseTable component, and the data extraction component is used to read MySQL library table data;

[0014] The field processing component is used to process the fields in the JSON file, assemble it into a JSON file, and save it to the server;

[0015] The large model interaction component is used to input the original JSON file, return the processed JSON file after being processed by the large model, and save the JSON data in the server;

[0016] The streaming data generation component is used to read the JSON data processed by the large model and generate streaming data to be processed by the subsequent business layer.

[0017] Preferably, the field processing component has the following functions:

[0018] ①Support custom configuration processing fields: configure the processing of fields containing names;

[0019] ②Support saving data as JSON and setting the save path on the server;

[0020] ③Supports setting the number of entries saved in each JSON file for analysis of large models (mainly to enable large models to maximize resource utilization and process data).

[0021] Preferably, the field processing component works as follows:

[0022] (1) Get the content in flowflie;

[0023] (2) Define the JSON path;

[0024] (3) Read the content in flowflie;

[0025] (4) Convert the JSON object into a formatted string;

[0026] (5) Write the JSON string to a file to generate a JSON file.

[0027] More preferably, the large model interaction component includes a large model calling interface and a custom input prompt, and the large model calling interface and the custom input prompt are used to interact with the large model.

[0028] More optimally, the working process of the large model interaction component is as follows:

[0029] (1) Read the JSON file content line by line;

[0030] (2) Assembly interface prompt;

[0031] (3) Assemble the content into the interface and write;

[0032] (4) Request the large model to call the interface;

[0033] (5) Get the corresponding content and save it to a JSON file;

[0034] (6) Define the JSON path;

[0035] (7) Get the content in flowflie;

[0036] (8) Convert the JSON object into a formatted string;

[0037] (9) Write the JSON string to a file to generate a JSON file.

[0038] Preferably, the JSON data reading component reads the file content line by line, and if the file content is not empty, converts the file into a JSON object and then writes the content into NiFi's flowfile.

[0039] An electronic device comprising: a memory and at least one processor;

[0040] wherein the memory stores computer-executable instructions;

[0041] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the JSON data processing method based on NiFi and a large model as described above.

[0042] A computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the JSON data processing method based on NiFi and a large model as described above is implemented.

[0043] The JSON data processing method, device, and medium based on NiFi and large models of the present invention have the following advantages:

[0044] (1) This invention is based on the JSON data processing technology that collaborates with the Apache NiFi data flow framework and the large model (LLM). By using JSON format files as the medium, NiFi and the large model can better utilize their own resources and process as much data as possible under limited resources. It is suitable for scenarios in the fields of finance, medical care, government affairs, etc. where structured data is converted to JSON data and data processing is required;

[0045] (2) The present invention extracts data through NiFi, constructs the extracted data into a JSON file, and then passes the constructed JSON file to the large model for processing and returns the processed JSON data to the local file. NiFi then reads the processed JSON file data and writes it to the business library to complete the data processing. This solves the contradiction between the processing capabilities of the existing data framework and the high latency and high resource consumption of the large model. By building a deep collaborative mechanism between the Apache NiFi stream processing engine and the large model, JSON data processing in complex data scenarios is realized;

[0046] (3) The present invention improves accuracy: compared with the traditional solution using fixed algorithm processing, the processing accuracy is improved;

[0047] (4) The present invention optimizes resources: compared with the scenario where a large model processes a single piece of data, processing multiple pieces of data in JSON greatly improves processing efficiency;

[0048] (V) The present invention expands native functions to develop custom components to connect to large models, and uses JSON files as intermediate media for data processing, which solves the technical problems of lack of complex semantic understanding capabilities, high real-time reasoning latency of large models, and poor adaptability to dynamic data in traditional data processing; among them, large model services include but are not limited to natural language processing (NLP) models or pre-trained language models (such as GPT series, BERT, etc.), which are used to identify sensitive information in unstructured text. NiFi's processors include custom processors or third-party extension components for integrating large model API calls and processing logic. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The present invention will be further described below with reference to the accompanying drawings.

[0050] AttachmentFigure 1 A flowchart of the JSON data processing method based on NiFi and large models;

[0051] Attachment Figure 2 Screenshot of the interface for reading MySQL database table;

[0052] Attachment Figure 3 Take a screenshot of the interface of the field processing component;

[0053] Attachment Figure 4 Take a screenshot of the interface of the large model interactive component;

[0054] Attachment Figure 5 A screenshot of the interface of the component that generates streaming data. DETAILED DESCRIPTION

[0055] The following detailed description is made of the JSON data processing method, device and medium based on NiFi and a large model of the present invention with reference to the accompanying drawings and specific embodiments.

[0056] Example 1:

[0057] As attached Figure 1 As shown, this embodiment provides a JSON data processing method based on NiFi and a large model, and the method is as follows:

[0058] S1. Read structured data through NiFi processor;

[0059] S2. Process the fields to be processed in the read structured data through the NiFi processor, assemble them into a JSON file, and save it on the server;

[0060] S3 passes the generated JSON file to the large model, which processes the JSON file and returns the processed file in the same format, writing it back to the server.

[0061] S4. Read the processed JSON file through the NiFi processor, read the content in the JSON file, and choose to replace the original data or save it again according to business needs.

[0062] The structured data in this embodiment includes library table data, JSON data and Excel data; wherein, the library table data and Excel data can be converted into JSON data.

[0063] The NiFi processor in this embodiment includes a data extraction component QueryDatabaseTable, a field processing component, a large model interaction component, and a streaming data generation component;

[0064] The data extraction component is used to read database table data, JSON data and Excel data; the data extraction component includes the QueryDatabaseTable component, which is used to read MySQL database table data, as shown in the attached figure. Figure 2 As shown;

[0065] The field processing component is used to process the fields in the JSON file, assemble it into a JSON file, and save it to the server;

[0066] The large model interaction component is used to input the original JSON file, return the processed JSON file after being processed by the large model, and save the JSON data in the server;

[0067] The streaming data generation component is used to read the JSON data processed by the large model and generate streaming data to be processed by the subsequent business layer.

[0068] As attached Figure 3 As shown, the field processing component in this embodiment has the following functions:

[0069] ①Support custom configuration processing fields: configure the processing of fields containing names;

[0070] ②Support saving data as JSON and setting the save path on the server;

[0071] ③Supports setting the number of entries saved in each JSON file for analysis of large models (mainly to enable large models to maximize resource utilization and process data).

[0072] The key codes of the field processing component in this embodiment are as follows:

[0073]

[0074]

[0075] The large model interaction component in this embodiment includes a large model calling interface and a custom input prompt, which are used to interact with the large model, as shown in the attached figure. Figure 4 As shown, the URL is the URL for accessing the large model, the recognition type is customized, such as recognizing text and audio, the recognition field is configured to identify the field in the JSON, and the customized large model prompt is written in the prompt.

[0076] The key codes of the large model interaction component are as follows:

[0077]

[0078]

[0079]

[0080]

[0081] For scenarios involving key information identification and processing, other tasks such as data conversion, data merging, and key information extraction can all be handled by constructing different prompts to process JSON data, and then obtain the data processed by the large model for subsequent business processing, as shown in the attached example. Figure 5 shown.

[0082] The key codes of the large model interaction component in this embodiment are as follows:

[0083]

[0084]

[0085] The JSON data reading component in this embodiment reads the file content line by line. If the file content is not empty, the file is converted into a JSON object and the content is written to NiFi's flowfile.

[0086] The complete process of this embodiment is: data reading: reading JSON, library tables and other data --> generating JSON files: processing the data into JSON files and saving them on the server --> large model processing JSON file processing: using the large model to process the JSON file content and save the processed JSON --> reading the processed JSON data --> data business processing: writing the processed data to the business library to provide external services.

[0087] Example 2:

[0088] This embodiment also provides an electronic device, including: a memory and at least one processor;

[0089] wherein the memory stores computer-executable instructions;

[0090] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the JSON data processing method based on NiFi and a large model in any embodiment of the present invention.

[0091] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0092] The memory can be used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, the memory can also include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state memory devices.

[0093] Example 3:

[0094] This embodiment also provides a computer-readable storage medium storing a plurality of instructions, which are loaded by a processor to cause the processor to execute the JSON data processing method, device, and medium based on NiFi and large models in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, wherein the storage medium stores software program code that implements the functions of any of the above embodiments, and a computer (or CPU or MPU) of the system or device can read and execute the program code stored in the storage medium.

[0095] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute part of the present invention.

[0096] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0097] In addition, it should be clear that the functions of any of the above embodiments can be achieved not only by executing the program code read by the computer, but also by enabling the operating system operating on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0098] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then based on the instructions of the program code, the CPU installed on the expansion board or expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A JSON data processing method based on NiFi and a large model, characterized in that, The method is as follows: Read structured data through NiFi processors; The NiFi processor processes the fields to be processed in the read structured data, assembles them into JSON files, and saves them on the server; The generated JSON file is passed to the large model, which processes the JSON file and returns the processed file in the same format and writes it back to the server; Read the processed JSON file through the NiFi processor, read the content in the JSON file, and choose to replace the original data or re-save it according to business needs.

2. The JSON data processing method based on NiFi and large model according to claim 1, characterized in that Structured data includes library table data, JSON data, and Excel data; among them, library table data and Excel data can both be converted into JSON data.

3. The JSON data processing method based on NiFi and large model according to claim 1 is characterized in that, The NiFi processor includes data extraction components, field processing components, large model interaction components, and streaming data generation components; Among them, the data extraction component is used to read library table data, JSON data and Excel data; among them, the data extraction component includes the QueryDatabaseTable component, and the data extraction component is used to read MySQL library table data; The field processing component is used to process the fields in the JSON file, assemble it into a JSON file, and save it to the server; The large model interaction component is used to input the original JSON file, return the processed JSON file after being processed by the large model, and save the JSON data in the server; The streaming data generation component is used to read the JSON data processed by the large model and generate streaming data to be processed by the subsequent business layer.

4. The JSON data processing method based on NiFi and large model according to claim 3 is characterized in that, The field processing component has the following functions: ①Support custom configuration processing fields: configure the processing of fields containing names; ②Support saving data as JSON and setting the save path on the server; ③Supports setting the number of entries saved in each JSON file for analysis of large models.

5. The JSON data processing method based on NiFi and large model according to claim 3 is characterized in that, The working process of the field processing component is as follows: (1) Get the content in flowflie; (2) Define the JSON path; (3) Read the content in flowflie; (4) Convert the JSON object into a formatted string; (5) Write the JSON string to a file to generate a JSON file.

6. The JSON data processing method based on NiFi and large model according to claim 3 is characterized in that, The large model interaction component includes a large model calling interface and a custom input prompt, which are used to interact with the large model.

7. The JSON data processing method based on NiFi and large model according to claim 3 is characterized in that, The working process of the large model interaction component is as follows: (1) Read the JSON file content line by line; (2) Assembly interface prompt; (3) Assemble the content into the interface and write; (4) Request the large model to call the interface; (5) Get the corresponding content and save it to a JSON file; (6) Define the JSON path; (7) Get the content in flowflie; (8) Convert the JSON object into a formatted string; (9) Write the JSON string to a file to generate a JSON file.

8. The JSON data processing method based on NiFi and large model according to claim 3 is characterized in that, The JSON data reading component reads the file content line by line. If the file content is not empty, it converts the file into a JSON object and then writes the content to NiFi's flowfile.

9. An electronic device, characterized in that: include: memory and at least one processor; wherein the memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the JSON data processing method based on NiFi and a large model according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the JSON data processing method based on NiFi and a large model as described in any one of claims 1 to 8 is implemented.

Images