A data processing method, apparatus and device

Abstract

The application provides a data processing method, device and equipment, metadata in an acquired flow file FlowFile is split into one or more records Record; a Record to be processed is acquired from the one or more Records, the Record to be processed is processed, and the processed Record is saved to a metadata part of the FlowFile. By splitting the FlowFile into one or more Records and storing information of the Records in the metadata part of the FlowFile, the information is stored in the memory, subsequent filtering, conversion and calculation of the Records can be operated based on the Records, without operating the whole FlowFile, and multiple reading and writing of the FlowFile are reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data collection, in particular to a data processing method, device and equipment. BACKGROUND

[0002] Apache NiFi (hereinafter referred to as NiFi) is a very powerful open source tool for realizing data collection, data cleaning, and loading data from the source end to the destination end (ETL, Extract-Transform-Load). NiFi supports graphical operation, is friendly to process designers, has more than 200 built-in processors to meet most data collection and cleaning needs, supports full-link tracking of data flow, real-time viewing of data flow, and flexible deployment such as single machine, cluster, and docker, and is suitable for various use environments. Due to the many advantages of NiFi, it is widely used in the ETL scene of the production environment.

[0003] When NiFi processes data, the object of processing is a FlowFile. As shown in the figure, a FlowFile contains two parts of content: metadata and business data. The metadata mainly includes the attributes of the FlowFile, such as a unique identifier, a name, a size, and other custom attributes; the business data is the ETL data contained in the FlowFile. The metadata is saved in the memory and can be modified. The business data cannot be modified and points to a real file in the local file system. Figure 1

[0004] In the related art, the object of the entire ETL process is a FlowFile. Since multiple processors are needed to process the data multiple times, the FlowFile needs to be read multiple times and new FlowFiles are generated, which results in a very slow processing speed of the entire ETL process and cannot process a large amount of data with high performance. With the increasing number of data access, the disadvantages of this "all storage and all retrieval" processing method are becoming more and more prominent. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a data processing method, device and equipment, which solves the problem of how to reduce the generation of new FlowFiles and reduce the reading and writing process of FlowFiles.

[0006] To solve the above technical problems, the technical solutions of the embodiments of the present application are as follows:

[0007] ​The embodiment of the present application provides a data processing method, and the method comprises the following steps:

[0008] Splitting metadata in the obtained flow file FlowFile into one or more records Record;

[0009] Obtaining a Record to be processed from the one or more Record, processing the Record to be processed, and saving the processed Record to a metadata part of the FlowFile.

[0010] In one preferred embodiment of the present application, splitting the metadata in the obtained FlowFile into one or more Record comprises the following steps:

[0011] According to the data type of the metadata, the metadata in the obtained flow file FlowFile is split into one or more Record.

[0012] In one preferred embodiment of the present application, according to the data type, the metadata in the obtained FlowFile is split into one or more Record, comprising the following steps:

[0013] If the data type of the metadata in the FlowFile comprises a comma separated value csv file and / or database data, the metadata in the FlowFile is split by line, and one line of metadata is split into one Record;

[0014] And / or,

[0015] If the data type of the metadata in the FlowFile is streaming data, the metadata in the FlowFile is split according to a message in the streaming data, and one message is split into one Record.

[0016] In one preferred embodiment of the present application, the information of the Record is stored in the metadata part of the FlowFile in a key-value manner.

[0017] In one preferred embodiment of the present application, the Record to be processed is obtained from the one or more Record, the Record to be processed is processed, and the processed Record is saved to the metadata part of the FlowFile, comprising the following steps:

[0018] Obtaining a Record to be filtered from the one or more Records, performing filtering processing on the Record to be filtered, and saving the Record obtained through the filtering processing to a metadata part of the FlowFile.

[0019] In one preferred embodiment of the present application, obtaining a Record to be processed from the one or more Records, performing processing on the Record to be processed, and saving the Record obtained through the processing to a metadata part of the FlowFile, further comprises:

[0020] Obtaining a Record to be converted from the one or more Records, performing conversion processing on the Record to be converted, and saving the Record obtained through the conversion processing to a metadata part of the FlowFile.

[0021] In one preferred embodiment of the present application, obtaining a Record to be processed from the one or more Records, performing processing on the Record to be processed, and saving the Record obtained through the processing to a metadata part of the FlowFile, further comprises:

[0022] Obtaining a Record to be calculated from the one or more Records, performing calculation processing on the Record to be calculated, and saving the Record obtained through the calculation processing to a metadata part of the FlowFile.

[0023] In one preferred embodiment of the present application, the method further comprises:

[0024] Storing one or more Records in the metadata part of the FlowFile into a database.

[0025] The embodiment of the present application further provides a data processing device, and the device comprises:

[0026] A first processing module is configured to split the obtained metadata in a FlowFile into one or more Records.

[0027] A second processing module is configured to obtain a Record to be processed from the one or more Records, perform processing on the Record to be processed, and save the Record obtained through the processing to a metadata part of the FlowFile.

[0028] In one preferred embodiment of the present application, the first processing module comprises:

[0029] The record generation unit is configured to split the metadata in the obtained FlowFile into one or more records according to a data type of the metadata.

[0030] In one preferred embodiment of the present application, the record generation unit is specifically configured to split the metadata in the FlowFile by line if the data type of the metadata in the FlowFile includes a comma-separated value (CSV) file and / or database data, and one line of metadata is split into one record.

[0031] If the data type of the metadata in the FlowFile is streaming data, the metadata in the FlowFile is split according to a message in the streaming data, and one message is split into one record.

[0032] In one preferred embodiment of the present application, the information of the record is stored in a metadata part of the FlowFile in a key-value manner.

[0033] In one preferred embodiment of the present application, the second processing module comprises:

[0034] The filtering unit is configured to obtain a record to be filtered from the one or more records, perform filtering processing on the record to be filtered, and save the record after the filtering processing to the metadata part of the FlowFile.

[0035] In one preferred embodiment of the present application, the second processing module further comprises:

[0036] The conversion unit is configured to obtain a record to be converted from the one or more records, perform conversion processing on the record to be converted, and save the record after the conversion processing to the metadata part of the FlowFile.

[0037] In one preferred embodiment of the present application, the second processing module further comprises:

[0038] The calculation unit is configured to obtain a record to be calculated from the one or more records, perform calculation processing on the record to be calculated, and save the record after the calculation processing to the metadata part of the FlowFile.

[0039] In one preferred embodiment of the present application, the second processing module further comprises:

[0040] a storage module configured to store one or more Records in the metadata portion of the FlowFile into a database.

[0041] The embodiment of the present application also provides a data acquisition device, which comprises a processor, a memory and a program stored in the memory and executable in the processor, and the program implements the steps of the data processing method when executed by the processor.

[0042] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the program implements the steps of the data processing method when executed by a processor.

[0043] The embodiment of the present application has the beneficial effects that: by splitting a FlowFile into one or more Records and storing the information of the Records in the metadata portion of the FlowFile, the subsequent filtering, conversion and calculation of the Records can be performed based on the Records, without the need to operate the whole FlowFile, thus avoiding the generation of intermediate FlowFiles and reducing the multiple reading and writing of the FlowFile. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 a structure diagram of a FlowFile in the related art;

[0045] Figure 2 a diagram for filtering, converting and calculating a read FlowFile in the related art;

[0046] Figure 3 a flow diagram of a data processing method in the embodiment of the present application;

[0047] Figure 4 a structure diagram of a data processing device in the embodiment of the present application;

[0048] Figure 5 a diagram for filtering, converting and calculating a read FlowFile in the embodiment of the present application;

[0049] Figure 6 a structure diagram of a data acquisition device in the embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the technical problems, technical solutions and advantages of the embodiments of the present application clearer, the following will be described in detail with reference to the drawings and specific embodiments.

[0051] With reference to the drawings and the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts are within the scope of the present application.

[0052] The term "comprising" and any of its variants, as used in the specification and claims of the present application, are intended to cover the process, method, system, product, or apparatus not only including the steps or units clearly listed, but also including other steps or units that are not clearly listed or inherent to such process, method, product, or apparatus. In addition, the use of "and / or" in the specification and claims of the present application indicates at least one of the connected objects, for example, A and / or B, which includes three cases of separate A, separate B, and A and B both exist.

[0053] In the embodiments of the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design presented as "exemplary" or "for example" in the embodiments of the present application should not be construed as being preferred or advantageous over other embodiments or design solutions. Rather, the use of "exemplary" or "for example" is intended to present concepts in a concrete manner.

[0054] The technology described herein is not limited to Long Time Evolution (LTE) / LTE-Advanced (LTE-A) systems, and can also be used in various wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency-Division Multiple Access (SC-FDMA), and other systems, for example, 5G NR, etc.

[0055] The terms "system" and "network" are often used interchangeably. CDMA systems implement radio technologies such as CDMA2000 and Universal Terrestrial Radio Access (UTRA). UTRA includes Wideband Code Division Multiple Access (WCDMA) and other CDMA variants. TDMA systems implement radio technologies such as the Global System for Mobile Communication (GSM). OFDMA systems implement radio technologies such as Ultra Mobile Broadband (UMB), Evolution-UTRA (E-UTRA), IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, and Flash-OFDM. UTRA and E-UTRA are part of the Universal Mobile Telecommunications System (UMTS). LTE and more advanced LTE versions (such as LTE-A) are newer versions of UMTS that use E-UTRA. UTRA, E-UTRA, UMTS, LTE, LTE-A, and GSM are described in documents from an organization called the 3rd Generation Partnership Project (3GPP). CDMA2000 and UMB are described in documents from an organization called 3rd Generation Partnership Project 2 (3GPP2). The technologies described herein can be used with the systems and radio technologies mentioned above, as well as with other systems and radio technologies.

[0056] In a typical ETL application scenario: the load processor reads the data and encapsulates it into several FlowFiles; three processors filter, transform, and perform calculations on the read FlowFiles; finally, the output processor writes the processed data into the database.

[0057] like Figure 2 As shown, the Processor's processing flow and the FlowFile's flow throughout the entire ETL process include:

[0058] 1. After the processor reads the data, it encapsulates the data into FlowFile A.

[0059] 2. The filter processor reads, filters, and encapsulates the new data as FlowFile B.

[0060] 3. The convert processor reads, converts, and encapsulates the new data as FlowFile C.

[0061] 4. The compute processor reads, computes, and encapsulates the new data as FlowFile D.

[0062] 5. The output processor reads FlowFile D and stores it in the database.

[0063] It can be found that in NiFi, a total of 4 FlowFiles, FlowFile A, FlowFile B, FlowFile C and FlowFile D, are generated, which need to be written to the local file system, which causes corresponding problems:

[0064] 1. Multiple FlowFiles for storing intermediate results, such as FlowFile B and FlowFile C, are generated, which occupy the storage space of the local file system.

[0065] 2. The FlowFile is read and written multiple times: FlowFile A is written once by the load processor and read once by the filter processor; similarly, FlowFile B, C and D are each read and written once. These read and write operations occupy a large amount of I / O bandwidth of the local file system and affect the efficient operation of the flow.

[0066] In view of these problems, a technique is needed to reduce the generation of intermediate flow files FlowFile and the read and write process of FlowFile.

[0067] Therefore, an embodiment of the present application provides a data processing method, as shown in the following figure, which comprises the following steps: Figure 3

[0068] Step 301: Splitting the metadata metadata in the acquired flow file FlowFile into one or more records Record;

[0069] Step 302: Acquiring a to-be-processed Record from the one or more Records, processing the to-be-processed Record, and saving the processed Record to the metadata part of the FlowFile.

[0070] ​The technical scheme provided by the application embodiment is that a flow file FlowFile is divided into a plurality of records Record, and information of the records Record is stored in a metadata part of the FlowFile, which is stored in the memory, and subsequent filtering, conversion and calculation of the records Record can be performed based on the records Record, without operation on the whole FlowFile, so that generation of an intermediate FlowFile is avoided, and multiple reading and writing of the FlowFile are reduced.

[0071] In one preferred embodiment of the application, the metadata in the acquired FlowFile is split into one or more records Record, including:

[0072] According to the data type of the metadata, the metadata in the acquired flow file FlowFile is split into one or more records Record. Record is a logical concept, which is a flow file FlowFile with metadata but without data. After the flow file FlowFile is split into a plurality of records Record, the records Record are new flow files FlowFile, which are independent of the flow file FlowFile before splitting.

[0073] In one preferred embodiment of the application, the metadata in the acquired flow file FlowFile is split into one or more records Record according to the data type, including:

[0074] If the data type of the metadata in the flow file FlowFile includes a comma-separated value csv file and / or database data, the metadata in the FlowFile is split by line, and one line of metadata is split into one Record;

[0075] and / or,

[0076] If the data type of the metadata in the FlowFile is streaming data, the metadata in the FlowFile is split according to a message in the streaming data, and one message is split into one Record. The streaming data refers to kafka data.

[0077] In one preferred embodiment of the application, the information of the Record is stored in the metadata part of the FlowFile in a key-value manner.

[0078] In one preferred embodiment of the present application, a Record to be processed is obtained from the one or more Records, the Record to be processed is processed, and the processed Record is saved to the metadata portion of the FlowFile, including:

[0079] A Record to be filtered is obtained from the one or more Records, the Record to be filtered is filtered, and the filtered Record is saved to the metadata portion of the FlowFile.

[0080] In one preferred embodiment of the present application, a Record to be processed is obtained from the one or more Records, the Record to be processed is processed, and the processed Record is saved to the metadata portion of the FlowFile, further including:

[0081] A Record to be converted is obtained from the one or more Records, the Record to be converted is converted, and the converted Record is saved to the metadata portion of the FlowFile.

[0082] In one preferred embodiment of the present application, a Record to be processed is obtained from the one or more Records, the Record to be processed is processed, and the processed Record is saved to the metadata portion of the FlowFile, further including:

[0083] A Record to be calculated is obtained from the one or more Records, the Record to be calculated is calculated, and the calculated Record is saved to the metadata portion of the FlowFile.

[0084] In one preferred embodiment of the present application, the method further includes:

[0085] Storing one or more Records in the metadata portion of the FlowFile to a database.

[0086] After being processed by each Processor, the metadata of the original FlowFile can be modified without the need to generate a new FlowFile. This avoids the generation of intermediate FlowFiles and reduces the multiple reading and writing of the FlowFile.

[0087] The technical means provided in the various embodiments are applied to the corresponding scenarios, and the process of processing the flow file FlowFile includes:

[0088] Step 1, after reading the data, it is encapsulated as FlowFile A.

[0089] Step 2, after reading FlowFile A, the data is split into several Records (a Record refers to a FlowFile only with a metadata part without a data part) according to the format. The splitting method is determined according to the original data type: the csv file and the database data are split by line, and one line of data is encapsulated as a Record; the streaming data of kafka is split by message, and one message is a Record. The information of the Record is stored in the metadata part of the FlowFile in a key-value manner, and these information can be read and modified in the subsequent Processor processing without generating a new FlowFile.

[0090] Step 3, after taking out the information of Record X from the memory, filtering, the data is still stored in the metadata part of the original FlowFile, that is, Record X1.

[0091] Step 4, after taking out the information of Record X1 from the memory, converting, the data is still stored in the metadata part of the original FlowFile, that is, Record X2.

[0092] Step 5, after taking out the information of Record X2 from the memory, calculating, the data is still stored in the metadata part of the original FlowFile, that is, Record X3.

[0093] Step 6, after taking out the information of Record X3 from the memory, storing it into the database.

[0094] The technical solution provided by the application embodiment only reads once, and then splits into individual Records, and these Records are stored in the memory, which greatly speeds up the processing speed of the ETL and reduces the hardware requirements of the ETL system.

[0095] The embodiment of the application provides a data processing device, as shown in Figure 4 The device comprises:

[0096] The first processing module 401 is configured to split the metadata in the obtained FlowFile into one or more records.

[0097] The second processing module 402 is configured to obtain a record to be processed from the one or more records, process the record to be processed, and save the processed record to the metadata part of the FlowFile.

[0098] In one preferred embodiment of the present application, the first processing module 401 comprises:

[0099] The record generation module is configured to split the metadata in the obtained FlowFile into one or more records according to the data type of the metadata.

[0100] In one preferred embodiment of the present application, the record generation module is specifically configured to split the metadata in the FlowFile by line if the data type of the metadata in the FlowFile comprises a comma-separated value (csv) file and / or database data, and split one line of metadata into one record.

[0101] If the data type of the metadata in the FlowFile is streaming data, the record generation module is specifically configured to split the metadata in the FlowFile by message in the streaming data, and split one message into one record.

[0102] In one preferred embodiment of the present application, the information of the record is stored in the metadata part of the FlowFile in a key-value manner.

[0103] In one preferred embodiment of the present application, the second processing module 402 comprises:

[0104] The filtering module is configured to obtain a record to be filtered from the one or more records, perform filtering processing on the record to be filtered, and save the record after the filtering processing to the metadata part of the FlowFile.

[0105] In one preferred embodiment of the present application, the second processing module 402 further comprises:

[0106] The conversion module is configured to obtain a Record to be converted from the one or more Records, perform conversion processing on the Record to be converted, and save the Record after conversion processing to a metadata part of the FlowFile.

[0107] In one preferred embodiment of the present application, the second processing module 402 further comprises:

[0108] The calculation unit is configured to obtain a Record to be calculated from the one or more Records, perform calculation processing on the Record to be calculated, and save the Record after calculation processing to the metadata part of the FlowFile.

[0109] In one preferred embodiment of the present application, the second processing module 402 further comprises:

[0110] The storage module is configured to store one or more Records in the metadata part of the FlowFile in a database.

[0111] The technical means provided in each embodiment is applied to the corresponding scene, for example, Figure 5 As shown in the figure, the loading processor 501 is used to implement the function of the first processing module 401, wherein the flow file split record processor 502 is used to implement the function of the record generation module in the first processing module 401; and the filtering processor 503, the conversion processor 504 and the calculation processor 505 are used to implement the function of the second processing module 402. Then, the processing of the FlowFile comprises:

[0112] The loading processor 501 encapsulates the data as the FlowFile A after reading the data.

[0113] The flow file split record processor 502 splits the data of the FlowFile A into a plurality of Records according to the format after reading the FlowFile A, wherein one Record refers to the FlowFile only having the metadata part and no data part. The splitting method is determined according to the original data type: the csv file and the database data are split by line, one line of data is encapsulated as one Record; the stream data of the kafka is split by message, one message is one Record. The information of the Record is stored in the metadata part of the FlowFile in the key-value mode, and these information can be read and modified in the subsequent Processor processing without generating a new FlowFile.

[0114] The filter processor 503 takes out the information of Record X from the memory, filters, and stores the data in the metadata part of the original FlowFile, that is, Record X1.

[0115] The conversion processor 504 takes out the information of Record X1 from the memory, converts, and stores the data in the metadata part of the original FlowFile, that is, Record X2.

[0116] The calculation processor 505 takes out the information of Record X2 from the memory, calculates, and stores the data in the metadata part of the original FlowFile, that is, Record X3.

[0117] The output processor 506 takes out the information of Record X3 from the memory and stores it in the database.

[0118] The technical solution provided by the application embodiment only reads once, then unpacks into individual Records, and the Records are stored in the memory, which greatly accelerates the processing speed of ETL and reduces the hardware requirements of the ETL system.

[0119] The application embodiment also provides a data acquisition device, as shown in Figure 6 The application embodiment also provides a data acquisition device, as shown in

[0120] The application embodiment also provides a computer readable storage medium having a computer program stored thereon, and the program is executed by a processor to implement the steps of the data processing method.

[0121] In the prior art, the object of the entire extraction, conversion, and loading to the destination (ETL) flow processing is FlowFile. Since multiple processors are required to process the data multiple times, the FlowFile needs to be read multiple times and new FlowFile is generated, which results in a very slow processing speed of the entire ETL flow and cannot process a large amount of data at high performance. With the increasing number of data access, the disadvantages of the entire storage and entire retrieval processing mode are increasingly prominent.

[0122] The technical solution provided by the application embodiment only reads once, then unpacks into individual Records, and the Records are stored in the memory, which greatly accelerates the processing speed of ETL and reduces the hardware requirements of the ETL system.

[0123] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage, or any other non-transmission medium that can be used to store information that can be accessed by a computing device to be detected. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0124] The above is the preferred embodiment of the present application, it should be pointed out that for those skilled in the art, without departing from the principles of the present application, can also make a number of improvements and refinements, these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A data processing method, characterized in that, The method includes: The metadata in the acquired FlowFile is split into one or more Records; Obtain the Record to be processed from the one or more Records, process the Record to be processed, and save the processed Record to the metadata part of the FlowFile; The metadata in the acquired FlowFile is split into one or more Records, including: Based on the data type of the metadata, the metadata in the obtained FlowFile is split into one or more Records; Based on the data type, the metadata in the obtained FlowFile is split into one or more Records, including: If the data type of the metadata in the FlowFile includes: comma-separated value CSV file and / or database data, then the metadata in the FlowFile is split by row, and one row of metadata is split into one Record; And / or, If the data type of the metadata in the FlowFile is streaming data, then the metadata in the FlowFile is split according to the message in the streaming data, and one message is split into one record; Record is a stream file with metadata but no data (FlowFile).

2. The method according to claim 1, characterized in that, The information in the Record is stored in the metadata section of the FlowFile using a key-value pair.

3. The method according to claim 1, characterized in that, Obtain the Record to be processed from one or more Records, process the Record to be processed, and save the processed Record to the metadata portion of the FlowFile, including: Obtain the Record to be filtered from the one or more Records, perform filtering on the Record to be filtered, and save the filtered Record to the metadata part of the FlowFile.

4. The method according to claim 3, characterized in that, The process includes obtaining a Record to be processed from one or more Records, processing the Record to be processed, and saving the processed Record to the metadata portion of the FlowFile, and also includes: Obtain the Record to be converted from one or more Records, perform conversion processing on the Record to be converted, and save the converted Record to the metadata part of the FlowFile.

5. The method according to claim 3 or 4, characterized in that, The process includes obtaining a Record to be processed from one or more Records, processing the Record to be processed, and saving the processed Record to the metadata portion of the FlowFile, and also includes: Obtain the record to be calculated from one or more records, perform calculation processing on the record to be calculated, and save the calculated record to the metadata part of the FlowFile.

6. The method according to claim 1, characterized in that, The method further includes: Store one or more Records from the metadata portion of the FlowFile into the database.

7. A data processing apparatus, characterized in that, The device includes: The first processing module is used to split the metadata in the acquired FlowFile into one or more Records; The second processing module is used to obtain the Record to be processed from the one or more Records, process the Record to be processed, and save the processed Record to the metadata part of the FlowFile; The first processing module includes: The record generation unit is used to split the metadata in the obtained FlowFile into one or more records according to the data type of the metadata; The record generation unit is specifically configured to: if the data type of the metadata in the FlowFile includes: comma-separated value CSV files and / or database data, then the metadata in the FlowFile is split by line, with one line of metadata split into one Record; and / or, if the data type of the metadata in the FlowFile is streaming data, then the metadata in the FlowFile is split by message in the streaming data, with one message split into one Record; Record is a stream file with metadata but no data (FlowFile).

8. The apparatus according to claim 7, characterized in that, The Record information is stored in the metadata section of the FlowFile using a key-value pair.

9. The apparatus according to claim 7, characterized in that, The second processing module includes: The filtering unit is used to obtain the Record to be filtered from the one or more Records, perform filtering processing on the Record to be filtered, and save the filtered Record to the metadata part of the FlowFile.

10. The apparatus according to claim 9, characterized in that, The second processing module further includes: The conversion unit is used to obtain the Record to be converted from the one or more Records, perform conversion processing on the Record to be converted, and save the converted Record to the metadata part of the FlowFile.

11. The apparatus according to claim 10, characterized in that, The second processing module further includes: The calculation unit is used to obtain the record to be calculated from the one or more records, perform calculation processing on the record to be calculated, and save the calculated record to the metadata part of the FlowFile.

12. The apparatus according to any one of claims 9-11, characterized in that, Also includes: A storage module is used to store one or more records from the metadata portion of the FlowFile into a database.

13. A data acquisition device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 6.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the data processing method as described in any one of claims 1 to 6.

Images