Distributed file analysis method and system

Abstract

The invention discloses a distributed file analysis method and system, and belongs to the technical field of file processing, distributed parallel processing is adopted, distributed file analysis is realized based on Flink + Kafka, and for a to-be-analyzed HDFS (Hadoop Distributed File System) file, the analysis efficiency is improved. And distributing file information needing to be analyzed to different hosts for executing analysis tasks in a balanced manner through a Kafka distributed message queue and an Flink distributed calculation engine, and executing file analysis operation in parallel. According to the invention, the distributed parallel analysis of the file can be realized, the high availability of the file analysis is ensured, the consistency of the file analysis is realized, and the file analysis is ensured not to be repeated or lost.

Description

Technical Field

[0001] The present invention relates to the technical field of file processing, and in particular to a distributed file parsing method and system. Background Art

[0002] Traditional file processing technology is mainly based on multi-threading and file IO technology to complete the parsing of a large number of files, parsing structured files and unstructured files into row data that can be used for database reading. In the real production environment, there are usually a large number of files that need to be parsed. Usually, Java applications or distributed computing engines directly read the HDFS file system according to the file path to complete the file parsing. The file parsing mode based on Java applications is a stand-alone mode, so there is a single point of failure, and the high availability of file parsing cannot be guaranteed. At the same time, the consistency of file parsing cannot be guaranteed. There may be repeated parsing of files. In addition, when there are too many files in some HDFS directories, the corresponding parsing client will be overloaded with processing pressure, resulting in an increase in the overall parsing delay and processing backlogs. Summary of the invention

[0003] The technical task of the present invention is to address the above shortcomings and provide a distributed file parsing method and system, which can realize distributed parallel parsing of files, ensure high availability of file parsing, achieve consistency of file parsing, and ensure that file parsing is not duplicated or lost.

[0004] The technical solution adopted by the present invention to solve its technical problem is:

[0005] A distributed file parsing method adopts the concept of distributed parallel processing and implements distributed file parsing based on Flink+Kafka. For HDFS files to be parsed, the file information to be parsed is evenly distributed to different hosts that perform parsing tasks through the Kafka distributed message queue and the Flink distributed computing engine. The file parsing operations are performed in parallel, which improves the file parsing efficiency, reduces the parsing delay, ensures the consistency of file parsing, and realizes high-availability parsing of files.

[0006] Furthermore, the implementation of this method is as follows:

[0007] By decomposing the file parsing process, after the file collection is written into the HDFS distributed file system, the file storage information such as the file name, the HDFS directory where the file is located, the HDFS connection address, the kerberos authentication ticket, etc. is formatted and written to the Kafka message queue as a parsing task message. Then, based on the Flink distributed computing engine consuming the parsing task messages in Kafka, the file parsing tasks are evenly scheduled to multiple nodes for distributed parallel parsing.

[0008] Furthermore, the specific implementation steps of the method include:

[0009] 1) File parsing task generation:

[0010] After writing files to HDFS, the upstream file production system writes metadata based on the HDFS file into the Kafka message queue. Each piece of file information corresponds to a file parsing task.

[0011] 2) File parsing task distribution:

[0012] Develop a Flink DataStream application and integrate the file parsing method into the Flink application operator. Use the Kafka interface provided by the Flink DataStream API to consume the file parsing task information in Kafka. Flink distributes the file parsing task information received from Kafka to the operator subtasks of each node. The operator task of each node applies the assigned file parsing task information to its integrated file parsing method to achieve distributed parallel parsing of tasks.

[0013] Furthermore, the metadata based on the HDFS file includes the file name, the HDFS directory where the file is located, the HDFS host address, the HDFS host port, and the Kerberos authentication ticket information.

[0014] Furthermore, each parsing task uses the file name, the HDFS directory where the file is located, and the HDFS host address as a unique primary key identifier;

[0015] Format each file parsing task information and write it to the Kafka message queue.

[0016] The present invention also claims to protect a distributed file parsing system, which implements distributed file parsing based on Flink+Kafka. For HDFS files to be parsed, the file information to be parsed is evenly distributed to different hosts that execute parsing tasks through the Kafka distributed message queue and the Flink distributed computing engine, and the file parsing operations are performed in parallel.

[0017] Furthermore, the system includes a file parsing task generation module and a file parsing task distribution module.

[0018] The file parsing task generation module:

[0019] After writing files to HDFS, the upstream file production system writes metadata based on the HDFS file into the Kafka message queue. Each piece of file information corresponds to a file parsing task.

[0020] The file parsing task distribution module:

[0021] Develop a Flink DataStream application and integrate the file parsing method into the Flink application operator. Use the Kafka interface provided by the Flink DataStream API to consume the file parsing task information in Kafka. Flink distributes the file parsing task information received from Kafka to the operator subtasks of each node. The operator task of each node applies the assigned file parsing task information to its integrated file parsing method to achieve distributed parallel parsing of tasks.

[0022] Furthermore, based on the metadata of the HDFS file, it includes the file name, the HDFS directory where the file is located, the HDFS host address, the HDFS host port, and the Kerberos authentication ticket information;

[0023] Each parsing task uses the file name, the HDFS directory where the file is located, and the HDFS host address as a unique primary key identifier; each file parsing task information is formatted and then written to the Kafka message queue.

[0024] The present invention also claims protection for a distributed file parsing device, comprising: at least one memory and at least one processor;

[0025] The at least one memory is used to store a machine-readable program;

[0026] The at least one processor is used to call the machine-readable program to implement the above method.

[0027] The present invention also claims protection for a computer-readable medium having computer instructions stored thereon, which implement the above method when executed by a processor.

[0028] Compared with the prior art, the distributed file parsing method and system of the present invention have the following beneficial effects:

[0029] From the perspective of high-availability distributed parallel processing, based on HDFS file metadata data as the analysis basis, comprehensive consideration of the high availability, consistency, concurrency, and balance of file processing, distributed file parsing is implemented based on Flink+Kafka to improve file parsing efficiency, reduce parsing latency, ensure consistency of file parsing, and achieve high-availability file parsing.

[0030] Currently, this algorithm is used to improve the file parsing efficiency of operators in a certain province on the server, shortening the file parsing delay from the original 30 minutes to 10 minutes, and ensuring the consistency of file parsing, achieving no loss or duplication of file parsing, and ensuring the high availability of file parsing in the event of failure of some nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flowchart of a distributed file parsing method provided by an embodiment of the present invention;

[0032] Figure 2 It is a diagram of metadata based on HDFS files provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0034] A distributed file parsing method is implemented based on Flink+Kafka. It adopts the concept of distributed parallel processing. For HDFS files to be parsed, the file information to be parsed is evenly distributed to different hosts that perform parsing tasks through the Kafka distributed message queue and the Flink distributed computing engine. The file parsing operations are performed in parallel, which improves the file parsing efficiency, reduces the parsing delay, ensures the consistency of file parsing, and realizes high-availability parsing of files.

[0035] The implementation of this method is as follows:

[0036] By decomposing the file parsing process, after the file collection is written into the HDFS distributed file system, the file storage information such as the file name, the HDFS directory where the file is located, the HDFS connection address, and the kerberos authentication ticket are formatted and written to the Kafka message queue as a parsing task message. Then, based on the Flink distributed computing engine, the parsing task messages in Kafka are consumed, and the file parsing tasks are evenly scheduled to multiple nodes for distributed parallel parsing. This algorithm can realize distributed parallel parsing of files, ensure high availability of file parsing, achieve consistency of file parsing, and ensure that file parsing is not duplicated or lost.

[0037] Combination Figure 1 As shown, the specific implementation steps of the method include:

[0038] 1. File parsing task generation:

[0039] Meta information based on HDFS files, including file name, HDFS directory where the file is located, HDFS host address, HDFS host port, and Kerberos authentication ticket information, such as Figure 2 shown.

[0040] After writing the file to HDFS, the upstream file production system will write the metadata based on the HDFS file, including the file name, the HDFS directory where the file is located, the HDFS host IP, the HDFS host port, the Kerberos authentication ticket and other information into the Kafka message queue. Each piece of file information corresponds to a file parsing task; each parsing task uses the file name, the HDFS directory where the file is located, and the HDFS host address as the unique primary key identifier; each piece of file parsing task information is formatted and then written to the Kafka message queue.

[0041] 2. File parsing task distribution:

[0042] Develop a Flink DataStream application and integrate the file parsing method into the Flink application operator. Use the Kafka interface provided by the Flink DataStream API to consume the file parsing task information in Kafka. Flink distributes the file parsing task information received from Kafka to the operator subtasks of each node. The operator task of each node applies the assigned file parsing task information to its integrated file parsing method to achieve distributed parallel parsing of tasks.

[0043] The embodiment of the present invention also provides a distributed file parsing system, which implements distributed file parsing based on Flink+Kafka. For HDFS files to be parsed, the file information to be parsed is evenly distributed to different hosts that perform parsing tasks through the Kafka distributed message queue and the Flink distributed computing engine, and the file parsing operation is performed in parallel. This improves the efficiency of file parsing, reduces the parsing delay, ensures the consistency of file parsing, and realizes high-availability parsing of files.

[0044] The system implements distributed file parsing as follows:

[0045] By decomposing the file parsing process, after the file collection is written into the HDFS distributed file system, the file storage information such as the file name, the HDFS directory where the file is located, the HDFS connection address, and the kerberos authentication ticket are formatted and written to the Kafka message queue as a parsing task message. Then, based on the Flink distributed computing engine, the parsing task messages in Kafka are consumed, and the file parsing tasks are evenly scheduled to multiple nodes for distributed parallel parsing. This algorithm can realize distributed parallel parsing of files, ensure high availability of file parsing, achieve consistency of file parsing, and ensure that file parsing is not duplicated or lost.

[0046] The system specifically includes a file parsing task generation module and a file parsing task distribution module.

[0047] The file parsing task generation module:

[0048] Meta information based on HDFS files, including file name, HDFS directory where the file is located, HDFS host address, HDFS host port, and Kerberos authentication ticket information;

[0049] After writing the file to HDFS, the upstream file production system will write the metadata based on the HDFS file, including the file name, the HDFS directory where the file is located, the HDFS host IP, the HDFS host port, the Kerberos authentication ticket and other information into the Kafka message queue. Each piece of file information corresponds to a file parsing task; each parsing task uses the file name, the HDFS directory where the file is located, and the HDFS host address as the unique primary key identifier; each piece of file parsing task information is formatted and then written to the Kafka message queue.

[0050] The file parsing task distribution module:

[0051] Develop a Flink DataStream application and integrate the file parsing method into the Flink application operator. Use the Kafka interface provided by the Flink DataStream API to consume the file parsing task information in Kafka. Flink distributes the file parsing task information received from Kafka to the operator subtasks of each node. The operator task of each node applies the assigned file parsing task information to its integrated file parsing method to achieve distributed parallel parsing of tasks.

[0052] An embodiment of the present invention further provides a distributed file parsing device, comprising: at least one memory and at least one processor;

[0053] The at least one memory is used to store a machine-readable program;

[0054] The at least one processor is used to call the machine-readable program to implement the distributed file parsing method described in the above embodiment.

[0055] The embodiment of the present invention further provides a computer-readable medium, on which computer instructions are stored, and when the computer instructions are executed by a processor, the distributed file parsing method described in the above embodiment is implemented. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0056] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0057] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0058] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0059] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0060] The present invention is shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. A distributed file parsing method, characterized in that: Distributed parallel processing is adopted to implement distributed file parsing based on Flink+Kafka. For the HDFS files to be parsed, the file information to be parsed is evenly distributed to different hosts that perform the parsing tasks through the Kafka distributed message queue and the Flink distributed computing engine, and the file parsing operations are performed in parallel.

2. A distributed file parsing method according to claim 1, characterized in that: The implementation of this method is as follows: By decomposing the file parsing process, after the file collection is written into the HDFS distributed file system, the file storage information is formatted and written into the Kafka message queue as a parsing task message. Then, based on the Flink distributed computing engine, the parsing task messages in Kafka are consumed and the file parsing tasks are evenly scheduled to multiple nodes for distributed parallel parsing.

3. A distributed file parsing method according to claim 1 or 2, characterized in that: The specific implementation steps of this method include: 1) File parsing task generation: After writing files to HDFS, the upstream file production system writes metadata based on the HDFS file into the Kafka message queue. Each piece of file information corresponds to a file parsing task. 2) File parsing task distribution: Develop a Flink DataStream application and integrate the file parsing method into the Flink application operator. Use the Kafka interface provided by the Flink DataStream API to consume the file parsing task information in Kafka. Flink distributes the file parsing task information received from Kafka to the operator subtasks of each node. The operator task of each node applies the assigned file parsing task information to its integrated file parsing method to achieve distributed parallel parsing of tasks.

4. A distributed file parsing method according to claim 3, characterized in that: Meta information based on HDFS files, including file name, HDFS directory where the file is located, HDFS host address, HDFS host port, and Kerberos authentication ticket information.

5. A distributed file parsing method according to claim 4, characterized in that: Each parsing task uses the file name, the HDFS directory where the file is located, and the HDFS host address as the unique primary key identifier; Format each file parsing task information and write it to the Kafka message queue.

6. A distributed file parsing system, characterized in that: Distributed file parsing is implemented based on Flink+Kafka. For HDFS files to be parsed, the file information to be parsed is evenly distributed to different hosts that perform parsing tasks through the Kafka distributed message queue and the Flink distributed computing engine, and the file parsing operations are performed in parallel.

7. A distributed file parsing system according to claim 1, characterized in that: The system includes a file parsing task generation module and a file parsing task distribution module. The file parsing task generation module: After writing files to HDFS, the upstream file production system writes metadata based on the HDFS file into the Kafka message queue. Each piece of file information corresponds to a file parsing task. The file parsing task distribution module: Develop a Flink DataStream application and integrate the file parsing method into the Flink application operator. Use the Kafka interface provided by the Flink DataStream API to consume the file parsing task information in Kafka. Flink distributes the file parsing task information received from Kafka to the operator subtasks of each node. The operator task of each node applies the assigned file parsing task information to its integrated file parsing method to achieve distributed parallel parsing of tasks.

8. A distributed file parsing system according to claim 6, characterized in that: The HDFS file-based meta information includes file name, HDFS directory where the file is located, HDFS host address, HDFS host port, and Kerberos authentication ticket information; Each parsing task uses the file name, the HDFS directory where the file is located, and the HDFS host address as a unique primary key identifier; each file parsing task information is formatted and then written to the Kafka message queue.

9. A distributed file parsing device, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is used to call the machine-readable program to implement the method described in any one of claims 1 to 5.

10. A computer-readable medium, characterized in that The computer readable medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images