Method and device for improving performance of flink writing to hdfs

Abstract

This invention discloses a method and apparatus for improving Flink's performance in writing to HDFS. The method includes: during the checkpoint snapshot phase, writing data from the Flink task process's memory to HDFS via file continuation, while simultaneously recording the offset of the written file; during the checkpoint recovery phase, truncating the file to the offset position before continuing writing; during the checkpoint snapshot phase, placing files that need to be closed while being written into an asynchronous queue, and managing asynchronous file closure operations in a state cache; processing using multiple threads; and optimizing CheckpointID by leveraging the strictly monotonically increasing characteristic of CheckpointID. This method and apparatus can ensure the stability of the data warehouse.

Description

Technical Field

[0001] This invention belongs to the field of real-time big data computing and relates to a method and apparatus for improving the data writing performance of real-time data warehouses, and more particularly to a method and apparatus for improving the writing performance of Flink to HDFS. Background Technology

[0002] In real-time data computation, a common approach to data integration on application platforms is pipelined, where a single Flink task writes to multiple streams. During data integration, hundreds of streams may exist within the same Kafka topic, and are written to multiple Hive tables by a single Flink task. This necessitates setting high parallelism for the task, resulting in a massive number of HDFS files being opened simultaneously, exacerbating cluster stability issues. When the Flink task performs a checkpoint, the HDFS sink performs operations such as flushing data and closing files. Consequently, Flink may encounter slow write speeds to individual HDFS files, leading to instability in the HDFS cluster and a decline in the overall throughput performance of the data warehouse. Summary of the Invention

[0003] To address the stability issues of Flink writing to HDFS clusters, this invention provides a method and apparatus for improving Flink's performance in writing to HDFS. This method significantly enhances the ability of Flink tasks to write to HDFS files, improves the efficiency of data writing and reading, and thus ensures the stability of the real-time data warehouse.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] In one embodiment of the present invention, a method for improving Flink's write performance to HDFS is proposed, the method comprising:

[0006] During the checkpoint snapshot phase, the in-memory data of the Flink task process is continued to be written to HDFS through file continuation, while the offset of the written file is recorded.

[0007] During the checkpoint recovery phase, the file is truncated to the offset position before writing continues;

[0008] During the checkpoint snapshot phase, files that need to be closed and are in a write state are placed in an asynchronous queue, and asynchronous file closure operations are managed in a state cache.

[0009] During the checkpoint snapshot phase, processing is performed using multiple threads;

[0010] During the checkpoint snapshot phase, the checkpoint ID is optimized by leveraging its strictly monotonically increasing characteristic.

[0011] Furthermore, the Checkpoint ID has been optimized, including:

[0012] The temporary directory on the sink side is {dump_path} / {next_cp_id}, where next_cp_id is defined as the latest cp_id + 1;

[0013] During the Checkpoint snapshot phase, the Sink saves the latest cp_id to the state cache and updates next_cp_id to cp_id+1.

[0014] During the Checkpoint completion phase, all data in the Sink's temporary directory that is less than or equal to the latest cp_id will be moved to the production directory;

[0015] During the Checkpoint recovery phase, the Sink restores the latest cp_id and moves all data in the Sink's temporary directory that is less than or equal to the latest cp_id to the production directory.

[0016] Furthermore, when a Flink task performs a checkpoint, a rolling strategy is defined so that when a file in the write state is closed, the write state will change to the append write state, and after a successful checkpoint, the append write state will change to the complete write state.

[0017] Furthermore, by setting the checkpoint interval, the speed, size, and quantity of certain files available to downstream systems can be controlled.

[0018] In one embodiment of the present invention, an apparatus for improving Flink's write performance to HDFS is also proposed, the apparatus comprising:

[0019] The file shutdown and frequency reduction module is used to continue writing the Flink task process's memory data to HDFS during the checkpoint snapshot phase by writing the file again, while recording the offset of the written file; during the checkpoint recovery phase, the file is truncated to the offset position and then the writing continues.

[0020] The asynchronous file closing module is used during the checkpoint snapshot phase to put files that need to be closed and are currently being written into an asynchronous queue, and at the same time, to manage asynchronous file closing operations in a state cache.

[0021] The stream parallelization processing module is used to process data using multiple threads during the checkpoint snapshot phase.

[0022] The Checkpoint ID optimization module is used to optimize the Checkpoint ID during the checkpoint snapshot phase by taking advantage of the strictly monotonically increasing characteristic of the Checkpoint ID.

[0023] Furthermore, the Checkpoint ID has been optimized, including:

[0024] The temporary directory on the sink side is {dump_path} / {next_cp_id}, where next_cp_id is defined as the latest cp_id + 1;

[0025] During the Checkpoint snapshot phase, the Sink saves the latest cp_id to the state cache and updates next_cp_id to cp_id+1.

[0026] During the Checkpoint completion phase, all data in the Sink's temporary directory that is less than or equal to the latest cp_id will be moved to the production directory;

[0027] During the Checkpoint recovery phase, the Sink restores the latest cp_id and moves all data in the Sink's temporary directory that is less than or equal to the latest cp_id to the production directory.

[0028] Furthermore, when a Flink task performs a checkpoint, a rolling strategy is defined so that when a file in the write state is closed, the write state will change to the append write state, and after a successful checkpoint, the append write state will change to the complete write state.

[0029] Furthermore, by setting the checkpoint interval, the speed, size, and quantity of certain files available to downstream systems can be controlled.

[0030] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it achieves the aforementioned improvement in Flink's write performance to HDFS.

[0031] In one embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that executes to improve the performance of Flink writing to HDFS.

[0032] Beneficial effects:

[0033] 1. This invention can reduce the frequency of closing HDFS files and greatly reduce the generation of small files.

[0034] 2. This invention adopts an asynchronous improvement in HDFS file closing, so that files can continue to be closed even when the fault is recovered.

[0035] 3. This invention performs parallel processing on real-time streams, increasing processing speed.

[0036] 4. In high-concurrency scenarios, the failure of the entire checkpoint can be quite noticeable due to the timeout or failure of individual Task Snapshots. This invention can improve the fault tolerance and stability of the system in the event of checkpoint failure. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the method for improving Flink's write performance to HDFS according to the present invention;

[0038] Figure 2 This is a schematic diagram of the RollingPolicy of the present invention;

[0039] Figure 3 This is a schematic diagram of the asynchronous queue state storage of the present invention;

[0040] Figure 4 This is a schematic diagram of the shutdown operation during the Checkpoint recovery phase of this invention;

[0041] Figure 5 This is a schematic diagram of the device structure for improving Flink's write performance to HDFS according to the present invention;

[0042] Figure 6 This is a schematic diagram of the computer device structure of the present invention. Detailed Implementation

[0043] The principles and spirit of the present invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0044] Those skilled in the art will recognize that embodiments of the present invention can be implemented as an apparatus, device, node, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0045] The embodiments of the present invention propose a method and apparatus for improving Flink's performance in writing to HDFS, which can significantly improve the ability of Flink tasks to write to HDFS files, improve the efficiency of data writing and reading, and thus ensure the stability of real-time data warehouses.

[0046] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments.

[0047] Figure 1 This is a schematic diagram of the method flow for improving Flink's write performance to HDFS according to the present invention. Figure 1 As shown, the method includes:

[0048] S1. Reduce the frequency of closing files.

[0049] The RollingPolicy defines that when a specified file being written to is closed, the writing state changes to Pending (indicating an ongoing append-writing process). After a successful checkpoint, the append-writing state changes to Finished (indicating a completed write process). Files in the Pending state will become Finished at the next checkpoint. By setting the checkpoint interval, the speed, size, and number of part files available to downstream systems can be controlled. During the Checkpoint Snapshot phase, files being written to are not closed. Instead, data is continuously written to HDFS, with the offset recorded. During the checkpoint recovery phase, files being written to are truncated to the offset position before writing continues, effectively removing redundant data. This reduces the frequency of file closes and significantly reduces the generation of small files.

[0050] like Figure 2 As shown, the file closing operation and the closing timeout operation that are in the writing state are processed asynchronously. Figure 2In the code, InProgressPart: the part (file) being written to; CloseingPartsForCurrentCheckpoint: the closed part (file) of the current checkpoint; PendingFileRecoverablesForCurrentCheckpoint: the recoverable pending file of the current checkpoint.

[0051] S2, Asynchronous improvements to file close.

[0052] Files that are currently being written and require closing are placed in an asynchronous queue. This ensures that the close operation does not block the main processing flow, improving throughput while maintaining HDFS stability. Asynchronous file close operations are managed in a state cache. This allows files to continue being closed even during checkpoint recovery.

[0053] The asynchronous operation here means that the file close operation is handed over to other threads (queues) for processing, while the current thread (queue) continues to the next operation without needing to handle the file close operation.

[0054] like Figure 3 As shown, the asynchronous file close operation during the checkpoint snapshot phase is managed within the state (state cache). The file is retrieved from the state cache and closed in an asynchronous thread (queue). Figure 3 In the code, InProgressPart: the part (file) being written; CloseingPartsForCurrentCheckpoint: the closed part (file) of the current checkpoint; PendingFileRecoverablesForCurrentCheckpoint: the recoverable suspended file of the current checkpoint; PendingFileRecoverablesPreCheckpoint: the file suspended to a recoverable pre-checkpoint.

[0055] like Figure 4 As shown, the closing operation and closing timeout operation of the currently written portion (file) are processed asynchronously. During the Checkpoint recovery phase, the files that need to be closed asynchronously are retrieved from the state (state cache) and closed. Figure 4 `CloseingPartsForCurrentCheckpoint`: Closed parts (files) of the current checkpoint; `PendingFileRecoverablesForCurrentCheckpoint`: Recoverable suspended files of the current checkpoint.

[0056] S3, Stream Parallelization Processing

[0057] For multiple streams, parallel processing is performed in the Checkpoint Snapshot stage. Parallel processing of multiple streams is actually about opening multiple buckets. The Flink framework natively performs serial processing in a loop, but after the parallelization transformation, it uses multi-threading to increase processing speed and reduce the occurrence of Checkpoint timeouts.

[0058] S4. Optimize Checkpoint ID

[0059] By fully leveraging the strictly monotonically increasing nature of Checkpoint IDs, which ensures that each checkpoint is larger than the previous one, temporary data with an ID less than or equal to the current Checkpoint ID can be submitted during the Checkpoint Complete phase. The specific optimization strategy is as follows:

[0060] The temporary directory on the Sink (data stream output mechanism) side is:

[0061] {dump_path} / {next_cp_id}, where {dump_path} is a variable representing the dump path, and {next_cp_id} is a variable representing the current latest Checkpoint ID, which is cp_id+1.

[0062] During the Checkpoint Snapshot phase, the Sink saves the latest cp_id to the State (state cache) and updates the next_cp_id to cp_id+1.

[0063] During the Checkpoint Complete phase, the Sink moves all data in the temporary directory that is less than or equal to the latest cp_id to the official directory (the path parameter passed in by the external caller).

[0064] During the Checkpoint Recover phase, the Sink restores the latest cp_id and moves all data in the temporary directory that is less than or equal to the latest cp_id to the production directory.

[0065] It should be noted that although the operation of the method of the present invention has been described in a specific order in the above embodiments and figures, this does not require or imply that the operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0066] To provide a clearer explanation of the methods for improving Flink's performance in writing to HDFS, a specific embodiment is described below. However, it is worth noting that this embodiment is only for better illustrating the present invention and does not constitute an undue limitation on the present invention.

[0067] Example:

[0068] 1. In Flink's checkpoint snapshot phase code, the file close operation is removed, and a method is added to record the write offset of the file. When the file needs to be closed, this offset is recorded in the state. Assuming Flink writes 1 million data entries to HDFS, requiring 1000 file closes, with each close taking 20ms, this invention can save 20 seconds and eliminate thread blocking caused by waiting to close files.

[0069] 2. When a cluster failure occurs and a checkpoint recovery is needed (checkpoint recovery phase), since the write offset of the file has already been recorded, writing only needs to continue from the offset position, which is equivalent to removing redundant data, thereby reducing the resources and time consumed by rewriting. When a 10G file is written to 8G and the network is interrupted, during checkpoint recovery, the storage offset is obtained, and writing continues from the offset position. At this time, only the remaining 2G of the file needs to be written.

[0070] 3. In step 1, files that need to be closed are placed in an asynchronous queue. The file close operation will be performed in a parallel thread. Asynchronous file close operations will no longer affect the main thread's data stream processing speed, reducing file write concurrency, increasing write speed, and reducing waiting time. This also needs to be managed in the state. Even after a fault recovery, files can continue to be closed. With this optimization, the write time for 1 million data entries is reduced by 20 seconds, and the data stream write performance is improved by 50%.

[0071] 4. Modify the code in the checkpoint snapshot phase by adding a method to implement parallel processing. Write to HDFS in parallel based on the number of file blocks. For example, for a 1GB file, with file blocks set to 128MB, the number of parallel write threads can be 8, potentially improving performance by up to 8 times. After the parallelization modification, processing through multiple threads increases processing speed and reduces checkpoint timeouts.

[0072] 5. Leveraging the strictly monotonically increasing nature of Checkpoint IDs, this invention enables the submission of temporary data with an ID less than or equal to the current Checkpoint ID during the Checkpoint Complete phase. This modification addresses the issue of write latency spikes in HDFS. Previously, Flink was prone to Checkpoint failures due to timeouts or failures of individual Task Snapshots. This invention solves the Checkpoint failure problem, effectively improving the system's fault tolerance and stability.

[0073] Based on the same inventive concept, this invention also proposes an apparatus for improving Flink's write performance to HDFS. The implementation of this apparatus can refer to the implementation of the method described above, and repeated details will not be repeated. The term "module" as used below can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0074] Figure 5 This is a schematic diagram of the device structure for improving Flink's write performance to HDFS according to the present invention. Figure 5 As shown, the device includes:

[0075] The file closing and frequency reduction module 101 is used to continue writing the Flink task process memory data to HDFS in the checkpoint snapshot stage by writing the file again, while recording the offset of the written file; in the checkpoint recovery stage, the file is truncated and the file is truncated to the offset position before continuing to write.

[0076] When a Flink task performs a checkpoint, a rolling strategy is used to define that when a file in the write state is closed, the write state will change to the append write state. After a successful checkpoint, the append write state will change to the complete write state.

[0077] By setting the checkpoint interval, the speed, size, and quantity of certain files available to downstream systems can be controlled.

[0078] The asynchronous file closing module 102 is used to place files that need to be closed and are currently being written into an asynchronous queue during the checkpoint snapshot phase, and to manage asynchronous file closing operations in a state cache.

[0079] The stream parallelization processing module 103 is used to perform multi-threaded processing during the checkpoint snapshot phase.

[0080] The Checkpoint ID optimization module 104 is used to optimize the Checkpoint ID during the checkpoint snapshot phase by taking advantage of the strictly monotonically increasing characteristic of the Checkpoint ID.

[0081] Optimize Checkpoint ID, including:

[0082] The temporary directory on the sink side is {dump_path} / {next_cp_id}, where next_cp_id is defined as the latest cp_id + 1;

[0083] During the Checkpoint snapshot phase, the Sink saves the latest cp_id to the state cache and updates next_cp_id to cp_id+1.

[0084] During the Checkpoint completion phase, all data in the Sink's temporary directory that is less than or equal to the latest cp_id will be moved to the production directory;

[0085] During the Checkpoint recovery phase, the Sink restores the latest cp_id and moves all data in the Sink's temporary directory that is less than or equal to the latest cp_id to the production directory.

[0086] It should be noted that although several modules of the apparatus for improving Flink's write performance to HDFS have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in a single module. Conversely, the features and functions of a single module described above can be further divided and embodied by multiple modules.

[0087] Based on the aforementioned inventive concept, such as Figure 6 As shown, the present invention also proposes a computer device 200, including a memory 210, a processor 220, and a computer program 230 stored in the memory 210 and executable on the processor 220. When the processor 220 executes the computer program 230, it implements the aforementioned method for improving Flink's write performance to HDFS.

[0088] Based on the aforementioned inventive concept, the present invention also proposes a computer-readable storage medium storing a computer program that executes the aforementioned method to improve Flink's performance in writing to HDFS.

[0089] The method and apparatus proposed in this invention improve the performance of Flink writing to HDFS, with HDFS write efficiency more than 10 times that of the original; solves the HDFS file close blocking problem in high-concurrency scenarios; effectively reduces the number of small files in HDFS, and reduces high-frequency writes by more than 10 times compared to the original; and greatly improves the fault tolerance and stability of the system by using checkpoints.

[0090] The technical terms used in this invention are explained as follows:

[0091] Apache Flink is an open-source stream processing framework used in distributed, high-performance, and highly available data stream applications. It can handle both finite and infinite data streams, that is, it can handle bounded and unbounded data streams.

[0092] Hadoop Distributed File System, or HDFS for short, is a distributed file system. HDFS is highly fault-tolerant and designed to be deployed on inexpensive hardware. It also provides high throughput access to application data, making it suitable for applications with very large datasets.

[0093] Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all action streams of data from consumers on a website.

[0094] Topic: Each message published to the Kafka cluster has a category, which is called a Topic.

[0095] Hive is a data warehouse tool based on Hadoop, used for data extraction, transformation, and loading. It is a mechanism for storing, querying, and analyzing large-scale data stored in Hadoop.

[0096] HDFS Sink is a mechanism used to ultimately output data streams to HDFS.

[0097] Flush: Writes memory data to disk.

[0098] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

[0099] Regarding the limitation of the scope of protection of this invention, those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solution of this invention are still within the scope of protection of this invention.

Claims

1. A method for improving Flink's write performance to HDFS, characterized in that, The method includes: During the checkpoint snapshot phase, the in-memory data of the Flink task process is continued to be written to HDFS through file continuation, while the offset of the written file is recorded. During the checkpoint recovery phase, the file is truncated to the offset position before writing continues; During the checkpoint snapshot phase, files that need to be closed and are in a write state are placed in an asynchronous queue, and asynchronous file closure operations are managed in a state cache. During the checkpoint snapshot phase, processing is performed using multiple threads; During the checkpoint snapshot phase, the Checkpoint ID is optimized by leveraging its strictly monotonically increasing characteristic, including: The temporary directory on the sink side is {dump_path} / {next_cp_id}, where next_cp_id is defined as the latest cp_id + 1; During the Checkpoint snapshot phase, the Sink saves the latest cp_id to the state cache and updates next_cp_id to cp_id+1. During the Checkpoint completion phase, all data in the Sink's temporary directory that is less than or equal to the latest cp_id will be moved to the production directory; During the Checkpoint recovery phase, the Sink restores the latest cp_id and moves all data in the Sink's temporary directory that is less than or equal to the latest cp_id to the production directory.

2. The method for improving Flink's write performance to HDFS according to claim 1, characterized in that, When a Flink task performs a checkpoint, a rolling strategy is used to define that when a file in the write state is closed, the write state will change to the append write state. After a successful checkpoint, the append write state will change to the complete write state.

3. The method for improving Flink's write performance to HDFS according to claim 2, characterized in that, By setting the checkpoint interval, the speed, size, and quantity of certain files available to downstream systems can be controlled.

4. An apparatus for improving Flink's write performance to HDFS, characterized in that, The device includes: The file shutdown and frequency reduction module is used to continue writing the Flink task process's memory data to HDFS during the checkpoint snapshot phase by writing the file again, while recording the offset of the written file; during the checkpoint recovery phase, the file is truncated to the offset position and then the writing continues. The asynchronous file closing module is used during the checkpoint snapshot phase to put files that need to be closed and are currently being written into an asynchronous queue, and at the same time, to manage asynchronous file closing operations in a state cache. The stream parallelization processing module is used to process data using multiple threads during the checkpoint snapshot phase. The Checkpoint ID optimization module is used to optimize Checkpoint IDs during the checkpoint snapshot phase by leveraging the strictly monotonically increasing nature of Checkpoint IDs. This includes: The temporary directory on the sink side is {dump_path} / {next_cp_id}, where next_cp_id is defined as the latest cp_id + 1; During the Checkpoint snapshot phase, the Sink saves the latest cp_id to the state cache and updates next_cp_id to cp_id+1. During the Checkpoint completion phase, all data in the Sink's temporary directory that is less than or equal to the latest cp_id will be moved to the production directory; During the Checkpoint recovery phase, the Sink restores the latest cp_id and moves all data in the Sink's temporary directory that is less than or equal to the latest cp_id to the production directory.

5. The apparatus for improving Flink's write performance to HDFS according to claim 4, characterized in that, When a Flink task performs a checkpoint, a rolling strategy is used to define that when a file in the write state is closed, the write state will change to the append write state. After a successful checkpoint, the append write state will change to the complete write state.

6. The apparatus for improving Flink's write performance to HDFS according to claim 5, characterized in that, By setting the checkpoint interval, the speed, size, and quantity of certain files available to downstream systems can be controlled.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1-3.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs the method according to any one of claims 1-3.

Images