Improve Flink HashMapStateBackend method by using key-value separation

Abstract

This invention discloses an improved method for Flink HashMapStateBackend using key-value separation. The method stores data keys and values ​​separately, with keys stored in storage nodes and values ​​stored in files on disk. The method includes: building and maintaining indexes for storage nodes according to access order; periodically monitoring memory usage, and performing key-value separation on storage nodes according to the index whenever memory usage exceeds a specified threshold; storing the separated values ​​in files according to the order of separation; considering modified values ​​of data that has undergone key-value separation as invalid, periodically calculating the failure rate, and cleaning up files with failure rates exceeding a specified threshold whenever the failure rate reaches that threshold. This invention solves the performance problem of Flink EmbeddedRocksDBStateBackend in Flink dual-stream Join job scenarios.

Description

Technical Field

[0001] This invention relates to the technical field of Flink stream computation and key-value storage, and in particular to a method for improving Flink HashMapStateBackend by using key-value separation. Background Technology

[0002] Flink provides two out-of-the-box state backends: FlinkHashMapStateBackend, which stores all data in memory, and Flink EmbeddedRocksDBStateBackend, which primarily stores data on disk. Flink HashMapStateBackend, as Flink's all-in-memory state backend, performs all data accesses in memory, offering excellent access performance. However, its data capacity is limited by the available memory in the cluster, making it suitable only for small-scale Flink jobs. As the data volume in a Flink job increases, it is prone to memory overflow exceptions, causing continuous job restarts and impacting production stability. Flink EmbeddedRocksDBStateBackend, storing data primarily on disk, can accommodate large amounts of data and is a general-purpose state backend for Flink jobs. However, because disk access performance is significantly lower than memory access performance, Flink EmbeddedRocksDBStateBackend's performance is inferior to FlinkHashMapStateBackend.

[0003] Key-value separation was originally used to optimize storage systems that use LSM-tree storage structures. Its core idea is to separate values ​​from the LSM-tree storage structure and store them separately, thereby reducing the amount of data stored in the LSM-tree storage structure, improving the performance of accessing the LSM-tree storage structure, and reducing the overhead of merging files in the LSM-tree storage structure.

[0004] The Flink dual-stream Join scenario is a classic Flink job scenario characterized by large data volumes and the following features: values ​​occupy significantly more storage space than keys, and values ​​are accessed far less frequently than keys. Currently, the Flink dual-stream Join scenario can only use FlinkEmbeddedRocksDBStateBackend. This invention improves FlinkHashMapStateBackend by employing key-value separation, providing a higher-performance state backend suitable for Flink dual-stream Join jobs. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings and deficiencies of the prior art and propose a method to improve Flink HashMapStateBackend by using key-value separation. This method stores the keys and values ​​of the data separately, keeping all keys in the memory storage nodes and storing the values ​​in files on the disk. On the one hand, this allows memory to hold more keys, thereby expanding the amount of data that FlinkHashMapStateBackend can hold, making it applicable to Flink dual-stream Join job scenarios. On the other hand, since the frequency of accessing values ​​is much lower than the frequency of accessing keys, key-value separation keeps the keys in memory, fully leveraging the advantages of memory as the primary storage for data access and its high access performance.

[0006] To achieve the above objectives, the technical solution provided by this invention is as follows: An improved Flink HashMapStateBackend method using key-value separation is employed. The data stored in the Flink HashMapStateBackend consists of two parts: a key and a value. Each data is stored in a memory-based storage node. This method improves Flink HashMapStateBackend by separating the key and value of the data, keeping all keys in the storage node and storing the values ​​in files on disk. The specific implementation of this method includes the following steps:

[0007] S1: Locate the storage node containing the data in the HashMap structure based on the data's key, and create and maintain an index for the storage node according to the access order;

[0008] S2: Periodically detect memory usage. Whenever memory usage exceeds a specified threshold, trigger key-value splitting. Based on the index built for storage nodes according to the access order, start key-value splitting from the least accessed storage node, separate the values ​​in the storage node and store them in the file, and store a number that can uniquely identify the value in the storage node. This number strictly increments according to the order in which the values ​​are separated. Key-value splitting ends when the number of separated values ​​reaches the specified amount of data.

[0009] S3: Values ​​separated from storage nodes are stored in files according to the order in which they were separated;

[0010] S4: For data that has been separated into key and value, modifying the value of the data will not modify the file. Instead, the value is considered invalid, and the number of invalid values ​​is recorded. The failure rate is calculated periodically. Whenever the failure rate reaches a specified threshold, files with a failure rate higher than the specified threshold are selected for cleaning. The still valid values ​​in the selected files are written to a new file, and the selected files are replaced with the new file. The selected files are deleted after being replaced, thus completing the cleaning.

[0011] Furthermore, the specific steps of step S1 are as follows:

[0012] S11: Based on the key of the data, find the storage node where the data is located in the HashMap structure;

[0013] S12: Organize the storage nodes into a doubly linked list structure according to the access order and index them. Each storage node has two pointers, which point to the storage node that was accessed before and the storage node that was accessed after. The storage node at the head of the doubly linked list is the most recently accessed storage node, and the storage node at the tail of the doubly linked list is the least recently accessed storage node.

[0014] S13: When accessing data, the storage node corresponding to the data becomes the most recently accessed storage node. Modify the pointer of the storage node and the pointers of related storage nodes, thereby moving the storage node to the head of the doubly linked list.

[0015] Furthermore, the specific steps of step S2 are as follows:

[0016] S21: Use the garbage collection time and memory usage provided by the Java Virtual Machine as indicators to measure memory usage. Periodically check the above two indicators. When the two indicators exceed the specified threshold, trigger key-value separation.

[0017] S22: When key-value separation is triggered, based on the index built for the storage nodes in the access order, starting from the least accessed storage node, key-value separation is performed on the storage nodes in the access order, the value of the storage node is separated and stored in the file, and a number that can uniquely identify the value is stored in the storage node. This number strictly increases in the order in which the values ​​are separated. Key-value separation ends when the separated values ​​reach the specified data volume.

[0018] Furthermore, the specific steps of step S3 are as follows:

[0019] S31: The values ​​separated from the storage node are first batched in the write buffer. When the batched values ​​reach the specified data volume, these batched values ​​are stored in the file currently being written.

[0020] S32: When the file currently being written reaches the specified file size, the file becomes an immutable file, and a new file is created to store the value that was detached later.

[0021] Furthermore, the specific steps of step S4 are as follows:

[0022] S41: Record the total number of values ​​contained in each file and the total number of invalid values ​​contained in the file. The failure rate of the file is equal to the total number of invalid values ​​contained in the file divided by the total number of values ​​contained in the file. Calculate the total failure rate of the separated values ​​periodically. The total failure rate is equal to the total number of invalid values ​​contained in all files divided by the total number of values ​​contained in all files.

[0023] S42: Whenever the total failure rate reaches a specified threshold, files with a failure rate higher than the specified threshold are selected for cleaning. The method for selecting files is as follows: First, the file with the highest failure rate is included in the selection. Then, the failure rate of the file with the closest creation time to the file is checked. If it is higher than the specified threshold, it is included in the selection. Then, the failure rate of the file with the closest creation time to the newly included file is checked. If it is higher than the specified threshold, it is included in the selection. The above process is repeated until the failure rate of the file with the closest creation time to the newly included file is no higher than the specified threshold, at which point the selection stops.

[0024] S43: Clean up invalid values ​​contained in the selected files: By searching all storage nodes, determine the valid values ​​contained in the selected files, write these valid values ​​into a new file, and replace the selected files with the new file. The selected files are then deleted after being replaced, thus completing the cleanup.

[0025] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0026] 1. This invention combines the characteristics of data in Flink dual-stream Join job scenarios to design a state backend suitable for Flink dual-stream Join job scenarios, which has better access performance than Flink EmbeddedRocksDBStateBackend.

[0027] 2. This invention improves Flink HashMapStateBackend by using key-value separation, expands the amount of data that Flink HashMapStateBackend can hold, and broadens the application scenarios of Flink HashMapStateBackend.

[0028] 3. This invention designs a method for cleaning up invalid values ​​that is compatible with the system architecture of Flink HashMapStateBackend which uses key-value separation. This method largely overcomes the problem that cleaning up invalid values ​​becomes a performance bottleneck in key-value separation schemes, reduces the impact of cleaning up invalid values ​​on accessed data, and conforms to the characteristics of high throughput and low latency of Flink stream computing. Attached Figure Description

[0029] Figure 1This is a diagram of the improved Flink HashMapStateBackend system architecture.

[0030] Figure 2 This is a schematic diagram of the storage node index.

[0031] Figure 3 This is a diagram illustrating the process of cleaning up invalid values. Detailed Implementation

[0032] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0033] This embodiment discloses a method for improving Flink HashMapStateBackend using key-value separation. The improved Flink HashMapStateBackend system architecture is as follows: Figure 1 As shown, it contains four modules: the original storage module FlinkHashMapStateBackend, where all storage nodes are located; the key-value separation module, which is responsible for periodically detecting memory usage and separating values ​​from storage nodes; the separated value storage module, which is responsible for storing the values ​​separated from storage nodes; and the invalid value cleanup module, which is responsible for selecting files with a failure rate higher than a specified threshold for cleanup.

[0034] The specific implementation of this method includes the following steps:

[0035] S1: Locate the storage node containing the data in the HashMap structure based on the data's key, and create and maintain an index for the storage node according to the access order. The specific steps are as follows:

[0036] like Figure 2 As shown, all storage nodes are indexed in a doubly linked list structure according to the access order. The index can be built and maintained using the Least Recently Used (LRU) algorithm: when accessing data, first find the storage node corresponding to the data, modify the pointer of that storage node and the pointers of related storage nodes, including the pointers of the two storage nodes that the current storage node points to, and the storage node currently at the head of the doubly linked list, so that the current storage node replaces the current storage node at the head of the doubly linked list and becomes the new head node of the doubly linked list.

[0037] When performing key-value separation on a storage node, it is equivalent to accessing the data corresponding to that storage node. Therefore, it is necessary to modify the pointer of that storage node and the pointers of related storage nodes, and move the storage node to the head of the doubly linked list.

[0038] S2: Periodically monitor memory usage. Whenever memory usage exceeds a specified threshold, trigger key-value splitting. Based on the index built for storage nodes according to access order, start key-value splitting from the least accessed storage node, separating the values ​​from that node and storing them in a file. Also store a unique identifier for each value in the storage node, with this unique identifier strictly incrementing according to the order in which the values ​​were split. Key-value splitting continues until the specified amount of data has been split. Details are as follows:

[0039] The system periodically detects memory usage, for example, by collecting the garbage collection time and memory usage provided by the Java Virtual Machine every 2 seconds.

[0040] The unique identifier of the value can be guaranteed by the following semantics to strictly increment the number according to the order in which the values ​​are separated: when the number is represented by a 64-bit binary number, for each separated value, the high 40 bits of the binary number use a binary number representing the time of separation of the value, and the low 24 bits use a binary number corresponding to a counter. The counter is incremented by one for each separated value.

[0041] The separation ends when the separated values ​​reach the specified data volume as follows: For each storage node being separated, the space occupied by the separated values ​​is calculated. During the key-value separation process, the space occupied by all separated values ​​is summed to obtain the separated data volume. When the summed separated data volume reaches the specified data volume, the key-value separation ends.

[0042] S3: Values ​​separated from the storage node are stored in the file according to the order in which they were separated. The specifics are as follows:

[0043] like Figure 1 As shown, the separated values ​​are first batched in the write buffer. When the batched values ​​in the write buffer reach a specified data size, such as 4KB, these batched separated values ​​are stored in the file currently being written.

[0044] Once the file currently being written reaches a specified size, such as 256MB, it becomes an immutable file. A new file is then created to store the values ​​that were subsequently separated. Immutable files can be stored in an array in chronological order of their creation.

[0045] When accessing a detached value, a binary search can be performed using a number that uniquely identifies the value: first, a binary search is performed in the write buffer; if the value is not found, a binary search is performed in the file currently being written; if the value is not found, a binary search is performed in the immutable file; once the immutable file containing the value is determined, a binary search is performed in that immutable file.

[0046] S4: For data that has been key-value separated, modifying the value of this data will not modify the file; instead, the value is considered invalid, and the number of invalid values ​​is recorded. The invalidity rate is calculated periodically. Whenever the invalidity rate reaches a specified threshold, files with an invalidity rate higher than the threshold are selected for cleanup. The still valid values ​​from the selected files are written to a new file, and the new file replaces the selected file. The selected file is deleted after being replaced, thus completing the cleanup. Specific details are as follows:

[0047] Each time a key-value pair of data is modified, the file storing that value is located, and the total number of invalid values ​​contained in that file is incremented by one. The timed calculation of the failure rate can be performed every 10,000 modifications to the key-value pair. When the total failure rate reaches a predetermined threshold, such as 50%, files with a failure rate exceeding the threshold are selected for cleanup. This cleanup process can be handled by an asynchronous thread.

[0048] The process for selecting files with a failure rate exceeding a specified threshold is as follows: Calculate the failure rate of all files, find the file with the highest failure rate and include it in the selection. Then, check the failure rate of files most recently created; if it exceeds the specified threshold, include it in the selection. Next, check the failure rate of files most recently created; if it exceeds the specified threshold, include it in the selection. Repeat this process until the failure rate of files most recently created does not exceed the specified threshold, at which point the selection stops. When cleaning up invalid values ​​contained in the selected files, determine the valid values ​​contained in the selected files by searching all storage nodes. Write these valid values ​​into a new file, replace the selected file with the new file, and delete the selected file after replacement, thus completing the cleanup. Figure 3 As shown, the color intensity represents the failure rate, with darker colors indicating higher failure rates. After cleaning up files with failure rates exceeding a specified threshold, the overall failure rate decreases. The process described in this section can be handled by an asynchronous thread, reducing the impact of cleaning up failure values ​​on accessed data.

[0049] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. An improved Flink HashMapStateBackend method employing key-value separation, wherein the data stored in the FlinkHashMapStateBackend consists of two parts: a key and a value, and each data is stored in a memory-based storage node. The method is characterized by... This method improves FlinkHashMapStateBackend by using a key-value separation approach, storing data keys and values ​​separately. Keys are all stored in the storage nodes, while values ​​are stored in files on disk. The specific implementation of this method includes the following steps: S1: Locate the storage node containing the data in the HashMap structure based on the data's key, and create and maintain an index for the storage node according to the access order; S2: Periodically detect memory usage. Whenever memory usage exceeds a specified threshold, trigger key-value splitting. Based on the index built for storage nodes according to the access order, start key-value splitting from the least accessed storage node, separate the values ​​in the storage node and store them in the file, and store a number that can uniquely identify the value in the storage node. This number strictly increments according to the order in which the values ​​are separated. Key-value splitting ends when the number of separated values ​​reaches the specified amount of data. S3: Values ​​separated from storage nodes are stored in files according to the order in which they were separated; S4: For data that has been separated into key and value, modifying the value of the data will not modify the file. Instead, the value is considered invalid, and the number of invalid values ​​is recorded. The failure rate is calculated periodically. Whenever the failure rate reaches a specified threshold, files with a failure rate higher than the specified threshold are selected for cleaning. The still valid values ​​in the selected files are written to a new file, and the selected files are replaced with the new file. The selected files are deleted after being replaced, thus completing the cleaning.

2. The method for improving Flink HashMapStateBackend using key-value separation according to claim 1, characterized in that, The specific steps for step S1 are as follows: S11: Based on the key of the data, find the storage node where the data is located in the HashMap structure; S12: Organize the storage nodes into a doubly linked list structure according to the access order and index them. Each storage node has two pointers, which point to the storage node that was accessed before and the storage node that was accessed after. The storage node at the head of the doubly linked list is the most recently accessed storage node, and the storage node at the tail of the doubly linked list is the least recently accessed storage node. S13: When accessing data, the storage node corresponding to the data becomes the most recently accessed storage node. Modify the pointer of the storage node and the pointers of related storage nodes, thereby moving the storage node to the head of the doubly linked list.

3. The method for improving Flink HashMapStateBackend using key-value separation according to claim 2, characterized in that, The specific steps for step S2 are as follows: S21: Use the garbage collection time and memory usage provided by the Java Virtual Machine as indicators to measure memory usage. Periodically check the above two indicators. When the two indicators exceed the specified threshold, trigger key-value separation. S22: When key-value separation is triggered, based on the index built for the storage nodes in the access order, starting from the least accessed storage node, key-value separation is performed on the storage nodes in the access order, the value of the storage node is separated and stored in the file, and a number that can uniquely identify the value is stored in the storage node. This number strictly increases in the order in which the values ​​are separated. Key-value separation ends when the separated values ​​reach the specified data volume.

4. The method for improving Flink HashMapStateBackend using key-value separation according to claim 3, characterized in that, The specific steps for step S3 are as follows: S31: The values ​​separated from the storage node are first batched in the write buffer. When the batched values ​​reach the specified data volume, these batched values ​​are stored in the file currently being written. S32: When the file currently being written reaches the specified file size, the file becomes an immutable file, and a new file is created to store the value that was detached later.

5. The method for improving Flink HashMapStateBackend using key-value separation according to claim 4, characterized in that, The specific steps for step S4 are as follows: S41: Record the total number of values ​​contained in each file and the total number of invalid values ​​contained in the file. The failure rate of the file is equal to the total number of invalid values ​​contained in the file divided by the total number of values ​​contained in the file. Calculate the total failure rate of the separated values ​​periodically. The total failure rate is equal to the total number of invalid values ​​contained in all files divided by the total number of values ​​contained in all files. S42: Whenever the total failure rate reaches a specified threshold, files with a failure rate higher than the specified threshold are selected for cleaning. The method for selecting files is as follows: First, the file with the highest failure rate is included in the selection. Then, the failure rate of the file with the closest creation time to the file is checked. If it is higher than the specified threshold, it is included in the selection. Then, the failure rate of the file with the closest creation time to the newly included file is checked. If it is higher than the specified threshold, it is included in the selection. The above process is repeated until the failure rate of the file with the closest creation time to the newly included file is no higher than the specified threshold, at which point the selection stops. S43: Clean up invalid values ​​contained in the selected files: By searching all storage nodes, determine the valid values ​​contained in the selected files, write these valid values ​​into a new file, and replace the selected files with the new file. The selected files are then deleted after being replaced, thus completing the cleanup.

Images