A data persistence storage method and device, electronic equipment and storage medium

Abstract

The application provides a data persistent storage method and device, electronic equipment and storage medium. The method comprises the following steps: writing state data into a local active prewrite log and an active memory cache according to each partition key; if the written capacity of the active memory cache meets a preset capacity threshold, or the written entries of the local active prewrite log meet a preset entry threshold, determining the active memory cache as an immutable memory cache, determining the local active prewrite log as a local immutable prewrite log, creating a new local active prewrite log and a new active memory cache to continue writing state data; if there is an unexpired local immutable prewrite log, persistently storing the immutable memory cache of the local immutable prewrite log to an external storage service; if the immutable memory cache is completed persistent storage, marking the corresponding local immutable prewrite log as expired to clean up expired logs and caches. The application decouples writing and storage, and improves storage efficiency and reliability.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to a data persistence storage method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of information technology, the speed and volume of data generation are constantly increasing. High throughput, low latency, and fast data processing have become crucial. Flink (an open-source distributed big data processing engine), as the most popular real-time computing engine, is highly favored for its high throughput and low latency capabilities. State operations are the core of its computation, and the efficiency of state operations during data processing directly affects the timeliness of computation. Among them, the keyedstatebackend for group operations carries most of the state load. Traditional keyedstatebackends are implemented using two methods: memory-based storage (memoryStateBackend) and high-performance embedded key-value database storage (RocksDBStateBackend). However, memoryStateBackend maintains all states in memory, which places extremely high demands on memory and is generally not suitable for ultra-large states. Furthermore, the recovery of quasi-states from task anomalies is very slow. RocksDBStateBackend, in the case of ultra-large states, requires data migration when adjusting the state distribution of tasks, which takes a long time and affects processing efficiency. Under frequent read and write operations, read amplification and write amplification problems are severe, seriously affecting read and write efficiency. In addition, it also has high requirements for memory space.

[0003] In related technologies, KeyedStateBackend is implemented using remote key-value (KV) storage devices. Data in the key-value storage devices is accessed and retrieved via network interaction, thus meeting the read and write requirements of ultra-large-scale states. However, since shards and keygroups are in a one-to-one correspondence, changes in the number of keygroups lead to shard reallocation. In ultra-large states, shard reallocation is time-consuming and heavily reliant on the capacity of external storage. Furthermore, direct writing to external storage places extremely high demands on its efficiency, severely impacting computational efficiency. Performance fluctuations in external storage and network fluctuations directly affect the efficiency of state write operations, and a large number of state write operations also put significant pressure on the backend data services. Summary of the Invention

[0004] This invention provides a data persistent storage method, apparatus, electronic device, and storage medium to solve the technical problems of low read / write efficiency for large-scale state data and external storage relying on key groups.

[0005] In one embodiment of the present invention, a data persistent storage method is provided, comprising: acquiring state data of different partition keys in the current partition, wherein the state data is determined based on key grouping operations performed on the acquired data stream; writing the corresponding state data to a local active write-ahead log and an active memory cache according to each partition key; if the written capacity of the active memory cache meets a preset capacity threshold, or the written entries of the local active write-ahead log meet a preset entry threshold, then the active memory cache is determined as an immutable memory cache, the local active write-ahead log is determined as a local immutable write-ahead log, and a new local active write-ahead log and a new active memory cache are created to continue writing the state data; if there are unexpired local immutable write-ahead logs, then the immutable memory cache corresponding to the local immutable write-ahead logs is persistently stored to an external storage service; if the immutable memory cache has completed persistent storage, then the local immutable write-ahead logs corresponding to the immutable memory caches are marked as expired to clean up expired local immutable write-ahead logs and corresponding immutable memory caches.

[0006] In one embodiment of the present invention, after performing key grouping operations on the acquired data stream, the method further includes: if a state read instruction exists, then reading the target data corresponding to the state read instruction from the active memory cache according to each partition key; if the target data is not read from the active memory cache, then reading the target data from the immutable memory cache; if the target data is not read from the immutable memory cache, then reading the target data from the read cache; if the target data is not read from the read cache, then reading the target data from the external storage service; if the target data is not read from the external storage service, then returning that the target data does not exist; if the target data is read from the external storage service and the target data has not expired, then adding the target data to the read cache, updating the expiration time of the target data in the external storage service, and returning the target data; if the target data is read from the external storage service and the target data has expired, then deleting the target data in the external storage service and returning that the target data does not exist.

[0007] In one embodiment of the present invention, cleaning up expired local immutable write-ahead logs and corresponding immutable memory caches includes: writing the state data in the immutable memory cache corresponding to the expired local immutable write-ahead logs into a read cache and deleting the immutable memory cache; and clearing the expired local immutable write-ahead logs.

[0008] In one embodiment of the present invention, before obtaining the status data of different partition keys in the current partition, the method further includes: reading the checkpoint information of the latest checkpoint through the job manager and starting the checkpoint coordinator to periodically perform checkpoint check operations, wherein the checkpoint is used to back up the write-ahead log; receiving the checkpoint information transmitted by the job manager through the task manager and performing a status data recovery operation to restore the backed-up status data; and receiving a data stream through the task manager to obtain grouped data of different data partitions.

[0009] In one embodiment of the present invention, performing a state data recovery operation includes: if the partition data volume of each data partition has not changed, loading a first write-ahead log from the local write-ahead log directory of the task manager; if the first write-ahead log is not loaded from the local write-ahead log directory, loading a second write-ahead log corresponding to the current partition from the checkpoint metadata in the checkpoint information; if the partition data volume of each data partition has changed, and the first write-ahead log exists in the local write-ahead log directory of the task manager, clearing the first write-ahead log, loading a third write-ahead log of all data partitions from the checkpoint information, and retaining the log record of the current partition after key grouping to obtain a fourth write-ahead log. Write logs; if the partition data volume of each data partition changes, and the first write-ahead log does not exist in the local write-ahead log directory of the task manager, then load the third write-ahead logs of all data partitions from the checkpoint information, and retain the log record of the current partition after key grouping to obtain the fourth write-ahead log; construct a target memory cache based on the first write-ahead log, the second write-ahead log, or the fourth write-ahead log; wherein, the first write-ahead log and the fourth write-ahead log are determined based on the local active write-ahead log and / or the local immutable write-ahead log that has not expired, and the second write-ahead log and the third write-ahead log are determined based on the cloud active write-ahead log and / or the cloud immutable write-ahead log that has not expired.

[0010] In one embodiment of the present invention, after key grouping, the log records of the current partition are retained to obtain a fourth write-ahead log, including: reading the cloud active write-ahead log in the third write-ahead log, performing key grouping according to the partition key in the cloud active write-ahead log, retaining the log records of the current partition, and writing the local active write-ahead log in the local write-ahead log directory; and / or, reading the unexpired cloud immutable write-ahead log in the third write-ahead log, performing key grouping according to the partition key in the unexpired cloud immutable write-ahead log, retaining the log records of the current partition, and writing the local immutable write-ahead log in the local write-ahead log directory.

[0011] In one embodiment of the present invention, after constructing the target memory cache, the process includes: starting a write-ahead log refresher to refresh the corresponding state data to the disk corresponding to the local active write-ahead log according to each partition key; starting a persistentizer to persistently store the immutable memory cache corresponding to the non-expired local immutable write-ahead log to an external storage service; starting a cleaner to clean up expired local immutable write-ahead logs and the corresponding immutable memory cache; and receiving the data stream through the task manager.

[0012] In one embodiment of the present invention, performing a checkpoint check operation includes: if there are unexpired local immutable write-ahead logs, persisting the local immutable write-ahead logs to the cloud storage service of the current checkpoint; if there are local active write-ahead logs, persisting the local active write-ahead logs to the cloud storage service of the current checkpoint; if the number of checkpoints is greater than a preset threshold, clearing the corresponding historical checkpoints and the cloud active write-ahead logs and / or cloud immutable write-ahead logs corresponding to the historical checkpoints; if there is expired state data in the current partition, clearing the expired state data in the external storage service.

[0013] In one embodiment of the present invention, a persistent data storage device is provided, comprising: a data acquisition module, configured to acquire state data of different partition keys in the current partition, wherein the state data is determined based on key grouping operations performed on the acquired data stream; a data writing module, configured to write the corresponding state data to a local active write-ahead log and an active memory cache respectively according to each partition key; and a conversion module, configured to determine the active memory cache as an immutable memory cache if the written capacity of the active memory cache meets a preset capacity threshold, or the written entries of the local active write-ahead log meet a preset entry threshold. The local active write-ahead log is identified as a local immutable write-ahead log, and a new local active write-ahead log and a new active memory cache are created to continue writing the state data; the persistent storage module is used to persistently store the immutable memory cache corresponding to the local immutable write-ahead log to an external storage service if there are any unexpired local immutable write-ahead logs; the cleanup module is used to mark the local immutable write-ahead log corresponding to the immutable memory cache as expired if the immutable memory cache has been persistently stored, so as to clean up the expired local immutable write-ahead logs and the corresponding immutable memory cache.

[0014] In one embodiment of the present invention, the device further includes a cloud storage service for persistently storing the corresponding local active write-ahead log and / or local immutable write-ahead log; the persistent storage module includes an external storage unified interface for abstracting read and write operations of state data in external storage services to isolate the differences between different external storage services; the external storage service includes a storage adaptation layer for adapting to the format requirements of different external storage services.

[0015] In one embodiment of the present invention, an electronic device is provided, the electronic device comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the data persistence storage method as described in any of the above embodiments.

[0016] In one embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a computer processor, the computer performs the data persistence storage method described in any of the above embodiments.

[0017] The beneficial effects of this invention are as follows: This invention provides a data persistent storage method, apparatus, electronic device, and storage medium. By using active memory caching and immutable memory caching, user write operations and persistent storage of state data are decoupled and do not affect each other, greatly improving the efficiency of state write operations. Furthermore, by adding a local active write-ahead log to the active memory cache, the reliability of state data is guaranteed. By converting the local active write-ahead log into a local immutable write-ahead log, the reliability of persistent storage of the immutable memory cache to external storage services is guaranteed. This allows external storage services to store state data without grouping data, relying entirely on the partition key, without depending on the number of key groups, and without the problem of external data re-partitioning. As a result, the storage of state data achieves memory-level efficiency, and can support ultra-large-scale state computation with relatively few memory resources.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0020] Figure 1A schematic diagram of an exemplary system architecture to which the technical solutions of embodiments of the present invention can be applied is shown; Figure 2 A flowchart illustrating a data persistence storage method according to an embodiment of the present invention is shown; Figure 3 A simplified flowchart of a data persistence storage method according to an embodiment of the present invention is shown. Figure 4 A schematic diagram of the state initialization process according to an embodiment of the present invention is shown; Figure 5 A schematic diagram of the inspection points according to an embodiment of the present invention is shown; Figure 6 A schematic diagram of the data writing process according to an embodiment of the present invention is shown; Figure 7 A schematic diagram of the data reading process according to an embodiment of the present invention is shown; Figure 8 A block diagram of a data persistence storage device according to an embodiment of the present invention is shown; Figure 9 A schematic diagram of a computer system suitable for implementing embodiments of the present invention is shown. Detailed Implementation

[0021] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0022] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the shape, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0023] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0024] Please see Figure 1 , Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of embodiments of the present invention can be applied is shown. For example... Figure 1As shown, the system architecture can include a caching layer, a write-ahead log layer, a storage layer, a storage adaptation layer, an executor layer, and a checkpoint layer. The caching layer includes a read cache (ReadCacheTTL), an active memory cache (ActiveMemCache), and an immutable memory cache (ImmutableMemCache). The read cache accelerates data retrieval efficiency, reduces the frequency of access to external storage services, and minimizes the performance impact of external storage services during data retrieval. The active memory cache is used for updating state data and accelerating read operations, improving the read / write efficiency of state data. The immutable memory cache is transformed based on the active memory cache reaching a preset capacity threshold; it is mainly used for subsequent persistent storage of state data to external storage services, accelerating data persistence efficiency, minimizing the performance impact of external storage services, and reducing the pressure on external storage services. The write-ahead log (WAL) layer includes a WALReader, active WALRecords corresponding to the active memory cache, and immutable WALRecords corresponding to the immutable memory cache. The WALReader reads active WALRecords and converts them into active memory cache, and reads immutable WALRecords and converts them into immutable memory cache. During checkpoint initialization, it reconstructs the previous active memory cache and unexpired immutable memory cache. Active WALRecords include local active WALRecords in the local WALRecord directory and cloud active WALRecords corresponding to cloud storage services. Immutable WALRecords include local immutable WALRecords in the local WALRecord directory and cloud immutable WALRecords corresponding to cloud storage services. The storage layer includes a unified interface for external storage, used to isolate the differences between different external storage services and abstract read / write operations on state data in external storage services. The storage adaptation layer includes adapters for various external storage services such as Redis, MySQL, Clickhouse, and TiDB, to efficiently implement read / write operations on state data according to the characteristics and syntax of different external storage services. The executor layer includes a write-ahead log flusher, immutable memory cache persisters, and an immutable memory cache and immutable write-ahead log cleaner. The write-ahead log flusher is used to refresh the write-ahead log data to files on disk periodically or line by line. The persisters are used to persist the state data of the immutable memory cache to external storage services in batches. The cleaner is used to mark the local immutable write-ahead logs corresponding to the immutable memory cache that have been persisted as expired, synchronize the immutable memory cache to the read cache, and then delete the immutable memory cache and the expired local immutable write-ahead logs.The checkpoint layer includes a checkpoint manager, a checkpoint metadata operator (Meta), and a checkpoint coordinator. The checkpoint manager manages the checkpoint metadata operator (Meta) and the checkpoint coordinator. The checkpoint metadata operator (Meta) reads and writes checkpoint metadata and manages the persistence of local active write-ahead logs and non-expired local immutable write-ahead logs corresponding to each checkpoint to cloud storage services, such as the Hadoop Distributed File System (HDFS) or Object Storage Service (OSS), to ensure the accuracy of state data. The checkpoint coordinator notifies the various subtasks of Flink (an open-source distributed big data processing engine) to perform checkpoint checks.

[0025] For example, the status data of different partition keys in the current partition is obtained, and the status data is determined based on key grouping operations on the obtained data stream; the corresponding status data is written to the local active write-ahead log and the active memory cache according to each partition key; if the written capacity of the active memory cache meets a preset capacity threshold, or the written entries of the local active write-ahead log meet a preset entry threshold, then the active memory cache is determined as an immutable memory cache, the local active write-ahead log is determined as a local immutable write-ahead log, and a new local active write-ahead log and a new active memory cache are created to continue writing status data; if there are unexpired local immutable write-ahead logs, the immutable memory cache corresponding to the local immutable write-ahead logs is persisted to an external storage service; if the immutable memory cache has been persisted, the local immutable write-ahead log corresponding to the immutable memory cache is marked as expired to clean up the expired local immutable write-ahead logs and the corresponding immutable memory cache.

[0026] In related technologies, there are technical problems such as low read / write efficiency for large-scale state data and reliance on key groups for external storage. To address the aforementioned technical problems, this invention provides a data persistence storage method, apparatus, electronic device, and storage medium. The implementation details of the technical solutions of the embodiments of this invention are described in detail below.

[0027] Please see Figure 2 , Figure 2 A flowchart illustrating a data persistence storage method according to an embodiment of the present invention is shown. Figure 2 As shown, in an exemplary embodiment, the data persistence storage method includes at least steps S210 to S250, which are described in detail below: Step S210: Obtain the status data of different partition keys in the current partition.

[0028] The status data is determined based on key grouping operations performed on the acquired data stream.

[0029] In one embodiment of the present invention, before obtaining the status data of different partition keys in the current partition, the method further includes: reading the checkpoint information of the latest checkpoint through the job manager and starting the checkpoint coordinator to periodically perform checkpoint check operations, wherein the checkpoint is used to back up the write-ahead log; receiving the checkpoint information transmitted by the job manager through the task manager and performing a status data recovery operation to restore the backed-up status data; and receiving the data stream through the task manager to obtain the grouped data of different data partitions.

[0030] In one embodiment of the present invention, please refer to Figure 3 , Figure 3 A simplified flowchart illustrating a data persistence storage method according to an embodiment of the present invention is shown. Figure 3 As shown, Flink's JobManager starts, loads the latest checkpoint information, and starts the checkpoint coordinator. Flink's TaskManager starts, initializes itself, receives checkpoint information from the JobManager, initializes checkpoints, and performs initialization operations, including state data recovery. The TaskManager receives state data, groups it using a key-by operation, and begins processing the grouped data, performing state read and write operations based on the partition key. The checkpoint coordinator sends a command to the TaskManager to execute the checkpoint check operation. The TaskManager executes the checkpoint check operation, and upon completion, sends the current task's checkpoint information to the checkpoint coordinator, indicating successful checkpoint check operation execution. Through this simplified Flink job execution process, efficient read / write operations and external storage persistence for extremely large amounts of state data can be achieved.

[0031] In one embodiment of the present invention, performing a state data recovery operation includes: if the partition data volume of each data partition has not changed, loading a first write-ahead log from the local write-ahead log directory of the task manager; if the first write-ahead log is not loaded from the local write-ahead log directory, loading a second write-ahead log corresponding to the current partition from the checkpoint metadata in the checkpoint information; if the partition data volume of each data partition has changed, and a first write-ahead log exists in the local write-ahead log directory of the task manager, clearing the first write-ahead log, loading a third write-ahead log of all data partitions from the checkpoint information, and retaining the log record of the current partition after key grouping to obtain a fourth write-ahead log. Write-ahead logs; if the partition data volume of each data partition changes and the first write-ahead log does not exist in the local write-ahead log directory of Task Manager, then the third write-ahead logs of all data partitions are loaded from the checkpoint information, and the log records of the current partition are retained after key grouping to obtain the fourth write-ahead log; construct the target memory cache based on the first write-ahead log, the second write-ahead log, or the fourth write-ahead log; wherein, the first write-ahead log and the fourth write-ahead log are determined based on the local active write-ahead logs and / or the local immutable write-ahead logs that have not expired, and the second write-ahead logs and the third write-ahead logs are determined based on the cloud active write-ahead logs and / or the cloud immutable write-ahead logs that have not expired.

[0032] In one embodiment of the present invention, after key grouping, the log records of the current partition are retained to obtain a fourth write-ahead log, including: after reading the cloud active write-ahead log in the third write-ahead log, key grouping is performed according to the partition key in the cloud active write-ahead log, and the log records of the current partition are retained, and local active write-ahead log is written in the local write-ahead log directory; and / or, after reading the cloud immutable write-ahead log that has not expired in the third write-ahead log, key grouping is performed according to the partition key in the cloud immutable write-ahead log that has not expired, and the log records of the current partition are retained, and local immutable write-ahead log is written in the local write-ahead log directory.

[0033] In one embodiment of the present invention, constructing a target memory cache based on a first write-ahead log, a second write-ahead log, or a fourth write-ahead log includes: reading the first write-ahead log or the second write-ahead log through a write-ahead log reader and constructing the target memory cache; or, constructing the target memory cache during the generation of the fourth write-ahead log.

[0034] In one embodiment of the present invention, the target memory cache includes an active memory cache corresponding to an active write-ahead log and an immutable memory cache corresponding to an immutable write-ahead log.

[0035] In one embodiment of the present invention, after constructing the target memory cache, the process includes: starting a write-ahead log refresher to refresh the corresponding state data to the disk corresponding to the local active write-ahead log according to each partition key; starting a persistentizer to persistently store the immutable memory cache corresponding to the local immutable write-ahead log that has not expired to an external storage service; starting a cleaner to clean up expired local immutable write-ahead logs and the corresponding immutable memory cache; and receiving a data stream through a task manager.

[0036] In one embodiment of the present invention, please refer to Figure 4 , Figure 4 A schematic diagram of the state initialization process according to an embodiment of the present invention is shown. Figure 4As shown, TaskManager loads the checkpoint information sent by JobManager. It determines whether the partition data volume of each data partition has changed; if it has not changed, i.e., no repartitioning, it first loads the first write-ahead log in the local write-ahead log (WAL) directory. If it is not in the local write-ahead log directory, it loads the second write-ahead log corresponding to the current partition from the checkpoint metadata in the checkpoint information; it reads the active write-ahead log corresponding to the first or second write-ahead log through the write-ahead log reader to build an active memory cache; it reads the immutable write-ahead log corresponding to the first or second write-ahead log through the write-ahead log reader to build an immutable memory cache. If changes occur, i.e., repartitioning, the system checks if the first write-ahead log exists in the local write-ahead log directory. If it does, it clears the first write-ahead log and loads the third write-ahead logs for all data partitions from the checkpoint information. The system then reads all active write-ahead logs from the cloud using the write-ahead log reader, groups these logs by their partition keys (KeyGroup), and retains only the log records for the current partition. Local active write-ahead logs are written to the local write-ahead log directory and flushed to disk. During this process, an active memory cache is built. Similarly, the system reads all immutable write-ahead logs from the cloud using the write-ahead log reader, groups these logs by their partition keys (KeyGroup), and retains only the log records for the current partition. Local immutable write-ahead logs are written to the local write-ahead log directory and flushed to disk. During this process, an immutable memory cache is built. Once the target memory cache is successfully built, executors such as the write-ahead log refresher, persistentizer, and cleaner are started. The entire state data recovery operation does not involve changes to the remote state; it only constructs the state from the write-ahead log used in checkpoints to ensure precise consistency. Compared to the entire state data, the data volume is very small, allowing for rapid construction and independence from the number of keyGroups. Furthermore, the most frequently accessed state data from the previous job is retained in the active memory cache, indirectly improving the efficiency of state data access and achieving hot loading of state data.

[0037] In one embodiment of the present invention, performing a checkpoint check operation includes: if there are unexpired local immutable write-ahead logs, persisting the local immutable write-ahead logs to the cloud storage service of the current checkpoint; if there are local active write-ahead logs, persisting the local active write-ahead logs to the cloud storage service of the current checkpoint; if the number of checkpoints is greater than a preset threshold, clearing the corresponding historical checkpoints and the cloud active write-ahead logs and / or cloud immutable write-ahead logs corresponding to the historical checkpoints; if there is expired state data in the current partition, clearing the expired state data in the external storage service.

[0038] In one embodiment of the present invention, please refer to Figure 5 , Figure 5 A schematic diagram of the inspection points according to an embodiment of the present invention is shown. Figure 5 As shown, the JobManager's checkpoint coordinator periodically sends instructions to the TaskManager to perform checkpoint operations. Upon receiving the instructions, the TaskManager begins the checkpoint operation. It checks if the local immutable write-ahead log (WAL) of the current immutable memory cache has expired. If not, it persists the existing WAL to the current checkpoint's cloud storage service, such as HDFS or OSS. It also persists the local active WAL of the current active memory cache to the current checkpoint's cloud storage service. Once both WAL persistence is successful, it reports a successful checkpoint execution to the checkpoint coordinator. The checkpoint expiration thread asynchronously performs a checkpoint release operation: based on a preset threshold, it calculates the historical checkpoints that need to be cleared, clears the WAL within these historical checkpoints, and releases space. The state expiration thread asynchronously performs a state expiration clearing operation: it checks if the state data in the current partition has expired; if so, it clears the corresponding state data. The specific implementation of the state expiration check can vary depending on the external storage adapter. The unified external storage interface only transmits the partition key; the external storage adapter uses the KeyGroup calculation rules to locate and process all relevant key states within the partition. Through the checkpointing steps described above, precise consistency of state semantics is achieved, ensuring rapid checkpoint execution. Expired state data is also cleaned up, freeing up external storage space. Since checkpoints persist only a small number of unexpired write-ahead logs, the checkpointing operation can be completed quickly, with negligible impact on data flow processing. Asynchronous checkpoint expiration and expired state cleanup have no impact on checkpoint execution; therefore, the entire checkpointing process has almost no impact on data processing.

[0039] Step S220: Write the corresponding status data to the local active write-ahead log and the active memory cache according to each partition key.

[0040] In one embodiment of the present invention, the corresponding status data is first written to the local active write-ahead log according to each partition key, and then written to the active memory cache.

[0041] In step S230, if the written capacity of the active memory cache meets the preset capacity threshold, or the written entries of the local active write-ahead log meet the preset entry threshold, then the active memory cache is determined as an immutable memory cache, the local active write-ahead log is determined as a local immutable write-ahead log, and a new local active write-ahead log and a new active memory cache are created to continue writing state data.

[0042] If there are unexpired local immutable write-ahead logs in step S240, then the immutable memory cache corresponding to the local immutable write-ahead logs is persistently stored in an external storage service.

[0043] In step S250, if the immutable memory cache has been persisted, the local immutable write-ahead log corresponding to the immutable memory cache is marked as expired in order to clean up the expired local immutable write-ahead log and the corresponding immutable memory cache.

[0044] In one embodiment of the present invention, cleaning up expired local immutable write-ahead logs and corresponding immutable memory caches includes: writing the state data in the immutable memory cache corresponding to the expired local immutable write-ahead logs into a read cache and deleting the immutable memory cache; clearing the expired local immutable write-ahead logs.

[0045] In one embodiment of the present invention, the keygroup to which the data partition belongs is determined by the partition key. The TaskManager can execute tasks for multiple data partitions, and one data partition corresponds to one active memory cache.

[0046] In one embodiment of the present invention, the number of immutable memory caches corresponding to a data partition can be dynamically adjusted based on a pre-configured initial number and maximum number, combined with the size of the current state data, to meet computational needs and control the generation parameters of the active memory cache and the generation parameters of the local active write-ahead log, such as capacity.

[0047] In one embodiment of the present invention, please refer to Figure 6 , Figure 6 A schematic diagram illustrating the data writing process according to an embodiment of the present invention is shown. Figure 6As shown, TaskManager receives data streams and writes state data based on the partition key during data processing. First, it checks if an active memory cache exists. If not, it allocates a contiguous block of memory as the active memory cache according to the preset configuration. Then, it creates a new file on the local write-ahead log directory of the disk, such as a Solid State Drive (SSD), which serves as the write-ahead log file for the active memory cache and is in an open state—that is, the local active write-ahead log. The state data to be written is first written to the local active write-ahead log. This process involves sequential writing to the write-ahead log file, achieving speeds comparable to memory. If the state data refresh configuration is set to immediate refresh, it is refreshed immediately to flush the written state data to the write-ahead log file on disk. If the state data refresh configuration is not set to immediate refresh, it is periodically flushed to the write-ahead log file on disk using a write-ahead file refresher. Periodic flushing effectively improves the throughput and efficiency of file writing. Even if the state data is not flushed to the write-ahead log file on disk after successful writing to the local active write-ahead log, the state data is immediately written to the active memory cache. The process involves several steps: First, determining if the written capacity of the active memory cache has reached a preset threshold. If so, creating a new active memory cache and a new local active write-ahead log (WAL) and converting the old active memory cache to an immutable memory cache, marking the corresponding local WAL as a local immutable WAL. Second, determining if the written entries in the local WAL have reached a preset entry threshold. If so, creating a new active memory cache and a new local WAL, converting the old active memory cache to an immutable memory cache, and marking the corresponding local WAL as a local immutable WAL. The immutable memory cache's persistent storage periodically checks for expired local immutable WAL logs. If found, the corresponding immutable memory cache is serialized and persisted to an external storage service in batches. If persistence is successful, the cleaner is notified to begin cleanup. The cleaner marks the local immutable WAL log of the immutable memory cache as expired, synchronizes the immutable memory cache to the read cache, and then deletes the immutable memory cache. Through the above data writing process, efficient writing of state data is achieved, ensuring writing speed and supporting the writing of extremely large states. External storage services can be dynamically expanded as needed. Furthermore, the amount of immutable memory cache can be dynamically adjusted based on the timeliness of the calculation, the size of the state, and memory resources to ensure efficient computation.

[0048] In one embodiment of the present invention, after performing key grouping operations on the acquired data stream, the method further includes: if a state read instruction exists, reading the target data corresponding to the state read instruction from the active memory cache according to each partition key; if the target data is not read from the active memory cache, reading the target data from the immutable memory cache; if the target data is not read from the immutable memory cache, reading the target data from the read cache; if the target data is not read from the read cache, reading the target data from the external storage service; if the target data is not read from the external storage service, returning that the target data does not exist; if the target data is read from the external storage service and the target data has not expired, adding the target data to the read cache, updating the expiration time of the target data in the external storage service, and returning the target data; if the target data is read from the external storage service and the target data has expired, deleting the target data in the external storage service and returning that the target data does not exist.

[0049] In one embodiment of the present invention, please refer to Figure 7 , Figure 7 A schematic diagram illustrating the data reading process according to an embodiment of the present invention is shown. Figure 7 As shown, when processing the data stream, TaskManager reads the target data corresponding to the state read command, first from the active memory cache. If the active memory cache does not contain the target data, it reads the target data from the immutable memory cache. If the immutable memory cache does not contain the target data, it reads the target data from the read cache. If the read cache does not contain the target data, it reads the target data from the external storage service. If the external storage service does not contain the target data, it returns "No target data found". If the target data is found in the external storage service, its expiration time is checked. If the target data has expired, it is deleted from the external storage service, and "No target data found" is returned. If it has not expired, the target data is added to the read cache, its expiration time in the external storage service is updated, and the target data is returned. Through the above data reading steps, efficient reading of state data can be achieved. The data in the active memory cache is the most frequently accessed, followed by the immutable memory cache, and then the read cache. In most cases, the data read has already been hit in the cache, greatly improving the efficiency of state data reading and significantly reducing the frequency of state reads from external storage services, thereby improving the overall efficiency of computation.

[0050] This invention can efficiently and quickly support operations on extremely large states. It can comprehensively consider computational needs, memory resource availability, and the pressure on external storage services to select appropriate memory and external storage resources to meet the needs of Flink jobs with different state sizes and computational timeframes, thereby replacing the memoryStateBackend and RocksDBStateBackend state backends.

[0051] Please see Figure 8 , Figure 8 A block diagram of a data persistent storage device according to an embodiment of the present invention is shown. This device can be applied to... Figure 1 The implementation environment shown is not limited to this embodiment. This device can also be applied to other exemplary implementation environments and specifically configured in other devices.

[0052] like Figure 8 As shown, a persistent data storage device 800 according to an embodiment of the present invention includes: a data acquisition module 810, a data writing module 820, a conversion module 830, a persistent storage module 840, and a persistent storage module 840.

[0053] Among them, the data acquisition module 810 is used to acquire the status data of different partition keys in the current partition. The status data is determined based on the key grouping operation of the acquired data stream. The data writing module 820 is used to write the corresponding status data to the local active write-ahead log and the active memory cache according to each partition key. The conversion module 830 is used to determine the active memory cache as an immutable memory cache and the local active write-ahead log as a local immutable write-ahead log if the written capacity of the active memory cache meets a preset capacity threshold or the written entries of the local active write-ahead log meet a preset entry threshold, and to create a new local active write-ahead log and a new active memory cache to continue writing state data. The persistent storage module 840 is used to persistently store the immutable memory cache corresponding to the local immutable write-ahead log to an external storage service if there is an unexpired local immutable write-ahead log. The cleanup module 850 is used to mark the local immutable write-ahead log corresponding to the immutable memory cache as expired if the immutable memory cache has been persisted, so as to clean up the expired local immutable write-ahead log and the corresponding immutable memory cache.

[0054] In one embodiment of the present invention, the data persistent storage device further includes a cloud storage service for persistently storing the corresponding local active write-ahead log and / or local immutable write-ahead log; the persistent storage module includes an external storage unified interface for abstracting the read and write operations of state data in the external storage service to isolate the differences between different external storage services; the external storage service includes a storage adaptation layer for adapting to the format requirements of different external storage services.

[0055] It should be noted that the persistent data storage device and the persistent data storage method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments and will not be repeated here. In practical applications, the persistent data storage device provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.

[0056] Embodiments of the present invention also provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the data persistence storage method provided in the above embodiments.

[0057] Please see Figure 9 , Figure 9 A schematic diagram of a computer system suitable for implementing embodiments of the present invention is shown. It should be noted that... Figure 9 The computer system 900 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0058] like Figure 9 As shown, the computer system 900 includes a central processing unit 901, which can perform various appropriate actions and processes based on a program stored in a read-only memory 902 or a program loaded from a storage section 908 into a random access memory 903, such as performing the methods described in the above embodiments. The random access memory 903 also stores various programs and data required for system operation. The central processing unit 901, the read-only memory 902, and the random access memory 903 are interconnected via a bus 904. An input / output interface 905 is also connected to the bus 904.

[0059] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. Removable media 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 910 as needed so that computer programs read from them can be installed into storage section 908 as needed.

[0060] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit 901, it performs various functions defined in the system of the present invention.

[0061] It should be noted that the computer-readable medium shown in the embodiments of the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0062] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0063] The units described in the embodiments of the present invention can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself. Therefore, the technical solutions according to the embodiments of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, portable hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of the present invention.

[0064] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer's processor, causes the computer to perform the data persistence storage method provided in the above embodiments. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0065] In the above embodiments, unless otherwise specified, the use of ordinal numbers such as "first" and "second" to describe common objects only indicates that they refer to different instances of the same object, rather than indicating that the objects being described must be in a given order, whether temporally, spatially, sequentially, or in any other way.

[0066] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for persistent data storage, characterized in that, The method includes: Obtain status data of different partition keys in the current partition, wherein the status data is determined based on key grouping operation on the obtained data stream; Based on each partition key, the corresponding status data is written to the local active write-ahead log and the active memory cache, respectively. If the written capacity of the active memory cache meets a preset capacity threshold, or the written entries of the local active write-ahead log meet a preset entry threshold, then the active memory cache is determined to be an immutable memory cache, the local active write-ahead log is determined to be a local immutable write-ahead log, and a new local active write-ahead log and a new active memory cache are created to continue writing the status data. If there are any unexpired local immutable write-ahead logs, then the immutable memory cache corresponding to the local immutable write-ahead logs will be persistently stored in an external storage service. If the immutable memory cache is persisted, the local immutable write-ahead log corresponding to the immutable memory cache is marked as expired in order to clean up the expired local immutable write-ahead log and the corresponding immutable memory cache.

2. The data persistent storage method according to claim 1, characterized in that, After performing key grouping operations on the acquired data stream, the following is also included: If a status read instruction exists, the target data corresponding to the status read instruction is read from the active memory cache according to each partition key; If the target data is not read from the active memory cache, then the target data is read from the immutable memory cache; If the target data is not read from the immutable memory cache, then the target data is read from the read cache; If the target data is not read from the read cache, then the target data is read from the external storage service. If the target data is not read from the external storage service, the system returns that the target data does not exist. If the target data is read from the external storage service and the target data has not expired, then the target data is added to the read cache, the expiration time of the target data in the external storage service is updated, and the target data is returned. If the target data is read from the external storage service and the target data has expired, then the target data in the external storage service is deleted, and the system returns that the target data does not exist.

3. The method according to claim 1, characterized in that, Clean up expired local immutable write-ahead logs and their corresponding immutable memory caches, including: Write the state data in the immutable memory cache corresponding to the expired local immutable write-ahead log to the read cache, and then delete the immutable memory cache. Clear the expired local immutable write-ahead log.

4. The data persistence storage method according to any one of claims 1-3, characterized in that, Before retrieving the status data for different partition keys in the current partition, the following steps are also included: The checkpoint information of the latest checkpoint is read through the job manager, and the checkpoint coordinator is started to perform checkpoint checks periodically. The checkpoints are used to back up the write-ahead log. The task manager receives checkpoint information transmitted by the job manager and performs a status data recovery operation to restore the backed-up status data. The task manager receives data streams to obtain grouped data from different data partitions.

5. The data persistent storage method according to claim 4, characterized in that, Perform state data recovery operations, including: If the amount of partition data in each of the data partitions remains unchanged, then the first write-ahead log is loaded from the local write-ahead log directory of the task manager; If the first write-ahead log is not loaded from the local write-ahead log directory, then the second write-ahead log corresponding to the current partition in the checkpoint metadata is loaded from the checkpoint information. If the partition data volume of each data partition changes, and the first write-ahead log exists in the local write-ahead log directory of the task manager, then the first write-ahead log is cleared, and the third write-ahead logs of all data partitions are loaded from the checkpoint information. After key grouping, the log record of the current partition is retained to obtain the fourth write-ahead log. If the partition data volume of each data partition changes and the first write-ahead log does not exist in the local write-ahead log directory of the task manager, then the third write-ahead log of all data partitions is loaded from the checkpoint information, and the log record of the current partition is retained after key grouping to obtain the fourth write-ahead log. Construct a target memory cache based on the first write-ahead log, the second write-ahead log, or the fourth write-ahead log; The first write-ahead log and the fourth write-ahead log are determined based on local active write-ahead logs and / or unexpired local immutable write-ahead logs, while the second write-ahead log and the third write-ahead log are determined based on cloud active write-ahead logs and / or unexpired cloud immutable write-ahead logs.

6. The data persistent storage method according to claim 5, characterized in that, After key grouping, the log records of the current partition are retained to obtain the fourth write-ahead log, which includes: After reading the cloud active write-ahead log in the third write-ahead log, the log is grouped into key groups according to the partition key in the cloud active write-ahead log, and the log record of the current partition is retained. The local active write-ahead log is written in the local write-ahead log directory. And / or, After reading the non-expired immutable cloud write-ahead logs in the third write-ahead log, the write-ahead logs are grouped into key groups according to the partition keys in the non-expired immutable cloud write-ahead logs, and the log records of the current partition are retained. The local immutable write-ahead logs are then written in the local write-ahead log directory.

7. The data persistent storage method according to claim 5, characterized in that, After building the target memory cache, the following steps are included: Start the write-ahead log refresher to refresh the corresponding status data to the disk corresponding to the local active write-ahead log according to each partition key; Start the persistent storage to persistently store the immutable memory cache corresponding to the local immutable write-ahead log that has not expired to an external storage service; Start the cleaner to clean up expired local immutable write-ahead logs and their corresponding immutable memory caches; The data stream is received through the task manager.

8. The data persistent storage method according to claim 4, characterized in that, Perform checkpoint inspection operations, including: If there is an unexpired local immutable write-ahead log, then the local immutable write-ahead log is persisted to the cloud storage service of the current checkpoint. If a local active write-ahead log exists, then the local active write-ahead log is persistently stored in the cloud storage service of the current checkpoint. If the number of checkpoints exceeds a preset threshold, then the corresponding historical checkpoints, as well as the cloud active write-ahead logs and / or cloud immutable write-ahead logs corresponding to the historical checkpoints, are cleared. If expired status data exists in the current partition, then the expired status data in the external storage service is cleared.

9. A persistent data storage device, characterized in that, The device includes: The data acquisition module is used to acquire the status data of different partition keys in the current partition. The status data is determined based on key grouping operations performed on the acquired data stream. The data writing module is used to write the corresponding status data to the local active write-ahead log and the active memory cache according to each partition key; The conversion module is used to determine the active memory cache as an immutable memory cache and the local active write-ahead log as a local immutable write-ahead log if the written capacity of the active memory cache meets a preset capacity threshold or the written entries of the local active write-ahead log meet a preset entry threshold, and to create a new local active write-ahead log and a new active memory cache to continue writing the status data. The persistent storage module is used to persistently store the immutable memory cache corresponding to the local immutable write-ahead log to an external storage service if there is an unexpired local immutable write-ahead log. The cleanup module is used to mark the local immutable write-ahead log corresponding to the immutable memory cache as expired if the immutable memory cache has been persistently stored, so as to clean up the expired local immutable write-ahead log and the corresponding immutable memory cache.

10. The data persistent storage device according to claim 9, characterized in that, The device also includes cloud storage services for persistently storing the corresponding local active write-ahead logs and / or local immutable write-ahead logs; The persistent storage module includes a unified external storage interface, which is used to abstract the read and write operations of state data in external storage services, so as to isolate the differences between different external storage services. The external storage service includes a storage adaptation layer for adapting to the format requirements of different external storage services.

11. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the data persistence storage method as described in any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the computer's processor, causes the computer to perform the data persistence storage method according to any one of claims 1 to 8.

Images