A Flink self-starting method based on file storage system

Abstract

The application discloses a kind of Flink self-starting method based on file storage system, including several checkpoints in file storage system, the method includes the following steps: determining the target checkpoint of target task from several checkpoints, the target checkpoint is the checkpoint of the newest generation time in the several checkpoints corresponding to the target task;According to the target checkpoint, the target task is started.The application selects the checkpoint of the newest generation time to start the target task, aims at accurately capturing the instant state snapshot before the interruption of target task, so as to get rid of the tedious dependence on manual creation of save point for target task, realizes the seamless automation of target task starting, significantly simplifies the operation process before online, greatly compresses the overall update cycle, to achieve the effect of fast start, so that the program can start faster.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of Flink self-starting, in particular to a Flink self-starting method based on a file storage system. BACKGROUND

[0002] It is known that Flink (Apache Flink) is a distributed open source stream processing framework for stateful computation on unbounded and bounded data streams. In Flink, savepoint is a key function that allows users to manually create a snapshot of the state of a stream processing job at any time and resume the job running at a later stage. In the case where the job needs to be stopped but is expected to be restarted at a later time, the savepoint can be triggered first, and then the job can be safely closed, and then the job can be restarted from the savepoint location, avoiding data loss and repeated processing.

[0003] In the prior art, every time the Flink task is restarted, the savepoint needs to be triggered first, and then the job is safely closed, and then the job is restarted from the savepoint location, and the update steps are troublesome and cumbersome, and every time the script interface of Flink needs to be entered to execute the script, which consumes a long time. SUMMARY

[0004] The technical problem to be solved by the application is to provide a Flink self-starting method based on a file storage system to solve the problem that the existing Flink task needs to trigger a savepoint, the update steps are troublesome and cumbersome, and every time the script interface of Flink needs to be entered to execute the script, which consumes a long time.

[0005] To solve the above technical problems, the technical scheme adopted by the application is:

[0006] A Flink self-starting method based on a file storage system, the file storage system comprising a plurality of checkpoints, the Flink self-starting method based on the file storage system comprising the following steps:

[0007] S1: determining a target checkpoint of a target task from the plurality of checkpoints, the target checkpoint being the checkpoint with the latest generation time among the plurality of checkpoints corresponding to the target task;

[0008] S2: starting the target task according to the target checkpoint.

[0009] Further, in the Flink self-starting method based on the file storage system, in step S1, the following steps are included:

[0010] S11: obtaining a task name of the target task;

[0011] S12: According to the task name, obtain a checkpoint storage directory associated with the task name;

[0012] S13: Determine the target checkpoint from all checkpoints included in the checkpoint storage directory.

[0013] Furthermore, in the Flink self-starting method based on the file storage system of the present invention, in step S13, the following steps are included:

[0014] S131: Arrange all checkpoints in the checkpoint storage directory in descending order according to their generation timestamps to form a descending sort table;

[0015] S132: Set the first checkpoint at the top of the sequence in the descending order list as the target checkpoint.

[0016] Furthermore, in the Flink self-starting method based on the file storage system of the present invention, in step S2, the following steps are included:

[0017] S21: Perform integrity verification on the target checkpoint;

[0018] S22: If the integrity verification passes, the file path of the target checkpoint is set as the startup path of the target task.

[0019] Furthermore, in the Flink self-starting method based on the file storage system described in the present invention, in step S21, it includes: determining whether there is a target file in the folder corresponding to the target checkpoint; when the target file exists, the integrity verification of the target checkpoint passes; when the target file does not exist, the integrity verification of the target checkpoint fails.

[0020] Furthermore, the Flink self-starting method based on the file storage system of the present invention further includes the following steps:

[0021] S3: Periodically generate checkpoints for the target task;

[0022] S4: Storing the generated checkpoint in the file storage system.

[0023] Furthermore, in the Flink self-starting method based on the file storage system of the present invention, in step S3, the following steps are included:

[0024] S31: Start the checkpoint generation mechanism by starting the checkpoint function;

[0025] S32: Set the checkpoint generation cycle;

[0026] S33: Periodically generate checkpoints according to the generation cycle.

[0027] Furthermore, in the Flink self-starting method based on the file storage system described in the present invention, in step S32, it includes: obtaining the attribute value of the preset indicator, and setting the checkpoint generation cycle according to the attribute value.

[0028] Furthermore, in the Flink self-starting method based on the file storage system described in the present invention, the preset indicators include at least one of the following: business processing speed, data update frequency, and fault tolerance recovery time.

[0029] Furthermore, the Flink self-starting method based on the file storage system of the present invention further includes the following steps:

[0030] S5: Obtain the task name of the target task and the generation timestamp of the checkpoint generated in step S3;

[0031] S6: Concatenate the task name and the generation timestamp to generate the name of the checkpoint.

[0032] The beneficial effect of the present invention is that if a target task is to be started (such as a version update), several checkpoints associated with the target task can be obtained from the file storage system first. Then, the checkpoint with the latest generation time (i.e., the target checkpoint) is determined from the several checkpoints, and then the task is started according to the target checkpoint, achieving the effect of generation from the target checkpoint, avoiding data loss and repeated processing, and eliminating the need to create a save point every time, shortening the update cycle, and speeding up the startup of the program. In other words, the present invention selects the checkpoint with the latest timestamp, aiming to accurately capture the instant state snapshot before the task is interrupted, thereby getting rid of the tedious reliance on manually creating save points for Flink tasks one by one, achieving seamless automation of the target task startup, significantly simplifying the operational process before going online, and greatly shortening the overall update cycle. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Fig. 1 This is a flow chart of an embodiment of the Flink self-starting method based on a file storage system according to the present invention;

[0034] Fig. 2 This is another flowchart of the Flink self-starting method based on the file storage system according to one embodiment of the present invention;

[0035] Fig. 3 This is another flowchart of the Flink self-starting method based on the file storage system according to one embodiment of the present invention;

[0036] Fig. 4 FIG2 is another flowchart of an embodiment of the Flink self-starting method based on a file storage system according to the present invention. DETAILED DESCRIPTION

[0037] To illustrate the technical content, achieved objectives and effects of the present invention in detail, the following description is given in conjunction with the embodiments and accompanying drawings.

[0038] The Flink autostart solution automatically starts an Apache Flink cluster or a single Flink application when the operating system boots, ensuring continuous and stable operation of large data stream processing systems without manual intervention. This Flink autostart solution can be used for long-running real-time stream processing tasks or batch processing jobs in production environments.

[0039] In Apache Flink, savepoints allow users to manually create a state snapshot of a stream processing job at any time and resume the job at a later stage. If a job needs to be stopped but hopes to be restarted at a later time, a savepoint can be triggered first, safely shutting down the job, and then restarting the job from the savepoint to avoid data loss and duplicate processing. In Apache Flink, checkpointing is the core mechanism for achieving state consistency and fault tolerance in stream processing jobs. Checkpointed state can be saved in reliable persistent storage systems such as HDFS, S3 (Amazon Simple Storage Service), GCS (Google Cloud Storage), etc., ensuring that state data can be recovered even in the event of a node failure or an entire cluster failure. When a job encounters a failure, Flink automatically resumes the job from the most recently successfully completed checkpoint, restarting all operators and loading the corresponding state to quickly resume job processing.

[0040] It's important to note that savepoints are user-controllable snapshots implemented on top of the checkpoint mechanism. They can be considered manually triggered checkpoints with additional uses (such as version upgrades and migrations). Savepoints and checkpoints share the same underlying state persistence and recovery mechanism, differing only in their triggering timing and purpose. In summary, Flink's checkpoint mechanism enables fast and accurate recovery of stream processing jobs in the face of failures by periodically creating globally consistent snapshots of job state and persisting them to reliable storage. This makes it a key technology for ensuring high availability and data consistency in large-scale stream processing applications.

[0041] The existing Flink task auto-start method requires triggering a savepoint each time, safely shutting down the job, and then restarting the job from the savepoint. This update process is cumbersome and requires entering the Flink script interface to execute the script each time. Therefore, it is crucial to avoid data loss and duplicate processing, eliminate the need for creating savepoints every time, shorten the update cycle, and enable faster program startup.

[0042] To do this, please refer to Figs. 1 to 4 As shown, in this embodiment, the present invention provides a Flink self-starting method based on a file storage system, wherein the file storage system includes several checkpoints. The method includes the following steps:

[0043] S1: Determine a target checkpoint of a target task from a plurality of checkpoints, where the target checkpoint is a checkpoint with the latest generation time among the plurality of checkpoints corresponding to the target task.

[0044] S2: Starting the target task according to the target checkpoint.

[0045] The structural principle / working principle of the present invention is briefly described as follows: In the present invention, if you want to start the target task (such as performing a version update), you can first actively connect to the file storage system. In this file storage system, there are several checkpoints corresponding to the target task. According to the generation time of the above several checkpoints, the checkpoint with the latest generation time (i.e., the target checkpoint) is determined, and then the task is started according to the target checkpoint, achieving the effect of generating from the target checkpoint, avoiding data loss and repeated processing, and not having to create a savepoint every time, shortening the update cycle, and speeding up the startup speed of the program. In other words, the present invention selects the checkpoint with the latest timestamp to accurately capture the instant state snapshot before the task is interrupted, thereby getting rid of the tedious dependence on manually creating savepoints for Flink tasks one by one, realizing seamless automation of task startup, significantly simplifying the pre-online operation process, and greatly compressing the overall update cycle. Specifically, the present invention has the following advantages and beneficial effects: (1) No need to manually create savepoints: For large version update scenarios, the traditional practice of manually entering each Flink task to trigger the creation of savepoints is abandoned. (2) One-click start of the task to be updated: After the update preparation phase is completed, the Flink task to be updated can be started with just one click. The system will intelligently identify and load the latest valid save point corresponding to each task, ensuring that the task resumes execution from the correct state. There is no need to manually search for and configure save point information, which greatly simplifies the startup process. (3) The update process is greatly streamlined and accelerated: Through the above-mentioned automation mechanism, the manual creation of save points and configuration of startup parameters, which originally required a lot of manpower and time, have been greatly simplified or even eliminated, thereby significantly shortening the entire update process. Without being trapped in trivial preliminary preparations, it is possible to quickly focus on and debug core issues such as the logic of the new version code, performance optimization, and exception handling, thereby improving the efficiency and quality of the update. (4) Focus on the essence of the program and improve problem-solving efficiency: The automated update process frees the team's attention from the mechanical steps of starting the program, allowing them to devote themselves to in-depth analysis of program logic, potential problems, and optimization space. This transformation helps to speed up problem discovery and resolution, enhance the accuracy of version iteration, and promote the efficient and stable development of the project.

[0046] It should be pointed out that the file storage system of the present invention can be a MinIO file storage system, HDFS (Hadoop Distributed File System), etc., which is not limited here. The MinIO file storage system is an open source object storage system designed for cloud native and containerized environments. It uses a distributed architecture to store and manage large-scale unstructured data, such as pictures, videos, log files, etc. In addition, the target task can be a corresponding Flink task. In this embodiment, Apache Flink is an open source computing platform for distributed data stream processing and batch data processing. It can provide functions that support both stream processing and batch processing applications based on the same Flink runtime (Flink Runtime). Therefore, a Flink task can be a stream processing task or a batch processing task, depending on the requirements and configuration of the application.

[0047] In an embodiment of the present invention, step S1 includes the following steps:

[0048] S11: Obtain the task name of the target task.

[0049] S12: According to the task name, obtain a checkpoint storage directory associated with the task name.

[0050] S13: Determine the target checkpoint from all checkpoints included in the checkpoint storage directory.

[0051] As can be seen from the above description, in the present invention, if the target checkpoint of a target task is to be determined from a number of checkpoints, the task name of the target task can be used as an input parameter. Based on this task name, the system can automatically locate the checkpoint storage directory associated with the task name. On this basis, the target checkpoint is selected from all the checkpoints contained in the checkpoint storage directory, that is, the checkpoint with the latest generation time. In summary, by obtaining the checkpoint storage directory associated with the task name, all checkpoints stored in the target task can be quickly determined, and then the checkpoint with the latest generation time (that is, the target checkpoint) can be quickly obtained. It should be pointed out that the checkpoint storage directory can contain a mapping relationship between all checkpoint names corresponding to the target task and file path information.

[0052] In an embodiment of the present invention, step S13 includes the following steps:

[0053] S131: Arrange all checkpoints in the checkpoint storage directory in descending order according to the generation timestamps to form a descending sort table.

[0054] S132: Set the first checkpoint at the top of the sequence in the descending order list as the target checkpoint.

[0055] As can be seen from the above description, in the present invention, all checkpoints in the checkpoint storage directory are sorted in descending order according to their embedded timestamps (i.e., generation timestamps) to form a descending sorted table. The first checkpoint at the top of the sequence is the checkpoint with the latest generation timestamp, i.e., the latest state snapshot. When determining the target checkpoint, the first checkpoint at the top of the sequence can be directly set as the target checkpoint, eliminating the need to compare the generation timestamps of all checkpoints in the checkpoint storage directory one by one. This saves time and resources, and allows for quick and accurate identification of the checkpoint with the latest generation time (i.e., the target checkpoint), shortening the update cycle and enabling faster program startup.

[0056] In an embodiment of the present invention, step S2 includes the following steps:

[0057] S21: Perform integrity verification on the target checkpoint.

[0058] S22: If the integrity verification passes, the file path of the target checkpoint is set as the startup path of the target task.

[0059] In the above-mentioned technical solution of the present invention, when Flink performs a checkpoint generation operation, although the system strives to ensure a smooth process, the possibility of checkpoint generation failure due to various factors is undeniable. Therefore, in scenarios where tasks rely on checkpoints for restart, particular attention must be paid to verifying the integrity of the checkpoints, aiming to promptly detect and eliminate invalid or damaged checkpoints caused by problems during the generation process. Only after confirming that the checkpoints fully meet expectations and are intact can tasks be safely launched from them, eliminating restart failures caused by using problematic checkpoints and thus ensuring the stable operation of stream processing jobs and the accuracy of data processing. Therefore, after finding the target checkpoint, its integrity can be immediately verified. Once the integrity of the target checkpoint is confirmed, the file path corresponding to the target checkpoint can be embedded as a required parameter for launching a new task, that is, the file path of the target checkpoint can be set as the startup path of the target task. In this way, Flink jobs can clearly know which target checkpoint to resume execution from during the startup initialization phase, eliminating the need for manual intervention throughout the entire process, effectively improving the convenience of task updates and the consistency of data processing.

[0060] It should be noted that if integrity verification fails, you can search for other recent checkpoints in the checkpoint storage directory. In some cases, such as when no checkpoints pass integrity verification or no checkpoints exist, the Flink task will be started directly without starting from a checkpoint.

[0061] In some preferred embodiments, step S21 includes the following steps: determining whether a target file exists in the folder corresponding to the target checkpoint; when the target file exists, the integrity verification of the target checkpoint passes; when the target file does not exist, the integrity verification of the target checkpoint fails.

[0062] As can be seen from the above description, in the present invention, Flink checkpoints exist in the form of folders. When Flink generates a checkpoint, it will generate a corresponding folder, but the _metadata file (i.e., the target file) in the folder will not be generated immediately, while the _metadata file should exist in the correct and valid folder. Therefore, in order to avoid the situation where only the folder is generated but no _metadata file is generated, the target checkpoint can be integrity verified. That is, the integrity of the target checkpoint is verified by checking whether the _metadata file exists. In the case that the _metadata file exists, the integrity verification of the target checkpoint passes. In the case that the _metadata file does not exist, the integrity verification of the target checkpoint fails. In summary, by verifying whether the target file exists in the folder corresponding to the target checkpoint, the integrity of the target checkpoint is verified, aiming to promptly discover and eliminate invalid or damaged checkpoints caused by problems in the generation process, so as to prevent restart failures caused by the use of problematic checkpoints, thereby ensuring the stable operation of the target task and the accuracy of data processing.

[0063] In an embodiment of the present invention, the method further comprises the following steps:

[0064] S3: Periodically generate checkpoints for the target task.

[0065] S4: Storing the generated checkpoint in the file storage system.

[0066] As can be seen from the above description, in the present invention, checkpoints can be periodically generated for the target task, and the periodically generated checkpoints can be stored in the file storage system, so that when the subsequent target task needs to be started, the task can be started according to the periodically generated checkpoints, achieving the effect of generation from the checkpoints, thereby ensuring that the instant status snapshot before the interruption of the target task can be accurately captured, without the need to manually create save points, that is, by periodically generating checkpoints for the target task for subsequent update startup, the traditional practice of manually triggering the creation of save points for the target task is abandoned, and the entire process does not require manual intervention, effectively improving the convenience and efficiency of target task updates.

[0067] It should be noted that the checkpoints periodically generated by Flink tasks (i.e., target tasks) are stored by default in the local file system of the task server. However, given the risk of data loss due to single node failures or even entire cluster failures, a more robust strategy can be adopted: storing checkpoints in a highly available, highly consistent persistent storage system, such as HDFS, Amazon S3, or Google Cloud Storage. These persistent storage systems, with their distributed architecture, multi-copy redundancy, and automatic failure recovery mechanisms, provide strong security for checkpoint data. Consequently, even in the face of extreme node failures or even cluster-level failures, Flink tasks can quickly retrieve checkpoint status information from these persistent storage systems, ensuring seamless and accurate resumption of the target task. This minimizes service interruption and the risk of data processing inconsistencies, effectively ensuring the security and business continuity of the entire stream processing system. Furthermore, after storing the generated checkpoints in the file storage system, integrity checks can be performed on the checkpoints to ensure their validity.

[0068] In an embodiment of the present invention, step S3 includes the following steps:

[0069] S31: Start the checkpoint generation mechanism by starting the checkpoint function.

[0070] S32: Set the checkpoint generation cycle.

[0071] S33: Periodically generate checkpoints according to the generation cycle.

[0072] As can be seen from the above description, in the present invention, by utilizing the Flink streaming computing engine, the checkpoint generation mechanism can be strategically enabled and optimized. First, the checkpoint function is enabled through meticulous configuration operations to ensure that it stably and efficiently ensures data consistency during system operation. Secondly, the checkpoint generation cycle is set. The cycle should be precisely adjusted according to actual business characteristics and requirements to achieve the best balance between performance and reliability. On this basis, checkpoints can be generated periodically according to the generation cycle set above. As can be seen from the above, by enabling the checkpoint function and setting the checkpoint generation cycle, checkpoints can be generated periodically so that when subsequent target tasks need to be started, tasks can be started according to the checkpoints generated in the cycle.

[0073] In an embodiment of the present invention, step S32 includes the following steps: obtaining an attribute value of a preset indicator, and setting a generation period of the checkpoint according to the attribute value.

[0074] It should be noted that, in actual applications, the preset indicators include at least one of the following: business processing speed, data update frequency, and fault-tolerant recovery time.

[0075] As can be seen from the above description, in the present invention, the generation of target task checkpoints is based on a generation cycle. Therefore, when generating checkpoints for target tasks, the generation cycle should not be too long, so as to avoid the excessive amount of data that needs to be replayed and processed during system startup or fault recovery, thereby causing significant pressure peaks on upstream and downstream services, affecting the stability and response efficiency of the overall system. Similarly, the generation cycle should not be too short, so as to avoid excessive consumption of computing resources by too frequent checkpoint generation operations, affecting real-time processing capabilities. To this end, the present invention sets the checkpoint generation cycle based on the attribute values ​​of preset indicators. The preset indicators can include factors such as business processing speed, data update frequency, and fault tolerance recovery time requirements. It should be noted that the preset indicators can be selected according to actual conditions and are not limited here. In summary, the present invention sets the checkpoint generation cycle based on the corresponding attribute values ​​of the preset indicators, which can scientifically and rationally determine the checkpoint generation cycle, thereby effectively ensuring the stability and response efficiency of the overall system, saving computing resources, and improving real-time processing capabilities.

[0076] In an embodiment of the present invention, the method further comprises the following steps:

[0077] S5: Obtain the task name of the target task and the generation timestamp of the checkpoint generated in step S3.

[0078] S6: Concatenate the task name and the generation timestamp to generate the name of the checkpoint.

[0079] As can be seen from the above description, in the present invention, when generating checkpoints for a target task, a set of standardized naming rules is followed, that is, the task name is closely combined with an accurate timestamp to form an easily recognizable and orderly identifier. Specifically, the name of each checkpoint is composed of the task name and the generation timestamp, with the task name set first and the generation timestamp reflecting the generation moment set after the task name. This ensures the clarity of the checkpoint name and the intuitiveness of the chronological sequence. When determining the target checkpoint of the target task, the checkpoint with the latest generation time associated with the target task (i.e., the target checkpoint) can be directly determined based on the checkpoint name, thereby improving the efficiency of determining the target checkpoint.

[0080] In summary, the Flink self-starting method based on the file storage system provided by the present invention has the following features: (1) No need to manually create savepoints: For large version update scenarios, the method aims to accurately capture the instant state snapshot before the task is interrupted, abandoning the traditional practice of manually triggering the creation of savepoints by entering each Flink task one by one, achieving seamless automation of task startup, significantly simplifying the pre-launch operation process, and greatly shortening the overall update cycle, so that the team can focus more on troubleshooting and optimizing the core problems of the program, rather than wasting energy on the initial stages of starting the program. (2) One-click startup of the task to be updated: After the update preparation phase is completed, the Flink task to be updated can be started with just one click. The system will intelligently identify and load the latest valid savepoint corresponding to each task, ensuring that the task resumes execution from the correct state. There is no need to manually search and configure savepoint information, which greatly simplifies the startup process. (3) The update process is greatly streamlined and accelerated: Through the above-mentioned automation mechanism, the manual creation of savepoints and configuration of startup parameters, which originally required a lot of manpower and time, are greatly simplified or even eliminated, thereby significantly shortening the entire update process. Without being bogged down in trivial preliminary preparations, teams can quickly focus on and debug core issues such as the new version's code logic, performance optimization, and exception handling, improving update efficiency and quality. (4) Focusing on the essence of the program and improving problem-solving efficiency: The automated update process frees the team's attention from the mechanical steps of launching the program, allowing them to fully devote themselves to in-depth analysis of program logic, potential problems, and optimization space. This shift helps speed up problem discovery and resolution, enhances the accuracy of version iteration, and promotes efficient and stable project development.

[0081] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's description and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A Flink self-starting method based on a file storage system, characterized in that: The file storage system includes several checkpoints, and the method includes the following steps: S1: Determine a target checkpoint for a target task from a plurality of checkpoints, where the target checkpoint is the checkpoint with the latest generation time among the plurality of checkpoints corresponding to the target task; S2: starting the target task according to the target checkpoint; In step S1, the following steps are included: S11: Obtain the task name of the target task; S12: According to the task name, obtain a checkpoint storage directory associated with the task name; S13: Determine the target checkpoint from all checkpoints included in the checkpoint storage directory; In step S2, the following steps are included: S21: Perform integrity verification on the target checkpoint; S22: If the integrity verification passes, setting the file path of the target checkpoint as the startup path of the target task; In step S21, the following steps are included: Determine whether a target file exists in the folder corresponding to the target checkpoint. If the target file exists, the integrity verification of the target checkpoint passes; if the target file does not exist, the integrity verification of the target checkpoint fails. Also includes the steps: S3: Periodically generate checkpoints for the target task; S4: storing the generated checkpoint in the file storage system; In step S3, the following steps are included: S31: Start the checkpoint generation mechanism by starting the checkpoint function; S32: Set the checkpoint generation cycle; S33: Periodically generate checkpoints according to the generation cycle.

2. The Flink self-starting method according to claim 1, characterized in that: In step S13, the following steps are included: S131: Arrange all checkpoints in the checkpoint storage directory in descending order according to their generation timestamps to form a descending sort table; S132: Set the first checkpoint at the top of the sequence in the descending order list as the target checkpoint.

3. The Flink self-starting method according to claim 1, characterized in that: In step S32, the following steps are included: obtaining the attribute value of the preset indicator, and setting the generation cycle of the checkpoint according to the attribute value.

4. The Flink self-starting method according to claim 3, characterized in that: The preset indicators include at least one of the following: business processing speed, data update frequency and fault tolerance recovery time.

5. The Flink self-starting method according to claim 1, characterized in that: The method further comprises the following steps: S5: Obtain the task name of the target task and the generation timestamp of the checkpoint generated in step S3; S6: Concatenate the task name and the generation timestamp to generate the name of the checkpoint.

Images