File storage system-based Flink target task starting method

Abstract

The invention discloses a Flink target task starting method based on a file storage system, the file storage system comprises a plurality of check points, and the method comprises the following steps: determining a target check point of a target task from the plurality of check points, the target check point is a check point with the latest generation time in a plurality of check points corresponding to the target task; and starting the target task according to the target check point. The check point with the latest generation time is selected to start the target task, and the instant state snapshot before the interruption of the target task is accurately captured, so that the tedious dependence of manually creating storage points for the target task one by one is eliminated, seamless automation of starting of the target task is realized, the operation process before online is remarkably simplified, and the efficiency of starting the target task is improved. The overall updating period is greatly compressed, so that the effect of quick starting is achieved, and the program is started more quickly.

Description

[0001] This case is a divisional application based on the invention patent filed on May 24, 2024, with application number 2024106544883 and titled "A Flink Self-Starting Method Based on a File Storage System". Technical Field

[0002] This invention relates to the field of Flink self-starting technology, and in particular to a method for starting a Flink target task based on a file storage system. Background Technology

[0003] As is well known, Flink (Apache Flink) is a distributed, open-source stream processing framework for stateful computation on unbounded and bounded data streams. In Flink, savepoints are a key feature that allows users to manually create a snapshot of the stream processing job's state at any time and resume the job at a later stage. In situations where a job needs to be stopped but can be restarted at a later time, a savepoint can be triggered first, the job can be safely shut down, and then restarted from the savepoint location, avoiding data loss and duplicate processing.

[0004] In existing technologies, each time a Flink task restarts, a savepoint needs to be triggered first, then the job needs to be safely closed, and then the job needs to be restarted from the savepoint location. The update process is troublesome and cumbersome, and each time the script needs to be executed in the Flink script interface, which takes a long time. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a Flink self-starting method based on a file storage system, so as to solve the problems that the startup of existing Flink tasks requires triggering savepoints, the update steps are troublesome and cumbersome, and the scripts must be executed in the Flink script interface every time, which consumes a lot of time.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A Flink self-starting method based on a file storage system, wherein the file storage system includes several checkpoints, the Flink self-starting method based on the file storage system includes the following steps: S1: Determine the target checkpoint for the target task from a number of checkpoints, wherein the target checkpoint is the checkpoint with the latest generation time among the number of checkpoints corresponding to the target task; S2: Start the target task based on the target checkpoint.

[0007] Furthermore, in the Flink self-starting method based on a file storage system described in this invention, step S1 includes the following steps: S11: Obtain the task name of the target task; S12: Based on the task name, obtain the checkpoint storage directory associated with the task name; S13: Determine the target checkpoint from all checkpoints contained in the checkpoint storage directory.

[0008] Furthermore, in the Flink self-starting method based on a file storage system described in this invention, step S13 includes the following steps: S131: Sort all checkpoints in the checkpoint storage directory in descending order of their generation timestamps to form a descending sorted list; S132: Set the first checkpoint at the top of the sequence in the descending list as the target checkpoint.

[0009] Furthermore, in the Flink self-starting method based on a file storage system described in this invention, step S2 includes the following steps: S21: Perform integrity verification on the target checkpoint; S22: If the integrity verification passes, set the file path of the target checkpoint to the startup path of the target task.

[0010] Furthermore, in the Flink self-starting method based on the file storage system described in this invention, step S21 includes: determining whether a target file exists in the folder corresponding to the target checkpoint; when the target file exists, the integrity verification of the target checkpoint passes; when the target file does not exist, the integrity verification of the target checkpoint fails.

[0011] Furthermore, the Flink self-starting method based on a file storage system described in this invention also includes the following steps: S3: Periodically generate checkpoints for the target task; S4: Store the generated checkpoints to the file storage system.

[0012] Furthermore, in the Flink self-starting method based on a file storage system described in this invention, step S3 includes the following steps: S31: Enable the checkpoint generation mechanism by activating the checkpoint function; S32: Set the checkpoint generation cycle; S33: Periodically generate checkpoints according to the generation cycle.

[0013] Furthermore, in the Flink self-starting method based on the file storage system described in this invention, step S32 includes: obtaining the attribute value of a preset indicator, and setting the generation cycle of the checkpoint according to the attribute value.

[0014] Furthermore, in the Flink self-starting method based on the file storage system described in this invention, the preset indicators include at least one of the following: business processing speed, data update frequency, and fault recovery time.

[0015] Furthermore, the Flink self-starting method based on a file storage system described in this invention also includes the following steps: S5: Obtain the task name of the target task and the generation timestamp of the checkpoint generated in step S3; S6: Combine and concatenate the task name and the generated timestamp to generate the name of the checkpoint.

[0016] The beneficial effects of this invention are as follows: To start a target task (such as performing a version update), several checkpoints associated with the target task can be obtained from the file storage system. Then, the checkpoint with the latest generation time (i.e., the target checkpoint) is determined from these checkpoints, and the task is started based on the target checkpoint, achieving the effect of generation from the target checkpoint. This avoids data loss and redundant processing, eliminates the need to create savepoints each time, shortens the update cycle, and speeds up program startup. In other words, this invention selects the checkpoint with the latest timestamp to accurately capture an instantaneous snapshot of the task's state before interruption, thereby eliminating the tedious dependence on manually creating savepoints for each Flink task. This achieves seamless automation of target task startup, significantly simplifies the pre-deployment operation process, and greatly shortens the overall update cycle. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating one implementation of the Flink self-starting method based on a file storage system according to the present invention. Figure 2 This is another flowchart illustrating one implementation of the Flink self-starting method based on a file storage system described in this invention. Figure 3 This is another flowchart illustrating one implementation of the Flink self-starting method based on a file storage system described in this invention. Figure 4 This is another flowchart illustrating one implementation of the Flink self-starting method based on a file storage system described in this invention. Detailed Implementation

[0018] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0019] The Flink self-starting solution refers to automatically starting an Apache Flink cluster or a single Flink application when the operating system boots up, ensuring the continuous and stable operation of a large data stream processing system without manual intervention. This Flink self-starting solution can be used for long-running real-time stream processing tasks or batch processing jobs in production environments.

[0020] In Apache Flink, savepoints allow users to manually create snapshots of a stream processing job's state at any time and resume the job at a later stage. If a job needs to be stopped but restarted at a later time, a savepoint can be triggered first, the job can be safely shut down, and then restarted from the savepoint location, avoiding data loss and duplicate processing. In Apache Flink, checkpointing is the core mechanism for achieving state consistency and fault tolerance in stream processing jobs. Checkpoint state can be stored in reliable persistent storage systems such as HDFS, S3 (Amazon Simple Storage Service), and GCS (Google Cloud Storage), ensuring that state data can be recovered even in the event of node or cluster failure. When a job encounters a failure, Flink automatically recovers the job from the most recently successfully completed checkpoint, restarts all operators, loads the corresponding state, and quickly resumes job processing.

[0021] It's important to note that savepoints are user-controlled snapshots implemented on top of the checkpoint mechanism. They can be viewed as manually triggered checkpoints with additional uses (such as version upgrades or migrations). Savepoints and checkpoints share the same underlying state persistence and recovery mechanism; only their triggering timing and purpose differ. In summary, Flink's checkpoint mechanism enables fast and accurate recovery of stream processing jobs in the face of failures by periodically creating globally consistent snapshots of job states and persisting them to reliable storage. It is a key technology for ensuring high availability and data consistency in large-scale stream processing applications.

[0022] In existing Flink task auto-start methods, each job requires triggering a savepoint, safely shutting down the job, and then restarting the job from the savepoint location. This update process is cumbersome and time-consuming, requiring the user to access Flink's scripting interface and execute scripts each time. Therefore, it is crucial to avoid data loss and duplicate processing, eliminate the need to create savepoints every time, shorten the update cycle, and enable faster program startup.

[0023] Therefore, please refer to Figures 1 to 4 As shown, in this embodiment, the present invention provides a Flink self-starting method based on a file storage system, wherein the file storage system includes several checkpoints, and the method includes the following steps: S1: Determine the target checkpoint for the target task from a number of checkpoints, wherein the target checkpoint is the checkpoint with the latest generation time among the several checkpoints corresponding to the target task.

[0024] S2: Start the target task based on the target checkpoint.

[0025] The structural principle / working principle of the present invention is briefly described as follows: In the present invention, if a target task is to be started (such as a version update), it can first actively connect to the file storage system. In the file storage system, there are several checkpoints corresponding to the target task. Based on the generation time of the above-mentioned checkpoints, the checkpoint with the latest generation time (i.e., the target checkpoint) is determined, and then the task is started according to the target checkpoint, so as to achieve the effect of being generated from the target checkpoint, avoiding data loss and duplicate processing, and not having to create a savepoint every time, shortening the update cycle and speeding up the program startup speed. In other words, the present invention selects the checkpoint with the latest timestamp, aiming to accurately capture the instantaneous state snapshot before the task is interrupted, thereby getting rid of the tedious dependence on manually creating savepoints for each Flink task, realizing seamless automation of task startup, significantly simplifying the operation process before going online, and greatly compressing the overall update cycle. Specifically, the present invention has the following advantages and beneficial effects: (1) No need to manually create savepoints: For large version update scenarios, the traditional practice of manually triggering the creation of savepoints by entering each Flink task one by one is abandoned. (2) One-click start of the task to be updated: After the update preparation stage is completed, the Flink task to be updated can be started with just one click. The system will intelligently identify and load the latest valid savepoints corresponding to each task, ensuring that the task resumes execution from the correct state. There is no need to manually search for and configure savepoint information, which greatly simplifies the startup process. (3) The update process is greatly simplified and accelerated: Through the above-mentioned automation mechanism, the manual creation of savepoints and configuration of startup parameters, which originally required a lot of manpower and time, are greatly simplified or even eliminated, thereby significantly shortening the entire update process. Without being burdened by tedious preliminary preparations, the team can quickly focus on and debug core issues such as the new version's code logic, performance optimization, and exception handling, thereby improving update efficiency and quality. (4) Focus on the essence of the program and improve problem-solving efficiency: The automated update process frees the team's attention from the mechanical steps of starting the program, allowing them to devote themselves to in-depth analysis of the program logic, potential problems, and optimization space. This transformation helps to improve the speed of problem discovery and resolution, enhance the accuracy of version iteration, and thus promote the efficient and stable development of the project.

[0026] It should be noted that the file storage system of this invention can be the MinIO file storage system, HDFS (Hadoop Distributed File System), etc., and is not limited thereto. The MinIO file storage system is an open-source object storage system designed specifically for cloud-native and containerized environments. It uses a distributed architecture to store and manage large-scale unstructured data, such as images, videos, and log files. Furthermore, the target task can be a corresponding Flink task. In this embodiment, Apache Flink is an open-source computing platform for distributed data stream processing and batch data processing. It can support both stream processing and batch processing applications based on the same Flink runtime. Therefore, a Flink task can be either a stream processing task or a batch processing task, depending on the application's requirements and configuration.

[0027] In an embodiment of the present invention, step S1 includes the following steps: S11: Obtain the task name of the target task.

[0028] S12: Based on the task name, obtain the checkpoint storage directory associated with the task name.

[0029] S13: Determine the target checkpoint from all checkpoints contained in the checkpoint storage directory.

[0030] As described above, in this invention, to determine the target checkpoint of a target task from several checkpoints, the task name of the target task can be used as an input parameter. Based on this task name, the system can automatically locate the checkpoint storage directory associated with that task name. Then, the target checkpoint, i.e., the checkpoint with the most recent generation time, is selected from all checkpoints contained in that checkpoint storage directory. In summary, by obtaining the checkpoint storage directory associated with the task name, all checkpoints stored for the target task can be quickly determined, and thus the checkpoint with the most recent generation time (i.e., the target checkpoint) can be quickly obtained. It should be noted that the checkpoint storage directory can contain a mapping relationship between all checkpoint names and file path information corresponding to the target task.

[0031] In an embodiment of the present invention, step S13 includes the following steps: S131: Sort all checkpoints in the checkpoint storage directory in descending order of their generation timestamps to form a descending sorted list.

[0032] S132: Set the first checkpoint at the top of the sequence in the descending list as the target checkpoint.

[0033] As described above, in this invention, all checkpoints in the checkpoint storage directory are sorted in descending order according to their embedded timestamps (i.e., generation timestamps) to form a descending sorted list. The first checkpoint at the top of the sequence is the checkpoint with the latest generation timestamp, i.e., the latest state snapshot. When determining the target checkpoint, the first checkpoint at the top of the sequence can be directly set as the target checkpoint, thus eliminating the need to compare the generation timestamps of all checkpoints in the checkpoint storage directory one by one, saving time and resources. This allows for quick and accurate identification of the checkpoint with the latest generation time (i.e., the target checkpoint), shortening the update cycle and enabling the program to start up faster.

[0034] In an embodiment of the present invention, step S2 includes the following steps: S21: Perform integrity verification on the target checkpoint.

[0035] S22: If the integrity verification passes, set the file path of the target checkpoint to the startup path of the target task.

[0036] In the above-described technical solution of this invention, during the checkpoint generation operation in Flink, although the system strives to ensure a smooth process, the possibility of checkpoint generation failure due to various factors cannot be denied. Therefore, for scenarios relying on checkpoints for task restart, the verification of checkpoint integrity must be given particular attention. This aims to promptly identify and eliminate invalid or damaged checkpoints caused by problems during the generation process. Only when the checkpoint is confirmed to be completely as expected and without damage can the task be started with confidence, thus preventing restart failures caused by using problematic checkpoints and ensuring the stable operation of stream processing jobs and the accuracy of data processing. Therefore, after finding the target checkpoint, its integrity can be verified immediately. Once the integrity of the target checkpoint is confirmed, the file path information corresponding to the target checkpoint can be embedded as a necessary parameter for starting a new task, that is, the file path of the target checkpoint is set as the startup path of the target task. In this way, the Flink job can clearly know from which target checkpoint to resume execution during the startup initialization phase. The entire process requires no manual intervention, effectively improving the convenience of task updates and the consistency of data processing.

[0037] It should be noted that if integrity verification fails, other recent checkpoints can be found in the checkpoint storage directory. In some cases, such as when no integrity verification checkpoint exists, or when no checkpoint exists at all, the Flink task will be started directly instead of starting from a checkpoint.

[0038] In some preferred embodiments, step S21 includes the following steps: determining whether a target file exists in the folder corresponding to the target checkpoint; when the target file exists, the integrity verification of the target checkpoint passes; when the target file does not exist, the integrity verification of the target checkpoint fails.

[0039] As described above, in this invention, Flink checkpoints exist in the form of folders. When Flink generates a checkpoint, it generates a corresponding folder. However, the `_metadata` file (i.e., the target file) within the folder is not generated immediately, even though it should exist in a valid folder. Therefore, to avoid the situation where only the folder is generated without the `_metadata` file, the target checkpoint's integrity can be verified. This is achieved by checking if the `_metadata` file exists. If the `_metadata` file exists, the integrity verification of the target checkpoint passes. If the `_metadata` file does not exist, the integrity verification fails. In summary, verifying the existence of the target file in the folder corresponding to the target checkpoint to verify its integrity aims to promptly identify and eliminate invalid or corrupted checkpoints caused by problems during the generation process. This prevents restart failures due to the use of problematic checkpoints, thereby ensuring the stable operation of the target task and the accuracy of data processing.

[0040] In an embodiment of the present invention, the method further includes the following steps: S3: Periodically generate checkpoints for the target task.

[0041] S4: Store the generated checkpoints to the file storage system.

[0042] As described above, in this invention, checkpoints can be periodically generated for the target task and stored in a file storage system. This allows the target task to be started according to the periodically generated checkpoints when it needs to be started later, achieving the effect of checkpoint generation. This ensures that an accurate snapshot of the target task's state before interruption can be captured without manually creating savepoints. By periodically generating checkpoints for the target task for subsequent update startup, the traditional practice of manually triggering savepoint creation for the target task is eliminated. The entire process requires no manual intervention, effectively improving the convenience and efficiency of target task updates.

[0043] It's important to note that checkpoints periodically generated by Flink tasks (i.e., the target task) can be stored by default in the local file system of the server hosting the task. However, considering the risk of data loss due to a single node failure or an entire cluster failure, a more robust strategy can be adopted: storing checkpoints in a highly available, strongly consistent persistent storage system, such as HDFS, Amazon S3, or Google Cloud Storage. These persistent storage systems, with their distributed architecture, multi-replica redundancy, and automatic fault recovery mechanisms, provide strong security for the data corresponding to the checkpoints. In this way, even in extreme cases of node failure or even cluster-level failure, Flink tasks can quickly retrieve the checkpoint state information from these persistent storage systems, ensuring that the target task can seamlessly and accurately resume execution. This minimizes service interruption time and the risk of data processing inconsistencies, effectively guaranteeing the security and business continuity of the entire stream processing system. Furthermore, after storing the generated checkpoints in the file storage system, integrity checks can be performed on the checkpoints to ensure their validity.

[0044] In an embodiment of the present invention, step S3 includes the following steps: S31: Enable the checkpoint generation mechanism by activating the checkpoint function.

[0045] S32: Set the checkpoint generation cycle.

[0046] S33: Periodically generate checkpoints according to the generation cycle.

[0047] As described above, this invention utilizes the Flink streaming computing engine to strategically enable and optimize the checkpoint generation mechanism. First, by meticulously configuring the checkpoint function, it ensures stable and efficient data consistency during system operation. Second, the checkpoint generation cycle is set, and this cycle should be precisely adjusted based on actual business characteristics and requirements to achieve the optimal balance between performance and reliability. Based on this, checkpoints can be generated periodically according to the set generation cycle. Therefore, by enabling the checkpoint function and setting the checkpoint generation cycle, checkpoints can be generated periodically, allowing tasks to be started based on the periodically generated checkpoints when subsequent target tasks require them.

[0048] In an embodiment of the present invention, step S32 includes the following steps: obtaining the attribute value of a preset indicator; and setting the generation cycle of the checkpoint based on the attribute value.

[0049] It should be noted that, in practical applications, the preset indicators include at least one of the following: business processing speed, data update frequency, and fault recovery time.

[0050] As described above, in this invention, the generation of target task checkpoints is based on a generation cycle. Therefore, the generation cycle for generating target task checkpoints should not be too long to avoid an excessively large amount of data requiring replay and processing during system startup or fault recovery, which could cause significant pressure peaks on upstream and downstream services, affecting the overall system's stability and response efficiency. Similarly, the generation cycle should not be too short to avoid excessive consumption of computing resources due to overly frequent checkpoint generation operations, affecting real-time processing capabilities. To this end, this invention sets the checkpoint generation cycle based on the attribute values ​​of preset indicators. These preset indicators may include factors such as business processing speed, data update frequency, and fault recovery time requirements. It should be noted that these preset indicators can be selected according to actual conditions and are not limited here. In summary, by setting the checkpoint generation cycle based on the corresponding attribute values ​​of preset indicators, this invention can scientifically and rationally determine the checkpoint generation cycle, thereby effectively ensuring the overall system's stability and response efficiency, saving computing resources, and improving real-time processing capabilities.

[0051] In an embodiment of the present invention, the method further includes the following steps: S5: Obtain the task name of the target task and the generation timestamp of the checkpoint generated in step S3.

[0052] S6: Combine and concatenate the task name and the generated timestamp to generate the name of the checkpoint.

[0053] As described above, in this invention, a standardized naming rule is followed during the process of generating checkpoints for the target task. This rule tightly combines the task name with a precise timestamp to form an easily identifiable and orderly identifier. Specifically, the name of each checkpoint is composed of the task name and the generation timestamp, with the task name placed first and the generation timestamp reflecting the generation time placed after the task name. This ensures the clarity of the checkpoint name and the intuitiveness of the time sequence, so that when determining the target checkpoint of the target task, the checkpoint with the latest generation time associated with the target task (i.e., the target checkpoint) can be directly determined based on the checkpoint name, thus improving the efficiency of determining the target checkpoint.

[0054] In summary, the Flink self-starting method based on the file storage system provided by this invention has the following advantages: (1) No need to manually create savepoints: For major version update scenarios, it aims to accurately capture the real-time state snapshot before the task is interrupted, abandoning the traditional practice of manually triggering savepoint creation for each Flink task, realizing seamless automation of task startup, significantly simplifying the pre-launch operation process, greatly compressing the overall update cycle, and enabling the team to focus more on troubleshooting and optimizing the core issues of the program, rather than spending energy on the initial stage of starting the program. (2) One-click startup of tasks to be updated: After the update preparation stage is completed, the Flink task to be updated can be started with just one click. The system will intelligently identify and load the latest valid savepoints corresponding to each task, ensuring that the task resumes execution from the correct state, without the need for manual search and configuration of savepoint information, which greatly simplifies the startup process. (3) The update process is greatly simplified and accelerated: Through the above automation mechanism, the manual creation of savepoints and configuration of startup parameters, which originally required a lot of manpower and time, are greatly simplified or even eliminated, thereby greatly shortening the entire update process. Without being bogged down in tedious preliminary preparations, teams can quickly shift their focus to core issues such as the new version's code logic, performance optimization, and exception handling, thereby improving update efficiency and quality. (4) Focusing on the essence of the program and improving problem-solving efficiency: The automated update process frees the team's attention from the mechanical steps of starting the program, allowing them to fully immerse themselves in in-depth analysis of the program logic, potential problems, and optimization space. This shift helps to improve the speed of problem discovery and resolution, enhance the accuracy of version iteration, and thus promote the efficient and stable development of the project.

[0055] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for launching a Flink target task based on a file storage system, characterized in that, The file storage system includes several checkpoints, and the method includes the following steps: S1: Determine the target checkpoint for the target task from a number of checkpoints, wherein the target checkpoint is the checkpoint with the latest generation time among the number of checkpoints corresponding to the target task; S2: Start the target task based on the target checkpoint; Step S2 includes the following steps: S21: Perform integrity verification on the target checkpoint; S22: If the integrity verification passes, set the file path of the target checkpoint to the startup path of the target task.

2. The method for starting a Flink target task according to claim 1, characterized in that, Step S1 includes the following steps: S11: Obtain the task name of the target task; S12: Based on the task name, obtain the checkpoint storage directory associated with the task name; S13: Determine the target checkpoint from all checkpoints contained in the checkpoint storage directory.

3. The method for starting a Flink target task according to claim 1, characterized in that, Step S13 includes the following steps: S131: Sort all checkpoints in the checkpoint storage directory in descending order of their generation timestamps to form a descending sorted list; S132: Set the first checkpoint at the top of the sequence in the descending list as the target checkpoint.

4. The method for starting a Flink target task according to claim 1, characterized in that, Step S21 includes the following steps: Determine whether the target file exists in the folder corresponding to the target checkpoint. If the target file exists, the integrity verification of the target checkpoint passes; if the target file does not exist, the integrity verification of the target checkpoint fails.

5. The method for starting a Flink target task according to claim 1, characterized in that, It also includes the following steps: S3: Periodically generate checkpoints for the target task; S4: Store the generated checkpoints to the file storage system.

6. The method for starting a Flink target task according to claim 5, characterized in that, Step S3 includes the following steps: S31: Enable the checkpoint generation mechanism by activating the checkpoint function; S32: Set the checkpoint generation cycle; S33: Periodically generate checkpoints according to the generation cycle.

7. The method for starting a Flink target task according to claim 6, characterized in that, Step S32 includes the following steps: obtaining the attribute value of a preset indicator, and setting the generation cycle of the checkpoint based on the attribute value.

8. The method for starting a Flink target task according to claim 7, characterized in that, The preset indicators include at least one of the following: business processing speed, data update frequency, and fault recovery time.

9. The method for starting a Flink target task according to claim 7, characterized in that, The method further includes the following steps: S5: Obtain the task name of the target task and the generation timestamp of the checkpoint generated in step S3; S6: Combine and concatenate the task name and the generated timestamp to generate the name of the checkpoint.

Images