Flink fault tolerance method based on k8s platform

Abstract

The invention discloses an Flink fault tolerance method based on a k8s platform. The method comprises the following steps: deploying an Flink cluster in the k8s platform; during Flink operation, periodically snapshotting a current container guide state, and storing the current container guide state as a check point; running a monitoring framework in the k8s, and monitoring the state of the Flink cluster; and when an Flink fault or a node fault is detected or predicted, a fault recovery process is triggered, the node is restarted or migrated to a healthy node, the Flink automatically recovers the container guide state from the latest check point, and the Flink operation is recovered. According to the method, a fault-tolerant mechanism of an Flink framework is integrated into a k8s platform; a check point storage mechanism of the Flink stores the boot state of the pod in the storage volume persistently, when a new pod is restarted, the stored state snapshot can be directly read, manual configuration is not needed, the restarting time of restarting after an exception is encountered is shortened, and the labor cost of operation and maintenance is saved.

Description

Technical Field

[0001] The present invention relates to the technical field of containerized operation and maintenance, and in particular to a Flink fault-tolerant method based on a k8s platform. Background Art

[0002] As containerized services mature, a more practical containerized orchestration tool, Kubernetes, has emerged. Kubernetes (K8s for short) is an open source container orchestration platform that is mainly used to automate the deployment, management, and expansion of containerized applications. Kubernetes has now become the industry standard for managing containerized applications. The power of Kubernetes lies in its high scalability, automated fault tolerance, and cloud-independent flexibility. It can manage thousands of containers and is suitable for different application scenarios such as microservices, batch processing, and streaming processing, so it is widely used in cloud-native applications. This tool makes up for the pain point of high management costs for a large number of microservice deployments.

[0003] However, in the service model of K8s, stateful services account for a considerable proportion. For the basic K8s migration features, K8s provides the Statefulset feature to implement container migration, but the Statefulset feature can only support the migration of storage volumes, which is equivalent to supporting the migration of databases on containers. There is no way to migrate the boot state stored in the memory, which results in the need to restart the system configuration and bridge configuration after each migration. This is reflected in the specific service as it takes too long to restart a new node or container, and each re-enabling of a node requires manpower and material resources. Summary of the invention

[0004] The present invention aims to solve one of the technical problems existing in the related art at least to a certain extent.

[0005] To this end, the present invention provides a Flink fault-tolerant method based on the k8s platform, which utilizes the save point check mechanism of Flink to support the storage of memory information and integrates the fault-tolerant mechanism of Flink into the k8s platform.

[0006] In order to achieve the above objectives, the present invention provides a Flink fault tolerance method based on the k8s platform, including:

[0007] Deploy the Flink cluster on the k8s platform;

[0008] When Flink is running, it periodically takes snapshots of the current container boot status and saves them as checkpoints.

[0009] Run the monitoring framework in k8s to monitor the status of the Flink cluster;

[0010] When a Flink failure or node failure is detected or predicted, the failure recovery process is triggered, the node is restarted or migrated to a healthy node, and Flink automatically restores the container boot state from the latest checkpoint and resumes the Flink job.

[0011] A further preferred technical solution of the present invention is that the Flink cluster is deployed in the k8s platform, including deploying a JobManager for coordinating and managing tasks and a TaskManager for executing tasks, and introducing configuration checkpoints and high availability mechanisms;

[0012] Use the configuration checkpoint mechanism to create checkpoints;

[0013] Leverage high availability mechanisms to automatically recover jobs from the latest checkpoint.

[0014] Preferably, the Flink cluster is deployed in the k8s platform, and the specific method is as follows:

[0015] Use kubectl and helm tools to manage k8s resources and deploy applications to ensure that the k8s platform is correctly configured;

[0016] On the k8s platform, create a separate namespace for Flink, add the Flink Helm Chart repository, and then deploy Flink in the namespace;

[0017] Customize Helm Chart parameters as needed, including the number of JobManagers and TaskManagers, and resource limits; or write a k8s resource list file to define the Flink cluster;

[0018] After deploying the Flink cluster, submit the Flink job through kubectl or the flink run command provided by Flink.

[0019] As required, configure Flink's storage and log persistent data storage through k8s's PersistentVolume and PersistentVolumeClaim.

[0020] Preferably, during the Flink operation, a snapshot of the current container boot state is periodically taken and saved as a checkpoint. The specific method is as follows:

[0021] In the source of the input stream, Flink periodically inserts fences into the data stream. When the fence reaches an operator, the operator takes a snapshot of the current container boot state and saves it to persistent storage as a checkpoint. Flink waits for all operators to process the current fence event before sending the next fence event to the data stream.

[0022] Preferably, the monitoring framework is run in k8s to monitor the status of the Flink cluster; including: obtaining the identification and monitoring data of the deployed Flink cluster from k8s through the monitoring framework deployed on the Apache FlinkDashboard cloud, and storing this information in the mapping table of the microservice, and triggering the fault recovery process when a failure of the JobManager or TaskManager node is detected.

[0023] As a preferred option, the Flink interface is exposed when the monitoring framework is running, and the fault prediction framework is used to predict the potential fault nodes, savepoint paths, and target healthy node indicators.

[0024] According to the indicators predicted by the fault prediction framework, the fault recovery service is started in advance. Flink calls TaskManager to enable the migration instruction, first executes the C / R operator, records the container boot status to the specified storage volume, and then the C / R executor restores the new service on the target node.

[0025] Preferably, the fault prediction framework is a dual-threaded prediction model of an offline process and an online process, the prediction model is trained according to offline process data, and real-time fault prediction is performed based on the trained prediction model using the online process.

[0026] Preferably, when a Flink failure or a node failure is detected or predicted, a failure recovery process is triggered, the node is restarted or migrated to a healthy node, Flink automatically restores the container boot state from the latest checkpoint, and restores the Flink job; the specific method is:

[0027] When a TaskManager or JobManager failure is detected, the Flink high availability mechanism will try to automatically recover the job from the latest checkpoint;

[0028] JobManager reads the most recent successful checkpoint from persistent storage and restarts the failed task on a healthy node; TaskManager recovers data from the checkpoint state and continues to process subsequent data streams;

[0029] After recovery is complete, Flink continues running the job from the state it was in when the failure occurred.

[0030] Preferably, when a TaskManager or JobManager failure is detected, the automatic restart strategy in k8s restarts the failed TaskManager or JobManager container.

[0031] Preferably, during the fault recovery process, Flink stores the checkpoint storage volume of the potential faulty node persistently.

[0032] Beneficial effects: The present invention integrates the fault-tolerant mechanism of the Flink framework into the k8s platform for the first time; Flink's checkpoint saving mechanism can persist the boot status of the pod in the storage volume, and when the new pod is restarted, the saved status snapshot can be directly read without manual configuration, which can greatly reduce the restart time after encountering an exception and save the labor cost of operation and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is the overall flow chart of the Flink fault-tolerant method based on the k8s platform of the present invention;

[0034] Figure 2 A flowchart for deploying Flink to k8s;

[0035] Figure 3 This is a schematic diagram of the fence mechanism when Flink processes data;

[0036] Figure 4 Schematic diagram of the failure recovery process from a checkpoint. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, and they should not be understood as limitations on the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In the description of the present invention, it should be understood that the terms used are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0038] Combine the following Figure 1-Figure 4The present invention describes the Flink fault tolerance method based on the k8s platform. The method mainly includes: deploying a Flink cluster in the container orchestration tool k8s; when processing stream tasks, the Flink cluster completes the checkpoint saving task through operator monitoring, and ensures the simultaneity of all operator processing tasks; when encountering a node failure, the disk data is saved by combining the migration mechanism provided by k8s with the Flink fault tolerance mechanism to realize the stateful migration feature; the configuration content and disk data saved by the Flink distributed snapshot algorithm are read on a new node to realize the checkpoint recovery function; although Flink is applied to stateful container migration, it currently only stays at the container runtime level. There has been no research on integrating the Flink project into an orchestration tool, such as k8s, and the present invention integrates this function into k8s.

[0039] k8s provides the statfulset feature, but this feature only supports retaining the storage status of the service, which can be simply understood as the database status of the service. However, stateful migration is not just about storage status. There are two other states: boot state and running state, which are stored in memory instead of storage disk. The boot state includes cache, configuration, library loading, and resource initialization settings. If this state is not retained during service migration, the service will have a long startup time due to reinitialization of the configurations mentioned above. The running state describes the current execution status of the service task. If this state is not saved, the node operation will restart. The characteristics of k8s itself do not save these states, but Flink can support saving these states, which can reduce the labor cost of containerized service operation and maintenance.

[0040] Example 1: In order to achieve transparent service continuity after a service interruption caused by a cloud infrastructure failure, the main goal of this example is to reallocate a running pod from one node in a K8s cluster to another node, or between two different clusters without terminating and restarting the pod. To this end, this example utilizes the concept of state snapshots when deployed. The application state in this example is runtime, that is, process information dumped in an "image" file, and can be restored to restore it at the exact moment before the reallocation. The specific steps of this example are as follows:

[0041] S1. Deploy a Flink cluster on the k8s platform; including deploying the JobManager for coordinating and managing tasks and the TaskManager for executing tasks, and introducing configuration checkpoints and high availability mechanisms; using the configuration checkpoint mechanism to create checkpoints; using the high availability mechanism to automatically restore jobs from the latest checkpoint. The specific methods are:

[0042] Use kubectl and helm tools to manage k8s resources and deploy applications to ensure that the k8s platform is correctly configured;

[0043] On the k8s platform, create a separate namespace for Flink to isolate resources, add the Flink HelmChart repository, and then deploy Flink in the namespace; use the Helm Chart officially provided by Flink to quickly deploy a Flink cluster on Kubernetes.

[0044] Customize Helm Chart parameters according to your needs, such as the number of JobManagers and TaskManagers, resource limits, etc. You can also directly write Kubernetes resource manifest files, such as YAML files, to define Flink clusters. For example, define the Deployment and Service of JobManager and TaskManager

[0045] After deploying the Flink cluster, you can submit Flink jobs through kubectl or the flink run command provided by Flink. For example, you can access the JobManager UI through kubectl port forwarding and then submit a job.

[0046] Configure Flink's storage (such as HDFS) and log persistence as required. You can use Kubernetes' PersistentVolume and PersistentVolumeClaim to implement persistent data storage. Use ConfigMap to configure Flink's log location so that it does not lose logs when the Pod is restarted.

[0047] S2. When running a Flink job, periodically take snapshots of the current container boot status and save them as checkpoints. After deploying the Flink cluster on the k8s platform, enable the checkpoint function in the Flink job configuration file (flink-conf.yaml) or code and set related parameters.

[0048] Ensure that state.checkpoints.dir uses a persistent storage path, such as S3, HDFS, or other distributed file systems, to prevent checkpoint data from being lost due to job restarts or node failures.

[0049] After submitting a Flink job with the checkpoint function enabled to a Flink cluster, you can start the job using the flink run command. After the job starts running, Flink automatically creates checkpoints at the configured time interval and stores them in the specified persistent storage location.

[0050] When saving checkpoints, the consistency of operator processing tasks must be ensured. The method used is:

[0051] In the input stream, Flink periodically inserts barriers into the data stream. In the output stream, Flink waits for all operators to process the current barrier event before inserting the next barrier event into the data stream. This ensures that all operators are synchronized at the same point in time.

[0052] When the fence reaches an operator, the operator will take a snapshot of its current state and save the state as a checkpoint. Only when all operators have completed the state saving of the checkpoint, Flink considers the checkpoint to be complete and then continues to process new data streams. This mechanism ensures data consistency and can effectively recover in the event of a failure.

[0053] S3. Run the monitoring framework in k8s to monitor the status of the Flink cluster. The monitoring framework obtains the identifier and monitoring data (such as CPU, memory, bandwidth, and traffic load) of the deployed Flink cluster from Kubernetes, and stores this information in the mapping table of the microservice. The monitoring framework is deployed on the Apache Flink Dashboard cloud.

[0054] Specifically, the monitoring framework deployed on the Apache FlinkDashboard cloud obtains the identification and monitoring data of the deployed Flink cluster from k8s, stores this information in the mapping table of the microservice, and triggers the fault recovery process when a failure of the JobManager or TaskManager node is detected.

[0055] When the monitoring framework is running, the Flink interface is exposed to the outside world and accepts fault prediction using the fault prediction framework. The main indicators predicted by the fault prediction framework after training from the past historical fault data are: potential fault nodes, save point paths, and target healthy nodes. That is, the past fault data is first added to the fault prediction model for prediction, potential fault nodes are marked, and fault recovery services are started in advance.

[0056] The fault prediction framework is a prediction model of dual threads of offline process and online process. The prediction model is trained according to the offline process data, and the online process is used to perform real-time fault prediction based on the trained prediction model. The training method for the prediction model in this embodiment is: use the VM workload data set from Bitbrain to train the model. The Bitbrains data set records the input / output values ​​of CPU, memory, network and disk within 2 months. Since the goal is to predict CPU failures, the CPU indicator is selected according to the percentage of the data set. Then they are normalized to the range of (0,1) before training. After that, the sliding window method is used to prepare the historical subsequence from the normalized data set as the input sequence of the model, where n time steps are used as input to predict the next time step in the next prediction.

[0057] S4: When a Flink failure or node failure is detected or predicted, the failure recovery process is triggered, the node is restarted or migrated to a healthy node, and Flink automatically restores the container boot state from the latest checkpoint and restores the Flink job. The specific method is:

[0058] When running a job in a Flink cluster, if a failure occurs, Flink will automatically try to recover from the latest successful checkpoint and restart the task on the target healthy node. If the Flink cluster has high availability (HA) configuration enabled, Flink will also automatically recover the job from the latest checkpoint when the JobManager crashes or the TaskManager fails to ensure task continuity and data consistency.

[0059] The JobManager reads the most recent successful checkpoint from persistent storage and restarts the failed task on a healthy node; the TaskManager recovers data from the checkpoint state and continues to process subsequent data streams; after the recovery is complete, Flink continues to run the job from the state when the failure occurred.

[0060] like Figure 4 As shown in the figure, in the fault recovery phase, Flink uses the checkpoint fault tolerance mechanism for fault recovery, which is based on a distributed snapshot algorithm: the Chandy-Lamport algorithm. It includes three parts: 1. Back up the intermediate state during runtime; 2. Restart the system after detecting the fault; 3. Read the intermediate state backed up in the first step to complete the initialization, and the system runs normally. After the new node is initialized, the Stateful feature of k8s itself can read the database state saved in the disk and complete the entire node restart process.

[0061] The pod migration process is performed through the following basic steps:

[0062] Send a request to migrate the pod to the system.

[0063] Inspect and capture the status of all containers in a pod.

[0064] Transfer the pod state to the target node. Restore the new pod from the checkpoint.

[0065] Terminate the old pod.

[0066] Redirect the service to the new pod.

[0067] During the fault recovery process, Flink will persist the checkpoint storage volume of the potential faulty node. When the RETAIN_ON_CANCELLATION configuration is enabled, the checkpoint will be retained even if the job is canceled or redeployed. To manage these checkpoint data and ensure that they do not grow indefinitely, you can clean up old checkpoint files regularly. Then, based on the indicators output by the fault prediction framework, read the bootstrap state configuration in the storage volume on the target healthy node to restore the new node.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A Flink fault tolerance method based on the k8s platform, characterized in that: include: Deploy the Flink cluster on the k8s platform; When Flink is running, it periodically takes snapshots of the current container boot status and saves them as checkpoints. Run the monitoring framework in k8s to monitor the status of the Flink cluster; When a Flink failure or node failure is detected or predicted, the failure recovery process is triggered, the node is restarted or migrated to a healthy node, and Flink automatically restores the container boot state from the latest checkpoint and resumes the Flink job.

2. According to the Flink fault-tolerant method based on the k8s platform according to claim 1, it is characterized in that: The deployment of the Flink cluster in the k8s platform includes the deployment of the JobManager for coordinating and managing tasks and the TaskManager for executing tasks, and the introduction of configuration checkpoints and high availability mechanisms; Use the configuration checkpoint mechanism to create checkpoints; Leverage high availability mechanisms to automatically recover jobs from the latest checkpoint.

3. The Flink fault-tolerant method based on the k8s platform according to claim 2, characterized in that: The specific method for deploying the Flink cluster on the k8s platform is as follows: Use kubectl and helm tools to manage k8s resources and deploy applications to ensure that the k8s platform is correctly configured; On the k8s platform, create a separate namespace for Flink, add the Flink Helm Chart repository, and then deploy Flink in the namespace; Customize Helm Chart parameters as needed, including the number of JobManagers and TaskManagers, and resource limits; or write a k8s resource list file to define the Flink cluster; After deploying the Flink cluster, submit the Flink job through kubectl or the flink run command provided by Flink. As required, configure Flink's storage and log persistent data storage through k8s's PersistentVolume and PersistentVolumeClaim.

4. According to the Flink fault-tolerant method based on the k8s platform according to claim 1, it is characterized in that: During the Flink job, the current container boot status is periodically snapshotted and saved as a checkpoint. The specific method is as follows: In the source of the input stream, Flink periodically inserts fences into the data stream. When the fence reaches an operator, the operator takes a snapshot of the current container boot state and saves it to persistent storage as a checkpoint. Flink waits for all operators to process the current fence event before sending the next fence event to the data stream.

5. The Flink fault-tolerant method based on the k8s platform according to claim 2, characterized in that: The monitoring framework is run in k8s to monitor the status of the Flink cluster; including: obtaining the identifier and monitoring data of the deployed Flink cluster from k8s through the monitoring framework deployed on the Apache FlinkDashboard cloud, and storing this information in the mapping table of the microservice, and triggering the fault recovery process when a failure of the JobManager or TaskManager node is detected.

6. According to the Flink fault-tolerant method based on the k8s platform according to claim 5, it is characterized in that: When the monitoring framework is running, the Flink interface is exposed to the outside world, and the fault prediction framework is used to predict the potential fault nodes, savepoint paths, and target healthy node indicators. According to the indicators predicted by the fault prediction framework, the fault recovery service is started in advance. Flink calls TaskManager to enable the migration instruction, first executes the C / R operator, records the container boot status to the specified storage volume, and then the C / R executor restores the new service on the target node.

7. The Flink fault-tolerant method based on the k8s platform according to claim 6 is characterized in that: The fault prediction framework is a dual-threaded prediction model of an offline process and an online process. The prediction model is trained according to offline process data, and real-time fault prediction is performed based on the trained prediction model using the online process.

8. The Flink fault-tolerant method based on the k8s platform according to claim 1 is characterized in that: When a Flink failure or node failure is detected or predicted, the failure recovery process is triggered, the node is restarted or migrated to a healthy node, Flink automatically restores the container boot state from the latest checkpoint, and restores the Flink job; the specific method is: When a TaskManager or JobManager failure is detected, the Flink high availability mechanism will try to automatically recover the job from the latest checkpoint; JobManager reads the most recent successful checkpoint from persistent storage and restarts the failed task on a healthy node; TaskManager recovers data from the checkpoint state and continues to process subsequent data streams; After recovery is complete, Flink continues running the job from the state it was in when the failure occurred.

9. The Flink fault-tolerant method based on the k8s platform according to claim 8, characterized in that: When a TaskManager or JobManager failure is detected, the automatic restart strategy in k8s restarts the failed TaskManager or JobManager container.

10. The Flink fault-tolerant method based on the k8s platform according to claim 8, characterized in that: During the fault recovery process, Flink stores the checkpoint storage volume of the potential faulty node persistently.

Images