Container cloud platform fault self-recovery system and method

Abstract

The invention provides a container cloud platform fault self-recovery system and method. The system comprises a control center unit, a recovery service unit, a recovery instruction unit and a database service unit. The control center unit is mainly responsible for displaying a work log, a task list and parameter configuration of the self-healing system and monitoring health states of the recovery service unit and the recovery instruction unit; the control center writes a task to be executed into the database service unit by configuring a fixed task; the recovery instruction unit is responsible for monitoring the k8s cluster state, reading error information generation tasks such as Node and Pod, and writing the tasks into the database service unit; partial tasks are processed by themselves or directly issued to a recovery service unit through an API (Application Program Interface); the recovery service unit mainly executes specific task implementation, including disk cleaning, memory cleaning, memory arrangement, configuration modification and container repair work; and the database service unit is mainly used for performing datamation storage on logs generated in the operation process of other modules. According to the invention, the efficiency of fault diagnosis and problem solving is improved.

Description

Technical Field

[0001] The present invention belongs to the field of container cloud platforms, and specifically designs a container cloud platform fault self-recovery system and method. Background Art

[0002] Kubernetes (k8s) is a general-purpose enterprise-level container orchestration container platform that can flexibly implement application deployment, load balancing, plug-in management and other functions. Automatic recovery is achieved through a series of automated mechanisms, aiming to improve the stability and availability of the system. The existing fault recovery methods are generally health checks, automatic expansion, fault migration, etc., and most of them still need to rely on manual recovery. These methods have limited fault detection and occupy too high a load. At the same time, the rapid detection mechanism may not be able to detect all types of faults in time, resulting in delays in fault switching. In some cases, fault switching may involve complex configuration and data synchronization issues, which will affect the rapid recovery of services. In addition, in actual work, the regularly backed up data may not be the latest, which will lead to inconsistent data after fault recovery, and the backup data system itself may also have problems, which will affect the reliability and timeliness of data recovery itself. Summary of the invention

[0003] The purpose of the present invention is to provide a container cloud platform fault self-recovery system and method to solve the above technical problems and improve the efficiency of fault diagnosis and problem solving.

[0004] The technical solution to achieve the purpose of the present invention is:

[0005] A container cloud platform fault self-recovery system includes a control center unit, a recovery service unit, a recovery instruction unit, and a database service unit, and is used to implement node exception processing, container group exception processing, and image exception processing, wherein:

[0006] The control center unit is used to display the work log, task list, parameter configuration of the self-healing system, and monitor the health status of the recovery command unit and the recovery service unit. The control center sends the tasks to be executed to the database service unit by configuring the tasks.

[0007] The recovery instruction unit is used to monitor the status of the container cloud platform cluster, read the error information to generate tasks, send them to the database service unit, process the tasks or directly send them to the recovery service unit through the API;

[0008] The recovery service unit is used to perform specific tasks, regularly read the status information of the node where the service is located, and perform cleanup or abnormal recovery work according to the configured threshold; regularly read its own tasks from the database service unit, perform cleanup or abnormal recovery operations; open the API interface to the recovery instruction unit to complete the abnormal recovery work specified by the recovery instruction unit;

[0009] The database service unit is deployed in a single point and is used to store data.

[0010] Furthermore, the control center unit includes a front-end module and a back-end module, wherein:

[0011] The front-end module includes the login interface, component monitoring interface, indicator monitoring interface, self-recovery task interface, and operation log, which are used for human-computer interaction, setting monitoring thresholds for memory, disk, and process, and fixed tasks;

[0012] After receiving the monitoring threshold and fixed tasks from the front-end module, the back-end module writes the information into the database service unit, reads the configuration variables to obtain the IP address and service port, forwards the request to the back-end virtual IP, and the request finally reaches the recovery service unit. It also receives the data from the recovery service unit and sends the data to the front-end module for display.

[0013] Furthermore, the node exception handling includes: the recovery instruction unit periodically traverses and queries the cluster node information to determine whether the node has memory pressure, disk pressure and process pressure. When the set threshold is reached, the lock of the processing task is obtained, and the recovery service unit on the corresponding node is called to clean up the node memory, disk, and zombie threads, and the operation log is written to the database service unit to release the database lock.

[0014] Furthermore, the container group exception processing includes: querying cluster information, traversing container group information, when the container group is in a pending state and the container state is unknown, the recovery instruction unit obtains the lock of the processing task, adopts a time-sharing operation algorithm, actively checks whether the corresponding node is abnormal, and then cleans up the disk and memory; when the exception processing reaches the set time and the container group state is still abnormal, the container group is judged to be abnormal, the abnormal container group is actively deleted, and the corresponding operation is written into the database service unit to release the database lock.

[0015] Furthermore, the image exception processing includes: querying cluster information, traversing container group information, and when the container group fails to pull an image, an image check error, or fails to create a container, the recovery instruction unit obtains the lock of the processing task, and repairs the specified image by modifying the image pulling strategy, or requesting the recovery service unit on the node where the container group is located, deleting the specified image and then reloading the image from the specified directory. When the container group status is still abnormal after the set time, the abnormal container group is deleted, and the corresponding operation is written to the database service unit to release the database lock.

[0016] Furthermore, the set time is 5 minutes.

[0017] A container cloud platform fault self-recovery method, comprising:

[0018] Configure tasks through the control center unit and send the tasks to be executed to the database service unit;

[0019] Monitor the status of the container cloud platform cluster through the recovery instruction unit, read the error information to generate tasks, send them to the database service unit, process the tasks or send them directly to the recovery service unit through the API;

[0020] The recovery service unit periodically reads the status information of the node where the service is located, and performs cleanup or abnormal recovery work according to the configured threshold; periodically reads its own tasks from the database service unit, performs cleanup or abnormal recovery operations; opens the API interface to the recovery instruction unit to complete the abnormal recovery work specified by the recovery instruction unit;

[0021] The database service unit is used to store the log information generated by each task operation.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] (1) High availability principle: each module can run the self-healing system independently and be deployed independently outside the K8S system to avoid the self-healing system itself being unable to work normally when the K8S function fails. When the control center itself is paralyzed, it does not affect the normal operation of the recovery command unit and recovery service unit on each host. The recovery command unit and recovery service unit can independently complete routine abnormal recovery work.

[0024] (2) Since there are multiple recovery instruction units in a multi-master cluster, the present invention needs to use the method of preempting the DB lock to achieve the consistency of multi-node scheduled tasks; each recovery instruction unit will go to the database to insert a task data before executing the scheduled task, and the DB lock is used to limit only one recovery instruction unit to successfully insert data in each time period, that is, only one recovery instruction unit can be successfully executed.

[0025] (3) It helps improve the efficiency of fault diagnosis and problem solving; the automated fault detection and recovery mechanism reduces the time and workload of human intervention and manual troubleshooting, thereby saving manpower and material resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram of the system.

[0027] Figure 2 This is a flowchart of node exception handling.

[0028] Figure 3 This is a flowchart of pod exception handling.

[0029] Figure 4 This is a flowchart of image exception processing. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application of the specification, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0031] k8s principle: k8s is an open source container orchestration platform for automated deployment, expansion and management of containerized applications. It mainly consists of the following parts: APISerer is the only interface of the k8s cluster and handles all API requests; Schedule is responsible for resource scheduling; kubelet runs on each node to ensure that the container is carried out as required and manages the life cycle of Pod (container group); users submit requests to APIServer through the API client, Server processes the request, Schedule selects the appropriate node according to the strategy, and kubelet starts the corresponding Pod as instructed.

[0032] Combination Figure 1 The present application provides a container cloud platform fault self-recovery system, including a control center unit, a recovery service unit, a recovery instruction unit, and a database service unit. The self-recovery system is designed for high availability, and each module can run independently and be independently deployed outside the K8S system to avoid the self-recovery system itself from failing to work properly when the k8s function fails. The container cloud platform fault self-recovery system supports conventional fault self-healing and probe health checks, and can also achieve coverage of multiple common container cloud platform fault self-healing.

[0033] Since there are multiple recovery instruction units in a multi-master cluster, it is necessary to use the method of preempting the db lock to achieve the consistency of multi-node scheduled tasks. Each recovery instruction unit will insert a task data into the database before executing the scheduled task. The db lock (database lock) is used to limit only one recovery instruction unit to successfully insert data in each time period, that is, only one recovery instruction unit is successfully executed.

[0034] The control center is mainly responsible for displaying the work log, task list, parameter configuration of the self-healing system, and monitoring the health status of the recovery command unit and the recovery service unit. The control center writes the routine tasks to be executed into the database service unit by configuring fixed tasks. The control center unit includes the front-end module recovery-center-fronter and the back-end module recovery-center-backend. The control center is mainly responsible for displaying the work log, task list, parameter configuration of the self-healing system, and monitoring the health status of the recovery command unit and the recovery service unit; the front-end module is mainly the interface display of the self-recovery system, including the login interface, component monitoring, indicator monitoring, self-recovery tasks and operation logs. The monitoring thresholds of memory, disk, pid, and fixed tasks are set through the interface. After receiving the monitoring indicator thresholds and fixed tasks, the back-end module will write the information into the database, and will also read the configuration variables to obtain the IP address and service port, forward the request to the back-end virtual IP, and the request will finally reach the recovery service unit. At the same time, it will receive the data of the recovery service unit and send the data to the front-end module. According to the principle of independent design, when the control center itself fails, it does not affect the normal operation of the recovery command unit and the recovery service unit on each host, and other functional components can complete the routine abnormal recovery work.

[0035] The recovery command unit is responsible for monitoring the status of the k8s cluster, reading error information such as Node and Pod, generating tasks and writing them into the database service unit; handling various pod and node exceptions, handling some tasks by itself or directly sending them to the recovery service unit through the API; at the same time, the recovery command unit is deployed in a high-availability manner, deployed on multiple (usually odd) management nodes, and provides services to the outside world through a unique VIP network address. When a recovery command unit fails, other recovery command units will take over the business, and the failure of the recovery command unit can be tolerated. At the same time, a working node will be elected to supplement the recovery command unit to keep the number of recovery command units in the system unchanged.

[0036] The recovery instruction unit accesses the APIServer component of k8s, periodically requests the cluster status, and obtains various abnormal statuses. The recovery instruction unit will call the corresponding API interface of the recovery service unit on the faulty node. The recovery service unit will implement the corresponding fault self-recovery process to realize the self-healing of the container cloud platform. The recovery instruction unit will write the corresponding self-recovery task into the database service unit. The database service unit will store the received information and return the received signal to the recovery instruction unit.

[0037] Combination Figure 2 In the specific design, the recovery instruction unit will periodically traverse and query the cluster node information to determine whether the node has memory pressure, disk pressure, and process pressure. When the set threshold is reached, the recovery instruction unit will obtain the lock of the processing task, and then call the recovery service unit on the corresponding node to clean up the node memory, disk, and pid zombie threads, and write the operation log into the data to release the db lock.

[0038] Combination Figure 3 When processing pod exceptions, the recovery instruction unit first queries the cluster information and traverses the pod information. When the pod is in pending (pending) or containerStatusUnknown (container status unknown) exceptions, the recovery instruction unit will get the lock of the processing task and adopt a time-sharing operation algorithm. It will actively check whether the corresponding node is abnormal, and then clean up the disk, clean up the memory and other operations; when the status is still pending or containerStatusUnknown (container status unknown) five minutes after creation, when the status is still pendingcontainer (with processing container) or StatusUnknown (status unknown), the pod is judged to be abnormal, and the abnormal pod is actively deleted, and the corresponding operations are written to the database to release the db lock.

[0039] Combination Figure 4 When processing image exceptions, the recovery instruction unit first queries the cluster information and traverses the pod information. When the pod has ImagePullBackoff (image pull failure), ImageInspectError (image check error), or CreateContainerError (container creation failure) exceptions, the recovery instruction unit will get the lock for processing tasks, and then modify the image pull strategy, or request the recovery service unit service on the node where the pod is located to repair the specified image, delete the specified image, and then reload the image from the specified directory. If the status is still abnormal five minutes after creation, the abnormal pod will be deleted, and the corresponding operations will be written to the database to release the db lock.

[0040] The recovery service unit mainly receives the scheduling commands of the recovery instruction unit and executes the corresponding tasks regularly, including cleaning the disk, cleaning the memory, organizing the memory, modifying the configuration, and deleting the image. At the same time, it receives the scheduling commands of the recovery instruction unit, executes the corresponding self-recovery operations, and completes the container self-recovery tasks.

[0041] During the specific design, the recovery service unit multi-processing task will first determine the node status, including whether the disk space, memory space, and image space have reached the threshold. If the threshold is reached, it will then determine whether the db lock has been grabbed. If the db lock has not been grabbed, the task will be placed in the waiting queue. If the db lock is grabbed, the cache, zombie processes, and memory will be cleaned up. Then, the kubelet on the corresponding node will be started to execute the fault recovery task, and then the preempted db lock will be released with a delay to continue the execution of the next task.

[0042] The database service unit mainly stores log information generated by each task when the container cloud platform fault self-recovery system is running, which is convenient for front-end display and offline analysis.

[0043] The database is deployed at a single point, and cluster deployment is not considered for the time being to avoid introducing new complexity. The system design requires that recovery-service and recovery-kube-service can still complete most of the abnormal recovery work after the database fails. After recovery-service / recovery-kube-service completes abnormal recovery, it writes the operation log to the database, and the control center reads the operation log and displays it to the front end. Recovery-service / recovery-kube-service pulls its own task information from MySQL and performs abnormal recovery work.

[0044] The present invention helps to improve the efficiency of fault diagnosis and problem solving; the automated fault detection and recovery mechanism reduces the time and workload of human intervention and manual troubleshooting, thereby saving manpower and material resources.

[0045] A container cloud platform fault self-recovery method, comprising:

[0046] Configure tasks through the control center unit and send the tasks to be executed to the database service unit;

[0047] Monitor the status of the container cloud platform cluster through the recovery instruction unit, read the error information to generate tasks, send them to the database service unit, process the tasks or send them directly to the recovery service unit through the API;

[0048] The recovery service unit periodically reads the status information of the node where the service is located, and performs cleanup or abnormal recovery work according to the configured threshold; periodically reads its own tasks from the database service unit, performs cleanup or abnormal recovery operations; opens the API interface to the recovery instruction unit to complete the abnormal recovery work specified by the recovery instruction unit;

[0049] The database service unit is used to store the log information generated by each task operation.

[0050] This embodiment also provides a container cloud platform fault self-recovery device, including: a memory, a processor and a computer program stored in the memory, and when the processor executes the computer program, the steps of the container cloud platform fault self-recovery method described in 7 are implemented.

[0051] A computer storage medium stores an executable program, and the executable program is executed by a processor to implement the steps of the container cloud platform fault self-recovery method.

[0052] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A container cloud platform fault self-recovery system, characterized by: It includes a control center unit, a recovery service unit, a recovery instruction unit, and a database service unit, which are used to implement node exception handling, container group exception handling, and image exception handling, wherein: The control center unit is used to display the work log, task list, parameter configuration of the self-healing system, and monitor the health status of the recovery command unit and the recovery service unit. The control center sends the tasks to be executed to the database service unit by configuring the tasks. The recovery instruction unit is used to monitor the status of the container cloud platform cluster, read the error information to generate tasks, send them to the database service unit, process the tasks or directly send them to the recovery service unit through the API; The recovery service unit is used to perform specific tasks, regularly read the status information of the node where the service is located, and perform cleanup or abnormal recovery work according to the configured threshold; regularly read its own tasks from the database service unit, perform cleanup or abnormal recovery operations; open the API interface to the recovery instruction unit to complete the abnormal recovery work specified by the recovery instruction unit; The database service unit is deployed in a single point and is used to store data.

2. A container cloud platform fault self-recovery system according to claim 1, characterized in that: The control center unit includes a front-end module and a back-end module, wherein: The front-end module includes the login interface, component monitoring interface, indicator monitoring interface, self-recovery task interface, and operation log, which are used for human-computer interaction, setting monitoring thresholds for memory, disk, and process, and fixed tasks; After receiving the monitoring threshold and fixed tasks from the front-end module, the back-end module writes the information into the database service unit, reads the configuration variables to obtain the IP address and service port, forwards the request to the back-end virtual IP, and the request finally reaches the recovery service unit. It also receives the data from the recovery service unit and sends the data to the front-end module for display.

3. A container cloud platform fault self-recovery system according to claim 1, characterized in that: The node exception processing includes: the recovery instruction unit periodically traverses and queries the cluster node information to determine whether the node has memory pressure, disk pressure and process pressure. When the set threshold is reached, the lock of the processing task is obtained, and the recovery service unit on the corresponding node is called to clean up the node memory, disk, and zombie threads, and the operation log is written to the database service unit to release the database lock.

4. A container cloud platform fault self-recovery system according to claim 1, characterized in that: The container group exception processing includes: querying cluster information, traversing container group information, when the container group is in pending processing and the container status is unknown, the recovery instruction unit obtains the lock of the processing task, adopts a time-sharing operation algorithm, actively checks whether the corresponding node is abnormal, and then cleans up the disk and memory; when the container group status is still abnormal after the exception processing reaches the set time, it is determined that the container group is abnormal, and the abnormal container group is actively deleted, and the corresponding operation is written into the database service unit to release the database lock.

5. A container cloud platform fault self-recovery system according to claim 4, characterized in that: The image exception processing includes: querying cluster information, traversing container group information, and when the container group fails to pull an image, an image check error, or fails to create a container, the recovery instruction unit obtains the lock of the processing task, and repairs the specified image by modifying the image pulling strategy, or requesting the recovery service unit on the node where the container group is located, deleting the specified image and then reloading the image from the specified directory. When the container group status is still abnormal after the set time, the abnormal container group is deleted, and the corresponding operation is written to the database service unit to release the database lock.

6. A container cloud platform fault self-recovery system according to claim 5, characterized in that: The setting time is 5 minutes.

7. A container cloud platform fault self-recovery method using the system described in any one of claims 1 to 6, characterized in that: include: Configure tasks through the control center unit and send the tasks to be executed to the database service unit; Monitor the status of the container cloud platform cluster through the recovery instruction unit, read the error information to generate tasks, send them to the database service unit, process the tasks or send them directly to the recovery service unit through the API; The recovery service unit periodically reads the status information of the node where the service is located, and performs cleanup or abnormal recovery work according to the configured threshold; periodically reads its own tasks from the database service unit, performs cleanup or abnormal recovery operations; opens the API interface to the recovery instruction unit to complete the abnormal recovery work specified by the recovery instruction unit; The database service unit is used to store the log information generated by each task operation.

8. A container cloud platform fault self-recovery device, characterized in that: include: A memory, a processor and a computer program stored in the memory, wherein the processor implements the steps of the container cloud platform fault self-recovery method described in claim 7 when executing the computer program.

9. A computer storage medium, characterized in that The computer storage medium stores an executable program, and the executable program is executed by the processor to implement the steps of the container cloud platform fault self-recovery method according to claim 7.

Images