A fast fault-tolerant method for container orchestration scheduling

Abstract

The application belongs to the technical field of computer system software, and particularly relates to a quick fault-tolerant method for container orchestration scheduling. The application adopts a container deployment mode designed for quick fault tolerance, plans the number of normal operation Pod replicas and the number of backup container replicas and resource requirements in advance, creates and starts the Pod required for normal operation, allocates resources on the available idle nodes according to the number of backup container replicas, and pre-creates the container application. When a node fails, the available backup replica is directly selected and the container is started without the process of scheduling and resource allocation, and a new backup replica is dynamically generated. The application reduces the conventional second-level container fault discovery and fault tolerance to the millisecond level, and meets the quick fault-tolerant requirement of critical business.

Description

Technical Field

[0001] This invention belongs to the field of computer system software technology, and specifically relates to a fast fault-tolerant method for container orchestration and scheduling. Background Technology

[0002] With the rapid development of general-purpose container and service technologies, containers, as lightweight virtualization technologies, enable applications to seamlessly migrate and run in different computing environments. They have become the best carrier for application deployment, dynamic migration, and microservices, and the embedded real-time field is also researching and following up on the implementation of container technology. However, in mission-critical scenarios, the high availability and rapid fault recovery of containerized applications become key factors determining system stability. Mission-critical containers have real-time and continuous requirements; services within containers cannot tolerate prolonged downtime or interruption. Even a brief period of service unavailability can lead to significant losses. If a simple migration and restart strategy is used when a container fails, it involves multiple steps such as image download and resource allocation, which is time-consuming and cannot meet the real-time requirements of mission-critical applications. Summary of the Invention

[0003] In view of this, and considering the situation where the extremely high availability and real-time requirements of critical business applications cannot be met by conventional container recovery mechanisms, this invention proposes a fast fault-tolerance method for container orchestration and scheduling. Through techniques such as pre-scheduling, resource pre-allocation, and rapid activation in case of failure, it ensures that a backup container can immediately take over when a container fails, thereby guaranteeing the continuity and stability of critical applications. This design can effectively reduce fault recovery time and better address the fault-tolerance requirements of critical applications.

[0004] The technical solution of this invention:

[0005] A fast fault-tolerance method for container orchestration and scheduling is used to achieve fast fault tolerance for mission-critical applications based on a container orchestration system, wherein the container orchestration system includes:

[0006] The management center is used for overall cluster status management, including interface servers, schedulers, and run controllers. The interface servers are responsible for providing access to and storage of various resource data; the schedulers are responsible for scheduling Pods; and the run controllers are responsible for creating Pod resource objects with running entities according to deployment strategies. One Pod corresponds to one or more containers.

[0007] The node agent is responsible for reporting basic information about nodes and applications, as well as their load status. Based on the Pod deployment requirements issued by the management center, it uses the standard CRI interface to call the container engine to pull images, allocate actual resources, and start Pods and containers.

[0008] The container engine is used to actually perform Pod and container lifecycle management functions and allocate network resources to containers through CNI plugins;

[0009] Specifically: The management center plans the number of Pod replicas and the number of spare container replicas and resource requirements for normal operation based on user needs; after creating and starting the Pods required for normal operation, the management center allocates resources on available idle nodes based on the number of spare container replicas, and pre-creates container applications.

[0010] Furthermore, based on the container orchestration system, a deployment strategy for rapid fault tolerance of critical business operations is proposed, which includes:

[0011] Dedicated resource types are used to describe fast fault tolerance strategies, and fast fault tolerance resources are integrated into the container orchestration system using custom resources.

[0012] Designed as a dedicated controller for handling fast fault tolerance strategies, this controller is directly integrated into the container orchestration system by interacting with an interface server.

[0013] By customizing the node agent, when a Pod has a specified label, only the standby Pod is started and a container is created, but the container is not started.

[0014] Furthermore, the method includes using dedicated resource types to describe fast fault tolerance strategies, the resource types including:

[0015] Resource type name and version;

[0016] Resource Name;

[0017] Resource specifications, including:

[0018] Pod tag selector, used to filter currently managed Pod objects;

[0019] A Pod template, which includes the Pod's scheduling and resource requirements, as well as a list of containers and images within the Pod;

[0020] The required number of Pod replicas;

[0021] The number of backup Pod replicas;

[0022] Resource status, including:

[0023] The number of replicas of a normally functioning Pod;

[0024] The number of replicas of the Pod waiting to run.

[0025] Furthermore, the method includes generating Pod replicas after processing the fast fault tolerance strategy using a dedicated controller, and selectively activating some Pod replicas based on the current state. The specific execution logic includes:

[0026] For the next pending fast fault-tolerant resource in the circular processing queue:

[0027] If the fast fault-tolerant resources are not deleted, obtain the number of normally running Pod replicas and the number of created but not started Pods, and compare them with the required number of Pod replicas and the number of spare Pod replicas. If the number of Pods is insufficient, create new Pods. If the number of Pods exceeds the limit, delete the extra Pods. If the number of normally running Pod replicas is less than the required number of Pod replicas, activate the created but not started Pods according to the difference.

[0028] If the fast fault-tolerant resource has been deleted, determine whether to delete all Pod sub-resources according to the standard deletion policy of container orchestration.

[0029] When a fast fault-tolerant resource changes, the resource is added to a queue and waits for processing in a loop.

[0030] When a Pod changes, including adding, updating, or deleting, if the Pod's owner is a fast fault-tolerant resource, the corresponding fast fault-tolerant resource is added to a queue and awaits processing in a loop.

[0031] Furthermore, the Pod does not start immediately after it is created and scheduled to the node. Instead, it is activated and started by the fast fault tolerance controller. When fault tolerance is required, the fast fault tolerance controller removes the specified label from the Pod. After receiving the update request, the node agent immediately starts the standby container.

[0032] In this embodiment, after a Pod fails, a backup Pod is immediately notified to start, thereby significantly improving fault tolerance performance.

[0033] The beneficial effects of this invention are:

[0034] (1) High availability: By pre-configuring backup Pod replicas, the application can quickly switch to the backup Pod in case of failure, thus ensuring high availability.

[0035] (2) Rapid fault recovery: Since the backup Pod is created and initialized in advance, all startup preparations have been completed. When a fault occurs, near-instantaneous cross-node failover can be achieved, greatly reducing recovery time.

[0036] (3) State consistency guarantee: Based on the container orchestration system, by sharing storage resources among Pod replicas, the standby Pod can immediately obtain all the record data of the faulty Pod after it starts up, thereby quickly restoring the running state and providing normal services to the outside world;

[0037] (4) Flexibility: Users can customize the number of replicas, Pod resources, and scheduling methods by using fast fault-tolerant resources according to the actual needs of different applications.

[0038] (5) Non-intrusive design: The rapid fault tolerance technology is integrated with the container orchestration system, and users can use the fault tolerance strategy to improve the availability of the application without making any modifications or special designs to the business application. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 A system architecture diagram for a fast fault-tolerant method for container orchestration and scheduling;

[0041] Figure 2 A schematic diagram illustrating the rapid fault-tolerant resource description;

[0042] Figure 3 A flowchart illustrating how a fast fault-tolerant controller processes new resources;

[0043] Figure 4 A flowchart illustrating fault tolerance processing for a fast fault-tolerant controller;

[0044] Figure 5 This is a diagram comparing fast fault tolerance and general fault tolerance processes. Detailed Implementation

[0045] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0046] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0047] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using other structures and / or functionalities besides one or more of the aspects set forth herein.

[0048] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The illustrations only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0049] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0050] In one embodiment of the present invention, a fast fault-tolerance method for container orchestration and scheduling is proposed, which is used to achieve fast fault tolerance for mission-critical applications based on a container orchestration system, wherein the container orchestration system includes:

[0051] The management center is used for overall cluster status management, including interface servers, schedulers, and run controllers. The interface servers are responsible for providing access to and storage of various resource data; the schedulers are responsible for scheduling Pods; and the run controllers are responsible for creating Pod resource objects with running entities according to deployment strategies. One Pod corresponds to one or more containers.

[0052] The node agent is responsible for reporting basic information about nodes and applications, as well as their load status. Based on the Pod deployment requirements issued by the management center, it uses the standard CRI interface to call the container engine to pull images, allocate actual resources, and start Pods and containers.

[0053] The container engine is used to actually perform Pod and container lifecycle management functions and allocate network resources to containers through CNI plugins;

[0054] Specifically: The management center plans the number of Pod replicas and the number of spare container replicas and resource requirements for normal operation based on user needs; after creating and starting the Pods required for normal operation, the management center allocates resources on available idle nodes based on the number of spare container replicas, and pre-creates container applications.

[0055] In this embodiment, based on the container orchestration system, a deployment strategy for rapid fault tolerance of critical business operations is implemented. This deployment strategy includes:

[0056] Dedicated resource types are used to describe fast fault tolerance strategies, and fast fault tolerance resources are integrated into the container orchestration system using custom resources.

[0057] Designed as a dedicated controller for handling fast fault tolerance strategies, this controller is directly integrated into the container orchestration system by interacting with an interface server.

[0058] By customizing the node agent, when a Pod has a specified label, only the standby Pod is started and a container is created, but the container is not started.

[0059] In this embodiment, the method includes using dedicated resource types to describe fast fault tolerance strategies, the resource types including:

[0060] Resource type name and version;

[0061] Resource Name;

[0062] Resource specifications, including:

[0063] Pod tag selector, used to filter currently managed Pod objects;

[0064] A Pod template, which includes the Pod's scheduling and resource requirements, as well as a list of containers and images within the Pod;

[0065] The required number of Pod replicas;

[0066] The number of backup Pod replicas;

[0067] Resource status, including:

[0068] The number of replicas of a normally functioning Pod;

[0069] The number of replicas of the Pod waiting to run.

[0070] In this embodiment, the method includes generating Pod replicas after processing the fast fault tolerance strategy using a dedicated controller, and selectively activating some Pod replicas based on the current state. The specific execution logic includes:

[0071] For the next pending fast fault-tolerant resource in the circular processing queue:

[0072] If the fast fault-tolerant resources are not deleted, obtain the number of normally running Pod replicas and the number of created but not started Pods, and compare them with the required number of Pod replicas and the number of spare Pod replicas. If the number of Pods is insufficient, create new Pods. If the number of Pods exceeds the limit, delete the extra Pods. If the number of normally running Pod replicas is less than the required number of Pod replicas, activate the created but not started Pods according to the difference.

[0073] If the fast fault-tolerant resource has been deleted, determine whether to delete all Pod sub-resources according to the standard deletion policy of container orchestration.

[0074] When a fast fault-tolerant resource changes, the resource is added to a queue and waits for processing in a loop.

[0075] When a Pod changes, including adding, updating, or deleting, if the Pod's owner is a fast fault-tolerant resource, the corresponding fast fault-tolerant resource is added to a queue and awaits processing in a loop.

[0076] In this embodiment, the Pod does not start immediately after it is created and scheduled to the node. Instead, it is activated and started by the fast fault tolerance controller. When fault tolerance is required, the fast fault tolerance controller removes the specified label from the Pod. After receiving the update request, the node agent immediately starts the backup container.

[0077] In this embodiment, after a Pod fails, a backup Pod is immediately notified to start, thereby significantly improving fault tolerance performance.

[0078] The fast fault-tolerant method for container orchestration and scheduling in this embodiment is implemented based on a container orchestration system and includes a dedicated fast fault-tolerant controller, comprising: a management center, node agents, a container engine, a fast fault-tolerant controller, and fast fault-tolerant resource descriptions, such as... Figure 1 As shown;

[0079] The fast fault-tolerant resource description in this embodiment can be defined using a JSON / YAML format file, and its content and meaning are as follows: Figure 2 As shown;

[0080] After a user creates a fast fault-tolerant resource, the fast fault-tolerant controller processes it, creates Pods through the interface server and node proxy, and activates a specified number of Pods. Figure 3 As shown;

[0081] When a node fails, the fast fault-tolerant controller handles the situation and activates a backup Pod, as follows: Figure 4 As shown;

[0082] When a node fails, general fault tolerance technologies require processes such as rebuilding the Pod, scheduling the Pod, allocating resources, creating the Pod and its container, and starting the container. In contrast, fast fault tolerance technologies only require activating the Pod and starting the container. Their processes are similar. Figure 5 As shown.

[0083] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A fast fault-tolerant method for container orchestration scheduling, characterized in that, A method for implementing fast fault tolerance for key business based on a container orchestration system, the container orchestration system comprising: a management center for overall state management of the cluster, including an interface server, a scheduler, and a running controller, wherein the interface server is responsible for providing access and storage of various resource data; the scheduler is responsible for scheduling Pods; and the running controller is responsible for creating Pod resource objects with running entities according to deployment strategies; a node agent responsible for reporting basic information of nodes and applications and load states of nodes and applications, and according to Pod deployment requirements issued by the management center, completing image pulling and actual resource allocation and starting Pods and containers through standard CRI interfaces by calling a container engine; a container engine for actually executing Pod and container lifecycle management functions and allocating network resources for containers through CNI plug-ins; wherein: the management center plans the number of normal running Pod replicas and the number of backup container replicas and resource requirements according to user needs; after creating and starting the Pods required for normal operation, the management center allocates resources on available idle nodes according to the number of backup container replicas, and pre-creates container applications; on the basis of the container orchestration system, a deployment strategy for fast fault tolerance for key business, the deployment strategy comprising: using a dedicated resource type to describe the fast fault tolerance strategy, and integrating the fast fault tolerance resources into the container orchestration system in the form of a custom resource; designing a dedicated controller for processing the fast fault tolerance strategy, and integrating the fast fault tolerance controller directly into the container orchestration system by interacting with the interface server; customizing the node agent, and when a Pod has a specified label, only starting a backup Pod and creating a container, and not starting a container; the method comprises using a dedicated resource type to describe the fast fault tolerance strategy, the resource type comprising: a resource type name and version; a resource name; a resource specification, including: a Pod label selector for filtering the currently managed Pod objects; a Pod template containing scheduling requirements and resource requirements of the Pod, as well as a list of containers and images in the Pod; the number of required Pod replicas; the number of backup Pod replicas; a resource state, including: the number of normal running Pod replicas; the number of waiting running Pod replicas.

2. The rapid fault-tolerant method of container orchestration scheduling according to claim 1, wherein, the method comprises generating Pod replicas after processing the fast fault tolerance strategy using a dedicated controller, and selectively activating part of the Pod replicas in combination with the current state, and the specific execution logic comprises: for the next fast fault tolerance resource to be processed in the loop processing queue: if the fast fault tolerance resource has not been deleted, obtaining the number of normal running Pod replicas and the number of created but not started Pods, and comparing them with the number of required Pod replicas and the number of backup Pod replicas, if the number of Pods is insufficient, creating a new Pod, if the number of Pods exceeds, deleting the excess Pods, and if the number of normal running Pod replicas is less than the number of required Pod replicas, activating the created but not started Pods according to the difference. If the fast fault-tolerant resource has been deleted, it is determined whether to delete all Pod sub-resources according to the standard deletion strategy of container orchestration; When the fast fault-tolerant resource changes, the resource is added to the queue and waits for the loop processing; When the Pod changes, the changes include addition, update and deletion, and if the owner of the Pod is the fast fault-tolerant resource, the corresponding fast fault-tolerant resource is added to the queue and waits for the loop processing.

3. The method of claim 2, wherein, The Pod is not started immediately after being created and scheduled to the node, but is started after being activated by the fast fault-tolerant controller; when fault tolerance is needed, the Pod is removed from the specified label by the fast fault-tolerant controller, and the standby container is started immediately after the node agent receives the update request.

4. The method of claim 3, wherein, After the Pod fails, the standby Pod is notified to start immediately, thereby greatly improving the fault tolerance performance.

Images