Rapid fault-tolerant method for container arrangement and scheduling

Abstract

The invention belongs to the technical field of computer system software, and particularly relates to a fast fault-tolerant method for container arrangement scheduling, which adopts a container deployment mode designed for fast fault tolerance, pre-plans the number of normally operating Pod copies, the number of standby container copies and resource requirements, and establishes and starts Pod required by normal operation after the Pod required by normal operation is established and started. And according to the number of the standby container copies, completing resource allocation and pre-creation of the container application at the available idle nodes. When the node has a fault, the available standby copy is directly selected and the container is started without scheduling and resource allocation processes, and a new standby copy is dynamically generated. Container fault discovery and fault tolerance at the conventional second level are reduced to be at the millisecond level, and the rapid fault tolerance requirement of key services is met.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer system software, and in particular relates to a fast fault-tolerant method for container orchestration and scheduling. Background Art

[0002] With the rapid development of container technology and service technology in general fields, containers, as lightweight virtualization technology, enable applications to migrate and run seamlessly in different computing environments. They have become the best carrier for application deployment, dynamic migration, and microservices. The embedded real-time field is also studying and following up on the implementation of container technology. However, in critical business scenarios, the high availability and rapid fault recovery of containerized applications have become key factors in determining system stability. Critical business containers have real-time and continuous requirements. The services in the container cannot tolerate long periods of downtime or interruption. Even short-term service unavailability may cause huge losses. If only a simple migration and restart strategy is adopted when a container fails, it will involve multiple steps such as image download and resource allocation, which takes a long time and cannot meet the real-time requirements of critical business applications. Summary of the invention

[0003] In view of this, in view of the fact that the extremely high requirements of key businesses for high availability and real-time performance cannot be met by conventional container recovery mechanisms, the present invention proposes a fast fault-tolerant method for container scheduling and scheduling. Through pre-scheduling, resource pre-allocation, and fast activation in case of failure, it is ensured that when a container fails, the backup container can take over immediately, thereby ensuring the continuity and stability of key applications. This design can effectively reduce the fault recovery time and better meet the fault-tolerant requirements of key applications.

[0004] The technical solution of the present invention:

[0005] A fast fault-tolerant method for container orchestration scheduling is used to implement fast fault-tolerant for key services based on a container orchestration system. The container orchestration system includes:

[0006] The management center is used for the overall status management of the cluster, including the interface server, scheduler and operation controller. The interface server is responsible for providing access and storage of various resource data; the scheduler is responsible for scheduling Pods; the operation controller is responsible for creating Pod resource objects with running entities according to the deployment strategy; a Pod corresponds to one or more containers;

[0007] The node agent is responsible for reporting the basic information of nodes and applications as well as the load status of nodes and applications. It also calls the container engine through the standard CRI interface to complete image pulling and actual resource allocation and start Pods and containers according to the Pod deployment requirements issued by the management center.

[0008] The container engine is used to actually perform Pod and container lifecycle management functions and allocate network resources to containers through CNI plug-ins;

[0009] Among them: the management center plans the number of normal running Pod copies, the number of spare container copies and resource requirements according to user needs; after creating and starting the Pods required for normal operation, the management center completes resource allocation and pre-creation of container applications on available idle nodes based on the number of spare container copies.

[0010] Furthermore, based on the container orchestration system, a deployment strategy for rapid fault tolerance of key services is provided, and the deployment strategy includes:

[0011] Use dedicated resource types to describe fast fault-tolerant strategies, and use custom resources to integrate fast fault-tolerant resources into the container orchestration system.

[0012] A dedicated controller is designed to handle fast fault tolerance policies. The fast fault tolerance controller is directly integrated into the container orchestration system by interacting with the interface server.

[0013] By custom designing the node agent, when a Pod has a specified label, only the standby Pod is started and the container is created, but the container is not started.

[0014] Further, the method includes using a dedicated resource type to describe a fast fault tolerance strategy, wherein the resource type includes:

[0015] Resource type name and version;

[0016] Resource name;

[0017] Resource specifications, including:

[0018] Pod label selector, used to filter the currently managed Pod objects;

[0019] Pod template, which includes the scheduling and resource requirements of the Pod, as well as the list of containers and images in the Pod;

[0020] The number of replicas of the Pod required;

[0021] The number of replicas of the standby Pod;

[0022] Resource status, including:

[0023] The number of replicas of the Pod that are running normally;

[0024] The number of replicas of the Pod waiting to run.

[0025] Furthermore, the method includes using a dedicated controller to process the fast fault tolerance strategy to generate Pod copies, and selectively activating some Pod copies in combination with the current state. The specific execution logic includes:

[0026] For the next pending fast fault-tolerant resource in the round-robin processing queue:

[0027] If the fast fault-tolerant resource is not deleted, obtain the number of normally running Pod replicas and the number of created but unstarted Pod replicas, and compare them with the required number of Pod replicas and the number of standby Pod replicas. If the number of Pod replicas is insufficient, create a new Pod; if the number of Pod replicas exceeds the limit, delete the redundant Pods; if the number of normally running Pod replicas is less than the required number of Pod replicas, activate the created but unstarted Pod based on the difference;

[0028] If the fast fault tolerance resource has been deleted, determine whether to delete all Pod subresources based on the standard deletion policy of container orchestration;

[0029] When fast fault-tolerant resources change, the resources are added to the queue and wait for cyclic processing;

[0030] When a Pod changes, including addition, update, and deletion, if the owner of the Pod is a fast fault-tolerant resource, the corresponding fast fault-tolerant resource will be added to the queue and wait for cyclic processing.

[0031] Furthermore, the Pod will not start immediately after being created and scheduled to the node, but will start after being activated by the fast fault-tolerant controller; when fault tolerance is required, the fast fault-tolerant controller removes the specified label from the Pod, and the node agent starts the backup container immediately after receiving the update request.

[0032] In this embodiment, after a Pod fails, the backup Pod is immediately notified to start, thereby greatly improving the fault tolerance performance.

[0033] Beneficial effects of the present invention:

[0034] (1) High availability: By pre-configuring backup Pod replicas, the application can be quickly switched to the backup Pod in the event of a failure, ensuring high availability of the application.

[0035] (2) Fast fault recovery: Since the backup Pod is created and initialized in advance, all startup preparations have been completed. When a fault occurs, cross-node fault switching can be achieved almost instantaneously, greatly reducing recovery time.

[0036] (3) Status consistency guarantee: Based on the container orchestration system, by sharing storage resources between Pod replicas, the backup Pod can immediately obtain all the record data of the failed Pod after it is started, so as to quickly restore the operating status and provide normal external services;

[0037] (4) Flexibility: Users can use fast fault-tolerant resources to customize the number of replicas, Pod resources, and scheduling methods according to the actual usage requirements of different applications;

[0038] (5) Non-intrusive design: Fast fault-tolerant technology is integrated with the container orchestration system. Users can use this fault-tolerant strategy to improve application availability without making any modifications or special designs to business applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 The overall system structure diagram of the fast fault-tolerant method for container orchestration and scheduling;

[0041] Figure 2 A schematic diagram for fast fault-tolerant resource description;

[0042] Figure 3 A schematic diagram of the process of fast fault-tolerant controller processing new resources;

[0043] Figure 4 A schematic diagram of the process of fault-tolerant processing for a fast fault-tolerant controller;

[0044] Figure 5 The figure is a comparison diagram of the fast fault tolerance and general fault tolerance processes. DETAILED DESCRIPTION

[0045] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0046] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0047] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0048] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0049] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0050] In one embodiment of the present invention, a fast fault tolerance method for container orchestration scheduling is proposed, which is used to implement fast fault tolerance for key services based on a container orchestration system. The container orchestration system includes:

[0051] The management center is used for the overall status management of the cluster, including the interface server, scheduler and operation controller. The interface server is responsible for providing access and storage of various resource data; the scheduler is responsible for scheduling Pods; the operation controller is responsible for creating Pod resource objects with running entities according to the deployment strategy; a Pod corresponds to one or more containers;

[0052] The node agent is responsible for reporting the basic information of nodes and applications as well as the load status of nodes and applications. It also calls the container engine through the standard CRI interface to complete image pulling and actual resource allocation and start Pods and containers according to the Pod deployment requirements issued by the management center.

[0053] The container engine is used to actually perform Pod and container lifecycle management functions and allocate network resources to containers through CNI plug-ins;

[0054] Among them: the management center plans the number of normal running Pod copies, the number of spare container copies and resource requirements according to user needs; after creating and starting the Pods required for normal operation, the management center completes resource allocation and pre-creation of container applications on available idle nodes based on the number of spare container copies.

[0055] In this embodiment, based on the container orchestration system, a deployment strategy for rapid fault tolerance of key services is provided, and the deployment strategy includes:

[0056] Use dedicated resource types to describe fast fault-tolerant strategies, and use custom resources to integrate fast fault-tolerant resources into the container orchestration system.

[0057] A dedicated controller is designed to handle fast fault tolerance policies. The fast fault tolerance controller is directly integrated into the container orchestration system by interacting with the interface server.

[0058] By custom designing the node agent, when a Pod has a specified label, only the standby Pod is started and the container is created, but the container is not started.

[0059] In this embodiment, the method includes using a dedicated resource type to describe a fast fault tolerance strategy, and the resource type includes:

[0060] Resource type name and version;

[0061] Resource name;

[0062] Resource specifications, including:

[0063] Pod label selector, used to filter the currently managed Pod objects;

[0064] Pod template, which includes the scheduling and resource requirements of the Pod, as well as the list of containers and images in the Pod;

[0065] The number of replicas of the Pod required;

[0066] The number of replicas of the standby Pod;

[0067] Resource status, including:

[0068] The number of replicas of the Pod that are running normally;

[0069] The number of replicas of the Pod waiting to run.

[0070] In this embodiment, the method includes using a dedicated controller to process the fast fault tolerance strategy to generate Pod copies, and selectively activating some Pod copies in combination with the current state. The specific execution logic includes:

[0071] For the next pending fast fault-tolerant resource in the round-robin processing queue:

[0072] If the fast fault-tolerant resource is not deleted, obtain the number of normally running Pod replicas and the number of created but unstarted Pod replicas, and compare them with the required number of Pod replicas and the number of standby Pod replicas. If the number of Pod replicas is insufficient, create a new Pod; if the number of Pod replicas exceeds the limit, delete the excess Pods. If the number of normally running Pod replicas is less than the required number of Pod replicas, activate the created but unstarted Pod based on the difference;

[0073] If the fast fault tolerance resource has been deleted, determine whether to delete all Pod subresources based on the standard deletion policy of container orchestration;

[0074] When fast fault-tolerant resources change, the resources are added to the queue and wait for cyclic processing;

[0075] When a Pod changes, including addition, update, and deletion, if the owner of the Pod is a fast fault-tolerant resource, the corresponding fast fault-tolerant resource will be added to the queue and wait for cyclic processing.

[0076] In this embodiment, the Pod is not started immediately after being created and scheduled to the node, but is started after being activated by the fast fault-tolerant controller; when fault tolerance is required, the fast fault-tolerant controller removes the specified label from the Pod, and the node agent immediately starts the backup container after receiving the update request.

[0077] In this embodiment, after a Pod fails, the backup Pod is immediately notified to start, thereby greatly improving the fault tolerance performance.

[0078] The fast fault-tolerant method for container orchestration scheduling of this embodiment is implemented based on the container orchestration system, and a dedicated fast fault-tolerant controller is designed, including: a management center, a node agent, a container engine, a fast fault-tolerant controller, and a fast fault-tolerant resource description, such as Figure 1 As shown;

[0079] The fast fault-tolerant resource description of this embodiment can be defined in a Json / Yaml format file, and its content and meaning are as follows: Figure 2 As shown;

[0080] When the user creates a fast fault-tolerant resource, the fast fault-tolerant controller processes it, creates Pods through the interface server and node agent, and activates the specified part of the Pod process as follows: Figure 3 As shown;

[0081] When a node fails, the fast fault-tolerant controller processes and activates the backup Pod as follows: Figure 4 As shown;

[0082] When a node fails, general fault tolerance technology needs to go through the process of rebuilding Pod, scheduling Pod, allocating resources, creating Pod and container, and starting container, while fast fault tolerance technology only needs to activate Pod and start container. The process is as follows: Figure 5 shown.

[0083] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A fast fault-tolerant method for container orchestration and scheduling, characterized in that: Used to implement fast fault tolerance for key services based on a container orchestration system, the container orchestration system comprising: The management center is used for the overall status management of the cluster, including the interface server, scheduler and operation controller. The interface server is responsible for providing access and storage of various resource data; the scheduler is responsible for scheduling Pods; the operation controller is responsible for creating Pod resource objects with running entities according to the deployment strategy; a Pod corresponds to one or more containers; The node agent is responsible for reporting the basic information of nodes and applications as well as the load status of nodes and applications. It also calls the container engine through the standard CRI interface to complete image pulling and actual resource allocation and start Pods and containers according to the Pod deployment requirements issued by the management center. The container engine is used to actually perform Pod and container lifecycle management functions and allocate network resources to containers through CNI plug-ins; Among them: the management center plans the number of normal running Pod copies, the number of spare container copies and resource requirements according to user needs; after creating and starting the Pods required for normal operation, the management center completes resource allocation and pre-creation of container applications on available idle nodes based on the number of spare container copies.

2. The fast fault-tolerant method for container orchestration and scheduling according to claim 1, characterized in that: Based on the container orchestration system, a deployment strategy for rapid fault tolerance of key services is provided. The deployment strategy includes: Use dedicated resource types to describe fast fault-tolerant strategies, and use custom resources to integrate fast fault-tolerant resources into the container orchestration system. A dedicated controller is designed to handle fast fault tolerance policies. The fast fault tolerance controller is directly integrated into the container orchestration system by interacting with the interface server. By custom designing the node agent, when a Pod has a specified label, only the standby Pod is started and the container is created, but the container is not started.

3. The fast fault-tolerant method for container orchestration and scheduling according to claim 2 is characterized in that: The method includes describing a fast fault tolerance strategy using a dedicated resource type, wherein the resource type includes: Resource type name and version; Resource name; Resource specifications, including: Pod label selector, used to filter the currently managed Pod objects; Pod template, which includes the scheduling and resource requirements of the Pod, as well as the list of containers and images in the Pod; The number of replicas of the Pod required; The number of replicas of the standby Pod; Resource status, including: The number of replicas of the Pod that are running normally; The number of replicas of the Pod waiting to run.

4. The fast fault-tolerant method for container orchestration and scheduling according to claim 3 is characterized in that: The method includes using a dedicated controller to process the fast fault tolerance strategy to generate Pod copies, and selectively activating some Pod copies in combination with the current state. The specific execution logic includes: For the next pending fast fault-tolerant resource in the round-robin processing queue: If the fast fault-tolerant resource is not deleted, obtain the number of normally running Pod replicas and the number of created but unstarted Pod replicas, and compare them with the required number of Pod replicas and the number of standby Pod replicas. If the number of Pod replicas is insufficient, create a new Pod; if the number of Pod replicas exceeds the limit, delete the excess Pods. If the number of normally running Pod replicas is less than the required number of Pod replicas, activate the created but unstarted Pod based on the difference; If the fast fault tolerance resource has been deleted, determine whether to delete all Pod subresources based on the standard deletion policy of container orchestration; When fast fault-tolerant resources change, the resources are added to the queue and wait for cyclic processing; When a Pod changes, including addition, update, and deletion, if the owner of the Pod is a fast fault-tolerant resource, the corresponding fast fault-tolerant resource will be added to the queue and wait for cyclic processing.

5. The fast fault-tolerant method for container orchestration and scheduling according to claim 4 is characterized in that: After the Pod is created and scheduled to the node, it will not start immediately, but will be started after being activated by the fast fault-tolerant controller; when fault tolerance is required, the fast fault-tolerant controller removes the specified label from the Pod, and the node agent starts the backup container immediately after receiving the update request.

6. The fast fault-tolerant method for container orchestration and scheduling according to claim 5, characterized in that: After a Pod fails, the backup Pod is immediately notified to start, greatly improving fault tolerance.

Images