A kubemark-based pod fault injection method and system in a kubernetes cluster

Abstract

The application provides a kubemark-based Pod fault injection method and system in a kubernetes cluster, which firstly provides a fault collection tool for collecting Pod fault information in a real cluster; meanwhile, virtual nodes with the same specifications as the nodes of the real cluster are created in a simulation cluster by using a modified Kubemark, and a CRI-Proxy (container runtime interface proxy) component is deployed to perform fault injection and container lifecycle management; a fault scene is simulated through a mode of "fault recording-fault broadcasting", and the stability and recovery capability of the system are tested by simulating Pod and container lifecycle abnormalities through fault injection, so that valuable performance test data and system optimization guidance are provided for the real cluster, resources are saved, test efficiency is improved, problems can be reproduced, potential defects can be found, and various risks can be avoided in advance.

Description

Technical Field

[0001] This invention relates to the technical field of cloud computing and container orchestration, and in particular to a method and system for Pod fault injection based on kubemark in a Kubernetes cluster. Background Technology

[0002] Kubernetes, an open-source container orchestration system, is widely used for the automatic deployment, elastic scaling, and management of containerized applications. It provides powerful resource management and scheduling capabilities, ensuring efficient utilization of underlying computing and storage resources. With the rapid development of AI technology, utilizing distributed computing resources for large-scale model training has become increasingly important. Kubernetes' capabilities meet the needs of large AI models in terms of resource requirements, deployment, and maintenance, making it the standard foundation for model training and inference environments.

[0003] While Kubernetes provides high availability and fault tolerance, in real-world production environments, systems can still face various unexpected situations. Developers and operations personnel need to construct various abnormal scenarios to test the system's stability and recovery capabilities. However, directly introducing faults into the production environment can be risky. Therefore, fault injection has become a crucial technique for testing the performance of systems and applications under various fault scenarios. However, traditional fault injection methods often operate directly on real nodes and Pods. In large-scale cluster environments, this makes it difficult to precisely control the scope and conditions of fault injection and can lead to resource waste.

[0004] Therefore, it is of great significance to develop a safe, efficient and easy-to-configure method to test the behavior of Kubernetes components under fault scenarios and provide valuable performance simulation data for real cluster environments. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention aims to provide a Pod fault injection method and system based on Kubemark. By utilizing and modifying Kubemark, a publicly available performance testing tool for Kubernetes, it simulates a large-scale Kubernetes cluster with logical specifications consistent with a real cluster, significantly reducing resource consumption and precisely controlling the scope and conditions of fault injection. It solves the problems of high cost of large-scale chaotic testing, difficulty in reproducing and troubleshooting complex issues, and low repeatability of testing methods in existing approaches. It uses a "fault recording-fault broadcasting" simulation mode to simulate and evaluate the performance of a real Kubernetes cluster.

[0006] The technical solution adopted in this invention is as follows:

[0007] A first aspect of the present invention: a Pod fault injection method based on kubemark in a Kubernetes cluster, the method comprising the following steps:

[0008] Step 1: In a real Kubernetes cluster, use a fault collection tool to collect real Pod fault information and generate a fault policy configuration file;

[0009] Step 2: Create virtual nodes in the simulation cluster using the modified Kubemark. Set the same resource configuration for each node as for the real nodes, and simulate a cluster of the same specifications as the real cluster.

[0010] Step 3: Deploy the CRI-Proxy component in the simulation cluster to receive Kubemark requests and perform fault injection on specified Pods.

[0011] Specifically, the fault collection tool in step one collects real Pod fault information based on the Prometheus cluster monitoring system, parses the collected Pod fault data, extracts the specific attributes of the fault, and then generates a configuration file that can be recognized when fault injection occurs in the simulated cluster.

[0012] Specifically, the actual Pod failure information includes CRI call interface errors and Pod status abnormalities; the specific attributes include Pod name, namespace, container name, failure type, failure time, and failure error message.

[0013] Specifically, the modified Kubemark in step two consists of: first, adding a client for the CRI-Proxy component to interact with CRI-Proxy; and second, rewriting all CRI implementations to forward the original CRI interface calls to CRI-Proxy, so that Kubemark is only responsible for receiving its response results.

[0014] Furthermore, the virtual node created in step two consists of two parts: Hollow Kubelet and HollowProxy. These two parts are used to simulate the Kubelet and Kube Proxy on the real node, respectively. They do not need to actually run containers or perform workloads, but instead mimic the behavior of the real node by simulating API calls and status reports.

[0015] Furthermore, step three is implemented through the following sub-steps:

[0016] (3.1) Create a remote runtime instance, start the gRPC server, and handle client requests;

[0017] (3.2) Construct a simulation service for remote container runtime. Specifically, simulate real runtime service and image service by implementing the CRI runtime service interface Fake_Runtime_Service and image service interface Fake_Image_Service respectively. At the same time, embed a fault handling mechanism in the implementation logic of the above interfaces so that it can simulate the corresponding fault scenario based on the information passed in the subsequent fault injection process.

[0018] (3.3) Load the fault policy configuration file and broadcast the fault information to the specified fault triggering stage according to the configuration file; define different types of faults and events for different stages of container and Pod lifecycle, and call the interface at the specified time to inject specific fault simulations. That is, each fault object will define the subject, time and fault type information of the fault occurrence. Multiple faults can be injected in the lifecycle of a Pod and the fault execution can be triggered at the specified time.

[0019] Furthermore, the lifecycle of the Pod and its internal containers in step (3.3) follows a strict temporal logic: the creation of the Pod is the starting point of the entire lifecycle; subsequently, its internal containers enter the creation phase in sequence; when the container finishes running, the container will terminate according to its own exit logic; finally, the Pod is deleted, marking the end of the entire lifecycle; based on this lifecycle, the state of the Pod in each stage is simulated, thereby realizing the control of the creation, running or deletion of the Pod outside of Kubernetes logic; similarly, fault injection capabilities can also be added based on this lifecycle. After the Pod is created, the container creation process is executed, at which time the preset creation fault is triggered; after the fault handling logic is executed, the container is officially created; when the container's running cycle ends, it terminates according to its own state; finally, the Pod is deleted, and the entire lifecycle is declared over.

[0020] Each fault object needs to define the fault occurrence time, duration, fault injection target, fault injection timing, fault completion status, and fault error message. The fault injection target specifies the Pod name, the namespace to which the Pod belongs, and the container name. The fault injection timing specifies the specific CRI interface, that is, specifies the logical steps to execute the fault.

[0021] A second aspect of the invention: The invention also discloses a Pod fault injection system based on kubemark in a Kubernetes cluster, the system comprising the following modules:

[0022] Fault policy configuration module: In a real Kubernetes cluster, a fault collection tool is used to collect real Pod fault information and generate a fault policy configuration file;

[0023] Simulation Cluster Construction Module: Virtual nodes are created in the simulation cluster using the modified Kubemark. Each node is configured with the same resources as the real nodes, simulating a cluster of the same specifications as the real cluster.

[0024] Fault Injection Module: Deploy the CRI-Proxy component in the simulation cluster to receive Kubemark requests and inject faults into specified Pods.

[0025] A third aspect of the present invention discloses an electronic device comprising a memory and a processor, wherein the memory is coupled to the processor; the memory is used to store program data, and the processor is used to execute the program data to implement the Pod fault injection method based on kubemark in a Kubernetes cluster.

[0026] A fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the Pod fault injection method and system based on kubemark in a Kubernetes cluster.

[0027] The beneficial effects of this invention are as follows:

[0028] This invention, by injecting faults on virtual nodes, enables extreme load and chaos testing without directly impacting the real production environment and running services. This saves resources compared to conducting fault testing on real large-scale clusters, avoids various risks, and improves the security of fault injection. Secondly, this invention accelerates the troubleshooting and remediation of faults. When encountering problems that are difficult to reproduce, it can replay abnormal events in the simulation environment, speeding up problem reproduction and localization, and rapidly debugging and verifying solutions. Finally, it helps technical teams identify potential problems and optimize system configurations in advance. In summary, this invention, by simulating real-world fault conditions in a simulation environment, helps identify potential risks and improves the stability, recoverability, and robustness of the system in the face of faults. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram illustrating the overall process of Pod fault injection based on Kubemark, as provided in another embodiment of the present invention.

[0031] Figure 2 This is a schematic diagram of the overall device flow of the present invention;

[0032] Figure 3 This is a system architecture diagram for Pod fault injection based on Kubemark, provided as an embodiment of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0035] This invention relates to a Pod fault injection method based on kubemark in a Kubernetes cluster, the specific steps of which include:

[0036] (1) In a real Kubernetes cluster, Pod failure information is monitored and obtained based on the Prometheus system, and then the Pod failure information is parsed and a failure policy configuration file is generated by the failure collection tool provided by this invention.

[0037] (2) Use the modified Kubemark to create virtual nodes in the simulation cluster. Set the same resource configuration as the real nodes for each node and simulate a cluster with the same specifications as the real cluster.

[0038] (3) Deploy the CRI-Proxy (Container Runtime Interface Proxy) component in the simulation cluster to receive Kubemark requests, perform fault injection and container lifecycle management, and simulate the fault scenarios of the real cluster.

[0039] Example 1: As Figure 1 As shown, the entire fault injection process includes the following:

[0040] Users submit Pod resource definitions via the Kubernetes API. The Kube-api-server receives and verifies the request, persists the Pod's metadata to the etcd database, and the Kubernetes scheduler schedules the Pod to a Hollow Node (virtual node) created by Kubemark.

[0041] On the virtual node, the Hollow Kubelet establishes a gRPC communication connection with the CRI_Server module of the CRI-Proxy component through the CRI-Proxy client. The Hollow Kubelet initiates a container creation request, which is forwarded to the subsequent fault injection logic module through CRI-Proxy.

[0042] CRI-Proxy reads and parses the pre-generated fault policy configuration file, broadcasts the fault information to the corresponding Pod lifecycle stage, executes the specific fault injection logic, and synchronously updates the container's state information after fault injection.

[0043] Example 2: As Figure 2 and Figure 3 The diagram shown illustrates the system flowchart and overall architecture of this invention for Pod fault injection based on Kubemark in a Kubernetes cluster. The system includes a fault policy configuration module, a simulation cluster construction module, and a fault injection module. It primarily involves two clusters: a real cluster and a simulation cluster. The real cluster is used for fault collection, and the simulation cluster is used for fault injection. Details are as follows:

[0044] The fault policy configuration module, in a real Kubernetes cluster, uses a fault collection tool to collect real Pod fault information and generate a fault policy configuration file. Specifically: First, for a real cluster, this invention provides a fault collection tool. This tool is responsible for capturing Pod fault information in the cluster, mainly including CRI call interface errors and Pod status anomalies. The tool exports all anomaly information based on the Prometheus cluster monitoring system, parses the collected data, and extracts the specific attributes of the fault, including Pod name, namespace, container name, fault type, fault time, and fault error message. This data is then used to generate a fault policy configuration file that can be recognized by the fault injection module deployed in the simulation cluster. Fault collection enables a "fault recording" function, recording and backing up fault information from the real cluster, which can then be repeatedly imported into the simulation cluster for broadcast playback.

[0045] The simulation cluster construction module uses a modified Kubemark to create virtual nodes within the simulation cluster. Each node is configured with the same resources as a real node, simulating a cluster of the same specifications as the real cluster; specifically as follows:

[0046] Secondly, this invention modifies Kubemark in two ways: first, by adding a CRI-Proxy client to the Hollow Kubelet for interaction with CRI-Proxy; and second, by rewriting all Container Runtime Interface (CRI) implementations, forwarding existing CRI interface calls to CRI-Proxy, which then executes the corresponding processing logic, while Kubemark only receives the responses. For the simulated cluster, the modified Kubemark is used to create virtual nodes. Each node can be configured with the same resources as real cluster nodes to simulate a cluster of the same specifications. The virtual nodes created by Kubemark are called HollowNodes, and they consist of two main parts: Hollow Kubelet and Hollow Proxy, which simulate the Kubelet and KubeProxy on the real nodes, respectively. However, they do not actually run containers or execute workloads; instead, they mimic the behavior of real nodes by simulating API calls and status reports.

[0047] The fault injection module deploys a CRI-Proxy component in the simulation cluster to receive requests from Kubemark and perform fault injection on specified Pods. Specifically:

[0048] This invention deploys the key component CRI-Proxy, which is a modification of the above-mentioned simulation cluster construction module, in the simulation cluster. This component consists of three modules: CRI_Server, Runtime_Service, and fault injection.

[0049] CRI_Server will start the gRPC server to handle client requests.

[0050] The `Runtime_Service` module creates a remote container runtime by implementing the CRI API. The CRI API is a standardized interface proposed by Kubernetes to decouple container runtimes (such as Docker and containerd) from Kubernetes core components (such as Kubelet). The CRI API mainly consists of two parts: `Runtime Service` and `Image Service`. The former defines a set of RPC methods for managing the container lifecycle, such as creating, starting, stopping, and deleting containers; the latter focuses on image management, providing functions such as pulling, inspecting, and deleting images. Correspondingly, this invention implements the CRI runtime service interface and image service interface in the `Runtime_Service` module through `Fake_Runtime_Service` and `Fake_Image_Service`, respectively, to simulate real runtime and image services. This method does not actually perform container creation or image pulling operations; it only simulates the results through API calls. Simultaneously, corresponding fault handling logic is added to these interface implementations, simulating corresponding faults based on information received by the fault injection module, such as specifying that a certain interface fails and returning a specific error message.

[0051] The fault injection module loads the fault policy configuration file and broadcasts fault information to specified stages according to the configuration file. It ensures the synchronization and updating of fault information on all virtual nodes in the simulation environment and provides accurate timing and context information for subsequent fault injection. Furthermore, this module manages and injects faults. Different types of faults and events can be defined for different stages of the container and Pod lifecycle. Each fault object defines the subject, time, and type of the fault. Multiple faults can be injected during the lifecycle of a Pod, triggering fault execution at specified times.

[0052] Specifically, the lifecycle of a Pod and its containers can be summarized as follows: the Pod is created, and then its internal containers sequentially enter the creation phase; when the container runtime ends, each container terminates sequentially according to its own exit logic; finally, the Pod is deleted, and the entire lifecycle ends. Based on this lifecycle, the state of a Pod at each stage can be simulated, thereby enabling control over the creation, operation, or deletion of Pods outside of Kubernetes logic. Also based on this lifecycle, fault injection capabilities can be added. After the Pod is created, the container creation process is executed, at which point a preset creation fault is triggered; after the fault handling logic is executed, the container is officially created; when the container runtime ends, it terminates according to its own state; finally, the Pod is deleted, and the entire lifecycle ends.

[0053] Each fault object needs to define the fault occurrence time, duration, fault injection target, fault injection timing, fault completion status, and fault error message. The fault injection target specifies the Pod name, the namespace to which the Pod belongs, and the container name; the fault injection timing specifies the specific CRI interface, i.e., at which step the fault logic is executed; the fault error message simulates the error feedback format and content when a real CRI interface call fails.

[0054] In summary, this invention can use limited resources to simulate a real cluster environment using Kubemark, simulate fault scenarios through a "fault recording-fault broadcasting" mode, and simulate abnormal situations in the Pod lifecycle through fault injection, thereby testing the stability and recovery capabilities of the system. It provides valuable performance test data and system optimization guidance for real clusters, saves resources, improves testing efficiency, and helps reproduce problems, discover potential defects, and help avoid various risks in advance.

[0055] Regarding the system in the above embodiments, the specific ways in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0056] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0057] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements the Pod fault injection method based on the Kubernetes cluster performance testing tool Kubemark as described in the above embodiments.

[0058] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0059] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0060] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A Pod fault injection method based on kubemark in a Kubernetes cluster, characterized in that, The method includes the following steps: Step 1: In a real Kubernetes cluster, use a fault collection tool to collect real Pod fault information and generate a fault policy configuration file; Step 2: Create virtual nodes in the simulation cluster using the modified Kubemark. Set the same resource configuration as the real nodes for each node and simulate a cluster of the same specifications as the real cluster. Step 3: Deploy the CRI-Proxy component in the simulation cluster to receive Kubemark requests and perform fault injection on specified Pods; this is implemented through the following sub-steps: (3.1) Create a remote runtime instance, start the gRPC server, and handle client requests; (3.2) Construct a simulation service for remote container runtime. Specifically, simulate real runtime service and image service by implementing the CRI runtime service interface Fake_Runtime_Service and image service interface Fake_Image_Service respectively. At the same time, embed a fault handling mechanism in the implementation logic of the above interfaces so that it can simulate the corresponding fault scenario based on the information passed in the subsequent fault injection process. (3.3) Load the fault policy configuration file and broadcast the fault information to the specified fault triggering stage according to the configuration file; define different types of faults and events for different stages of container and Pod lifecycle, and call the interface at the specified time to inject specific fault simulations. That is, each fault object will define the subject, time and fault type information of the fault occurrence. Multiple faults can be injected in the lifecycle of a Pod and the fault execution can be triggered at the specified time.

2. The Pod fault injection method according to claim 1, characterized in that, The fault acquisition tool in step one collects real Pod fault information based on the Prometheus cluster monitoring system, parses the collected Pod fault data, extracts the specific attributes of the fault, and then generates a configuration file that can be recognized when injecting faults in the simulated cluster.

3. The Pod fault injection method according to claim 2, characterized in that, The actual Pod failure information includes CRI call interface errors and Pod status abnormalities; the specific attributes include Pod name, namespace, container name, failure type, failure time, and failure error message.

4. The Pod fault injection method according to claim 1, characterized in that, The modified Kubemark in step two is specifically as follows: First, a client for the CRI-Proxy component is added to interact with CRI-Proxy; second, all CRI implementations are rewritten, and the original CRI interface calls are forwarded to CRI-Proxy, so that Kubemark is only responsible for receiving its response results.

5. The Pod fault injection method according to claim 1, characterized in that, The virtual node created in step two consists of two parts: Hollow Kubelet and Hollow Proxy. These two parts are used to simulate the Kubelet and Kube Proxy on the real node, respectively. They do not need to actually run containers or perform workloads, but instead mimic the behavior of the real node by simulating API calls and status reports.

6. The Pod fault injection method according to claim 1, characterized in that, The lifecycle of the Pod and its internal containers in step (3.3) follows a strict temporal logic: the creation of the Pod is the starting point of the entire lifecycle; subsequently, its internal containers enter the creation phase in sequence; when the container finishes running, the container will terminate according to its own exit logic; finally, the Pod is deleted, marking the end of the entire lifecycle; based on this lifecycle, the state of the Pod in each stage is simulated, thereby realizing the control of the creation, running or deletion of the Pod outside of Kubernetes logic; similarly, fault injection capability can also be added based on this lifecycle. After the Pod is created, the container creation process is executed, at which time the preset creation fault is triggered; Once the fault handling logic is executed, the container is officially created; when the container's runtime cycle ends, it terminates according to its own state; finally, the Pod is deleted, and the entire lifecycle is declared over. Each fault object needs to define the fault occurrence time, duration, fault injection target, fault injection timing, fault completion status, and fault error message. The fault injection target specifies the Pod name, the namespace to which the Pod belongs, and the container name. The fault injection timing specifies the specific CRI interface, that is, specifies the logical steps to execute for the fault.

7. A Pod fault injection system based on kubemark in a Kubernetes cluster, characterized in that, The system includes the following modules: Fault policy configuration module: In a real Kubernetes cluster, a fault collection tool is used to collect real Pod fault information and generate a fault policy configuration file; Simulation Cluster Construction Module: Virtual nodes are created in the simulation cluster using the modified Kubemark. Each node is configured with the same resources as the real nodes, simulating a cluster of the same specifications as the real cluster. Fault Injection Module: Deploy the CRI-Proxy component in the simulation cluster to receive requests from Kubemark and inject faults into specified Pods; this is implemented through the following steps: Create a remote runtime instance, start the gRPC server, and handle client requests; A simulation service for remote container runtime is built. Specifically, the runtime service interface Fake_Runtime_Service and the image service interface Fake_Image_Service of CRI are implemented to simulate real runtime services and image services, respectively. At the same time, a fault handling mechanism is embedded in the implementation logic of the above interfaces so that it can simulate corresponding fault scenarios based on the information passed in the subsequent fault injection process. Load the fault policy configuration file and broadcast the fault information to the specified fault triggering stage according to the configuration file; define different types of faults and events for different stages of container and Pod lifecycle, and call the interface at the specified time to inject specific fault simulations. That is, each fault object will define the subject, time and fault type information of the fault occurrence. Multiple faults can be injected in the lifecycle of a Pod and the fault execution can be triggered at the specified time.

8. An electronic device comprising a memory and a processor, wherein, The memory is coupled to the processor; characterized in that the memory is used to store program data, and the processor is used to execute the program data to implement the Pod fault injection method based on kubemark in a Kubernetes cluster as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the Pod fault injection method and system based on kubemark in a Kubernetes cluster as described in any one of claims 1-6.

Images