Application Fault Drill Method and System Based on Kubernetes Cluster

Abstract

The present invention provides an application fault drill method and system based on a kubernetes cluster, relating to the field of cloud native technologies, including: Step S1: Submit a task orchestration taskYml chaos experiment data to the APIserver server. After the TaskController task flow controller watches the event, it immediately processes it and persists the experiment data through etcd; Step S2: The controller ChaosController parses the experiment object, selects a corresponding fault injection method according to the operation type of the experiment object, and manages the life cycle of the experiment; Step S3: According to the fault injection type, the simulated fault injection component Agent executes a predetermined logic. The present invention can actively inject some abnormal fault states in software or hardware aspects into the application under a certain pressure, simulate the fault scenarios that the application may encounter during actual production operation, locate the factors affecting system stability, and improve the resilience of the distributed system.

Description

Technical Field

[0001] The present invention relates to the field of cloud native technologies, and specifically, to an application fault drill method and system based on a kubernetes cluster. Background Art

[0002] In the field of cloud native, the distributed application architectures deployed based on Kubernetes are becoming increasingly complex, and problems such as unpredictability are becoming increasingly prominent. Existing software testing strategies focus on preventing foreseeable system risks and cannot meet the system stability requirements in the Kubernetes environment. Actively detecting application faults has become an effective solution. Summary of the Invention

[0003] Aiming at the defects in the prior art, the present invention provides an application fault drill method and system based on a kubernetes cluster.

[0004] According to an application fault drill method and system based on a kubernetes cluster provided by the present invention, the solution is as follows:

[0005] In the first aspect, an application fault drill method based on a kubernetes cluster is provided. The method includes:

[0006] TaskFlow Controller TaskController: Controls the life cycle of the drill task;

[0007] Controller ChaosController: Controls the life cycle of the experiment. After the task object is decomposed by the TaskFlow Controller TaskController, it will become a unit controlled by the ChaosController;

[0008] Component Agent: Used to inject relevant abnormal interferences into the specified application;

[0009] Explosion Radius Control Component ExplosionController: Used to control the explosion radius of the experiment and select the current object by using the native label selection ability of Kubernetes;

[0010] The implementation process of the method includes:

[0011] Step S1: Submit a task orchestration taskYml chaos experiment data to the APIserver server. After the TaskFlow Controller TaskController watches the event, it processes immediately and persists the experiment data through etcd;

[0012] Step S2: The controller ChaosController parses the experimental object, selects the corresponding fault injection method according to the operation type of the experimental object, and manages the life cycle of the experiment;

[0013] Step S3: According to the fault injection type, the simulated fault injection component Agent executes the predetermined logic.

[0014] Preferably, the step S1 includes:

[0015] Step S1.1: The task flow controller TaskController parses the task object according to the submitted task orchestration taskYml, performs persistent storage on the task object data, and determines the task flow type according to the taskType in the metadata of the task object. If it is a single experiment task flow, step S1.2 is executed; if it is a multi-experiment task flow, step S1.3 is executed;

[0016] Step S1.2: Package the single experiment chaosObject object according to the chaosTemplate of the task object, and then submit it to the APIserver server. Subsequently, the controller ChaosController listens to the Chaos object, executes the specific experiment, and feeds back the experiment status through the Reconcile mechanism;

[0017] Step S1.3: For the execution of multi-experiment tasks, determine whether the task object is executed serially or in parallel according to the taskType;

[0018] Step S1.4: Define the timing task type through the Schedule in the chaosTemplate. If it is a single execution task, the Job method is used to execute it; if it is executed multiple times, the CronJob method is used.

[0019] Preferably, the step S1.3 specifically includes:

[0020] If it is executed serially, the single experiment chaosObject parsed from the task object is queued into the TaskQueue, and then dequeued and executed in sequence. The execution of the Chaos object is still controlled by the controller ChaosController and the result is fed back. After each experiment is executed, the task flow controller TaskController is notified in a callback manner. TaskController caches the number of executed experiments, and then compares it with the total number of this task object to determine the execution status of the task;

[0021] If it is executed in parallel, the method of step S1.2 is used to execute it.

[0022] Preferably, step S2 includes:

[0023] After the controller ChaosController captures the chaosObject object, it parses the main data of the object and determines the injection method according to the value of chaosType;

[0024] For fault injection of non-life-cycle management types, according to the taskTarget and action content, an AgentObject object is combined, and then the node information is confirmed according to the experimental object information, so as to send the experimental data to the specific Agent program; the Agent execution result is monitored through the Reconcile callback, and a recovery instruction is sent to the Agent after the experiment ends;

[0025] For experimental actions related to Pod life-cycle management, fault injection is implemented by calling the Pod-related APIs of the API server. After the injection is completed, the completion status of the experiment is verified by listening to the Pod time, and then ChaosController updates the status of this task, and one task is completed.

[0026] Preferably, step S3 includes:

[0027] Step S3.1: The Agent parses the AgentObject, obtains the experimental data, and parses the experimental data to determine whether the experiment is for the node where the Pod is located or for the Pod itself, and then implements the specific experiment;

[0028] Step S3.2: According to the different experimental types submitted, the Agent executes the corresponding logic;

[0029] Step S3.3: At the end of the duration of the fault injection session, destroy the Agent chaos injection program, exit the namespaces of the application Pod with the injected fault, and then send the experimental result to the controller ChaosController.

[0030] Preferably, step S3.1 further includes: If it is for the Pod itself, enter the namespaces of the Pod through the attachNS() function of the Agent to operate on the experimental object. If it is for the nodes Nodes, the experiment injection is directly completed on the node.

[0031] Preferably, step S3.2 further includes: According to the experimental object and the experimental type type, call the injectFault() function to inject the experiment, and then the callback returns the result. The injection method of the injectFault() function is determined according to type.

[0032] In a second aspect, an application fault rehearsal system based on a kubernetes cluster is provided. The system includes:

[0033] TaskFlow Controller TaskController: Controls the life cycle of rehearsal tasks;

[0034] Controller ChaosController: Controls the life cycle of experiments. After the task object is decomposed by the TaskFlow Controller TaskController, it becomes a unit controlled by ChaosController;

[0035] Component Agent: Used to inject relevant abnormal interferences into the specified application;

[0036] Explosion Radius Control Component ExplosionController: Used to control the explosion radius of the experiment, and uses the native label selection ability of Kubernetes to select the current object;

[0037] The system includes:

[0038] Module M1: Submits a task orchestration taskYml chaos experiment data to the APIserver server. After the TaskFlow Controller TaskController watches the event, it processes immediately and persists the experiment data through etcd;

[0039] Module M2: The Controller ChaosController parses the experiment object, selects the corresponding fault injection method according to the operation type of the experiment object, and manages the life cycle of the experiment;

[0040] Module M3: According to the fault injection type, the simulated fault injection component Agent executes the predetermined logic.

[0041] Preferably, the module M1 includes:

[0042] Module M1.1: The TaskFlow Controller TaskController parses the task object according to the submitted task orchestration taskYml, persists and stores the task object data, and judges the task flow type according to the taskType in the metadata of the task object. If it is a single experiment task flow, execute Module M1.2, and for a multi-experiment task flow, execute Module M1.3;

[0043] Module M1.2: Package the single - experiment chaosObject object according to the chaosTemplate of the task object, and then submit it to the APIserver server. Subsequently, the controller ChaosController monitors the Chaos object, executes the specific experiment, and feedbacks the experiment status through the Reconcile mechanism;

[0044] Module M1.3: Execution of multi - experiment tasks, determine whether the task object is executed serially or in parallel according to the taskType;

[0045] Module M1.4: In Module M1.2 and Module M1.3, the experiment task object type is a scheduled task type. If it is a single - execution task, the Job method is used to execute it. If it is a multiple - execution task, the CronJob method is used;

[0046] The specific content of the said Module M1.3 includes:

[0047] If it is executed serially, the single - experiment chaosObject parsed from the task object is put into the TaskQueue, and then taken out of the queue and executed in turn. The execution of the Chaos object is still controlled by the controller ChaosController and the result is feedback. After each experiment is executed, the task - flow controller TaskController is notified in a callback manner. TaskController caches the number of executed experiments and then compares it with the total number of this task object to determine the execution status of the task;

[0048] If it is executed in parallel, the method of Module M1.2 is used for execution.

[0049] Preferably, the said Module M2 includes:

[0050] After the controller ChaosController captures the chaosObject object, it parses the main data of the object and determines the injection method according to the value of chaosType;

[0051] For non - life - cycle - management - type fault injection, according to the taskTarget and action content, combine the AgentObject object, and then confirm the node information according to the experiment target object information, so as to send the experimental data to the specific Agent program; Monitor the Agent execution result through the Reconcile callback, and send a recovery instruction to the Agent after the experiment ends;

[0052] For the experimental actions related to Pod lifecycle management, fault injection is implemented by calling the Pod-related APIs of the API server. After the injection is completed, the completion status of the experiment is verified by listening to the Pod time, and then the ChaosController updates the status of this task, and one task is completed.

[0053] The module M3 includes:

[0054] Module M3.1: The Agent parses the AgentObject, obtains the experimental data, and parses the experimental data to determine whether the experiment is for the node where the Pod is located or for the Pod itself, and then implements the specific experiment.

[0055] Module M3.2: According to the different experimental types submitted, the Agent executes the corresponding logic.

[0056] Module M3.3: At the end of the duration of the fault injection session, destroy the program for Agent chaos injection, exit the namespaces of the application Pod injected with faults, and then send the experimental results to the controller ChaosController.

[0057] The module M3.1 further includes: If it is for the Pod itself, enter the namespaces of the pod through the attachNS() function of the Agent to operate on the experimental object. If it is for the nodes Nodes, directly complete the experimental injection on the node.

[0058] The module M3.2 further includes: According to the experimental object and the experimental type type, call the injectFault() function to inject the experiment, and then call back the returned result. The injection method of the injectFault() function is determined according to type.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] 1. The present invention is mainly a fault drill system designed for the Kubernetes cluster. It tries to reuse the Kubernetes infrastructure to implement the function of application fault drill. For example: use the operator plug-in mechanism of Kubernetes to define certain fault types and implement the corresponding controllers; classify and manage the fault injection experiments, and design a separate CRD for each type of fault, and the ChaosController controller is used to implement the full life cycle management of the fault injection. By using the Kubeconfig configuration file method to manage the information of multiple clusters, the ability to inject faults into multi-cluster applications is realized. In this way, the deployment and use difficulty are reduced.

[0061] 2. The present invention provides a proactive fault injection solution implemented based on Kubernetes. According to the characteristics of actual fault injection and the applicability of the method, different implementation components are designed for different types of fault injection. The data transfer between components and controllers, etc. utilizes the capabilities of Kubernetes itself;

[0062] 3. The present invention implements a multi-dimensional blast radius controller, which controls the target experimental object by means of custom tags and network policies, designs different directions of fault injection from the dimension of the influence range after injecting faults, and limits the influence range of faults;

[0063] 4. The present invention provides a capability of work task flow and experiment scheduling to achieve an orderly injection simulation of multiple faults and non-preemptive resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, objectives, and advantages of the present invention will become more apparent:

[0065] Figure 1 is the overall framework schematic diagram;

[0066] Figure 2 is the schematic diagram of the data structure package;

[0067] Figure 3 is the schematic diagram of TaskQueue;

[0068] Figure 4 is the schematic diagram of Schedule;

[0069] Figure 5 is the schematic diagram of the Agent executing the fault injection function;

[0070] Figure 6 is the schematic diagram of the processing fault logic of ExplosionController. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0071] The following specifically describes the present invention in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several changes and improvements can still be made. These all fall within the protection scope of the present invention.

[0072] The embodiment of the present invention provides an application fault drill method based on a kubernetes cluster, referring to Figure 1 as shown, wherein, the main components and function designs involved are as follows:

[0073] 1) TaskFlow Controller TaskController: The TaskController controls the life cycle of the drill task. Through the extended API resource object TaskFlow, it orchestrates and controls a chaos drill task. The task flow contains one or more experiments, which are executed in parallel or serially. It will execute the fault experiments in the chaosList according to different methods, and determine whether the subsequent experiments depend on the previous ones to continue execution by setting feedback.

[0074] TaskController includes three core designs, namely the TaskFlow data structure, the TaskQueue task queue, and the Schedule scheduling. Single-experiment tasks and tasks that do not involve timed scheduling do not use the latter two modules.

[0075] Refer to Figure 2 As shown, TaskFlow defines the task flow structure, which is designed using the standard Kubernetes API object. The data structure contains metadata and the ChaosTemplate experiment object data, mainly including: task identification, experiment list, scheduling type, target object, experiment actions, etc.

[0076] Refer to Figure 3 As shown, TaskQueue is a first-in, first-out queue for the task flow. For multiple experiments executed serially, tasks enter the queue first and leave the queue after completion. Tasks for single experiments do not enter the queue. The experiments in the chaosList for multi-experiment tasks executed serially enter the TaskQueue queue as a whole. The termination of an experiment is based on the task unit, that is, when the task ends, the experiment will automatically end, and then all experiments leave the queue.

[0077] Refer to Figure 4 As shown, the Schedule defined by chaosTemplate includes two types of experiment object scheduling, namely periodic scheduling and aperiodic scheduling, which are implemented based on CornJob and Job, and use the Reconcile mechanism to implement result monitoring.

[0078] 2) Controller ChaosController: A chaos experiment controller designed in the operator mode of Kubernetes. This controller is the only component that can control the experiment lifecycle. After the task object is decomposed by the TaskController, it becomes a unit controlled by the ChaosController. The controller will use different methods to complete the experiment according to the specific type of the experiment. In this design, three methods are implemented in total. One method is to call the Pod-oriented API, such as delete, edit, patch, etc. The second method is to inject a Sidecar container at the time of Pod creation by designing ChaosWebhook, and inject faults through this container. The third method is to encapsulate the fault injection program as an Agent and deploy it to the cluster through the DaemonSet method. Using the characteristics of the container itself, through the method of sharing container namespaces, the ability of non-intrusive fault injection into the application is realized. After the fault injection is completed, the controller perceives the experiment result through the Reconcile feedback mechanism and controls the complete lifecycle of an experiment.

[0079] 3) Component Agent: The Agent is a simulated fault injection component used to inject relevant abnormal interferences into the specified application, including faults at the software and hardware levels, and is deployed on each node of the cluster through the daemonset method of Kubernetes. The Agent encapsulates the logic of fault handling, and the update logic requires resetting the Agent's Pod. The Agent provides services by exposing ports externally, shielding the underlying implementation details, and enabling seamless updates of the fault injection implementation method. It mainly provides 4 methods, which implement fault injection, result acquisition, experiment termination, and entry into namespaces. Among them, the method for entering namespaces: attachNS(nsId,*), the experiment injection method: injectFault(targetId,type,***), getResult(targetId,*), stopFault(targetId,*).

[0080] The implementation method of the Agent component in this solution mainly includes: encapsulating the call interfaces for operating on Linux container Cgroups and Namespaces, the Java method-level bytecode injection tool Byteman, and script tools for fault injection at the Pod level, application method level, and host level, such as host pressure and TC network traffic control. The Agent executes the fault injection function with reference to Figure 5 as shown.

[0081] 4) Explosion Controller: The ExplosionController component controls the blast radius of the experiment. It leverages Kubernetes' native label-based selection capabilities to select current objects and provides experimental object location capabilities based on multiple granularities, such as Pod, Label, and Service. To simulate abnormal network traffic failure scenarios, this component supports north-south traffic control, simulating network failures on the request side and the server side. It combines labels and Network Policies to identify experimental orchestration objects to achieve precise control of the blast radius.

[0082] The north-south dimension controls the object of fault injection. Taking A requesting service B as an example, the southbound dimension injects the fault into B, affecting A's request response through B's fault. The northbound dimension injects the fault into A, simulating the occurrence of the fault by delaying A's request.

[0083] The ExplosionController uses object selectors to locate the target object. The granularity ranges from small to large: Pod, Label, and Service. Service selects the injected object from the perspective of the service, Pod from the perspective of a single instance, and Label from the perspective of metadata. When selecting the target injection object from the perspective of a Service, the ExplosionController parses the Service's backend Endpoint object, queries for valid backend Pod objects, and then uses the Pod selector to inject faults into the specific instance.

[0084] ExplosionController's fault handling logic reference Figure 6 shown.

[0085] Specifically, the implementation process of this method includes:

[0086] Step S1: Submit a task orchestration taskYml chaos experiment data to the API server. The TaskController controller watches the event and processes it immediately, and etcd will persist the data.

[0087] Specifically, step S1 includes:

[0088] Step S1.1: The TaskController in the task flow controller parses the task object from the submitted taskYml, persistently stores the data of this object, and determines the task flow type according to the taskType in the metadata of the task object. If it is a single experiment task flow, step S1.2 is executed; if it is a multi-experiment task flow, step S1.3 is executed.

[0089] Step S1.2: Package the single experiment chaosObject object according to the chaosTemplate of the task object, and then submit it to the APIserver server. Subsequently, the controller ChaosController listens to this Chaos object, executes the specific experiment, and then feeds back the experiment status through the Reconcile mechanism.

[0090] Execution of multi-experiment tasks: Determine whether the task object is executed serially or in parallel according to taskType. If it is executed serially, the single experiment chaosObject parsed from the Task is queued into the TaskQueue, and then dequeued and executed in sequence. The execution of chaos is still controlled by the ChaosController and the results are fed back. After each experiment is executed, the TaskController is notified in a callback manner. The TaskController caches the number of executed experiments and then compares it with the total number of this Task to determine the execution status of the task. If it is executed in parallel, the method in step S1.2 is used for execution.

[0091] Step S1.4: The Schedule in the chaosTemplate of the task object in step S1.2 and step S1.3 defines the type of scheduled task. If it is a single execution task, it is executed in the Job way; if it is executed multiple times, the CronJob way is used. Both ways rely on the capabilities of Kubernetes itself.

[0092] Step S2: The controller ChaosController parses the experiment object, and then selects the corresponding fault injection method according to the object operation type to manage the life cycle of the experiment.

[0093] Specifically, step S2 includes: After the controller ChaosController captures the chaosObject object, it parses the main data of the object, including chaosType, chaosTarget, action, and targetSelector. These 4 objects contain the dependency information of the fault injection type. According to the value of chaosType, the injection method is determined, which are Pod lifecycle management type, daemonset shared namesapces type injection, and sidecar injection. For non-lifecycle management type fault injection, then according to the taskTarget and action content, the AgentObject object is combined, and then the node information where it is located is confirmed according to the experimental target information, so as to send the experimental data to the specific Agent program; the Agent execution result is monitored through the Reconcile callback, and a recovery instruction is sent to the Agent after the experiment ends.

[0094] For the experimental actions of the Pod lifecycle management type, fault injection is implemented by calling the POD-related APIs of the API server. After the injection is completed, the completion status of the experiment is verified by listening to the Pod time, and then ChaosController updates the status of this task, and one task is completed.

[0095] Step S3: According to the fault injection type, the simulated fault injection component Agent executes the predetermined logic.

[0096] Specifically, step S3 includes:

[0097] Step S3.1: Agent parses AgentObject, obtains the experimental data, and then parses the experimental data to determine whether the experiment is for the node where the Pod is located or for the Pod itself, and then implements the specific experiment. If it is for the Pod itself, the attachNS() function of Agent is used to enter the namespaces of the pod to operate on the experimental object. If it is for the nodes, the experiment injection is directly completed on the node.

[0098] Step S3.2: According to different submitted experiment types, the Agent executes corresponding logic. First, based on the experimental object and the experiment type type, the injectFault() function is called to inject the experiment. Then the callback returns the result. The injection method of injectFault() is determined according to type. For example, to add latency to a specified method of a Java program, the Byteman component based on the fault injection technology is used to perform bytecode-level latency script injection to achieve latency at the method level. To perform resource occupancy type fault injection, encapsulated programs such as the stress pressure tool are started to implement. During the experiment, stopFault() can be accepted simultaneously to terminate an experiment, and getResult() is used to obtain the experiment result.

[0099] Step S3.3: At the end of the duration of the fault injection session, destroy the programs injected chaotically by the Agent, exit the namespaces of the application Pods injected with faults, and then send the experiment results to the controller ChaosController.

[0100] The present invention also provides an application fault drill system based on a kubernetes cluster. Those skilled in the art can understand the application fault drill method provided by the present invention as the specific implementation manner of the application fault drill system based on a kubernetes cluster, that is, the application fault drill system based on a kubernetes cluster can be implemented by executing the step process of the application fault drill method based on a kubernetes cluster.

[0101] The embodiment of the present invention provides an application fault drill method and system based on a kubernetes cluster, which can actively inject some software or hardware abnormal fault states into an application under a certain pressure, simulate the fault scenarios that the application may encounter during actual production operation, locate the factors affecting system stability, and improve the resilience of the distributed system.

[0102] The fault drill system of the present invention is mainly designed for the Kubernetes cluster. It tries to reuse the Kubernetes infrastructure to implement the function of application fault drill. For example, the operator plug-in mechanism of Kubernetes is used to define certain fault types and implement the corresponding controllers; the fault injection experiments are classified and managed, and each type of fault has an independent CRD designed, and the ChaosController controller is used to implement the full life cycle management of fault injection. By adopting the Kubeconfig configuration file method to manage multiple cluster information, the fault injection ability for multi-cluster applications is realized. In the above way, the deployment and usage difficulties are reduced.

[0103] The present invention provides a proactive fault injection solution implemented based on Kubernetes. According to the characteristics of actual fault injection and the applicability of the method, different implementation components are designed for different types of fault injection. The data transfer between components and controllers, etc. relies on the capabilities of Kubernetes itself. The present invention implements a multi-dimensional blast radius controller, which controls the target experimental object through custom labels and network policies, and designs different directions of fault injection from the dimension of the influence range after injecting faults to limit the influence range of the faults. The present invention provides an ability of a work task flow and an experiment schedule to achieve an orderly injection simulation of multiple faults and non-preemptive resources.

[0104] Those skilled in the art know that in addition to implementing the system and its various devices, modules, and units provided by the present invention in the form of pure computer-readable program code, the method steps can be logically programmed to enable the system and its various devices, modules, and units provided by the present invention to be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers, etc. to achieve the same functions. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be regarded as the structures within the hardware component; the devices, modules, and units for implementing various functions can also be regarded as both software modules for implementing the method and the structures within the hardware component.

[0105] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. Without conflict, the embodiments of the present application and the features in the embodiments can be combined arbitrarily with each other.

Claims

1. An application fault drill method based on a kubernetes cluster, characterized in that, Including: Task flow controller TaskController: Controls the life cycle of the drill task; Controller ChaosController: Controls the life cycle of the experiment. After the task object is decomposed by the task flow controller TaskController, it will become the unit controlled by ChaosController; Component Agent: Used to inject relevant abnormal interferences into the specified application; Explosion radius control component ExplosionController: Used to control the explosion radius of the experiment, and uses the native label selection ability of Kubernetes to select the current object; The implementation process of the method includes: Step S1: Submit a task orchestration taskYml chaos experiment data to the APIserver server. After the TaskController task flow controller watches the event, it immediately processes it and persists the experiment data through etcd; Step S2: The controller ChaosController parses the experiment object, selects the corresponding fault injection method according to the operation type of the experiment object, and manages the life cycle of the experiment; Step S3: According to the fault injection type, the simulated fault injection component Agent executes the predetermined logic; The said Step S1 includes: Step S1.1: The task flow controller TaskController parses the task object according to the submitted task orchestration taskYml, persists the task object data, and judges the task flow type according to the taskType in the metadata of the task object. If it is a single experiment task flow, execute Step S1.2; if it is a multi-experiment task flow, execute Step S1.3; Step S1.2: Package the single experiment chaosObject object according to the chaosTemplate of the task object, and then submit it to the APIserver server. Subsequently, the controller ChaosController listens to the Chaos object, executes the specific experiment, and feeds back the experiment status through the Reconcile mechanism; Step S1.3: For the execution of multi-experiment tasks, judge whether the task object is executed serially or in parallel according to the taskType; Step S1.4: Define the timing task type through the Schedule in the chaosTemplate. If it is a single execution task, use the Job method to execute it; if it is a multiple execution, use the CronJob method; The said Step S1.3 specifically includes: If it is executed serially, the single experiment chaosObject parsed from the task object is enqueued into the TaskQueue, and then dequeued and executed in sequence. The execution of the Chaos object is still controlled by the controller ChaosController and the result is fed back. After each experiment is executed, the task flow controller TaskController is notified in a callback manner. TaskController caches the number of executed experiments and then compares it with the total number of this task object to determine the execution status of the task; If it is executed in parallel, the method of step S1.2 is used for execution.

2. The application fault drill method based on the kubernetes cluster according to claim 1, wherein The said step S2 includes: After the controller ChaosController captures the chaosObject object, it parses the main data of the object and decides the injection method according to the value of chaosType; For the fault injection of non-life-cycle management type, according to the taskTarget and action content, the AgentObject object is combined, and then the node information is confirmed according to the experiment target object information, so as to send the experimental data to the specific Agent program; The Agent execution result is monitored through the Reconcile callback, and a recovery instruction is sent to the Agent after the experiment ends; For the experimental actions of Pod life-cycle management type, the fault injection is realized by calling the Pod-related APIs of the API server. After the injection is completed, the completion status of the experiment is verified by listening to the Pod time, and then ChaosController updates the status of this task, and one task is completed.

3. The application fault drill method based on the kubernetes cluster according to claim 1, characterized in that The said step S3 includes: Step S3.1: The Agent parses the AgentObject, obtains the experimental data, and parses the experimental data to judge whether the experiment is for the node where the Pod is located or for the Pod itself, and then implements the specific experiment; Step S3.2: According to the different experimental types submitted, the Agent executes the corresponding logic; Step S3.3: At the end of the duration of the fault injection link, the Agent chaos injection program is destroyed, the namespaces of the application Pod injected with the fault are exited, and then the experimental result is sent to the controller ChaosController.

4. The application fault drill method based on the kubernetes cluster according to claim 3, wherein, The said step S3.1 further includes: If it is for the Pod itself, the attachNS() function of the Agent is used to enter the namespaces of the Pod to operate on the experimental object. If it is for the nodes Nodes, the experimental injection is directly completed on the node.

5. The application fault drill method based on the kubernetes cluster according to claim 4, characterized in that The said step S3.2 further includes: According to the experimental object and the experimental type type, the injectFault() function is called to inject the experiment, and then the result is returned by callback. The injection method of the injectFault() function is determined according to type.

6. An application fault drill system based on a kubernetes cluster, characterized in that, It includes: Task flow controller TaskController: Controls the life cycle of the drill task; Controller ChaosController: Controls the lifecycle of the experiment. After the task object is decomposed by the task flow controller TaskController, it becomes the unit controlled by ChaosController; Component Agent: Used to inject relevant abnormal interferences into the specified application; Explosion Radius Control Component ExplosionController: Used to control the explosion radius of the experiment, and uses the native label selection ability of Kubernetes to select the current object; The system includes: Module M1: Submits a task orchestration taskYml chaos experiment data to the APIserver server. After the TaskController task flow controller watches the event, it processes immediately and persists the experiment data through etcd; Module M2: The controller ChaosController parses the experiment object, selects the corresponding fault injection method according to the operation type of the experiment object, and manages the lifecycle of the experiment; Module M3: According to the fault injection type, the simulated fault injection component Agent executes the predetermined logic; The module M1 includes: Module M1.1: The task flow controller TaskController parses the task object according to the submitted task orchestration taskYml, persists the task object data, and judges the task flow type according to the taskType in the metadata of the task object. If it is a single experiment task flow, execute module M1.2, and for a multi-experiment task flow, execute module M1.3; Module M1.2: Encapsulates a single experiment chaosObject object according to the chaosTemplate of the task object, and then submits it to the APIserver server. Subsequently, the controller ChaosController listens to the Chaos object, executes the specific experiment, and feedbacks the experiment status through the Reconcile mechanism; Module M1.3: Execution of multi-experiment tasks, judging whether the task object is executed serially or in parallel according to taskType; Module M1.4: Defines the timing task type through the Schedule in the chaosTemplate. If it is a single execution task, it is executed in the Job mode. If it is executed multiple times, the CronJob mode is used; The module M1.3 specifically includes: If it is executed serially, the single experiment chaosObject parsed from the task object is queued into the TaskQueue, and then dequeued and executed in turn. The execution of the Chaos object is still controlled by the controller ChaosController and the result is feedback. After each experiment is executed, the task flow controller TaskController is notified in a callback manner. TaskController caches the number of executed experiments, and then compares it with the total number of this task object to determine the execution status of the task; If it is executed in parallel, it is executed in the way of module M1.

2.

7. The application fault drill system based on the kubernetes cluster according to claim 6, characterized in that, The module M2 includes: After the controller ChaosController captures the chaosObject object, it parses the main data of the object and decides the injection method according to the value of chaosType; For non-life-cycle management type fault injection, according to the taskTarget and action content, combine the AgentObject object, then confirm the node information according to the experimental target object information, and thus send the experimental data to the specific Agent program; monitor the Agent execution result through the Reconcile callback, and send a recovery instruction to the Agent after the experiment ends; For the experimental actions related to Pod life cycle management, implement fault injection by calling the Pod-related APIs of the API server. After the injection is completed, verify the completion status of the experiment by listening to the Pod time, and then ChaosController updates the status of this task, and one task is completed; The module M3 includes: Module M3.1: The Agent parses the AgentObject, obtains the experimental data, and parses the experimental data to determine whether the experiment is for the node where the Pod is located or for the Pod itself, and then implements the specific experiment; Module M3.2: According to the different experimental types submitted, the Agent executes the corresponding logic; Module M3.3: At the end of the duration of the fault injection link, destroy the Agent chaos injection program, exit the namespaces of the application Pod injected with the fault, and then send the experimental result to the controller ChaosController; The module M3.1 further includes: If it is for the Pod itself, enter the namespaces of the Pod through the attachNS() function of the Agent to operate on the experimental object. If it is for the nodes Nodes, directly complete the experimental injection on the node; The module M3.2 further includes: According to the experimental object and the experimental type type, call the injectFault() function to inject the experiment, and then call back the return result. The injection method of the injectFault() function is determined according to type.

Images