A chaos engineering platform with fault injection verification capabilities

Abstract

The present invention discloses a chaos engineering platform including a fault injection verification function. After initiating an experimental request, the task execution program converts the fault task into a fault execution request according to the execution order of the fault tasks in the workflow and issues it. The API service issues a fault injection request corresponding to the fault model to the operator or chaos agent based on the injection object of the fault model. During the execution of the fault injection request, the indicator monitoring task corresponding to the fault injection request is sent to the operator or chaos agent once every preset time interval. The chaos agent obtains the corresponding PID of the injection object according to the type of the injection object in the fault injection request, installs an agent to the PID process, executes the corresponding fault model through the agent, and executes the indicator monitoring task during the execution of the fault model to obtain the fault injection status through the API service, thereby realizing the mixed fault orchestration of multiple injection objects and further realizing the verification of the execution status of the fault task during the fault injection process.

Description

Technical Field

[0001] The present invention relates to the field of chaos engineering, and in particular to a chaos engineering platform including a fault injection verification function. Background Art

[0002] Alibaba is one of the earliest companies in China to explore chaos engineering. ChaosBlade is Alibaba's open-source chaos engineering project launched in 2019, designed to help enterprises solve high availability issues in cloud-native processes through chaos engineering. ChaosBlade provides a rich set of fault injection capabilities, such as for resource failures like memory, CPU, disk, and network, as well as code-level failures like those in Java and C++. ChaosMesh is an excellent open-source chaos engineering platform that supports a wide range of fault types, but its drawbacks include incomplete experimental objects and different experiments for Kubernetes objects and host objects, making it difficult to implement mixed fault orchestration for both types of objects. While existing chaos engineering technologies provide fault orchestration, they lack the ability to orchestrate both standalone and cloud-native applications, making it difficult to cope with the current heterogeneous internet environment. Furthermore, their use cases are limited, such as restricting application deployment methods, supporting only fault injection, and resulting in a complex usage process.

[0003] At the same time, failure observability and verification are highly valued features for chaos engineering users. Users can intuitively observe the fault injection process and, based on key injection time points, observe changes in system metrics, making it easier to infer system bottlenecks and areas for improving system resilience. Steady-state observation is a crucial component of chaos engineering experiments, but currently, open source chaos platforms generally lack this feature, making it less user-friendly for users without a foundation in chaos engineering. Summary of the Invention

[0004] To achieve mixed fault orchestration for multiple injection targets and verify the execution status of faulty tasks and real-time observation of corresponding system indicators during the fault injection process, the present invention proposes a chaos engineering platform with fault injection verification function. The chaos engineering platform includes:

[0005] The management platform is used to set up experimental models, fault tasks, and indicator monitoring tasks corresponding to the fault tasks, and to initiate experimental requests. The experimental model includes the settings of fault combinations and fault combination orchestration methods. The fault combinations include one or more fault groups, and a fault group includes one or more fault tasks. The fault tasks are used to set injection objects and fault parameters. The indicator monitoring tasks are used to set the system indicators to be monitored and the acquisition commands corresponding to the system indicators.

[0006] The scheduling platform is used to receive experimental requests and includes a scheduler and a task execution program. The scheduler is used to generate a workflow based on the arrangement of fault combinations, and the workflow is the execution order of fault tasks. The task execution program is used to convert the fault tasks into fault execution requests based on the execution order of the fault tasks in the workflow and issue them after receiving the experimental request.

[0007] A fault injection platform is used to set an injection object model, which is used to set the injection object and set the selection method of the object to be injected; the fault injection platform includes an API service, an operator, and a Chaos Agent; the API service is used to receive a fault execution request issued by a task execution program and convert it into a corresponding fault model, and based on the injection object of the fault model, it sends a fault injection request corresponding to the fault model to the operator or Chaos Agent, and during the execution of the fault injection request, it sends an indicator monitoring task corresponding to the fault injection request to the operator or Chaos Agent once every preset time interval; the operator is used to receive the fault injection request and indicator monitoring task and forward them to the Chaos Agent; the Chaos The Agent is used to receive fault injection requests and indicator monitoring tasks issued by the API service and the Operator, and when the injection object in the fault injection request is the object to be injected, it obtains its corresponding PID according to the type of the injection object in the fault injection request, installs the Agent to the PID process, executes the corresponding fault model through the Agent, and executes the received indicator monitoring task in the process of executing the fault model to obtain the system indicator monitoring value, and returns the system indicator monitoring value to the API service; the API service is also used to obtain the fault injection status by comparing the system indicator monitoring value with the fault indicator setting value corresponding to the fault model;

[0008] The UI display module is used to display the system indicator monitoring value, fault indicator setting value and fault injection status corresponding to each fault task in real time.

[0009] Furthermore, the arrangement method is: the fault tasks within the fault group are executed serially, and the fault groups are executed serially, or the fault tasks within the fault group are executed serially and the fault groups are executed in parallel.

[0010] Furthermore, the selection method includes randomly selecting the object to be injected from the set injection objects, selecting all the set injection objects as the objects to be injected, and setting a preset percentage of the set injection objects as the objects to be injected.

[0011] Furthermore, the types of the injection objects include stand-alone machines and Kubernetes clusters; the stand-alone machines include stand-alone processes and stand-alone containers; the Kubernetes cluster includes pods, nodes, and pod containers. The Kubernetes cluster consists of a group of nodes running containerized applications. The Pod includes one or more pod containers, and the Pod runs on a node.

[0012] Furthermore, in the fault injection platform, when the injection object of the fault model is a pod, a node or a pod container, the API service sends a fault injection request and an indicator monitoring task corresponding to the fault model to the Operator.

[0013] Furthermore, in the fault injection platform, when the type of the injection object is a Kubernetes cluster, the PID is obtained by entering the corresponding namespace, and the namespace is used to store the PID of the program process, and the program process is a pod process, a node process or a pod container process.

[0014] Furthermore, the failure types of the faulty tasks include Java failure, DNS failure, Pod failure, disk failure and network failure.

[0015] Furthermore, the fault parameters include fault ID, fault type and fault action field; the fault action field includes: kill pod process, kill pod container process, IP resolution error, method delay, custom exception, memory overflow, CPU full load, IO limit, read and write speed limit, network rejection fault, network packet loss fault, network delay fault and network packet disorder fault.

[0016] Compared with the prior art, the present invention has at least the following beneficial effects:

[0017] (1) The present invention sets an experimental model and sets the fault combination and the arrangement method of the fault combination in the experimental model. After initiating the experimental request, the task execution program converts the fault task into a fault execution request according to the execution order of the fault task in the workflow and sends it. The API service receives the fault execution request issued by the task execution program and converts it into a corresponding fault model, and sends a fault injection request corresponding to the fault model to the Operator or Chaos Agent based on the injection object of the fault model. During the execution of the fault injection request, the indicator monitoring task corresponding to the fault injection request is sent to the Operator or Chaos Agent once every preset time interval. The Chaos The Agent receives the fault injection request and indicator monitoring task issued by the API service and the Operator, obtains the corresponding PID according to the type of the injection object in the fault injection request, and installs the Agent to the PID process, executes the corresponding fault model through the Agent, and executes the received indicator monitoring task in the process of executing the fault model to obtain the system indicator monitoring value, and returns the system indicator monitoring value to the API service to obtain the fault injection status through the API service, that is, to verify whether the injection of the fault model is effective; the present invention realizes fault mixing by setting the scheduling order of the fault tasks in the experimental model, and starts the experiment based on the execution order of the mixed fault tasks. In addition, the present invention obtains the PID under different paths by the type of the injection object, thereby realizing the mixed fault injection of different injection objects on the basis of fault mixing. On this basis, the present invention executes the indicator monitoring task multiple times during the execution of the fault injection request, so that the present invention further realizes the verification of the execution status of the fault task and the real-time observation of its corresponding system indicators during the fault injection process on the basis of realizing the mixed fault scheduling of multiple injection objects;

[0018] (2) The present invention sets the serial execution of fault tasks within a fault group and serial execution between fault groups through the experimental model, or serial execution of fault tasks within a fault group and parallel execution between fault groups, thereby achieving mixed encoding of multiple faults, that is, achieving mixed injection of multiple faults;

[0019] (3) The present invention supports mixed fault injection for single-machine processes, single-machine containers, and Kubernetes clusters, pods, nodes, and pod containers. This solves the problem in the prior art of being unable to orchestrate mixed faults for Kubernetes objects and host objects, and improves the ability to cope with the current heterogeneous Internet environment.

[0020] (4) The present invention sets an Operator in the fault injection platform. When the injection object of the fault task is a pod, node, or pod container, the Operator receives the fault injection request issued by the API service, thereby making the injection of fault tasks compatible with both a single machine and a Kubernetes cluster, thereby realizing mixed fault injection of multiple injection objects;

[0021] (5) The present invention sets the selection method of the object to be injected through the injection object model, thereby improving the flexibility of fault injection;

[0022] (6) The present invention executes the received indicator monitoring task in the process of executing the fault model to obtain the system indicator monitoring value, and returns the system indicator monitoring value to the API service; the API service obtains the fault injection status by comparing the system indicator monitoring value with the fault indicator setting value corresponding to the fault model, thereby realizing the verification of the execution status of the fault task;

[0023] (7) The present invention realizes steady-state observation of the system indicators corresponding to each fault task by displaying the system indicator monitoring value, fault indicator setting value and fault injection status corresponding to each fault task in real time through the UI display module. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is an architecture diagram of a chaos engineering platform that supports mixed orchestration of multiple injection objects.

[0025] Figure 2 Verify command execution graph for DNS failure;

[0026] Figure 3 It is a diagram of fault combination arrangement in series and parallel mode;

[0027] Figure 4 It is a fault combination diagram in DAG (directed acyclic) mode;

[0028] Figure 5 Delay fault model for java methods;

[0029] Figure 6 A model for limiting the number of read and write times for disk failures;

[0030] Figure 7 Model read and write speed limits for disk failures;

[0031] Figure 8 Verify command execution graph for disk write speed limit failure. DETAILED DESCRIPTION

[0032] The following are specific embodiments of the present invention and the accompanying drawings to further describe the technical solutions of the present invention, but the present invention is not limited to these embodiments.

[0033] Example 1

[0034] In order to realize mixed fault arrangement of multiple injection objects and verify the execution status of fault tasks and real-time observation of corresponding system indicators during the fault injection process, such as Figure 1 As shown, the present invention proposes a chaos engineering platform that includes a fault injection verification function. Chaos engineering can artificially introduce abnormal states of software or hardware into the system to create fault scenarios, thereby determining optimization strategies based on the system's behavior under various pressures. The application of chaos engineering can verify and evaluate the system's ability to resist disturbances and maintain normal operation (stability), identify unknown hidden dangers in advance and repair them, and thus ensure that the system can better resist out-of-control conditions in the production environment and improve overall stability. In order to specifically address the uncertainty of distributed systems, chaos engineering can be regarded as performing a series of experiments to discover the weaknesses of the system;

[0035] The chaos engineering platform includes:

[0036] The management platform is used to set up experimental models, fault tasks, and indicator monitoring tasks corresponding to the fault tasks, and to initiate experimental requests. The experimental model includes the settings of fault combinations and fault combination orchestration methods. The fault combinations include one or more fault groups, and a fault group includes one or more fault tasks. The fault tasks are used to set injection objects and fault parameters. The indicator monitoring tasks are used to set the system indicators to be monitored (including system energy consumption indicators and fault task indicators) and the acquisition commands corresponding to the system indicators.

[0037] The arrangement method is: the fault tasks within the fault group are executed serially, and the fault groups are executed serially, or the fault tasks within the fault group are executed serially and the fault groups are executed in parallel.

[0038] This embodiment Figure 3 The figure shows the orchestration relationship between the fault tasks within a fault group (fault 1, fault 2, fault 3, fault 4) and the orchestration relationship between fault groups (Group 1, Group 2, ..., Group N).

[0039] Figure 4 Fault orchestration is performed in a DAG (directed acyclic graph) format. Each fault task (fault 1, fault 2, fault 3, fault 4, fault 5, fault 6) has zero or more predecessor tasks, and execution (scheduling) is performed in a topologically sorted manner.

[0040] The fault parameters include a fault ID, a fault type, and a fault action field; the fault action field includes: kill pod process (pod-kill), kill pod container process (pod-container-kill), IP resolution error (dns-fault), method delay (java-delay), custom exception (java-exception), memory overflow (java-mem), CPU full load (java-cpu), IO limit (disk-iops), read and write speed limit (disk-bps), network rejection fault (network-reject), network packet loss fault (network-loss), network delay fault (network-delay), and network packet disorder fault (network-reorder).

[0041] The scheduling platform is used to receive experimental requests and includes a scheduler and a task execution program. The scheduler is used to generate a workflow based on the arrangement of fault combinations, and the workflow is the execution order of fault tasks. The task execution program is used to convert the fault tasks into fault execution requests based on the execution order of the fault tasks in the workflow and issue them after receiving the experimental request.

[0042] It should be noted that the scheduler is also used to record the running status of each faulty task in the workflow (running, running completed, running error, etc.). These status information also include a completion time, which limits the execution time of each faulty task. The scheduler and the task execution program interact through asynchronous calls and responses. The task execution program is responsible for the execution of tasks in the experimental request, converts the faulty task into the corresponding faulty execution request and sends it, and sends the running status of the faulty task to the scheduler. The scheduling platform also includes a supervisory program for monitoring and changing the status of each faulty task. It will regularly poll the status of each faulty task. If the running status detected is a timeout or a failure, the faulty task will be restored or other remedial measures will be performed.

[0043] In this embodiment, an experiment may include one or more fault groups. Multiple fault groups can be executed in parallel or serially. A fault group can also contain fault tasks of multiple fault types. The scheduler in this embodiment is mainly used to limit the order of execution of fault tasks.

[0044] What needs to be explained is:

[0045] The experiment in this embodiment refers to a process in which a user sets a fault combination and a fault combination arrangement method to test system stability, and the scheduling platform schedules and executes the fault injection.

[0046] A fault group represents a set of fault tasks, each of which is executed in serial fashion.

[0047] Choreography refers to the order and method (selection method) in which the fault tasks are injected during the experiment.

[0048] A fault injection platform is used to set an injection object model, wherein the injection object model is used to set the injection object and set a selection method for the object to be injected; the selection method includes randomly selecting the object to be injected from the set injection objects, selecting all the set injection objects as the objects to be injected, and setting a preset percentage of the set injection objects as the objects to be injected; the fault injection platform includes an API service, an Operator, and a Chaos Agent; the API service is used to receive a fault execution request issued by a task execution program and convert it into a corresponding fault model, and based on the injection object of the fault model, send a fault injection request corresponding to the fault model to the Operator or Chaos Agent, and during the execution of the fault injection request, send an indicator monitoring task corresponding to the fault injection request to the Operator or Chaos Agent once every preset time interval (since the fault task in an experiment is one-time, and indicator monitoring is a continuous process, the indicator monitoring task will be repeatedly executed at a certain interval); the Operator is used to receive the fault injection request and the indicator monitoring task and forward them to the Chaos Agent; the Chaos The Agent is used to receive fault injection requests and indicator monitoring tasks issued by the API service and the Operator, and when the injection object in the fault injection request is the object to be injected, it obtains its corresponding PID according to the type of the injection object in the fault injection request, installs the Agent to the PID process, executes the corresponding fault model through the Agent, and executes the received indicator monitoring task in the process of executing the fault model to obtain the system indicator monitoring value, and returns the system indicator monitoring value to the API service; the API service is also used to obtain the fault injection status by comparing the system indicator monitoring value with the fault indicator setting value corresponding to the fault model;

[0049] It should be noted that the system indicator monitoring value includes the system energy consumption monitoring indicator value (such as CPU usage, memory usage, etc.) and the fault task indicator monitoring value. This embodiment specifically obtains the fault injection status through the fault task indicator monitoring value and the fault indicator setting value corresponding to the fault model.

[0050] In addition, since steady-state observation is an important part of chaos engineering experiments, steady-state needs to be judged based on the system's energy consumption monitoring indicator values. Steady-state indicators (i.e., system energy consumption indicator values before the fault model is executed or when the system is in normal operation) need to be collected and compared with the system energy consumption monitoring indicator values under chaos engineering experiments, and the stable data range of the system energy consumption monitoring indicator needs to be determined. Steady-state indicators and system energy consumption monitoring indicators correspond to different usage scenarios. By monitoring system energy consumption indicators, computer resources can be reasonably configured based on the system energy consumption monitoring indicator values. Steady-state determination has the following two aspects:

[0051] 1) The system energy consumption monitoring indicator value needs to be evaluated within a preset time range (to determine whether it is within the stable data range. If so, it means that the system is still in a steady state during the fault injection). Otherwise, a sudden change in a short period of time cannot explain the problem;

[0052] 2) Ensure the real-time performance of the system energy consumption monitoring index value. The steady-state monitoring of the present invention includes configuring the system indicators in the index monitoring task on the management platform, generally including the indicator type, URL, parameters, and steady-state time range, data range and error. The API service automatically pulls the indicators and memorizes the calculations to determine whether the system is in a steady state.

[0053] This example uses DNS failure verification and disk failure verification as examples to illustrate:

[0054] 1. DNS failure verification

[0055] DNS fault verification mainly verifies the domain name and the resolved IP (i.e. the fault task indicator to be monitored). This embodiment uses the ping command for testing. If the ping command is used on the domain name (i.e. the acquisition command corresponding to the fault task indicator), the IP address corresponding to the domain name will be output. It is determined by judging whether the resolved IP address is the same as the set IP (i.e. the fault indicator setting value corresponding to the fault model). If they are the same, it means that the fault injection is effective (the fault injection status is effective). The return value is 0, indicating that the corresponding domain name is not connected, otherwise it returns 1 (the fault injection status is failed). Figure 2 As shown, assuming the injected domain name is "www.baidu.com" and the corresponding IP is set to "1.2.3.4", before injection Figure 2 As shown in the upper part, after injection Figure 2 Shown in the lower part.

[0056] 2. Disk failure verification

[0057] Disk failure verification uses the iostat command (i.e., the acquisition command). Iostat can easily test the iops and read / write rates of a certain partition of the system. To simulate disk operations, we use the dd command. dd can easily convert and copy files. The dd command is "dd if= / dev / zero of= / testfile bs=1024Kcount=2000oflag=direct". Test the writing of a 200M file and use iostat monitoring. Here, the write speed limit is tested. "bps_write_limit" (i.e., the fault task indicator to be monitored) is set to 2097152 (the fault indicator setting value corresponding to the fault model), i.e., 2M / s. Before injecting the fault, Figure 8 As shown in the lower part, after injecting the write speed limit Figure 8 As shown in the upper part, it can be seen that the disk write speed is limited to 2M / s.

[0058] It should be noted that, in addition to the four selection methods of randomly selecting the object to be injected from the set injection objects, selecting all the set injection objects as the objects to be injected, setting a preset percentage of the set injection objects as the objects to be injected (i.e., randomly selecting a preset percentage of the set injection objects as the objects to be injected), and specifying a specific Pod, Pod container, Node, host or host process as the object to be injected, this embodiment also includes selection through the injection object selector:

[0059] In addition to being able to accurately specify Pods, Pod containers, Nodes, hosts, and host processes, this embodiment also supports injection in the form of specified selectors, and currently supports label selectors and service selectors. A label is a key-value pair attached to a pod, and the label is used to specify the identification attributes of the injection object. Labels can be attached to pods when they are created, and can be added and modified at any time. Each pod will have a set of labels, and each label is stored as a key-value pair. Users can specify a set of labels on the management platform. As long as the pod matches one of the labels, it can be used as an object to be injected. Through the label selector, you only need to specify the label to select a group of pods or hosts that meet the conditions as the objects to be injected.

[0060] The injection target types include standalone machines and Kubernetes clusters. Standalone machines include standalone processes and standalone containers. Kubernetes clusters include pods, nodes, and pod containers. A Kubernetes cluster consists of a group of nodes running containerized applications. A pod includes one or more pod containers, and the pod runs on a node. Each node can be a physical machine or a virtual machine.

[0061] In the fault injection platform, when the injection object of the fault model is a pod, node or pod container, the API service sends a fault injection request and indicator monitoring task corresponding to the fault model to the Operator.

[0062] In the fault injection platform, when the type of the injection object is a Kubernetes cluster, the PID is obtained by entering the corresponding namespace, where the namespace is used to store the PID of the program process, which is a pod process, a node process, or a pod container process.

[0063] The failure types of the faulty tasks include Java failure, DNS failure, Pod failure, disk failure and network failure.

[0064] The fault models corresponding to each fault type are described in detail below:

[0065] 1. Java failures mainly include:

[0066] (1) Class method delay: when calling a class method, it returns after a specified delay, and the action field is delay;

[0067] (2) Customize the method return value. When the method returns, it returns a specified value. The action field is java-return.

[0068] (3) Throw a custom exception. Throw a custom exception in the specified method, and the action field is java-exception;

[0069] (4) The Java process CPU is fully loaded, and the action field is java-cpu;

[0070] (5) Memory overflow, including stack overflow and heap overflow, the action field is java-stack and java-heap;

[0071] This example takes class method delay as an example to show how Figure 5 The Java method delay fault model shown in the figure, where kind indicates the fault type, the action field is delay, class indicates the package class name (specific package and class), method indicates the class method name, which must be a method in the class, delay indicates the method delay time, pid indicates the Java process ID, and duration indicates the duration of the fault.

[0072] 2. DNS failure

[0073] DNS (Domain Name System) is primarily responsible for resolving host names into IP addresses, completing the conversion from domain names to host-identifiable IP addresses. A DNS fault can resolve a set domain name into a set IP address (fault indicator setting value). The DNS fault model primarily includes the fields kind, domains, ip, and action. Kind indicates the fault type (DNSChaos), domains indicates a list of domain names, allowing for the specification of multiple domain names and supporting wildcards for domain names, ip indicates resolving the specified domain name into an IP address, and action is dns-fault.

[0074] 3. Pod failure

[0075] A pod is the smallest unit of Kubernetes resource scheduling, consisting of one or more containers. You can kill a selected pod or its containers using the "pod-kill" or "pod-container-kill" action. Pod failures require no parameters other than "action" and "kind" (failure type). These parameters are used in conjunction with the "target" field, which specifies the injection target.

[0076] 4. Disk failure

[0077] There are two main types of disk failures: one is I / O limit, which limits the number of I / Os per second for a specified process. You can specify read mode limit and write mode limit separately. 2) The other is read and write speed limit, which limits the read or write rate of a specified process. It also supports read limit, write limit, or both read and write limit.

[0078] Take the read and write times limit as an example, Figure 6 The figure shows the read and write limit model for disk failures. The action is "disk-iops", the iops_read_limit field is the read limit per second, and the iops_write_limit is the write limit per second. When both parameters are 0, the read and write limit is canceled.

[0079] Take the read and write speed limits as an example. Figure 7 The figure shows the read and write speed limit model for disk failures. The action field is "disk-bps", the bps_read_limit field is the read speed limit per second, and the bps_write_limit field is the write speed limit per second. When both parameters are 0, the read and write speed limits are canceled.

[0080] 5. Network failure

[0081] The network is a very important part of the real world, and network failures are very likely to occur. By simulating network failures, we can simulate the stability of the system in the face of poor network environments or even network attacks, thereby improving the system's confidence in facing network failures. This embodiment mainly supports four types of network failures, namely:

[0082] (1) Network rejection failure: requests are rejected at a certain ratio, and the action field is network-reject;

[0083] (2) Network packet loss failure: network data packets are discarded at a certain ratio, and the action field is network-loss.

[0084] (3) Network delay failure: delay the packet at a certain ratio, and the action field is network-delay.

[0085] (4) Network packet out-of-order failure: packets are processed out of order with a specified delay, and the action field is network-reorder.

[0086] This embodiment takes the network rejection fault as an example for detailed description. The network rejection fault model corresponding to the network rejection fault includes the following fields: kind, action, protocol, nic, ips, probability, and duration.

[0087] Kind is the fault type (the value is NetworkChaos), the action field is network-reject, the protocol field can be tcp, udp, or all, and all represents both protocols. nic represents the network card name, which defaults to eth0. ips represents the matching IP address (the value is ["1.1.1.1"]). You can use ip, ip:port, or cidr+port. Currently, only IPv4 is supported. Probability represents the rejection probability (the value is 30). Duration represents the fault duration (the value is 40s).

[0088] The UI display module is used to display the system indicator monitoring value, fault indicator setting value and fault injection status corresponding to each fault task in real time.

[0089] The present invention sets an experimental model and sets the fault combination and the arrangement mode of the fault combination in the experimental model. After initiating the experimental request, the task execution program converts the fault task into a fault execution request according to the execution order of the fault task in the workflow and sends it. The API service receives the fault execution request issued by the task execution program and converts it into a corresponding fault model. Based on the injection object of the fault model, it sends a fault injection request corresponding to the fault model to the Operator or Chaos Agent. During the execution of the fault injection request, the indicator monitoring task corresponding to the fault injection request is sent to the Operator or Chaos Agent once every preset time interval. The Chaos The Agent receives the fault injection request and indicator monitoring task issued by the API service and the Operator, obtains the corresponding PID according to the type of the injection object in the fault injection request, and installs the Agent to the PID process. The corresponding fault model is executed by the Agent, and the received indicator monitoring task is executed during the execution of the fault model to obtain the system indicator monitoring value, and the system indicator monitoring value is returned to the API service to obtain the fault injection status through the API service, that is, to verify whether the injection of the fault model is effective; the present invention realizes fault mixing by setting the scheduling order of the fault tasks in the experimental model, and starts the experiment based on the execution order of the mixed fault tasks. In addition, the present invention obtains the PID under different paths according to the type of the injection object, thereby realizing fault mixed injection of different injection objects on the basis of fault mixing. On this basis, the present invention executes the indicator monitoring task multiple times during the execution of the fault injection request. Therefore, the present invention further realizes the verification of the execution status of the fault task and the real-time observation of its corresponding system indicators during the fault injection process on the basis of realizing mixed fault scheduling of multiple injection objects.

[0090] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0091] In addition, in the present invention, descriptions such as "first," "second," and "one" are for descriptive purposes only and should not be understood to indicate or imply their relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include at least one of such features. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0092] In the present invention, unless otherwise specified or limited, the terms "connection" and "fixation" should be understood in a broad sense. For example, "fixation" can mean fixed connection, detachable connection, or integration; mechanical connection or electrical connection; direct connection or indirect connection through an intermediate medium; internal communication between two elements or interaction between two elements, unless otherwise specified. Those skilled in the art will be able to understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0093] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in this field can implement it. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

Claims

1. A chaos engineering platform with fault injection verification function, characterized in that: The chaos engineering platform includes: The management platform is used to set up experimental models, fault tasks, and indicator monitoring tasks corresponding to the fault tasks, and to initiate experimental requests. The experimental model includes the settings of fault combinations and fault combination orchestration methods. The fault combinations include one or more fault groups, and a fault group includes one or more fault tasks. The fault tasks are used to set injection objects and fault parameters. The indicator monitoring tasks are used to set the system indicators to be monitored and the acquisition commands corresponding to the system indicators. The scheduling platform is used to receive experimental requests and includes a scheduler and a task execution program. The scheduler is used to generate a workflow based on the arrangement of fault combinations, and the workflow includes the execution order of fault tasks. The task execution program is used to convert the fault tasks into fault execution requests according to the execution order of the fault tasks in the workflow and issue them after receiving the experimental request. A fault injection platform is used to set an injection object model, which is used to set the injection object and set the selection method of the object to be injected; the fault injection platform includes an API service, an operator, and a Chaos Agent; the API service is used to receive a fault execution request issued by a task execution program and convert it into a corresponding fault model, and based on the injection object of the fault model, it sends a fault injection request corresponding to the fault model to the operator or Chaos Agent, and during the execution of the fault injection request, it sends an indicator monitoring task corresponding to the fault injection request to the operator or Chaos Agent once every preset time interval; the operator is used to receive the fault injection request and indicator monitoring task and forward them to the Chaos Agent; the Chaos The Agent is used to receive fault injection requests and indicator monitoring tasks issued by the API service and the Operator, and when the injection object in the fault injection request is the object to be injected, it obtains its corresponding PID according to the type of the injection object in the fault injection request, installs the Agent to the process corresponding to the PID, executes the corresponding fault model through the Agent, and executes the received indicator monitoring task in the process of executing the fault model to obtain the system indicator monitoring value, and returns the system indicator monitoring value to the API service; the API service is also used to obtain the fault injection status by comparing the system indicator monitoring value with the fault indicator setting value corresponding to the fault model; The UI display module is used to display the system indicator monitoring value, fault indicator setting value and fault injection status corresponding to each fault task in real time.

2. A chaos engineering platform with fault injection verification function according to claim 1, characterized in that: The arrangement method is: the fault tasks within the fault group are executed serially, and the fault groups are executed serially, or the fault tasks within the fault group are executed serially and the fault groups are executed in parallel.

3. A chaos engineering platform with fault injection verification function according to claim 2, characterized in that: The selection method includes randomly selecting the object to be injected from the set injection objects, selecting all the set injection objects as the objects to be injected, and setting a preset percentage of the set injection objects as the objects to be injected.

4. A chaos engineering platform with fault injection verification function according to claim 3, characterized in that: The types of injection objects include stand-alone machines and Kubernetes clusters; the stand-alone machine includes stand-alone processes and stand-alone containers; the Kubernetes cluster consists of a group of nodes running containerized applications, Pods run on nodes, and the Pods include one or more pod containers.

5. A chaos engineering platform with fault injection verification function according to claim 4, characterized in that: In the fault injection platform, when the injection object of the fault model is a pod, node or pod container, the API service sends a fault injection request and indicator monitoring task corresponding to the fault model to the Operator.

6. A chaos engineering platform with fault injection verification function according to claim 5, characterized in that: In the fault injection platform, when the type of the injection object is a Kubernetes cluster, the PID is obtained by entering the corresponding namespace, where the namespace is used to store the PID of the program process, which is a pod process, a node process, or a pod container process.

7. A chaos engineering platform with fault injection verification function according to claim 6, characterized in that: The failure types of the faulty tasks include Java failure, DNS failure, Pod failure, disk failure and network failure.

8. A chaos engineering platform with fault injection verification function according to claim 7, characterized in that: The fault parameters include fault ID, fault type and fault action fields; The fault action field includes: kill pod process, kill pod container process, IP resolution error, method delay, custom exception, memory overflow, CPU full load, IO limit, read and write speed limit, network rejection fault, network packet loss fault, network delay fault and network packet disorder fault.

Images