Cloud native non-consistency fault detection method for distributed application

Abstract

The invention discloses a cloud native non-consistency fault detection method for a distributed application, and the method comprises the steps: deploying a Kubernetes cloud native platform in a local environment, deploying a to-be-detected distributed application on the Kubernetes cloud native platform as a detection object, and operating a work load in the detection object; injecting a plurality of cloud native non-consistency fault modes into the detection object, wherein the modes comprise unstable IP address distribution, DNS service delay and SIGTERM signal termination application program; based on a cluster health monitoring algorithm and test judgment logic of different fault modes, whether a fault exists or not is judged by monitoring the number and the state of Pod. The method is a testing tool for the cloud native non-consistency fault in the distributed application, the cloud native non-consistency fault can be detected in an end-to-end mode, and the stability and reliability of the distributed application are ensured.

Description

Technical Field

[0001] The present invention belongs to the field of fault detection for distributed applications, and particularly relates to a cloud-native non-consistent fault detection method for distributed applications. Background Art

[0002] Cloud Native is a software approach for building, deploying, and managing applications in a cloud computing environment. It has changed the basic practices of modern system development and deployment: developers utilize cloud services (such as Microsoft Azure, Amazon AWS, Google GCE, etc.) and deploy systems in a virtualized cloud environment (such as Kubernetes, or simply k8s), rather than on local physical machines. Cloud computing abstracts hardware resources into highly elastic and easily scalable abstract computing resources through system-level virtualization technology, and allocates them in real-time on demand according to user needs, and then charges according to the actual usage. Compared with the traditional physical machine-based deployment method, cloud computing provides higher flexibility, scalability, and availability, and thus has received extensive attention and application.

[0003] Although cloud-native technology has been widely applied in the industrial community, distributed applications still play an important role. To support cloud computing, developers need to migrate early distributed applications (such as Zookeeper, Cassandra, Kafka, etc.) from traditional bare-metal server environments to cloud environments. However, some distributed applications migrated to the cloud-native environment may run normally in the bare-metal server environment but fail in the cloud-native environment under the same configuration, input, and execution order. This phenomenon is called cloud-native non-consistent fault.

[0004] Taking the failure with JIRA number ZOOKEEPER-2184 as an example, ZooKeeper is a distributed coordination application used to save and synchronize configurations and data among multiple nodes. ZooKeeper clients access by resolving the domain name of the ZooKeeper server into an IP address. When ZooKeeper is deployed on the Kubernetes cloud-native platform, the ZooKeeper server may encounter failures or be scheduled for restart. However, Kubernetes will assign a new IP address to the restarted ZooKeeper server, causing the ZooKeeper clients still accessing with the old IP address to be unable to connect to the restarted server and thus unable to read or write any data. This failure does not occur in the bare-metal server environment because the IP address of the ZooKeeper server deployed on a physical machine remains unchanged after restart; however, in the cloud-native environment, the IP address of the server will be re-assigned after restart. Such a failure that occurs in the cloud-native environment but not on traditional physical devices is exactly the cloud-native non-uniformity failure.

[0005] Avoiding cloud-native non-uniformity failures is crucial for the reliability of cloud computing because many important cloud services still rely on traditional distributed applications. Due to the existence of cloud-native non-uniformity failures, traditional distributed applications face new challenges when migrating to the cloud-native environment, which may lead to various impacts such as service interruption, function failure, data loss, decreased reliability, and decreased performance. Therefore, a detection tool for cloud-native non-uniformity failures of distributed applications is needed. Summary of the Invention

[0006] Aiming at the deficiencies of the prior art, the purpose of the embodiments of this application is to provide a method for detecting cloud-native non-uniformity failures of distributed applications.

[0007] According to the first aspect of the embodiments of this application, a method for detecting cloud-native non-uniformity failures of distributed applications is provided, including:

[0008] S1: Deploy the Kubernetes cloud-native platform in the local environment, deploy the distributed application to be detected on the Kubernetes cloud-native platform as the detection object, and run a workload in the detection object;

[0009] S2: Inject several cloud-native non-uniformity failure modes into the detection object, including unstable IP address allocation, DNS service latency, and SIGTERM signal to terminate the application;

[0010] S3: Based on the cluster health monitoring algorithm and the test determination logic of different failure modes, determine whether there is a failure by monitoring the number and status of Pods.

[0011] Furthermore, step S1 includes:

[0012] S11: Deploy the Kubernetes cloud-native platform on the local device;

[0013] S12: Deploy the distributed application to be detected as the detection object on the Kubernetes cloud-native platform through the k8s configuration file or the Operator mode;

[0014] S13: Initialize the detection object;

[0015] S14: Run a workload in the detection object to simulate user input.

[0016] Furthermore, in step S12, create an independent namespace for each distributed application through the kubectl command-line tool.

[0017] Furthermore, for the IP address allocation instability mode, change its IP address by restarting a Pod;

[0018] For the DNS service latency failure mode, modify the coredns component of Kubernetes and add an idle waiting time to its service discovery function;

[0019] For the SIGTERM signal to terminate the application mode, modify the default termination waiting time of each detection object, delete a Pod, and record the time it takes for the Pod to terminate running from receiving the SIGTERM signal.

[0020] Furthermore, in step S3, based on the test determination logic for different failure modes, determine whether there is a failure by monitoring the number and status of Pods, including:

[0021] For the failure of IP address allocation instability, monitor the behavior of the IP address change of the Pods of the detection object, and determine whether the distributed application can resume normal operation after the IP change through the cluster health monitoring algorithm. If it cannot resume, it is determined that the detection object has an IP address allocation instability failure;

[0022] For the failure of DNS service latency, monitor the behavior of the detection object when DNS service latency occurs, and determine whether the detection object can operate normally in the presence of DNS latency through the cluster health monitoring algorithm. If it cannot operate normally, it is determined that the detection object has a DNS service latency failure;

[0023] For the failure of the application terminated by the SIGTERM signal, based on the time it takes for the Pod to terminate its operation from receiving the SIGTERM signal, determine whether the detection object can normally exit the operation within the limited time, and use the cluster health monitoring algorithm to determine whether the detection object is in a normal operating state. If it cannot normally exit the operation or the detection object is not in a normal operating state, it is determined that the detection object has a failure of the application terminated by the SIGTERM signal.

[0024] Further, in step S3, the cluster health monitoring algorithm includes:

[0025] Obtain all current Pods and their statuses;

[0026] If the number of Pods meets the expectation and all Pods are in the running state, it is determined that the detection object is in a normal operating state;

[0027] If the number of Pods does not meet the expectation or any one Pod is not in the running state, wait for 10×2 retry seconds and then re-obtain all current Pods and their statuses and make a determination of the running state; if the normal operating conditions are still not met within the limit of the maximum number of retries, it is determined that the detection object is not in a normal operating state.

[0028] According to the second aspect of the embodiments of the present application, there is provided a cloud-native non-uniformity fault detection device for distributed applications, including:

[0029] An environment deployment module, configured to deploy a Kubernetes cloud-native platform in a local environment, deploy a distributed application to be detected as a detection object on the Kubernetes cloud-native platform, and run a workload in the detection object;

[0030] A fault injection module, configured to inject several cloud-native non-uniformity fault modes into the detection object, including unstable IP address allocation, DNS service delay, and SIGTERM signal terminating the application;

[0031] A test determination module, configured to determine whether there is a fault by monitoring the number and status of Pods based on the cluster health monitoring algorithm and the test determination logic of different fault modes.

[0032] According to the third aspect of the embodiments of the present application, there is provided a computer program product, including computer programs / instructions, which when executed by a processor implement the method described in the first aspect.

[0033] According to the fourth aspect of the embodiments of the present application, there is provided an electronic device, including:

[0034] One or more processors;

[0035] A memory for storing one or more programs;

[0036] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect.

[0037] According to the fifth aspect of the embodiments of the present application, there is provided a computer-readable storage medium, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method as described in the first aspect are implemented.

[0038] Compared with the prior art, the technical innovation points of the present invention are: 1. End-to-end test method: Compared with traditional unit tests, this test method runs tests on the real cloud-native platform Kubernetes and has inputs in a real cloud-native environment. 2. Precise fault injection and test determination logic: It can inject specific fault modes and determine whether the faults can be correctly handled according to the fault modes, with higher precision compared to general fault injection methods. 3. Good scalability: The detection objects, injected fault modes, and test determination logic of the present invention can be conveniently added and extended, and the present invention itself can also be used as a process in continuous integration for the development process of distributed applications. Therefore, the beneficial effects of the present invention are:

[0039] 1) Automatically test cloud-native non-conformance vulnerabilities: The present invention is the first automatic test tool for cloud-native non-conformance vulnerabilities in distributed applications. The present invention can inject fault modes of cloud-native non-conformance, simulate the process of fault occurrence, and verify whether the distributed application can handle the faults.

[0040] 2) Effectiveness: The present invention conducts end-to-end tests in a real cloud-native environment. Compared with unit tests, the test scheme of the present invention includes inputs and test determination logic in a real cloud-native environment, and the test effectiveness is stronger.

[0041] 3) Scalability: The present invention supports multiple distributed applications as test objects and can be extended to more distributed applications and fault modes. In addition, the present invention can be used as a step in continuous integration for the development process of distributed applications.

[0042] Therefore, the present invention has good prospects for popularization and application.

[0043] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. Description of the Drawings

[0044] The accompanying drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0045] Figure 1 is a cloud-native non-uniform fault detection method for distributed applications shown according to an exemplary embodiment.

[0046] Figure 2 A block diagram of a cloud-native non-uniform fault detection device for distributed applications shown according to an exemplary embodiment.

[0047] Figure 3 is a schematic diagram of an electronic device shown according to an exemplary embodiment. Detailed implementation manners

[0048] Here, the exemplary embodiments will be described in detail, and the examples are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present application.

[0049] The terms used in the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The singular forms "a", "the", and "said" used in the present application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0050] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0051] Figure 1 is a flowchart of a cloud-native non-uniform fault detection method for distributed applications shown according to an exemplary embodiment. As Figure 1 shown, this method is applied to a terminal and includes the following steps:

[0052] Step S1: Deploy the Kubernetes cloud-native platform in the local environment, deploy the distributed application to be detected as the detection object on the Kubernetes cloud-native platform, and run the workload in the detection object; this step may include the following sub-steps:

[0053] Step S11: Deploy the Kubernetes cloud-native platform on the local device. Deploy it through the kind (Kubernetes in Docker) tool. The Kubernetes platform deployed in this way uses a Docker container as a Kubernetes cluster, without the need to create a cluster on multiple physical devices, and the deployment is more convenient and suitable for use as a test.

[0054] Step S12: Deploy the distributed application to be detected as the detection object on the Kubernetes cloud-native platform through the k8s configuration file or the Operator mode. Specifically, in this application, the kubectl command-line tool is used to create an independent namespace for each distributed application to isolate the object to be detected and avoid the mutual influence between the system components of Kubernetes and the detection object. The distributed applications that are currently supported are shown in Table 1 below:

[0055] Table 1 Distributed applications supported by Kubernetes

[0056]

[0057]

[0058] Step S13: Initialize the detection object. Specifically, after all detection objects are deployed, count the number and status of the Pods of the detection object, and wait for all Pods to be in the running state, where a Pod is the smallest deployable computing unit that can be created and managed in Kubernetes.

[0059] Step S14: Run the workload in the detection object to simulate user input, such as the YCSB (Yahoo! Cloud Serving Benchmark) test set, or test sets for specific detection objects such as Cassandra-stress and Redis Benchmark as the workload.

[0060] Step S2: Inject several cloud-native non-conformance fault modes into the detection object, including unstable IP address allocation, DNS service latency, and SIGTERM signal to terminate the application. After the deployment is completed in Step S1 and the Pod list of the distributed application is obtained, inject cloud-native non-conformance fault modes into the detection object. In this embodiment, the above three cloud-native non-conformances can be supported. Specifically, for each injected fault mode, perform the following actions:

[0061] For the unstable IP address allocation mode, change the IP address of a Pod by restarting it to simulate unstable IP address allocation;

[0062] For the DNS service latency fault mode, modify the coredns component of Kubernetes and add a certain idle waiting time to its service discovery function to simulate DNS latency;

[0063] For the SIGTERM signal to terminate the application mode, modify the default termination waiting time (terminationGracePeriodSeconds value) of each detection object, then delete a Pod, and record the time it takes for the Pod to terminate running from receiving the SIGTERM signal to simulate the behavior of terminating the application using the SIGTERM signal in the cloud-native environment.

[0064] Step S3: Based on the cluster health monitoring algorithm and the test determination logic for different fault modes, determine whether there is a fault by monitoring the number and status of Pods;

[0065] Specifically, the test determination includes the following process:

[0066] Monitor the number and health status of Pods to determine the running status of the distributed application. If the number of Pods does not meet the expectation or not all enter the Running state after a certain period of time, it is regarded as a failure in the health status check of the distributed application. And this tool uses an exponential backoff strategy to retry multiple times when querying the number and status of Pods to reduce the possibility of false alarms caused by network environment fluctuations.

[0067] For the fault of unstable IP address allocation, monitor the behavior of the IP address change of the Pods of the detection object, and determine whether the distributed application can resume the normal running state after the IP change through the cluster health monitoring algorithm. If it cannot be restored, it is determined that the detection object has a fault of unstable IP address allocation.

[0068] For the failure of DNS service latency, monitor the behavior of the detection object when DNS service latency occurs, and use the cluster health monitoring algorithm to determine whether the detection object can operate normally in the presence of DNS latency. If it cannot operate normally, it is determined that the detection object has a failure of DNS service latency.

[0069] For the failure of terminating the application with the SIGTERM signal, based on the time it takes for the Pod recorded in step S2 from receiving the SIGTERM signal to terminating the operation, determine whether the detection object can normally exit the operation within the default termination waiting time, and use the cluster health monitoring algorithm to determine whether the detection object is in a normal operating state. If it cannot normally exit the operation or the detection object is not in a normal operating state, it is determined that the detection object has a failure of terminating the application with the SIGTERM signal.

[0070] Specifically, the cluster health monitoring algorithm accepts two inputs, namely the expected number of Pods and the maximum number of retries max_retry, and maintains the currently retried number retry within the limit of the maximum number of retries (retry ≤ max_retry). Use the kubectl_get_pods() API provided by Kubernetes to obtain all current Pods and their statuses. It is regarded as healthy if and only if the number of Pods meets the expectation and all Pods are in the running state (Running state). If the number of Pods does not meet the expectation or any one Pod is not in the running state, wait for 10×2 retry seconds, and then check again. If the check still fails within the limit of the maximum number of retries, it is regarded as unhealthy.

[0071] Finally, this application will also record the running results of the workload to determine whether the detection object can correctly execute the workload in the case of including failures. If the running results of the workload are incorrect in a certain failure mode, it is also determined that the detection object has a problem with this failure mode.

[0072] In summary, this application deploys the distributed application to be detected on the Kubernetes cloud-native platform, and completes the cloud-native non-uniformity fault detection for the distributed application through the following steps: First, deploy the Kubernetes cloud-native platform in the local environment and deploy the distributed application to be detected as the detection object, and run the workload in the detection object to simulate user input; Second, inject cloud-native non-uniformity fault modes such as unstable IP address allocation, DNS service latency, and SIGTERM signal termination into the detection object; Finally, based on the cluster health monitoring algorithm, by monitoring the number and status of Pods, and combining the running results of the workload, determine whether the detection object has various failures, and complete the cloud-native non-uniformity fault detection for the distributed application.

[0073] Corresponding to the foregoing embodiments of the cloud-native non-uniformity fault detection method for distributed applications, the present application also provides an embodiment of a cloud-native non-uniformity fault detection device for distributed applications.

[0074] Figure 2 It is a block diagram of a cloud-native non-uniformity fault detection device for distributed applications shown according to an exemplary embodiment. Refer to Figure 2 This device may include:

[0075] An environment deployment module 21, configured to deploy a Kubernetes cloud-native platform in a local environment, deploy a distributed application to be detected as a detection object on the Kubernetes cloud-native platform, and run a workload in the detection object;

[0076] A fault injection module 22, configured to inject several cloud-native non-uniformity fault modes into the detection object, including unstable IP address allocation, DNS service latency, and SIGTERM signal to terminate the application;

[0077] A test determination module 23, configured to determine whether there is a fault by monitoring the number and status of Pods based on a cluster health monitoring algorithm and test determination logics for different fault modes.

[0078] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments of the method, and will not be elaborated herein.

[0079] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial descriptions of the method embodiments. The device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present application. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0080] Correspondingly, the present application also provides a computer program product, including a computer program / instructions, which when executed by a processor, implement the cloud-native non-uniformity fault detection method for distributed applications as described above.

[0081] Correspondingly, the present application further provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the cloud-native non-uniformity fault detection method for distributed applications as described above. As Figure 3 shown, it is a hardware structure diagram of a device with any data processing capability where the cloud-native non-uniformity fault detection device for distributed applications provided by an embodiment of the present invention is located. In addition to Figure 3 the processors, memory, and network interfaces shown, any device with data processing capability where the device in the embodiment is located usually includes other hardware according to the actual functions of the device with any data processing capability, which will not be elaborated here.

[0082] Correspondingly, the present application further provides a computer-readable storage medium, on which computer instructions are stored, and when the instructions are executed by a processor, the cloud-native non-uniformity fault detection method for distributed applications as described above is implemented. The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device, such as a plug-in hard disk, a Smart Media Card (SMC), an SD card, a Flash Card, etc. equipped on the device. Further, the computer-readable storage medium may also include both the internal storage unit of any device with data processing capability and the external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store the data that has been output or will be output.

[0083] Those skilled in the art will readily think of other implementation schemes of the present application after considering the specification and practicing the content disclosed herein. The present application aims to cover any variations, uses, or adaptive changes of the present application, and these variations, uses, or adaptive changes follow the general principles of the present application and include the common general knowledge or conventional technical means in the technical field not disclosed in the present application.

Claims

1. A cloud-native inconsistency fault detection method for distributed applications, characterized in that: include: S1: deploying a Kubernetes cloud-native platform in a local environment, deploying a distributed application to be detected on the Kubernetes cloud-native platform as a detection object, and running a workload in the detection object; S2: Injects several cloud-native inconsistency failure modes into the detection object, including unstable IP address allocation, DNS service delay, and SIGTERM signal termination of the application; S3: Based on the cluster health monitoring algorithm and the test judgment logic of different failure modes, it determines whether there is a failure by monitoring the number and status of Pods.

2. The method according to claim 1, characterized in that: Step S1 includes: S11: Deploy Kubernetes cloud-native platform on local devices; S12: Deploy the distributed application to be tested on the Kubernetes cloud native platform as the test object through the k8s configuration file or Operator mode; S13: Initializing the detection object; S14: Running a workload in the detection object to simulate user input.

3. The method according to claim 2, characterized in that In step S12, an independent namespace is created for each distributed application using the kubectl command line tool.

4. The method according to claim 1, characterized in that: For unstable IP address allocation mode, restart a Pod to change its IP address; For the DNS service delay failure mode, modify the coredns component of Kubernetes and add an idle waiting time to the service discovery function. For the SIGTERM signal to terminate the application mode, modify the default termination waiting time of each detection object, delete a Pod and record the time it takes for the Pod to terminate from receiving the SIGTERM signal.

5. The method according to claim 1, characterized in that: In step S3, based on the test judgment logic of different failure modes, it is determined whether there is a failure by monitoring the number and status of Pods, including: For the failure of unstable IP address allocation, monitor the IP address change behavior of the Pod of the detection object, and use the cluster health monitoring algorithm to determine whether the distributed application can restore normal operation after the IP change. If it cannot be restored, it is determined that the detection object has an unstable IP address allocation failure; For DNS service delay failures, monitor the behavior of the detection object when DNS service delay occurs, and use the cluster health monitoring algorithm to determine whether the detection object can operate normally in the presence of DNS delay. If it cannot operate normally, it is determined that the detection object has a DNS service delay failure; For the failure of terminating the application with the SIGTERM signal, the time taken for the Pod to terminate from receiving the SIGTERM signal is used to determine whether the detection object can exit normally within the limited time, and the cluster health monitoring algorithm is used to determine whether the detection object is in a normal running state. If it cannot exit normally or the detection object is not in a normal running state, it is determined that the detection object has a failure of terminating the application with the SIGTERM signal.

6. The method according to claim 1, characterized in that In step S3, the cluster health monitoring algorithm includes: Get all current Pods and their status; If the number of Pods meets expectations and all Pods are in the running state, the detected object is determined to be in a normal running state; If the number of Pods does not meet the expectation or any Pod is not in the running state, wait 10×2 retry After a few seconds, all current Pods and their status are retrieved again, and the running status is determined; if the normal running conditions are still not met within the maximum retry count limit, the detection object is determined to be not in a normal running state.

7. A cloud-native inconsistency fault detection device for distributed applications, characterized in that: include: An environment deployment module, used to deploy the Kubernetes cloud native platform in the local environment, deploy the distributed application to be detected on the Kubernetes cloud native platform as the detection object, and run the workload in the detection object; A fault injection module is used to inject several cloud-native inconsistency failure modes into the detection object, including unstable IP address allocation, DNS service delays, and SIGTERM signal termination of the application; The test judgment module is used to determine whether there is a fault by monitoring the number and status of Pods based on the cluster health monitoring algorithm and the test judgment logic of different failure modes.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images