Automated testing method and device for failure recovery mechanism of distributed system

Abstract

The present invention discloses an automated testing method and apparatus for failure recovery mechanisms in distributed systems. The method comprises: injecting an initial fault sequence into the distributed system to obtain target system operating information; generating and mutating a fault sequence seq′ based on the target system operating information, system-specific constraints, and user-specific constraints to update a fault sequence queue; selecting a fault sequence seq from the fault sequence queue; testing the distributed system using the fault sequence seq, obtaining a defect report or updating the fault sequence queue, and then returning to selecting a fault sequence seq from the fault sequence queue; and obtaining a test result for the distributed system based on the defect report. The present invention can cover more system code and expose failure recovery defects.

Description

Technical Field

[0001] The present invention belongs to the field of software technology, and in particular relates to an automated testing method and device for a failure recovery mechanism of a distributed system. Background Art

[0002] In the era of big data and cloud computing, distributed systems have become a key pillar of modern applications, providing reliable support for users. Distributed systems, represented by the distributed storage system HDFS, the cluster management service YARN, the distributed computing framework MapReduce, and the distributed coordination service ZooKeeper, are widely deployed and used by major internet companies such as Alibaba, Baidu, Tencent, and Google. Large-scale distributed systems manage a large number of distributed software components, hardware, and their configurations, typically consisting of tens of thousands of physical machines (i.e., compute nodes). These numerous compute nodes must work together, run various complex protocols, and face various non-deterministic scenarios (such as network and failure scenarios).

[0003] Reliability and fault tolerance are the primary goals of modern distributed systems. Due to power outages, hardware errors, software errors, and other reasons, nodes in distributed clusters will inevitably fail (crash), making the nodes inaccessible. As the scale of nodes in distributed clusters increases, node failures become more common. When a node fails, the tasks running on the node and the node's memory data are immediately lost, and other active nodes in the cluster may not work properly (such as when the master node fails). To cope with node failures, developers have introduced various complex automatic failure recovery mechanisms in distributed systems. For example, the write-ahead log mechanism in HBase persists every write request to a log file. Once a node fails, the data before the node failure can be restored by replaying the log.

[0004] However, correctly handling node failures is challenging for distributed systems. Failure recovery in distributed systems is also prone to errors. Node failures can occur at any time on any node, leading to a variety of failure scenarios, such as partial updates to persistent state and loss of in-memory data. The inherent complexity of distributed systems exacerbates the complexity of failure recovery. Developers struggle to predict all possible failure scenarios and ensure that failure recovery mechanisms are correctly implemented. Furthermore, during the system testing phase, it's not feasible to thoroughly test distributed systems by automatically injecting node failures. Consequently, incorrect failure recovery mechanisms and their implementation can introduce complex crash recovery bugs, resulting in serious consequences such as cluster outages, node inaccessibility, and data loss, impacting the reliability, availability, and performance of distributed systems.

[0005] Existing work that can be used to detect failure recovery defects and test the correctness of distributed system failure recovery is mainly based on fault injection technology, including implementation-level distributed system model checkers, distributed system testing frameworks based on fault injection, and defect detection technology based on program analysis technology. Implementation-level distributed system model checkers such as Modist (Junfeng Yang et al. "MODIST: Transparent Model Checking of Unmodified Distributed Systems". Proc. NSDI 2009.), SAMC (Tanakorn Leesatapornwongsa et al. "SAMC: Semantic-Aware Model Checking for Fast Discovery of Deep Bugs in Cloud Systems". Proc. OSDI 2014.), and FlyMC (Jeffrey F Lukman et al. "FlyMC: Highly Scalable Testing of Complex Interleavings in Distributed Systems". Proc EuroSys 2019.) systematically enumerate all possible execution paths by intercepting various nondeterministic events in distributed systems (such as network messages and faults) and reordering their execution, thereby exposing defects that are only triggered under specific execution paths. These distributed system model checkers do not consider faults in their testing or most only consider injecting single node failures. Furthermore, since these distributed system model checkers detect more than just defects related to node failures, they explore a large amount of state space unrelated to node failures before exposing a failure recovery defect. The biggest problem facing distributed model checkers is the explosion of state search space when applied to real distributed systems. Distributed system testing frameworks based on fault injection emphasize injecting various faults, including node failures / restarts, into the system to test the system's behavior under abnormal conditions. Random fault injection frameworks such as Chaos Monkey (https: / / netflix.github.io / chaosmonkey / ) and Jepsen (https: / / github.com / jepsen-io / jepsen) can randomly inject node failures in an attempt to trigger a defect through multiple runs.PreFail (Pallavi Joshi et al. "PREFAIL: A Programmable Tool for Multiple-Failure Injection". Proc. OOPSLA 2011.) is a programmable fault injection tool that allows testers to customize various fault injection strategies. Fate (Haryadi S. Gunawi et al. "FATE and DESTINI: A Framework for Cloud Recovery Testing". Proc. NSDI 2011.) uses abstract fault IDs to systematically search the fault scenario space and test multiple fault combinations. Compared to distributed system model checkers, these fault injection-based distributed system testing frameworks do not systematically control the execution order of various non-deterministic events. Their main problem is the explosion of the fault scenario space. Work that uses program analysis techniques to detect failure recovery defects, such as FCatch (Haopeng Liu et al. "FCatch: Automatically Detecting Time-of-Fault Bugs in Cloud Systems". Proc. ASPLOS 2018.) and CrashTuner (Jie Lu et al. "CrashTuner: Detecting Crash-Recovery Bugs in Cloud Systems via Meta-Info Analysis". Proc. SOSP 2019.), performs defect detection based on specific defect patterns. These methods can only cover a limited space of fault scenarios.

[0006] In summary, existing work on distributed system failure recovery defect detection either only covers a limited space of fault scenarios, relies on the experience of testers, or faces the problem of space explosion. Therefore, these methods still have limitations in systematically testing distributed systems and exposing failure recovery defects. Summary of the Invention

[0007] The present invention aims to address the problem of an explosion of fault scenario space in existing fault injection-based distributed system testing methods by providing a method and apparatus for automated testing of failure recovery mechanisms in distributed systems. This method systematically tests the fault scenario space by collecting system runtime information to adjust node failure / restart injection and adjusting fault scenario testing priorities based on an optimized fault scenario selection strategy. This allows for faster coverage of more system code, particularly recovery-related code, and exposes failure recovery defects.

[0008] The technical solution of the present invention includes:

[0009] An automated testing method for a failure recovery mechanism of a distributed system, the method comprising:

[0010] Injecting an initial fault sequence into the distributed system to obtain target system operation information; wherein the initial fault sequence does not contain any fault;

[0011] Generate and mutate a fault sequence seq′ based on the target system operation information, system-specific constraints, and user-specific constraints to update the fault sequence queue; wherein the system-specific constraints represent the node downtime that the target cluster in the target system can tolerate at the same time, and the user-specific constraints represent the maximum number of faults in the fault sequence seq′;

[0012] Select a fault sequence seq from the fault sequence queue;

[0013] After testing the distributed system using the fault sequence seq and obtaining a defect report or updating the fault sequence queue, returning to selecting a fault sequence seq from the fault sequence queue;

[0014] Based on the defect report, a test result of the distributed system is obtained.

[0015] Furthermore, the generating and mutating the fault sequence seq′ based on the target system operation information, the system-specific constraints and the user-specific constraints to obtain the fault sequence queue includes:

[0016] Based on the operation information, obtain I / O related information and code coverage;

[0017] In the case where the code coverage includes code blocks not covered by the previous fault sequence being tested, or the last fault in the fault sequence being tested is a node failure fault, then,

[0018] Generate a fault sequence seq′ based on the I / O related information; wherein the I / O related information includes: a node ID of the executed I / O operation, a call stack, an operation source path, an operation target path, and a timestamp; the operation source path and the operation target path include: a node ID and a file path;

[0019] In the case where the number of faults in the fault sequence seq′ is equal to the maximum number of faults in the user-specific constraint, the fault sequence queue is not updated;

[0020] If the number of faults in the fault sequence seq′ is less than the maximum number of faults in the user-specified constraint, then for the I / O points after the last I / O point in the fault sequence seq′ where the fault event is a node failure or a node restart, each operation selects an I / O point and adds the node failure fault or the node restart fault to obtain a new set of fault sequences and adds them to the fault sequence queue;

[0021] In the case that the code coverage includes code blocks covered by previously tested fault sequences and the last fault in the fault sequence seq is not a node failure fault, the fault sequence seq is discarded and the fault sequence queue is not updated.

[0022] Furthermore, for the I / O points after the last I / O point in the fault sequence seq′ where the fault event is a node failure or a node restart, each operation selects an I / O point and adds the node failure fault or the node restart fault to obtain a new set of fault sequences to be added to the fault sequence queue, including:

[0023] Find the position loc of the last I / O point in the fault sequence seq′ where the fault event is a node failure or a node restart, and create an index value i of the I / O point after the last I / O point; wherein the initial value of the index value i is loc+1;

[0024] Create a mutation fault sequence set mutates; the initial value of the mutation fault sequence set mutates is empty;

[0025] When the fault event corresponding to the i-th I / O point is a no-fault event, determine whether the injected node failure event and the node restart event meet the system-specific constraints, and add the fault sequence that meets the system-specific constraints to the mutated fault sequence set mutates;

[0026] If the fault event corresponding to the i-th I / O point is a node failure event or a node restart event, skip the i-th I / O point and process the i+1-th I / O point;

[0027] After all I / O points after the last I / O point are processed, the mutated fault sequence set mutates is added to a fault sequence queue.

[0028] Furthermore, the selecting a fault sequence seq from the fault sequence queue includes:

[0029] Selecting a set of suspected fault sequence candidate sets from the fault sequence queue;

[0030] Performing a priority score on each fault sequence in the candidate set of suspected fault sequences;

[0031] According to the scoring results, a fault sequence is determined;

[0032] Furthermore, selecting a set of suspicious fault sequence candidate sets from the fault sequence queue includes:

[0033] Generate a suspicious fault sequence candidate set candidates; the initial value of the suspicious fault sequence candidate set candidates is empty;

[0034] When strategy one is applied with a certain probability, based on the fault IDs of all fault sequences in the fault sequence queue, fault sequences with globally untested fault IDs are added to the set of candidates for the suspected fault sequence. Strategy one is applied to preferentially select fault sequences with globally untested fault IDs for testing. The fault ID is a hash value calculated from the call stack and fault event type of the I / O point corresponding to the last fault in a fault sequence.

[0035] When the current suspicious fault sequence candidate set candidates is not empty, the current suspicious fault sequence candidate set candidates is used as the suspicious fault sequence candidate set;

[0036] When the current set of suspected fault sequence candidates is empty and strategy 2 is applied with a certain probability, all fault sequences in the fault sequence queue whose last fault occurs at an I / O point during the recovery process are found and added to the set of suspected fault sequence candidates. Strategy 2 is applied to preferentially test fault sequences whose last fault occurs at an I / O point during the recovery process. The I / O point during the recovery process is determined by determining whether the I / O point appears after the fault injection test and has a new call stack.

[0037] When the current suspicious fault sequence candidate set candidates is not empty, the current suspicious fault sequence candidate set candidates is used as the suspicious fault sequence candidate set;

[0038] When the current set of suspected fault sequence candidates is empty and strategy three is applied with a certain probability, the fault IDs of all fault sequences in the fault sequence queue are calculated. For the mutated fault sequence group mutates obtained by mutating the same fault sequence seq′, if there is a fault sequence with a fault ID that has not been tested for the mutated fault sequence group mutates, then the fault sequence is added to the set of suspected fault sequence candidates. In particular, the application strategy three is to give priority to testing fault sequences with untested fault IDs in the same mutated group.

[0039] When the current suspicious fault sequence candidate set candidates is not empty, the current suspicious fault sequence candidate set candidates is used as the suspicious fault sequence candidate set;

[0040] When the current suspicious fault sequence candidate set candidates is empty, all fault sequences in the fault sequence queue are taken as the suspicious fault sequence candidate set.

[0041] Furthermore, the step of performing priority scoring on each fault sequence in the candidate set of suspected fault sequences includes:

[0042] Calculating a first priority score based on the execution speed of the fault sequence; wherein a fault sequence with a faster execution speed obtains a higher first priority score than a fault sequence with a slower execution speed;

[0043] Calculating a second priority score based on the code coverage corresponding to the fault sequence; wherein a fault sequence with a high code coverage rate obtains a higher second priority score than a fault sequence with a low code coverage rate;

[0044] Calculating a third priority score based on the waiting rounds of the fault sequence in the fault sequence queue; wherein a fault sequence with more waiting rounds obtains a higher third priority score than a fault sequence with fewer waiting rounds;

[0045] Calculating a fourth priority score based on the number of faults in the fault sequence; wherein, if the number of faults is no more than a set number, the fault sequence with a larger number of faults obtains a higher fourth priority score than the fault sequence with a smaller number of faults; and if the number of faults is more than the set number, the fault sequence with a smaller number of faults obtains a higher fourth priority score than the fault sequence with a larger number of faults;

[0046] The first priority score, the second priority score, the third priority score and the fourth priority score are integrated to obtain a priority score of the corresponding fault sequence.

[0047] Furthermore, after testing the distributed system using the fault sequence seq and obtaining a defect report or updating the fault sequence queue, returning to selecting a fault sequence seq from the fault sequence queue includes:

[0048] During the process of injecting the fault sequence seq into the distributed system, if at least one fault cannot be injected, interrupting the injection process, adding the fault sequence seq to the fault sequence queue, and then returning to selecting a fault sequence seq from the fault sequence queue;

[0049] During the process of injecting the fault sequence seq into the distributed system, if all faults of the fault sequence seq are successfully injected and the fault symptom checker of the workload in the distributed system does not confirm a potential defect, then the fault sequence queue is updated based on the operation information of the distributed system, and the process returns to selecting a fault sequence seq from the fault sequence queue;

[0050] In the process of injecting the fault sequence seq into the distributed system, all faults of the fault sequence seq are successfully injected, and the fault symptom checker of the workload in the distributed system confirms the potential defect or the confirmation time exceeds the set time, then a defect report is generated and returned to the process of selecting a fault sequence seq from the fault sequence queue.

[0051] An automated testing device for a failure recovery mechanism of a distributed system, the device comprising:

[0052] An information acquisition module, configured to inject an initial fault sequence into the distributed system to obtain target system operation information; wherein the initial fault sequence does not include any fault;

[0053] a queue update module, configured to generate and mutate a fault sequence seq′ based on the target system operation information, system-specific constraints, and user-specific constraints, so as to update the fault sequence queue; wherein the system-specific constraints represent the node downtime that the target cluster in the target system can tolerate at the same time, and the user-specific constraints represent the maximum number of faults in the fault sequence seq′;

[0054] A fault sequence selection module selects a fault sequence seq from the fault sequence queue;

[0055] a system testing module, configured to test the distributed system using the fault sequence seq, obtain a defect report or update the fault sequence queue, and then return to selecting a fault sequence seq from the fault sequence queue;

[0056] A result generating module is used to obtain the test result of the distributed system based on the defect report.

[0057] A storage medium stores a computer program, wherein the computer program is configured to execute any of the above methods when running.

[0058] An electronic device, characterized in that it includes a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute any of the above methods.

[0059] Compared with the prior art, the present invention has the following technical advantages:

[0060] 1. The method of the present invention targets the failure recovery mechanism of distributed systems. It generates various fault scenarios including multiple node failures and node restart combinations through system runtime feedback for testing. It can systematically test the fault scenario space of the target distributed cluster, cover the situations that may be encountered in the production environment as fully as possible, and help distributed system developers to promptly discover failure recovery defects in the system.

[0061] 2. When generating and selecting fault scenarios for testing, the present invention is guided by code coverage and gives priority to testing fault scenarios that are easy to increase code coverage and trigger failure recovery defects. This can more effectively explore the fault scenario space within limited time and resources, greatly improving testing efficiency.

[0062] 3. The system runtime information collected by this invention includes I / O-related information and code coverage information, which can be easily adapted to new distributed systems. For a new target system, developers only need to identify the I / O-related operations of the system's application layer or use the JDK-level I / O operation identification provided by this invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 Flowchart of the test model of the method of the present invention.

[0064] Figure 2 An example diagram of the overall architecture of the method of the present invention.

[0065] Figure 3 Example diagram of initial fault sequence generation and mutation.

[0066] Figure 4 Example diagram of general fault sequence generation and mutation. DETAILED DESCRIPTION

[0067] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only specific embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0068] The technical solution of the present invention includes an automated testing method for the failure recovery mechanism of a distributed system. In this automated testing method, a failure scenario involving one or more node failures / restart failures is abstracted into a failure sequence consisting of I / O points arranged in ascending timestamp order. Through system runtime feedback, a series of failure sequences are automatically generated to test the failure recovery mechanism of the target distributed system, quickly covering more failure recovery-related code and exposing failure recovery defects.

[0069] Specifically, the present invention is based on Figure 1 The test model shown: collects system runtime information by running the target distributed system and the specified workload. If the code coverage increases or the last fault in the current fault sequence under test is a node failure, fault sequence generation and mutation are performed. If the maximum test time is not reached and the fault sequence queue is not empty, a new fault sequence is selected from the fault sequence queue for testing, and fault injection is performed according to the fault sequence under test.

[0070] The following combination Figure 2 The specific implementation architecture example diagram in the figure is used to describe the method of the present invention in detail.

[0071] Step 1: Inject the initial fault sequence into the distributed system to obtain the target system operation information.

[0072] The present invention first initializes an empty fault sequence to be tested (seq) (i.e., no faults are injected into the target system), drives the target cluster and workload to run for the first time, and collects system runtime information through dynamic code instrumentation technology. The system runtime information includes code coverage information and I / O-related information. The code coverage information includes: information about the basic code blocks covered by the target cluster during this run; the I / O-related information includes: the node ID, call stack, operation source path, operation target path, and timestamp of the I / O operations performed by the target cluster during this run; the I / O operations include: file I / O and network I / O; the operation source path and operation target path include: node ID and file path.

[0073] Step 2: Based on the target system operation information, system-specific constraints and user-specific constraints, generate and mutate the fault sequence seq′ to obtain the fault sequence queue.

[0074] Based on the system runtime information, system-specific constraints and user-specific constraints, fault sequences are generated and mutated, and added to the fault sequence queue. Specifically, it is first necessary to collect and record the runtime information of all nodes in the target cluster, and compare the code coverage information of this run with the code coverage information of previous tests. If code coverage is increased in this run, or the last fault injected in this run is a node failure fault, a fault sequence seq′ is generated based on the collected I / O related information. The I / O points in the fault sequence seq′ are the I / O points actually executed in this test, and are arranged in ascending order by timestamp. The fault events corresponding to these I / O points are the fault events actually injected when fault injection is performed according to the fault sequence seq in this test: node failure, node restart or no fault. For the first run, the fault event corresponding to each I / O point is a no-fault event. As Figure 3 As shown in Figure 1, after the initial operation, an initial fault sequence containing 4 I / O points is obtained. Figure 4 As shown in the figure, during normal operation, after a fault injection test is performed according to a tested fault sequence seq, a fault sequence seq′ containing 4 I / O points is obtained based on the I / O related information collected during this operation. Compared with the tested fault sequence seq during this operation, the I / O points Crash due to node failure n2 (Even if node n2 fails) and disappears, the new I / O point Appear.

[0075] Secondly, based on a series of constraints, the present invention generates a new variant fault sequence by injecting a new fault (node ​​failure fault or node restart fault) into a fault-free I / O point after the I / O point where the last fault is located in seq′. By traversing all fault-free I / O points after the I / O point where the last fault is located in seq′, the present invention finally obtains a series of variant fault sequences by mutating seq′ and adds them to the fault sequence queue. Here, the series of constraints include general constraints (such as only restarting failed nodes and only failing active nodes), system-specific constraints (such as the number of nodes that fail at the same time in the target cluster can be tolerated by the target cluster), and user-specified constraints (such as the number of injected faults does not exceed the maximum number of faults specified by the user). Finally, all variant fault sequences obtained based on the mutation of seq′ in this step are added to the fault sequence queue.

[0076] In one example, the fault sequence queue update process includes:

[0077] 1) Determine whether the current operation is the initial operation:

[0078] If yes, go to step 2);

[0079] If not, proceed to step 3);

[0080] 2) Generate an initial fault sequence seq′ based on the collected I / O operations and related information. The I / O points in the initial fault sequence corresponding to the I / O related information in this operation are sorted in ascending order by timestamp. The fault event corresponding to each I / O point is a non-fault event, and then proceed to step 6).

[0081] 3) Based on the collected runtime coverage information, determine whether this test has increased code coverage:

[0082] If yes, go to step 5);

[0083] If not, proceed to step 4);

[0084] 4) Determine whether the last fault in the fault sequence seq is a node failure event:

[0085] If yes, go to step 5);

[0086] If not, proceed to step 13);

[0087] 5) Based on the collected I / O operations and related information, as well as the fault injection results, the fault sequence seq is calibrated to generate the fault sequence seq′. The I / O-related information corresponding to the I / O points in seq′ is the I / O operations and related information actually executed during this run, and the fault events corresponding to the I / O points in seq′ are the node failure events, node restart events, or no-fault events actually injected into the I / O points during this run.

[0088] 6) Determine whether the number of faults in the fault sequence seq′ is equal to the maximum number of faults specified in the user-specified constraint:

[0089] If yes, proceed to step 13);

[0090] If not, proceed to step 7);

[0091] 7) For the fault sequence seq′, find the position of the last I / O point in the sequence where the fault event is a node failure or a node restart, which is loc, create an index value i with an initial value of loc+1, and create a set of mutated fault sequences mutates with an initial value of empty;

[0092] 8) For the i-th I / O point in the fault sequence seq′, determine whether the corresponding fault event is a no-fault event:

[0093] If yes, go to step 9);

[0094] If not, proceed to step 11);

[0095] 9) Determine whether injecting a node failure event into the i-th I / O point in the fault sequence seq′ satisfies a series of constraints:

[0096] If so, generate a new mutated fault sequence m, whose initial value is the same as seq′, so that the fault event corresponding to the i-th I / O point in m becomes a node failure event, add m to the set mutates, and go to step 10);

[0097] If not, proceed to step 10);

[0098] 10) Determine whether injecting a node restart event at the i-th I / O point in the fault sequence seq′ satisfies a series of constraints:

[0099] If so, generate a new mutated fault sequence m, whose initial value is the same as seq′, so that the fault event corresponding to the i-th I / O point in m becomes a node restart event, add m to the set mutates, and go to step 11);

[0100] If not, proceed to step 11);

[0101] 11) Set the value of i to i+1 and determine whether the position pointed to by i is greater than the position of the last I / O point in seq′:

[0102] If yes, proceed to step 12);

[0103] If not, proceed to step 8);

[0104] 12) Add the mutated fault sequence mutates to the fault sequence queue;

[0105] 13) Enter the subsequent fault sequence selection stage.

[0106] Step 3: Select a fault sequence seq from the fault sequence queue.

[0107] If the current test time has not reached the pre-specified maximum test time and the fault sequence queue is not empty, a set of suspicious fault sequences is selected from the fault sequence queue based on the pre-configured probability. These suspicious fault sequences are then prioritized and scored. Based on the priority scores, a fault sequence seq is randomly selected from the candidate suspicious fault sequences for testing. The specific steps for selecting a fault sequence are as follows: first, a set of candidate suspicious fault sequences is selected, then each fault sequence in the candidate set is prioritized and, based on the scores, a fault sequence is randomly selected from the candidate set for testing.

[0108] Step 3.1: Select a set of suspected fault sequence candidates.

[0109] The specific steps for selecting a candidate set of suspected fault sequences are as follows: First, a candidate set of suspected fault sequences, named candidates, is generated, with its initial value set to empty. Next, a random number (random) between [0 and 1) is generated and the value of random is determined to be less than the probability of applying Strategy 1, such as 99%. If so, the fault IDs of all fault sequences in the fault sequence queue are calculated, and globally, any fault sequences with untested fault IDs are added to candidates. If candidates is not empty, it is returned as a candidate set of suspected fault sequences.

[0110] Furthermore, the fault ID of the fault sequence is a hash value calculated from the call stack of the I / O point corresponding to the last fault in a fault sequence and the fault event type.

[0111] If candidates is empty, a random number between [0, 1) is generated and the probability of applying Strategy 2, such as 95%, is determined. If so, all fault sequences in the fault sequence queue whose last fault occurs at an I / O point during the recovery process are found and added to candidates. The I / O point during the recovery process is determined by determining whether the I / O point appears after the fault injection test and has a new call stack. If candidates is not empty, it is returned as a candidate set of suspected fault sequences.

[0112] If candidates is empty, a random number between [0, 1) is generated and the probability of applying strategy three is determined, e.g., 90%. If so, the fault IDs of all fault sequences in the fault sequence queue are calculated. For a set of mutated fault sequences mutates derived from the same fault sequence seq′, if a sequence m in the mutates has a fault ID that has not been tested for the mutates, m is added to candidates. If candidates is not empty, it is returned as a candidate set of suspected fault sequences.

[0113] Finally, if candidates is still empty, all the sequences to be tested in the fault sequence queue are returned as a set of suspected fault sequence candidates.

[0114] In one example, strategy 1 prioritizes testing fault sequences with globally untested fault IDs. Strategy 2 prioritizes testing fault sequences whose last fault occurs during recovery. Strategy 3 prioritizes testing fault sequences with untested fault IDs within the same mutation group. Fault sequences within the same mutation group are those derived from mutating the same fault sequence seq′.

[0115] In addition, the probabilities of the application-optimized failure scenario selection strategies one, two, and three are configurable parameters, and based on the configurable parameters, the effect of the method when applied to different systems and different workloads is adjusted.

[0116] Step 3.2: Prioritize each fault sequence in the candidate set.

[0117] For the obtained candidate set of suspected fault sequences, the method of the present invention scores the fault sequences therein according to the following principles: improve the test priority of fault sequences with short execution time; improve the test priority of fault sequences with high code coverage; improve the test priority of fault sequences with high waiting rounds in the fault sequence queue (wherein waiting rounds refer to the rounds that a fault sequence goes through in the fault sequence queue before being tested. Each time a fault sequence is selected from the fault sequence for testing is a round); when the number of faults in the fault sequence is no more than six, improve the test priority of fault sequences with more faults; when the number of faults is greater than six, reduce the test priority of fault sequences with more faults. Based on the priority score, a fault sequence is randomly selected from the candidate set of suspected fault sequences for testing.

[0118] Step 3.3: Randomly select a fault sequence from the scores for testing.

[0119] The present invention integrates the scores of various fault sequences to obtain a final priority score for the fault sequence. Finally, based on the priority score, a fault sequence is randomly selected from the candidate set of suspicious fault sequences for testing.

[0120] Step 4: Use the fault sequence seq to test the distributed system, obtain a defect report or update the fault sequence queue, and then return to selecting a fault sequence seq from the fault sequence queue.

[0121] According to the current fault sequence under test, the target cluster and workload are run, system runtime information is collected and recorded, and the I / O related information collected at the I / O operation points of the target system is reported to the fault injection test controller. The controller controls the execution of the target system and the injection of faults according to the fault sequence under test. Specifically, the fault injection test controller runs the workload to drive the operation of the target cluster. The target cluster collects relevant information at all I / O points and reports it to the fault injection test controller and waits for the decision of the fault injection test controller. The fault injection test controller decides to inject a node failure fault, a node restart fault, or to allow the reporting node to continue running at a certain node based on the current fault sequence under test seq.

[0122] In this step, first, the present invention collects information at each I / O point of the target system and reports it to a unified fault injection test controller. The fault injection test controller compares the collected I / O point information with the I / O point information in the fault sequence to be tested one by one to inject faults in sequence and control the execution of the target system. If a reported I / O point information is consistent with the I / O point information where the fault to be injected is located, the fault injection controller injects the corresponding node failure fault or node restart fault by calling a predefined script (for node restart faults, the fault injection controller will also notify the node that reported the I / O point information to continue running). Otherwise, the fault injection controller notifies the node that reported the I / O point information to continue running.

[0123] Secondly, during the operation of the current test workload, and after the workload operation is completed, the method described in the present invention will run a predefined fault symptom checker to confirm potential defects. Specifically, the checker of the present invention will check general fault symptoms (such as FATAL, ERROR and Exception in the operation log, and node crashes), and will also check operation-specific fault symptoms (such as operations returning error codes or reading outdated data). Based on the checker that has been implemented in the present invention, users can easily implement specific checkers for other workloads. In actual use, testers can reduce defect omissions and defect false alarms by designing more sophisticated fault symptom checkers for specific workloads under specific target systems.

[0124] In one example, for a fault injection test, after the workload is completed, if there is a fault in the fault sequence under test that cannot be successfully injected, the fault sequence will be re-added to the fault sequence queue and retested in the subsequent test process; if all faults in the fault sequence under test are successfully injected and this test passes all predefined fault symptom checkers, the runtime information of this test system is collected to enter the subsequent fault sequence generation and mutation stage; if all faults in the fault sequence under test are successfully injected and this test does not pass all predefined fault symptom checkers, a defect report is generated, and then a new fault sequence is selected from the fault sequence queue for testing; if the workload in this test cannot end within the pre-configured timeout period, the fault injection test is re-performed according to the fault sequence under test. If the new test still cannot end within 2 times the timeout period, a pending defect report is generated, and then a new fault sequence is selected from the fault sequence queue for testing.

[0125] Step 5: Based on the defect report, obtain the test results of the distributed system.

[0126] If the current test time has not ended and there are still fault sequences to be tested in the fault sequence queue (ie the fault sequence queue is not empty), a fault sequence is selected from them for further testing. Otherwise, the present invention generates a final test result based on the defect reports of each test.

[0127] In summary, the method of the present invention targets the failure recovery mechanism of distributed systems, and generates various fault scenarios including multiple node failures and node restart combinations through system runtime feedback for testing. It can systematically test the fault scenario space of the target distributed cluster, cover the situations that may be encountered in the generation environment as fully as possible, and help distributed system developers to promptly discover failure recovery defects in the system.

[0128] When generating and selecting fault scenarios for testing, the present invention is guided by code coverage and prioritizes testing fault scenarios that are easy to increase code coverage and trigger failure recovery defects. This can more effectively explore the fault scenario space within limited time and resources, greatly improving testing efficiency.

[0129] The system runtime information collected by this invention includes I / O-related information and code coverage information, which can be easily adapted to new distributed systems. For a new target system, developers only need to identify the I / O-related operations of the system application layer or use the I / O operation identification provided by the JDK layer provided by this invention.

[0130] Although the specific implementation process and example drawings of the present invention are disclosed for illustrative purposes, their purpose is to facilitate understanding of the content of the present invention and to facilitate implementation accordingly, those skilled in the art will appreciate that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the illustrated implementation process and example drawings.

Claims

1. An automated testing method for failure recovery mechanisms of distributed systems, characterized in that: The method comprises: Injecting an initial fault sequence into the distributed system to obtain target system operation information; wherein the initial fault sequence does not contain any fault; Based on the target system operation information, the system-specific constraints, and the user-specific constraints, a fault sequence seq′ is generated and mutated to update a fault sequence queue; wherein the system-specific constraints represent the node downtime that a target cluster in the target system can tolerate at the same time, and the user-specific constraints represent the maximum number of faults in the fault sequence seq′. The generating and mutating the fault sequence seq′ based on the target system operation information, the system-specific constraints, and the user-specific constraints to obtain a fault sequence queue includes: Based on the operation information, obtain I / O related information and code coverage; In the case where the code coverage includes code blocks not covered by the previous fault sequence being tested, or the last fault in the fault sequence being tested is a node failure fault, then, Generate a fault sequence seq′ based on the I / O related information; wherein the I / O related information includes: a node ID of the executed I / O operation, a call stack, an operation source path, an operation target path, and a timestamp; the operation source path and the operation target path include: a node ID and a file path; In the case where the number of faults in the fault sequence seq′ is equal to the maximum number of faults in the user-specific constraint, the fault sequence queue is not updated; If the number of faults in the fault sequence seq′ is less than the maximum number of faults in the user-specified constraint, then for the I / O points after the last I / O point in the fault sequence seq′ where the fault event is a node failure or a node restart, each operation selects an I / O point and adds the node failure fault or the node restart fault to obtain a new set of fault sequences and adds them to the fault sequence queue; If the code coverage includes a code block covered by a previously tested fault sequence and the last fault in the fault sequence seq is not a node failure fault, the fault sequence seq is discarded and the fault sequence queue is not updated; Select a fault sequence seq from the fault sequence queue; After testing the distributed system using the fault sequence seq and obtaining a defect report or updating the fault sequence queue, returning to selecting a fault sequence seq from the fault sequence queue; Based on the defect report, a test result of the distributed system is obtained.

2. The method according to claim 1, wherein For the I / O points after the last I / O point in the fault sequence seq′ where the fault event is a node failure or a node restart, each operation selects an I / O point and adds the node failure fault or the node restart fault to obtain a new set of fault sequences to be added to the fault sequence queue, including: Find the position loc of the last I / O point in the fault sequence seq′ where the fault event is a node failure or a node restart, and create an index value i of the I / O point after the last I / O point; wherein the initial value of the index value i is loc+1; Create a mutation fault sequence set mutates; the initial value of the mutation fault sequence set mutates is empty; When the fault event corresponding to the i-th I / O point is a no-fault event, determine whether the injected node failure event and the node restart event meet the system-specific constraints, and add the fault sequence that meets the system-specific constraints to the mutated fault sequence set mutates; If the fault event corresponding to the i-th I / O point is a node failure event or a node restart event, skip the i-th I / O point and process the i+1-th I / O point; After all I / O points after the last I / O point are processed, the mutated fault sequence set mutates is added to a fault sequence queue.

3. The method according to claim 1, wherein The selecting a fault sequence seq from the fault sequence queue comprises: Selecting a set of suspected fault sequence candidate sets from the fault sequence queue; Performing a priority score on each fault sequence in the candidate set of suspected fault sequences; According to the scoring results, a fault sequence is determined.

4. The method according to claim 3, wherein The selecting a set of suspicious fault sequence candidate sets from the fault sequence queue includes: Generate a suspicious fault sequence candidate set candidates; the initial value of the suspicious fault sequence candidate set candidates is empty; When strategy one is applied with a certain probability, based on the fault IDs of all fault sequences in the fault sequence queue, fault sequences with globally untested fault IDs are added to the set of candidates for the suspected fault sequence. Strategy one is applied to preferentially select fault sequences with globally untested fault IDs for testing. The fault ID is a hash value calculated from the call stack and fault event type of the I / O point corresponding to the last fault in a fault sequence. When the current suspicious fault sequence candidate set candidates is not empty, the current suspicious fault sequence candidate set candidates is used as the suspicious fault sequence candidate set; When the current set of suspected fault sequence candidates is empty and strategy 2 is applied with a certain probability, all fault sequences in the fault sequence queue whose last fault occurs at an I / O point during the recovery process are found and added to the set of suspected fault sequence candidates. Strategy 2 is applied to preferentially test fault sequences whose last fault occurs at an I / O point during the recovery process. The I / O point during the recovery process is determined by determining whether the I / O point appears after the fault injection test and has a new call stack. When the current suspicious fault sequence candidate set candidates is not empty, the current suspicious fault sequence candidate set candidates is used as the suspicious fault sequence candidate set; When the current set of suspected fault sequence candidates is empty and strategy three is applied with a certain probability, the fault IDs of all fault sequences in the fault sequence queue are calculated, and the fault sequence group mutates obtained by mutating the same fault sequence seq′ is added to the set of suspected fault sequence candidates if there is a fault sequence with a fault ID that has not been tested for the mutated fault sequence group mutates. The application strategy three is to give priority to testing fault sequences with untested fault IDs in the same mutated group. When the current suspicious fault sequence candidate set candidates is not empty, the current suspicious fault sequence candidate set candidates is used as the suspicious fault sequence candidate set; When the current suspicious fault sequence candidate set candidates is empty, all fault sequences in the fault sequence queue are taken as the suspicious fault sequence candidate set.

5. The method according to claim 3, wherein The step of performing priority scoring on each fault sequence in the candidate set of suspected fault sequences includes: Calculating a first priority score based on the execution speed of the fault sequence; wherein a fault sequence with a faster execution speed obtains a higher first priority score than a fault sequence with a slower execution speed; Calculating a second priority score based on the code coverage corresponding to the fault sequence; wherein a fault sequence with a high code coverage rate obtains a higher second priority score than a fault sequence with a low code coverage rate; Calculating a third priority score based on the waiting rounds of the fault sequence in the fault sequence queue; wherein a fault sequence with more waiting rounds obtains a higher third priority score than a fault sequence with fewer waiting rounds; Calculating a fourth priority score based on the number of faults in the fault sequence; wherein, if the number of faults is no more than a set number, the fault sequence with a larger number of faults obtains a higher fourth priority score than the fault sequence with a smaller number of faults; and if the number of faults is more than the set number, the fault sequence with a smaller number of faults obtains a higher fourth priority score than the fault sequence with a larger number of faults; The first priority score, the second priority score, the third priority score and the fourth priority score are integrated to obtain a priority score of the corresponding fault sequence.

6. The method according to claim 1, wherein After testing the distributed system using the fault sequence seq and obtaining a defect report or updating the fault sequence queue, returning to selecting a fault sequence seq from the fault sequence queue includes: During the process of injecting the fault sequence seq into the distributed system, if at least one fault cannot be injected, interrupting the injection process, adding the fault sequence seq to the fault sequence queue, and then returning to selecting a fault sequence seq from the fault sequence queue; During the process of injecting the fault sequence seq into the distributed system, if all faults of the fault sequence seq are successfully injected and the fault symptom checker of the workload in the distributed system does not confirm a potential defect, then the fault sequence queue is updated based on the operation information of the distributed system, and the process returns to selecting a fault sequence seq from the fault sequence queue; During the process of injecting the fault sequence seq into the distributed system, all faults of the fault sequence seq are successfully injected, and the fault symptom checker of the workload in the distributed system confirms the potential defect or the confirmation time exceeds the set time, then a defect report is generated and returned to the selection of a fault sequence seq from the fault sequence queue.

7. An automated testing device for failure recovery mechanisms of distributed systems, characterized in that: The device comprises: An information acquisition module, configured to inject an initial fault sequence into the distributed system to obtain target system operation information; wherein the initial fault sequence does not include any fault; A queue updating module is configured to generate and mutate a fault sequence seq′ based on the target system operation information, system-specific constraints, and user-specific constraints to update a fault sequence queue; wherein the system-specific constraints represent the node downtime that a target cluster in the target system can tolerate at the same time, and the user-specific constraints represent the maximum number of faults in the fault sequence seq′. The generation and mutation of the fault sequence seq′ based on the target system operation information, system-specific constraints, and user-specific constraints to obtain a fault sequence queue includes: Based on the operation information, obtain I / O related information and code coverage; In the case where the code coverage includes code blocks not covered by the previous fault sequence being tested, or the last fault in the fault sequence being tested is a node failure fault, then, Generate a fault sequence seq′ based on the I / O related information; wherein the I / O related information includes: a node ID of the executed I / O operation, a call stack, an operation source path, an operation target path, and a timestamp; the operation source path and the operation target path include: a node ID and a file path; In the case where the number of faults in the fault sequence seq′ is equal to the maximum number of faults in the user-specific constraint, the fault sequence queue is not updated; If the number of faults in the fault sequence seq′ is less than the maximum number of faults in the user-specified constraint, then for the I / O points after the last I / O point in the fault sequence seq′ where the fault event is a node failure or a node restart, each operation selects an I / O point and adds the node failure fault or the node restart fault to obtain a new set of fault sequences and adds them to the fault sequence queue; If the code coverage includes a code block covered by a previously tested fault sequence and the last fault in the fault sequence seq is not a node failure fault, the fault sequence seq is discarded and the fault sequence queue is not updated; A fault sequence selection module selects a fault sequence seq from the fault sequence queue; a system testing module, configured to test the distributed system using the fault sequence seq, obtain a defect report or update the fault sequence queue, and then return to selecting a fault sequence seq from the fault sequence queue; A result generating module is used to obtain the test result of the distributed system based on the defect report.

8. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 6.

9. A storage medium storing a computer program, wherein: The computer program is configured to execute the method according to any one of claims 1 to 6 when executed.

Images