A chaos engineering test case generation method, device and electronic equipment

Abstract

This disclosure provides a method, apparatus, electronic device, and readable medium for generating chaos engineering test cases. The method includes a seed test case editing module generating multiple chaos engineering test seed cases; a test case orchestration management library orchestrating the seed test cases according to test case orchestration rules to generate multiple test cases; a chaos engineering front-end probe importing the generated test cases into a blockchain node and a fuzzy test data mutation module, which mutates the test parameters in the test cases according to mutation rules to generate new test parameters; the chaos engineering front-end probe importing the test cases including the new test parameters into the blockchain node, collecting the operating parameters of the blockchain system where the blockchain node resides, filtering problematic test cases, and submitting them to a test feedback module; and the test feedback module extracting the feedback problematic test cases to update the test case orchestration rules in the test case orchestration management library, thereby improving the completeness and reliability of chaos engineering testing.

Description

Technical Field

[0001] This disclosure relates to the field of blockchain technology, and in particular to a method, apparatus and electronic device for generating chaos engineering test cases. Background Technology

[0002] Chaos engineering is a practice that intentionally and in a controlled manner induces failures in a production or pre-production environment to understand the impact of these failures and plan better defense postures and incident maintenance strategies. Unlike traditional forward verification (verifying that a system possesses a certain function), chaos engineering is a reverse testing approach. It primarily involves proactively injecting various errors into the system and then observing whether the system remains in a normal, stable, and service-available state, thereby identifying potential problems in advance and improving system availability. In current software system testing, due to the complexity of the program system, the vast input space, and limitations such as time and cost, test cases constructed by testers typically only cover a subset of program behavior; this is also known as deterministic testing methods (i.e., the input test data is deterministic, and the number of test cases is limited). A single test case can be tested individually; however, software systems are highly complex, and test cases have various combinations and configurations (e.g., serial or parallel combinations). Potential relationships exist between different test cases, which may lead to system crashes. As the number of test cases increases, the combinations and configurations between them increase exponentially, making it difficult to manually traverse all combinations and configurations to construct test cases efficiently. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, and readable medium for generating chaos engineering test cases. By incorporating fuzzy testing concepts into the chaos engineering testing system, a greater number of fault injection test cases are constructed to cover as many test boundary conditions and abnormal situations as possible. Furthermore, the test case orchestration rules are optimized to adjust the composition and combination of test cases, thereby improving the completeness and reliability of chaos engineering testing.

[0004] To achieve the above technical objectives, the embodiments of this disclosure adopt the following technical solutions:

[0005] In a first aspect, embodiments of this disclosure provide a method for generating chaos engineering test cases, the method comprising:

[0006] The seed test case editing module generates multiple chaos engineering test seed test cases, where each seed test case includes one or more test parameters;

[0007] The test case orchestration management library orchestrates seed test cases to generate multiple test cases according to test case orchestration rules, which include the composition and combination method of seed test cases;

[0008] The Chaos Engineering front-end probe imports the generated test cases into the blockchain node and the fuzzy test data mutation module. The fuzzy test data mutation module mutates the test parameters in the test cases according to the mutation rules to generate new test parameters.

[0009] The chaos engineering front-end probe will import test cases, including new test parameters, into the blockchain node;

[0010] The front-end probe of the chaos engineering collects the operating parameters of the blockchain system where the blockchain node is located, filters problem test cases, and submits them to the test feedback module.

[0011] The test feedback module extracts the problem test cases from the feedback to update the test case orchestration rules in the test case orchestration management library.

[0012] In some possible implementations, the method further includes:

[0013] The initial orchestration management module generates seed test case combinations and combination methods based on the known relationships between test cases.

[0014] In some possible implementations, the test feedback module extracts the reported problem test cases to update the test case orchestration rules in the test case orchestration management library, including:

[0015] Extract the combination and combination methods of seed test cases from the problem test cases to supplement or replace the test case orchestration rules in the test case orchestration management library.

[0016] In some possible implementations, the chaos engineering front-end probe collects the operating parameters of the blockchain system where the blockchain nodes reside and filters problem test cases, including:

[0017] Determine the range of stable operating parameters for the blockchain system;

[0018] Collect the blockchain system's operating parameters after importing multiple sets of test parameters for any test case into the blockchain node;

[0019] For any set of test parameters, if the collected operating parameters of the blockchain system exceed the range of stable operating parameters of the blockchain system, the set of test parameters is identified as abnormal test parameters; if the number or proportion of abnormal test parameters in multiple sets of test parameters exceeds the threshold, the test cases are filtered and identified as problematic test cases.

[0020] In some possible implementations, the mutation rules of the fuzz test data mutation module include one or more of the following:

[0021] Randomly reverse the bits of the test parameters;

[0022] Perform random arithmetic operations on the integer digits of the test parameters;

[0023] Randomly select values ​​from a specific list of boundary values ​​to replace some of the test parameter values;

[0024] Randomly select values ​​or strings from the data dictionary to replace part of the values ​​or strings in the test parameters.

[0025] In some possible implementations, the test parameters in the test cases are mutated to generate new test parameters, including:

[0026] Identify the core test parameters from one or more test parameters, mutate the core test parameters, and generate new test parameters.

[0027] Secondly, embodiments of this disclosure provide a chaos engineering test case generation apparatus, the apparatus comprising:

[0028] The seed test case editing module is configured to generate multiple chaos engineering test seed test cases, where each seed test case includes one or more test parameters;

[0029] The test case orchestration management library is configured to orchestrate seed test cases to generate multiple test cases according to test case orchestration rules, where the test case orchestration rules include the composition and combination method of seed test cases;

[0030] The chaos engineering front-end probe is configured to import generated test cases into the blockchain node and fuzz test data mutation module; import test cases including new test parameters into the blockchain node; collect the running parameters of the blockchain system where the blockchain node is located, filter problematic test cases, and submit them to the test feedback module.

[0031] The fuzz test data mutation module is configured to mutate the test parameters in the test cases according to mutation rules to generate new test parameters.

[0032] The test feedback module is configured to extract problematic test cases from feedback to update the test case orchestration rules in the test case orchestration management library.

[0033] In some possible implementations, the device also includes an initial orchestration management module configured to generate seed test case combinations and combinations based on the relationships between known test cases.

[0034] In some possible implementations,

[0035] The test feedback module is configured to extract problematic test cases from feedback to update the test case orchestration rules in the test case orchestration management library, including:

[0036] Extract the combination and combination methods of seed test cases from the problem test cases to supplement or replace the test case orchestration rules in the test case orchestration management library.

[0037] In some possible implementations, the chaos engineering front-end probe is configured to collect operational parameters of the blockchain system where the blockchain nodes reside, and to screen problem test cases, including:

[0038] Determine the range of stable operating parameters for the blockchain system;

[0039] Collect the blockchain system's operating parameters after importing multiple sets of test parameters for any test case into the blockchain node;

[0040] For any set of test parameters, if the collected operating parameters of the blockchain system exceed the range of stable operating parameters of the blockchain system, the set of test parameters is identified as abnormal test parameters; if the number or proportion of abnormal test parameters in multiple sets of test parameters exceeds the threshold, the test cases are filtered and identified as problematic test cases.

[0041] In some possible implementations, the mutation rules of the fuzz test data mutation module include one or more of the following:

[0042] Randomly reverse the bits of the test parameters;

[0043] Perform random arithmetic operations on the integer digits of the test parameters;

[0044] Randomly select values ​​from a specific list of boundary values ​​to replace some of the test parameter values;

[0045] Randomly select values ​​or strings from the data dictionary to replace part of the values ​​or strings in the test parameters.

[0046] In some possible implementations, the fuzz test data mutation module identifies one or more core test parameters among the test parameters, mutates the core test parameters, and generates new test parameters.

[0047] Thirdly, embodiments of this application provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect.

[0048] Fourthly, embodiments of this application provide a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in the first aspect.

[0049] The first aspect of the technical solution provided by the embodiments of this disclosure brings at least the following beneficial effects: The method generates multiple chaotic engineering test seed cases through a seed case editing module; the test case orchestration management library orchestrates the seed cases according to the test case orchestration rules to generate multiple test cases; the chaotic engineering front-end probe imports the generated test cases into the blockchain node and the fuzzy test data mutation module, and the fuzzy test data mutation module mutates the test parameters in the test cases according to the mutation rules to generate new test parameters; the chaotic engineering front-end probe imports the test cases including the new test parameters into the blockchain node, collects the operating parameters of the blockchain system where the blockchain node is located, filters problem test cases, and submits them to the test feedback module; the test feedback module extracts the feedback problem test cases to update the test case orchestration rules of the test case orchestration management library, thereby improving the completeness and reliability of chaotic engineering testing. The chaotic engineering test case generation method of the embodiments of this disclosure combines the concept of fuzzy testing in the chaotic engineering testing system to construct a larger number of fault injection test cases to cover as many test boundary conditions and abnormal situations as possible, and further optimizes the test case orchestration rules to adjust the combination composition and combination method of test cases, thereby improving the completeness and reliability of chaotic engineering testing.

[0050] It should be noted that the technical effects of any of the implementation methods in the second to fourth aspects can be found in the technical effects of the corresponding implementation methods in the first aspect, and will not be repeated here.

[0051] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments of this disclosure will be briefly described below. Clearly, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit the scope of this disclosure.

[0053] Figure 1 A schematic diagram of a chaos engineering test case generation apparatus according to at least one embodiment of the present disclosure is shown.

[0054] Figure 2 A schematic diagram illustrating the main steps of a method for generating chaos engineering test cases according to at least one embodiment of the present disclosure is shown.

[0055] Figure 3 A schematic diagram of an electronic device according to at least one embodiment of the present disclosure is shown;

[0056] Figure 4 A schematic diagram of a computer-readable medium according to at least one embodiment of the present disclosure is shown. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0058] In the following text, any methods, apparatus, examples, and contents that do not fully correspond to the scope defined by the claims are not derived from the present invention. Such methods, apparatus, examples, and contents, as well as all subsequent descriptions, are for illustrative purposes only, or to highlight specific aspects or features of the claims.

[0059] Note that the examples described below are merely specific examples and are not intended to limit the embodiments of this disclosure to the specific shapes, hardware, connections, operations, values, conditions, data, sequences, etc., shown and described. Those skilled in the art can utilize the concepts of this disclosure to construct further embodiments not mentioned herein by reading this specification.

[0060] The terminology used in this disclosure is that which is currently widely used in the art in consideration of the functionality of this disclosure; however, these terms may vary depending on the intent, precedent, or new technology of those skilled in the art. Furthermore, specific terms may be chosen by the applicant, and in such cases, their detailed meanings will be described in the detailed description of this disclosure. Therefore, the terminology used in this specification should not be construed as simple names, but rather based on the meaning of the terms and the overall description of this disclosure.

[0061] To better understand the embodiments of this disclosure, the relevant terms involved in this disclosure will first be defined and explained.

[0062] Chaos engineering is a technique that uses an experimental approach to improve the resilience and fault tolerance of a system. Its core lies in proactively introducing faulty or chaotic conditions to observe the system's behavior under these abnormal circumstances. The goal of chaos engineering is to discover unknown weaknesses in a system and proactively prevent potential customer harm. It typically conducts experiments in production environments to ensure the system can withstand unpredictable events in the real world. Chaos engineering emphasizes exploring new information through experimentation rather than validating known results.

[0063] Fuzzing is a type of software testing that automatically or semi-automatically generates large amounts of random, invalid, and unexpected test data and feeds it into a program to discover errors and security vulnerabilities. It focuses particularly on the behavior of software under unpredictable conditions, providing a comprehensive assessment of software quality. The goal of fuzzing is to discover defects as early as possible, reducing the cost and effort of fixing them. It stress-tests software through automated error detection, ensuring stability under abnormal conditions or when handling malformed input.

[0064] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0065] Existing chaos engineering testing methods generally employ traditional deterministic testing approaches. This involves testers designing deterministic test cases with specific parameters based on the fault type, resulting in a very limited number of such cases. For example, in a network packet loss scenario, multiple test cases could be set with packet loss rates of 1%, 10%, and 20%. However, the actual packet loss rate ranges from 0 to 100%, meaning an unlimited number of test cases could theoretically be created. Similarly, in a test scenario based on a Java int parameter's range, testers typically select specific values ​​like a, b, and c as parameters and set several test cases. However, this parameter range is (0-2,147,483,647), and in chaos engineering testing, it can also take negative values ​​or exceed the upper bound. Therefore, billions of test cases could be constructed based on this parameter.

[0066] Considering the complexity of blockchain distributed systems, insufficient or unevenly distributed deterministic test cases in chaos engineering testing can have several impacts: First, insufficient or unevenly distributed test cases may fail to cover more potentially problematic scenarios, thus failing to discover more potential issues and affecting the completeness of chaos engineering simulation testing (i.e., more problems are actually undiscovered). Second, even if all tests pass, insufficient or unevenly distributed test cases cannot completely guarantee the correctness of the test results. For example, three test cases may pass, but other uncovered potential test cases may fail, thus affecting the reliability of the test results (i.e., the test appears to pass with a small number of test cases, but actually fails with a large number of test cases). More importantly, due to the complexity of blockchain systems, the correlation between different test cases may allow individual test cases to pass, but in a real-world production environment, the cumulative effect of multiple different test cases can still cause system crashes. However, exhaustively combining all possible test case configurations and methods is impossible in actual testing. Therefore, it is urgent to propose a feasible chaos engineering testing method to ensure the completeness and reliability of the tests.

[0067] Based on the above problems, this disclosure proposes a method for generating chaos engineering test cases in a blockchain scenario. Drawing on the concept of fuzz testing, it proposes a test data mutation mechanism to construct a larger number of fault injection test cases, so as to cover as many test boundary conditions and abnormal situations as possible. Furthermore, it optimizes the test case orchestration rules, adjusts the composition and combination of test cases, and improves the completeness and reliability of chaos engineering testing.

[0068] Figure 1A schematic diagram of a chaos engineering test case generation apparatus 100 according to at least one embodiment of the present disclosure is shown. The test case generation apparatus 100 includes a chaos engineering backend test management system 110, a chaos engineering frontend probe 121, and a fuzzy testing data mutation module 122. The chaos engineering frontend probe 121 and the fuzzy testing data mutation module 122 can be deployed on a blockchain node 120 of a blockchain system. The blockchain node 120 can be a personal laptop, desktop computer, or mobile device, or a server or other blockchain-specific device. This device is equipped with blockchain node software and connects to other nodes via a network to participate in blockchain transaction verification, data storage, and other tasks. The chaos engineering frontend probe 121 can be implemented, for example, using an SDK (Software Development Kit), or added as a plugin to existing software (e.g., blockchain node software). The chaos engineering frontend probe 121 in this disclosure can both inject test cases into the blockchain system and collect the operating parameters of the blockchain system. For example, a blockchain node application is deployed on the computer of blockchain node 120, and a probe program is also installed. The probe program has high system privileges and can inject test cases into the blockchain node application, the blockchain system, the network, or the disk. Simultaneously, the probe program can easily access or send requests to obtain various data from blockchain node 120 and perform data processing and analysis. The fuzzy testing data mutation module 122, based on fuzzy testing principles and data mutation rules, constructs test parameters based on the arranged seed test cases. It can be a separately configured software, installed on the computer of blockchain node 120 along with the chaos engineering front-end probe 121, or directly integrated into the chaos engineering front-end probe 121, or installed as independent software on other devices, communicating with the chaos engineering front-end probe 121 via network connection, etc. This disclosure does not limit this. Alternatively, if the chaos engineering front-end probe 121 is deployed as an independent software program on a device other than blockchain node 120, it can connect to blockchain node 120 via network connection, inject test cases into the blockchain system, and collect the operating parameters of the blockchain system.

[0069] The Chaos Engineering Backend Test Management System 110 includes a seed test case editing module 111, an initial orchestration management module 112, a test case orchestration management library 113, and a test feedback module 114. These modules can be set independently or integrated as needed, for example, into a single Chaos Engineering Backend Test Management System software. This disclosure does not limit this integration. Specifically, the seed test case editing module 111 is used by testers to edit and design specific test case parameters to form seed test cases; the test case orchestration management library 113 is used by testers to manage, orchestrate, and set various types of test cases; the test feedback module 114 feeds back test cases that have encountered problems returned by the Chaos Engineering frontend probe 121 to the test case orchestration management library 113 to update the test case orchestration rules of the library; the initial orchestration management module 112 is optional and is used to initialize the test case orchestration rules.

[0070] The chaos engineering test case generation device of this embodiment interacts according to the following process: The seed test case editing module 111 can generate multiple chaos engineering test seed test cases according to the blockchain system, each seed test case including one or more test parameters (step S101); the initial orchestration management module 112 generates the seed test case combination structure and combination method according to the known association relationship of test cases to initialize the test case orchestration rules (step S102); the test case orchestration management library 113 orchestrates the seed test cases according to the test case orchestration rules to generate multiple test cases, wherein the test case orchestration rules include the seed test case combination structure and combination method (step S103); the generated test cases are then processed by the chaos engineering system. The front-end probe 121 imports the blockchain node 120 and the fuzzy test data mutation module 122 (step S104). The fuzzy test data mutation module 122 mutates the test parameters in the test cases according to the mutation rules to generate new test parameters. The test cases including the new test parameters are imported into the blockchain node 120 through the chaos engineering front-end probe 121 (step S105). The chaos engineering front-end probe 121 collects the blockchain system operation parameters, filters the problematic test cases and submits them to the test feedback module 114 (step S106). The test feedback module 114 extracts the problematic test cases to update the test case orchestration rules of the test case orchestration management library 113 (step S107).

[0071] Optionally, the test feedback module 114 extracts the problem test cases from the feedback to update the test case orchestration rules of the test case orchestration management library 113. It is set to extract the combination and combination method of seed test cases from the problem test cases to supplement or replace the test case orchestration rules of the test case orchestration management library 113.

[0072] Optionally, the chaos engineering front-end probe 121 collects blockchain system operating parameters and filters problematic test cases, including determining the stable parameter range of the blockchain system; it collects the blockchain system operating parameters after importing multiple sets of test parameters for any test case into blockchain node 120; for any set of test parameters, when the collected blockchain system operating parameters exceed the stable parameter range of the blockchain system, the set of test parameters is identified as abnormal test parameters; when the number or proportion of abnormal test parameters in multiple sets of test parameters exceeds a threshold, the test case is filtered and identified as a problematic test case.

[0073] Optionally, the mutation rules of the fuzz test data mutation module 122 may include one or more of the following:

[0074] Randomly reverse the bits of the test parameters;

[0075] Perform random arithmetic operations on the integer digits of the test parameters;

[0076] Randomly select values ​​from a specific list of boundary values ​​to replace some of the test parameter values;

[0077] Randomly select values ​​or strings from the data dictionary to replace parts of the values ​​or strings in the test parameters.

[0078] Optionally, the fuzz test data mutation module 122 determines one or more core test parameters among the test parameters, mutates the core test parameters, and generates new test parameters.

[0079] Figure 2 A schematic diagram of the main steps of a chaos engineering test case generation method 200 according to at least one embodiment of the present disclosure is shown.

[0080] In step S210, the seed test case editing module 111 generates multiple chaos engineering test seed test cases, each of which includes one or more test parameters. In blockchain chaos testing, faults can include various types, such as host faults, network faults, disk faults, time errors, and file faults. Each fault type can be further subdivided. Taking network faults as an example, they can be further divided into sub-fault types such as network corruption, network latency, network loss, network bandwidth, network port occupancy, network partitioning, and DNS faults. Each sub-fault can have one or more test parameters set, and a corresponding seed test case is generated. It is understandable that some of the one or more test parameters are more important, and their changes can significantly affect the stability of the blockchain system, thus serving as core test parameters; while other parameters have a smaller impact on system stability. For example, in the network loss sub-fault, packet loss rate has a significant impact and can be used as a core test parameter; buffer also has some impact, but its impact is smaller in the network loss sub-fault. Since there are many types of sub-faults, and each sub-fault can include multiple test parameters, in actual testing, preferably, when a seed test case includes multiple test parameters, the core test parameters can be mutated first during the mutation process of the fuzzy test data mutation module 122; that is, in the seed test cases of network loss, the core test parameter - packet loss rate can be mutated first according to the mutation rules.

[0081] Understandably, the test parameters in the test cases are injected into the blockchain node 120 through the chaos engineering front-end probe 121, which can modify the settings of the blockchain system. For example, it can modify the file access permissions of the blockchain node, put pressure on the container, put pressure on the disk, occupy network ports, etc., in order to conduct chaos engineering tests.

[0082] In step S220, the test case orchestration management library 113 orchestrates seed test cases according to test case orchestration rules to generate multiple test cases. These orchestration rules include the composition and combination method of seed test cases. Due to the complexity of blockchain systems, a large number of seed test cases require chaos engineering testing. When each seed test case is tested individually, if multiple test parameters are included, the correlation between these parameters may affect the test. For example, taking a blockchain system with four nodes as an example, during network bandwidth testing, if the bandwidth between two nodes is set to 10Mb / s, the blockchain system is stable; however, when the bandwidth between the two nodes is 5Mb / s, the blockchain system cannot complete the consensus process and crashes; and when the bandwidth between the three nodes is 10Mb / s, the system also crashes. In other words, the number of problematic nodes and network bandwidth jointly affect the test results, and the two factors are correlated. The above scenario is only a simplification; in practice, the test parameters need to be continuously mutated through the fuzzy test data mutation module 122 to complete extensive testing in order to obtain the network bandwidth test results.

[0083] More significantly, the relationships between different seed test cases also affect the test results, and these relationships are even more difficult to obtain during actual testing. For example, assuming the system has only 3 seed test cases (A, B, C), and each seed test case includes only 1 test parameter, when only considering the composition of the seed test cases, the test case orchestration management library 113 can set up 7 orchestration methods as shown in Table 1; while considering both the composition of the seed test cases and various combinations, the test case orchestration management library 113 can set up no fewer than 31 orchestration methods as shown in Table 2.

[0084] Table 1. Composition of Seed Use Cases

[0085] Serial Number Seed use case composition 1 A 2 B 3 C 4 A, B 5 B, C 6 A, C 7 A, B, C

[0086] Table 2. Seed Use Case Combinations and Combination Methods

[0087]

[0088]

[0089] As the example above shows, when the number of seed test cases and test parameters increase, the number of orchestration methods available for testing becomes extremely large and inexhaustible. Therefore, preferably, the correlation between seed test cases can be obtained based on fault tree analysis, or the correlation can be extracted based on fault scenarios that occur during testing in production environments, and stored in the initial orchestration management module 112 as the initial input to the test case orchestration management library 113. For example, assuming that the preset fault tree analysis shows that seed test case A and seed test case B are strongly correlated in sequence, then when orchestrating test cases, their sequential combination can be tested first, as shown in Table 3 below, where orchestration methods 7 can be tested first.

[0090] Table 3 Priority Testing Methods

[0091]

[0092]

[0093] Understandably, if different test cases in parallel orchestration include one or more identical parameters, then the identical parameters can be deduplicated; if different test cases in serial orchestration include one or more identical parameters, then deduplication of identical parameters is not necessary, because test parameters may change at any time during production.

[0094] Optionally, the initial rules of the test case orchestration management library 113 can also be set from simple to complex. For example, for the case of 3 seed test cases, first orchestrate the case of a single seed test case (there are only 3 possibilities) to find the key seed test case; then orchestrate the case of 2 seed test cases (there are 9 different ways) to find the key relationships; and then feed back this information, i.e., the problem test case, through the test feedback module 114, and further orchestrate test cases based on the key seed test cases and key relationships found.

[0095] In step S230, the chaos engineering front-end probe 121 imports the generated test cases into the blockchain node 120 and the fuzzy test data mutation module 122. The fuzzy test data mutation module 122 mutates the test parameters in the test cases according to the mutation rules, generating new test parameters. It can be understood that after the test cases are injected into the chaos engineering front-end probe 121, they are used for blockchain system testing via the blockchain node 120, and simultaneously input into the fuzzy test data mutation module 122 to mutate the test parameters. Specific mutation methods can employ one or more of the following:

[0096] 1) Random Bit Reversal. One or more bits, or even a byte (8 bits), of the test data are flipped, with 0s becoming 1s and 1s becoming 0s. When the bits are flipped, the original character or number changes, thus achieving test data mutation. Specifically, the fuzzy test data mutation module 122 obtains the test case file from the chaos engineering front-end probe 121, converts the integer or string into its binary representation; randomly selects one or more bits to flip; converts the modified binary representation back to an integer or character; completes the test data mutation, and writes the new test data (byte array or string) back to the test case file.

[0097] 2) Arithmetic addition and subtraction methods. Addition and subtraction operations are performed on the test data. For example, the original data 10 is increased to 20 through addition, and the original data 5000 is reduced to 2500 through subtraction. Specifically, for example, the fuzzy test data mutation module 122 obtains the test case file from the chaos engineering front-end probe 121, reads the test data, identifies the integers within it, adds to the data, completes the test data mutation, and writes the new test data (byte array or string) back to the test case file.

[0098] 3) Specific Boundary Value Replacement Method. This method replaces test data with specific boundary values, such as 0, -1, MAX_INT (this value varies depending on the language; for example, MAX_INT is 2,147,483,647 in Java), MIN_INT (this value also varies depending on the language; for example, MAX_INT is -2,147,483,648 in Java), and NULL, and stores them in a specific boundary value list. Specifically, for example, the fuzzy test data mutation module 122 obtains the test case file from the chaos engineering front-end probe 121, reads the test data from the test case file as a byte array or string, and uses regular expressions or string matching algorithms to identify integers, floating-point numbers, strings, etc.; it randomly selects a value from the specific boundary value list and replaces a portion of the original value with the selected replacement value, completing the test data mutation. The modified test data is then written back to the test case file.

[0099] 4) Dictionary Replacement Method. This method replaces some numerical values ​​or strings in the test data with predefined numerical values ​​or strings from the data dictionary. Specifically, for example, the fuzzy test data mutation module 122 obtains the test case file from the chaos engineering front-end probe 121, reads the test data in the test case file as a byte array or string, and uses regular expressions or string matching algorithms to identify integer, floating-point, and string values. It then randomly selects a numerical value or string from the data dictionary and replaces some of the test parameter's numerical value or string with the selected replacement value, completing the test data mutation. The modified test data is then written back to the test case file.

[0100] 5) Combination Method. The above methods can be combined to mutate the test data. For example, combining Method 1 and Method 2, Method 1 can be mutated first, and then Method 2 can be executed to mutate the test data based on the mutation result of Method 1. The specific combination method and combination order are not limited in this disclosure.

[0101] In step S240, the chaos engineering front-end probe 121 imports test cases including new test parameters into the blockchain node 120 to test the operation of the blockchain system.

[0102] In step S250, the chaos engineering front-end probe 120 collects the operating parameters of the blockchain system where the blockchain node 120 is located, filters problematic test cases, and submits them to the test feedback module 114. Specifically, first, the stable parameter range of the blockchain system is determined, including one or more operating parameters. For a test case constructed based on orchestration rules, multiple sets of test parameters are generated based on the fuzzy test data mutation module 122. After importing each set of test parameters into the blockchain node, the operating parameters of the blockchain system are collected. For any set of test parameters, when the collected operating parameters of the blockchain system exceed the stable parameter range of the blockchain system, the set of test parameters is identified as abnormal test parameters; if the number or proportion of abnormal test parameters in multiple sets of test parameters exceeds a threshold, the test case is filtered and identified as a problematic test case. For example, assuming that for test case AA, the fuzzy test data mutation module 122 generates 49 sets of test parameters, plus the original unmutated set of test parameters, for a total of 50 sets of test parameters. A threshold number (e.g., 30) or a threshold proportion (e.g., 60%) can be set to determine problematic test cases. For example, if 35 out of 50 sets of test parameters are imported into the blockchain system, and these 35 sets of blockchain system operating parameters exceed the range of stable operating parameters of the blockchain system, that is, there are 35 sets of abnormal test parameters, and the number and proportion of these abnormal test parameters both exceed the threshold, then test case AA is identified as a problematic test case.

[0103] In step S260, the test feedback module 114 extracts the problematic test cases from the feedback to update the test case orchestration rules in the test case orchestration management library. For example, assuming seven test cases are generated according to the orchestration method in Table 3, and are tested separately, the second test case is identified as a problematic test case. This means that the three seed test cases A, B, and C are sequentially tested, demonstrating a sequential relationship between them. This relationship can then be used to supplement or replace the orchestration rules in the test case orchestration management library 113.

[0104] Optionally, repeating steps S220-260 can iterate the orchestration rules and generate new test cases based on the orchestration rules.

[0105] The chaotic engineering test case generation method of this disclosure combines fuzzy testing concepts in the chaotic engineering test system to construct a larger number of fault injection test cases, so as to cover as many test boundary conditions and abnormal situations as possible, and further optimize the test case arrangement rules to adjust the composition and combination of test cases, thereby improving the completeness and reliability of chaotic engineering testing.

[0106] It should be noted that the above application scenarios are merely exemplary, intended to describe one or more aspects of this disclosure in specific scenarios. However, these aspects are not essential, and various modifications can be made to the application scenario. It is readily understood that the specific application scenarios described in this disclosure are not limited.

[0107] At least some embodiments of this disclosure also provide an electronic device. Figure 3 A schematic diagram of an electronic device 300 according to at least one embodiment of the present disclosure is shown.

[0108] like Figure 3 As shown, the electronic device 300 includes one or more processors 310 and a memory 320. The memory 320 includes one or more computer program modules 321. These computer program modules 321 are stored in the memory 320 and are executed by the processor 310. Each computer program module 321 includes instructions for executing a chaos engineering test case generation method and its additional aspects according to at least one embodiment of the present disclosure. When executed by the processor 310, these instructions can perform one or more steps of the chaos engineering test case generation method and its additional aspects according to at least one embodiment of the present disclosure. The memory 320 and the processor 310 can be interconnected via a bus system and / or other forms of connection mechanisms (not shown). For example, the bus may be a Peripheral Component Interconnect Standard (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0109] For example, processor 310 may be a central processing unit (CPU), a digital signal processor (DSP), or other processing unit with data processing and / or program execution capabilities, such as a field-programmable gate array (FPGA). Processor 310 may be a general-purpose processor or a special-purpose processor, capable of controlling other components in electronic device 300 to perform desired functions.

[0110] Exemplarily, memory 320 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules 321 may be stored on the computer-readable storage medium, and processor 310 may run one or more computer program modules 321 to implement various functions of electronic device 300. The computer program modules include multiple computer-executable instructions. Various application programs and various data, as well as various data used and / or generated by the application programs, may also be stored in the computer-readable storage medium.

[0111] For example, electronic device 300 may also include input devices such as touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, and gyroscopes; output devices such as liquid crystal displays, speakers, and vibrators; storage devices such as magnetic tapes and hard disks (HDDs or SDDs); and communication devices such as network interface cards like LAN cards and modems. The communication devices allow electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data and perform communication processing via networks such as the Internet. A drive is connected to the I / O interface as needed. Removable storage media, such as disks, optical disks, magneto-optical disks, and semiconductor memories, are installed on the drive as needed so that computer programs read from them can be installed into the storage device as required.

[0112] For example, the electronic device 300 may further include a peripheral interface (not shown in the figure). This peripheral interface can be various types of interfaces, such as a USB interface, a Lightning interface, etc. The communication device can communicate wirelessly with networks and other devices, such as the Internet, intranets and / or wireless networks such as cellular telephone networks, wireless local area networks (LANs) and / or metropolitan area networks (MANs). Wireless communication can use any of a variety of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (e.g., based on IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n standards), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email, instant messaging, and / or Short Message Service (SMS), or any other suitable communication protocol.

[0113] The electronic device 300 may be, for example, a system-on-a-chip (SOC) or a device including the SOC. For instance, it can be any device such as a mobile phone, tablet computer, laptop computer, e-reader, game console, television, digital photo frame, navigator, home appliance, communication base station, industrial controller, server, etc., or any combination of data processing devices and hardware. The embodiments of this disclosure do not limit this. The specific functions and technical effects of the electronic device 300 can be found in the above description of the chaos engineering test case generation method and its additional aspects according to at least one embodiment of this disclosure, and will not be repeated here.

[0114] Figure 4 A schematic diagram of a readable storage medium 400 according to at least one embodiment of the present disclosure is shown.

[0115] like Figure 4 As shown, a computer program 410 is stored on a readable storage medium 400, which is a computer-readable storage medium. When the computer program 410 is executed by a processor, it performs one or more steps of the chaos engineering test case generation method and its additional aspects as described above.

[0116] For example, when the program code is read by a computer, the computer can execute the program code stored in the computer storage medium to perform one or more steps to implement, for example, the chaos engineering test case generation method and its additional aspects according to at least one embodiment of the present disclosure.

[0117] For example, the readable storage medium may include a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, and other readable storage media or any combination thereof. The readable storage medium 400 may be a non-transitory readable storage medium.

[0118] At least some of the embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other.

[0119] It should be noted that, in this disclosure, relational terms such as "first," "second," etc., are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved; that is, the preceding or following operations are not necessarily executed precisely in sequence. Instead, various steps may be processed in reverse order or simultaneously as needed. Furthermore, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0121] The units described in the embodiments of this disclosure can be implemented in software or hardware. The described units can also be located in a processor. The names of these units do not, in some cases, constitute a limitation on the unit itself.

[0122] The following points should be noted regarding this disclosure:

[0123] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0124] (2) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0125] The above are merely exemplary embodiments of this disclosure and are not intended to limit the scope of protection of this disclosure, which is determined by the appended claims.

Claims

1. A method for generating test cases for chaotic engineering, characterized in that, The method includes: The seed test case editing module generates multiple chaos engineering test seed test cases, wherein each seed test case includes one or more test parameters; The test case orchestration management library orchestrates the seed test cases according to the test case orchestration rules to generate multiple test cases, wherein the test case orchestration rules include the composition and combination method of the seed test cases; The Chaos Engineering Front-End Probe imports the generated test cases into the blockchain node and the Fuzzy Test Data Mutation Module. The Fuzzy Test Data Mutation Module mutates the test parameters in the test cases according to the mutation rules to generate new test parameters. The chaos engineering front-end probe imports test cases, including the new test parameters, into the blockchain node; The chaos engineering front-end probe collects the operating parameters of the blockchain system where the blockchain node is located, filters problem test cases, and submits them to the test feedback module. The test feedback module extracts the problematic test cases from the feedback to update the test case orchestration rules in the test case orchestration management library.

2. The method according to claim 1, characterized in that, The method further includes: The initial orchestration management module generates seed test case combinations and combination methods based on the known relationships between test cases.

3. The method according to claim 2, characterized in that, The test feedback module extracts the problematic test cases from the feedback to update the test case orchestration rules in the test case orchestration management library, including: Extract the combination composition and combination method of seed test cases from the problem test cases to supplement or replace the test case orchestration rules of the test case orchestration management library.

4. The method according to claim 1, characterized in that, The chaos engineering front-end probe collects the operating parameters of the blockchain system where the blockchain node is located, and filters problem test cases, including: Determine the range of stable operating parameters for the blockchain system; Collect the blockchain system operation parameters after importing multiple sets of test parameters for any of the test cases into the blockchain node; For any set of test parameters, if the collected operating parameters of the blockchain system exceed the range of stable operating parameters of the blockchain system, the set of test parameters is identified as abnormal test parameters; if the number or proportion of abnormal test parameters in the multiple sets of test parameters exceeds a threshold, the test case is filtered and identified as a problem test case.

5. The method according to claim 1, characterized in that, The mutation rules of the fuzz test data mutation module include one or more of the following: The bits of the test parameters are randomly reversed; Perform random arithmetic operations on the integer digits of the test parameters; Randomly select values ​​from a specific list of boundary values ​​to replace some of the values ​​of the test parameters; Randomly select values ​​or strings from the data dictionary to replace part of the values ​​or strings of the test parameters.

6. The method according to claim 5, characterized in that, The process of mutating the test parameters in the test cases to generate new test parameters includes: Identify the core test parameter among the one or more test parameters, and mutate the core test parameter to generate a new test parameter.

7. A chaotic engineering test case generation device, characterized in that, The device includes: The seed test case editing module is configured to generate multiple chaos engineering test seed test cases, wherein each seed test case includes one or more test parameters; The test case orchestration management library is configured to orchestrate the seed test cases to generate multiple test cases according to the test case orchestration rules, wherein the test case orchestration rules include the composition and combination method of the seed test cases; The chaos engineering front-end probe is configured to import the generated test cases into the blockchain node and the fuzz test data mutation module; import test cases including new test parameters into the blockchain node; collect the operating parameters of the blockchain system where the blockchain node is located, filter problematic test cases, and submit them to the test feedback module. The fuzzy test data mutation module is configured to mutate the test parameters in the test cases according to the mutation rules to generate the new test parameters. The test feedback module is configured to extract the problem test cases from the feedback to update the test case orchestration rules of the test case orchestration management library.

8. The apparatus according to claim 7, characterized in that, The device also includes an initial orchestration management module, configured to generate seed test case combinations and combination methods based on the known relationships between test cases.

9. The apparatus according to claim 8, characterized in that, The test feedback module is configured to extract problematic test cases from the feedback to update the test case orchestration rules in the test case orchestration management library, including: Extract the combination composition and combination method of seed test cases from the problem test cases to supplement or replace the test case orchestration rules of the test case orchestration management library.

10. The apparatus according to claim 7, characterized in that, The chaos engineering front-end probe is configured to collect the operating parameters of the blockchain system where the blockchain node resides, and to screen problem test cases, including: Determine the range of stable operating parameters for the blockchain system; Collect the blockchain system operation parameters after importing multiple sets of test parameters for any of the test cases into the blockchain node; For any set of test parameters, if the collected operating parameters of the blockchain system exceed the range of stable operating parameters of the blockchain system, the set of test parameters is identified as abnormal test parameters; if the number or proportion of abnormal test parameters in the multiple sets of test parameters exceeds a threshold, the test case is filtered and identified as a problem test case.

11. The apparatus according to claim 7, characterized in that, The mutation rules of the fuzz test data mutation module include one or more of the following: The bits of the test parameters are randomly reversed; Perform random arithmetic operations on the integer digits of the test parameters; Randomly select values ​​from a specific list of boundary values ​​to replace some of the values ​​of the test parameters; Randomly select values ​​or strings from the data dictionary to replace part of the values ​​or strings of the test parameters.

12. The apparatus according to claim 11, characterized in that, The fuzzy test data mutation module determines the core test parameter among the one or more test parameters, mutates the core test parameter, and generates new test parameters.

13. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

14. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

Images