Fault simulation method and system, and distributed system testing method

Abstract

The present application discloses a fault simulation method and system. The method includes: obtaining complex fault configuration information; the complex fault configuration information includes complex fault component configuration information and its logical relationships; creating a complex fault injection task based on the complex fault component configuration information and its logical relationships included in the complex fault configuration information, the complex fault injection task including fault injection subtasks and their execution sequence; calling the fault injection subtasks based on the execution sequence and establishing an executable workflow object including the fault injection subtasks; executing the workflow object, calling the execution components required by the fault injection subtasks to perform corresponding operations of the complex fault components, and thus injecting complex faults into the system under test.

Description

Technical Field

[0001] The present invention relates to the field of testing technology, and in particular to a fault simulation method and system, and a distributed system testing method. Background Art

[0002] Failures are inevitable during software system operation. With the rise of distributed systems and microservices, system scale is growing, the call relationships and timings between system applications are becoming more complex, and the scenarios in which failures occur are also becoming increasingly complex. Therefore, it is often necessary to simulate system failures to promptly identify potential system issues or to replay system failures for timely troubleshooting and ensure normal system operation. To this end, testers are constantly seeking new methods for system robustness testing.

[0003] A common system testing approach currently is chaos engineering: proactively creating controllable failures in distributed system production environments to verify whether the system's fault tolerance meets requirements. Automated testing systems based on chaos engineering can implement fault simulation, fault discovery, and fault recovery processes for single machines, single applications, or local networks within specific scenarios. Some testing systems can autonomously and randomly create failures to uncover more robustness issues within the tested system, and can also artificially trigger single failures through clients or APIs. This simple testing approach, which simulates single failures (such as a single network interface card delay or CPU exhaustion on a single machine), is not suitable for distributed system scenarios.

[0004] Failures in distributed system scenarios are usually complex failures caused by multiple fault causes. A failure is often the result of the superposition of multiple different fault causes occurring on different nodes (which can be distributed nodes such as hosts and switches). These fault causes occur in a certain time sequence relationship (for example, simultaneously, or in a predecessor and successor time sequence relationship).

[0005] In distributed system robustness testing, to observe the performance of the system under test in specific fault scenarios or to simulate and replay historical faults, people often need to simulate complex fault scenarios and control the faults to occur automatically at specific times. Existing automated testing systems based on chaos engineering concepts do not have the ability to simulate complex faults. Therefore, they cannot effectively simulate complex faults in distributed system scenarios and cannot meet the testing requirements of distributed systems. Summary of the Invention

[0006] In view of the above problems, the present invention is proposed to provide a fault simulation method and system, and a distributed system testing method that overcomes the above problems or at least partially solves the above problems.

[0007] An embodiment of the present invention provides a fault simulation method, including:

[0008] Acquire complex fault configuration information; the complex fault configuration information includes complex fault component configuration information and logical relationships thereof;

[0009] Creating a complex fault injection task according to the complex fault component configuration information and the logical relationship thereof included in the complex fault configuration information, wherein the complex fault injection task includes fault injection subtasks and their execution sequence;

[0010] According to the execution sequence, calling the fault injection subtask and establishing an executable workflow object including the fault injection subtask;

[0011] The workflow object is executed, and the execution component required by the fault injection subtask is called to execute the corresponding operation of the complex fault component, thereby injecting the complex fault into the system under test.

[0012] In some optional embodiments, obtaining complex fault configuration information includes:

[0013] According to the fault identifier included in the input fault injection instruction, complex fault configuration information matching the fault identifier stored in the database is queried and acquired.

[0014] In some optional embodiments, creating a complex fault injection task according to the complex fault component configuration information and the logical relationship thereof included in the complex fault configuration information includes:

[0015] Generating a task name of a complex fault injection task according to the complex task name included in the complex fault configuration information;

[0016] Generating a task description of a complex fault injection task according to the complex task description included in the complex fault configuration information;

[0017] According to the complex fault component configuration information and its logical relationship included in the complex fault configuration information, the subtask name, subtask description information, subtask command parameters and subtask execution time of the fault injection subtask in the complex fault injection task are generated respectively; the complex fault component includes simple faults, pre-fault preparation and post-fault processing.

[0018] In some optional embodiments, calling the fault injection subtask according to the execution sequence and establishing an executable workflow object including the fault injection subtask include:

[0019] Periodically scan the fault injection subtasks in the complex fault injection task that are created and are in the callable state. When the callable fault injection subtask is determined to be executable based on its execution time, it is marked as executable.

[0020] The workflow engine is called to periodically scan the fault injection subtasks in the created complex fault injection task that are in the executable state, and add them as executable workflow object fault injection subtasks based on the execution sequence of the fault injection subtasks.

[0021] In some optional embodiments, executing the workflow object and calling the execution component required by the fault injection subtask to perform corresponding operations of the complex fault component to inject the complex fault into the system under test includes:

[0022] The fault injection subtask to be executed is obtained from the workflow object in sequence. For the fault injection subtask to be executed, the corresponding execution component is called according to the command parameters of the fault injection subtask to execute the operations required by the corresponding complex fault component until the fault injection subtask in the workflow object is completed.

[0023] In some optional embodiments, sequentially obtaining the fault injection subtasks to be executed from the workflow object includes:

[0024] A fault injection subtask whose predecessor has been successfully executed and whose execution time has arrived is obtained from the workflow object as the fault injection subtask to be executed currently.

[0025] In some optional embodiments, according to the command parameters of the fault injection subtask, the corresponding execution component is called to perform the operations required by the corresponding complex fault component, including:

[0026] When the currently executed fault injection subtask is a simple fault injection task, based on the subtask command parameters, the local fault simulation plug-in of the target device is called to simulate the corresponding simple fault, and / or the external system device is called to inject the corresponding simple fault on the target device;

[0027] When the currently executed fault injection subtask is a pre-fault preparation task and / or post-fault processing task for a simple fault simulated using a local fault simulation plug-in, the pre-fault preparation action component and / or post-fault processing action component of the target device is called according to the subtask command parameters to perform the corresponding pre-fault preparation and / or post-fault processing operations;

[0028] When the currently executed fault injection subtask is a pre-fault preparation task and / or post-fault processing task for a simple fault injected using an external system device, an action pre-fault preparation operation instruction and / or post-fault processing operation instruction is sent to the corresponding external system device according to the subtask command parameters.

[0029] In some optional embodiments, the above method further includes:

[0030] The input configuration information of the complex fault component is obtained, and complex fault configuration information including the complex fault component configuration information and its logical relationship is generated; the complex fault component includes simple faults, pre-fault preparations and post-fault processing.

[0031] In some optional embodiments, the step of obtaining the input configuration information of the complex fault component includes:

[0032] Obtain the complex fault component name, complex fault component description, command parameters and execution time of the complex fault component;

[0033] The execution time is the time point of the complex fault component or the delay time relative to the predecessor component;

[0034] The command parameters include at least one of a target device, action content, duration, and construction action indicator of a complex fault component.

[0035] An embodiment of the present invention further provides a fault simulation system, comprising:

[0036] Fault injection subsystem for:

[0037] Acquire complex fault configuration information, wherein the complex fault configuration information includes complex fault component configuration information and logical relationships thereof;

[0038] Creating a complex fault injection task according to the complex fault component configuration information and the logical relationship thereof included in the complex fault configuration information, wherein the complex fault injection task includes fault injection subtasks and their execution sequence;

[0039] According to the execution sequence, calling the fault injection subtask and establishing an executable workflow object including the fault injection subtask;

[0040] Executing the workflow object, calling the execution component required by the fault injection subtask to perform corresponding operations of the complex fault component, thereby injecting complex faults into the system under test;

[0041] Workflow engine, used to control the execution of workflow objects.

[0042] In some optional embodiments, the above system further includes:

[0043] The fault arrangement subsystem is used to obtain complex fault component configuration information and arrange the complex fault configuration information according to the complex fault component configuration information and its logical relationship.

[0044] In some optional embodiments, the fault orchestration subsystem includes:

[0045] Complex fault scenario orchestration interface, used to obtain configuration information of complex fault components input by users;

[0046] The fault description management module is used to generate complex fault configuration information including complex fault component configuration information and its logical relationship based on the input complex fault component configuration information, and call the basic data engine to store the compiled complex fault configuration information in the database; the complex fault component includes simple faults, pre-fault preparation and post-fault processing.

[0047] In some optional embodiments, the fault injection subsystem includes:

[0048] Fault injection interface, used to obtain fault injection instructions input by users;

[0049] The fault injection engine is configured to obtain pre-programmed complex fault configuration information based on an input fault injection instruction; create a complex fault injection task including fault injection subtasks and their execution sequence based on the complex fault component configuration information and their logical relationships included in the complex fault configuration information; call executable fault injection subtasks in a polling manner based on the execution sequence to establish an executable workflow object including the executable fault injection subtasks; execute the workflow object, call the execution components required by the fault injection subtasks to perform corresponding operations of the complex fault component, and thereby inject a complex fault into the system under test;

[0050] The execution component is used to perform the corresponding operations of the fault component.

[0051] In some optional embodiments, the fault injection engine includes a fault injection task creation component configured to:

[0052] According to the fault identifier included in the fault injection instruction, calling the basic data engine of the basic framework layer to query and obtain the complex fault configuration information matching the fault identifier stored in the database;

[0053] Generate a task name in a complex fault injection task according to the complex task name included in the complex fault configuration information;

[0054] Generating a task description in a complex fault injection task according to the complex task description included in the complex fault configuration information;

[0055] According to the complex fault component configuration information and its logical relationship included in the complex fault configuration information, the subtask name, subtask description information, subtask command parameters and subtask execution time of the fault injection subtask in the complex fault injection task are generated respectively; the complex task components include simple faults, pre-fault preparation and post-fault processing.

[0056] In some optional embodiments, the fault injection engine includes a fault injection task scheduling component, configured to:

[0057] Periodically scan the fault injection subtasks in the complex fault injection task that are created and are in the callable state. When the callable fault injection subtask is determined to be executable based on its execution time, it is marked as executable.

[0058] The workflow engine is called to periodically scan the fault injection subtasks in the created complex fault injection task that are in the executable state, and add them as fault injection subtasks in the executable workflow object based on the execution sequence of the fault injection subtasks.

[0059] In some optional embodiments, the fault injection engine includes a fault injection action component configured to:

[0060] Under the control of the workflow engine, the fault injection subtask to be executed is obtained from the workflow object in sequence. For the fault injection subtask to be executed, the corresponding execution component is called according to the subtask command parameters to perform the operations required by the corresponding complex fault component until the fault injection subtask in the workflow object is completed.

[0061] An embodiment of the present invention further provides a distributed system testing method, comprising:

[0062] Obtain system test instructions through the test instruction input interface, and display the corresponding fault injection interface to the user according to the test requirements included in the system test instructions;

[0063] Obtain fault injection instructions through the fault injection interface, obtain the arranged complex fault configuration information according to the fault injection instructions, and use the above-mentioned fault simulation method to inject complex faults into the system under test;

[0064] Obtain performance data of the system under test after complex fault injection.

[0065] In some optional embodiments, the above method further includes:

[0066] The complex fault component configuration information is obtained through the complex fault scenario arrangement interface, and the complex fault configuration information is pre-arranged according to the obtained complex fault component configuration information and its logical relationship.

[0067] In some optional embodiments, the above method further includes:

[0068] The test requirements and the obtained performance data of the system under test are provided to the test terminal and displayed to the user through the test result display interface.

[0069] An embodiment of the present invention further provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-mentioned fault simulation method or the above-mentioned distributed system testing method.

[0070] An embodiment of the present invention also provides a computer device, comprising: a memory, a processor, and a computer program and instructions stored in the memory and executable on the processor; wherein, when the processor executes the program, the above-mentioned fault simulation method or the above-mentioned distributed system testing method is implemented.

[0071] The beneficial effects of the above technical solutions provided by the embodiments of the present invention include at least:

[0072] For the complex faults to be injected, a complex fault injection task including fault injection subtasks and their execution sequence is created according to the complex fault configuration information, and the currently executable fault injection subtask in the complex fault injection task is called based on the execution sequence, and the executable fault injection subtask is added to the workflow object. Based on the workflow engine, each complex fault injection subtask in the complex fault injection task is executed according to the above timing relationship to inject complex faults into each target device in the distributed system for system testing. This method can realize the automatic arrangement and injection of complex faults, and control the injection timing of each simple fault in the complex fault through the execution sequence, so that multiple simple faults included in the complex fault can be injected into the test system in a timely and orderly manner. This method has the fault simulation capability of complex fault scenarios, can well simulate the complex faults in distributed system scenarios, and meet the testing requirements of distributed systems.

[0073] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purposes and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0074] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0076] Figure 1 This is a flowchart of a fault simulation method in Embodiment 1 of the present invention;

[0077] Figure 2 This is a schematic diagram of the structure of a fault simulation system in Embodiment 1 of the present invention;

[0078] Figure 3 This is a diagram illustrating a specific architecture of a fault simulation system in the second embodiment of the present invention;

[0079] Figure 4 This is an example diagram of a directed acyclic graph in the second embodiment of the present invention;

[0080] Figure 5 This is an example diagram of arranging complex faults in the second embodiment of the present invention;

[0081] Figure 6 This is an example diagram of a complex fault injection process in the second embodiment of the present invention;

[0082] Figure 7 This is an example diagram of a workflow object in the second embodiment of the present invention;

[0083] Figure 8 This is an example diagram of a workflow object executing a complex fault component operation in the second embodiment of the present invention;

[0084] Figure 9 This is a flow chart of a distributed system testing method in Embodiment 3 of the present invention;

[0085] Figure 10 This is a schematic diagram of the structure of the test system in Example 3 of the present invention. DETAILED DESCRIPTION

[0086] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0087] A distributed system refers to a software system built on a network. In a distributed system, a group of computer devices presents a unified whole to the user.

[0088] Chaos engineering is a system robustness testing concept that actively creates controllable failures in a distributed system production environment to verify whether the system's fault tolerance meets requirements or to obtain unknown knowledge about the robustness of the distributed system.

[0089] In order to solve the problem in the existing technology that the automated testing system based on chaos engineering does not have the ability to simulate complex faults and cannot meet the testing requirements of distributed systems, an embodiment of the present invention provides a fault simulation method, which arranges complex faults based on logical relationships and forms complex fault injection tasks based on timing relationships. Task scheduling is performed through polling and task execution is implemented based on workflow technology. It can well simulate complex faults in distributed system scenarios and meet the testing requirements of distributed systems.

[0090] Example 1

[0091] The first embodiment of the present invention provides a fault simulation method that can be used for testing distributed systems. By simulating faults caused by various reasons on various devices in the distributed system, complex faults can be injected into the distributed system and the scene when the fault occurs can be simulated and replayed. The process of the method is as follows: Figure 1 As shown, the following steps are included:

[0092] Step S101: Acquire complex fault configuration information.

[0093] In this step, when a complex task needs to be injected into the system under test, the programmed complex fault configuration information is obtained according to the input fault injection instruction. The obtained complex fault configuration information includes complex fault component configuration information and its logical relationship.

[0094] The system under test can be a distributed system. Complex fault configuration information can be obtained through the fault simulation system to realize the injection of complex faults. The fault simulation system can provide a human-computer interaction interface through the test terminal, such as a fault injection interface and a complex fault scenario orchestration interface, to realize the acquisition of fault injection instructions, complex fault configuration information, etc.

[0095] Complex faults can be pre-arranged and the obtained complex fault configuration information can be stored in a database. When a fault injection instruction input by the user is received, the complex fault configuration information matching the fault identifier stored in the database is queried and obtained based on the fault identifier included in the input fault injection instruction.

[0096] A complex fault typically refers to a fault caused by multiple factors and consisting of multiple simple faults. These simple faults can occur in a specific temporal relationship, such as simultaneously, as a predecessor, or as a successor. To simulate complex faults, pre-fault preparation and post-fault processing must be performed for the simple faults within the complex fault to be injected. Therefore, a complex fault injection typically involves multiple fault injection subtasks, each responsible for injecting different simple faults, performing pre-fault preparation, and performing post-fault processing.

[0097] Complex fault configuration information is pre-arranged for complex fault injection. Because complex faults include multiple fault injection subtasks, each fault injection subtask corresponds to a complex fault component. Complex fault components include simple faults, pre-fault preparation, and post-fault processing. Therefore, the corresponding complex fault configuration information includes the configuration information for multiple complex fault components. To ensure that each complex fault subtask executes in the correct sequence, the complex fault configuration information also includes the logical relationships between these complex fault components, such as the execution time and order.

[0098] Step S102: Create a complex fault injection task based on the complex fault component configuration information and its logical relationship included in the complex fault configuration information, wherein the complex fault injection task includes fault injection subtasks and their execution sequence.

[0099] Create a complex fault injection task based on the complex fault configuration information to inject complex faults. This can be achieved through the fault simulation system. The created complex fault injection task can include the task name, task description, configuration information of each complex fault component and its logical relationship. Specifically, it can include:

[0100] Generate a task name in a complex fault injection task according to the complex task name included in the complex fault configuration information;

[0101] Generate a task description in a complex fault injection task according to the complex task description included in the complex fault configuration information;

[0102] Based on the complex fault configuration information, including the complex fault component configuration information and its logical relationships, the subtask name, subtask description, subtask command parameters, and subtask execution time of each fault injection subtask in the complex fault injection task are generated. Complex fault components include simple faults, pre-fault preparation, and post-fault processing. Each complex fault component can create a corresponding fault injection subtask.

[0103] For example, the task name of a complex fault injection task may be Complex Fault 1, and the task description may describe the relevant content of Complex Fault 1, such as testing system performance during network card delay and CPU exhaustion. The configuration information of Complex Fault Component 1 may include the configuration information of Complex Fault Injection Subtask 1, including: the subtask name is Subtask 1, the subtask description is Network Card Delay, subtask command parameters may include delaying the network card of device A and the delay time, the network card of device B and the delay time, etc., the subtask execution time may be immediate or after a period of time, etc.; the configuration information of Complex Fault Injection Subtask 2 may also be included, including: the subtask name is Subtask 2, the subtask description is CPU exhaustion, subtask command parameters may include CPU exhaustion of device A and device B, etc., the subtask execution time may be immediate or after a period of time, etc. The subtask command parameters vary for different fault injection subtasks and are set according to their respective circumstances.

[0104] Step S103: According to the execution sequence, the fault injection subtask is called, and an executable workflow object including the fault injection subtask is established.

[0105] In this step, the complex fault injection tasks created are periodically scanned for fault injection subtasks that are in a callable state. When the callable fault injection subtask is determined to be executable based on its execution time, it is marked as executable. The workflow engine is called to periodically scan the complex fault injection tasks created for fault injection subtasks that are in an executable state. Based on the execution sequence of the fault injection subtask, it is added as a fault injection subtask in the executable workflow object, which can also be called a subtask node.

[0106] Polling can be used to check whether the execution time of each fault injection subtask in a complex fault injection task meets the requirements. The fault injection subtasks can then be scheduled based on the execution time, creating an executable workflow object. Whether the execution time meets the requirements can be determined based on the execution sequence. The execution sequence can include the execution time of the fault injection subtask. The execution time meets the requirements if it reaches or is no later than the execution time. The execution sequence can also include predecessor or successor subtasks of the fault injection subtask. The execution time meets the requirements if all predecessor subtasks have been executed.

[0107] Continuing with the previous example, the execution order of the two fault injection subtasks, NIC delay and CPU exhaustion, is to execute the CPU exhaustion subtask after the NIC delay subtask is completed. In this case, the two subtasks are executed in a round-robin manner. Based on the execution time, the NIC delay subtask is scheduled first and then the CPU exhaustion subtask is scheduled. An executable workflow object is created, which includes the execution sequence of the two subtasks.

[0108] Step S104: Execute the established executable workflow object, call the execution component required by the fault injection subtask to execute the corresponding operation of the complex fault component, and realize the injection of complex faults into the system under test.

[0109] The fault injection subtasks currently to be executed are sequentially obtained from the execution workflow object. For each fault injection subtask currently to be executed, the corresponding execution component is called according to the command parameters of the fault injection subtask to perform the operations required by the corresponding fault component. This process continues until the fault injection subtasks in the workflow object are completed, completing the complex fault injection into the system under test. When executing the fault injection subtasks in the workflow object, they are executed sequentially according to the predecessor and successor relationships between subtasks. After all fault injection subtasks in the complex fault injection task have been executed, the complex fault injection is considered complete. Complex fault injection is achieved by injecting the fault components of the complex fault in a timed manner.

[0110] Sequentially acquiring the fault injection subtasks to be executed currently from the workflow object, including: acquiring from the workflow object a fault injection subtask whose predecessor fault injection subtask has been successfully executed and whose execution time has arrived, as the fault injection subtask to be executed currently.

[0111] If the current fault injection subtask fails or encounters an exception, all subsequent fault injection subtasks will be stopped. This means that the fault injection subtasks in a workflow object will only execute if the above requirements are met. If failures or exceptions occur, you can troubleshoot the causes of the failure and exception. If the current fault injection subtask successfully executes after troubleshooting, you can continue executing subsequent fault injection subtasks in the workflow object.

[0112] In practical applications, a fault injection subtask without a predecessor fault injection subtask can be regarded as a special case in which the predecessor fault injection subtask has been successfully executed.

[0113] Continuing with the previous example, we first execute the network card delay subtask in the workflow object, and then execute the CPU exhaustion subtask after completion. After both subtasks are completed, the injection of complex fault 1 is completed.

[0114] Based on the command parameters of the fault injection subtask, the corresponding execution component is called to perform the operations required by the corresponding fault component. The corresponding operations can be performed locally on the target device or by calling external system devices according to the needs of the subtask, including:

[0115] When the currently executed fault injection subtask is a simple fault injection task, based on the command parameters of the subtask, the local fault simulation plug-in of the target device is called to simulate the corresponding simple fault, and / or the external system device is called to inject the corresponding simple fault on the target device;

[0116] When the currently executed fault injection subtask is a pre-fault preparation task and / or post-fault processing task for a simple fault simulated using a local fault simulation plug-in, the pre-fault preparation action component and / or post-fault processing action component of the target device is called according to the command parameters of the subtask to perform the corresponding pre-fault preparation and / or post-fault processing operations;

[0117] When the currently executed fault injection subtask is a pre-fault preparation task and / or post-fault processing task for a simple fault injected using an external system device, the pre-fault preparation operation instruction and / or post-fault processing operation instruction is sent to the corresponding external device according to the command parameters of the subtask.

[0118] The above-mentioned fault simulation method also includes the arrangement of complex fault configuration information, which includes: obtaining the input configuration information of each complex fault component, generating complex fault configuration information including complex fault component configuration information and its logical relationship; complex fault components include simple faults, pre-fault preparation and post-fault processing.

[0119] The step of obtaining the configuration information of each complex fault component includes: obtaining the name of the complex fault component, the description of the complex fault component, the command parameters of the complex fault component, and the execution time;

[0120] The execution time is the time point of the complex fault component or the delay duration relative to the predecessor component;

[0121] The command parameters include at least one of a target device, action content, duration, and construction action indicator of the complex fault component.

[0122] Based on the same inventive concept, the embodiment of the present invention further provides a fault simulation system, the structure of which is as follows: Figure 2 As shown, it includes a fault injection subsystem 22 and a workflow engine 23;

[0123] The fault injection subsystem 22 is configured to obtain complex fault configuration information, including complex fault component configuration information and its logical relationships; create a complex fault injection task based on the complex fault component configuration information and its logical relationships included in the complex fault configuration information, the complex fault injection task including fault injection subtasks and their execution sequence; call the fault injection subtasks according to the execution sequence, establish an executable workflow object including the fault injection subtasks; execute the workflow object, call the execution components required by the fault injection subtasks to perform corresponding operations of the complex fault components, and thus inject complex faults into the system under test;

[0124] The workflow engine 23 is used to control the execution of workflow objects.

[0125] The above system further includes: a fault arrangement subsystem 21, which is used to obtain complex fault component configuration information and arrange the complex fault configuration information according to the complex fault component configuration information and its logical relationship.

[0126] The fault orchestration subsystem 21 is specifically used to obtain the configuration information of each complex fault component input through the complex fault scenario orchestration interface, generate complex fault configuration information including the complex fault component configuration information and its logical relationship, and call the basic data engine to store the orchestrated complex fault configuration information in the database; the complex fault components include simple faults, pre-fault preparations and post-fault processing.

[0127] The fault injection subsystem 22 is used to obtain the arranged complex fault configuration information based on the input fault injection instruction, including: the fault injection engine in the fault injection subsystem obtains the input fault injection instruction through the fault injection interface, and based on the fault identifier included in the fault injection instruction, calls the basic data engine of the basic framework layer to query and obtain the complex fault configuration information stored in the database that matches the fault identifier.

[0128] The fault injection subsystem 22 is used to create a complex fault injection task including fault injection subtasks and their execution timing based on the complex fault component configuration information and the logical relationship thereof included in the complex fault configuration information, including: the fault injection engine in the fault injection subsystem generates a task name in the complex fault injection task based on the complex task name included in the complex fault configuration information; generates a task description in the complex fault injection task based on the complex task description included in the complex fault configuration information; and generates a subtask name, subtask description information, subtask command parameters and subtask execution time of the fault injection subtasks in the complex fault injection task based on the complex fault component configuration information and the logical relationship thereof included in the complex fault configuration information.

[0129] The fault injection subsystem 22 is configured to call the executable fault injection subtasks in a polling manner according to the execution sequence, and establish an executable workflow object including the executable fault injection subtasks, including:

[0130] The fault injection engine in the fault injection subsystem periodically scans the created complex fault injection tasks for fault injection subtasks in a callable state. When the callable fault injection subtask is determined to be executable based on its execution time, it is marked as executable.

[0131] The workflow engine is called to periodically scan the fault injection subtasks in the created complex fault injection task that are in the executable state, and add them as fault injection subtasks in the executable workflow object based on the execution sequence of the fault injection subtasks.

[0132] The fault injection subsystem 22 is used to execute workflow objects, call the action execution components required by each fault injection subtask to perform the corresponding operations of each fault component, and inject complex faults into the system under test, including:

[0133] Under the control of the workflow engine, the fault injection engine in the fault injection subsystem sequentially obtains the fault injection subtask to be executed from the workflow object. For the fault injection subtask to be executed, the corresponding execution component is called according to the subtask command parameters to perform the operations required by the corresponding fault component until the fault injection subtask in the workflow object is executed, completing the complex fault injection of the system under test.

[0134] Example 2

[0135] The second embodiment of the present invention provides a specific implementation architecture of the above fault simulation system. Figure 3 As shown, the overall structure of the system is divided into three layers: user interface layer, work logic layer and basic framework layer.

[0136] The user interface layer provides an operation interface for users of the system and interacts with the work logic layer via API. The user interface layer provides an operation interface for users, obtains user input through the operation interface, and calls the corresponding functional modules of the work logic layer through API, passing the obtained input information to the work logic layer.

[0137] The user interface layer also provides users with a complex fault scenario orchestration interface and a fault injection interface. The complex fault scenario orchestration interface retrieves user-edited complex fault configuration information, including the configuration information for each simple fault and their logical relationships. The fault injection interface receives user-entered fault injection instructions, such as which complex fault to inject. Users can select a pre-programmed complex fault by entering or selecting a fault identifier.

[0138] The working logic layer includes a fault description management module for implementing complex fault orchestration and a fault injection engine for implementing complex fault injection, as well as a remote agent and a fault simulation plug-in. It also includes the APIs between the fault description management module and the fault injection engine and the user interface layer, the remote or local call interfaces between the fault description management module and the fault injection engine and the basic framework layer, and the remote call interface between the fault injection engine and the remote agent and the fault simulation plug-in.

[0139] The fault description management module converts the configuration information format of complex faults obtained from the operation interface and provides the compiled complex fault configuration information to the basic framework layer for storage.

[0140] The fault injection engine includes functional modules such as fault injection task creation, fault injection task scheduling, and fault injection action components. The fault injection task creation module creates a complex fault injection task including simple fault injection subtasks and their execution sequence based on the simple fault configuration information and their logical relationships included in the complex fault configuration information; the fault injection task scheduling module calls the currently executable simple fault injection subtasks through polling based on the execution sequence; the fault injection action component can call the workflow engine and call remote agents and fault simulation plug-ins, etc. The workflow engine can establish an executable workflow object including executable simple fault injection subtasks based on the execution sequence and control the execution of the workflow object. The fault injection action component executes the workflow object to call the fault simulation device corresponding to each simple fault injection subtask to inject each simple fault included in the complex fault into the system under test. The fault simulation device can be a target machine or other external system device. The target machine can be equipped with a remote agent and a fault simulation plug-in to implement the call during fault injection.

[0141] The basic framework layer includes functional modules such as the basic data engine, external system calls, and workflow engines. It also includes a call interface with the database and a remote or local call interface with the work logic layer.

[0142] The basic data engine can call the database to store the configuration information of the arranged complex faults in the database, and call the configuration information of the arranged complex faults from the database.

[0143] The external system call module implements the calling function between external systems. For example, when it is necessary to call the equipment in the external system to implement fault action, an external system call can be made.

[0144] The workflow engine implements the creation of workflow objects and controls the execution of workflow objects.

[0145] The above is a description of the fault simulation system provided by the embodiment of the present invention from the perspective of system architecture. The system can be divided into three subsystems based on the implemented functions: a fault orchestration subsystem, a fault injection subsystem, and a workflow engine.

[0146] The fault orchestration subsystem includes a complex fault scenario orchestration interface at the user interface level, a fault description management module at the work logic level, and the underlying framework modules it relies on, such as the basic data engine. The fault injection subsystem includes a fault injection interface at the user interface level and a fault injection engine at the work logic level, along with the underlying framework modules it relies on, such as the workflow engine, basic data engine, and some external system calls. The fault injection subsystem may also include a remote agent, fault simulation plug-in, and external system call modules. The fault injection subsystem, in conjunction with the workflow engine, implements complex fault injection.

[0147] like Figure 3 As shown, the fault orchestration subsystem includes a complex fault scenario orchestration interface and a fault description management module.

[0148] The complex fault scenario orchestration interface is used to obtain the configuration information of complex fault components entered by the user.

[0149] The fault description management module is used to generate complex fault configuration information including complex fault component configuration information and its logical relationships based on the input complex fault component configuration information, and call the basic data engine to store the compiled complex fault configuration information in the database; complex fault components include simple faults, pre-fault preparation and post-fault processing.

[0150] The fault injection subsystem includes a fault injection interface and a fault injection engine, and may also include an execution component.

[0151] The fault injection interface is used to obtain fault injection instructions input by the user.

[0152] The fault injection engine is configured to obtain pre-programmed complex fault configuration information based on input fault injection instructions; create a complex fault injection task including fault injection subtasks and their execution sequence based on the complex fault component configuration information and their logical relationships included in the complex fault configuration information; call executable fault injection subtasks in a polling manner based on the execution sequence to establish an executable workflow object including the executable fault injection subtasks; execute the workflow object, call the execution components required by the fault injection subtasks to perform corresponding operations of the complex fault component, and thus inject complex faults into the system under test.

[0153] The execution component is used to execute the corresponding actions of the complex fault component. The execution component can be a local fault simulation plug-in or other action execution component of the target device, or it can be an external system device.

[0154] The fault injection engine may include a fault injection task creation component, a fault injection task scheduling component, and a fault injection action component.

[0155] The fault injection task creation component is used to obtain the arranged complex fault configuration information according to the input fault injection instruction, and create a complex fault injection task including fault injection subtasks and their execution sequence according to the complex fault component configuration information and their logical relationships included in the complex fault configuration information.

[0156] The above fault injection task creates components, specifically for:

[0157] According to the fault identifier included in the fault injection instruction, the basic data engine of the basic framework layer is called to query and obtain the complex fault configuration information matching the fault identifier stored in the database;

[0158] Generate a task name in a complex fault injection task according to the complex task name included in the complex fault configuration information;

[0159] Generate a task description in a complex fault injection task according to the complex task description included in the complex fault configuration information;

[0160] According to the complex fault component configuration information and its logical relationship included in the complex fault configuration information, the subtask name, subtask description information, subtask command parameters and subtask execution time of the fault injection subtask in the complex fault injection task are generated respectively; the complex fault component includes simple faults, pre-fault preparation and post-fault processing.

[0161] The fault injection task scheduling component is used to call the executable fault injection subtasks in a polling manner according to the execution sequence, and establish an executable workflow object including the executable fault injection subtasks.

[0162] The above-mentioned fault injection task scheduling component is specifically used to: periodically scan the fault injection subtasks in the created complex fault injection tasks that are in the callable state, and when it is determined that the callable fault injection subtask is executable based on the execution time of the callable fault injection subtask, mark it as executable state; call the workflow engine, periodically scan the fault injection subtasks in the created complex fault injection tasks that are in the executable state, and add them as fault injection subtasks in the executable workflow object based on the execution sequence of the fault injection subtasks.

[0163] The fault injection action component is used to execute the workflow object, call the action execution component required by the fault injection subtask to perform the corresponding operations of the complex fault component, and realize the injection of complex faults into the system under test.

[0164] The above-mentioned fault injection action component is specifically used to: under the control of the workflow engine, sequentially obtain the fault injection subtasks to be executed from the workflow object, and for the fault injection subtask to be executed, call the corresponding execution component according to the subtask command parameters to perform the operations required by the corresponding fault component until all fault injection subtasks in the workflow object are executed, thereby completing the complex fault injection of the system under test.

[0165] The following describes in detail the specific implementation process of the above-mentioned fault simulation system to implement complex fault arrangement and complex fault injection.

[0166] The fault orchestration subsystem implements the orchestration of complex faults, obtains the input configuration information of each simple fault, and generates complex fault configuration information including simple fault configuration information and its logical relationships.

[0167] Users can edit the components of a complex fault scenario and their relationships through the user interface. These components are referred to as fault scenario components, including the fault cause (referred to as a simple fault) and other components (pre-fault preparation and post-fault cleanup). Relationships between components within a complex fault scenario include parallel, predecessor, and successor. Specifically, a simple fault can be considered a complex fault with a single cause. The fault description management component allows for the addition, deletion, modification, and querying of complex fault description data.

[0168] When orchestrating complex faults, the dependencies between complex fault components can be described using a directed acyclic graph (DAG). A directed acyclic graph (DAG) consists of nodes and edges. It is a graph in which any edge has a direction and there are no loops. If an edge points from node A to node B, then A is called the predecessor of B and B is the successor of A. An example of a directed acyclic graph is Figure 4 As shown, it includes four nodes: preparation before abnormal injection, simple fault 1, simple fault 2, and cleanup after abnormal injection. Among them, the predecessor nodes of simple fault 1 and simple fault 2 are preparation before abnormal injection, and the successor nodes are cleanup after abnormal injection.

[0169] Examples of orchestrating complex failures include Figure 5 As shown in the figure, the complex fault scenario orchestration interface consists of two parts: the directed acyclic graph display part and the parameter configuration part. The directed acyclic graph display part can display and edit the directed acyclic graph, starting from the START node, continuing to pre-fault preparation, simple fault 1 and simple fault 2, post-fault cleanup, and ending at the END node. The nodes in the directed acyclic graph can be edited, and the nodes can be added, deleted, and modified. The parameter configuration part can configure the parameters of the complex fault component corresponding to each node, that is, enter the configuration information of the complex fault component. As shown in the figure, the complex fault scenario orchestration interface consists of two parts: the directed acyclic graph display part and the parameter configuration part. Figure 5 As shown, when simple fault 2 is clicked, the name, description, parameter 1, parameter 2, parameter 3, parameter 4, etc. of simple fault 2 can be entered. The input parameters can be the target device for fault injection, the delay time relative to the predecessor component (action), or other parameters.

[0170] Users edit graphically through the interface (displayed as a directed acyclic graph). The graph and complex fault configuration information (i.e., complex fault description) are converted to each other through the fault description management module of the working logic layer. The graph is converted into a complex fault description to formalize complex fault scenarios and facilitate automated processing. When converting a graph into a complex fault description, the names in the parameters of complex fault components (such as simple faults) are converted into descriptive information in the complex fault description through index queries. An example of a graph converted into complex fault configuration information (i.e., complex fault description) is as follows:

[0171] name:Name Of The Task

[0172] description:Description Of The Task

[0173] actions

[0174] -name:Action1

[0175] description:Description Of Action1

[0176] next:[Action2, Action3]

[0177] command:Command1

[0178] -name:Action2

[0179] description:Description Of Action2

[0180] next:[Action4]

[0181] command:Command2

[0182] -name:Action3

[0183] description:Description Of Action3

[0184] next:[Action4]

[0185] command: Command3

[0186] -name:Action4

[0187] description:Description Of Action4

[0188] next:[NULL]

[0189] command:Command4

[0190] The name in the first line of the complex fault configuration information is the name of the complex fault injection task, the description in the second line is the description information of the complex fault injection task, and the multiple actions that follow, each action corresponds to a complex fault component. Each complex fault component includes a name (name), description information (description), command parameters (command) and execution time (such as subsequent action).

[0191] In complex fault configuration information, variable parameters are specified with capital letters, while fixed parameters are specified in lowercase letters. Actions are used to describe complex fault components in complex fault scenarios, and the command in each action represents the command to be executed. For example, when the action is a simple fault, the command parameters may include: a) the target device for fault injection, which can be represented by the host IP address, network switch device ID, etc.; b) the delay time relative to the preceding action; and additional command parameters may also be included, such as those controlling the fault duration and fault severity.

[0192] The fault injection subsystem implements complex fault injection and can execute fault injection tasks based on user-provided parameters. This subsystem works in conjunction with the workflow engine. Because the execution of a complex fault injection task can be divided into multiple steps, each step can perform an operation corresponding to a complex fault component. These steps can also become fault injection subtasks. These fault injection subtasks can have predecessor, successor, or parallel temporal relationships. The workflow engine enables the execution of subtasks based on the temporal relationships of fault injection subtasks during system runtime.

[0193] The complex fault injection process of distributed systems can be found in Figure 6 As shown in the figure, solid arrows indicate data flow, dashed arrows indicate control flow, dashed boxes indicate task states, and solid boxes indicate system components. The process of complex fault injection is described as follows:

[0194] The user initiates a fault injection task through the fault injection interface.

[0195] The user initiates a task by specifying the ID of the complex fault configuration information to be injected through the UI. The complex fault configuration information exists in the database, with the ID as the primary key. After entering the ID, the corresponding complex fault configuration information can be found based on the ID. The user interface can also be used to call the API to pass task parameters, such as the user-specified ID.

[0196] The fault injection task creation component creates complex fault injection tasks based on the relevant parameters in the complex fault configuration information entered by the user.

[0197] The parameters in the complex fault configuration information may include: a) the ID of the complex fault configuration information; b) the execution time. The execution time can be a scheduled execution time, an execution time represented by a predecessor or successor relationship, or it can be set to empty. In other words, the execution time parameter can be configured selectively. If it is configured not to specify this parameter, it means immediate execution. If it is specified, it will be executed at the specified time.

[0198] like Figure 6As shown, the output of task creation is the task description of the created complex fault injection task. This may include the complex task name, complex task description information, and complex fault component configuration information shown in the above example. It may also include the runtime status of each fault injection subtask (such as the task start time and current task status). The created complex fault injection task description can be stored in the database, and the task runtime status can be set to created, indicating that it is callable.

[0199] The fault injection task scheduling component performs the scheduling task function and can call the created complex fault injection tasks stored in the database. The task scheduling execution process includes: the fault injection task scheduling component periodically scans the callable complex fault injection tasks, such as tasks with the status of created. The period can be set, for example, it can be set to 1 second. In a round of scanning, for a task with a status of created, if the user specifies a scheduled execution time, if it is not later than the current time, then the task is marked as executable, otherwise no processing is performed; if the user does not specify an execution time, then the task is directly marked as executable. Set the running status of the executable task to ready (i.e. executable status), and update the modified task description to the database.

[0200] After marking the task as executable, you can call the workflow engine to execute each subtask in the complex fault injection task. The task execution process includes:

[0201] a) The workflow engine periodically scans tasks in the ready state. The period can be set, for example, to 1 second.

[0202] b) For a task in the ready state, according to the actions part in the complex fault configuration information, the description of each complex fault component in the complex fault configuration information and their timing relationship, call the create method of the workflow engine to create an actual executable workflow object.

[0203] See also Figure 7 As shown, the workflow is a directed acyclic graph (DAG). Figure 7 The example workflow object includes 4 subtask nodes, corresponding to Figure 4 The four complex fault components shown in Figure 1 are logically workflow objects, representing an overall task. Each subtask node is an executable piece of code, called a fault injection subtask, which performs simple fault injection, preparation before fault injection, or post-fault cleanup. Each fault injection subtask in the workflow object corresponds to an action in the complex fault configuration. The sequential relationship between fault injection subtasks in the workflow object is consistent with the predecessor relationship between actions in the complex fault configuration.

[0204] c) Call the start method of the workflow object to start executing the workflow. The execution of the workflow object meets the following requirements:

[0205] For any fault injection subtask (except for fault injection subtasks without predecessors), this fault injection subtask can only be executed after all predecessor fault injection subtasks are completed;

[0206] If any fault injection subtask fails to execute or throws a code exception, the execution of all subsequent fault injection subtasks will be stopped;

[0207] For any fault injection subtask, when all the predecessor fault injection subtasks are executed successfully, the current fault injection subtask will be executed after waiting for the corresponding time according to the pre-configured delay time (in the command parameter of the action in the fault description). Figure 6 As shown, if an exception occurs during task execution, the task status is set to failed; otherwise, the task status is set to success. The modified task description is updated in the database.

[0208] Execute the workflow object to execute each fault injection subtask in the workflow object in sequence. The fault injection subtask can correspond to simple fault injection, fault injection preparation and fault injection post-injection cleanup. The execution process of the fault injection subtask can be seen in Figure 8 shown.

[0209] For example, when the fault injection subtask is to perform simple fault injection, there are two ways to implement simple fault injection:

[0210] Method 1: Call the fault simulation plug-in on the target device to simulate a simple fault. In the complex fault configuration information, the command parameter of the action includes the target device to be called and the corresponding fault simulation plug-in information. The fault simulation is directly executed on the target device, such as a single disk space is full, a single machine CPU is exhausted, and a single network card is delayed.

[0211] For the fault injection execution process of the fault simulation plug-in method, see Figure 8 As shown, the fault injection action component in the fault injection engine (located in the node of the workflow object) remotely calls the agent on the target device (target machine), and at the same time passes in the subtask parameters, including the fault name (such as a single disk space is full, a single CPU is exhausted, a single network card delay) and other parameters (such as the fault severity and fault duration). The agent then calls the local fault simulation plug-in.

[0212] Method 2: Call the API of external system devices to indirectly inject faults, such as shutting down a network switch.

[0213] For the fault injection execution process of the external system call method, see Figure 8 As shown, the fault injection action component in the fault injection engine calls the API of the external system (such as the switch management system), and the external system agent performs the specified action.

[0214] The fault injection subtasks are similar to the pre-fault injection preparation and post-fault injection cleanup processes, and are also divided into two cases:

[0215] Case 1: Fault injection preparation and post-fault injection cleanup using a fault simulation plug-in. For example, a single disk space exhaustion fault occurs. Before fault injection, the agent calls the disk selection plug-in to randomly select one of the multiple data disks on the local machine and record the current disk space level (for fault recovery). After fault injection, the agent sends a disk space cleanup instruction to the single disk space level recovery plug-in, with the goal of restoring the disk space level to the pre-fault injection level. Figure 8 As shown, other action components remotely call the agent on the target device (target machine) and pass in subtask parameters. Then the agent calls the local action execution plug-in to perform related actions such as fault injection preparation and post-fault injection cleanup.

[0216] Case 2: Preparation before fault injection and cleanup after fault injection using external system calls. Taking the case of a network switch shutdown fault as an example, before fault injection, the action component sends a command to the switch management system to check whether the switch status is normal and then obtain switch operation permissions; after fault injection, the action execution component sends a network switch startup command to the switch management system. Figure 8 As shown, other action components in the fault injection engine call the API of the external system, and the external system agent performs the specified action.

[0217] Alternatively, fault injection can be performed without using an agent and using SSH or other similar remote command execution methods instead.

[0218] Example 3

[0219] The third embodiment of the present invention provides a distributed system test system and method. The structure of the distributed system test system is as follows: Figure 9 As shown, it includes: a test terminal 1 and a fault simulation system 2.

[0220] Test terminal 1, used to obtain system test instructions through the test instruction input interface, and obtain fault injection instructions through the fault injection interface;

[0221] The fault simulation system 2 is used to display the corresponding fault injection interface to the user according to the test requirements included in the system test instructions; and according to the fault injection instructions, obtain the arranged complex fault configuration information, inject complex faults into the system under test 3, and obtain the performance data of the system under test after the complex fault injection.

[0222] The fault simulation system 2 is further configured to provide the test requirements and the acquired performance data of the system under test to the test terminal; correspondingly, the test terminal 1 is further configured to display the test requirements and performance data to the user through a test result display interface.

[0223] The test terminal 1 can be installed with a test client, which provides the tester with various human-computer interaction interfaces involved in the testing process. For example, a test instruction input interface for obtaining test instructions, through which the user can enter test instructions, such as instructions for starting a test, instructions containing test requirements, etc.; another example: a fault injection interface, through which the user can enter fault injection instructions and fault injection-related information; another example: a complex fault scenario arrangement interface, through which the user can enter complex fault component configuration information and other information related to complex fault arrangement; another example: an information display interface, which can display information such as the test requirements entered by the user and the corresponding test results to the user.

[0224] The structure and function of the fault simulation system 2 refer to the relevant descriptions in the above-mentioned embodiment 1 and embodiment 2, and will not be repeated here.

[0225] Optionally, the above system also includes an external system 4, which is used to call the execution components required by the fault injection subtask according to the calling instructions of the fault simulation system 2, and perform corresponding operations of the fault injection subtask on the corresponding target device in the system under test 3.

[0226] When the complex fault injection task of the system under test 3 requires the external system to implement the injection of one or more fault injection subtasks, the external system 4 can be called by the fault simulation system 2. The fault simulation system 2 can call one or more devices in the external system required by the fault injection subtask according to the needs to implement the simulation injection of the corresponding fault injection subtask.

[0227] like Figure 9 As shown, multiple test terminals 1 can be deployed for use by different testers. Fault simulation system 2 can be implemented on a single server, a server cluster, or even a cloud device. Tested system 3 can include one or more network node devices such as cloud devices, personal computers, servers, mainframes, hosts, and switches. External system 4 can include one or more external system devices such as cloud devices, personal computers, servers, mainframes, and so on.

[0228] The distributed system testing method provided by the embodiment of the present invention has the following process: Figure 10 As shown, the following steps are included:

[0229] Step S201: obtaining a system test instruction through a test instruction input interface, and displaying a corresponding fault injection interface to the user according to the test requirements included in the system test instruction;

[0230] The test instructions can be obtained and the fault injection interface can be displayed through the human-computer interaction interface on the test terminal.

[0231] Step S202: Obtain a fault injection instruction through the fault injection interface, obtain the compiled complex fault configuration information according to the fault injection instruction, and inject the complex fault into the system under test using the above-mentioned fault simulation method;

[0232] Step S203: Acquire performance data of the system under test after complex fault injection.

[0233] After injecting complex faults into the system under test, the various performance indicators of the system under test are monitored and the corresponding performance data is obtained in order to understand the operating status and fault tolerance of the system under test when the injected fault occurs.

[0234] Optionally, the above method further includes:

[0235] Step S204: providing the test requirements and the acquired performance data of the system under test to the test terminal, and displaying them to the user through the test result display interface.

[0236] The information display interface on the test terminal can be used to display to users relevant data on various performance indicators of the tested system after complex fault injection, such as CPU usage, memory usage, system fault information, etc.

[0237] Optionally, the above method further includes: obtaining complex fault component configuration information through a complex fault scenario arrangement interface, and pre-arranging the complex fault configuration information according to the obtained complex fault component configuration information and its logical relationship.

[0238] The contents of the above-mentioned distributed system testing method and system have been described in Embodiment 1 and Embodiment 2 and will not be repeated here.

[0239] An embodiment of the present invention further provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-mentioned fault simulation method or the above-mentioned distributed system testing method.

[0240] An embodiment of the present invention also provides a computer device, comprising: a memory, a processor, and a computer program and instructions stored in the memory and executable on the processor; wherein, when the processor executes the program, the above-mentioned fault simulation method or the above-mentioned distributed system testing method is implemented.

[0241] The above-mentioned method and system of the embodiments of the present invention are a distributed system fault simulation system designed under the guidance of chaos engineering concepts. They are capable of simulating complex faults and feature complex fault orchestration and scheduled occurrence. They can be used in cloud platform services or products such as proprietary clouds and block storage. Leveraging the concept of a directed acyclic graph (DAG), the method and system orchestrates complex faults, ensuring the orderly execution of each component involved in a complex fault. A polling approach is used to check whether time requirements are met to implement the timing of complex fault tasks. Within complex faults, relative time delays are employed to implement the timing of each sub-step, ensuring that complex fault tasks and their included simple tasks or actions are executed on the required timeframes. Complex fault injection, implemented through workflow technology, ensures orderly and rational execution without manual intervention. Leveraging the concepts of agents and agent plug-ins, they implement single-target simple fault injection and unify both agent and external system call approaches for fault injection. This enables automated execution of every step from fault orchestration to fault task creation, execution, and fault injection, enabling the simulation and re-creation of complex faults. When this fault simulation method is used for testing distributed systems, it can greatly improve the automation level of system testing.

[0242] Unless otherwise specifically stated, terms such as process, calculate, compute, determine, display, and the like may refer to the actions and / or processes of one or more processing or computing systems, or similar devices, that manipulate and convert data represented as physical (e.g., electronic) quantities within registers or memories of a processing system into other data similarly represented as physical quantities within the memories, registers, or other such information storage, transmission, or display devices of the processing system. Information and signals may be represented using any of a variety of different techniques and methods. For example, data, instructions, commands, information, signals, bits, symbols, and chips referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, light fields or particles, or any combination thereof.

[0243] It should be understood that the specific order or hierarchy of steps in the disclosed processes is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of the present disclosure. The accompanying method claims present elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0244] In the foregoing detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that embodiments of the claimed subject matter require more features than are expressly recited in each claim. On the contrary, as reflected in the appended claims, the invention comprises less than all the features of any individual disclosed embodiment. The appended claims are therefore hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0245] Those skilled in the art will also appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments herein may be implemented as electronic hardware, computer software, or a combination thereof. In order to clearly illustrate the interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps described above are generally described around their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. A skilled person may implement the described functions in an adaptable manner for each specific application, but such implementation decisions should not be interpreted as departing from the scope of protection of this disclosure.

[0246] The steps of the methods or algorithms described in conjunction with the embodiments herein may be directly embodied as hardware, software modules executed by a processor, or a combination thereof. The software module may be located in a RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, register, hard disk, removable disk, CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is connected to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium may also be an integral part of the processor. The processor and storage medium may be located in an ASIC. The ASIC may be located in a user terminal. Of course, the processor and storage medium may also be present in a user terminal as discrete components.

[0247] For software implementation, the techniques described in this application can be implemented using modules (e.g., procedures, functions, etc.) that perform the functions described in this application. These software codes can be stored in a memory unit and executed by a processor. The memory unit can be implemented within the processor or external to the processor. In the latter case, it is communicatively coupled to the processor via various means, which are well known in the art.

[0248] The foregoing description includes examples of one or more embodiments. Of course, it is not possible to describe all possible combinations of components or methods for the purposes of describing the above embodiments, but one of ordinary skill in the art will recognize that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to encompass all such changes, modifications and variations that fall within the scope of the appended claims. Furthermore, to the extent the term "comprising" is used in the specification or claims, the term is intended to be encompassed in a manner similar to the term "including," as explained in terms of "including," used as a transitional word in the claims. Furthermore, any use of the term "or" in the specification of the claims is intended to mean a "non-exclusive or."

Claims

1. A fault simulation method, comprising: Acquire complex fault configuration information, wherein the complex fault configuration information includes complex fault component configuration information and logical relationships thereof; Creating a complex fault injection task according to the complex fault component configuration information and the logical relationship thereof included in the complex fault configuration information includes: Generating a task name of a complex fault injection task according to the complex task name included in the complex fault configuration information; Generating a task description of a complex fault injection task according to the complex task description included in the complex fault configuration information; Generate subtask names, subtask description information, subtask command parameters, and subtask execution time of fault injection subtasks in a complex fault injection task according to the complex fault component configuration information and its logical relationship included in the complex fault configuration information; the complex fault components include simple faults, pre-fault preparation, and post-fault processing; The complex fault injection task includes fault injection subtasks and their execution sequence; According to the execution sequence, calling the fault injection subtask and establishing an executable workflow object including the fault injection subtask; The workflow object is executed, and the execution component required by the fault injection subtask is called to execute the corresponding operation of the complex fault component, thereby injecting the complex fault into the system under test.

2. The method according to claim 1, wherein The obtaining of complex fault configuration information includes: According to the fault identifier included in the input fault injection instruction, complex fault configuration information matching the fault identifier stored in the database is queried and acquired.

3. The method according to claim 1, wherein According to the execution sequence, the fault injection subtask is called, and an executable workflow object including the fault injection subtask is established, including: Periodically scan the fault injection subtasks in the complex fault injection task that are created and are in the callable state. When the callable fault injection subtask is determined to be executable based on its execution time, it is marked as executable. The workflow engine is called to periodically scan the fault injection subtasks in the created complex fault injection task that are in the executable state, and add them as fault injection subtasks in the executable workflow object based on the execution sequence of the fault injection subtasks.

4. The method according to claim 1, wherein Executing the workflow object, calling the execution components required by the fault injection subtask to perform corresponding operations of the complex fault component, and injecting complex faults into the system under test, including: The fault injection subtask to be executed is obtained from the workflow object in sequence. For the fault injection subtask to be executed, the corresponding execution component is called according to the command parameters of the fault injection subtask to execute the operations required by the corresponding complex fault component until the fault injection subtask in the workflow object is completed.

5. The method according to claim 4, wherein: Obtain the fault injection subtasks currently to be executed from the workflow object in sequence, including: A fault injection subtask whose predecessor has been successfully executed and whose execution time has arrived is obtained from the workflow object as the fault injection subtask to be executed currently.

6. The method of claim 4, wherein: Based on the command parameters of the fault injection subtask, the corresponding execution component is called to perform the operations required by the corresponding complex fault component, including: When the currently executed fault injection subtask is a simple fault injection task, based on the subtask command parameters, the local fault simulation plug-in of the target device is called to simulate the corresponding simple fault, and / or the external system device is called to inject the corresponding simple fault on the target device; When the currently executed fault injection subtask is a pre-fault preparation task and / or post-fault processing task for a simple fault simulated using a local fault simulation plug-in, the pre-fault preparation action component and / or post-fault processing action component of the target device is called according to the subtask command parameters to perform the corresponding pre-fault preparation and / or post-fault processing operations; When the currently executed fault injection subtask is a pre-fault preparation task and / or post-fault processing task for a simple fault injected using an external system device, an action pre-fault preparation operation instruction and / or post-fault processing operation instruction is sent to the corresponding external system device according to the subtask command parameters.

7. The method according to any one of claims 1 to 6, further comprising: Obtaining input configuration information of complex fault components, and generating complex fault configuration information including the complex fault component configuration information and its logical relationship; The complex fault components include simple faults, pre-fault preparations and post-fault processing.

8. The method of claim 7, wherein: The step of obtaining the input configuration information of the complex fault component includes: Obtain the complex fault component name, complex fault component description, command parameters and execution time of the complex fault component; The execution time is the time point of the complex fault component or the delay time relative to the predecessor component; The command parameters include at least one of a target device, action content, duration, and construction action indicator of a complex fault component.

9. A fault simulation system comprising: Fault injection subsystem for: Acquire complex fault configuration information, wherein the complex fault configuration information includes complex fault component configuration information and logical relationships thereof; Creating a complex fault injection task according to the complex fault component configuration information and the logical relationship thereof included in the complex fault configuration information, wherein the complex fault injection task includes fault injection subtasks and their execution sequence; According to the execution sequence, calling the fault injection subtask and establishing an executable workflow object including the fault injection subtask; Executing the workflow object, calling the execution component required by the fault injection subtask to perform corresponding operations of the complex fault component, thereby injecting complex faults into the system under test; The fault orchestration subsystem includes: Complex fault scenario orchestration interface, used to obtain configuration information of complex fault components input by users; The fault description management module is used to generate complex fault configuration information including complex fault component configuration information and its logical relationships based on the input complex fault component configuration information, and call the basic data engine to store the compiled complex fault configuration information in the database; the complex fault component includes simple faults, pre-fault preparation and post-fault processing; The fault injection engine is configured to obtain pre-programmed complex fault configuration information based on an input fault injection instruction; create a complex fault injection task including fault injection subtasks and their execution sequence based on the complex fault component configuration information and their logical relationships included in the complex fault configuration information; call executable fault injection subtasks in a polling manner based on the execution sequence to establish an executable workflow object including the executable fault injection subtasks; execute the workflow object to call the execution components required by the fault injection subtasks to perform corresponding operations on the complex fault component; The fault injection engine includes a fault injection task creation component for: According to the fault identifier included in the fault injection instruction, calling the basic data engine of the basic framework layer to query and obtain the complex fault configuration information matching the fault identifier stored in the database; Generating a task name of a complex fault injection task according to the complex task name included in the complex fault configuration information; Generating a task description of a complex fault injection task according to the complex task description included in the complex fault configuration information; According to the complex fault component configuration information and its logical relationship included in the complex fault configuration information, the subtask name, subtask description information, subtask command parameters and subtask execution time of the fault injection subtask in the complex fault injection task are generated respectively; the complex fault component includes simple fault, pre-fault preparation and post-fault processing A workflow engine is used to control the execution of the workflow object.

10. The system of claim 9: The fault arrangement subsystem is also used to obtain complex fault component configuration information and arrange the complex fault configuration information according to the complex fault component configuration information and its logical relationship.

11. The system of claim 9, wherein: The fault injection subsystem includes: Fault injection interface, used to obtain fault injection instructions input by users; Execution component, used to perform corresponding operations of complex fault components.

12. The system of claim 10, wherein: The fault injection engine includes a fault injection task scheduling component for: Periodically scan the fault injection subtasks in the complex fault injection task that are created and are in the callable state. When the callable fault injection subtask is determined to be executable based on its execution time, it is marked as executable. The workflow engine is called to periodically scan the fault injection subtasks in the created complex fault injection task that are in the executable state, and add them as fault injection subtasks in the executable workflow object based on the execution sequence of the fault injection subtasks.

13. The system of claim 12, wherein the fault injection engine comprises a fault injection action component configured to: Under the control of the workflow engine, the fault injection subtask to be executed is obtained from the workflow object in sequence. For the fault injection subtask to be executed, the corresponding execution component is called according to the subtask command parameters to perform the operations required by the corresponding complex fault component until the fault injection subtask in the workflow object is completed.

14. A distributed system testing method, comprising: Obtain system test instructions through the test instruction input interface, and display the corresponding fault injection interface to the user according to the test requirements included in the system test instructions; Obtain a fault injection instruction through a fault injection interface, obtain the arranged complex fault configuration information according to the fault injection instruction, and inject the complex fault into the system under test using the fault simulation method according to any one of claims 1 to 8; Obtain performance data of the system under test after complex fault injection.

15. The method of claim 14, further comprising: The complex fault component configuration information is obtained through the complex fault scenario arrangement interface, and the complex fault configuration information is pre-arranged according to the obtained complex fault component configuration information and its logical relationship.

16. The method according to claim 14 or 15, further comprising: The test requirements and the obtained performance data of the system under test are provided to the test terminal and displayed to the user through the test result display interface.

17. A computer-readable storage medium having computer instructions stored thereon, wherein when the instructions are executed by a processor, the fault simulation method according to any one of claims 1 to 8 or the distributed system testing method according to any one of claims 14 to 16 is implemented.

18. A computer device comprising: a memory for storing computer program instructions; A processor is configured to execute the computer program instructions to implement the fault simulation method according to any one of claims 1 to 8 or the distributed system testing method according to any one of claims 14 to 16.

Images