Fault injection method, device, storage medium and program product

Abstract

The present invention provides a fault injection method, device, storage medium and program product, which relate to the field of computer technology. The method is applied to a test end, and the method includes: obtaining fault injection configuration parameters and read-write operation instructions corresponding to a target fault scenario, the read-write operation instructions are used to trigger the target fault scenario; sending the fault injection configuration parameters and read-write operation instructions corresponding to the target fault scenario to a setting hardware driver in the operating system kernel of the tested end, so that the setting hardware driver completes the injection of the target fault scenario according to the fault injection configuration parameters corresponding to the target fault scenario during the process of controlling the corresponding setting hardware to execute the read-write operation instructions corresponding to the target fault scenario; obtaining the response information of the operating system kernel to the target fault scenario by setting the hardware driver. This scheme has a good test effect on fault injection.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a fault injection method, device, storage medium and program product. Background Art

[0002] Fault injection is a testing method used to introduce errors or faults into a target system to test the target system's ability to cope with the situation and to see how easily the target system can be damaged and its ability to recover after being disturbed by other factors such as stress, thereby proving the resilience and fragility of the target system.

[0003] With the rapid development of cloud computing technology, Non-Volatile Memory Express (NVMe) is widely used as a high-speed storage interface. NVMe drivers are very important as a bridge between operating systems and NVMe hardware devices. Therefore, in order to ensure the stability of the operating system, it is urgent to propose a test method for efficiently testing the operating system kernel based on NVMe drivers. Summary of the invention

[0004] The embodiment of the present invention provides a fault injection method, a device, a storage medium and a program product, which realizes efficient testing of an operating system kernel through fault injection of a driver.

[0005] In a first aspect, an embodiment of the present invention provides a fault injection method, which is applied to a test end, and the method includes:

[0006] Obtaining fault injection configuration parameters and read / write operation instructions corresponding to a target fault scenario, wherein the read / write operation instructions are used to trigger the target fault scenario;

[0007] Sending the fault injection configuration parameters and read / write operation instructions corresponding to the target fault scenario to the setting hardware driver in the operating system kernel of the tested end, so that the setting hardware driver completes the injection of the target fault scenario according to the fault injection configuration parameters corresponding to the target fault scenario during the process of controlling the corresponding setting hardware to execute the read / write operation instructions corresponding to the target fault scenario;

[0008] The response information of the operating system kernel to the target fault scenario is obtained through the set hardware driver.

[0009] In a second aspect, an embodiment of the present invention provides a fault injection device, applied to a test end, the device comprising:

[0010] An acquisition module, used to acquire fault injection configuration parameters and read / write operation instructions corresponding to a target fault scenario, wherein the read / write operation instructions are used to trigger the target fault scenario;

[0011] A sending module, used to send the fault injection configuration parameters and read / write operation instructions corresponding to the target fault scenario to a setting hardware driver in the operating system kernel of the tested end, so that the setting hardware driver completes the injection of the target fault scenario according to the fault injection configuration parameters corresponding to the target fault scenario in the process of controlling the corresponding setting hardware to execute the read / write operation instructions corresponding to the target fault scenario;

[0012] A response module is used to obtain response information of the operating system kernel to the target fault scenario through the set hardware driver.

[0013] In a third aspect, an embodiment of the present invention provides a fault injection method, which is applied to a setting hardware driver in an operating system kernel of a tested end, and the method includes:

[0014] Receive the fault injection configuration parameters and read / write operation instructions corresponding to the target fault scenario sent by the test end;

[0015] In the process of controlling the corresponding setting hardware to execute the read and write operation instructions, the injection of the target fault scenario is completed based on the fault injection configuration parameters;

[0016] Sending response information of the operating system kernel to the target fault scenario to the test end.

[0017] In a fourth aspect, an embodiment of the present invention provides a fault injection device, which is applied to a setting hardware driver in an operating system kernel of a tested end, and the device includes:

[0018] A receiving module, used to receive fault injection configuration parameters and read and write operation instructions corresponding to the target fault scenario sent by the test end;

[0019] A control module, used for controlling the corresponding setting hardware to complete the injection of the target fault scenario based on the fault injection configuration parameters during the process of executing the read and write operation instructions;

[0020] A sending module is used to send the response information of the operating system kernel to the target fault scenario to the test end.

[0021] In a fifth aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the fault injection method as described in the first aspect or the third aspect.

[0022] In a sixth aspect, an embodiment of the present invention provides a non-temporary machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor executes the fault injection method as described in the first aspect or the third aspect.

[0023] In a seventh aspect, an embodiment of the present invention provides a computer program product, comprising: a computer program, which, when executed by a processor of an electronic device, enables the processor to execute the fault injection method as described in the first aspect or the third aspect.

[0024] The fault injection method provided by the embodiment of the present invention pregenerates a fault scenario library, wherein different fault scenarios are defined by different fault injection configuration parameters and corresponding read-write operation instructions. When the test end performs a fault injection test on the operating system kernel each time, the fault injection configuration parameters and read-write operation instructions corresponding to the target fault scenario are obtained from the fault scenario library, and the fault injection configuration parameters and read-write operation instructions are sent to the setting hardware driver (such as ‌NVMe driver) in the operating system kernel of the tested end, thereby realizing the automatic injection of fault parameters related to the target fault scenario into the tested end, improving work efficiency, and laying the foundation for the smooth progress of subsequent fault injection. After receiving the fault injection configuration parameters and read-write operation instructions corresponding to the target fault scenario, the tested end can enable the setting hardware driver to complete the injection of the target fault scenario according to the fault injection configuration parameters corresponding to the target fault scenario in the process of controlling the corresponding setting hardware (such as a certain ‌NVMe hardware) to execute the read-write operation instructions corresponding to the target fault scenario, thereby realizing the simulation of the target fault scenario during the operation of the operating system. Afterwards, by setting the hardware driver to obtain the response information of the operating system kernel to the target fault scenario, the reaction of the operating system kernel after the fault injection of the target fault scenario by setting the hardware driver can be obtained, and the fault injection test is completed. Moreover, the entire fault injection test process uses the hardware driver as the medium. By using the set fault scenario library, a variety of fault scenarios can be covered, and the test effect is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 A schematic diagram of the structure of a fault injection system provided by an embodiment of the present invention;

[0027] Figure 2 A flowchart of a fault injection method provided by an embodiment of the present invention;

[0028] Figure 3 A flowchart of a fault injection method provided by an embodiment of the present invention;

[0029] Figure 4 A schematic diagram of an application of a fault injection method provided by an embodiment of the present invention;

[0030] Figure 5 A flowchart of another fault injection method provided by an embodiment of the present invention;

[0031] Figure 6 A schematic diagram of the structure of a fault injection device provided by an embodiment of the present invention;

[0032] Figure 7 A schematic diagram of the structure of another fault injection device provided by an embodiment of the present invention;

[0033] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0036] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the case where there is no conflict between the embodiments, the following embodiments and the features in the embodiments can be combined with each other. In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0037] First, the terms or concepts involved in the embodiments of the present invention are explained:

[0038] Fault injection: is a testing method used to introduce errors or faults into the target system to test the target system's ability to cope with the faults, and to see how easily the target system can be damaged and its ability to recover after being disturbed by other factors such as stress, thereby proving the resilience and fragility of the target system.

[0039] Non-Volatile Memory Express (NVMe): is a high-performance, low-latency storage protocol and interface standard designed for non-volatile storage media (such as solid-state drives).

[0040] The embodiment of the present invention provides the following ideas to complete the test problem for the operating system kernel: First, the fault injection configuration parameters and read-write operation instructions corresponding to the target fault scenario are obtained to realize the simulation of the required fault scenario. Secondly, since the operating system kernel will interact with some hardware during the working process, and the interaction is realized through the driver corresponding to the hardware, the results of the hardware executed under the read-write operation instructions of the operating system kernel will also be fed back to the operating system kernel through the driver. Therefore, the hardware driver can be used as a bridge to realize the fault injection test of the operating system kernel. Thus, each time a test is performed, a target fault scenario can be read from the fault scenario library, and the fault injection configuration parameters and read-write operation instructions corresponding to the target fault scenario are sent to the setting hardware driver in the operating system kernel of the tested end, so that the setting hardware driver completes the injection of the target fault scenario according to the fault injection configuration parameters corresponding to the target fault scenario in the process of controlling the corresponding setting hardware to execute the read-write operation instructions corresponding to the target fault scenario. It should be understood that since the interaction between the operating system kernel and the hardware is carried out through the hardware driver, by performing fault injection for the target fault scenario through the hardware driver and observing the response information of the operating system kernel to the target fault scenario, the reaction of the operating system kernel after the target fault scenario is injected by setting the hardware driver, thereby completing the fault injection test.

[0041] The following is an introduction to the fault injection solution provided by the embodiment of the present invention.

[0042] Figure 1 A structural diagram of a fault injection system provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the system includes a test end 10 and a tested end 20. The test end 10 and the tested end 20 can be terminal devices such as computers, mobile phones, tablet computers, etc. The test end 10 and the tested end 20 can be the same or different. Specifically, as an implementation method, the test end 10 and the tested end 20 can both be computers.

[0043] As another implementation, the test end 10 can be a computer, and the tested end 20 can be a virtual machine. It should be understood that the fault injection test is mainly performed on the tested end 20. In practical applications, after each fault injection test is completed, in order to ensure that the next fault injection test will not be disturbed, the cache files of the tested end 20 will be cleared and the tested end 20 will be restarted. Restarting the computer takes a long time. In contrast, the virtual machine can reduce time consumption and improve work efficiency. In addition, if the tested end 20 is a computer, then when it is damaged, it needs to be repaired, which not only affects the smooth progress of the fault injection test, but also generates additional maintenance costs. By setting the tested end 20 as a virtual machine, when the virtual machine is damaged, it only needs to recreate a new virtual machine, which can ensure the efficiency of the fault injection test and will not generate additional maintenance costs.

[0044] In actual applications, the test end 10 obtains fault injection configuration parameters and read / write operation instructions corresponding to the target fault scenario, and sends the fault injection configuration parameters and read / write operation instructions corresponding to the target fault scenario to the set hardware driver in the operating system kernel of the tested end 20.

[0045] The tested end 20 receives the fault injection configuration parameters and read / write operation instructions corresponding to the target fault scenario sent by the testing end 10, and controls the corresponding set hardware to execute the read / write operation instructions, completes the injection of the target fault scenario based on the fault injection configuration parameters, and sends the response information of the operating system kernel to the target fault scenario to the testing end 10.

[0046] The test end 10 obtains the response information of the operating system kernel to the target fault scenario by setting the hardware driver.

[0047] For a detailed description of the interaction process between the testing terminal 10 and the tested terminal 20, please refer to the following embodiments, which will not be described in detail here.

[0048] Figure 2 A flowchart of a fault injection method provided by an embodiment of the present invention, the method is applied to a test end, such as Figure 2 As shown, the method comprises the following steps:

[0049] 201. Obtain fault injection configuration parameters and read / write operation instructions corresponding to a target fault scenario, where the read / write operation instructions are used to trigger the target fault scenario.

[0050] 202. Send the fault injection configuration parameters and read / write operation instructions corresponding to the target fault scenario to the setting hardware driver in the operating system kernel of the tested end, so that the setting hardware driver completes the injection of the target fault scenario according to the fault injection configuration parameters corresponding to the target fault scenario during the process of controlling the corresponding setting hardware to execute the read / write operation instructions corresponding to the target fault scenario.

[0051] 203. Obtain response information of the operating system kernel to the target fault scenario by setting the hardware driver.

[0052] In actual applications, a set fault scenario library can be created in advance on the test end. The set fault scenario library contains multiple fault injection configuration parameters and multiple read and write operation instructions (also called IO operation instructions). Different fault injection configuration parameters and different read and write operation instruction combinations can simulate a variety of different fault scenarios. The various different fault scenarios can include instructions, queues, power and operating system status, data transmission and storage, management and security, etc. Fault injection configuration parameters may include:

[0053] status_code: Fault status code, represented by the execution status field defined in the completion queue entity of the hardware operation instruction of the hardware. It is used to define the abnormal situation caused by the hardware completing an operation instruction, such as power loss, invalid operation code contained in the read and write operation instructions, etc.

[0054] dont_retry: used to define whether to turn on the command retry mechanism. For example, if dont_retry is set to 0, it means that if a read or write operation instruction fails to complete, the read or write operation instruction can be retried. If dont_retry is set to 1, it means that if a read or write operation instruction fails to complete, the read or write operation instruction cannot be retried.

[0055] Interval: indicates the interval at which the fault occurs. For example, the interval can be 10, 30, 60, etc., which are not listed here one by one.

[0056] space: indicates the margin for fault occurrence. For example, if the margin is set to 100, then starting from 100, the value of the margin will decrease by 1 each time a read or write operation instruction is executed. When the value of the margin decreases to 0, fault injection can be performed.

[0057] probability: indicates the probability of a fault. For example, the probability of a fault can be 0, 50%, 100%, etc. In particular, in actual applications, if you feel that the probability is still too high when it is set to 1%, you can set the probability to 100% and use it with Interval (such as increasing the interval between faults) to reduce the probability of a fault.

[0058] times: indicates the maximum number of times a fault occurs. For example, the maximum number of times a fault occurs can be 1, 2, or 10 times.

[0059] The read and write operation instructions can be used for fault injection testing at different levels (such as the file system layer, general block layer, etc.). The file system layer is the part of the operating system responsible for managing and storing files. It is used to handle file creation, deletion, reading and writing operations. The general block layer is the interface for processing storage devices (such as hard disks, SSDs, etc.) in the operating system. It provides the basic functions of reading and writing storage device blocks. By performing fault injection tests at different levels through read and write operation instructions, fault injection test results at different levels can be obtained, which improves the accuracy of fault injection testing.

[0060] To facilitate understanding, the following examples illustrate different fault scenarios composed of multiple fault injection configuration parameters and multiple read and write operation instructions:

[0061] Fault scenario 1: Assume that the parameter value of "status_code" is set to a non-zero value such as "0x5", indicating power failure, which defines that the abnormal situation caused by the hardware completing a certain operation instruction is power loss. "dont_retry" is defined to not open the instruction retry mechanism, that is, the parameter value of "dont_retry" can be set to 1. "Interval" is defined as 2 for the interval of fault occurrence. "space" is defined as 0 for the margin of fault occurrence. "probability" is defined as 100% for the probability of fault occurrence. "times" is defined as the maximum number of fault occurrences as 1. Read and write operation instructions can be set to operation instructions related to "read", for example, read operations on a file, read and write operations on a buffer, etc.

[0062] Fault scenario 2: Assume that the parameter value of "status_code" is set to a non-zero value such as "0x5", which defines that the abnormal situation caused by the hardware completing a certain operation instruction is power loss. The instruction retry mechanism is turned on by "dont_retry", that is, the parameter value of "dont_retry" can be set to 0. The interval of fault occurrence is defined as 2 by "Interval". The margin of fault occurrence is defined as 0 by "space". The probability of fault occurrence is defined as 100% by "probability". The maximum number of fault occurrences is defined as 1 by "times". The read and write operation instructions can be set to operation instructions related to "read", for example, reading a file, reading and writing a buffer, etc.

[0063] It should be understood that as long as any fault injection configuration parameter or read / write operation instruction is different, two different fault scenarios can be simulated, such as the above-mentioned "fault scenario 1" and "fault scenario 2", the difference between these two fault scenarios is only the parameter "dont_retry". Of course, the above-mentioned "fault scenario 1" and "fault scenario 2" are only two specific implementation methods of fault simulation, and two different fault scenarios can also be defined by setting different "Interval", different "probability", different "times", different "probability" and "Interval" and other different parameter combinations, which are not listed here one by one.

[0064] Based on the above, in the specific implementation, firstly, the test end is remotely connected to the machine (virtual machine or physical machine) on which the operating system kernel is mounted, so as to control the operating system kernel to clear the log records. It should be understood that the tested end may generate some log records that are not related to the fault injection test when it is started. In order to avoid the log records from interfering with the test results of the fault injection test in the subsequent fault injection test, the operating system kernel can be controlled to clear the log records.

[0065] Afterwards, the test end obtains the fault injection configuration parameters and read / write operation instructions corresponding to the target fault scenario in the above-mentioned setting scenario library, and sends the fault injection configuration parameters and read / write operation instructions to the setting hardware driver (such as NVMe hardware driver) in the operating system kernel of the tested end.

[0066] After receiving the fault injection configuration parameters and the read-write operation instructions, the tested end controls the corresponding setting hardware to execute the read-write operation instructions, and completes the injection of the target fault scenario based on the fault injection configuration parameters. Specifically, the tested end decomposes the read-write operation instructions corresponding to the target fault scenario into hardware operation instructions corresponding to the setting hardware through the setting hardware driver, and sends them to the setting hardware for execution, and the setting hardware generates a completion queue entity corresponding to the hardware operation instruction, so that the setting hardware driver modifies the value of the execution status field of the hardware operation instruction in the completion queue entity to the fault status code corresponding to the target fault scenario.

[0067] For ease of understanding, for example, assume that the hardware driver is set to an NVMe hardware driver, the hardware is set to NVMe hardware, and the hardware operation instructions corresponding to the NVMe hardware are NVMe operation instructions. Then, in the specific implementation, the NVMe hardware driver will decompose the read and write operation instructions corresponding to the target fault scenario into multiple NVMe operation instructions and send them to the NVMe hardware for execution. After the NVMe hardware executes multiple NVMe operation instructions, it will generate a corresponding completion queue entity (Completion Queue Entry, CQE for short) in the completion queue. It should be noted that the completion queue contains multiple completion queue entities CQE, which record the execution status of the hardware operation instructions. By modifying the execution status to the fault status code status_code corresponding to the target fault scenario, the simulation of the target fault scenario can be completed. Assuming that when the hardware operation instruction is executed normally, the fault status code corresponding to its execution status is "0", and "0" is modified by the fault status code corresponding to the target fault scenario (it can be modified to any non-0 fault status code corresponding to the target fault scenario, for example, it can be modified from "0" to "1"), the simulation of the target fault scenario can be completed. At this time, the response information of the operating system kernel to the target fault scenario (such as successful execution of NVMe operation instructions, failure of NVMe operation instructions, and error information generated by failure of NVMe operation instructions) is obtained, and the response information is sent to the test end.

[0068] The fault injection method provided by the embodiment of the present invention can simulate a variety of fault scenarios by setting the fault injection configuration parameters and read-write operation instructions in the fault scenario library, so that the entire fault injection test process can cover a variety of fault scenarios, and the test effect is good. By sending the fault injection configuration parameters and read-write operation instructions corresponding to the target fault scenario to the setting hardware driver in the operating system kernel of the test end at the test end, the setting hardware driver completes the injection of the target fault scenario according to the fault injection configuration parameters corresponding to the target fault scenario during the process of controlling the corresponding setting hardware to execute the read-write operation instructions corresponding to the target fault scenario, and then obtains the reaction of the operating system kernel after the target fault scenario is injected by the setting hardware driver, realizing the fault injection operation for the setting hardware driver, and obtaining the fault injection configuration parameters and read-write operation instructions corresponding to the target fault scenario from the setting fault scenario library during the fault injection process, so that the fault injection test can cover a variety of scenarios, the test range is wider, and the test effect is better.

[0069] Figure 3 A flowchart of a fault injection method provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, the method specifically comprises the following steps:

[0070] 301. Obtain fault injection configuration parameters and read / write operation instructions corresponding to a target fault scenario, where the read / write operation instructions are used to trigger the target fault scenario.

[0071] 302. Send the fault injection configuration parameters and read / write operation instructions corresponding to the target fault scenario to the setting hardware driver in the operating system kernel of the tested end, so that the setting hardware driver completes the injection of the target fault scenario according to the fault injection configuration parameters corresponding to the target fault scenario while controlling the corresponding setting hardware to execute the read / write operation instructions corresponding to the target fault scenario.

[0072] 303. Obtain response information of the operating system kernel to the target fault scenario by setting the hardware driver.

[0073] 304. Obtain log records generated by the operating system kernel in a target fault scenario.

[0074] 305. Determine the fault type generated by the operating system kernel based on the log record and response information.

[0075] The process of obtaining the response information in step 301 to step 303 is not described in detail here, and reference may be made to the above embodiment.

[0076] For step 304-step 305, it should be understood that the operating system kernel will generate a series of log data (such as function call data, fault injection configuration parameters corresponding to the injected fault scenario, etc.) when performing a fault injection test under the target fault scenario, forming a log record. It should be additionally explained here that after completing the injection of the target fault scenario, the operating system kernel can be controlled to clear the log record generated under the target fault scenario to ensure that when the next fault injection test is performed, the test result of the next fault injection test will not be interfered by the log record, thereby ensuring the accuracy of the next fault injection test.

[0077] It should be understood that the response information indicates whether an error has occurred, and the log record records relevant information about the fault. In actual applications, the type of fault generated by the operating system kernel can be determined based on the log record and response information, such as file system error, mount mode error, hardware failure, etc.

[0078] By obtaining the log records generated by the operating system kernel in the target fault scenario, the fault type generated by the operating system kernel can be determined based on the log records and response information, laying the foundation for subsequent verification of the operating system kernel.

[0079] Furthermore, if the fault type is a fault type that can trigger a verification operation, the consistency of read and write data of the operating system kernel is verified by a verification tool.

[0080] It should be understood that after completing the fault injection test, in order to verify the resilience of the operating system kernel and determine whether the operating system kernel can return to normal after the fault disappears, the operating system kernel can be verified for read and write data consistency through a verification tool when the fault type is a fault type that can trigger a verification operation. The verification tool can be a disk I / O performance test tool (Flexible I / O Tester‌, referred to as FIO), and the fault type that can trigger a verification operation can be a file system error, a mount method error, a hardware failure, etc.

[0081] If the verification tool outputs the set error field, it means that the read-write data consistency verification has failed, and there may be some problems with the operating system kernel. Anyway, if the verification tool does not output the set error field, it means that the read-write data consistency verification has succeeded, and the operating system kernel has returned to normal after the fault disappears.

[0082] By using the verification tool to verify the read and write data consistency of the operating system kernel when the fault type is a fault type that can trigger a verification operation, it can be determined whether the operating system kernel can return to normal after the fault disappears. If it can return to normal, it means that the self-recovery ability of the operating system kernel is strong, and no processing is required at this time. If it cannot return to normal, it means that the self-recovery ability of the operating system kernel is weak, and subsequent staff will need to perform relevant processing.

[0083] If the fault type is that the file system is read-only, no verification operation is performed. It should be understood that in this case, writing operations to the file system should be avoided to prevent data loss or damage.

[0084] Based on the above, after obtaining the fault injection configuration parameters and read / write operation instructions, response information and verification results corresponding to the target fault scenario, in order to provide effective data support for subsequent analysis and optimization of the operating system kernel, the embodiment of the present invention stores the fault injection configuration parameters and read / write operation instructions, response information and verification results corresponding to the target fault scenario in the fault library corresponding to the operating system kernel. For ease of understanding, the fault library is illustrated below in conjunction with the following Table 1:

[0085]

[0086]

[0087] In Table 1 above, “id”, “fault_name”, “dont_retry”, “status_code”, “status”, “status_name”, “status_code_info”, “times”, “probability”, “interval”, and “space” represent fault injection configuration parameters, “IO_command” represents read and write operation instructions, “expected_result” and “dmesg_log” represent response information, and “FIO_result” and “check_result” represent verification results.

[0088] Further, in the embodiment of the present invention, the operating system kernel is one of multiple versions of the operating system kernel. For multiple versions of the operating system kernel, it is necessary to determine whether the performance of the higher version of the operating system kernel meets expectations. The specific determination steps are as follows:

[0089] Obtain the fault libraries corresponding to multiple versions of operating system kernels;

[0090] For a target fault scenario, determine the response information and verification results corresponding to the multiple versions of the operating system kernel in the target fault scenario from the fault libraries corresponding to the multiple versions of the operating system kernel;

[0091] Compare the response information and verification results of multiple versions of the operating system kernel in the target fault scenario to determine whether the performance of the higher version of the operating system kernel is better than that of the lower version of the operating system kernel.

[0092] In actual applications, for target fault scenarios, the response information and verification results corresponding to the multiple versions of the operating system kernel in the target fault scenario are determined from the fault libraries corresponding to the multiple versions of the operating system kernel. If there is no difference in the response information and verification results between the fault libraries corresponding to the multiple versions of the operating system kernel, or the response information and verification results corresponding to the higher version of the operating system kernel are better than the response information and verification results corresponding to the lower version of the operating system kernel, it can be determined that the performance of the higher version of the operating system kernel is better than that of the lower version of the operating system kernel, and the higher version of the operating system kernel has better stability and fault recovery capabilities.

[0093] If the response information and verification results corresponding to the higher version of the operating system kernel are inferior to those corresponding to the lower version of the operating system kernel, it can be determined that the performance of the higher version of the operating system kernel is inferior to that of the lower version of the operating system kernel. At this time, it is necessary to conduct an in-depth analysis of the fault library corresponding to the higher version of the operating system kernel to find the problems.

[0094] By comparing the corresponding response information and verification results of multiple versions of the operating system kernel in the target fault scenario, it can be determined whether the performance of the higher version of the operating system kernel is better than that of the lower version of the operating system kernel. The corresponding operation can be determined according to the judgment result of the performance of the operating system kernel (for example, if it is determined that the performance of the higher version of the operating system kernel is worse than that of the lower version of the operating system kernel, the higher version of the operating system kernel can be optimized, or the higher version of the operating system kernel can be rolled back to the lower version), ensuring that the performance of the higher version of the operating system kernel finally obtained meets expectations.

[0095] The following is an example of the solution of the present invention in stages with reference to specific application scenarios:

[0096] Assume that the test end is a computer and the tested end is a virtual machine. Figure 4 As shown, the test end includes a control module, a fault injection module, a data analysis module and a data collection module.

[0097] 1. Preprocessing stage:

[0098] Start the virtual machine, download the required version of the operating system kernel code, and recompile the kernel code. After that, run the kernel code on the virtual machine and restart the virtual machine to switch the operating system kernel version on the virtual machine to the required version.

[0099] Multiple fault injection configuration parameters and multiple read-write operation instructions are combined and arranged to obtain multiple fault scenarios, and the data corresponding to the multiple fault scenarios are saved in a comma-separated values ​​(CSV) file to obtain a set fault scenario library. The CSV file is a commonly used electronic file format for storing and exchanging tabular data. The CSV file contains multiple rows of data, each row of data contains a series of fault injection configuration parameters and corresponding read-write operation instructions. This series of fault injection configuration parameters and corresponding read-write operation instructions constitute a fault scenario, that is, each row of data in the CSV file forms a fault scenario.

[0100] 2. Practical application stage:

[0101] S1. The test end establishes a remote connection with the operating system kernel of the tested end, and sends an activation instruction to the tested end through the control module to activate and initialize the tested end. Specifically, the test end controls the operating system kernel to clear log records through the control module.

[0102] S2. The test end obtains the fault injection configuration parameters and read and write operation instructions corresponding to the target fault scenario in the above-mentioned setting scenario library (i.e., a row of data in the above-mentioned CSV file) through the fault injection module and sends them to the setting hardware driver in the operating system kernel of the tested end to modify the content of the debugfs file interface of the tested end and simulate the target fault scenario. Specifically, the content of the debugfs file interface can be modified to the fault injection configuration parameters corresponding to the target fault scenario.

[0103] Furthermore, it is assumed that the hardware driver is set to be an NVMe hardware driver, the hardware is set to be NVMe hardware, and the hardware operation instruction corresponding to the NVMe hardware is an NVMe operation instruction. Then, in the specific implementation, the NVMe hardware driver will decompose the read and write operation instructions corresponding to the target fault scenario into multiple NVMe operation instructions and send them to the NVMe hardware for execution. The NVMe hardware generates a corresponding completion queue entity CQE based on the execution status of multiple NVMe operation instructions. Assuming that when the hardware operation instruction is executed normally, the fault status code corresponding to its execution status is "0", and "0" is modified by the fault status code corresponding to the target fault scenario (it can be modified to any non-0 fault status code corresponding to the target fault scenario, for example, it can be modified from "0" to "1"), the simulation of the target fault scenario can be completed. At this time, the response information of the operating system kernel to the target fault scenario is obtained (such as the successful execution of the NVMe operation instruction, the failure of the NVMe operation instruction, and the error information generated by the failure of the NVMe operation instruction), and the response information is sent to the test end.

[0104] During this process, the FIO tool can be used to verify whether the fault injection test is successful. Specifically, when the err field output by the FIO tool is 0, it means that no problem was encountered during the test, that is, the test was successful. When the err field is not 0, it means that the test failed and the test process was terminated.

[0105] S3. The test end obtains the log records generated by the operating system kernel of the tested end in the target fault scenario through the data collection module, and determines the fault type generated by the operating system kernel based on the log records and response information through the data analysis module. If the fault type is a fault type that can trigger a verification operation, the verification tool verifies the consistency of read and write data of the operating system kernel.

[0106] Specifically, take the example of using the FIO tool to open the verify option to verify the read and write data consistency of the NVMe device, and check whether there are error fields such as "err!=0" in the FIO tool output. If so, it means that the read and write data consistency verification has failed, and there may be some problems with the operating system kernel. Anyway, if the verification tool does not output the above error fields, it means that the read and write data consistency verification is successful, and the operating system kernel returns to normal after the fault disappears.

[0107] S4. After the fault disappears, delete the files generated during the fault injection process, format the NVMe hardware, restart the virtual machine, etc.

[0108] S5. Store the fault injection configuration parameters and read / write operation instructions, response information and verification results corresponding to the target fault scenario in the fault library corresponding to the operating system kernel.

[0109] S6. For the target fault scenario, determine the response information and verification results corresponding to the multiple versions of the operating system kernel in the target fault scenario from the fault libraries corresponding to the multiple versions of the operating system kernel. If there is no difference in the response information and verification results between the fault libraries corresponding to the multiple versions of the operating system kernel, or the response information and verification results corresponding to the higher version of the operating system kernel are better than the response information and verification results corresponding to the lower version of the operating system kernel, it can be determined that the performance of the higher version of the operating system kernel is better than that of the lower version of the operating system kernel, and the higher version of the operating system kernel has better stability and fault recovery capabilities.

[0110] If the response information and verification results corresponding to the higher version of the operating system kernel are inferior to those corresponding to the lower version of the operating system kernel, it can be determined that the performance of the higher version of the operating system kernel is inferior to the performance of the lower version of the operating system kernel. At this time, it is necessary to conduct in-depth analysis of the fault library corresponding to the higher version of the operating system kernel and the fault library corresponding to the lower version of the operating system kernel.

[0111] Based on the above, the fault injection method provided by the embodiment of the present invention can realize the automation of the whole process of obtaining fault injection configuration parameters, implementing fault injection, collecting and analyzing operating system kernel data, fault clearing and forming a fault library, thereby improving the test efficiency. In addition, the embodiment of the present invention can effectively verify the fault detection capability, ensure that the operating system kernel can timely and accurately identify various errors that occur in the process of setting the driver operation instruction processing, evaluate the fault tolerance and recovery mechanism, and verify the self-recovery capability of the operating system kernel when encountering a fault.

[0112] Figure 5 A flowchart of another fault injection method provided by an embodiment of the present invention, the method is applied to the setting hardware driver in the operating system kernel of the tested end, such as Figure 5 As shown, the method includes:

[0113] 501. Receive fault injection configuration parameters and read / write operation instructions corresponding to a target fault scenario sent by a test end.

[0114] 502. In a process of controlling the corresponding setting hardware to execute the read and write operation instructions, the injection of the target fault scenario is completed based on the fault injection configuration parameters.

[0115] 503. Send the response information of the operating system kernel to the target fault scenario to the test end.

[0116] Among them, the fault injection configuration parameters include a fault status code, which is represented by an execution status field defined in a completion queue entity of a hardware operation instruction of the setting hardware; in the process of controlling the corresponding setting hardware to execute the read and write operation instructions, the injection of the target fault scenario is completed based on the fault injection configuration parameters, including:

[0117] Decomposing the read and write operation instructions into hardware operation instructions corresponding to the set hardware and sending them to the set hardware for execution, so that the set hardware generates a completion queue entity corresponding to the hardware operation instructions;

[0118] The execution status of the hardware operation instruction in the completion queue entity is modified to the fault status code corresponding to the target fault scenario.

[0119] In addition, the fault injection configuration parameters also include at least one of the following parameters: a parameter indicating whether the hardware operation instruction is to enable retry, a parameter indicating the probability of occurrence of the target fault scenario, a parameter indicating the interval of occurrence of the target fault scenario, and a parameter indicating the count margin of occurrence of the target fault scenario. The value of the execution status field of the hardware operation instruction in the completion queue entity is modified to the fault status code corresponding to the target fault scenario, including:

[0120] Modify the value of at least one parameter in the setting file interface to a parameter value corresponding to the target fault scenario;

[0121] The completion queue entity is read through the setting file interface to modify the value of the execution status field of the hardware operation instruction in the completion queue entity to the fault status code corresponding to the target fault scenario.

[0122] In specific implementation, the test end obtains the fault injection configuration parameters and read / write operation instructions corresponding to the target fault scenario in the above-mentioned setting scenario library through the fault injection module and sends them to the setting hardware driver in the operating system kernel of the tested end to modify the content of the setting file interface of the tested end to simulate the target fault scenario. Among them, the setting file interface can be a debugfs file interface.

[0123] Figure 5 The method shown can execute the steps in the aforementioned embodiments. For detailed execution process and technical effects, please refer to the description in the aforementioned embodiments, which will not be repeated here.

[0124] The fault injection device of one or more embodiments of the present invention will be described in detail below. Those skilled in the art will appreciate that these devices can be configured using commercially available hardware components through the steps taught in this solution.

[0125] Figure 6The present invention provides a schematic diagram of the structure of a fault injection device provided in an embodiment of the present invention. The device is applied to a test end and comprises: an acquisition module 61 , a sending module 62 and a response module 63 .

[0126] The acquisition module 61 is used to acquire the fault injection configuration parameters and read / write operation instructions corresponding to the target fault scenario, wherein the read / write operation instructions are used to trigger the target fault scenario.

[0127] The sending module 62 is used to send the fault injection configuration parameters and read-write operation instructions corresponding to the target fault scenario to the setting hardware driver in the operating system kernel of the tested end, so that the setting hardware driver completes the injection of the target fault scenario according to the fault injection configuration parameters corresponding to the target fault scenario during the process of controlling the corresponding setting hardware to execute the read-write operation instructions corresponding to the target fault scenario.

[0128] The response module 63 is used to obtain the response information of the operating system kernel to the target fault scenario through the set hardware driver.

[0129] Among them, optionally, the fault injection configuration parameters include a fault status code, and the fault status code is represented by the value of the execution status field defined in the completion queue entity of the hardware operation instruction of the set hardware; the set hardware driver decomposes the read and write operation instructions corresponding to the target fault scenario into hardware operation instructions corresponding to the set hardware and sends them to the set hardware for execution, and the set hardware generates a completion queue entity corresponding to the hardware operation instruction, so that the set hardware driver modifies the value of the execution status field of the hardware operation instruction in the completion queue entity to the fault status code corresponding to the target fault scenario.

[0130] Optionally, the device further includes: a log record acquisition module and a type determination module.

[0131] The log record acquisition module is used to acquire the log records generated by the operating system kernel in the target fault scenario.

[0132] The type determination module is used to determine the fault type generated by the operating system kernel according to the log record and the response information.

[0133] Optionally, the device further comprises: a verification module, configured to verify the consistency of read and write data of the operating system kernel through a verification tool if the fault type is a fault type that can trigger a verification operation.

[0134] Among them, optionally, the device also includes: a storage module, used to store the fault injection configuration parameters and read and write operation instructions corresponding to the target fault scenario, the response information and the verification result of the verification into a fault library corresponding to the operating system kernel.

[0135] Wherein, optionally, the operating system kernel is one of multiple versions of operating system kernels, and the device further includes: a fault library acquisition module, a data determination module and a comparison module.

[0136] The fault library acquisition module is used to acquire the fault libraries corresponding to the multiple versions of the operating system kernels.

[0137] The data determination module is used to determine, for the target fault scenario, response information and verification results corresponding to each of the multiple versions of the operating system kernel under the target fault scenario from the fault libraries corresponding to each of the multiple versions of the operating system kernel.

[0138] The comparison module is used to compare the response information and verification results corresponding to the multiple versions of the operating system kernel in the target fault scenario to determine whether the performance of the higher version of the operating system kernel is better than that of the lower version of the operating system kernel.

[0139] Optionally, the device further includes: a connection module, configured to establish a remote connection with the operating system kernel to control the operating system kernel to clear log records.

[0140] Optionally, the device further includes: a control module, configured to control the operating system kernel to clear log records generated under the target fault scenario after completing the injection of the target fault scenario.

[0141] Figure 6 The device shown can execute the steps in the aforementioned embodiments. For detailed execution process and technical effects, please refer to the description in the aforementioned embodiments, which will not be repeated here.

[0142] Figure 7 The present invention provides a schematic diagram of a fault injection device according to an embodiment of the present invention, which is applied to a hardware driver set in the operating system kernel of the tested terminal. Figure 7 As shown, the device includes: a receiving module 71, an injection module 72 and a sending module 73.

[0143] The receiving module 71 is used to receive the fault injection configuration parameters and read / write operation instructions corresponding to the target fault scenario sent by the test end.

[0144] The injection module 72 is used to control the corresponding setting hardware to execute the read and write operation instructions, and complete the injection of the target fault scenario based on the fault injection configuration parameters.

[0145] The sending module 73 is used to send the response information of the operating system kernel to the target fault scenario to the test end.

[0146] Among them, optionally, the fault injection configuration parameters include a fault status code, and the fault status code is represented by the value of the execution status field defined in the completion queue entity of the hardware operation instruction of the set hardware. The fault injection configuration parameters also include at least one of the following parameters: a parameter indicating whether the hardware operation instruction enables retry, a parameter indicating the probability of occurrence of the target fault scenario, a parameter indicating the occurrence interval of the target fault scenario, and a parameter indicating the occurrence count remainder of the target fault scenario.

[0147] The injection module 72 is specifically used to: decompose the read and write operation instructions into hardware operation instructions corresponding to the set hardware and send them to the set hardware for execution, so that the set hardware generates a completion queue entity corresponding to the hardware operation instructions; modify the value of the execution status field of the hardware operation instructions in the completion queue entity to the fault status code corresponding to the target fault scenario. Also, modify the value of at least one parameter in the setting file interface to a parameter value corresponding to the target fault scenario; read the completion queue entity through the setting file interface to modify the value of the execution status field of the hardware operation instructions in the completion queue entity to the fault status code corresponding to the target fault scenario.

[0148] Figure 7 The device shown can execute the steps in the aforementioned embodiments. For detailed execution process and technical effects, please refer to the description in the aforementioned embodiments, which will not be repeated here.

[0149] In one possible design, the above Figure 6 or Figure 7 The structure of the fault injection device shown can be implemented as an electronic device. Figure 8 As shown, the electronic device may include: a processor 81, a memory 82, and a communication interface 83. The memory 82 stores executable codes, and when the executable codes are executed by the processor 81, the processor 81 can at least implement the fault injection method provided in the above embodiments.

[0150] In addition, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the fault injection method provided in the aforementioned embodiment.

[0151] In addition, an embodiment of the present invention provides a computer program product, which includes: a computer program, when the computer program is executed by a processor of an electronic device, the processor can at least implement the fault injection method provided in the above embodiment.

[0152] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.

[0153] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and application programs. Based on such an understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fault injection method, characterized in that: Applied to the test end, the method includes: Obtaining fault injection configuration parameters and read / write operation instructions corresponding to a target fault scenario, wherein the read / write operation instructions are used to trigger the target fault scenario; Sending the fault injection configuration parameters and read / write operation instructions corresponding to the target fault scenario to the setting hardware driver in the operating system kernel of the tested end, so that the setting hardware driver completes the injection of the target fault scenario according to the fault injection configuration parameters corresponding to the target fault scenario during the process of controlling the corresponding setting hardware to execute the read / write operation instructions corresponding to the target fault scenario; Obtaining response information of the operating system kernel to the target fault scenario through the set hardware driver; For the target fault scenario, determine the response information and read-write data consistency verification results corresponding to the multiple versions of the operating system kernel in the target fault scenario from the fault libraries corresponding to the multiple versions of the operating system kernel; compare the response information and read-write data consistency verification results corresponding to the multiple versions of the operating system kernel in the target fault scenario to determine whether the performance of the higher version of the operating system kernel is better than that of the lower version of the operating system kernel, the read-write data consistency verification result is the result of reading and writing data consistency verification of the corresponding operating system kernel, and the operating system kernel is one of the multiple versions of the operating system kernel.

2. The method according to claim 1, characterized in that The fault injection configuration parameter includes a fault status code, and the fault status code is represented by the value of the execution status field defined in the completion queue entity of the hardware operation instruction of the setting hardware; The set hardware driver decomposes the read and write operation instructions corresponding to the target fault scenario into hardware operation instructions corresponding to the set hardware and sends them to the set hardware for execution. The set hardware generates a completion queue entity corresponding to the hardware operation instructions, so that the set hardware driver modifies the value of the execution status field of the hardware operation instructions in the completion queue entity to the fault status code corresponding to the target fault scenario.

3. The method according to claim 1, characterized in that The method further comprises: Obtaining log records generated by the operating system kernel in the target fault scenario; The fault type generated by the operating system kernel is determined according to the log record and the response information.

4. The method according to claim 3, characterized in that The method further comprises: If the fault type is a fault type that can trigger a verification operation, the consistency of read and write data of the operating system kernel is verified by a verification tool.

5. The method according to claim 4, characterized in that The method further comprises: The fault injection configuration parameters and read / write operation instructions corresponding to the target fault scenario, the response information and the verification result of the verification are stored in a fault library corresponding to the operating system kernel.

6. The method according to any one of claims 1 to 5, characterized in that Before sending the fault injection configuration parameters and read / write operation instructions corresponding to the target fault scenario to the setting hardware driver in the operating system kernel of the tested end, the method further includes: A remote connection is established with the operating system kernel to control the operating system kernel to clear log records.

7. The method according to any one of claims 3 to 5, characterized in that: The method further comprises: After the injection of the target fault scenario is completed, the operating system kernel is controlled to clear the log records generated under the target fault scenario.

8. A fault injection method, characterized in that: The method for setting a hardware driver in the operating system kernel of the tested terminal includes: Receive the fault injection configuration parameters and read / write operation instructions corresponding to the target fault scenario sent by the test end; In the process of controlling the corresponding setting hardware to execute the read and write operation instructions, the injection of the target fault scenario is completed based on the fault injection configuration parameters; Sending response information of the operating system kernel to the target fault scenario to the test end; Among them, the test end is used to determine, for the target fault scenario, from the fault libraries corresponding to each of the multiple versions of the operating system kernel, the verification results of the response information and read-write data consistency corresponding to the multiple versions of the operating system kernel under the target fault scenario; compare the verification results of the response information and read-write data consistency corresponding to the multiple versions of the operating system kernel under the target fault scenario to determine whether the performance of the higher version of the operating system kernel is better than that of the lower version of the operating system kernel, and the verification result of the read-write data consistency is the result of reading and writing data consistency verification of the corresponding operating system kernel, and the operating system kernel is one of the multiple versions of the operating system kernel.

9. The method according to claim 8, characterized in that The fault injection configuration parameter includes a fault status code, and the fault status code is represented by the value of the execution status field defined in the completion queue entity of the hardware operation instruction of the setting hardware; In the process of controlling the setting hardware corresponding to the read and write operation instructions, the injection of the target fault scenario is completed based on the fault injection configuration parameters, including: Decomposing the read and write operation instructions into hardware operation instructions corresponding to the set hardware and sending them to the set hardware for execution, so that the set hardware generates a completion queue entity corresponding to the hardware operation instructions; The value of the execution status field of the hardware operation instruction in the completion queue entity is modified to the fault status code corresponding to the target fault scenario.

10. The method according to claim 9, characterized in that The fault injection configuration parameters also include at least one of the following parameters: a parameter indicating whether the hardware operation instruction turns on retry, a parameter indicating the probability of occurrence of the target fault scenario, a parameter indicating the occurrence interval of the target fault scenario, and a parameter indicating the occurrence count margin of the target fault scenario; The step of modifying the value of the execution status field of the hardware operation instruction in the completion queue entity to a fault status code corresponding to the target fault scenario includes: Modify the value of at least one parameter in the setting file interface to a parameter value corresponding to the target fault scenario; The completion queue entity is read through the setting file interface to modify the value of the execution status field of the hardware operation instruction in the completion queue entity to the fault status code corresponding to the target fault scenario.

11. An electronic device, characterized in that: include: A memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the fault injection method as described in any one of claims 1 to 7 or claims 8 to 10.

12. A non-transitory machine-readable storage medium, characterized in that: The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor is caused to execute the fault injection method according to any one of claims 1 to 7 or claims 8 to 10.

13. A computer program product, characterized in that include: A computer program, when executed by a processor of an electronic device, causes the processor to perform the fault injection method according to any one of claims 1 to 7 or claims 8 to 10.

Images