Fault injection method and device

Abstract

The present disclosure provides a fault injection method and device, which relates to the field of storage technology, and the method includes: obtaining a fault injection command; determining the fault type of the target fault to be injected according to the fault injection command; determining a target injection path corresponding to the fault type from a first injection path and a second injection path, the first injection path being the path between the disk management process and the master control node, and the second injection path being the path between the disk management process and the disk; generating a target fault instruction corresponding to the fault type, and executing the target fault instruction based on the target injection path. The present disclosure not only improves the flexibility and pertinence of fault injection, but also enhances the test coverage of distributed storage systems, so that developers and testers can identify and solve potential problems in a timely manner before a fault occurs, thereby improving the exception handling capability for distributed storage systems.

Description

Technical Field

[0001] The present disclosure relates to the field of storage technology, and in particular to a fault injection method and device. Background Art

[0002] With the rapid development of information technology, distributed storage systems have been widely used in big data processing, cloud computing and enterprise applications. These systems are usually composed of multiple storage nodes and can provide high-availability and high-reliability storage services. However, as the scale of the system increases, the failure rate of storage devices also increases, especially when using new storage media such as NVMeSSD, the types and complexity of failures are also increasing.

[0003] In distributed storage systems, disk failures may lead to data loss, service interruption, and system performance degradation. Therefore, timely detection and handling of disk failures is the key to ensuring system stability and reliability. However, traditional fault injection methods are often limited to simple fault types and cannot fully cover the various complex faults that may occur in distributed storage systems. In particular, there is a lack of effective injection methods for disk drive failures that are difficult to simulate. Summary of the invention

[0004] The embodiments of the present disclosure provide a fault injection method and device, aiming to solve the problems existing in the above-mentioned background technology.

[0005] In order to solve the above technical problems, the present disclosure is implemented as follows: In a first aspect, an embodiment of the present disclosure provides a fault injection method, the method comprising: Get the fault injection command; Determining the fault type of the target fault to be injected according to the fault injection command; Determine a target injection path corresponding to the fault type from a first injection path and a second injection path, wherein the first injection path is a path between a disk management process and a master control node, and the second injection path is a path between the disk management process and a disk; A target fault instruction corresponding to the fault type is generated, and the target fault instruction is executed based on the target injection path.

[0006] Optionally, determining a target injection path corresponding to the fault type from the first injection path and the second injection path includes: Determine the fault type of the target fault, the fault type at least including: disk heartbeat fault, disk status fault and disk drive fault; In a case where the failure type of the target failure is a disk heartbeat failure or a disk status failure, determining the first injection path as the target injection path; In a case where the failure type of the target failure is a disk drive failure, the second injection path is determined as the target injection path.

[0007] Optionally, executing the target fault instruction based on the target injection path includes: In a case where the failure type of the target failure is a disk heartbeat failure, determining a target disk from a storage cluster managed by the disk management process according to the fault injection command; A target fault instruction corresponding to the disk heartbeat fault is executed based on the first injection path, and the status information of the target disk is deleted from the heartbeat information list.

[0008] Optionally, executing the target fault instruction based on the target injection path includes: In a case where the failure type of the target failure is a disk status failure, determining a target disk from a storage cluster managed by the disk management process according to the fault injection command; The target fault instruction corresponding to the disk status fault is executed based on the first injection path, and the status information of the target disk is modified to the abnormal status specified by the fault injection command in the heartbeat information list.

[0009] Optionally, executing the target fault instruction based on the target injection path includes: In a case where the fault type of the target fault is a disk drive fault, determining a target disk from a storage cluster managed by the disk management process according to the fault injection command; Determine the drive controller of the target disk from the local information of the target disk; Determine the failure subtype of the disk drive failure; Executing a target fault instruction corresponding to the fault subtype of the disk drive fault based on the second injection path, marking a fault mark corresponding to the fault subtype for the drive controller of the target disk; The method further includes: generating a target fault for the I / O request according to the fault mark when the drive controller of the target disk receives the I / O request.

[0010] Optionally, generating a target fault for the I / O request according to the fault tag includes: According to the fault mark, determining the fault subtype of the disk drive fault, the fault subtype at least including: I / O submission fault, I / O completion fault and I / O timeout fault; In the case where the failure subtype of the disk drive failure is an I / O submission failure, determining a return value set by the failure flag; responding to the I / O request, returning the return value as the status information of the target disk to the master control node; In the case where the fault subtype of the disk drive fault is an I / O completion fault, determining the fault error code set by the fault tag; responding to the completion event of the I / O request, obtaining a callback function of the completion event of the I / O request, and overwriting the error code in the callback function with the fault error code; In the case where the failure subtype of the disk drive failure is an I / O timeout failure, in response to a completion event of the I / O request, it is checked whether the processing time of the I / O request is greater than a preset timeout processing time; in the case where the processing time of the I / O request is not greater than the timeout processing time, a delayed return process is performed on the completion request representing the completion of the execution of the I / O request until the processing time of the I / O request is greater than the timeout processing time.

[0011] Optionally, generating a target fault for the I / O request according to the fault tag includes: According to the fault mark, determining the fault subtype of the disk drive fault, the fault subtype at least including a data silent fault and an I / O suspension fault; In the case where the failure subtype of the disk drive failure is a data silent failure, intercepting the I / O request, and returning a return value indicating successful execution of the I / O request as the status information of the target disk to the master control node; In a case where the fault subtype of the disk drive fault is an I / O suspension fault, the state of the I / O request is set to a suspension state; checking whether the suspension time of the I / O request is greater than a preset suspension timeout time; and in a case where the suspension time of the I / O request is not greater than the suspension timeout time, maintaining the state of the I / O request in a suspension state until the suspension time of the I / O request is greater than the suspension timeout time.

[0012] Optionally, the method further comprises: When the disk management process receives an I / O request, writing the target fault instruction into a first linked list; Determining whether the target fault instruction needs to be submitted to the drive controller; In a case where the target fault instruction does not need to be submitted to the drive controller, reading the target fault instruction from the first linked list; Determining configuration information of the target fault instruction, and constructing a target fault request for the I / O request based on the target fault instruction, wherein the target fault request includes meta-information corresponding to the configuration information of the target fault instruction; Writing the target fault request into a second linked list; The first linked list and the second linked list are polled, and a corresponding target fault is generated based on configuration information of the target fault instruction in the first linked list or meta information of the target fault request in the second linked list.

[0013] Optionally, the method further comprises: In the case where the target fault instruction needs to be submitted to the drive controller, determining the fault subtype of the disk drive fault; In a case where the fault subtype of the disk drive fault is an I / O timeout fault, monitoring a completion request indicating that the I / O request has been completed; When the completion request is detected, writing the completion request into a third linked list; The first linked list and the third linked list are polled, and a corresponding I / O timeout fault is generated based on the configuration information of the target fault instruction in the first linked list, and the I / O timeout fault is used to perform corresponding delayed return processing on the completion request in the third linked list.

[0014] In a second aspect, an embodiment of the present disclosure provides a fault injection device, the device comprising: An acquisition module, used to obtain fault injection commands; A first determination module, used to determine the fault type of the target fault to be injected according to the fault injection command; A second determination module is used to determine a target injection path corresponding to the fault type from a first injection path and a second injection path, wherein the first injection path is a path between the disk management process and the master control node, and the second injection path is a path between the disk management process and the disk; The fault injection module is used to generate a target fault instruction corresponding to the fault type and execute the target fault instruction based on the target injection path.

[0015] Optionally, the second determining module includes: A first judgment submodule is used to judge the fault type of the target fault, wherein the fault type at least includes: disk heartbeat fault, disk status fault and disk drive fault; A first determination submodule, configured to determine the first injection path as the target injection path when the fault type of the target fault is a disk heartbeat fault or a disk status fault; The second determining submodule is configured to determine the second injection path as the target injection path when the fault type of the target fault is a disk drive fault.

[0016] Optionally, the fault injection module includes: A third determination submodule is used to determine a target disk from the storage cluster managed by the disk management process according to the fault injection command when the fault type of the target fault is a disk heartbeat fault; The first injection submodule is used to execute the target fault instruction corresponding to the disk heartbeat fault based on the first injection path, and delete the status information of the target disk from the heartbeat information list.

[0017] Optionally, the fault injection module includes: A fourth determination submodule is used to determine a target disk from the storage cluster managed by the disk management process according to the fault injection command when the fault type of the target fault is a disk status fault; The second injection submodule is used to execute the target fault instruction corresponding to the disk status fault based on the first injection path, and modify the status information of the target disk in the heartbeat information list to the abnormal state specified by the fault injection command.

[0018] Optionally, the fault injection module includes: a fifth determination submodule, configured to determine a target disk from a storage cluster managed by the disk management process according to the fault injection command when the fault type of the target fault is a disk drive fault; A sixth determination submodule, configured to determine a drive controller of the target disk from local information of the target disk; a seventh determination submodule, used to determine a fault subtype of the disk drive fault; a marking submodule, configured to execute a target fault instruction corresponding to the fault subtype of the disk drive fault based on the second injection path, and mark a fault mark corresponding to the fault subtype for the drive controller of the target disk; Optionally, the device further comprises: A generating module is used to generate a target fault for the I / O request according to the fault mark when the drive controller of the target disk receives the I / O request.

[0019] Optionally, the generating module includes: A second judgment submodule is used to judge the fault subtype of the disk drive fault according to the fault mark, and the fault subtype at least includes: I / O submission fault, I / O completion fault and I / O timeout fault; A first drive fault injection submodule is used to determine the return value set by the fault mark when the fault subtype of the disk drive fault is an I / O submission fault; respond to the I / O request and return the return value as the status information of the target disk to the master control node; A second drive fault injection submodule is used to determine the fault error code set by the fault mark when the fault subtype of the disk drive fault is an I / O completion fault; respond to the completion event of the I / O request, obtain a callback function of the completion event of the I / O request, and overwrite the error code in the callback function with the fault error code; The third drive fault injection submodule is used to check whether the processing time of the I / O request is greater than a preset timeout processing time in response to a completion event of the I / O request when the fault subtype of the disk drive fault is an I / O timeout fault; and if the processing time of the I / O request is not greater than the timeout processing time, perform delayed return processing on the completion request representing the completion of the execution of the I / O request until the processing time of the I / O request is greater than the timeout processing time.

[0020] Optionally, the generating module includes: A third judgment submodule, configured to judge a fault subtype of the disk drive fault according to the fault mark, wherein the fault subtype includes at least a data silent fault and an I / O suspension fault; a fourth drive fault injection submodule, configured to intercept the I / O request when the fault subtype of the disk drive fault is a data silent fault, and return a return value indicating successful execution of the I / O request as the status information of the target disk to the master control node; The fifth drive fault injection submodule is used to set the state of the I / O request to a suspended state when the fault subtype of the disk drive fault is an I / O suspension fault; check whether the suspension time of the I / O request is greater than a preset suspension timeout; and if the suspension time of the I / O request is not greater than the suspension timeout, maintain the state of the I / O request in a suspended state until the suspension time of the I / O request is greater than the suspension timeout.

[0021] Optionally, the device further comprises: A first writing module, configured to write the target fault instruction into a first linked list when the disk management process receives an I / O request; A first judgment module, used to judge whether the target fault instruction needs to be submitted to the drive controller; a reading module, configured to read the target fault instruction from the first linked list when the target fault instruction does not need to be submitted to the drive controller; A construction module, configured to determine configuration information of the target fault instruction, and construct a target fault request for the I / O request based on the target fault instruction, wherein the target fault request includes meta-information corresponding to the configuration information of the target fault instruction; A second writing module, used for writing the target fault request into a second linked list; The first polling module is used to poll the first linked list and the second linked list, and generate a corresponding target fault based on the configuration information of the target fault instruction in the first linked list or the meta information of the target fault request in the second linked list.

[0022] Optionally, the device further comprises: A second determination module, configured to determine a fault subtype of the disk drive fault when the target fault instruction needs to be submitted to the drive controller; A monitoring module, configured to monitor a completion request indicating that the I / O request has been completed when the fault subtype of the disk drive fault is an I / O timeout fault; A third writing module is used for writing the completion request into a third linked list when the completion request is detected; A second polling module is used to poll the first linked list and the third linked list, and generate a corresponding I / O timeout fault based on the configuration information of the target fault instruction in the first linked list, and the I / O timeout fault is used to perform corresponding delayed return processing on the completion request in the third linked list.

[0023] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: According to different fault types, the present invention flexibly selects appropriate target injection paths from the first injection path (the path between the disk management process and the master node) and the second injection path (the path between the disk management process and the disk) for fault injection, effectively simulating multiple fault scenarios, including not only common disk heartbeat faults and disk status faults, but also hard-to-reproduce disk drive faults, covering the simulation of up to dozens of error codes. This not only improves the flexibility and pertinence of fault injection, but also enhances the test coverage of distributed storage systems, enabling developers and testers to identify and solve potential problems in a timely manner before faults occur, thereby improving the exception handling capabilities for distributed storage systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 is a schematic diagram of a distributed storage system application architecture provided by an embodiment of the present disclosure; Figure 2 is a flow chart of a fault injection method provided by an embodiment of the present disclosure; Figure 3 is a schematic diagram of a disk management process for implementing fault injection in one embodiment of the present disclosure; Figure 4 It is a fault injection schematic diagram of a disk heartbeat fault and a disk status fault in one embodiment of the present disclosure; Figure 5 is a schematic diagram of a fault injection of a disk drive failure in one embodiment of the present disclosure; Figure 6 is a flowchart of implementing a disk drive failure in one embodiment of the present disclosure; Figure 7 It is a structural block diagram of a fault injection device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0027] In modern distributed storage systems, with the surge in data volume and the continuous evolution of storage technology, traditional fault injection methods mainly rely on simple simulation methods, and can usually only test limited fault types, such as basic disk write errors or read errors. They often lack flexibility and cannot truly reflect complex fault scenarios, making it difficult to effectively identify and handle potential problems in actual operation. In addition, during the fault simulation process, traditional methods are unable to perform fine-grained control over specific storage media or fault types, resulting in insufficient test coverage and difficulty meeting the requirements of modern distributed storage systems for high availability and high reliability. In view of this, the present disclosure aims to expand the range of fault types that can be simulated and to accurately simulate complex fault scenarios.

[0028] Figure 1is a schematic diagram of a distributed storage system application architecture provided by an embodiment of the present disclosure, such as Figure 1 As shown in the figure, in the application architecture, the disk fault injection module is an independent service responsible for receiving fault injection commands from testers and distributing these fault injection commands to the corresponding disk management processes. The disk fault injection module is configured with a fault distribution module that can process fault injection requests, select the appropriate disk management process, and send RPC messages to obtain cluster views. Testers specify the disk to be injected and the fault type through the configuration file, and the fault injection module parses the configuration file and executes the corresponding fault injection command.

[0029] Each disk management process is responsible for managing multiple NVMe SSD disks on its node and regularly reports disk status information to the master node (master), including health status and fault status. After receiving the fault injection command, the disk management process will execute the corresponding fault code according to the fault type to simulate the fault behavior (such as heartbeat failure, status failure or drive failure). The disk management process is also responsible for clearing the fault mark when the fault is recovered to ensure that the disk resumes normal operation.

[0030] The disk cluster consists of multiple NVMe SSD disks, and each disk management process manages a group of disks. The disk cluster is designed with a three-copy strategy, and each disk is connected through a network.

[0031] The master node is the metadata management service of the entire storage cluster. It is responsible for recording the distribution information of user data in the cluster and uniformly managing the disk resources in the cluster. The master node regularly receives heartbeat information from the disk management process to determine the status of each disk. When a disk failure is detected, the master node initiates data migration or recovery operations to ensure data reliability and high availability of the system.

[0032] Figure 2 is a flow chart of a fault injection method provided by an embodiment of the present disclosure, such as Figure 2 As shown, the method includes: Step S101: Obtain a fault injection command.

[0033] The fault injection command is an instruction used to trigger a specific fault type in a distributed storage system. The fault injection command is issued by the tester through the fault distribution module to simulate various disk failures in order to verify the system's processing capabilities under abnormal conditions. The fault injection command can include multiple fault types, such as disk heartbeat failure, disk status failure, and disk drive failure.

[0034] For example, in response to a disk heartbeat failure, the tester issues a corresponding fault injection command to instruct the system to stop reporting the heartbeat signal of a specific target disk (for example, disk d0). Once the fault injection command is executed, the disk management process will no longer report the status information of the target disk to the master node, and the master node will mark the target disk as a fault state, thereby triggering the corresponding fault handling mechanism.

[0035] Step S102: Determine the fault type of the target fault to be injected according to the fault injection command.

[0036] In step S102, according to the fault injection command obtained in step S101, the specific fault type of the target fault to be injected is determined.

[0037] In this step, the disk management process parses the fault injection command received from the fault distribution module. The fault injection command contains the configuration information of the target fault, including at least: the ID of the target disk, the fault type and the fault level. According to the parsed information, the fault type specified in the fault injection command is determined. The determined fault type is recorded for subsequent operations. The fault type can be stored in a log, or the corresponding state variable can be maintained. In addition, the disk management process can feedback confirmation information of the fault type to the fault distribution module.

[0038] Step S103: determine a target injection path corresponding to the fault type from a first injection path and a second injection path, wherein the first injection path is a path between the disk management process and the master control node, and the second injection path is a path between the disk management process and the disk.

[0039] In the disclosed embodiment, two main injection paths are defined in the disk management process, namely, the first injection path and the second injection path, wherein the first injection path is the communication path between the disk management process and the master control node, and is used to transmit disk status information and fault status information. The second injection path is the communication path between the disk management process and the actual disk, and is used to directly interact with the disk and perform specific fault injection operations.

[0040] According to the fault type determined in step S102, it is determined through which injection path the fault type should be injected. Specifically, determining a target injection path corresponding to the fault type from the first injection path and the second injection path includes: Step S1031 , determining the fault type of the target fault, wherein the fault type at least includes: disk heartbeat fault, disk status fault and disk drive fault.

[0041] Determine the specific fault type of the target fault determined in step S102. Among them, a disk heartbeat failure refers to the disk failing to report status information to the master control node on time, simulating the situation where the disk is disconnected. A disk status failure refers to the disk status being modified to an abnormal state, such as "failed" or "unavailable". A disk drive failure refers to a failure simulated at the disk drive level, such as I / O submission failure, I / O timeout, etc. By determining the fault type, a basis is provided for subsequent path selection.

[0042] Step S1032: when the fault type of the target fault is a disk heartbeat fault or a disk status fault, determine the first injection path as the target injection path.

[0043] If the type of the target fault is determined to be a disk heartbeat fault or a disk status fault in step S1031, the first injection path will be selected as the target injection path because these two types of faults involve the transmission of status information between the disk management process and the master control node. It is understandable that a disk heartbeat fault requires stopping the reporting of the heartbeat signal of a specific disk through the first injection path to simulate the situation where the target disk is disconnected. A disk status fault requires modifying the state of the target disk to a specified abnormal state through the first injection path so that the master control node can identify and take corresponding fault handling measures.

[0044] Step S1033: when the fault type of the target fault is a disk drive fault, determine the second injection path as the target injection path.

[0045] If the target fault type is determined to be a disk drive fault in step S1031, the second injection path will be selected as the target injection path, because the injection of the disk drive fault involves direct interaction between the disk management process and the actual disk. In this case, the second injection path is used to communicate with the disk controller to simulate the fault behavior at the driver level, such as failure and timeout of I / O requests.

[0046] The above embodiment selects a suitable injection path according to different fault types, laying a foundation for subsequent fault instruction execution, so as to effectively simulate various disk failure scenarios.

[0047] Step S104: Generate a target fault instruction corresponding to the fault type, and execute the target fault instruction based on the target injection path.

[0048] Figure 3 FIG. 1 is a schematic diagram of a disk management process for implementing fault injection in one embodiment of the present disclosure. Figure 3As shown, according to the determined fault type, a corresponding fault instruction is constructed. Based on the previously selected target injection path, the generated target fault instruction is issued and executed in the corresponding disk management process to achieve fault injection. It should be noted that the generated fault instruction needs to conform to the predefined format to ensure that it can be correctly parsed and executed. The target fault instruction contains configuration information such as the fault type, target disk ID, fault parameters, etc., and can be stored in an instruction queue or a linked list. According to the target injection path (first injection path or second injection path) determined in step S103, it is determined how to send the fault instruction. If the first injection path is selected, the instruction will be sent to the master node; if the second injection path is selected, the instruction will be sent directly to the target disk. The target fault instruction is sent through a corresponding communication mechanism (such as a message queue, a network request, etc.).

[0049] According to different fault types, the present invention flexibly selects appropriate target injection paths from the first injection path (the path between the disk management process and the master node) and the second injection path (the path between the disk management process and the disk) for fault injection, effectively simulating multiple fault scenarios, including not only common disk heartbeat faults and disk status faults, but also hard-to-reproduce disk drive faults, covering the simulation of up to dozens of error codes. This not only improves the flexibility and pertinence of fault injection, but also enhances the test coverage of distributed storage systems, enabling developers and testers to identify and solve potential problems in a timely manner before faults occur, thereby improving the exception handling capabilities for distributed storage systems.

[0050] In an optional embodiment, executing the target fault instruction based on the target injection path includes: when the fault type of the target fault is a disk heartbeat failure, according to the fault injection command, determining the target disk from the storage cluster managed by the disk management process; executing the target fault instruction corresponding to the disk heartbeat failure based on the first injection path, and deleting the status information of the target disk from the heartbeat information list.

[0051] When it is determined that the target fault type is a disk heartbeat fault, a series of specific operations are performed to simulate the disk heartbeat fault and verify the processing capacity of the distributed storage system in this case. Specifically, the target disk is determined according to the fault injection command, and the disk management process parses the fault injection command received from the fault distribution module. According to the parsed target disk ID, the disk management process searches for the corresponding target disk in the storage cluster it manages. The disk information in the storage cluster is maintained in a data structure for quick search. When the target disk is determined, due to the characteristics of the disk heartbeat fault, the fault injection is performed through the first injection path (i.e., the path between the disk management process and the master node).

[0052] The specific operation of executing the target fault instruction is to delete the status information of the target disk from the heartbeat information list. The heartbeat information list is a collection of all disk status records, which is used to monitor the health status of the disk. Deleting the status information of the target disk means that the master node will no longer receive the heartbeat signal of the target disk, simulating the situation of the disk being disconnected. The master node will find that the target disk has not reported the status in the next status check, and thus mark it as a fault state.

[0053] When it is found that the status information of the target disk has been deleted, the master node will detect that the target disk has not reported the heartbeat signal on time and take corresponding measures according to the preset fault handling mechanism. For example, the master node may redirect the I / O request to the target disk to other normally working disks to ensure the availability of the application architecture and the integrity of the data. In addition, this fault injection event is recorded for subsequent fault analysis and system monitoring. Log information can help the development and testing team evaluate the performance of the application architecture under abnormal conditions and make corresponding optimizations.

[0054] Through the above steps, the target fault instruction for disk heartbeat failure can be effectively executed. Not only does it simulate the situation of disk loss, but it also triggers the fault handling mechanism of the master node by deleting the heartbeat information, which can help developers and testers identify and solve potential problems before the actual failure occurs, thereby improving the reliability and exception handling capabilities of the distributed storage system application architecture.

[0055] In an optional embodiment, executing the target fault instruction based on the target injection path includes: when the fault type of the target fault is a disk status failure, determining the target disk from the storage cluster managed by the disk management process according to the fault injection command; executing the target fault instruction corresponding to the disk status failure based on the first injection path, and modifying the status information of the target disk in the heartbeat information list to the abnormal state specified by the fault injection command.

[0056] Figure 4 FIG. 1 is a schematic diagram of a fault injection of a disk heartbeat fault and a disk status fault in an embodiment of the present disclosure. Figure 4As shown, when the target fault type is a disk status fault, the disk management process parses the fault injection command received from the fault distribution module. According to the parsed target disk ID, the disk management process searches for the corresponding target disk in the storage cluster it manages. After determining the target disk, the application architecture can perform subsequent fault injection operations on the disk. The injection of the disk status fault still needs to be performed through the first injection path. It should be noted that the present disclosure considers injecting two major types of faults, disk heartbeat faults and disk status faults, based on the first injection path because these two types of faults are closely related to the interaction between the disk management process and the master control node in the distributed storage system. First, the aforementioned disk heartbeat failure directly affects the monitoring and management of the health status of the disk by the master control node. Therefore, injecting the heartbeat failure through the first injection path can effectively simulate the scenario of the disk being disconnected. Secondly, the disk status failure to be described here involves abnormal operation of the disk, such as disk process hanging, inconsistent status information, etc., which also needs to be reflected and processed through the interaction between the disk management process and the master control node. Injecting these two types of faults based on the first injection path is equivalent to the upstream injection logic based on the disk management process, which facilitates the master control node to make corresponding decisions and take necessary recovery measures to better reproduce the two types of faults.

[0057] The specific operation of executing the target fault instruction is to modify the state information of the target disk to the abnormal state specified by the fault injection command. The abnormal state can be "failure", "unavailable" or other predefined states. It involves finding the current state of the target disk in the heartbeat information list and updating it to the new abnormal state.

[0058] When the state information of the target disk is modified to an abnormal state, the master node detects the state change of the target disk and takes corresponding measures according to the preset fault handling mechanism. For example, the master node may redirect the I / O request to the disk to other disks that are working normally.

[0059] Through the above steps, the target fault instruction for the disk status fault is effectively executed. The present disclosure not only simulates the abnormal disk status, but also triggers the fault handling mechanism of the master control node by modifying the status information. It helps developers and testers to identify and solve potential problems before the actual fault occurs, thereby improving the exception handling capability of the distributed storage system application architecture.

[0060] In an optional embodiment, executing the target fault instruction based on the target injection path includes: when the fault type of the target fault is a disk drive failure, determining the target disk from the storage cluster managed by the disk management process according to the fault injection command; determining the drive controller of the target disk from the local information of the target disk; determining the fault subtype of the disk drive failure; executing the target fault instruction corresponding to the fault subtype of the disk drive failure based on the second injection path, and marking the fault mark corresponding to the fault subtype for the drive controller of the target disk.

[0061] Figure 5 FIG. 1 is a schematic diagram of a fault injection of a disk drive failure in an embodiment of the present disclosure. Figure 5 As shown in the figure, when the target fault type is a disk drive fault, similar to the process described above, the disk management process parses the received fault injection command. The disk management process searches for the corresponding disk in the storage cluster it manages based on the parsed target disk ID. The drive controller information is extracted from the local information of the target disk. Each disk has a set of local information, including the configuration, status and other related parameters of its drive controller. The disk management process obtains the specific data of the drive controller by accessing the local information. The drive controller, as a bridge between the disk and the master node, is responsible for converting I / O requests into operations that the disk can understand, and at the same time feeds back the disk's response to the upper layer. Further, the specific subtype of the disk drive fault is determined. The disk drive fault includes multiple subtypes, such as I / O submission fault, I / O completion fault, I / O timeout fault, etc. According to the information in the fault injection command, the specific fault subtype is judged and classified. Different fault subtypes will result in different responses and processing mechanisms.

[0062] Finally, based on the second injection path, the target fault instruction corresponding to the subtype of the disk drive failure will be executed. The injection of the disk drive failure needs to be performed through the second injection path (i.e., the path between the disk management process and the disk). In contrast to the first injection path, the second injection path is the downstream injection logic for the disk drive process. Considering that the disk drive failure involves the communication and data transmission between the disk drive controller and the disk management process. Fault injection through the second injection path can directly affect the behavior of the drive controller, thereby more realistically simulating the impact of the drive failure. Unlike disk heartbeat failures and disk status failures, the occurrence of disk drive failures often does not depend on the upstream logic of the disk management process. Therefore, the present disclosure aims to ensure that the stability and fault handling capabilities of the driver layer are independently tested without affecting the normal communication between the disk management process and the master control node.

[0063] The specific operation of executing the target fault instruction is to mark the drive controller of the target disk with a fault mark corresponding to the fault subtype for subsequent fault processing and response mechanism, indicating how the drive controller should process when receiving an I / O request.

[0064] In an optional implementation, the method further includes: when the drive controller of the target disk receives an I / O request, generating a target fault for the I / O request according to the fault mark.

[0065] The I / O request comes from the upper layer application or system component in the user state, and the I / O request may include read and write operations. The drive controller of the target disk is responsible for processing the I / O request and interacting with the disk for data. When the drive controller receives the I / O request, it checks the fault flag related to the I / O request. According to the fault flag, the drive controller will decide how to respond to the request, thereby generating a corresponding target fault.

[0066] Specifically, in an optional implementation manner, generating a target fault for the I / O request according to the fault mark includes: According to the fault mark, the fault subtype of the disk drive fault is determined, and the fault subtype includes at least: I / O submission fault, I / O completion fault and I / O timeout fault; in the case where the fault subtype of the disk drive fault is I / O submission fault, the return value set by the fault mark is determined; in response to the I / O request, the return value is returned to the master control node as the status information of the target disk; in the case where the fault subtype of the disk drive fault is I / O completion fault, the fault error code set by the fault mark is determined; in response to the completion of the I / O request, the fault error code is returned to the master control node; in the case where the fault subtype of the disk drive fault is I / O completion fault, the fault error code is returned to the master control node; in response to the completion of the I / O request, the fault error code is returned to the master control node; in response to the completion of the I / O request, the fault subtype of the disk drive fault is I / O submission fault, the return value is returned to the master control node as the status information of the target disk; in response to the completion of the I / O request, the fault error code is returned to the master control node ... subtype of the disk drive fault is I / O submission fault, the return value is returned to the master control node as the status information of the target disk event, obtaining a callback function of the completion event of the I / O request, and overwriting the error code in the callback function with the fault error code; in a case where the fault subtype of the disk drive fault is an I / O timeout fault, in response to the completion event of the I / O request, checking whether the processing time of the I / O request is greater than a preset timeout processing time; in a case where the processing time of the I / O request is not greater than the timeout processing time, performing a delayed return process on the completion request representing the completion of the I / O request until the processing time of the I / O request is greater than the timeout processing time.

[0067] The fault subtypes include at least but not limited to one of the following: an I / O submission fault, indicating that an error occurred when submitting an I / O request; an I / O completion fault, indicating that an error occurred when completing an I / O request; an I / O timeout fault, indicating that the I / O request was not responded to within a predetermined time.

[0068] If the fault subtype of the disk drive fault is an I / O submission fault, the return value is determined according to the fault tag. For example, the return value may be ERROR_DISK_NOT_RESPONDING, indicating that the disk failed to respond to the request. After receiving the I / O request, the drive controller immediately returns this return value as the status information of the target disk to the master control node. In this way, the master control node will know that there is a problem with the disk and can take appropriate measures.

[0069] If the fault subtype of the disk drive fault is an I / O completion fault, a fault error code is determined based on the fault tag, such as IO_ERROR_MEDIA_ERROR, indicating a media error. When the completion event of the I / O request is triggered, the drive controller obtains the callback function of the completion event of the request. The drive controller replaces the error code in the callback function with the fault error code set in the fault tag, and then feeds back the error to the upper-layer master node.

[0070] If the fault subtype of the disk drive failure is an I / O timeout fault, the drive controller checks the request processing time in the I / O request completion event. If the processing time does not exceed the preset timeout processing time (for example, 100 milliseconds), the drive controller will delay the return processing of the I / O request completion request and keep the request status as unfinished. When the processing time exceeds the set timeout limit, the drive controller will start processing the I / O request completion event and mark it as timeout completion.

[0071] Through the above fault injection method for disk drive failure, different types of target failures can be generated according to the fault markers, and corresponding measures can be taken during the processing of I / O requests.

[0072] In an optional embodiment, generating a target fault for the I / O request based on the fault mark includes: judging a fault subtype of the disk drive fault based on the fault mark, the fault subtype including at least a data silence fault and an I / O suspension fault; if the fault subtype of the disk drive fault is a data silence fault, intercepting the I / O request, and returning a return value representing successful execution of the I / O request as status information of the target disk to the master control node; if the fault subtype of the disk drive fault is an I / O suspension fault, setting the status of the I / O request to a suspended state; checking whether the suspension time of the I / O request is greater than a preset suspension timeout; and if the suspension time of the I / O request is not greater than the suspension timeout, maintaining the status of the I / O request in a suspended state until the suspension time of the I / O request is greater than the suspension timeout.

[0073] The fault subtypes include at least but not limited to the following: data silent fault and I / O suspension fault. Among them, data silent fault refers to the disk drive process, although the master control node believes that the I / O request has been successfully executed, but in fact the request does not write or update any data to the disk. The data silent fault is usually not discovered in a short time because the disk management process will return a successful status, causing the master control node to continue to perform subsequent operations. I / O suspension fault means that the I / O request is set to a suspended state during the processing process, resulting in the request not being responded to in time. This type of fault may be caused by resource competition, device busyness or other system problems. Here, this embodiment further classifies the disk drive fault into two types of sub-faults, the first type is the I / O submission fault, I / O completion fault and I / O timeout fault mentioned above, and the second type is the data silent fault and I / O suspension fault described here. The first type of fault is directly related to the submission and completion process of the I / O request, and can reflect the errors and timeouts that may occur in the request processing stage in real time. The second type of failure focuses on more complex failure scenarios, such as data silence and I / O suspension, which often show their impact only after the I / O request is submitted, so different strategies need to be adopted when handling them.

[0074] When the fault subtype is determined to be a data silent fault based on the fault tag, the driver controller intercepts the I / O request (for example, writing data to the disk) issued by the application. The driver controller returns the return value (for example, the return value is SUCCESS) indicating the successful execution of the I / O request directly to the master control node without actually submitting the request to the disk. That is, the master control node believes that the data has been successfully written to the disk, but in fact the data has not been written.

[0075] If the fault subtype is determined to be an I / O suspension fault based on the fault mark, when an I / O request arrives, the driver controller sets the state of the I / O request to a suspended state, that is, the I / O request is temporarily shelved and not actually processed. The driver controller checks whether the suspension time of the I / O request is greater than a preset suspension timeout (for example, set to 100 milliseconds). If the suspension time does not exceed the preset timeout limit, the driver controller will keep the state of the I / O request in a suspended state until the suspension time exceeds the timeout limit. Once the suspension time exceeds the preset timeout, the driver controller can take appropriate measures, such as returning a timeout error or marking the request as completed.

[0076] In an optional embodiment, the method also includes: when the disk management process receives an I / O request, writing the target fault instruction into a first linked list; determining whether the target fault instruction needs to be submitted to the drive controller; when the target fault instruction does not need to be submitted to the drive controller, reading the target fault instruction from the first linked list; determining the configuration information of the target fault instruction, and constructing a target fault request for the I / O request based on the target fault instruction, the target fault request including meta-information corresponding to the configuration information of the target fault instruction; writing the target fault request into a second linked list; polling the first linked list and the second linked list, and generating a corresponding target fault based on the configuration information of the target fault instruction in the first linked list or the meta-information of the target fault request in the second linked list.

[0077] Regarding the injection strategy of disk drive failures, the present disclosure considers using linked lists to store different types of failures to achieve the purpose of polling and harvesting failures. Figure 6 is a flowchart of implementing a disk drive failure in one embodiment of the present disclosure, such as Figure 6 As shown, when the disk management process receives an I / O request, a target fault instruction is generated according to the fault mark. At this time, the target fault instruction is written into the first linked list. The first linked list is used to store the fault commands to be processed. Next, it is determined whether the target fault instruction needs to be submitted to the drive controller. This judgment is based on the type of fault instruction and the current state of the application architecture. If the target fault instruction does not need to be submitted to the drive controller, it means that the instruction can be processed without interfering with the actual I / O operation.

[0078] When it is determined that the target fault instruction does not need to be submitted to the drive controller, the target fault instruction is read out from the first linked list to ensure the update and management of the first linked list, and to avoid the retention of useless or processed fault instructions in the first linked list. Subsequently, a target fault request for the I / O request is constructed according to the configuration information of the target fault instruction. The target fault request contains meta-information corresponding to the target fault instruction to ensure that the characteristics and handling methods of the fault can be accurately reflected during the fault injection process. The constructed target fault request will be written into the second linked list. The second linked list is used to store the prepared fault request. By storing the fault request separately from the fault command, the fault injection can be managed and scheduled more flexibly. The first linked list and the second linked list are polled to check the contents in the first linked list and the second linked list. Based on the configuration information of the target fault instruction in the first linked list, or the meta-information of the target fault request in the second linked list, the corresponding target fault is generated.

[0079] Through linked list management, the application architecture can efficiently store, process and execute different types of fault injection requests. The separate storage of the first linked list and the second linked list makes fault management more flexible and clear, which can improve the accuracy and response speed of fault injection, thereby better simulating real fault scenarios during testing and verification.

[0080] In an optional embodiment, the method also includes: when the target fault instruction needs to be submitted to the drive controller, determining the fault subtype of the disk drive fault; when the fault subtype of the disk drive fault is an I / O timeout fault, monitoring a completion request representing the completion of the I / O request execution; when the completion request is detected, writing the completion request into a third linked list; polling the first linked list and the third linked list, generating a corresponding I / O timeout fault based on the configuration information of the target fault instruction in the first linked list, the I / O timeout fault being used to perform corresponding delayed return processing on the completion request in the third linked list.

[0081] like Figure 6 As shown, when it is determined that the target fault instruction needs to be submitted to the drive controller, if the fault subtype is determined to be an I / O timeout fault, it is necessary to monitor the completion request that represents the completion of the I / O request execution. When it is detected that the completion request of the drive controller exists, the I / O request is written into the third linked list (delayed completion request linked list). The third linked list is used to store completion requests that have been completed but need to be returned with a delay, so as to control when the completion information is returned to the application, thereby realizing fault simulation.

[0082] It should be noted that the polling mechanism disclosed in the present invention involves monitoring of multiple linked lists and events. As mentioned above, the first linked list stores target fault instructions, including fault commands to be processed. The second linked list stores fault requests, including constructed fault requests. The third linked list stores completion requests, including requests that have been completed but need to be returned with a delay. The completion event of the I / O request monitors the actual completion of the I / O request. Figure 6 The I / O polling thread shown in the figure checks and processes different states and requests in real time by polling these four parts. Specifically, the corresponding I / O timeout fault is generated according to the configuration information of the target fault instruction in the first linked list, and the delayed return processing is performed on the completion request in the third linked list. This mechanism allows the system to flexibly manage and respond to various fault conditions during the fault injection process, thereby simulating real fault scenarios. This verifies the system's responsiveness and stability in the face of different fault conditions.

[0083] Figure 7 is a structural block diagram of a fault injection device provided by an embodiment of the present disclosure, such as Figure 7 As shown, the device comprises: An acquisition module 201 is used to acquire a fault injection command; A first determination module 202, configured to determine a fault type of a target fault to be injected according to the fault injection command; A second determination module 203 is used to determine a target injection path corresponding to the fault type from a first injection path and a second injection path, wherein the first injection path is a path between the disk management process and the master control node, and the second injection path is a path between the disk management process and the disk; The fault injection module 204 is configured to generate a target fault instruction corresponding to the fault type and execute the target fault instruction based on the target injection path.

[0084] In an optional implementation, the second determining module includes: A first judgment submodule is used to judge the fault type of the target fault, wherein the fault type at least includes: disk heartbeat fault, disk status fault and disk drive fault; A first determination submodule, configured to determine the first injection path as the target injection path when the fault type of the target fault is a disk heartbeat fault or a disk status fault; The second determining submodule is configured to determine the second injection path as the target injection path when the fault type of the target fault is a disk drive fault.

[0085] In an optional implementation, the fault injection module includes: A third determination submodule is used to determine a target disk from the storage cluster managed by the disk management process according to the fault injection command when the fault type of the target fault is a disk heartbeat fault; The first injection submodule is used to execute the target fault instruction corresponding to the disk heartbeat fault based on the first injection path, and delete the status information of the target disk from the heartbeat information list.

[0086] In an optional implementation, the fault injection module includes: A fourth determination submodule is used to determine a target disk from the storage cluster managed by the disk management process according to the fault injection command when the fault type of the target fault is a disk status fault; The second injection submodule is used to execute the target fault instruction corresponding to the disk status fault based on the first injection path, and modify the status information of the target disk in the heartbeat information list to the abnormal state specified by the fault injection command.

[0087] In an optional implementation, the fault injection module includes: a fifth determination submodule, configured to determine a target disk from a storage cluster managed by the disk management process according to the fault injection command when the fault type of the target fault is a disk drive fault; A sixth determination submodule, configured to determine a drive controller of the target disk from local information of the target disk; a seventh determination submodule, used to determine a fault subtype of the disk drive fault; a marking submodule, configured to execute a target fault instruction corresponding to the fault subtype of the disk drive fault based on the second injection path, and mark a fault mark corresponding to the fault subtype for the drive controller of the target disk; In an optional embodiment, the device further comprises: A generating module is used to generate a target fault for the I / O request according to the fault mark when the drive controller of the target disk receives the I / O request.

[0088] In an optional implementation, the generating module includes: A second judgment submodule is used to judge the fault subtype of the disk drive fault according to the fault mark, and the fault subtype at least includes: I / O submission fault, I / O completion fault and I / O timeout fault; A first drive fault injection submodule is used to determine the return value set by the fault mark when the fault subtype of the disk drive fault is an I / O submission fault; respond to the I / O request and return the return value as the status information of the target disk to the master control node; A second drive fault injection submodule is used to determine the fault error code set by the fault mark when the fault subtype of the disk drive fault is an I / O completion fault; respond to the completion event of the I / O request, obtain a callback function of the completion event of the I / O request, and overwrite the error code in the callback function with the fault error code; The third drive fault injection submodule is used to check whether the processing time of the I / O request is greater than a preset timeout processing time in response to a completion event of the I / O request when the fault subtype of the disk drive fault is an I / O timeout fault; and if the processing time of the I / O request is not greater than the timeout processing time, perform delayed return processing on the completion request representing the completion of the execution of the I / O request until the processing time of the I / O request is greater than the timeout processing time.

[0089] In an optional implementation, the generating module includes: A third judgment submodule, configured to judge a fault subtype of the disk drive fault according to the fault mark, wherein the fault subtype includes at least a data silent fault and an I / O suspension fault; a fourth drive fault injection submodule, configured to intercept the I / O request when the fault subtype of the disk drive fault is a data silent fault, and return a return value indicating successful execution of the I / O request as the status information of the target disk to the master control node; The fifth drive fault injection submodule is used to set the state of the I / O request to a suspended state when the fault subtype of the disk drive fault is an I / O suspension fault; check whether the suspension time of the I / O request is greater than a preset suspension timeout; and if the suspension time of the I / O request is not greater than the suspension timeout, maintain the state of the I / O request in a suspended state until the suspension time of the I / O request is greater than the suspension timeout.

[0090] In an optional embodiment, the device further comprises: A first writing module, configured to write the target fault instruction into a first linked list when the disk management process receives an I / O request; A first judgment module, used to judge whether the target fault instruction needs to be submitted to the drive controller; a reading module, configured to read the target fault instruction from the first linked list when the target fault instruction does not need to be submitted to the drive controller; A construction module, configured to determine configuration information of the target fault instruction, and construct a target fault request for the I / O request based on the target fault instruction, wherein the target fault request includes meta-information corresponding to the configuration information of the target fault instruction; A second writing module, used for writing the target fault request into a second linked list; The first polling module is used to poll the first linked list and the second linked list, and generate a corresponding target fault based on the configuration information of the target fault instruction in the first linked list or the meta information of the target fault request in the second linked list.

[0091] In an optional embodiment, the device further comprises: A second determination module, configured to determine a fault subtype of the disk drive fault when the target fault instruction needs to be submitted to the drive controller; A monitoring module, configured to monitor a completion request indicating that the I / O request has been completed when the fault subtype of the disk drive fault is an I / O timeout fault; A third writing module is used for writing the completion request into a third linked list when the completion request is detected; A second polling module is used to poll the first linked list and the third linked list, and generate a corresponding I / O timeout fault based on the configuration information of the target fault instruction in the first linked list, and the I / O timeout fault is used to perform corresponding delayed return processing on the completion request in the third linked list.

[0092] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, devices, electronic devices, and storage media. Therefore, the embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0093] The embodiments of the present disclosure are described with reference to the flowcharts and / or block diagrams of the methods and devices according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing terminal device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0094] Although the preferred embodiments of the present disclosure have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present disclosure.

[0095] Finally, it should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "include..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements. A fault injection method and device provided by the present disclosure are described in detail above. The principle and implementation method of the present disclosure are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present disclosure; at the same time, for those skilled in the art, according to the idea of ​​the present disclosure, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present disclosure.

Claims

1. A fault injection method, characterized in that: The method comprises: Get the fault injection command; Determining the fault type of the target fault to be injected according to the fault injection command; Determine a target injection path corresponding to the fault type from a first injection path and a second injection path, wherein the first injection path is a path between a disk management process and a master control node, and the second injection path is a path between the disk management process and a disk; A target fault instruction corresponding to the fault type is generated, and the target fault instruction is executed based on the target injection path.

2. The method according to claim 1, characterized in that The step of determining a target injection path corresponding to the fault type from the first injection path and the second injection path comprises: Determine the fault type of the target fault, the fault type at least including: disk heartbeat fault, disk status fault and disk drive fault; In a case where the failure type of the target failure is a disk heartbeat failure or a disk status failure, determining the first injection path as the target injection path; In a case where the failure type of the target failure is a disk drive failure, the second injection path is determined as the target injection path.

3. The method according to claim 2, characterized in that The executing the target fault instruction based on the target injection path includes: In a case where the failure type of the target failure is a disk heartbeat failure, determining a target disk from a storage cluster managed by the disk management process according to the fault injection command; A target fault instruction corresponding to the disk heartbeat fault is executed based on the first injection path, and the status information of the target disk is deleted from the heartbeat information list.

4. The method according to claim 2, characterized in that: The executing the target fault instruction based on the target injection path includes: In a case where the failure type of the target failure is a disk status failure, determining a target disk from a storage cluster managed by the disk management process according to the fault injection command; The target fault instruction corresponding to the disk status fault is executed based on the first injection path, and the status information of the target disk is modified to the abnormal status specified by the fault injection command in the heartbeat information list.

5. The method according to claim 2, characterized in that: The executing the target fault instruction based on the target injection path includes: In a case where the fault type of the target fault is a disk drive fault, determining a target disk from a storage cluster managed by the disk management process according to the fault injection command; Determine the drive controller of the target disk from the local information of the target disk; Determine the failure subtype of the disk drive failure; Executing a target fault instruction corresponding to the fault subtype of the disk drive fault based on the second injection path, marking a fault mark corresponding to the fault subtype for the drive controller of the target disk; The method further includes: generating a target fault for the I / O request according to the fault mark when the drive controller of the target disk receives the I / O request.

6. The method according to claim 5, characterized in that Generating a target fault for the I / O request according to the fault tag includes: According to the fault mark, determining the fault subtype of the disk drive fault, the fault subtype at least including: I / O submission fault, I / O completion fault and I / O timeout fault; In the case where the failure subtype of the disk drive failure is an I / O submission failure, determining a return value set by the failure flag; responding to the I / O request, returning the return value as the status information of the target disk to the master control node; In the case where the fault subtype of the disk drive fault is an I / O completion fault, determining the fault error code set by the fault tag; responding to the completion event of the I / O request, obtaining a callback function of the completion event of the I / O request, and overwriting the error code in the callback function with the fault error code; In the case where the failure subtype of the disk drive failure is an I / O timeout failure, in response to a completion event of the I / O request, it is checked whether the processing time of the I / O request is greater than a preset timeout processing time; in the case where the processing time of the I / O request is not greater than the timeout processing time, a delayed return process is performed on the completion request representing the completion of the execution of the I / O request until the processing time of the I / O request is greater than the timeout processing time.

7. The method according to claim 5, characterized in that Generating a target fault for the I / O request according to the fault tag includes: According to the fault mark, determining the fault subtype of the disk drive fault, the fault subtype at least including a data silent fault and an I / O suspension fault; In the case where the failure subtype of the disk drive failure is a data silent failure, intercepting the I / O request, and returning a return value indicating successful execution of the I / O request as the status information of the target disk to the master control node; In a case where the fault subtype of the disk drive fault is an I / O suspension fault, the state of the I / O request is set to a suspension state; checking whether the suspension time of the I / O request is greater than a preset suspension timeout time; and in a case where the suspension time of the I / O request is not greater than the suspension timeout time, maintaining the state of the I / O request in a suspension state until the suspension time of the I / O request is greater than the suspension timeout time.

8. The method according to claim 5, characterized in that The method further comprises: When the disk management process receives an I / O request, writing the target fault instruction into a first linked list; Determining whether the target fault instruction needs to be submitted to the drive controller; In a case where the target fault instruction does not need to be submitted to the drive controller, reading the target fault instruction from the first linked list; Determining configuration information of the target fault instruction, and constructing a target fault request for the I / O request based on the target fault instruction, wherein the target fault request includes meta-information corresponding to the configuration information of the target fault instruction; Writing the target fault request into a second linked list; The first linked list and the second linked list are polled, and a corresponding target fault is generated based on configuration information of the target fault instruction in the first linked list or meta information of the target fault request in the second linked list.

9. The method according to claim 8, characterized in that The method further comprises: In the case where the target fault instruction needs to be submitted to the drive controller, determining the fault subtype of the disk drive fault; In a case where the fault subtype of the disk drive fault is an I / O timeout fault, monitoring a completion request indicating that the I / O request has been completed; When the completion request is detected, writing the completion request into a third linked list; The first linked list and the third linked list are polled, and a corresponding I / O timeout fault is generated based on the configuration information of the target fault instruction in the first linked list, and the I / O timeout fault is used to perform corresponding delayed return processing on the completion request in the third linked list.

10. A fault injection device, characterized in that: The device comprises: An acquisition module, used to obtain fault injection commands; A first determination module, used to determine the fault type of the target fault to be injected according to the fault injection command; A second determination module is used to determine a target injection path corresponding to the fault type from a first injection path and a second injection path, wherein the first injection path is a path between the disk management process and the master control node, and the second injection path is a path between the disk management process and the disk; The fault injection module is used to generate a target fault instruction corresponding to the fault type and execute the target fault instruction based on the target injection path.

Images