Storage system I / O path tracking and fault testing method, system and equipment

Abstract

The invention relates to the technical field of storage fault detection, and provides a storage system I / O path tracking and fault testing method, system and equipment, and the method comprises the steps: generating a test file through a client, and writing the generated test file as a test I / O into a storage cluster; the written test I / O is stored through the key service, and test I / O inflow information is tracked and collected at a key service entry; test I / O inflow information is collected through a client, a fault decision is made according to the collected test I / O inflow information, and a fault is injected into the storage cluster according to the made fault decision; and judging whether the storage cluster with the injection fault can be self-recovered or not, and deciding whether to carry out the next round of fault injection or not according to a judgment result. According to the method, the information can be automatically collected and sent to the client to decide the fault point, the robustness of the storage system facing the fault is tested, the weaknesses existing in the storage system are accurately identified and solved, and the fault tolerance and the restorability of the storage system are improved.

Description

Technical Field

[0001] The present application relates to the technical field of storage fault detection, and in particular to a storage system I / O path tracking and fault testing method, system and device. Background Art

[0002] With the rapid development of information technology, storage systems have become an indispensable part of modern computing environments. They not only need to process the growing amount of data, but also must ensure the reliability of data and the stability of the system. In this context, it is particularly important to test the stability and robustness of storage systems. Traditional testing methods mainly rely on simulating multiple real failure scenarios and injecting these scenarios into the storage cluster to evaluate the system's performance in the face of abnormal situations.

[0003] However, these traditional methods have some limitations. First, they usually do not involve correlation with storage I / O traces, which means that the test may not accurately simulate the behavior of the system under the actual workload. Second, due to the lack of in-depth analysis of I / O patterns, these methods may not be able to accurately identify and resolve possible weaknesses in the system before the production environment. This may lead to unforeseen problems in actual deployment, thus affecting the reliability and performance of the system.

[0004] Therefore, how to provide a new testing method that closely combines fault injection with storage I / O traces to more accurately simulate actual workloads, identify and resolve potential problems before the production environment, improve the stability and robustness of the storage system, and ensure that the system can maintain normal operation in the face of various failure scenarios, thereby protecting critical data from loss, has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of this, in order to overcome the deficiencies of the prior art, the present application aims to provide a storage system I / O path tracking and fault testing method, system and device.

[0006] According to a first aspect of the present application, a storage system I / O path tracing and fault testing method is provided, the method comprising: Generate a test file through the client, and write the generated test file into the storage cluster as a test I / O; The test I / O written by the key service storage is tracked and collected at the key service entrance; Collect test I / O inflow information through the client, make a fault decision based on the collected test I / O inflow information, and inject a fault into the storage cluster based on the made fault decision; Determine whether the storage cluster with injected faults can recover by itself, and decide whether to perform the next round of fault injection based on the judgment result.

[0007] Optionally, in the storage system I / O path tracking and fault testing method of the present application, generating a test file through a client, and writing the generated test file into a storage cluster as a test I / O includes: Generate a marked test file through the client and send a test file write request to the storage cluster; The storage cluster receives the test file write request sent by the client, and writes the marked test file as a test I / O to multiple storage servers of the storage cluster through a hash algorithm.

[0008] Optionally, in the storage system I / O path tracking and fault testing method of the present application, the test I / O written by the key service storage is tracked and collected at the key service entrance. The information of the test I / O inflow includes: Each storage server uses multiple key services to store the written test I / O; An information collection point is set at the entrance of each key service, and the set information collection point is used to periodically track and test the path of I / O passing through the key service.

[0009] Optionally, in the storage system I / O path tracking and fault testing method of the present application, the test I / O written by the key service storage is tracked and collected at the key service entrance. The information of the test I / O inflow includes: When the information collection point at the key service entrance detects a test I / O inflow within the detection period, the test I / O inflow information is returned to the client, and the test I / O inflow information includes the storage server name, the key service name, the test I / O inflow time and the test I / O information; When the information collection point of the key service entrance does not detect the test I / O inflow within the detection cycle, the detection failure information is returned to the client, and the detection failure information includes the storage server name, key service name, test I / O timeout and test I / O information.

[0010] Optionally, in the storage system I / O path tracking and fault testing method of the present application, the test I / O written by the key service storage is tracked and collected at the key service entrance. The method also includes: A detection flag is set at each information collection point, and the client uses the detection flag to open or close the information collection point of the key service entrance; Each time the client writes a marked test file, a corresponding round of information collection is performed.

[0011] Optionally, in the storage system I / O path tracking and fault testing method of the present application, collecting test I / O inflow information through a client, formulating a fault decision according to the collected test I / O inflow information, and injecting a fault into the storage cluster according to the formulated fault decision, including: The client collects the test I / O inflow information returned by the storage server of the storage cluster, stores the collected test I / O inflow information in the information collection set, and stores the storage cluster fault mode library in the fault set; Orthogonal processing is performed on the information collection set and the fault set to obtain a target fault mode library. In each round of information collection, a fault mode is selected from the target fault mode library through a random selection algorithm, and the selected fault mode is stored in the fault mode set. Inject faults into the storage cluster based on the fault mode, and shut down all information collection points through the client.

[0012] Optionally, in the storage system I / O path tracking and fault testing method of the present application, in each round of information collection, a fault mode is selected from the target fault mode library by a random selection algorithm, and the selected fault mode is stored in a fault mode set, including: before storing the fault mode in the fault mode set, comparing the fault mode to be stored with the fault modes already stored in the fault mode set; when the fault mode to be stored is different from the fault modes already stored in the fault mode set, storing the fault mode to be stored in the fault mode set; when the fault mode to be stored is the same as the fault mode already stored in the fault mode set, abandoning the storage of the fault mode.

[0013] Optionally, in the storage system I / O path tracking and fault testing method of the present application, judging whether the storage cluster into which the fault is injected can self-recover, and deciding whether to perform the next round of fault injection according to the judgment result, include: Set the storage cluster recovery time threshold. If the storage cluster with injected fault recovers within the set storage cluster recovery time threshold, end the current round of fault testing and enable the information collection point for the next round of fault testing. If the storage cluster injected with the fault does not recover within the set storage cluster recovery time threshold, an error event is reported.

[0014] According to a second aspect of the present application, a storage system I / O path tracing and fault testing system is provided, the system comprising: The client is used to generate a test file and write the generated test file into the storage cluster as a test I / O; collect the test I / O inflow information, make a fault decision based on the collected test I / O inflow information, and inject the fault into the storage cluster based on the fault decision; determine whether the storage cluster with the injected fault can recover by itself, and decide whether to perform the next round of fault injection based on the determination result; The storage cluster includes multiple storage servers. Each storage server includes multiple key services. The key services are used to store the test I / O written, and track and collect the test I / O inflow information at the key service entrance.

[0015] According to a third aspect of the present application, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect of the present application when executing the program.

[0016] The storage system I / O path tracking and fault testing method and system of the present application bind the storage system failure mode with the I / O storage track, and accurately test the robustness of the storage system by performing fixed-point collection of the storage system I / O path and the client independently deciding to inject faults. In practical applications, the storage system I / O path tracking and fault testing method and system of the present application does not require manual setting of fault points, and can automatically collect information and hand it over to the client to decide on the fault point; by purposefully introducing chaos and unstable factors in the storage system, the robustness of the storage system in the face of faults is tested; it helps to accurately identify and resolve possible weaknesses in the system before or during the production environment, and improve the fault tolerance and recoverability of the storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is an example diagram of the architecture of a storage system I / O path tracking and fault testing system according to an embodiment of the present application; Figure 2 A flowchart of a storage system I / O path tracking and fault testing method according to an embodiment of the present application; Figure 3 exemplifying the execution of the storage system I / O path tracing and fault testing method according to an embodiment of the present application; Figure 4 A schematic diagram of the structure of the device provided in this application. DETAILED DESCRIPTION

[0019] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0020] It should be noted that the following embodiments and features in the embodiments may be combined with each other in the absence of conflict; and, based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making any creative work are within the scope of protection of the present disclosure.

[0021] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0022] This application aims to provide an improved testing method that associates fault injection with storage I / O traces to achieve a more accurate assessment of the stability and robustness of the storage system. With this method, the behavior of the system under actual workloads can be better understood, and potential weaknesses can be identified and fixed before deployment, thereby improving the overall performance and reliability of the storage system. A marked test file is generated by the client, written into the storage system, and collection points are set on key components of the storage system. Once the target I / O is detected, the information is actively reported to the client. The client collects information and autonomously decides what kind of fault to inject into what service of the storage cluster to observe the cluster performance to accurately verify the robustness of the system.

[0023] Figure 1 FIG. 1 is an example diagram of the architecture of a storage system I / O path tracking and fault testing system according to an embodiment of the present application, such as Figure 1 As shown, the system may include: The client is used to generate a test file and write the generated test file into the storage cluster as a test I / O; collect the test I / O inflow information, make a fault decision based on the collected test I / O inflow information, and inject the fault into the storage cluster based on the fault decision; determine whether the storage cluster with the injected fault can recover by itself, and decide whether to perform the next round of fault injection based on the determination result; The storage cluster includes multiple storage servers. Each storage server includes multiple key services. The key services are used to store the test I / O written, and track and collect the test I / O inflow information at the key service entrance.

[0024] Based on the above system, an embodiment of the present application provides a storage system I / O path tracking and fault testing method, which is described below through the following embodiments.

[0025] Figure 2 FIG. 1 is a flowchart of a method for tracing and testing I / O paths in a storage system according to an embodiment of the present application. Figure 2 As shown, the storage system I / O path tracking and fault testing method includes the following steps: Step S101: Generate a test file through a client, and write the generated test file into a storage cluster as a test I / O.

[0026] As an optional example, in this embodiment, a marked test file is generated by the client, and a test file write request is sent to the storage cluster; the storage cluster receives the test file write request sent by the client, and writes the marked test file as a test I / O to multiple storage servers of the storage cluster through a hash algorithm.

[0027] Step S102: The test I / O written by the key service storage is tracked and collected at the key service entrance.

[0028] In this embodiment, each storage server uses multiple key services to store the written test I / O; an information collection point is set at the entrance of each key service, and the set information collection point is used to periodically track the path of the test I / O through the key service.

[0029] As an optional example, when the information collection point of the key service entrance detects a test I / O inflow within the detection cycle, the test I / O inflow information is returned to the client, and the test I / O inflow information includes the storage server name, the key service name, the test I / O inflow time and the test I / O information; when the information collection point of the key service entrance does not detect the test I / O inflow within the detection cycle, the detection failure information is returned to the client, and the detection failure information includes the storage server name, the key service name, the test I / O timeout time and the test I / O information.

[0030] In this embodiment, a detection flag is set at each information collection point, and the client opens or closes the information collection point of the key service entrance through the detection flag; each time the client writes a marked test file, a corresponding round of information collection is performed.

[0031] Step S103: collecting test I / O inflow information through the client, formulating a fault decision according to the collected test I / O inflow information, and injecting a fault into the storage cluster according to the formulated fault decision.

[0032] As an optional example, in this embodiment, the client collects the test I / O inflow information returned by the storage server of the storage cluster, stores the collected test I / O inflow information in the information collection set, and stores the storage cluster fault mode library in the fault set. It should be noted that before storing the fault mode in the fault mode set, this embodiment compares the fault mode to be stored with the fault modes already stored in the fault mode set. When the fault mode to be stored is different from the fault modes already stored in the fault mode set, the fault mode to be stored is stored in the fault mode set. When the fault mode to be stored is the same as the fault mode already stored in the fault mode set, the storage of the fault mode is abandoned.

[0033] In this embodiment, after the storage cluster fault mode library is stored in the fault set, the information collection set and the fault set are orthogonally processed to obtain the target fault mode library. In each round of information collection, a fault mode is selected from the target fault mode library through a random selection algorithm, and the selected fault mode is stored in the fault mode set; faults are injected into the storage cluster according to the fault mode, and all information collection points are closed through the client.

[0034] Step S104: determine whether the storage cluster with injected faults can recover by itself, and decide whether to perform the next round of fault injection based on the determination result.

[0035] As an optional example, in this embodiment, a storage cluster recovery time threshold is set. If the storage cluster into which the fault is injected recovers within the set storage cluster recovery time threshold, the current round of fault testing is ended, and the information collection point is enabled for the next round of fault testing. If the storage cluster into which the fault is injected does not recover within the set storage cluster recovery time threshold, an error event is reported.

[0036] The storage system I / O path tracing and fault testing method of this embodiment is described in detail below in a specific scenario. Figure 3 FIG. 1 is an example diagram of executing the storage system I / O path tracking and fault testing method according to an embodiment of the present application, such as Figure 3 As shown, in the scenario of this example, the storage system I / O path tracking and fault testing method of this embodiment is performed according to the following steps: 1. The client generates a marked test file so that the tested file can be detected later.

[0037] 2. The client writes the test file to the storage cluster, and provides an information collection mechanism to monitor the marked file I / O. Specifically: 1) In the distributed storage cluster, a test I / O write request is received from the client, and the test file is written to multiple storage servers (hereinafter referred to as nodes) using a hash algorithm; 2) There are multiple key services in each node to complete the storage of test I / O data, such as Figure 4 There are services A, B, and C to cooperate in completing the writing; 3) Set up a test I / O information collection point at the entrance of each key service and perform periodic detection to track the path of the test I / O through the key service; 4) During the detection cycle of each key service entrance, after detecting the test I / O inflow, a message is returned to the client, including the cluster node name, service name, detection I / O inflow time, and test I / O information (such as node 1, service A, time time, and file tag); if the test I / O inflow is not detected during the detection cycle, a detection failure message is returned to the client, including the cluster node name, service name, detection I / O timeout time, and test I / O information (such as node 1, service A, time timeout, and file tag); 5) At the same time, set the detection flag at each sampling point. If the client wants to close the entry information collection of a certain service, it can be adjusted through this switch; 6) According to the test I / O collection mechanism, the three nodes execute the same mechanism and the collected information is returned to the client. Each time the client writes a marked test file, it is regarded as a round of information collection for the cluster.

[0038] 3. The client sets up a fault decision mechanism, and decides what kind of fault to handle for a certain node based on the returned collected information. Specifically: 1) Collect the messages returned by all detection points in the storage cluster and store them in set M; 2) Store the storage cluster failure mode library (including network, process, node and other failure modes) in set N; 3) The set M and the set N are combined by the orthogonal array method to obtain the target fault mode library F, namely, the fault mode, fault cluster node, and fault service; and a fault mode F1 is selected from F through a random selection algorithm, such as faulting the process of service A of node 1; 4) Put the fault mode F1 selected in each round into a set D. Compare the fault mode F1 in the next round with the fault modes in set D. F1 is not included in D, that is, the fault mode in each round is different. 5) Inject a fault into the storage cluster according to the fault method of F1, and at the same time, the client sends a message to shut down the information collection of all sampling points.

[0039] 4. After the fault injection, the cluster health detection mechanism is relied upon to detect whether the system can recover by itself, so as to decide whether to perform the next round of fault injection. Specifically: 1) After the fault is injected, if the cluster recovers within T time, the current round of fault testing is completed, and the information collection is reported to start the next round of fault testing; 2) If the cluster recovery time is greater than T, an error event is reported to inform manual cluster troubleshooting to confirm whether a storage system error is detected.

[0040] The storage system I / O path tracking and fault testing method and system of this embodiment binds the storage system failure mode to the I / O storage track, and through fixed-point collection of the storage system I / O path, the client independently decides to inject faults, and accurately tests the robustness of the storage system. In practical applications, the storage system I / O path tracking and fault testing method and system of this embodiment does not require manual setting of fault points, and can automatically collect information and hand it over to the client to decide on the fault point; by purposefully introducing chaos and unstable factors in the system, the robustness of the storage system in the face of faults is tested; it helps to accurately identify and resolve possible weaknesses in the storage system before or during the production environment, and improve the fault tolerance and recoverability of the storage system.

[0041] like Figure 4 As shown, the present application also provides a device, including a processor 210, a communication interface 220, a memory 230 for storing a computer program executable by the processor, and a communication bus 240. The processor 210, the communication interface 220, and the memory 230 communicate with each other through the communication bus 240. The processor 210 implements the above-mentioned storage system I / O path tracking and fault testing method by running the executable computer program.

[0042] Among them, the computer program in the memory 230 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0043] The system embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected based on actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0044] Through the description of the above implementation modes, those skilled in the art can clearly understand that each implementation mode can be implemented by means of software plus a necessary general hardware platform, or of course by hardware. Based on such an understanding, the above technical solution can essentially or in other words be embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0045] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A storage system I / O path tracking and fault testing method, characterized in that: The method comprises: Generate a test file through the client, and write the generated test file into the storage cluster as a test I / O; The test I / O written by the key service storage is tracked and collected at the key service entrance; Collect test I / O inflow information through the client, make a fault decision based on the collected test I / O inflow information, and inject a fault into the storage cluster based on the made fault decision; Determine whether the storage cluster with injected faults can recover by itself, and decide whether to perform the next round of fault injection based on the judgment result.

2. The storage system I / O path tracing and fault testing method according to claim 1, characterized in that: Generate a test file through the client and write the generated test file to the storage cluster as a test I / O, including: Generate a marked test file through the client and send a test file write request to the storage cluster; The storage cluster receives the test file write request sent by the client, and writes the marked test file as a test I / O to multiple storage servers of the storage cluster through a hash algorithm.

3. The storage system I / O path tracing and fault testing method according to claim 1, characterized in that: The test I / O written by the key service storage is tracked and collected at the key service entrance, including: Each storage server uses multiple key services to store the written test I / O; An information collection point is set at the entrance of each key service, and the set information collection point is used to periodically track and test the path of I / O passing through the key service.

4. The storage system I / O path tracing and fault testing method according to claim 1, characterized in that: The test I / O written by the key service storage is tracked and collected at the key service entrance, including: When the information collection point at the key service entrance detects a test I / O inflow within the detection period, the test I / O inflow information is returned to the client, and the test I / O inflow information includes the storage server name, the key service name, the test I / O inflow time and the test I / O information; When the information collection point of the key service entrance does not detect the test I / O inflow within the detection cycle, the detection failure information is returned to the client, and the detection failure information includes the storage server name, key service name, test I / O timeout and test I / O information.

5. The storage system I / O path tracing and fault testing method according to claim 1, characterized in that: It is characterized in that The test I / O written by the key service storage is tracked and collected at the key service entrance. It also includes: A detection flag is set at each information collection point, and the client uses the detection flag to open or close the information collection point of the key service entrance; Each time the client writes a marked test file, a corresponding round of information collection is performed.

6. The storage system I / O path tracing and fault testing method according to claim 1, characterized in that: Collect test I / O inflow information through the client, make a fault decision based on the collected test I / O inflow information, and inject a fault into the storage cluster based on the made fault decision, including: The client collects the test I / O inflow information returned by the storage server of the storage cluster, stores the collected test I / O inflow information in the information collection set, and stores the storage cluster fault mode library in the fault set; Orthogonal processing is performed on the information collection set and the fault set to obtain a target fault mode library. In each round of information collection, a fault mode is selected from the target fault mode library through a random selection algorithm, and the selected fault mode is stored in the fault mode set. Inject faults into the storage cluster based on the fault mode, and shut down all information collection points through the client.

7. The storage system I / O path tracing and fault testing method according to claim 6, characterized in that: In each round of information collection, a fault mode is selected from the target fault mode library through a random selection algorithm, and the selected fault mode is stored in a fault mode set, including: before storing the fault mode in the fault mode set, comparing the fault mode to be stored with the fault modes stored in the fault mode set; when the fault mode to be stored is different from the fault modes stored in the fault mode set, storing the fault mode to be stored in the fault mode set; when the fault mode to be stored is the same as the fault mode stored in the fault mode set, abandoning the storage of the fault mode.

8. The storage system I / O path tracing and fault testing method according to claim 1, characterized in that: Determine whether the storage cluster with injected faults can recover by itself, and decide whether to perform the next round of fault injection based on the judgment result, including: Set the storage cluster recovery time threshold. If the storage cluster with injected fault recovers within the set storage cluster recovery time threshold, end the current round of fault testing and enable the information collection point for the next round of fault testing. If the storage cluster injected with the fault does not recover within the set storage cluster recovery time threshold, an error event is reported.

9. A storage system I / O path tracking and fault testing system, characterized in that: The system comprises: The client is used to generate a test file and write the generated test file into the storage cluster as a test I / O; collect the test I / O inflow information, make a fault decision based on the collected test I / O inflow information, and inject the fault into the storage cluster based on the fault decision; determine whether the storage cluster with the injected fault can recover by itself, and decide whether to perform the next round of fault injection based on the determination result; The storage cluster includes multiple storage servers. Each storage server includes multiple key services. The key services are used to store the test I / O written, and track and collect the test I / O inflow information at the key service entrance.

10. A computer device, characterized in that: The computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method according to any one of claims 1 to 8 when executing the program.

Images