Test method and device of storage system, electronic equipment and medium

Abstract

The invention provides a test method and device of a storage system, electronic equipment and a medium. The method comprises the following steps: combining at least one fault in a fault test set with at least one node in a storage platform; in the process that the storage platform operates the target service, according to a preset fault injection strategy, injecting each fault in the at least one fault into a node combined with the fault; test data associated with each fault is collected.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of computer technology, and particularly relates to a test method and device of a storage system, an electronic device and a medium. BACKGROUND

[0002] With the continuous expansion of the scale and increasing complexity of software systems, the reliability of the system becomes a key factor to ensure business continuity and user experience. The traditional reliability testing method verifies the system through pre-set fault scenarios, such as simulating hardware failure, network failure, process failure, resource exhaustion, etc. However, the traditional testing method relies on manual design of pre-set fault scenarios, which is difficult to comprehensively cover all possible fault combinations in the system; due to the complexity and interaction of the system, many potential fault scenarios may be ignored, resulting in test results that cannot truly reflect the reliability of the system. SUMMARY

[0003] To overcome the problems in the related art, the present specification provides a test method and device of a storage system, an electronic device and a medium.

[0004] According to a first aspect of an embodiment of the present specification, a test method of a storage system is provided, the method comprising: combining at least one fault in a fault test set with at least one node in a storage platform; during the process of running a target service in the storage platform, according to a pre-set fault injection strategy, injecting each fault in the at least one fault into the node combined with the fault; collecting test data related to each fault.

[0005] According to a second aspect of an embodiment of the present specification, a test device of a storage system is provided, comprising: a combination module configured to combine at least one fault in a fault test set with at least one node in a storage platform; an injection module configured to, during the process of running a target service in the storage platform, according to a pre-set fault injection strategy, inject each fault in the at least one fault into the node combined with the fault; and a collection module configured to collect test data related to each fault.

[0006] According to a third aspect of an embodiment of the present specification, an electronic device is provided, comprising:

[0007] a processor;

[0008] a memory for storing processor-executable instructions;

[0009] The processor is configured to execute the test method of the storage system of the first aspect or any of its corresponding embodiments.

[0010] According to a fourth aspect of the embodiments of the present specification, a computer readable storage medium is provided, and the computer readable storage medium stores computer instructions for causing a computer to execute the test method of the storage system according to the first aspect or any one of the corresponding embodiments thereof.

[0011] The technical solutions provided by the embodiments of the present specification can include the following beneficial effects:

[0012] In the embodiments of the present specification, by combining the faults in the fault test set with the nodes in the storage platform, fault injection is performed based on the combination, which can more fully cover the fault scenarios, effectively simulate the uncertainty scenarios in the real environment, and thus the test results can more truly reflect the reliability of the system.

[0013] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present specification. BRIEF DESCRIPTION OF DRAWINGS

[0014] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present specification and serve to explain the principles of the present specification together with the specification.

[0015] Figure 1 is a flowchart of a test method of a storage system according to an exemplary embodiment of the present specification.

[0016] Figure 2 is a schematic diagram of a test method of a storage system according to another exemplary embodiment of the present specification.

[0017] Figure 3 is a hardware structure diagram of a computer device in which the test device of the storage system according to the embodiments of the present specification is located.

[0018] Figure 4 is a block diagram of a test device of a storage system according to an exemplary embodiment of the present specification. DETAILED DESCRIPTION

[0019] The exemplary embodiments will be described in detail hereinafter with reference to the attached drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all the embodiments consistent with the present specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present specification, as detailed in the appended claims.

[0020] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in this specification, the singular forms "a," "an" and "the" include plural referents unless the context clearly dictates otherwise. It will further be understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It will be understood that the terms "and / or," "and / or," "at least one of," and "one or more of" as used herein refer to and encompass any one of the indicated possibilities, individual or combined.

[0021] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It will be understood that the terms "and / or," "at least one of," and "one or more of" as used herein refer to and encompass any one of the indicated possibilities, individual or combined.

[0022] Next, the embodiments of the present specification are described in detail.

[0023] The test method of the storage system provided by the embodiments of the present specification is described in detail below. As shown in Figure 1 Figure 1 is a flowchart of a test method of a storage system according to an exemplary embodiment of the present specification. The test method of the storage system provided by the embodiments of the present specification can include the steps as shown below.

[0024] In step 110, at least one fault in the fault test set is combined with at least one node in the storage platform.

[0025] According to the embodiments of the present specification, the fault test set can include at least one fault. The node may, for example, include a storage node, a computing node, a management node, and the like.

[0026] In step 120, during the process of running the target service in the storage platform, each fault in the at least one fault is injected into the node combined with the fault according to a preset fault injection strategy.

[0027] According to the embodiments of the present specification, the target service can be set according to actual test needs. The preset fault injection strategy can be used to indicate the way of fault injection.

[0028] In step 130, test data related to each fault is collected.

[0029] According to the embodiments of the present specification, test points can be set during the fault process and / or when the fault is recovered, the storage system is checked at each test point, and test data is collected.

[0030] ​Exemplarily, the test data can include, for example, whether the fault injection is successful, the fault recovery time, data consistency, service drop zero time, resource usage record (memory, CPU, etc. usage), memory leakage in each module, log assertion, etc. According to an embodiment of the present specification, by combining the faults in the fault test set with the nodes in the storage platform, fault injection based on the combination can cover the fault scenarios more fully and effectively simulate the uncertainty scenarios in the real environment.

[0031] Optionally, the test data can also be summarized and a test report can be output to evaluate the reliability of the storage platform.

[0032] Optionally, the combination of at least one fault in the fault test set and at least one node in the storage platform can adopt a specified combination manner or a random combination manner.

[0033] According to an embodiment of the present specification, at least one fault can be randomly selected from the fault test set and at least one node can be randomly selected from the storage platform. Then, at least one fault and at least one node are randomly combined.

[0034] According to another embodiment of the present specification, at least one node can be specified in the storage platform in advance, which is referred to as a specified node. Based on this, at least one fault can be randomly selected from the fault test set. Then, each of the at least one fault is randomly combined with at least one specified node in the storage platform.

[0035] According to still another embodiment of the present specification, at least one fault can be specified in the fault test set in advance, which is referred to as a specified fault. Based on this, at least one node can be randomly selected from the storage platform. Then, each of the at least one node is randomly combined with at least one specified fault in the fault test set.

[0036] Optionally, for example, a fault test set configuration file can be set in advance, which can include, for example, the fault test set and parameters related to the faults in the fault test set, etc. The parameters can include, for example, the fault type of the fault, the fault parameter range and the fault interval time range, and the process of randomly selecting at least one fault from the fault test set can include, for example: randomly selecting at least one fault type; determining a target fault of the at least one fault type in the fault test set; randomly selecting a fault parameter from the fault parameter range of the target fault as the fault parameter of the target fault; and randomly selecting a fault interval time from the fault interval time range of the target fault as the interval time of the target fault.

[0037] According to the embodiments of the present specification, by automatically combining faults in a specified or random manner, the fault scenarios existing in the storage platform can be more comprehensively traversed and covered, the test results can be automatically analyzed, the test efficiency and test quality can be improved, and the test results can more truly reflect the reliability of the system.

[0038] In addition, by injecting faults (including fault combinations) randomly to simulate unpredictable fault scenarios, the uncertainty in the real environment can be effectively simulated, and the performance of the storage platform in chaotic scenarios can be verified.

[0039] Optionally, the preset fault injection strategy can be, for example, a dense fault injection strategy or a non-dense fault injection strategy.

[0040] In the process of injecting each fault in the at least one fault into the node of the fault combination according to the preset fault injection strategy, if the preset fault injection strategy is a dense fault injection strategy, for the faults other than the first injected fault, the fault is injected before the recovery of the previous fault. If the preset fault injection strategy is a non-dense fault injection strategy, for the faults other than the first injected fault, the fault is injected after the recovery of the previous fault. The non-dense fault injection strategy can be used to manufacture various random chaotic fault scenarios.

[0041] By introducing the dense fault injection strategy and the non-dense fault injection strategy, the reliability of the new version of the storage system can be verified in a short time, and the demand for continuous delivery of rapid version verification can be met.

[0042] Optionally, the faults, the parameters related to the faults, and the nodes of the fault combinations can be recorded in a fault test set configuration file. The faults can be injected into the nodes according to the fault test set configuration file. For example, in the embodiments, the fault test set configuration file can include, for example, node IP, node role, fault type, fault operation, fault parameter range, repetition number, fault interval time (in the case where the repetition number is greater than 1, the fault is repeatedly injected according to the fault interval time), business drop zero duration (or IO drop to 0 duration, i.e., the duration of the storage platform being stuck), selected identifier (used to indicate whether to inject the fault this time), and the like.

[0043] Optionally, for example, an IO model set configuration file can be set in advance, and the IO model set configuration file can include at least one IO model. The IO model can be used to specify the data size of a single read-write operation transmission. Based on this, an IO model can be randomly selected from the IO model set configuration file; and the IO configuration when the storage platform runs the target business can be set according to the IO model, for example, including automatically deploying and running related IO tools, OP (operation) combination scripts, and the like.

[0044] Optionally, for example, a scenario service set can be pre-set. The service model in the scenario service set can customize the operation strategy of the service, such as the running time and the combination range, and the like. Based on this, for example, a target service can be generated according to a service model in the scenario service set, or the target service can be generated according to a combination of multiple service models in the scenario service set.

[0045] Optionally, for example, a test platform configuration file can be pre-set for configuring the storage platform to be tested.

[0046] According to the embodiments of the present specification, in order to solve the problems that the storage system fault test is difficult to comprehensively cover the complex scenarios of users and the test efficiency is low, the chaos engineering method is introduced, combined with the automation technology, the fault scene is randomly generated, including randomly combining the fault type, fault object, fault parameter, fault time, IO model, and the like, to obtain the fault scene, so as to simulate the randomness and diversity of the fault in the real environment of the user. In addition, the intensive fault and non-intensive fault long stability test method can be introduced to solve the demand of version rapid verification and continuous delivery.

[0047] The tester can maintain the fault test set configuration file, customize the faults and related parameters (supporting random and specific value) to be injected. The fault test set configuration file can be a table file, and the table management fault data is intuitive and easy to use. The table file can be, for example, an excel file. The table can include, for example, node IP, node role, fault type, fault operation, parameter (such as range), repetition number, fault interval, drop 0 specification, and the like.

[0048] As shown in FIG. 8, it is a schematic diagram of another test method of a storage system according to an exemplary embodiment. The present embodiment describes a processing procedure of the test of the storage system on the basis of the foregoing embodiment, including the following steps: Figure 2

[0049] 201, configuration analysis: analyze the fault test set configuration file FIT.xlsx file and the test platform (TestBed) configuration file, for the preparation of the following fault injection;

[0050] 202, data preparation: analyze the IO model set configuration file, generate the corresponding IO simulation configuration, and prepare the test scripts and tools. The IO model can be used to specify the data size of a single read-write operation transmission.

[0051] 203, service running: select a service from the scenario service set, run the service on the corresponding storage platform, and configure the service running strategy according to the TestBed configuration file;

[0052] ​204, Fault injection: generate fault related parameters according to the configuration requirements in the fault test set configuration file. During business operation, one or more nodes can be randomly selected to inject random faults, or specified nodes and fault types can be supported. According to the version plan, intensive fault and non-intensive fault strategies are formulated to meet the short-time reliability verification of the version; under the non-intensive fault strategy, the next fault is performed after the environment is restored to normal after the end of the fault; under the intensive fault strategy, the next fault is performed without waiting for the end of the previous fault, aiming to create various random chaos scenes.

[0053] 205, Result analysis: check the test points at the time of the above faults and the time of fault recovery, such as checking data consistency, business zero-time check, resource usage record, memory leakage in each module, log assertion, and other observation items, and backfill the results;

[0054] 206, Output report: test results are summarized and test reports are output for evaluating the reliability of the storage system.

[0055] According to the embodiments of the present specification, monitoring information and running results can be automatically collected and test results can be automatically analyzed.

[0056] Businesses can run and traverse continuously according to test requirements. Faults are continuously injected, and after each fault injection, result analysis is performed, and after the result analysis is completed, the next fault is injected, and the cycle is continued until the test period ends.

[0057] According to the test method of the storage system of the present specification, one-key deployment can be used, and the user has no technical requirements and good usability. In addition, the running results can be automatically analyzed to improve the test efficiency.

[0058] Corresponding to the embodiments of the foregoing method, the present specification also provides embodiments of a test device of a storage system and a terminal to which the test device is applied.

[0059] The embodiments of the test device of the storage system of the present specification can be applied to a computer device, such as a server or a terminal device. The device embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a logically meaningful device, it is formed by a processor reading corresponding computer program instructions in a non-volatile memory into a memory for running. From the hardware level, as shown in Figure 3 Figure 3 ​In addition to the processor 310, the memory 330, the network interface 320, and the non-volatile memory 340 shown, the server or electronic device in which the apparatus 331 is located in embodiments can also include other hardware, as appropriate for the actual functionality of the computer device, which is not shown.

[0060] As shown in Figure 4 Figure 4 is a block diagram of a testing device of a storage system according to an exemplary embodiment, the device comprising:

[0061] a combination module 410, configured to combine at least one fault in the fault test set with at least one node in the storage platform;

[0062] an injection module 420, configured to inject each fault in the at least one fault into the node combined with the fault according to a preset fault injection strategy in a process in which the storage platform runs a target service;

[0063] a collection module 430, configured to collect test data related to each fault.

[0064] Optionally, the combination module can comprise:

[0065] a first selection sub-module, configured to randomly select at least one fault from the fault test set;

[0066] a second selection sub-module, configured to randomly select at least one node from the storage platform;

[0067] a first combination sub-module, configured to randomly combine the at least one fault with the at least one node.

[0068] Optionally, the combination module can comprise:

[0069] a first selection sub-module, configured to randomly select at least one fault from the fault test set;

[0070] a second combination sub-module, configured to randomly combine each fault in the at least one fault with at least one designated node in the storage platform.

[0071] Optionally, the combination module can comprise:

[0072] a third selection sub-module, configured to randomly select at least one node from the storage platform;

[0073] a third combination sub-module, configured to randomly combine each node in the at least one node with at least one designated fault in the fault test set.

[0074] ​Optionally, the fault test set comprises at least one fault, and a fault type, a fault parameter range and a fault interval time range of the fault; the first selection submodule can comprise:

[0075] a type selection unit configured to randomly select at least one fault type;

[0076] a fault determination unit configured to determine a target fault of the at least one fault type in the fault test set;

[0077] a parameter selection unit configured to randomly select a fault parameter from a fault parameter range of the target fault as a fault parameter of the target fault;

[0078] a time interval selection unit configured to randomly select a fault interval time from a fault interval time range of the target fault as an interval time of the target fault.

[0079] Optionally, the preset fault injection strategy is a dense fault injection strategy or a non-dense fault injection strategy; the injection module can comprise:

[0080] a first injection submodule configured to, if the preset fault injection strategy is the dense fault injection strategy, for a fault other than the first injected fault, inject the fault before a recovery of a previous fault of the fault;

[0081] a second injection submodule configured to, if the preset fault injection strategy is the non-dense fault injection strategy, for a fault other than the first injected fault, inject the fault after a recovery of a previous fault of the fault.

[0082] Optionally, the apparatus can further comprise:

[0083] an IO model selection module configured to randomly select an IO model from an IO model set configuration file;

[0084] an IO configuration module configured to set an IO configuration of the storage platform running the target service according to the IO model.

[0085] Correspondingly, the specification also provides an electronic device, which comprises a processor, a memory for storing processor-executable instructions, wherein the processor is configured to: combine at least one fault in a fault test set with at least one node in a storage platform; in a process of the storage platform running a target service, according to a preset fault injection strategy, inject each fault in the at least one fault into the node combined with the fault; and collect test data related to each fault.

[0086] According to the embodiments of the present specification, by combining the faults in the fault test set with the nodes in the storage platform, the fault injection based on the combination can cover the fault scenarios more fully and effectively simulate the uncertainty scenarios in the real environment.

[0087] The implementation process of the functions and roles of each module in the above device is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0088] For the device embodiment, since it basically corresponds to the method embodiment, the related parts can be referred to the part of the method embodiment. The device embodiments described above are only schematic, and the modules shown as separate components can or can not be physically separate, and the components shown as modules can or can not be physical modules, i.e. they can be located in one place or distributed on multiple network modules. Some or all of the modules can be selected to achieve the purpose of the present specification according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0089] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve the desired result. In addition, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are possible or can be advantageous.

[0090] Those skilled in the art will readily conceive other embodiments of the present specification upon considering the specification and practicing the invention claimed herein. The present specification is intended to cover any variations, uses, or adaptations of the present specification that follow the general principles of the present specification and include common general knowledge or conventional technical means in the art not claimed in the present specification. The specification and examples are only considered as exemplary, and the true scope and spirit of the present specification are indicated by the following claims.

[0091] It should be understood that the present specification is not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present specification is limited only by the appended claims.

[0092] The above only describes the preferred embodiments of the present specification, and does not limit the present specification, and any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present specification shall be included in the scope of protection of the present specification.

Claims

1. A testing method for a storage system, characterized in that, The method includes: Combine at least one fault from the fault test set with at least one node from the storage platform; During the operation of the target service on the storage platform, each of the at least one faults is injected into the node that is combined with the fault, according to a preset fault injection strategy. Collect test data associated with each of the aforementioned faults.

2. The method according to claim 1, characterized in that, The step of combining at least one fault in the fault test set with at least one node in the storage platform includes: Randomly select at least one fault from the fault test set; Randomly select at least one node from the storage platform; The at least one fault is randomly combined with the at least one node.

3. The method according to claim 1, characterized in that, The step of combining at least one fault in the fault test set with at least one node in the storage platform includes: Randomly select at least one fault from the fault test set; Each of the at least one faults is randomly combined with at least one designated node in the storage platform.

4. The method according to claim 1, characterized in that, The step of combining at least one fault in the fault test set with at least one node in the storage platform includes: Randomly select at least one node from the storage platform; Each of the at least one node is randomly combined with at least one specified fault in the fault test set.

5. The method according to claim 2 or 3, characterized in that, The fault configuration in the fault test set includes fault type, fault parameter range, and fault interval time range. The step of randomly selecting at least one fault from the fault test set includes: Randomly select at least one fault type; Determine the target fault of at least one fault type from the fault test set; Randomly select a fault parameter from the range of fault parameters of the target fault, and use it as the fault parameter of the target fault; Randomly select a fault time interval from the range of fault intervals of the target fault, and use it as the interval of the target fault.

6. The method according to claim 1, characterized in that, The preset fault injection strategy is either a dense fault injection strategy or a non-dense fault injection strategy. The step of injecting each of the at least one faults into a node combined with the faults according to a preset fault injection strategy includes: If the preset fault injection strategy is a dense fault injection strategy, then for faults other than the first injected fault, the fault will be injected before the previous fault is recovered. If the preset fault injection strategy is a non-intensive fault injection strategy, then for faults other than the first injected fault, the fault will be injected after the previous fault is recovered.

7. The method according to claim 1, characterized in that, The method further includes: Randomly select an I / O model from the input / output I / O model set configuration file; Based on the IO model, configure the IO settings of the storage platform when running the target service.

8. A testing apparatus for a storage system, characterized in that, The device includes: A combination module is used to combine at least one fault in the fault test set with at least one node in the storage platform; An injection module is used to inject each of the at least one faults into a node that combines the faults, according to a preset fault injection strategy, during the operation of the target service on the storage platform. The collection module is used to collect test data related to each of the faults.

9. An electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to perform a test method for the storage system according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the test method of the storage system according to any one of claims 1 to 7.

Images