Method, electronic device and computer program product for testing a storage system

Abstract

Embodiments of the present disclosure relate to a method, an electronic device and a computer program product for testing a storage system. The method comprises: obtaining a result of performing a first test on the storage system by using a test case; if the result indicates that the storage system fails the first test, performing a second test on the storage system based on a problem existing in the storage system; and if the result indicates that the storage system passes the first test, determining a security level of the test case based on the result. Embodiments of the present disclosure can effectively improve the testing performance and system reliability of the storage system.

Description

Technical Field

[0001] The embodiments of this disclosure generally relate to the field of testing storage systems, and more specifically to methods, electronic devices, and computer program products for testing storage systems. Background Technology

[0002] In the storage systems industry, customers often want to increase the load on their storage systems. However, under higher or extreme loads, storage systems can become unreliable or even unavailable. For example, modern storage systems can become unreliable under complex configurations, multiple workloads and system resource pressures, numerous feature interactions, and unexpected single failure insertions. If a storage system becomes unreliable, it may encounter many system-level problems during operation, such as system emergencies, data unavailability or data loss (DUDL), and significant performance degradation.

[0003] Customers of storage systems are eager to know the extent to which they should load their systems when they encounter problems in complex scenarios, but this question is often difficult to answer. Because reliability is very different from functionality and simple performance metrics, it is difficult to determine reliability using predefined test sets. Summary of the Invention

[0004] Embodiments of this disclosure provide methods, electronic devices, and computer program products for testing storage systems.

[0005] In a first aspect of this disclosure, a method for testing a storage system is provided. The method includes: obtaining the result of performing a first test on the storage system using test cases; if the result indicates that the storage system has failed the first test, performing a second test on the storage system based on the identified problems; and if the result indicates that the storage system has passed the first test, determining a security level for the test cases based on the result.

[0006] In a second aspect of this disclosure, an electronic device is provided, comprising: at least one processing unit; at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform an action, the action comprising: obtaining the result of performing a first test on a storage system using test cases; if the result indicates that the storage system has failed the first test, performing a second test on the storage system based on a problem existing in the storage system; and if the result indicates that the storage system has passed the first test, determining a security level of the test cases based on the result.

[0007] In a third aspect of this disclosure, a computer program product is provided. The computer program product is tangibly stored in a non-transitory computer storage medium and includes machine-executable instructions. When executed by a device, the machine-executable instructions cause the device to perform any step of the method described in the first aspect of this disclosure.

[0008] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or essential features of this disclosure, nor is it intended to limit the scope of this disclosure. Attached Figure Description

[0009] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0010] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0011] Figure 2 A flowchart illustrating an example method for testing a storage system according to an embodiment of the present disclosure is shown;

[0012] Figure 3 A flowchart illustrating an example method for expanding the scope of test cases to determine a security level, according to embodiments of the present disclosure, is shown.

[0013] Figure 4 A flowchart illustrating an example method for narrowing the scope of test cases to perform tests on a storage system, according to embodiments of the present disclosure, is shown; and

[0014] Figure 5 A schematic block diagram of an example device that can be used to implement embodiments of the present disclosure is shown.

[0015] In the various figures, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0016] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0017] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0018] As mentioned above, storage systems can become unreliable or even unavailable under higher or extreme loads. For example, modern storage systems can become unreliable under complex configurations, multiple workloads and system resource pressures, numerous feature interactions, and unexpected single fault insertions. In such situations, unexpected problems may occur, such as system emergency / reboots, which can lead to DUDL (Duration-Extended Flow) or a precipitous drop in performance. Customers of storage systems encountering problems in complex scenarios are eager to know to what extent they should load their systems, but this question is often difficult to answer, even for research and development personnel. In many cases, when customers encounter problems in complex scenarios, electrical engineers have to consult research and development teams, which often requires the efforts of different testing teams, such as searching through hundreds of system test scenarios to see if similar scenarios have been tested, or attempting to reproduce the customer's scenario later on the test server to aid in the investigation. Therefore, this technical field needs to document, summarize, and compare effective and rapid integration methods for a large number of system test scenarios and customer scenarios.

[0019] Embodiments of this disclosure provide a scheme for testing storage systems to address one or more of the aforementioned problems and other potential issues. According to embodiments of this disclosure, test cases can be used to perform tests on the storage system, and the security level of the test cases can be determined based on the test results, or the storage system can be retested. The method for testing storage systems disclosed in this disclosure can effectively improve the testing performance and system reliability of storage systems.

[0020] The following will be referenced Figures 1 to 5 The embodiments of this disclosure are described in detail. Figure 1 A block diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. (See diagram for example.) Figure 1 As shown, environment 100 includes computing device 110 and storage system 120. It should be understood that the structure of environment 100 is described for illustrative purposes only and does not imply any limitation on the scope of this disclosure. For example, embodiments of this disclosure can also be applied to environments different from environment 100.

[0021] The computing device 110 can be, for example, any physical computer, virtual machine, server, etc., running the user application. The storage system 120 can, for example, send a request to the computing device 110 to execute a test and scenario-based runtime data, and / or receive test results for multiple test cases from the computing device 110. In response to receiving a request to execute a test from the storage system 120, the computing device 110 can read scenario-based runtime data from the storage system 120. The storage system 120 can include any currently known or future-developed non-volatile storage media, such as hard disks, solid-state drives (SSDs), or disk arrays.

[0022] The computing device 110 can perform tests on the storage system 120 based on multiple test cases. For example, when the computing device 110 receives a request to perform tests from the storage system 120, it can perform tests on the storage system 120 based on multiple test cases, determine the security level of the test cases, and provide the test results to the storage system 120. Furthermore, the computing device 110 can adjust the test cases based on the test results, accurately determining the boundaries of the tests by expanding or narrowing the scope of the test cases.

[0023] exist Figure 1 In the illustrated embodiment, the computing device 110 may utilize the test module 111 deployed therein to implement the aforementioned testing process and / or test case adjustment process. It should be understood that the test module 111 is merely exemplary and not limiting, and the computing device 110 may also deploy other suitable modules or components to implement the aforementioned tests.

[0024] The technical solution disclosed herein can cover as many test cases as possible based on multiple predefined test scenario dimensions, in order to generate the security level (also known as the system security boundary) of test cases using a limited number (e.g., hundreds) of tests. In this way, the solution can not only help developers improve system reliability during the development phase, but also effectively reflect the system reliability of the client, especially when the system security boundary of the storage system is automatically updated from its early development phase to the production-ready version (RTM version).

[0025] The technical solution disclosed herein is based on testing a storage system, also known as system testing. Testing a storage system requires meeting the following specific requirements. First, the system testing involved in this paper is implemented using multiple test cases, a testing process distinct from functional testing. Test cases are generated based on predefined test scenario dimensions. Test cases are scenario-based, and each test scenario can include various input / output (IO) workloads, mixed use of features, mixed system operations, etc. It should be understood that in this paper, test cases and test scenarios are interchangeable.

[0026] Secondly, test scenario attributes typically involve concurrency, repeatability, randomness, and long duration. In other words, when generating test scenarios, attributes such as concurrency, repeatability, randomness, and long duration usually need to be considered. Concurrency increases the diversity and complexity of testing. A test scenario typically includes multiple features. When focusing on one feature, it's necessary to consider, for example, whether the feature's existence affects other features, whether it coexists with other features, and how the customer expects to use the feature. Repeatability refers to repeatedly performing an operation, such as turning a feature on or off, creating or deleting objects, expanding or shrinking storage objects, etc. Randomness broadens the product's coverage. For example, adding randomness to test scenarios can include randomly changing IO data patterns, randomizing the start and stop schedules of test components, and performing random operations on storage objects. Long duration relates to the execution time of the test scenario; for example, test scenario execution typically lasts three to five days, or even more than a month.

[0027] Furthermore, the scope of each test case or test scenario should be scalable and / or scalable. This means that after a test passes multiple times, the scope of the test case or test scenario can be expanded and then tested again. If a test fails, or if a test consistently fails, the scope of the test case or test scenario can be scaldped down in a specific way and then tested again.

[0028] The above briefly introduced the conditions that test cases or test scenarios typically need to meet. Next, we will combine... Figures 2-4 The various aspects of the technology disclosed herein are discussed in detail.

[0029] Figure 2 A flowchart illustrating an example method 200 for testing a storage system according to an embodiment of the present disclosure is shown schematically. For example, method 200 may be performed by, for example... Figure 1 The method is executed by the computing device 110 shown. It should be understood that method 200 may also include additional actions not shown and / or the actions shown may be omitted; the scope of this disclosure is not limited in this respect. The following is in conjunction with… Figure 1Let me describe method 200 in detail.

[0030] At box 210, computing device 110 obtains the result of performing a first test on the storage system using test cases (also referred to herein as the “first result”).

[0031] In some embodiments, in response to receiving a request to perform a test and scenario-based runtime data from the storage system 120, the computing device 110 may first automatically generate test cases suitable for a specific scenario based on the scenario-based runtime data received from the storage system 120, and then perform tests on the storage system 120 using the test module 111 and the test cases. In some embodiments, as hundreds of system test cases are executed in the test module 111, the result of each test is acquired, and then the test result is used in specific aspects (e.g., over many test cycles) to refine, improve, and rerun the test case. The computing device 110 may also obtain test cases suitable for a specific scenario from an existing test case library.

[0032] After comparing customer scenario usage data with system test status, multiple dimensions of the test scenario can be defined, also known as test scenario dimensions. For example, embodiments of this disclosure refer to the following test scenario for description. This test scenario can have several independent dimensions, such as object configuration, external workload, system resource usage, and feature usage; these four dimensions are commonly used. The test scenario may also additionally include a filtering factor dimension. In the context of this disclosure, this is also referred to as 4+1 dimensions. It should be understood that the above test scenarios and their dimensions are exemplary and not limiting, and embodiments of this disclosure can also be implemented using other appropriate test scenarios and / or dimensions.

[0033] Storage systems typically have a large variety of objects, such as drives, logical unit numbers (LUNs), file snapshots, replication sessions, migration sessions, online compression (ILC) objects / online deduplication (ILD) objects, etc. These objects have system limits in terms of configuration; that is, the values ​​of these objects should fall within the range of [0, the object's maximum value]. It should be understood that a configurable object at least means that the number and / or size of the object is configurable; in other words, the number and / or size of the object is scalable and / or scalable.

[0034] In the external workload dimension, such as IO workload and related loads loaded onto the storage system, the metrics of external workloads should usually be defined by performance testing, such as bandwidth, input / output operations per second (IOPS), response time, etc.

[0035] System resource usage refers to the status of system resource usage, including CPU utilization, operating system (OS) memory usage, and system reserved memory usage.

[0036] In the feature usage dimension, the features involved are those enabled or used by the storage system, namely security-related features, NAS-related features or operations, etc. It should be understood that these features are configurable, that is, at least the number and / or size of the features are configurable; in other words, the number and / or size of the features are scalable and / or scalable.

[0037] Other factors that are difficult to categorize into the above four dimensions, or other factors that are more suitable to be considered as filtering factors, are included in the "Filtering Factors" dimension. Filtering factors include models, specific IO parameters, etc.

[0038] It should be noted that the definitions of each dimension above are for illustrative purposes only and do not imply any limitation on the scope of this disclosure. Similarly, each dimension has configurable characteristics. This means that, in a particular respect, the configurable characteristics on each dimension are scalable and / or scalable. Furthermore, it should be understood that, for a specific test scenario, such as a real-world client-side storage system scenario (the client scenario), features (sometimes also called items) in any of the above dimensions can be defined and used as needed to adapt to different testing requirements. For example, for the object configuration dimension, all kinds of objects that the client wants to interact with can be features in the test scenario.

[0039] To facilitate comparison between customer scenario usage data and system test status, each feature in the test scenario can be standardized. As an example, and not a limitation, "Number of LUNs" in the object configuration dimension and "Total IOPS" in the external workload dimension are selected as two features / items in a test case or test scenario. For items with explicit limits, the limit can be specified as 100. For example, "Number of LUNs," if the system-defined limit for this item is 2000, the number of LUNs can be standardized to a fraction between [0, 100], where 2000 LUNs correspond to a fraction of 100, and 1000 LUNs correspond to a fraction of 50. For items without explicitly defined limits, the maximum value can be specified as a fraction of 100. For example, "Total IOPS," if this item reaches a maximum value of 200,000, the total IOPS can be standardized to a fraction between [0, 100], where a total IOPS of 200,000 corresponds to a fraction of 100. Note that this item is updatable; that is, its maximum value is updatable, and when a new maximum value is reached, the range of its fraction is also updated. It is important to note that the standardization steps described above are not mandatory, but rather for ease of description and intuitive comparison. The absence of standardization steps or the use of different standardization methods will not affect the final comparison between customer scenario usage data and system test results, and therefore will not affect the load recommendation for the customer's system.

[0040] In some embodiments, for example, computing device 110 may categorize and organize test results according to test scenarios, such as collecting metric data for specific test scenario dimensions based on multiple dimensions of the test scenario. For example, the test scenario described herein may have the aforementioned 4+1 independent dimensions. It should be understood that the specific test scenario dimensions described above are given for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0041] At box 220, computing device 110 determines whether the result of performing the first test indicates that the storage system has passed the first test. The term "result" may refer to, for example, whether specific characteristics of the storage system are present in a particular scenario, whether the storage system passes the test, the security of the storage system, etc.

[0042] At box 230, if the result indicates that the storage system passes the first test, computing device 110 determines the security level of the test case based on the result of the first test.

[0043] In some embodiments, after the storage system 120 passes the test for a test case, the features of that test case can be considered safe. Similarly, features below those of the test case can be considered safe. Thus, features from the above 4+1 independent dimensions that have passed the test for the test case can be considered a system security boundary, also known as a system security level. Furthermore, all metric data from the above 4+1 independent dimensions can be recorded for future comparison with data from specific user scenarios.

[0044] Table 1 below illustrates an example of customer scenario data according to the techniques described herein. In this example, features or items from the above 4+1 independent dimensions, as shown, indicate that after addressing existing problems and potential adverse effects in the storage system, the storage system can survive if the following conditions are met simultaneously. This means that the system security boundary has been improved on the development side and detected by the system testing side. In this case, the storage system can maintain reliability under the same or lower conditions, and the storage system can also be subjected to greater stress from this situation, such as increasing the load. Then, after increasing the load on the storage system based on the features or items of the 4+1 independent dimensions, new test cases are generated to determine whether the storage system is still reliable under the new test cases, and if it is still reliable, to determine the new security level (system security boundary). Doing so enables prediction of the specific data situation of the client. In other words, the techniques disclosed herein can provide the ability to reasonably recommend system load for the client-side storage system, and even the ability to predict the reliability of the client-side storage system.

[0045] Table 1. Test scenario dimension data for test cases of the storage system that passed the test.

[0046]

[0047] At box 240, if the result indicates that the storage system has failed the first test, computing device 110 performs a second test on the storage system based on the problem with the storage system.

[0048] In some embodiments, if storage system 120 fails the tests for the test cases, it indicates that there may be some problems in storage system 120. In this case, developers and / or testers may attempt to resolve these problems and then perform a second test on storage system 120 based on the problems found. It should be understood that problems in the storage system can be of various types. For example, system emergencies, data unavailability or data loss (DUDL), significant performance degradation, precipitous performance drops, etc. When a storage system exhibits similar problems, the storage system should be considered unreliable, or even unavailable.

[0049] Figure 3 A flowchart of an example method 300 for expanding the scope of test cases to determine a security level, according to embodiments of the present disclosure, is shown. Method 300 can be used as follows: Figure 2 One example implementation of block 230. According to some embodiments, method 300 can be implemented by, for example... Figure 1 The computing device 110 shown performs the following: If the result of the first test indicates that the storage system 120 passes the first test, the storage system 120 is secure under that test case, meaning that the scope of the original test case is within the system security boundary. On the other hand, in this case, the true system security boundary is uncertain, and both the client and / or the test end need to understand the extent to which the workload can be increased under the existing conditions. According to some embodiments, the computing device 110 can establish, adjust, and broaden the system security boundary by implementing method 300.

[0050] At box 310, computing device 110 expands the scope of the test case as an expanded test case.

[0051] As mentioned earlier, the scope of each test case or test scenario should be scalable and / or scalable. For example, but not limited to, expanding the scope of a test case may include increasing the number of features or items included in the test case, increasing the size of features or items, adding new features or items to the test case, or adding new test steps or the order of test sequences, etc., according to the tester's design purpose.

[0052] At box 320, computing device 110 performs a third test on storage system 120 using extended test cases. Then, at box 330, computing device 110 determines whether storage system 120 passes the third test. If it passes the third test, computing device 110 determines the security level of the extended test cases at box 340. In this way, the security level of the extended test cases covers more features or items than the original test cases, thereby improving the system security boundary. In other words, under the security level of the extended test cases, storage system 120 can withstand more system stress or workload. Similarly, test cases can be further extended and then used to perform tests on the storage system according to the techniques of this disclosure to determine a broader security boundary.

[0053] Figure 4 A flowchart of an example method 400 for narrowing the scope of test cases to perform tests on a storage system, according to embodiments of the present disclosure, is shown. Method 400 can be used as follows: Figure 2 One example implementation of the block 240. According to some embodiments, method 400 can be implemented by, for example... Figure 1 The computing device 110 shown performs the following: If the result of the first test indicates that the storage system 120 has failed the first test, the storage system 120 is unreliable and / or insecure under that test case, meaning that the scope of the original test case exceeds the system security boundary in some respects. In this case, the client and / or the test end need to understand which aspects of the workload and / or other factors are the main causes of the problem with the storage system 120. According to some embodiments, the computing device 110 can address the problems existing in the storage system 120 by implementing method 400, adjusting, narrowing, improving, and enhancing the system security boundary.

[0054] In some situations, the storage system 120 may become overloaded and begin to become unreliable or even unavailable, such as in the aforementioned system emergencies, data unavailability or data loss (DUDL), significant performance degradation, or a precipitous drop in performance. Figure 4 As shown, at box 410, computing device 110 determines whether the problem with storage system 120 can be resolved. If, for example, excessive workload causes the problem with the storage system to be unresolved, the process proceeds to box 420. At box 420, computing device 110 narrows down the scope of the test cases as reduced test cases.

[0055] As mentioned earlier, the scope of each test case or test scenario should be scalable and / or scalable. For example, but not limited to, reducing the scope of a test case may include reducing the number of features or items included in the test case, reducing the size of features or items, removing features or items from the test case, or reducing the order of test steps or test sequences, depending on the tester's design objectives.

[0056] Then, at box 430, computing device 110 performs a second test on storage system 120 using reduced test cases. In this way, the reduced test cases lower the system security boundary by covering fewer features or items compared to the original test cases. In other words, at the reduced test case security level, storage system 120 becomes more secure, or more reliable, due to less system stress or workload. Similarly, test cases can be further reduced and then used to perform tests on the storage system according to the techniques of this disclosure to ensure the reliability and security of the storage system within the security boundary.

[0057] According to some embodiments, if Figure 4 Box 410 indicates that the problem in storage system 120 can be resolved. After the problem is resolved, computing device 110 can perform a second test on storage system 120 using test cases. It should be understood that the environment of the test cases changes regardless of whether the aforementioned problem in storage system 120 can be resolved.

[0058] The techniques disclosed herein can effectively detect system security boundaries and correspondingly improve system testing processes to determine system reliability, as well as provide load recommendations for customer application scenarios. Many methods can be used to match customer scenarios with recorded system test scenarios, as shown in Table 2 below, but these methods are merely examples and not limitations. It should be understood that these comparison methods can be used individually or in combination.

[0059] Table 2. Methods for comparing customer scenarios and recorded system test scenarios

[0060]

[0061] Furthermore, after comparing customer scenarios and recorded system test scenarios using the methods shown in Table 2, different matching results may be obtained, as shown in Table 3 below.

[0062] Table 3. Analysis of Matching Results between Customer Scenarios and Recorded System Test Scenarios

[0063]

[0064] According to some embodiments, in the matching and / or comparison of customer scenarios and recorded system test scenarios, one or more items or features may exceed the original system security boundaries. Using the techniques of this disclosure, after the scope of the current test case (TS) is expanded to an expanded test case (TS+), such as Figure 3 As shown in box 310, the extended test case will be logged. Table 4 below lists matches for multiple items from 4+1 dimensions. It should be understood that the multiple items in Table 4 are listed for illustrative purposes only and do not imply any limitation on the scope of this disclosure. For example, items from the “Object Configuration” dimension may also include “Number of LUNs”, items from “External Workloads” may also include “IO Mode”, items from the “Feature Usage” dimension may also include “Snapshot Creation” and “LUN Migration”, and so on.

[0065] As shown in Table 4 below, from the current test case TS to the extended test case TS+, while keeping other items unchanged, only the items "Number of File Snapshots" increased from 3800 to 4000, "OS Memory" increased from 90% to 92%, and "NAS Module Reserved (PNAS) Memory" increased from 85% to 90%. Compared to the customer scenario data, all three items fall within the scope of the current test case TS and the extended test case TS+, in other words, they fall within their system security boundaries. It is worth noting that the item "Multi-user" in the customer scenario data far exceeds the security boundaries of the current test case TS and the extended test case TS+. Therefore, there is a gap between the user scenario and the test case for this "Multi-user" item, and the testing side should expand the scope of the corresponding test cases to fill this gap. It should be understood that the example of comparing the customer scenario and the recorded system test scenario is described for illustrative purposes only and does not imply any limitation on the scope of this disclosure. In this way, the technology of this disclosure can enable the continuous refinement and improvement of system test scenarios during the R&D phase to identify, establish, adjust, and broaden the system security boundaries.

[0066] Table 4. A typical example comparing customer scenarios and recorded system test scenarios.

[0067]

[0068]

[0069] Figure 5 A schematic block diagram of an example device 500 that can be used to implement embodiments of the present disclosure is shown. For example, such as Figure 1 The computing device 110 shown can be implemented by electronic device 500. For example... Figure 5As shown, device 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 502 or loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. CPU 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0070] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0071] The various processes and handling described above, such as methods 200, 300, and / or 400, can be executed by processing unit 501. For example, in some embodiments, methods 200, 300, and / or 400 can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by CPU 501, one or more actions of methods 200, 300, and / or 400 described above can be performed.

[0072] This disclosure can be a method, electronic device, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0073] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0074] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0075] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0076] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, electronic devices (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0077] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing electronic equipment to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing electronic equipment, they create an electronic device that implements the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing electronic equipment, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0078] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing electronic device, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing electronic device, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing electronic device, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0079] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0080] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for determining and indicating whether a storage system can reliably perform client scenarios with corresponding values ​​for corresponding dimensions of an operation, comprising: Multiple test cases are executed on the storage system, each test case being reliably executed by the storage system and having a pattern of corresponding values ​​for the dimension of the operation, and for each test case, test case data corresponding to the pattern including the values ​​is recorded; The customer scenario is compared with the recorded test case data for the test case to identify one or more test cases whose value patterns are similar to the value patterns of the customer scenario. as well as Coverage for the customer scenario is calculated based on the identified similar test cases, and a corresponding reliability output is generated based on the calculated coverage, the reliability output being selected from: (1) a secure output generated when the customer scenario is fully covered by the identified similar test cases in all dimensions, the secure output indicating that the storage system can reliably execute the customer scenario; and (2) an insecure output generated when the customer scenario is not covered by the identified similar test cases in most of the dimensions, the insecure output indicating that the storage system cannot reliably execute the customer scenario. Executing the multiple test cases includes: Obtain the results of performing a first test on the storage system using test cases; If the result indicates that the storage system has failed the first test, a second test is performed on the storage system based on the problems identified; and If the result indicates that the storage system passes the first test, the security level of the test case is determined based on the result. Furthermore, performing the second test on the storage system based on the problems existing in the storage system includes: if the problems existing in the storage system can be resolved, then performing the second test on the storage system using the test cases after the problems are resolved.

2. The method according to claim 1, further comprising: If the result indicates that the storage system passes the first test, the scope of the test case is expanded to become an expanded test case; A third test is performed on the storage system using the extended test cases. as well as If the storage system passes the third test, the security level of the extended test cases is determined.

3. The method of claim 2, wherein expanding the scope of the test cases includes at least one of the following: Increase the number of features included in the test cases; and Increase the size of the feature.

4. The method of claim 1, wherein performing a second test on the storage system based on the problems existing in the storage system comprises: If the problems existing in the storage system cannot be resolved... The scope of the test cases is narrowed down to the reduced test cases, and... The second test is performed on the storage system using the reduced test cases.

5. The method of claim 4, wherein narrowing the scope of the test cases includes at least one of the following: Reduce the number of features included in the test cases; and Reduce the size of the feature.

6. The method of claim 1, wherein the reliability output is further selected from: (3) an intermediate output generated when the customer scenario is not covered by the identified similar test cases in some dimensions of the dimension, the intermediate output indicating that the storage system can execute the customer scenario but with the risk of unreliability, and The dimensions mentioned include object configuration dimension, external workload dimension, system resource dimension, and feature usage dimension.

7. The method according to claim 1, wherein the test cases are generated based on predefined test scenario dimensions.

8. An electronic device, comprising: At least one processing unit; At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform an action, the action including: Multiple test cases are executed on the storage system, each test case being reliably executed by the storage system and having a pattern of corresponding values ​​for the dimension of the operation, and for each test case, the corresponding test case data including the values ​​of the pattern is recorded; The customer scenario is compared with the recorded test case data for the test cases to identify one or more test cases whose value patterns are similar to the value patterns of the customer scenario; and Coverage for the customer scenario is calculated based on the identified similar test cases, and a corresponding reliability output is generated based on the calculated coverage, the reliability output being selected from: (1) a secure output generated when the customer scenario is fully covered by the identified similar test cases in all dimensions, the secure output indicating that the storage system can reliably execute the customer scenario; and (2) an insecure output generated when the customer scenario is not covered by the identified similar test cases in most of the dimensions, the insecure output indicating that the storage system cannot reliably execute the customer scenario. Executing the multiple test cases includes: Obtain the results of performing a first test on the storage system using test cases; If the result indicates that the storage system has failed the first test, a second test is performed on the storage system based on the problems identified; and If the result indicates that the storage system passes the first test, the security level of the test case is determined based on the result. Furthermore, performing the second test on the storage system based on the problems existing in the storage system includes: if the problems existing in the storage system can be resolved, then performing the second test on the storage system using the test cases after the problems are resolved.

9. The electronic device according to claim 8, wherein the action further includes: If the result indicates that the storage system passes the first test, the scope of the test case is expanded to become an expanded test case; A third test is performed on the storage system using the extended test cases. as well as If the storage system passes the third test, the security level of the extended test cases is determined.

10. The electronic device of claim 9, wherein expanding the scope of the test cases includes at least one of the following: Increase the number of features included in the test cases; and Increase the size of the feature.

11. The electronic device of claim 8, wherein performing a second test on the storage system based on problems existing in the storage system comprises: If the problems existing in the storage system cannot be resolved... The scope of the test cases is narrowed down to the reduced test cases, and... The second test is performed on the storage system using the reduced test cases.

12. The electronic device of claim 11, wherein narrowing the scope of the test cases includes at least one of the following: Reduce the number of features included in the test cases; and Reduce the size of the feature.

13. The electronic device of claim 8, wherein the reliability output is further selected from: (3) an intermediate output generated when the customer scenario is not covered by the identified similar test cases in some dimensions of the dimension, the intermediate output indicating that the storage system can perform the customer scenario but with the risk of unreliability, and The dimensions mentioned include object configuration dimension, external workload dimension, system resource dimension, and feature usage dimension.

14. The electronic device according to claim 8, wherein the test cases are generated based on predefined test scenario dimensions.

15. A computer program product tangibly stored in a computer storage medium and comprising machine-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1-7.

Images