Autonomous software testing machine aimed at storing product defect detection

Abstract

Provided herein are methods, systems, and non-transitory processor-readable storage media for tuning a system. An example method includes providing, by a tuning module, a tuning action based on a state associated with a test case; executing, by a tuning agent, the test case using the tuning action; evaluating, by a tuning evaluation module, the tuning action with respect to a long-term reward; and determining a similarity between a subsequent execution of the test case and an evaluation of a tuning action of at least one previous execution of the test case to tune the tuning action.

Description

Technical Field

[0001] This field generally relates to information processing systems, and more specifically to the effective reproduction of defects in such systems. Background Technology

[0002] In Agile software testing, the tuning of test case parameters plays a crucial role. Agile is an iterative methodology for project management and software development. Tuning test parameters involves identifying combinations of test parameters that, among various test configuration factors (such as, for example, the size of the test object, workload, duration, etc.), trigger failures or are highly likely to trigger test failures or defects. Identifying optimal test parameters increases the probability of triggering valid vulnerabilities and defects within the information processing system. Summary of the Invention

[0003] The illustrative embodiments provide techniques for implementing a tuning system in a storage system. For example, the illustrative embodiments include a tuning module that provides tuning actions based on the state associated with a test case. A tuning agent uses the tuning actions to execute the test cases, and a tuning evaluation module evaluates the tuning actions with respect to long-term rewards. The tuning agent determines the similarity between subsequent executions of a test case and evaluations of at least one previously executed tuning action for the test case to tune the tuning action. Other types of processing devices may be used in other embodiments.

[0004] These and other illustrative embodiments include, but are not limited to, apparatuses, systems, methods, and processor-readable storage media. Attached Figure Description

[0005] Figure 1 An information processing system including a tuning agent and an automated testing client is shown in an illustrative embodiment.

[0006] Figure 2 A high-level flowchart of the process for tuning the system is shown in an illustrative embodiment.

[0007] Figure 3 A flowchart illustrating the process for the plot framework used in the tuning system is shown in an illustrative embodiment.

[0008] Figure 4 A flowchart illustrating the process for system tuning in an illustrative embodiment is shown.

[0009] Figure 5 and Figure 6 An example of a processing platform that can be used to implement at least a portion of distributed and multi-level server authentication embodiments is shown. Detailed Implementation

[0010] This document will describe illustrative embodiments with reference to exemplary computer networks and associated computers, servers, network devices, or other types of processing devices. However, it should be understood that these and other embodiments are not limited to use with the specific illustrative network and device configurations shown. Therefore, the term "computer network" as used herein is intended to be interpreted broadly to encompass, for example, any system comprising multiple networked processing devices.

[0011] The following describes a technique for implementing a tuning system, which can be used to provide tuning modules, etc., that provide tuning actions based on the state associated with test cases. A tuning agent uses tuning actions to execute test cases, and a tuning evaluation module evaluates the tuning actions with regard to long-term rewards. The tuning agent determines the similarity between subsequent executions of a test case and the evaluation of at least one previously executed tuning action for that test case to tune the tuning action.

[0012] Effectively tuning test parameters in complex information processing systems can be a time-consuming and costly task. Multiple combinations of test parameter values ​​may be tried, and there is often insufficient time to determine the optimal combination to optimize defect triggering in complex information processing systems.

[0013] Conventional methods for tuning test parameters can be problematic. Typically, testers rely on their tuning experience or in-depth analysis of source code to manually tune parameters. Neither of these methods is effective. Conventional techniques fail to share tuning experience and expertise among testers. They do not provide the means to tune test parameters, learn from previous tuned tests, or make judgments based on those tests.

[0014] In contrast, in at least some implementations of the present technology as described herein, in a tuning system, the tuning module provides tuning actions based on the state associated with a test case. A tuning agent uses the tuning actions to execute the test cases, and a tuning evaluation module evaluates the tuning actions with regard to long-term rewards. The tuning agent determines the similarity between subsequent executions of a test case and evaluations of at least one previously executed tuning action for the test case to tune the tuning action.

[0015] Therefore, the goal of current technology is to provide methods and systems for optimizing systems. Another goal is to use trial and error to iteratively learn optimal optimization strategies and then adaptively reuse these results for subsequent similar testing work. Yet another goal is to identify the optimal parameters for testing without exploring multiple combinations of test parameter values. In other words, to identify parameters in information processing systems that are likely to trigger vulnerabilities / flaws within a small test configuration and using a limited number of trials, thereby discovering vulnerabilities / flaws as early as possible.

[0016] In at least some implementations of the current techniques described herein, the use of a tuning system can provide one or more of the following advantages: a tuning system that iteratively tunes, learns, and judges test parameters to discover defects as much as possible; recommends test parameters that are more likely to trigger defects; reduces manual costs; saves time in exploring multiple combinations of test parameters; and does not rely on the experience of human testers.

[0017] Compared to conventional techniques, in at least some implementations of the present technology as described herein, in a tuning system, the tuning module provides tuning actions based on the state associated with a test case. A tuning agent uses the tuning actions to execute test cases, and a tuning evaluation module evaluates the tuning actions with regard to long-term rewards. The tuning agent determines the similarity between subsequent executions of a test case and evaluations of at least one previously executed tuning action for the test case to tune the tuning action. The tuning system is implemented by at least one processing device including a processor coupled to memory.

[0018] In an example embodiment of the current technology, the tuning agent receives tuning requests for test cases executed on the automated testing client from the automated testing client. A status detection module acquires the status associated with the test cases. A system monitor module monitors the runtime status during test case execution. A reward calculation module evaluates the rewards generated by the tuning actions applied to the execution of the test cases. The tuning agent includes a status detection module, a tuning module, a system monitoring module, a reward calculation module, and a tuning evaluation module.

[0019] In an example embodiment of the current technology, the state includes at least one test parameter, a test bench identifier, and the test runtime environment state at a certain point in time.

[0020] In example embodiments of the present technology, the test runtime environment state includes at least one of the following: average total input / output operations per second (IOPS), central processing unit (CPU) utilization, and average physical space utilization during test case execution.

[0021] In an example embodiment of the current technology, the tuning agent receives test case information and runtime information associated with the test case, wherein previous execution of the test case did not result in an error.

[0022] In an example embodiment of the current technology, the initial training of test cases is performed by selecting a random state in which tuning actions are applied to the execution of the test cases.

[0023] In an example embodiment of the current technology, at least one parameter is randomly generated for use when applying tuning actions to the execution of test cases.

[0024] In example embodiments of the current technology, the tuning strategy is customized for test cases by defining at least one of the following: tuning parameters, maximum and minimum parameter values, tuning intervals, and the state associated with the test case that triggers the end of the tuning process.

[0025] In an example embodiment of the current technology, a tuning strategy is applied to tuning actions to determine modifications to at least one test parameter for subsequent execution of a test case.

[0026] In a sample embodiment of the current technology, the evaluation module evaluates the tuning action to maximize long-term rewards.

[0027] In an example embodiment of the current technology, the reward calculation module determines the ratio between the test result score and the test configuration score.

[0028] In an example embodiment of the current technology, the test result score indicates the severity of the fault that occurred during the execution of the test case.

[0029] In an example embodiment of the current technology, the test configuration score is a weighted sum of configuration factors, which include at least one of the following: input / output operations per second (IOPS), central processing unit (CPU) utilization, and duration of the test case.

[0030] In an example embodiment of the current technology, the tuning evaluation module determines the best tuning strategy to be applied to the tuning action when the tuning action is applied to the execution of the test case.

[0031] In an example embodiment of the current technology, the tuning evaluation module determines recommended parameters for the execution of test cases to increase the probability of detecting errors during the execution of test cases.

[0032] In an example embodiment of the prior art, the degree of matching between test case details associated with subsequent execution of a test case and test case details associated with at least one of a plurality of previous executions of a test case, each of which has a corresponding tuning action, is calculated, wherein the test case details include test case information, test bench information, and test runtime information.

[0033] In an example embodiment of the present technology, at least one of a plurality of previous executions of a test case is determined to have a matching degree higher than a matching degree threshold. A corresponding tuning action associated with at least one of the plurality of previous executions of the test case is used in subsequent executions of the test case.

[0034] In an example embodiment of the current technology, the average of at least one previous execution of a test case that is above the matching threshold is calculated.

[0035] In an example embodiment of the current technology, at least one of the multiple previous executions of a test case is determined to have a matching degree not exceeding a matching degree threshold. Initial training for subsequent executions of the test case is performed by selecting randomly tuned actions based on probabilities associated with the state.

[0036] Figure 1 A computer network (also referred to herein as an information processing system) 100 configured according to an illustrative embodiment is shown. The computer network 100 includes an automated test client 102. The automated test client 102 is coupled to a network 104, wherein in this embodiment, network 104 is assumed to represent a subnetwork or other relevant portion of the larger computer network 100. In the example embodiment, additional automated test clients (not shown) coupled to network 104 may be present. Therefore, both elements 100 and 104 are referred to herein as examples of "networks," but in... Figure 1 In the context of this embodiment, the latter is assumed to be a component of the former. Also coupled to network 104 is storage system 105. Such storage system may include any of many different types of storage devices, including network-attached storage devices (NAS), storage area networks (SAN), directly attached storage devices (DAS), and distributed DAS, as well as combinations of these and other storage types (including software-defined storage devices).

[0037] Storage system 105 includes a tuning agent 110, which can be described as a test case parameter tuning agent based on adaptive learning. In an example embodiment, tuning agent 110 can be deployed as a service in the cloud. The tuning agent consists of a state detection module 112, a tuning action selection module 114, a system monitor module 116, a reward calculation module 118, and a tuning evaluation module 120.

[0038] The status detection module 112 acquires the status of the test cases, such as the current test case. In an example embodiment, the tuning agent 110 receives tuning requests for test cases executed on the automated test client 102. The tuning action selection module 114 observes the current status s(t) and provides a tuning action a(t) to the tuning agent 110. In an example embodiment, the current status s(t) includes at least one test parameter, a testbench identifier, and the test runtime environment status at time point t.

[0039] System monitor 116 monitors the runtime status of the system under test during the execution of each test case (e.g., data unavailability or data loss (DU / DL), emergency, performance degradation, memory leak, critical error messages in the system log, etc.).

[0040] The reward calculation module 118 calculates the reward for action a(t) in state s(t) based on the severity level of the vulnerability / flaw and / or the system in the test state and / or the final configuration in which the vulnerability / flaw occurs.

[0041] The optimization evaluation module 120 uses a reinforcement learning algorithm to compute the optimization evaluation based on state s(t), action a(t), reward, and the next state (t+1). The optimization evaluation Q(s,a) is a mapping between states and actions that maximizes the long-term reward. In other words, Q(s,a) is the value of taking action a in state t.

[0042] The automated testing client 102 may include, for example, a server and / or parts of one or more server systems, as well as devices such as mobile phones, laptops, tablets, desktop computers, or other types of computing devices. Such devices are examples of what is more generally referred to herein as “processing devices.” Some of these processing devices are also collectively referred to herein as “computers.”

[0043] In some embodiments, the automated testing client 102 includes a corresponding computer associated with a specific company, organization, or other enterprise. Additionally, at least a portion of the computer network 100 may also be referred to herein as collectively including an "enterprise network." As those skilled in the art will understand, many other operational scenarios involving various types and arrangements of processing devices and networks are possible.

[0044] Furthermore, it should be understood that the term “user” in this context and elsewhere in this document is intended to be interpreted broadly to encompass entities such as people, hardware, software, or firmware, as well as various combinations of these entities.

[0045] Assuming network 104 includes a portion of a global computer network (such as the Internet), other types of networks may be part of computer network 100, including wide area networks (WANs), local area networks (LANs), satellite networks, telephone or cable networks, cellular networks, wireless networks (such as Wi-Fi or WiMAX networks), or portions or combinations of these and other types of networks. Therefore, in some embodiments, computer network 100 includes a combination of multiple different types of networks, each including a processing device configured to communicate using Internet Protocol (IP) or other relevant communication protocols.

[0046] Also associated with storage system 105 are one or more input / output devices, which illustratively include a keyboard, display, or any other type of input / output device in combination. Such input / output devices can be used, for example, to support one or more user interfaces of storage system 105, and to support communication between storage system 105 and other related systems and devices not explicitly shown. For example, a user can invoke test cases on automated test client 102. One or more input / output devices can also be associated with any automated test client 102.

[0047] Additionally, Figure 1 The storage system 105 is assumed to be implemented using at least one processing device. Each such processing device typically includes at least one processor and associated memory, and implements one or more functional modules for controlling certain features of the storage system 105.

[0048] More specifically, the storage system 105 in this embodiment may include a processor coupled to a memory and a network interface.

[0049] Processors illustratively include microprocessors, microcontrollers, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other types of processing circuitry, as well as portions or combinations of these circuitry elements.

[0050] Memory illustratively includes any combination of random access memory (RAM), read-only memory (ROM), or other types of memory. The memory and other memory disclosed herein can be considered as examples of what is more generally referred to as a "processor-readable storage medium" that stores executable computer program code or other types of software programs.

[0051] One or more embodiments include articles of manufacture, such as computer-readable storage media. Examples of articles of manufacture include, but are not limited to, storage devices such as storage disks, storage arrays, or integrated circuits containing memory, as well as a variety of other types of computer program products. The term “article of manufacture” as used herein should be understood to exclude transient propagation signals. These and other references to “disk” herein are intended to refer generally to storage devices, including solid-state drives (SSDs), and therefore should not be considered, or in any way limited to, rotating magnetic media.

[0052] The network interface allows the storage system 105 to communicate with the automated test client 102 via network 104, and illustratively includes one or more conventional transceivers.

[0053] The tuning system can be implemented, at least partially, as software stored in memory and executed by a processor, and can reside in any of the storage system 105 and / or the automated test client 102. The tuning system can be a standalone plug-in that can be included within a processing device. This processing device can be any of the storage system 105, the automated test client 102, or any other processing device. The tuning system can reside on a processing device separate from the storage system 105 and / or the automated test client 102. In this example scenario, any storage system 105 and the automated test client 102 can send and receive messages to and from a separate processing device to access the tuning system.

[0054] It should be understood that Figure 1 The specific set of elements for storage system 105 shown (related to the automated test client 102 of computer network 100) is presented by way of illustrative example only, and additional or alternative elements may be used in other embodiments. Thus, another embodiment includes additional or alternative systems, devices, and other network entities, as well as different arrangements of modules and other components. For example, in at least one embodiment, one or more storage systems in storage system 105 may be on the same processing platform and / or as part of the same processing platform.

[0055] References, for example Figure 2 , Figure 3 and Figure 4 The flowchart is used to describe in more detail the exemplary process of using the storage system 105 and the automated test client 102 in the example tuning system of the computer network 100.

[0056] Now for reference Figure 2 This figure illustrates a high-level flowchart of the process for tuning the system in an illustrative embodiment. It should be understood that this particular process is merely an example, and additional or alternative processes may be performed in other embodiments.

[0057] At position 221, baseline test cases for initial learning are identified. In the example embodiment, test cases can be selected for specific test areas.

[0058] At point 222, the tuning strategy is customized. For example, the parameters used for tuning can be customized, such as defining the state space, maximum value, minimum value, tuning interval, and the state to end tuning. An example of these values ​​is listed below:

[0059]

[0060] At point 223, initial learning is performed. The purpose of initial learning is to gain experience that can be used to perform transfer training more quickly, in order to identify the optimal parameters for testing without exploring multiple combinations of test parameter values. In other words, it identifies parameters that are highly likely to trigger vulnerabilities / flaws in the information processing system within a small test configuration and a limited number of trials. In the example embodiment, the configuration can be calculated by various factors such as the size of the test object, workload, duration, etc.

[0061] In an example embodiment, this training can be performed within a scenario framework. In each scenario, the tuning agent 110 starts in a random state and performs actions such as... Figure 3 The training steps are a fixed number, as shown. For example, the tuning agent 110 generates random parameter combinations at time t. t Optimize agent 110 in parameters. t Execute test cases to obtain the current state S under the condition of t ={parameters t test_bed, runtime_info t}

[0062] At position 224, the tuning agent 110 stores the tuning evaluation of Q(s,a) and test reports for fault test cases, where Q(s,a) is the value of action a taken in state t. The test report may include test case information, test parameters that triggered the fault, test bench information, and the test runtime environment state. The following is a list of test case reports including fault information:

[0063] <Test Case Information>

[0064] Test case name:<Test Case Name>

[0065] Test area:<Test Area ID>

[0066] Test keywords:<Keyword1,keyword2,...,KeywordN>

[0067] <Test parameters trigger fault>

[0068] Parameter1 = 200

[0069] parameter2 = 4

[0070] parameter3 = 45

[0071] <Test Bench Information>

[0072] hardware:<hardware>

[0073] platform: <platform>

[0074] <Test Runtime Information>:

[0075] average_total_IOPS = 60K

[0076] average_CPU_Util = 70%

[0077] average_physical_space_usage=40%

[0078] At point 225, as described above, the similarity between the new tuning request and the generated test failure report is calculated. The similarity is calculated based on test case information, testbench information, and test runtime information. In the example embodiment, a matching threshold can be configured. In this example scenario, generated test failure reports with a matching score higher than the matching threshold can be considered "reusable experience," meaning that tuning actions (e.g., specific combinations of test parameters) can be applied to new tuning requests.

[0079] At position 226, tuning agent 110 tunes the parameters during the transfer learning step based on the generated test failure report and the calculated similarity. Tuning agent 110 returns a recommended combination of test parameters that has been determined to have a high probability of triggering the vulnerability / flaw.

[0080] At position 227, the optimization agent 110 continues to update the optimization assessment.

[0081] Figure 3 This is a flowchart of the process of using the plot framework in the tuning system of the illustrative embodiment. Figure 3 A more detailed description is provided in Figure 2 The plot framework described at 223. It should be understood that this particular process is merely an example, and additional or alternative processes may be performed in other embodiments.

[0082] At 330, the tuning agent initializes the runtime evaluation Q(s,a) using episode numbers and episode training steps. For example, tuning agent 110 generates random parameter combinations at time t. t Optimize agent 110 in parameters. t Execute test cases to obtain the current state St = {parameters} t ,test_bed,runtime_infor t }

[0083] At position 331, optimize agent 110 to initialize ε.

[0084] At point 332, the tuning agent 110 determines whether the scenario has ended. If yes, the tuning evaluation process ends. If no, at point 333, the tuning agent 110 generates random parameters and obtains state s(t) as the current state.

[0085] At 334, the tuning agent 110 determines whether the plot frame has reached the final training step or state. If not, at 335, the tuning agent 110 generates new parameters. t+1 In an example embodiment, the tuning agent may select a random tuning action based on the probability of ε. In another example embodiment, the tuning agent 110 selects the best tuning action that has been observed. In other words, those tuning actions with the highest Q(s,a). The tuning agent 110 then generates new parameters. t+1 .

[0086] At position 336, optimize agent 110 using new parameters. t+1 To execute test cases and evaluate the reward Rt (Rt = test result score / test configuration score), and determine the next state S. t+1 ={parameters t+1 ,test_bed t runtime_info t+1 }

[0087] At position 337, optimize agent 110 to state S. t+1 Set to the current state. At 341, tune agent 110 and repeat steps 334 through 337 until the final training step or state is reached.

[0088] At position 334, when the final training step or state has been reached, the tuning agent 110 approaches the optimal tuning strategy at position 338. In the example embodiment, the tuning agent 110 uses reinforcement learning and (s t ,a t ,R t ,s t+1 The optimal policy is approximated by updating Q(s,a). Multiple RL algorithms exist that can be used to extrapolate the optimal policy. RL algorithms can include, but are not limited to, Q-learning, Deep Q-Network (DQN) algorithms, and Dual Deep Q-Network (DDQN) algorithms.

[0089] At 339, the agent 110 gradually decreases ε, and at 342, the process advances to the next episode within the plot frame.

[0090] Figure 4 A flowchart illustrating a process for tuning a system in an illustrative embodiment is shown. It should be understood that this particular process is merely an example, and additional or alternative processes may be performed in other embodiments.

[0091] In an example embodiment, tuning agent 110 receives tuning requests from automated test client 102 for test cases executed on automated test client 102. Tuning agent 110 includes a status detection module 112, a tuning module 114, a system monitor module 116, a reward calculation module 118, and a tuning evaluation module 120.

[0092] In the example embodiment, tuning agent 110 can receive test case information and test bench information. In the example embodiment, tuning agent 110 can receive tuning requests for test cases executed without generating any vulnerabilities / defects. In this example scenario, tuning agent 110 receives test case information and runtime information associated with the test cases.

[0093] In the example embodiment, the status detection module 112 obtains the status associated with the test case. The status detection module 112 obtains the status of the current test case, which includes parameters associated with the tuning request.

[0094] In the example embodiment, the state includes at least one test parameter, a test bench identifier, and the test runtime environment state at a certain point in time. In the example embodiment, the state space can be defined as S = {S1, S2, ... S}. n }, where each S t ={parameters t ,test_bed,runtime_info t In the example embodiment, parameters t This represents a combination of test parameters, and `test_bed` is a static value representing testbed information, which may include, but is not limited to, hardware or platform information. In the example embodiment, the test runtime environment state includes at least one of the following: average total IOPS, average CPU utilization, and average physical space utilization during test case execution. In the example embodiment, `runtime_info` t The value represents the state of the test runtime environment, which may include, but is not limited to, rounded values ​​of average total IOPS, average CPU utilization, and average physical space utilization during test case execution. t Examples could be:

[0095] <parameters>

[0096] Parameter1 = 200

[0097] parameter2 = 4

[0098] parameter3 = 45

[0099] <Test Platform>

[0100] hardware: <hardware>

[0101] platform: <platform>

[0102] <Runtime Information>

[0103] average_total_IOPS = 60(K)

[0104] average_CPU_Util = 70 (percentage)

[0105] average_physical_space_usage = 40 (percentage)

[0106] In the example embodiment, the system monitor module 116 monitors the runtime status during the execution of test cases. The system monitor module 116 monitors the runtime status of the test system during the execution of each test case, for example, data unavailability or data loss (DU / DL), emergencies, performance degradation, memory leaks, critical error messages in the system logs, etc.

[0107] At 400, tuning module 114 provides tuning actions based on the state associated with the test case. In an example embodiment, tuning agent 110 selects where to apply tuning actions such as Figure 2 The initial training of the test cases is performed by executing random states as explained in the documentation. In the example embodiment, the tuning agent 110 randomly generates the state in which tuning actions are applied, such as... Figure 3 At least one parameter used when executing the test case being explained.

[0108] In example embodiments, such as Figure 2 As explained at point 222, tuning agent 110 customizes the tuning strategy for test cases by defining at least one of the following: tuning parameters, maximum and minimum parameter values, tuning intervals, and the state associated with the test case that triggers the end of the testing effort. In the example embodiment, again as Figure 2 As explained at point 222, tuning agent 110 applies tuning strategies to tuning actions to determine modifications to at least one test parameter for subsequent execution of the test case.

[0109] At position 402, tuning agent 110 uses tuning actions to execute test cases.

[0110] At 404, the tuning evaluation module 120 evaluates the tuning action in terms of long-term rewards. In an example embodiment, test parameters are tuned for test cases to effectively determine optimal test parameters, which can be determined using reinforcement learning, wherein rewards are received for actions performed correctly and penalties are received for actions not performed correctly. In an example embodiment, the tuning agent 110 observes the environment S at time t. t State, Selected Action a t And receive reward Rt.

[0111] In an example embodiment, tuning agent 110 determines the optimal tuning strategy to be applied to the tuning actions when applying tuning actions to the execution of test cases. The goal is to determine the optimal tuning strategy π* that maximizes long-term rewards.

[0112] In the example embodiment, given a state s, there may be two tuning actions, where a1 indicates increasing a first parameter and a2 indicates decreasing the first parameter. In the example embodiment, when a test case is executed using tuning actions a1 and a2 (alone), the test result may be that the tuning actions do not lead to the triggering of a vulnerability or defect, meaning that the immediate reward of tuning actions a1 and / or a2 is negative. However, in this example embodiment, the long-term value of tuning action a1 (denoted as Q(s,a1)) may be determined to be greater than the long-term value of tuning action a2 (denoted as Q(s,a2)).

[0113] To find the optimal tuning strategy, the action-value function Q... π (s,a) defines the value of action a in state s under policy π. In the example embodiment, there are multiple methods for extrapolating the optimal policy π. Reinforcement learning algorithms can include, but are not limited to, Q-learning, Deep Q-Network (DQN) algorithms, and Dual Deep Q-Network (DDQN) algorithms. In the reinforcement learning framework, the reward Rt signal indicates what is good in an immediate sense, while the value function Q... π (s, a) then estimates how "good" action a is in a given state s in the long run. Therefore, in the example embodiment, the goal of tuning agent 110 is to maximize long-run values ​​rather than immediate rewards.

[0114] In an example embodiment, the reward calculation module 118 evaluates the rewards generated by tuning actions applied to the execution of test cases. In an example embodiment, the reward calculation module 118 evaluates tuning actions to maximize long-term rewards. In an example embodiment, the reward function R can be defined as guiding the tuning agent 110 towards tuning actions that increase the probability of quickly and effectively triggering vulnerabilities / defects when executing test cases. Therefore, the goal of the tuning agent 110 is to find critical vulnerabilities with fewer trials and iterations and a smaller final configuration. Another goal is to maximize the likelihood of triggering these critical vulnerabilities.

[0115] In the example embodiment, the reward is determined using the ratio between the test result score and the test configuration score, for example:

[0116] Rt = Test result score / Test configuration score.

[0117] In the example implementation, the test result score indicates the severity of the failure that occurred during the execution of the test case. The following are examples of test result scores for the system based on the severity level of the vulnerability / flaw or the test status:

[0118] Test status Test result score Faults with critical vulnerabilities 10 Faults with vulnerabilities 8 Passing under performance degradation 6 If multiple error messages are found in the system log, then... 5 Passed without any errors or performance degradation. -5

[0119] In the example embodiment, the test configuration score is a weighted sum of configuration factors, which include at least one of the following: IOPS, CPU utilization, and duration of the test case. In the example embodiment, the test configuration score is based on the final configuration in which the vulnerability / flaw occurs. For example:

[0120] test_configuration_score=W1*scale_score+W2*IOPS_score+W3*CPU_utilization_score+W4*duration_score, where (Wi indicates the weight of configuration factor i).

[0121] In the example embodiment, weights are defined by their importance and can be predefined by the customer. The sum of the weights equals 1.

[0122] In the example embodiment, the tuning evaluation module 120 determines recommended parameters for the execution of test cases to increase the probability of detecting errors during the execution of test cases. In the example embodiment, multiple test parameters may exist, such as, but not limited to, workload (e.g., the workload of the automated test client 102), test case duration, data pattern (which is included in the workload), size (meaning, for example, the number of test objects in the storage system, the number of storage resource objects, such as volumes, LUNs, file systems, etc.).

[0123] Depending on various factors, such as the test area or test characteristics, some combinations may be more likely to trigger vulnerabilities / flaws than others. In an example embodiment, tuning agent 110 begins tuning actions using an initial combination of parameters, where each tuning action modifies a single parameter value.

[0124] In the example embodiment, tuning agent 110 observes the current state s(t) and provides a tuning action a(t) to the environment. At each time step t, each tuning action modifies a single input value (e.g., increases or decreases a parameter) based on a tuning strategy to obtain the next state. Below is an example where the test case has three parameters and the tuning strategy is customized according to the list of tuning actions:

[0125]

[0126] In the example implementation, for the current parameters t ={p1,p2,p3}, the optimization actions are as follows:

[0127] action Parameter modification New parameters 1 No action {p1,p2,p3} 2 p1<-p1+25 {p1+25,P2,p3} 3 p1<-p1-25 {p1-25,p2,p3} 4 p2<-P2*(1+1 / 4) {p1,p2*(1+1 / 4),p3} 5 p3<-p3-15 {p1,p2,p3-15}

[0128] In the example embodiment, for tuning action 1, no modification is made to the test parameters. For tuning action 2, tuning agent 110 will execute test cases in step (t+1) using the new parameters {p1+25, P2, p3} and obtain new runtime information. The next state is then S. t+1 ={parameters t+1 ,test_bed t runtime_info t+1 }, where parameters t+1 This refers to one of the new parameters; `test_bed` is a static value representing testbed information. `runtime_info` t+1 This indicates the test runtime environment state at step t+1.

[0129] At 406, tuning agent 110 determines the similarity between the subsequent execution of a test case and the evaluation of a tuning action related to at least one previous execution of the test case, in order to tune the tuning action. In the example embodiment, the similarity of a new tuning request is related to, for example, Figure 2 The similarity is calculated based on the 224 test case reports generated. The similarity is calculated using test case information, testbench information, and test runtime information.

[0130] In an example embodiment, tuning agent 110 calculates the matching degree between test case details associated with a subsequent execution of a test case and test case details associated with at least one of a plurality of previous executions of the test case, wherein each of the previous executions has its corresponding tuning action. In the example embodiment, test case details include test case information, testbench information, and test runtime information. In other words, the matching degree is used to calculate the similarity between a new tuning request for a test case executed on automated test client 102 and a previously executed test case.

[0131] In an example embodiment, tuning agent 110 determines that at least one of the multiple previous executions of a test case has a matching degree higher than a matching degree threshold.

[0132] In the example embodiment, the matching degree threshold can be configured. In the example embodiment, tuning agent 110 uses a corresponding tuning action associated with at least one of a plurality of previous executions of the test case for subsequent executions of the test case. In the example embodiment, all historical fault information with a matching degree higher than the matching degree threshold can be considered "reusable" tuning actions. In other words, if the test case associated with a new tuning request has a matching degree higher than the matching degree threshold, the tuning action associated with a previously executed test case can be used.

[0133] In an example embodiment, if there are multiple test reports for a specific tuning request, each test report having a matching degree higher than a matching degree threshold, the tuning agent 110 calculates the average of at least one previous execution of the test case that has a matching degree higher than the matching degree threshold. For example, if there are N test failure reports:

[0134] Test Fault Reports 1, 2, 3, ... N

[0135] Test case information Test bench information Test runtime information

[0136] The mean squared depth (MDT) threshold is 7. There exists a tuning request with the following matching records:

[0137]

[0138]

[0139] In this example embodiment, there are two test failure reports with a matching degree higher than the matching degree threshold. Therefore, the average matching degree is calculated as (9+8) / 2 = 8.5.

[0140] In example embodiments, such as Figure 2 As explained at point 226, parameters associated with test failure reports where the match rate exceeds a match rate threshold will be considered as initial values ​​for transmission tuning. In the example explained above, there are two recommended parameter combinations (where parameters...). t =PC i ):

[0141] PC1:<P1=50,P2=5,P3=24> MD1 = 9

[0142] PC2:<P1=100,P2=7,P3=12> MD2 = 8

[0143] In the example embodiment, the parameter ε is set based on the average matching degree. In the example embodiment, if a new tuning request is more similar to a previous tuning request in the initial learning, the tuning agent 110 utilizes previous experience. The ε parameter can also be referred to as the exploration and development tradeoff parameter. Based on the current state, a random action with a probability of (1-ε) is selected using probability ε, where (1-ε) indicates the tuning action with the highest Q(s,a) observed so far. The parameter ε is calculated (this is also in...). Figure 3 (Described at points 331 and 339 in the text):

[0144] ε i ={1,MD1≥MDT (Cannot utilize (multiple) prior experiences. ε needs to be explored)

[0145] ε i ={1 / MD1>MDT (higher similarity. May be able to leverage prior experience)

[0146] In the example embodiment, the tuning agent 110 will use the recommended parameter combination as described above as the initial parameters, instead of... Figure 3 The generation of random parameter combinations is described at point 333.

[0147] In an example embodiment, tuning agent 110 determines that at least one of a plurality of previous executions of a test case does not have a matching degree higher than a matching degree threshold. In this example scenario, tuning agent 110 performs initial training for subsequent executions of the test case by selecting random tuning actions based on probabilities associated with the state.

[0148] Therefore, combining Figure 2 , Figure 3 and Figure 4 The specific processing operations and other functions described in the flowcharts are presented by way of illustrative example only and should not be construed as limiting the scope of this disclosure in any way. For example, the order of process steps may be changed in other embodiments, or some steps may be performed simultaneously rather than sequentially.

[0149] The illustrative embodiments described above offer significant advantages over conventional methods. For example, some embodiments are configured to significantly improve the tuning of test parameters that trigger vulnerabilities. For example, some embodiments are configured to provide at least one of the following: using trial and error to iteratively learn the optimal tuning strategy and then adaptively reusing those results for subsequent similar testing work; identifying the optimal parameters for testing without exploring multiple combinations of test parameter values; identifying parameters that are likely to trigger vulnerabilities / flaws in the information processing system with a small test configuration and a limited number of trials; and discovering vulnerabilities / flaws as early as possible. These and other embodiments can effectively improve the way data is protected compared to conventional methods.

[0150] It should be understood that the specific advantages described above and elsewhere herein are associated with specific illustrative embodiments and do not need to appear in other embodiments. Furthermore, the features and functions of the specific type of information processing system illustrated in the figures and described above are merely exemplary, and many other arrangements may be used in other embodiments.

[0151] As previously described, at least a portion of the information processing system 100 may be implemented using one or more processing platforms. A given such processing platform includes at least one processing device comprising a processor coupled to memory. In some embodiments, the processor and memory include corresponding processor and memory elements using virtual machines or containers provided by one or more underlying physical machines. The term "processing device" as used herein is intended to be interpreted broadly to encompass a variety of different arrangements of physical processors, memory, and other device components, as well as virtual instances of these components. For example, a "processing device" in some embodiments may include one or more virtual processors or execute across one or more virtual processors. Thus, a processing device may be physical or virtual and may execute across one or more physical or virtual processors. It should also be noted that a given virtual device may be mapped to a portion of a physical device.

[0152] Some illustrative embodiments of a processing platform for implementing at least a portion of an information processing system include a cloud infrastructure that includes virtual machines implemented using hypervisors running on a physical infrastructure. The cloud infrastructure also includes a set of applications running on the respective virtual machines under the control of the hypervisors. Multiple hypervisors may also be used, each using at least one underlying physical machine to provide the set of virtual machines. Different sets of virtual machines provided by one or more hypervisors can be used to configure multiple instances of various components of the system.

[0153] These and other types of cloud infrastructure can be used to provide environments also referred to herein as multi-tenant environments. One or more system components or portions thereof are implemented, for example, for tenant use in such a multi-tenant environment.

[0154] As previously stated, the cloud infrastructure disclosed herein may include cloud-based systems. In illustrative embodiments, virtual machines provided in such systems may be used to implement at least a portion of a computer system.

[0155] In some embodiments, the cloud infrastructure additionally or alternatively includes multiple containers implemented using container host devices. For example, as detailed herein, a given container in the cloud infrastructure illustratively includes Docker containers or other types of Linux containers (LXC). Containers run on virtual machines in a multi-tenant environment, but other arrangements are also possible. Containers are utilized to implement various different types of functionality within system 100. For example, containers can be used to implement corresponding processing devices that provide computing and / or storage services for cloud-based systems. Similarly, containers can be used in combination with other virtualization infrastructure, such as virtual machines implemented using hypervisors.

[0156] Now refer to Figure 5 and Figure 6 To describe illustrative embodiments of the processing platform in more detail. Although described in the context of system 100, these platforms can also be used in other embodiments to implement at least a portion of other information processing systems.

[0157] Figure 5 An example processing platform including cloud infrastructure 500 is illustrated. Cloud infrastructure 500 includes a combination of physical and virtual processing resources for implementing at least a portion of information processing system 100. Cloud infrastructure 500 includes multiple virtual machines (VMs) and / or container sets 502-1, 502-2, ... 502-L implemented using virtualization infrastructure 504. Virtualization infrastructure 504 runs on physical infrastructure 505 and illustratively includes one or more hypervisors and / or operating system-level virtualization infrastructures. Operating system-level virtualization infrastructures illustratively include kernel control groups of Linux operating systems or other types of operating systems.

[0158] Cloud infrastructure 500 also includes application sets 510-1, 510-2, ... 510-L running under the control of virtualization infrastructure 504 on corresponding VM / container sets 502-1, 502-2, ... 502-L. VM / container set 502 includes corresponding VMs, corresponding sets of one or more containers, or corresponding sets of one or more containers running within VMs. Figure 5 In some implementations of the embodiments, VM / container set 502 includes a corresponding VM implemented using virtualization infrastructure 504, which includes at least one hypervisor.

[0159] A hypervisor platform can be used to implement a hypervisor within a virtualized infrastructure 504, wherein the hypervisor platform has an associated virtual infrastructure management system. The underlying physical machines include one or more distributed processing platforms, and the one or more distributed processing platforms include one or more storage systems.

[0160] exist Figure 5 In other implementations of the embodiments, VM / container set 502 includes corresponding containers implemented using virtualization infrastructure 504, which provides operating system-level virtualization capabilities, such as support for Docker containers running on bare metal hosts or on VMs. The containers are implemented illustratively using the corresponding kernel control groups of the operating system.

[0161] As can be clearly seen from the above, one or more processing modules or other components of system 100 can each run on a computer, server, storage device, or other processing platform element. The given examples of such elements are considered as examples of what is more generally referred to herein as a "processing device". Figure 4 The cloud infrastructure 500 shown can represent at least a portion of a processing platform. Another example of such a processing platform is... Figure 6 The processing platform 600 shown.

[0162] The processing platform 600 in this embodiment includes part of the system 100 and includes multiple processing devices that communicate with each other via a network 604. The multiple processing devices are referred to as 602-1, 602-2, 602-3, ..., 602-K.

[0163] Network 604 includes any type of network, such as global computer networks including the Internet, WANs, LANs, satellite networks, telephone or cable networks, cellular networks, wireless networks such as Wi-Fi or WiMAX networks, or portions or combinations of these networks and other types of networks.

[0164] The processing device 602-1 in the processing platform 600 includes a processor 610 coupled to a memory 612.

[0165] Processor 610 includes a microprocessor, microcontroller, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other types of processing circuitry, and portions or combinations of these circuitry elements.

[0166] Memory 612 includes any combination of random access memory (RAM), read-only memory (ROM), or other types of memory. Memory 612 and other memories disclosed herein should be considered as illustrative examples of executable program code stored in what is more generally referred to as a "processor-readable storage medium."

[0167] Articles of manufacture including such processor-readable storage media are considered illustrative embodiments. A given such article of manufacture includes, for example, any memory array, memory disk, or integrated circuit containing RAM, ROM, or other electronic memory, or several other types of computer program products. The term "article of manufacture" as used herein should be understood to exclude transient propagating signals. Many other types of computer program products including processor-readable storage media may be used.

[0168] The processing device 602-1 also includes a network interface circuit 614, which is used to interface the processing device with the network 604 and other system components, and may include a conventional transceiver.

[0169] Assume that the other processing devices 602 of the processing platform 600 are configured in a manner similar to that shown in the processing device 602-1 in the figure.

[0170] Similarly, the specific processing platform 600 shown in the figure is presented only as an example, and the system 100 may include additional or alternative processing platforms and many different processing platforms in any combination, each of which includes one or more computers, servers, storage devices or other processing devices.

[0171] For example, other processing platforms used to implement the illustrative embodiments may include different types of virtualization infrastructure, replacing or supplementing the virtualization infrastructure that includes virtual machines. Such virtualization infrastructure illustratively includes container-based virtualization infrastructure configured to provide Docker containers or other types of LXC.

[0172] As another example, in some embodiments, the components of a given processing platform may include a converged infrastructure.

[0173] Therefore, it should be understood that in other embodiments, different arrangements of additional or alternative elements may be used. At least a subset of these elements may be implemented together on a common processing platform, or each such element may be implemented on a separate processing platform.

[0174] Furthermore, numerous other arrangements of computers, servers, storage products or devices, or other components within the information processing system 100 are also possible. These components can communicate with other elements of the information processing system 100 via any type of network or other communication medium.

[0175] For example, specific types of storage products that can be used to implement a given storage system in the illustrative embodiments include all-flash and hybrid flash storage arrays, scale-out all-flash storage arrays, scale-out NAS clusters, or other types of storage arrays. In the illustrative embodiments, combinations of these and several other storage products can also be used to implement a given storage system.

[0176] It should be emphasized again that the embodiments described above are presented for illustrative purposes only. Many variations and other alternative embodiments can be used. Furthermore, in other embodiments, the specific configurations of the system and device elements illustrated in the drawings, as well as the associated processing operations, can vary. Thus, for example, the specific types of processing devices, modules, systems, and resources deployed in a given embodiment, and their corresponding configurations, can vary. Moreover, the various assumptions made above in describing the illustrative embodiments should also be considered exemplary and not as requirements or limitations of this disclosure. Many other alternative embodiments within the scope of the appended claims will be apparent to those skilled in the art.< / platform> < / hardware> < / platform> < / hardware>

Claims

1. A method comprising: The tuning module provides tuning actions based on the state associated with the test cases; The tuning agent uses the tuning actions to execute test cases; The tuning action is evaluated by the tuning evaluation module regarding long-term rewards, wherein the rewards are determined using the ratio between the test result score and the test configuration score, wherein the test configuration score is a weighted sum of configuration factors, including input / output operations per second (IOPS), average CPU utilization, and the duration of the test case. as well as The method involves determining the similarity between subsequent executions of the test case and evaluations of at least one previously executed tuning action of the test case to tune the tuning action, wherein the method is implemented by at least one processing device, the at least one processing device including a processor coupled to memory.

2. The method according to claim 1, further comprising: The tuning agent receives tuning requests from the automated testing client for the test cases executed on the automated testing client; The state associated with the test case is obtained by the state detection module; The runtime status of the test cases is monitored by the system monitor module during execution; as well as The reward calculated by the reward calculation module is evaluated for the reward generated by the tuning action applied to the execution of the test case; The tuning agent includes the status detection module, the tuning module, the system monitoring module, the reward calculation module, and the tuning evaluation module.

3. The method according to claim 2, wherein the state includes at least one test parameter, test bench identifier, and test runtime environment state at a certain point in time.

4. The method of claim 3, wherein the test runtime environment state includes at least one of the following: average total input / output operations per second (IOPS), average CPU utilization, and average physical space utilization during the execution of the test cases.

5. The method according to claim 1, wherein the tuning action provided by the tuning module includes: The initial training of the test cases is performed by selecting a random state in which the tuning action is applied to the execution of the test cases.

6. The method according to claim 5, further comprising: At least one parameter is randomly generated when the tuning action is applied to the execution of the test case.

7. The method of claim 1, wherein the tuning action provided by the tuning module comprises: Customize the tuning strategy for the test cases by specifying at least one of the following: i) Parameter tuning; ii) Maximum and minimum values ​​of the parameter; iii) Tuning interval; and iv) The state associated with the test case that triggered the end of the tuning process.

8. The method of claim 1, wherein the tuning action provided by the tuning module comprises: The tuning strategy is applied to the tuning action to determine modifications to at least one test parameter for subsequent execution of the test case.

9. The method of claim 2, wherein evaluating the tuning action by the evaluation module with respect to long-term rewards comprises: Evaluate the tuning actions to maximize the long-term reward.

10. The method of claim 2, wherein evaluating the reward by the reward calculation module comprises: Determine the ratio between the test result score and the test configuration score.

11. The method of claim 10, wherein the test result score identifies the severity of a failure that occurred during the execution of the test case.

12. The method of claim 10, wherein the test configuration score is based on the final configuration in which the vulnerability or defect occurs.

13. The method of claim 2, wherein evaluating the tuning action by the tuning evaluation module with respect to the long-term reward comprises: Determine the optimal tuning strategy to be applied to the tuning action when the tuning action is applied to the execution of the test case.

14. The method of claim 2, wherein evaluating the tuning action by the tuning evaluation module with respect to the long-term reward comprises: Determine recommended parameters for the execution of the test cases to increase the probability of detecting errors during the execution of the test cases.

15. The method according to claim 1, further comprising: Calculate the degree of match between the test case details associated with the subsequent execution of the test case and the test case details associated with at least one of the plurality of previous executions of the test case, each of the plurality of previous executions having a corresponding tuning action, wherein the test case details include test case information, test bench information and test runtime information.

16. The method of claim 15, further comprising: Determine that at least one of the plurality of previous executions of the test case has a matching degree higher than the matching degree threshold; as well as The corresponding tuning action associated with at least one of the multiple previous executions of the test case is used in the subsequent execution of the test case.

17. The method of claim 15, further comprising: Calculate the average of at least one previous execution among multiple previous executions of the test case that are above the matching threshold.

18. The method of claim 15, further comprising: Determine that at least one of the plurality of previous executions of the test case does not have a matching degree higher than the matching degree threshold; as well as The initial training for the subsequent execution of the test case is performed by selecting randomized tuning actions based on probabilities associated with the state.

19. A system comprising: At least one processing device, including a processor coupled to memory; The at least one processing device is configured to: The tuning module provides tuning actions based on the state associated with the test cases; The tuning agent executes the test cases using the tuning actions. The tuning action is evaluated by the tuning evaluation module regarding long-term rewards, wherein the rewards are determined using the ratio between the test result score and the test configuration score, wherein the test configuration score is a weighted sum of configuration factors, including input / output operations per second (IOPS), average CPU utilization, and the duration of the test case. as well as Determine the similarity between subsequent executions of the test case and the evaluation of at least one previously executed tuning action of the test case to tune the tuning action.

20. A computer program product comprising a non-transitory processor-readable storage medium wherein program code of one or more software programs is stored, wherein the program code, when executed by at least one processing device, causes the at least one processing device to: The tuning module provides tuning actions based on the state associated with the test cases; The tuning agent executes the test cases using the tuning actions. The tuning action is evaluated by the tuning evaluation module regarding long-term rewards, wherein the rewards are determined using the ratio between the test result score and the test configuration score, wherein the test configuration score is a weighted sum of configuration factors, including input / output operations per second (IOPS), average CPU utilization, and the duration of the test case. as well as Determine the similarity between subsequent executions of the test case and the evaluation of at least one previously executed tuning action of the test case to tune the tuning action.

Images