Method and apparatus for improving test case efficiency for vulnerability detection

Abstract

Embodiments of the present disclosure relate to methods and apparatuses for improving test case efficiency for vulnerability detection. A method for testing operation of a system under test (SUT) includes obtaining, by a test evaluation system, an automation job request for implementing an automation job on the SUT, obtaining a current state of the SUT in response to the automation job request, performing factor analysis on the current state using a test result database to determine a set of factors that potentially cause a failure of the SUT; determining an action based on the set of factors using an experience module and an exploration-exploitation trade-off (EET) value; modifying the automation job based on the action to obtain a new automation job, executing the new automation job on the SUT to obtain a result; calculating a reward corresponding to the result, and updating the experience module based on the reward, the current state, the action, and a new state of the SUT.

Description

Background Technology

[0001] Performing operational testing on a computing environment can be time-consuming and resource-intensive. During operational testing, any number of factors can cause the system under test (e.g., the computing environment) to fail. Identifying the factors that cause failure can be beneficial for the purposes of operational testing. Discovering these factors during operational testing can be a tedious task. Attached Figure Description

[0002] Some embodiments of the present invention will be described with reference to the accompanying drawings. However, the drawings are merely illustrative of certain aspects or embodiments of the invention and are not intended to limit the scope of the claims.

[0003] Figure 1 A schematic diagram of a system according to one or more embodiments of the present invention is shown.

[0004] Figure 2A A schematic diagram illustrating the relationship between states, actions, and rewards according to one or more embodiments of the present invention is shown.

[0005] Figure 2B A schematic diagram of an empirical function according to one or more embodiments of the present invention is shown.

[0006] Figures 3A to 3B A flowchart for implementing and updating automated test jobs according to one or more embodiments of the present invention is shown.

[0007] Figure 4 A schematic diagram of a computing device according to one or more embodiments of the present invention is shown. Specific Implementation

[0008] Specific embodiments will be described with reference to the accompanying drawings. In the following description, numerous details are set forth as details of the invention. It will be understood by those skilled in the art that one or more embodiments of the invention may be practiced without these specific details, and numerous variations or modifications may be made without departing from the scope of the invention. To avoid obscuring the description, certain details known to those skilled in the art have been omitted.

[0009] In the following description of the accompanying drawings, any component described in the drawings may be equivalent in various embodiments of the invention to one or more similarly named components described in any other drawing. For the sake of brevity, descriptions of these components will not be repeated for each drawing. Therefore, embodiments of each component in each drawing are assumed by reference to be optionally present in each other drawing having one or more similarly named components. Furthermore, any description of components in the drawings should be interpreted as optional embodiments that may be implemented additionally, in conjunction with, or alternatively as corresponding similarly named components described in any other drawing.

[0010] Throughout this disclosure, elements in the figures may be labeled A through N, A through P, A through M, or A through L. As used herein, the foregoing notation means that an element may include any number of items and is not required to include the same number of elements as any other element labeled A through N, A through P, A through M, or A through L. For example, a data structure may include a first element labeled A and a second element labeled N. This notation convention means that the data structure may include any number of elements. Another data structure also labeled A through N may also include any number of elements. The number of elements in the first data structure and the number of elements in the second data structure may be the same or different.

[0011] As used herein, the terms "operational connection," "operably connected," or "operational connection" refer to a direct or indirect connection between elements, components, and / or devices that allows the elements to interact in a certain way. For example, the term "operably connected" can refer to a direct connection (e.g., a wired connection between two devices or components) or an indirect connection (e.g., any number of devices or components connected "operably connected" via wired and / or wireless connections). Therefore, any path that information may take can be considered an operational connection.

[0012] Embodiments of the present invention include systems and methods for managing operational testing of computing environments (e.g., storage devices). These operational tests can be managed using an automated job framework that performs the operational tests. Embodiments of the present invention utilize reinforcement learning and a test result database to determine actions to be applied to the automated jobs to modify the operational tests, with the aim of identifying factors influencing computing environment failures. Embodiments of the present invention include functions for monitoring actions applied to the modified automated jobs, identifying the impact of the modified automated jobs on the system under test (SUT), and determining actions to be applied to the automated jobs based on the state of the SUT, as well as the results of implementing the modified automated jobs and optimizing strategies for future modifications. This optimization strategy determination can provide efficient operational testing for multiple SUTs, enabling the identification of more failure points in the SUTs with fewer attempts.

[0013] Various embodiments of the present invention will now be described.

[0014] Figure 1 A schematic diagram of a system according to one or more embodiments of the present invention is shown. The system (100) includes an automated operation (150), a computing environment (130), and a test and evaluation system (110). Figure 1 The system (100) may include additional, different and / or other components without departing from the present invention.

[0015] In one or more embodiments, the computing environment (130) includes the functionality to provide computing services to a user (not shown) or other entity. The computing environment (130) may be implemented as a computing device (e.g., see [link to documentation]). Figure 4 These computing devices can use any number of computing components (132, 138).

[0016] In one or more embodiments, at least a portion of the computing components (132, 138) performs operational testing on the computing environment (130). The portion of the computing components (132, 136) undergoing operational testing may be referred to as the System Under Test (SUT) (136). The SUT (136) can be operationally tested using an automated operating system (150) and a test evaluation system (110). For example, the SUT (136) may be implemented as a storage system, wherein the computing components (132, 138) are storage devices that store data generated or retrieved from the computing environment (130).

[0017] In one or more embodiments, the computing environment (130) is implemented as a computing device (e.g., Figure 4The computing device may be, for example, a laptop computer, a desktop computer, a server, a distributed computing system, or a cloud resource (e.g., a third-party storage system accessible via a wired or wireless connection). The computing device may include one or more processors, memory (e.g., random access memory), and persistent storage (e.g., disk drives, solid-state drives, etc.). The computing device may include instructions stored in persistent storage that, when executed by the processor(s) of the computing device, cause the computing device to perform the functions of the computing environment (130) described in this application.

[0018] In one or more embodiments disclosed herein, the computing environment (130) is implemented as a logical device. This logical device can utilize the computing resources of any number of computing devices to provide the functionality of the computing environment (130) described herein.

[0019] In one or more embodiments, the automated job system (150) includes functionality to perform operational tests. Specifically, the automated job system (150) may include any computing resources for issuing input / output operations (IOPS), storage requests, and / or other read / write requests to impose a computational load on the SUT (136). The IOPS (or other requests) issued by the automated job system (150) may be used to test the operability of the SUT (136).

[0020] For example, the operation level issued by the automated job system (150) may be high enough that these operations cause the computational load on the SUT (136) to approach or exceed its limits. Thus, the SUT (136) may fail (e.g., during service operations and / or requests). This failure may be caused by one or more operational issues (also known as bugs). These operational issues may be evaluated for future iterations of the SUT (136) to improve it.

[0021] The configuration (e.g., selection) in the implemented operations may affect test cases (e.g., the set of operations associated with a given SUT(136) at a given moment) including the failure of SUT(136). Thus, it may be beneficial to identify the factors causing failures by repeating tests while minimizing the amount of test execution.

[0022] To help increase the set of identified factors while reducing the number of tests, embodiments of the present invention may utilize a test evaluation system (110). This test evaluation system (110) may include functionality for evaluating test case configurations (also known as automated jobs) to determine the set of factors limiting the operability of the SUT (136). To perform this functionality, the test evaluation system (110) includes a factor analysis module (112), a state collection module (114), a reward calculation module (116), an experience module (118), a recommendation module (120), a similarity analysis module (122), a test results database (124), and a reinforcement agent (126). The test evaluation system (110) may include additional, fewer, and / or different components without departing from the present invention.

[0023] In one or more embodiments, the factor analysis module (112) analyzes which factors (also known as automation job parameters) have the greatest impact on test failures. Different combinations of factors may lead to one or more vulnerabilities for different combinations of computing components under test in the computing environment (130).

[0024] In one or more embodiments, the status collection module (114) acquires the status of the test cases from the automated operating system (110) and / or the system under test (SUT) (136). Figure 2A (Discussion in the middle) Related information.

[0025] In one or more embodiments, the reward calculation module (116) includes the ability to calculate the reward for an action (t) in state S(t) for a given test case (“t”) based on the goal of modifying the automated job to find critical vulnerabilities with fewer attempts and thereby reduce computational costs.

[0026] In one or more embodiments, the experience module (118) applies an experience function based on the current state (“s”), action (“a”), and next state. The experience Q(s,a) is a mapping from state pairs and actions to the maximum long-term reward. For more details on the experience function, see [reference needed]. Figure 2B .

[0027] In one or more embodiments, the recommendation module (120) observes the current state and provides job modification actions (a) for the current automated job. For example, the recommendation module (120) can use an experience module to identify actions that maximize the reward. The recommendation module can recommend the next action by selecting (e.g., (i) an identified action, or (ii) a random action). This selection is in... Figures 3A to 3B The method will be discussed further in the future.

[0028] In one or more embodiments, the similarity analysis module (122) analyzes the similarity between two or more states. The similarity analysis module can determine a similarity value between state pairs, and a state pair can be considered similar based on any predefined threshold. The similarity value can be determined by any mechanism without departing from the invention.

[0029] In one or more embodiments, the test results database (124) stores items such as test cases (e.g., Figure 2A The status information defined in the document, the test results of the corresponding automated job, the operation information of the automated job, and any other information for evaluating the system under test (SUT) (136).

[0030] In one or more embodiments, the reinforcement agent (126) uses a reinforcement learning algorithm to update the empirical function discussed throughout this disclosure. This empirical function may be updated based on, for example, the current state, an action, a reward for the action performed on the current state, and the next state after performing the action. Reinforcement learning algorithms used to update the empirical function include, but are not limited to, Q-learning algorithms, Deep Q-Network (DQN) algorithms, and Dual Deep Q-Network (DDQN) algorithms. For further details on calculating the reward, refer to [reference needed]. Figure 2A .

[0031] Figure 2A A schematic diagram illustrating the relationship between states, actions, and rewards according to one or more embodiments of the present invention is shown. As described above, a state (210) may be associated with a test case. A state (210) may include, for example, test case information, automated job information, 214, and SUT information. The state (210) of a test case may include additional, fewer, and / or different information without departing from the present invention.

[0032] In one or more embodiments, the test case information (212) involves static information about the test cases, such as test name, test-related features, test area, and test parameters.

[0033] In one or more embodiments, the automated job information (214) relates to automated job parameters (some of which can be changed by actions) that configure a set of operations to be performed in the corresponding test.

[0034] In one or more embodiments, the system under test (SUT) information (216) is a static value related to the computing components in the corresponding SUT. Examples of SUT information include hardware information, operating system information, firmware information, and other information without departing from the invention.

[0035] The SUT information (216) can further specify runtime information related to the dynamic information of the SUT, such as performance status, capacity usage status, rounded value of average total IOPS (input / output operations per second), average CPU utilization, average latency, and average physical space usage during automated job execution.

[0036] In one or more embodiments, the test evaluation system (110) discussed above... Figure 1 ) Observe the current state (210) at each attempt and take an action (220) that specifies the modification of the automation parameters of the current automation job (214).

[0037] In one or more embodiments, the result of applying action (220) to the current state (210) can be used to calculate a reward (230). The reward (230) can represent a value related to the objective of how much the action causes SUT operability failure. For example, the reward (210) can be based on the failure level caused by the action and the testing cost (e.g., a value representing the amount of resources required to apply the action). For example, a failure causing a critical vulnerability can result in a higher failure level than a test that does not cause SUT failure. A moderate failure level could be a test failure without a critical vulnerability. The lowest failure level could be a test passing without any error messages. The reward (230) can be calculated as a ratio of failure level to testing cost.

[0038] Figure 2B A schematic diagram of an empirical function according to one or more embodiments of the present invention is shown. As described above, the empirical function (240) consists of an empirical module (118, Figure 1 The empirical function (240) can produce an output (250) based on inputs including the current state (242) and an action (also called the next action) (244). The output (250) can include the next state (252) and a calculated long-term value (254). The next state (252) can represent the expected state resulting from applying changes associated with the next action (244) to the current state (242). For example, as Figure 2A As shown, action (210) can be a change to the automated job parameters of state (242). The next state (252) can specify the information of the current state (242) with the updated automated job parameters, replacing the previous automated job parameters.

[0039] The calculated long-term value (254) represents the maximum expected value that selecting the next action (244) to execute on the current state (242) could lead to the corresponding SUT achieving its failure objective. For example, in a scenario where the next action (244) causes the next state (252) to develop in a direction that is highly likely to lead to failure, the next action (244) applied to the current state (242) may be associated with a higher calculated long-term value (254). Conversely, in a scenario where the next action (244) guides the next state (252) in a direction that is less likely to lead to failure (or may require a considerably high computational cost to achieve failure), the next action (244) applied to the current state (242) may be associated with a lower computational long-term value (254).

[0040] Figures 3A to 3B A flowchart illustrating a method for implementing and updating automated jobs for testing according to one or more embodiments of the present invention is shown. The method is as follows... Figures 3A to 3B As shown, it can be evaluated by, for example, a test evaluation system (110, Figure 1 )implement. Figure 1 Other components of the system shown may perform Figures 3A to 3B The method is described without departing from the present invention. Although the steps in the flowchart are presented and described sequentially, those skilled in the art will understand that some or all of the steps may be performed in a different order, may be combined or omitted, and some or all of the steps may be performed in parallel.

[0041] Figures 3A to 3B The method can be executed for automated job sets.

[0042] Go to Figure 3A In step 300, an automated job request for testing the SUT is obtained from the automated job system.

[0043] In step 302, the current status of the test cases is obtained. As discussed throughout this disclosure, this status may include test case information for the most recent tests applied to the SUT, automation job parameters for previous automated jobs, and SUT information for the SUT.

[0044] In step 304, factor analysis is performed using the test results database and the current state to determine the set of factors contributing to the test failure. In one or more embodiments, the factor analysis is performed by a factor analysis module. Factor analysis includes analyzing the test results database to identify the factors (e.g., automation job parameters) that have the greatest impact on the test failure. These factors can be determined using any machine learning algorithm without departing from the present invention.

[0045] In one or more embodiments, after factor analysis, the action space is updated to specify the set of actions affected by the determined set of factors. For example, factor analysis can result in narrowing down the action space to a subset of all possible actions applicable to a given state, given the total number of possible actions that can be applied to it. By updating the action space using the determined set of factors, the accuracy and speed of multiple iterations of testing are improved.

[0046] In step 306, a determination is made regarding whether the current state is specified in the experience module. In one or more embodiments, the determination in step 306 is based on whether, at the current time point, for any action, there exists any long-term value that can be determined using the current state applied to the experience function. If the current state is specified in the experience module, the method proceeds to step 310; otherwise, the method proceeds to step 308.

[0047] In step 308, a determination is made as to whether the experience module identifies and specifies a state similar to the current state. In one or more embodiments, a similarity module is used to determine step 308, which determines whether the current state is similar to another state specified in the experience module. If a similar state exists, the method proceeds to step 310; otherwise, the method proceeds to step 312.

[0048] In step 310, the Explore and Exploit (EET) value associated with the current state and / or similar states is reduced from a previously set value. In one or more embodiments, the EET value is a value between 0 and 1, representing the probability that the next action to be determined (e.g., in step 314) is a random action. For states that are not frequently tried, the EET value can be high, thus encouraging exploration of possible actions and learning from the results. Conversely, for states that have been extensively tested, the EET value can be relatively low, thus encouraging the exploitation of known actions to achieve the desired objective of addressing operational problems (or failures) in the SUT. Therefore, the test evaluation system utilizes exploration (higher EET value) for newer, less tested states and utilizes exploration (lower EET value) for more extensively tested states. The EET value may decrease with each subsequent round of testing.

[0049] In step 312, after determining that the current state and similar states are not specified in the experience module, the EET value is set to 1 (or 100%). As mentioned above, a higher EET value can be associated with states that have been tested but have fewer states with lower EET values. Therefore, given that the current state is not specified in the experience module and has not yet been tested, the EET value is set to the highest value (e.g., 1) to encourage the exploration of possible actions.

[0050] In step 314, an action is determined using the experience module and the EET value. In one or more embodiments, the action is determined by identifying a known action that provides the highest long-term value for a given current state and applying the probability of a random action determined by the currently set EET value. If the probability of selecting a random action is satisfied, the random action is determined as the action; otherwise, the known action is determined as the action.

[0051] In step 316, the automated job associated with the automated job request is modified based on the determined action to initiate a new automated job. In one or more embodiments, the automated job is modified to obtain a new automated job by applying the action determined in step 314 (e.g., either a known action that maximizes a long-term value or a random action) to the automated job parameters of the current state.

[0052] A new automated job can be initiated by sending it to the automated job system on the SUT. The automated job system then executes the new automated job on the SUT.

[0053] Go to Figure 3B After the new automated job is executed, in step 318, the result of the new automated job is obtained. The result may specify whether operational problems were introduced in the SUT, whether the automated job was executed successfully, whether any errors were introduced, and / or any other information specifying the application of the automated job result to the SUT.

[0054] In step 320, a reward is calculated based on the result. In one or more embodiments, such as in Figure 2A As discussed earlier, the reward is calculated using the current state. The reward is calculated based on the results obtained in step 318 and the computational cost of applying the new automated job.

[0055] In step 322, the experience module is updated using the current state, the determined set of actions, the new state after applying these actions, and the calculated reward. In one or more embodiments, the experience module is updated based on the outcome (e.g., whether a failure occurred, whether a critical failure occurred, or whether a requirement was successfully met), the calculated reward, the current state, the new state, and the determined actions. For example, the experience module can be updated using a reinforcement learning algorithm to approximate the optimal policy. In this way, the functionality of the experience module can achieve accuracy over multiple iterations of state, action, and reward.

[0056] In step 324, it is determined whether the obtained result indicates an operational problem (such as a vulnerability) in the SUT. If an operational problem is indicated, the method proceeds to step 330; otherwise, the method proceeds to step 326.

[0057] In step 326, based on the results of the new automated job, it is determined whether a trial threshold has been reached to attempt to raise an operational problem in the SUT. If the trial threshold is reached, the method proceeds to step 330; otherwise, the method proceeds to step 328.

[0058] In step 328, the new state is set as the current state. After step 328, the method returns. Figure 3A In step 304, the next attempt for the next state of the test case triggers an operational failure and evaluates the identified actions(s). This allows for multiple test iterations of the SUT until a failure occurs (e.g., a result is detected via an operational problem) or the attempt threshold is reached.

[0059] In step 330, the test result database is updated based on the results and new status, according to the occurrence of an operational problem or the reaching of a trial threshold. This allows future iterations of automated jobs to use the updated test result database to identify factors contributing to future test failures.

[0060] Example

[0061] To illustrate various aspects of the invention, examples according to one or more embodiments of the invention are described below. Turning to this example, consider a scenario where a system under test (SUT) package is used to store a storage device system containing a large amount of data.

[0062] and Figure 2A Similar example states discussed in the text may include the following information: the test name is "TestName:Test_QC112047_BasicMetroTest", the test characteristic is "MetroReplication", the test region is "BasicLifecycletest", the session number is "10", and the operating system is "Linux". Automated job information includes parameters such as the number of cycles, load profile, load type, number of volumes, pre-fill percentage, and local protection policy. In other words, the test will repeat a defined number of cycles, apply a predefined load profile, create multiple volumes, consume a pre-fill percentage of the SUT's storage capacity, and configure a local protection policy for each volume using a predefined load type. These automated job parameters can be modified for evaluation to determine how they affect the operability of the SUT.

[0063] Additional SUT information includes the SUT's system hardware, system platform, and driver information. For a given set of automated job parameters, SUT information also includes dynamic information such as average total IOPS (input / output operations per second), average CPU utilization percentage, average IOPS latency, and average physical space usage.

[0064] To define the action space, use the following table to define the automation job set parameters in each row, define the state space, and the potential actions for each parameter:

[0065]

[0066]

[0067] Table 1: Automated operation parameters and corresponding possible actions.

[0068] In this example, the list of acceptable local protection policies includes the following: none, once a day, once every 16 hours, once every 8 hours, once every 4 hours, once every 2 hours, once every hour, and once every 30 minutes.

[0069] The list of acceptable IO tools includes the following: vdbench, fio, dbc, and iometer.

[0070] The list of acceptable I / O modes includes the following: Profile 1, Profile 2, Profile 3, and Profile 4.

[0071] The following table lists each possible action for the current set of automated jobs:

[0072]

[0073]

[0074] Table 2: List of actions and modifications for automated tasks.

[0075] As shown above, each action represents a modification to the parameters of the automated job (such as loop count, pre-filling, load profile, etc.). Table 2 above provides an example operation space of possible operations that can be applied to the state in each test iteration. Following... Figure 3A After performing factor analysis in step 304, a set of factors considered likely to influence the test and increase the chance of failure in the SUT can be identified. For example, for the first factor analysis iteration, the factors influencing the test could be the number of cycles (p2), the load profile (p6), and the local protection strategy (p4). Therefore, the action space shown in Table 2 can be updated to list some of the possible actions for the identified factors affecting p2, p4, and p6. Accordingly, the action space can be updated to include only actions 1, 3, 5, and 7. For each test iteration, the action space is updated to accurately reflect the list of actions that may produce favorable results.

[0076] according to Figures 3A to 3BThe method is used to test the example SUT above, starting from the current state including the following parameters: P1: 10%; P2: 5; P3: 10; P4: once a day; P5: vdbench; P6: profile 1.

[0077] For each test iteration, an action is selected from the updated action space to apply to a state that includes modifying one of the parameters P1-P6 according to Table 2. The result may include confirmation of whether a failure occurred, whether an operational problem such as a critical or non-critical vulnerability occurred, or whether an error message was issued.

[0078] To calculate the reward for each state, the following table maps the result set to the corresponding test result score:

[0079]

[0080]

[0081] Table 4: Test status and corresponding test result scores.

[0082] As shown in Table 4, failures with critical vulnerabilities are mapped to the highest test result score, while passes without any error messages or operational issues (e.g., no failures) are mapped to negative rewards, because it is undesirable to not be close to the load limit when testing the operability of the SUT.

[0083] In this example, the reward is calculated as the ratio of the test result score to the computational cost of implementing the automated task. The computational cost can be calculated as follows: Cost = W1P1 + W2P2 + W3P3 + W3P3 + W4P4 + W5P5, where each parameter P1 to P5 corresponds to a value among the automated task parameters defined in Table 1, and each weight W1 to W5 is a predefined value based on the impact of the corresponding parameter on the computational cost.

[0084] To determine the next action in the test, the test evaluation system applies an empirical function to each state to calculate the expected long-term value of a given action in achieving the objective.

[0085] End of example

[0086] As described above, embodiments of the present invention can be implemented using computing devices. Figure 4A schematic diagram of a computing device according to one or more embodiments of the present invention is shown. The computing device (400) may include one or more computer processors (402), non-persistent storage (404) (e.g., volatile memory such as random access memory (RAM), cache memory), persistent storage (406) (e.g., hard disk, optical disc drive or digital versatile disc (DVD) drive, flash memory, etc.), communication interface (412) (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), input device (410), output device (408), and many other elements and functions not shown. These components will be described below.

[0087] In one embodiment of the invention, the (multiple) computer processors (402) may be integrated circuits for processing instructions. For example, the (multiple) computer processors may be the core or microcore of one or more processors. The computing device (400) may also include one or more input devices (410), such as a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device. Furthermore, the communication interface (412) may include an integrated circuit for connecting the computer (400) to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, a mobile network, or any other type of network) and / or another device (e.g., another computing device).

[0088] In one embodiment of the invention, the computing device (400) may include one or more output devices (408), such as a screen (e.g., a liquid crystal display (LCD), a plasma display, a touch screen, a cathode ray tube (CRT) monitor, a projector, or other display device), a printer, an external storage device, or any other output device. The one or more output devices may be the same as or different from the input devices(s). The input and output devices(s) may be locally or remotely connected to the computer processor(s)(402), non-persistent storage(404), and persistent storage(406). Many different types of computing devices exist, and the aforementioned input and output devices(s) may take other forms.

[0089] One or more embodiments of the present invention may be implemented using instructions executed by a processor of one or more data management devices. Furthermore, these instructions may correspond to computer-readable instructions stored on one or more non-transitory computer-readable media.

[0090] One or more embodiments of the present invention can improve the operation of one or more computing devices. More specifically, embodiments of the present invention enable efficient data management of a distributed namespace (DNS) by processing a large number of pages from multiple snapshots of the DNS. This efficient data management is achieved by allocating a subset of all pages to multiple parallel processes for each computing node. Embodiments of the present invention reduce the logical traversal of allocated pages by determining a subset of allocated pages for physical enumeration. To prevent redundancy in the physical enumeration of pages, embodiments of the present invention provide an algorithm for determining a subset of allocated pages by excluding any pages not referenced in the data snapshots and filtering those pages referenced by multiple snapshots, such that only one computing node contains a given page in the subset of pages used for physical enumeration. By providing a method for parallel processing of DNS while reducing redundancy, embodiments of the present invention improve the operation of computing devices by efficiently managing computing resources by a DNS system of computing devices using DNS services.

[0091] Therefore, embodiments of the present invention can solve the problem of inefficient use of computing resources. This problem arises from the technical nature of the environment utilized by the file system.

[0092] The problems discussed above should be understood as examples of problems solved by the embodiments disclosed in this invention, and the invention should not be limited to solving the same or similar problems. The disclosed invention is broadly applicable to solving a range of problems beyond those discussed herein.

[0093] While the invention has been described with reference to a limited number of embodiments, those skilled in the art will understand, upon benefit from this disclosure, that other embodiments can be devised without departing from the scope of the invention. Therefore, the scope of the invention should be defined only by the appended claims.

Claims

1. A method for testing the operation of a system under test (SUT), the method comprising: The test and evaluation system obtains automation job requests for implementing automated jobs on the SUT; In response to the automated job request, obtain the current status of the SUT; Perform factor analysis on the current state using the test results database to identify a set of factors that could potentially cause the failure of the SUT; Based on the aforementioned factor set, actions are determined using an empirical module and by exploring and utilizing trade-off EET values. The experience module indicates a long-term value for performing the action in the current state. The EET value corresponds to the probability that the determined action is random, and The long-term value corresponds to a value that indicates the expected result of the SUT experiencing the failure when the action is performed in the current state; Modify the automated job based on the action to obtain a new automated job; Execute the new automated job on the SUT to obtain the results; Calculate the reward corresponding to the result; as well as The experience module is updated based on the reward, the current state, the action, and the current state of the SUT.

2. The method according to claim 1, further comprising: After updating the aforementioned experience module; Make a determination that the attempt threshold has not yet been reached; Based on the determination, the factor analysis is performed in the new state to determine a new set of factors; Determine the new EET value based on the new state; The second action is determined using the experience module, the new state, and the new EET value. Modify the new automated job based on the second action to obtain a third automated job; The third automated task is executed on the SUT to obtain the second result; Calculate the second reward corresponding to the second result; as well as The experience module is updated based on the second reward, the new state, the second action, and the third state of the SUT.

3. The method of claim 1, wherein the SUT is a storage system comprising one or more storage devices.

4. The method of claim 1, wherein the current state is not specified in the experience module, and wherein a similar state specified in the experience module is used to determine the action.

5. The method of claim 1, wherein the current state is not specified in the experience module, wherein a similar state is not specified in the experience module, and wherein the EET value is set to 100%.

6. The method of claim 1, wherein the result indicates the fault of the SUT, and wherein the test result database is updated to specify the fault, the current state, and the action.

7. The method according to claim 1, wherein the current state includes test case information, automated job information, and SUT information.

8. The method of claim 7, wherein the factor set corresponds to a portion of the automated job, and wherein the action corresponds to at least one factor in the factor set of the automated job specified by the automated job information.

9. A non-transient computer-readable medium comprising computer-readable programming code, which, when executed by a computer processor, causes the computer processor to perform a method for testing a system under test (SUT), the method comprising: The test and evaluation system obtains automation job requests for implementing automated jobs on the SUT; In response to the automated job request, obtain the current status of the SUT; Perform factor analysis on the current state using the test results database to identify a set of factors that could potentially cause the failure of the SUT; Based on the aforementioned factor set, actions are determined using an empirical module and by exploring and utilizing trade-off EET values. The experience module indicates a long-term value for performing the action in the current state. The EET value corresponds to the probability that the determined action is random, and The long-term value corresponds to a value that indicates the expected result of the SUT experiencing the failure when the action is performed in the current state; Modify the automated job based on the action to obtain a new automated job; Execute the new automated job on the SUT to obtain the results; Calculate the reward corresponding to the result; as well as The experience module is updated based on the reward, the current state, the action, and the current state of the SUT.

10. The non-transient computer-readable medium according to claim 9, further comprising: After updating the aforementioned experience module; Make a determination that the attempt threshold has not yet been reached; Based on the determination, the factor analysis is performed in the new state to determine a new set of factors; Determine the new EET value based on the new state; The second action is determined using the experience module, the new state, and the new EET value. Modify the new automated job based on the second action to obtain a third automated job; The third automated task is executed on the SUT to obtain the second result; Calculate the second reward corresponding to the second result; as well as The experience module is updated based on the second reward, the new state, the second action, and the third state of the SUT.

11. The non-transient computer-readable medium of claim 9, wherein the SUT comprises a storage system of one or more storage devices.

12. The non-transient computer-readable medium of claim 9, wherein the current state is not specified in the empirical module, and wherein a similar state specified in the empirical module is used to determine the action.

13. The non-transient computer-readable medium of claim 9, wherein the current state is not specified in the empirical module, wherein a similar state is not specified in the empirical module, and wherein the EET value is set to 100%.

14. The non-transient computer-readable medium of claim 9, wherein the result indicates the fault of the SUT, and wherein the test result database is updated to specify the fault, the current state, and the action.

15. The non-transient computer-readable medium of claim 9, wherein the current state includes test case information, automated job information, and SUT information.

16. The non-transient computer-readable medium of claim 15, wherein the set of factors corresponds to a portion of the automated job, and wherein the action corresponds to modifying at least one factor in the set of factors in the automated job specified by the automated job information.

17. A system comprising: The testing and evaluation system operates on the processor. The system under test (SUT) is a storage system that includes one or more storage devices. as well as The memory includes instructions, and when executed by the processor, the instruction execution method includes: Obtain the automation job request used to implement the automation job on SUT; In response to the automated job request, obtain the current status of the SUT. The current status includes test case information, automated job information, and SUT information; Perform factor analysis on the current state using the test results database to identify a set of factors that could potentially cause the failure of the SUT; Based on the aforementioned factor set, actions are determined using an empirical module and by exploring and utilizing trade-off EET values. The action described therein corresponds to modifying at least one factor in the set of factors in the automated task. The experience module indicates a long-term value for performing the action at the current state. The EET value corresponds to the probability that the determined action is random, and The long-term value corresponds to a value that indicates the expected result of the SUT experiencing the failure when the action is performed in the current state; Modify the automated job based on the action to obtain a new automated job; Execute the new automated job on the SUT to obtain the results; The results indicate the fault of the SUT, and the test result database is updated to specify the fault, the current state, and the action. Calculate the reward corresponding to the result; and The experience module is updated based on the reward, the current state, the action, and the current state of the SUT.

18. The system of claim 17, further comprising: After updating the aforementioned experience module; Make a determination that the attempt threshold has not yet been reached; Based on the determination, the factor analysis is performed in the new state to determine a new set of factors; Determine a new EET value based on the new state; The second action is determined using the experience module, the new state, and the new EET value. Based on the second action, modify the new automated job to obtain a third automated job; The third automated task is executed on the SUT to obtain the second result; Calculate the second reward corresponding to the second result; as well as The experience module is updated based on the second reward, the new state, the second action, and the third state of the SUT.

19. The system of claim 17, wherein the current state is not specified in the experience module, and wherein a similar state specified in the experience module is used to determine the action.

20. The system of claim 17, wherein the current state is not specified in the experience module, wherein a similar state is not specified in the experience module, and wherein the EET value is set to 100%.

Images