A test case scheduling method, device and storage device for improving effectiveness of automatic testing

Abstract

The application discloses a test case scheduling method and device for improving the effectiveness of automatic testing and a storage device, and the method comprises the following steps: acquiring test parameters in response to code submission, wherein the test parameters comprise test duration and test cases; starting basic testing; determining a changed firmware module according to a function module that needs to be changed, and determining a weighting value and an execution times of the changed firmware module test; and performing the test case corresponding to the changed firmware module according to the weighting value and the execution times. The application can guarantee the normal operation of the basic function of the function module by setting a basic test set, and can determine whether the change of one function module affects the basic function of other function modules and the like by screening the changed function module, so that the testability is improved.

Description

Technical Field

[0001] This application relates to the field of storage technology, and more specifically, to a test case scheduling method, apparatus, and storage device for improving the effectiveness of automated testing. Background Technology

[0002] Currently, when performing automated regression testing on storage devices, multiple rounds of testing are typically executed according to a fixed set of manually selected test cases, or different reference factors (such as priority or weighting factors) are set for the test cases to determine which test items to execute first. However, because the matching rules are fixed, the scheduling result is deterministic for each test object, making it impossible to achieve precise test coverage and to implement enhanced testing for related changes.

[0003] Therefore, how to provide a test case scheduling method that can achieve accurate test coverage has become an urgent problem for those skilled in the art. Summary of the Invention

[0004] The purpose of this application is to provide a test case scheduling method, apparatus, and device to improve the effectiveness of automated testing and avoid inaccurate test coverage caused by fixed matching rules. By setting weights and execution counts for test items associated with functional modules, and by changing the weights and execution counts, more accurate test coverage can be achieved, increasing the probability of discovering problems.

[0005] To achieve the above objectives, as a first aspect of this application, this application proposes a test case scheduling method to improve the effectiveness of automated testing, comprising the following steps: obtaining test parameters in response to code submission, the test parameters including test duration and test cases; starting basic testing; determining the modified firmware module according to the functional modules to be modified, and determining the weighting value and execution number of the modified firmware module test execution; and executing the test cases corresponding to the modified firmware module for testing according to the weighting value and execution number.

[0006] The test case scheduling method for improving the effectiveness of automated testing as described above further includes setting different tags for each test case based on the test items contained in each test case.

[0007] The test case scheduling method for improving the effectiveness of automated testing as described above, wherein the modified firmware module includes:

[0008] Based on the required firmware file path and keyword search in the modified code, determine the functional modules that need to be modified. The modified firmware modules include tags for associated test cases.

[0009] The test case scheduling method for improving the effectiveness of automated testing as described above includes determining the weighting value and execution number of tests for the modified firmware module, which involves: assigning a weighting value to each modified firmware module corresponding to a test item; and assigning the execution number of each test item according to its relevance to the modified firmware module.

[0010] The test case scheduling method for improving the effectiveness of automated testing as described above, wherein executing test cases corresponding to the modified firmware module based on weighted values ​​and execution counts includes: prioritizing the execution of the test item with the largest weighted value; after executing the test item with the largest weighted value according to the number of executions in response, switching to the test item with the next largest weighted value to continue testing.

[0011] The test case scheduling method for improving the effectiveness of automated testing, as described above, includes allocating the number of executions of corresponding test items based on their relevance to the modified firmware module. This includes allocating the number of executions of corresponding test items based on their relevance to the modified firmware module, the time required for test execution, and the probability of test errors.

[0012] The test case scheduling method for improving the effectiveness of automated testing as described above further includes: checking whether the test duration has been reached after one round of test case execution; if the test duration has been reached, calculating the test results; if the test duration has not been reached, re-executing the test cases.

[0013] The test case scheduling method for improving the effectiveness of automated testing as described above further includes: after each round of test execution, checking whether there are any test case errors; if there are no errors, continuing to the next round of testing to obtain the corresponding test results; wherein the next round of testing is to execute the test cases corresponding to the remaining tags according to the weight values.

[0014] As a second aspect of this application, this application provides a test case scheduling device for improving the effectiveness of automated testing, comprising: a test parameter acquisition module, used to acquire test parameters in response to code submission, the test parameters including test duration and test cases; a basic test startup module, used to start basic tests; a modified firmware module weighting value and execution count determination module, used to determine the modified firmware module according to the functional modules to be modified, and determine the weighting value and execution count of the modified firmware module test execution; and a test case execution module, used to execute the test cases corresponding to the modified firmware module for testing according to the weighting value and execution count.

[0015] As a third aspect of this application, this application provides a storage device comprising: a processor and a memory, wherein the memory includes an intelligent control program, and when the intelligent control program is executed by the processor, it implements the test case scheduling method described above for improving the effectiveness of automated testing.

[0016] This application has the following beneficial effects:

[0017] (1) By setting up a basic test set, this application can ensure the normal operation of the basic functions of the functional modules. On this basis, by screening and modifying a functional module, it can determine whether it affects the basic functions of other functional modules, thus enhancing the testability.

[0018] (2) This application associates the functional modules that need to be modified with test cases and sets the weighting value and execution number of the corresponding test items. By changing the weighting value and execution number of the corresponding test items, more accurate test coverage can be achieved for the changed parts, which increases the probability of finding problems. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0020] Figure 1 This is a flowchart illustrating a test case scheduling method for improving the effectiveness of automated testing according to an embodiment of this application.

[0021] Figure 2 This is a flowchart of a method for performing automated testing according to an embodiment of this application;

[0022] Figure 3 This is a schematic diagram of the internal structure of a test case scheduling device for improving the effectiveness of automated testing, provided according to an embodiment of this application.

[0023] Figure 4 This is a schematic diagram of the structure of a device provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. Example 1

[0025] like Figure 1 As shown, this embodiment provides a test case scheduling method to improve the effectiveness of automated testing, specifically including the following steps:

[0026] Step S1: Retrieve test parameters in response to code submission. Test parameters include test duration and test cases.

[0027] Each code commit will automatically trigger a test. The first step is to obtain the test parameters, which include the test duration, test cases, and a list of test machine hosts.

[0028] This embodiment allows adding test cases or test machines during testing. If a new test machine is added, it takes effect immediately, and test cases are assigned to and executed for the new test machine. If a new test case is added, it takes effect in the next round of testing.

[0029] One of the test cases may contain one or more test items. Each test case is assigned a different label based on the test items it contains. The label indicates the specific test item covered by the test case.

[0030] When a test case contains multiple test items, such as IO (input / output), Power Cycle (battery cycle), Reset, and FW Update (firmware upgrade), then the test case is labeled with multiple different tags based on these multiple test items.

[0031] Step S2: Start the basic test.

[0032] Basic tests are a special type of test item that includes the most basic IO, VU, or Reset tests. They can be understood as a minimal complete set of tests or Smoke tests.

[0033] For example, if there are 20 test cases for I / O, then select two test cases from these 20, one for sequential read / write and one for random read / write, and mark them both as Basic (basic test set). Again, as an example, if there are multiple test cases for Reset, then select one test case from among these test cases that includes all Reset methods and mark it as Basic.

[0034] Regardless of which functional module is modified in the firmware, the Basic contains a minimal test set for all functional modules, ensuring that all functional modules can function properly.

[0035] For example, if the IO module is modified, Basic testing can ensure that even if the modification occurs, the device can still perform IO, Reset, firmware upgrade and downgrade operations normally.

[0036] The basic test is executed once. After the basic test is completed, step S3 is executed.

[0037] Step S3: Determine the firmware module to be modified based on the functional modules to be modified, and determine the weighting value and number of executions for the modified firmware module test.

[0038] The firmware modules to be modified include the functional modules that need to be modified by searching for keywords in the modified code based on the path of the firmware file to be modified. The functional modules that need to be modified constitute the firmware modules to be modified, and the firmware modules to be modified contain tags for related test cases.

[0039] The modified firmware modules are labeled and associated with the test cases. After association, each modified firmware module may correspond to multiple test items.

[0040] As is understandable, test items are the keywords of test cases. Keywords can be represented by tags. When fixed software changes are associated with test items through tags, they are also associated with the test cases corresponding to the test items.

[0041] One or more functional modules can be modified at a time. If the bootloader module is modified, the modification will primarily affect firmware upgrades, and secondarily affect resets. Therefore, the corresponding test items will be firmware upgrade and reset, and the bootloader module will include the tags "upgrade" and "reset". Again, as an example, if the journal and garbage collection (GC) modules are modified, the corresponding test items such as Journal, IO, GC, and Reset will all be affected, and will include the relevant tags.

[0042] Each modified firmware module is assigned a weight for a corresponding test item based on its focus. The weight determines the execution priority of each test item. The higher the weight of a test item for a modified firmware module, the higher the priority of that test item. The weight range is [0, 100].

[0043] Test items unrelated to the modified firmware modules have their weights set to 0 and are not tested. For example, modifying the bootloader module has no impact on I / O, so the I / O weight is set to 0.

[0044] The loop (number of executions) for each test item is assigned based on its relevance to the modified firmware module. Relevance includes the degree of relevance to the modified firmware module, the time required for test execution, and the probability of encountering problems during the test.

[0045] Specifically, considering only the relevance to a fixed module, the more relevant the test item is to the modified firmware module, the higher its weighting value and the greater the number of executions. However, if the probability of encountering problems is also considered, for example, when testing for a firmware update, a few loops are generally sufficient to ensure mutual upgrades between versions. But for a reset test, a few loops are often insufficient to detect problems, potentially requiring more loops. Therefore, sometimes it's necessary to allocate more loops to test items with low weighting values ​​to ensure test reliability. If execution time is considered, such as for IO tests, which have longer execution times, fewer loops can be allocated if the test item has a higher weighting value.

[0046] The dictionary format for Weight and Loop is: {modified firmware module 1: {test item 1: {Weight1, Loop1}, test item 2: {Weight2, Loop2} ... test item n: {Weightn, Loopn}} ,... modified firmware module N}.

[0047] For example, if modifications are made to the Bootloader module, the FW Update test should be performed first. Therefore, the Weight for this test should be set to 100, and the Loop (number of executions) to 10, ensuring the upgrade operation executes smoothly. Next, the Reset and Power cycle tests should be performed, with the Weight set to 50 and the Loop to 20. I / O has less correlation with the Bootloader module, so the Weight and Loop should be set to 1. The I / O test should be performed only after the FW Update, Reset, and Power cycle tests are completed.

[0048] It is worth noting that if there are test items with the same weight, the execution order is randomly selected.

[0049] It is worth noting that the Loop of the basic test set is set to 1 and is called first regardless of the weighting value. After the basic test set is completed, the remaining tests are executed according to the weighting value.

[0050] For example, if changes are made to the IO module, after the basic test set is completed, the remaining Label tests should be called according to Weight. At this time, the IO weight is the highest. Since the IO test takes a long time, there are fewer Loops. For example, the test runs 2 to 5 Loops.

[0051] For example, the assigned weighting value is related to whether there are firmware changes. The more relevant it is to firmware changes, the larger the weighting value and the more times it is executed. The probability of problems being encountered during testing is also taken into consideration. For example, in upgrade / downgrade testing, a few loops are usually enough to ensure mutual upgrades between versions. However, in reset testing, a few loops are generally unlikely to encounter problems. Therefore, sometimes it is necessary to allocate more loops to those with lower weighting values ​​to ensure the reliability of the test.

[0052] Step S4: Execute the test cases corresponding to the modified firmware module according to the weighted value and the number of executions.

[0053] Based on the files in the modified paths and keywords in the modified code, we can determine which functional modules have been modified and then perform tests.

[0054] Specifically, during the testing process, based on the aforementioned association between the modified firmware modules and test cases, the associated test cases are determined in the test parameters, resulting in the corresponding test items, their weighted values, and the number of executions. Test items are executed according to their weighted values, prioritizing the test item with the highest weighted value. After the test item with the highest weighted value has been executed according to the required number of executions, the test switches to the next highest weighted value to continue testing.

[0055] The test items that iterate through all modified firmware modules include two iteration methods. One method is to execute the same test cases on all test machines, which takes a long time and is suitable for SKU (Stock Keeping Unit) coverage. The other method is to assign different test cases to each test machine, with each test case executed only once, which is suitable for fast test coverage.

[0056] After all the test items associated with the modified firmware module have been executed according to the set loop, check whether the test duration has been reached.

[0057] If the test duration is not reached, the test case associated with the modified firmware module will be re-executed. Once the test duration is reached, the test results will be tallied.

[0058] After analyzing the test results, it is also necessary to check if any test cases have reported errors.

[0059] Specifically, after each test case completes one round, if the test duration is reached, a check is performed to see if any test cases have failed. If a test case has failed, the corresponding test machine exits the test, and the "debug.log" log file is automatically collected. A JIRA (project and event tracking software) is created and copied to relevant personnel. If no test cases have failed, the remaining test machines continue to the next round of testing until the test duration is reached again, at which point the test results are obtained and checked.

[0060] The next round of testing involves executing the test cases corresponding to the remaining labels according to their weight values.

[0061] For example, such as Figure 2 The diagram shown illustrates the automated testing process provided in this embodiment, which includes the following steps:

[0062] Step Q1: The process begins.

[0063] Step Q2: Obtain test parameters.

[0064] Step Q3: Check if the test duration has been reached.

[0065] The process ends when the test duration is reached.

[0066] If the test duration is not reached, proceed to step Q4.

[0067] Step Q4: Obtain a list containing test cases and test machine hosts.

[0068] Step Q5: Obtain the test items corresponding to the test cases.

[0069] Step Q6: Obtain the modified firmware module by modifying the path, file name, and keywords.

[0070] Step Q7: Obtain the weighted value and execution count of the test item corresponding to the modified firmware module.

[0071] Step Q8: Perform the test item as a basic test.

[0072] Step Q9: Obtain the test results and check if any test cases are reporting errors.

[0073] The process ends when a test case reports an error.

[0074] If no test cases report errors, the current test is considered to have been executed successfully, and step Q10 is executed.

[0075] Step Q10: Execute the remaining test items in the order of the set weighted values.

[0076] Tests are executed starting with those having the largest weighted values.

[0077] After executing the next test item with the next higher weight, proceed to step Q11.

[0078] Step Q11: Obtain the test results and check if any test cases report errors.

[0079] If no test cases report errors, the current test is considered to have been executed successfully, and the process returns to step Q3 to determine whether the test duration has been reached.

[0080] The process ends when a test case reports an error.

[0081] Example 2

[0082] like Figure 3 As shown, this application provides a test case scheduling device 100 to improve the effectiveness of automated testing, comprising:

[0083] The test parameter acquisition module 101 is used to acquire test parameters in response to code submission. The test parameters include test duration and test cases.

[0084] Basic test startup module 102 is used to start basic tests.

[0085] The modified firmware module weighting value and execution count determination module 103 is used to determine the modified firmware module according to the functional module to be modified, and to determine the weighting value and execution count of the modified firmware module test execution.

[0086] The test case execution module 104 is used to execute test cases corresponding to the modified firmware module according to the weighting value and the number of executions. Example 3

[0087] like Figure 4 As shown, this application provides a storage device 200, which includes a processor 201 and a memory 202. The memory 202 includes an intelligent control program. When the intelligent control program is executed by the processor 201, it implements the test case scheduling method described above for improving the effectiveness of automated testing.

[0088] This application also provides a computer storage medium storing computer instructions, which, when invoked, are used to execute the test case scheduling method for improving the effectiveness of automated testing.

[0089] The embodiments disclosed in this invention provide a computer-readable storage medium storing computer program instructions. When the computer program instructions are executed on a computer, the computer executes the aforementioned test case scheduling method for improving the effectiveness of automated testing.

[0090] This invention provides a processor for processing the above-described test case scheduling method to improve the effectiveness of automated testing.

[0091] In this embodiment of the invention, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0092] The various methods, steps, and logic diagrams disclosed in the embodiments of this invention can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor reads information from the storage medium and, in conjunction with its hardware, completes the steps of the above methods.

[0093] The storage medium can be memory, such as volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0094] Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM).

[0095] This application has the following beneficial effects:

[0096] (1) By setting up a basic test set, this application can ensure the normal operation of the basic functions of the functional modules. On this basis, by screening and modifying a functional module, it can determine whether it affects the basic functions of other functional modules, thus enhancing the testability.

[0097] (2) This application associates the functional modules that need to be modified with test cases and sets the weighting value and execution number of the corresponding test items. By changing the weighting value and execution number of the corresponding test items, more accurate test coverage can be achieved, and the probability of finding problems can be increased.

[0098] Although the examples referenced in this application are described for illustrative purposes only and not for limiting the scope of this application, changes, additions and / or deletions to the implementation may be made without departing from the scope of this application.

[0099] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A test case scheduling method to improve the effectiveness of automated testing, characterized in that, Includes the following steps: In response to code submission, test parameters are obtained, including test duration and test cases; Start basic testing; Determine the firmware modules to be modified based on the required functional modules, and determine the weighting value and number of executions for testing the modified firmware modules; Tests are performed on the corresponding test cases of the modified firmware module based on the weighted value and the number of executions. Each test case contains one or more test items, and the method further includes: Set different labels for each test case based on the test items it contains; The modified firmware module includes: Based on the required firmware file path and keyword search in the modified code, determine the functional modules that need to be modified. The modified firmware modules include tags for associated test cases.

2. The test case scheduling method for improving the effectiveness of automated testing as described in claim 1, characterized in that, The weighted values ​​and number of executions for determining the modified firmware module tests include: Assign a weighted value for each modified firmware module to the corresponding test item; The number of times each test item is executed is assigned based on its relevance to the modified firmware module.

3. The test case scheduling method for improving the effectiveness of automated testing as described in claim 2, characterized in that, The process of executing test cases corresponding to the modified firmware module based on weighted values ​​and execution counts includes: Execute the test item with the highest weighted value first; After executing the test item with the highest weighting value according to the number of times the response is completed, switch to the test item with the next highest weighting value to continue testing.

4. The test case scheduling method for improving the effectiveness of automated testing as described in claim 2, characterized in that, The number of times each test item is executed is assigned based on its relevance to the modified firmware module, including: The number of times each test item is executed is allocated based on its relevance to the modified firmware module, the time required for test execution, and the probability of encountering problems during testing.

5. The test case scheduling method for improving the effectiveness of automated testing as described in any one of claims 1-4, characterized in that, The method further includes: After each round of test case execution, check whether the test duration has been reached. If the test duration is reached, the test results will be statistically analyzed. If the test duration is not reached, the test case will be re-executed.

6. The test case scheduling method for improving the effectiveness of automated testing as described in claim 5, characterized in that, Also includes: After each round of testing is completed, check if any test cases report errors; If no errors are reported, continue to the next round of testing to obtain the corresponding test results; The next round of testing involves executing the test cases corresponding to the remaining labels according to their weight values.

7. A test case scheduling device for improving the effectiveness of automated testing, characterized in that, include: The test parameter acquisition module is used to obtain test parameters in response to code submissions. Test parameters include test duration and test cases. The basic test startup module is used to start basic tests; The modified firmware module weighting value and execution count determination module is used to determine the modified firmware module according to the functional module to be modified, and to determine the weighting value and execution count of the modified firmware module test execution; The test case execution module is used to execute test cases corresponding to the modified firmware modules based on weighted values ​​and execution counts. Each test case contains one or more test items, and the device further includes: Set different labels for each test case based on the test items it contains; The modified firmware module includes: Based on the required firmware file path and keyword search in the modified code, determine the functional modules that need to be modified. The modified firmware modules include tags for associated test cases.

8. A storage device, characterized in that, The storage device includes: a processor and memory. The memory includes an intelligent control program, which, when executed by the processor, implements a test case scheduling method for improving the effectiveness of automated testing as described in any one of claims 1-6.

Images