Test method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a test method and device, electronic equipment and a storage medium. An optimization method comprises the following steps: acquiring test data of a plurality of test suites when testing target software; wherein each test suite comprises a plurality of test scripts; determining a first index value and / or a second index value corresponding to each test script according to the test data; wherein the first index value represents the probability that the test script misreports the target software defect, and the second index value represents the performance that the test script identifies the target software defect; determining a target test script in the plurality of test scripts according to the first index value and / or the second index value; and optimizing the target test script, and / or determining a target test suite in the plurality of test suites according to the execution frequency of the plurality of test suites, and transferring the target test script into the target test suite. Through the embodiment of the invention, the quality and execution efficiency of software testing can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer software testing, and in particular to a testing method and device, electronic equipment, and storage medium. Background Art

[0002] In the development cycle of software products, software testing is not only to ensure the quality of software products, but also to verify whether the various components or modules of the software meet their design requirements and whether they can be properly integrated to produce the expected results. The process of software testing includes the design, development and execution of various test cases. Each test case corresponds to a specific function or a single interface of a software module and contains several test conditions, test scripts and expected results of each major step. Test scripts can be executed manually or through automated testing tools.

[0003] In the related art, after the development of the test script is completed, the tester will define one or more complete test suites according to the test requirements, each test suite consists of multiple test scripts, and then automatically and repeatedly execute the test suite regularly, and the execution frequency of each test suite is different. However, with the continuous increase of test scripts, the functions and modules of the tested software are also constantly changing and iterating. When the test scripts are automatically executed, a large number of false positive failures (referring to test failures that are not caused by software product defects) will occur, or some scripts have become invalid, which will make the execution time of the test scripts increase exponentially, and the efficiency of discovering the real defects of the software product will be greatly reduced. Summary of the invention

[0004] In view of the above problems, a testing method and device, electronic device, and storage medium are proposed to overcome the above problems or at least partially solve the above problems, including:

[0005] A testing method, comprising:

[0006] Acquire test data of multiple test suites when testing the target software; wherein each of the test suites includes multiple test scripts;

[0007] Determine a first indicator value and / or a second indicator value corresponding to each of the test scripts according to the test data; wherein the first indicator value represents the probability that the test script falsely reports the target software defect, and the second indicator value represents the performance of the test script in identifying the target software defect;

[0008] Determine a target test script among the multiple test scripts according to the first indicator value and / or the second indicator value;

[0009] The target test script is optimized, and / or a target test suite is determined among the multiple test suites according to execution frequencies of the multiple test suites, and the target test script is transferred to the target test suite.

[0010] Optionally, the test data includes multiple types, and determining the first indicator value and / or the second indicator value corresponding to each test script according to the test data includes:

[0011] In the test data, multiple target test data are determined, and a weight value corresponding to each of the target test data is determined;

[0012] According to the test data and the weight value, a first indicator value and / or a second indicator value corresponding to each of the test scripts is determined.

[0013] Optionally, before obtaining test data of multiple test suites when testing the target software, the method further includes:

[0014] Testing the target software using the multiple test suites;

[0015] After optimizing the target test script and / or determining a target test suite from among the multiple test suites according to the execution frequencies of the multiple test suites and transferring the target test script to the target test suite, the method further includes:

[0016] The target software is tested using the plurality of test suites including the target test scripts.

[0017] Optionally, the first indicator value and the second indicator value are numerical indicator values, and determining a target test script from the multiple test scripts according to the first indicator value and / or the second indicator value includes:

[0018] Sorting the multiple test scripts according to the first indicator value to obtain a first sorting, and / or sorting the multiple test scripts according to the second indicator value to obtain a second sorting;

[0019] The test scripts that conform to a first preset order in the first sorting and / or conform to a second preset order in the second sorting are determined as the target test scripts.

[0020] Optionally, determining a target test suite from among the multiple test suites according to execution frequencies of the multiple test suites includes:

[0021] Obtaining execution frequencies of the multiple test suites;

[0022] Among the multiple test suites, the test suite whose execution frequency is less than or equal to a preset execution frequency is determined as the target test suite.

[0023] Optionally, determining a weight value corresponding to each type of target test data includes:

[0024] Determining the development stage corresponding to the target software;

[0025] A weight value corresponding to each type of the target test data is determined according to the development stage.

[0026] Optionally, the test data includes the number of times the test script is run within a first preset time period, a first test failure rate of test failures caused by defects other than the target software within the first preset time period, a second test failure rate of test failures caused by defects other than the target software within a second preset time period, and the number of test failures caused by undetermined causes within the first preset time period; the test data includes the average running time of the test script within a third preset time period, the first number of target software defects identified by the test script within the third preset time period, the third test failure rate of test failures caused by the target software defects within the third preset time period, and the second number of code blocks or code lines tested for the target software;

[0027] Determining a plurality of target test data in the test data, and determining a weight value corresponding to each of the target test data, includes:

[0028] Determine first weight values ​​corresponding to the first test failure rate, the second test failure rate, and the number of test failures, respectively, and / or determine second weight values ​​corresponding to the average running time, the first number, the third test failure rate, and the second number, respectively;

[0029] Determining the first indicator value and / or the second indicator value corresponding to each of the test scripts according to the test data and the weight value includes:

[0030] Determine the first indicator value corresponding to each of the test scripts based on the number of runs, the first test failure rate, the second test failure rate, the number of test failures, and the first weight value, and / or determine the second indicator value corresponding to each of the test scripts based on the average running time, the first number, the third test failure rate, the second number, and the second weight value.

[0031] A testing device, comprising:

[0032] A test data acquisition module, used to acquire test data of multiple test suites when testing the target software; wherein each of the test suites includes multiple test scripts;

[0033] An indicator determination module, used to determine a first indicator value and / or a second indicator value corresponding to each of the test scripts according to the test data; wherein the first indicator value represents the probability that the test script falsely reports a defect of the target software, and the second indicator value represents the performance of the test script in identifying defects of the target software;

[0034] a target test script determination module, configured to determine a target test script from among the plurality of test scripts according to the first indicator value and the second indicator value;

[0035] The target test script processing module is used to optimize the target test script and / or determine a target test suite among the multiple test suites according to the execution frequencies of the multiple test suites and transfer the target test script to the target test suite.

[0036] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the above-mentioned test method when executed by the processor.

[0037] A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and the computer program implements the above-mentioned testing method when executed by a processor.

[0038] The embodiments of the present invention have the following advantages: by acquiring test data of multiple test suites when testing target software, wherein each test suite includes multiple test scripts, a first indicator value and a second indicator value corresponding to each test script are determined according to the test data, wherein the first indicator value represents the probability of the test script falsely reporting a target software defect, and the second indicator value represents the performance of the test script in identifying the target software defect, a target test script is determined from multiple test scripts according to the first indicator value and / or the second indicator value, the target test script is optimized, and / or a target test suite is determined from multiple test suites according to the execution frequency of multiple test suites, and the target test script is transferred to the target test suite, thereby realizing automatic optimization of test scripts that need to be optimized, and / or dynamic adjustment of test scripts in the test suite, so that the test script can adapt to the testing requirements of the software in a timely manner, test the software in a more targeted manner, and effectively improve the testing efficiency of software testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0040] Figure 1 is a flow chart of steps of a testing method provided by one embodiment of the present invention;

[0041] Figure 2 is a software testing system architecture diagram provided by an embodiment of the present invention;

[0042] Figure 3 A diagram of a software and hardware organizational structure for software testing provided by an embodiment of the present invention;

[0043] Figure 4 It is an iterative flow chart of software testing provided by one embodiment of the present invention;

[0044] Figure 5 It is a structural block diagram of a testing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] In the development cycle of software products, software testing is not only to ensure the quality of software products, but also to verify whether the various components or modules of the software meet their design requirements and whether they can be properly integrated to produce the expected results. The process of software testing includes the design, development and execution of various test cases. Each test case corresponds to a specific function or a single interface of a software module and contains several test conditions, test scripts and expected results of each major step. Test scripts can be executed manually or through automated testing tools.

[0047] In the related art, after the development of the test script is completed, the tester will define one or more complete test suites according to the test requirements, each test suite consists of multiple test scripts, and then automatically and repeatedly execute the test suite regularly, and the execution frequency of each test suite is different. However, with the continuous increase of test scripts, the functions and modules of the tested software are also constantly changing and iterating. When the test scripts are automatically executed, a large number of false positive failures (referring to test failures that are not caused by software product defects) will occur, or some scripts have become invalid, which will make the execution time of the test scripts increase exponentially, and the efficiency of discovering the real defects of the software product will be greatly reduced.

[0048] For example, when the execution of a test script may conflict with other simultaneously executed test script resources, timing problems caused by the response speed of the software under test, unavailability of external resources (such as network resources), random failures or intermittent race conditions, etc., the test script may be falsely judged to fail.

[0049] Reference Figure 1 , shows a flow chart of a test method provided by an embodiment of the present invention, which may specifically include the following steps:

[0050] Step 101, obtaining test data of multiple test suites when testing target software; wherein each of the test suites includes multiple test scripts;

[0051] A test suite consists of multiple test scripts, which can be divided and integrated into different test cases to generate a test suite. The test suite can be executed immediately to test the target software, or it can be executed periodically according to the preset execution frequency according to the test requirements in software automation testing.

[0052] Test data refers to relevant data in the process of testing the target software, such as the test results after executing the test script to test a function or module in the target software. The test results can be test pass, test failure, target software output not meeting expectations, etc.; also include logs generated during the test, data output by the target software, error information of the target software, test time, test code blocks or code lines, etc.

[0053] In actual applications, after executing multiple test suites to test the target software and obtaining test data, the test data can be saved in the test database and wait for being called. For a software product composed of many modules and having multiple versions or releases, the test database contains a large number of test scripts, and each test script may have test data with execution history of several months or years.

[0054] Step 102, determining a first indicator value and a second indicator value corresponding to each of the test scripts according to the test data; wherein the first indicator value represents the probability that the test script falsely reports the target software defect, and the second indicator value represents the performance of the test script in identifying the target software defect;

[0055] The probability of a test script falsely reporting a defect in the target software refers to the probability that a test failure that is not caused by a defect in the target software itself is erroneously reported by the test script, that is, the probability that the test script falsely determines a failure. When the test fails for the target software due to external reasons such as network resources, defects or limitations of the test script itself, it can be considered that the test script falsely reports a defect in the target software.

[0056] The performance of the test script in identifying defects in the target software, that is, the ability of the test script to accurately identify defects in the target software according to the expected design.

[0057] In a specific implementation, the first indicator value and the second indicator value can be numerical indicator values, such as score values. When the score value corresponding to the first indicator value is larger, the probability of a false positive failure in the test script is greater, and the reliability of the test script is lower; when the score value corresponding to the second indicator value is larger, the performance of the test script in identifying defects in the target software is higher, and the test script can more effectively and accurately identify defects in the target software that do not meet expectations.

[0058] In practical applications, after obtaining the test data, preset rules, preset algorithms, etc. may be determined to perform data analysis and processing on the test data, thereby determining the first indicator value and the second indicator value.

[0059] In actual applications, multiple test suites can also be set up on different software platforms or hardware platforms to test the target software, obtain test data for different software platforms and hardware platforms, and then obtain the first indicator value and / or second indicator value corresponding to the different software platforms and hardware platforms based on the test data. Then, the first indicator value and / or second indicator value corresponding to the different software platforms and hardware platforms are integrated through a preset algorithm or a pre-trained model. The first indicator value and / or second indicator value after data integration can reflect the comprehensive performance of the test script on multiple platforms, thereby meeting the optimization needs of multi-platform software testing.

[0060] In some embodiments of the present invention, the test data includes multiple types, and determining the first indicator value and / or the second indicator value corresponding to each test script according to the test data includes:

[0061] In the test data, multiple target test data are determined, and a weight value corresponding to each of the target test data is determined;

[0062] According to the test data and the weight value, a first indicator value and / or a second indicator value corresponding to each of the test scripts is determined.

[0063] In this embodiment, the test data includes multiple types, and the degree of influence of each test data on the first index value and / or the second index value is different. Therefore, after obtaining the test data, it is necessary to determine which target test data need to be assigned weight values, and then further determine the weight values ​​corresponding to the target test data. The first index value and / or the second index value determined by the test data and the weight value can better meet the needs of software testing and improve the accuracy of the first index value and / or the second index value in reflecting the relevant indicators.

[0064] In some embodiments of the present invention, the test data includes the number of times the test script is run within a first preset time period, a first test failure rate of test failures caused by defects other than the target software within the first preset time period, a second test failure rate of test failures caused by defects other than the target software within a second preset time period, and the number of test failures caused by undetermined causes within the first preset time period; the test data includes the average running time of the test script within a third preset time period, the first number of target software defects identified by the test script within the third preset time period, the third test failure rate of test failures caused by the target software defects within the third preset time period, and the second number of code blocks or code lines tested for the target software;

[0065] Determining a plurality of target test data in the test data, and determining a weight value corresponding to each of the target test data, includes:

[0066] Determine first weight values ​​corresponding to the first test failure rate, the second test failure rate, and the number of test failures, respectively, and / or determine second weight values ​​corresponding to the average running time, the first number, the third test failure rate, and the second number, respectively;

[0067] Determining the first indicator value and / or the second indicator value corresponding to each of the test scripts according to the test data and the weight value includes:

[0068] Determine the first indicator value corresponding to each of the test scripts based on the number of runs, the first test failure rate, the second test failure rate, the number of test failures, and the first weight value, and / or determine the second indicator value corresponding to each of the test scripts based on the average running time, the first number, the third test failure rate, the second number, and the second weight value.

[0069] The first preset time period may be a time period within the test script running cycle, such as the past week or month, and may be specifically determined according to actual application requirements.

[0070] The second preset time period may be another time period within the test script running cycle that is different from the first preset time period, such as a time period that is smaller or larger than the first preset time period. For example, if the first preset time period is the past month, the second preset time period may be set to the past three months to obtain more data samples.

[0071] The number of times the test script is run, that is, the number of times the test script executes the test on the target software and outputs the test results, is counted after each test script is executed.

[0072] In this embodiment, test failure can be divided into: test failure caused by target software defects, test failure caused by non-target software defects, and test failure caused by undetermined reasons. Test failure caused by target software defects refers to the failure of test script execution that can be determined to be caused by defects in the target software itself. Test failure caused by non-target software defects refers to the test failure that can be determined to be caused by factors other than defects in the target software itself. Test failure caused by undetermined reasons means that it cannot be determined whether the test failure is caused by a target software defect or a non-target software defect, that is, a test failure to be analyzed. In actual applications, when a test failure occurs due to an undetermined reason, the failure can be re-determined as a test failure caused by a target software defect or a test failure caused by a non-target software defect after further analysis and confirmation by the tester.

[0073] The first test failure rate is the ratio of the number of test failures caused by non-target software defects to the total number of test failures within the first preset time period; similarly, the second test failure rate is the ratio of the number of test failures caused by non-target software defects to the total number of test failures within the second preset time period.

[0074] The number of test failures for which the cause was not determined, that is, the total number of test failures for which the cause could not be determined and are pending analysis.

[0075] In a specific implementation, by obtaining test data such as the number of runs, the first test failure rate, the second test failure rate, and the number of test failures, and since the first test failure rate, the second test failure rate, the number of test failures, and the like have different degrees of importance in affecting the first index value, the first weight values ​​corresponding to the above test data can be determined respectively to improve the accuracy of the first index value. Furthermore, by analyzing and processing these test data in combination with the first weight value according to preset rules, algorithms, models, etc., the first index value can be determined to characterize the probability of the test script falsely reporting a target software defect.

[0076] As an example, the first indicator value may be determined by the following formula:

[0077] SCORE1=RUN P1 *(K1*FF P1 +K2*FF P2 +K3*INV P1 )

[0078] Among them, SCORE1 is the first index value, P1 is the first preset time period, P2 is the second preset time period, RUN P1 is the number of runs, FF P1 is the first test failure rate, FF P2 is the second test failure rate, INV P1 is the number of test failures caused by undetermined reasons, K1, K2, and K3 are the first weight values, where K1 is the weight value corresponding to the first test failure rate, K2 is the weight value corresponding to the second test failure rate, and K3 is the weight value corresponding to the number of test failures.

[0079] In actual applications, when the test script generates more errors and fails for a long time, the larger the setting values ​​of K1, K2, and K3 are, and the smaller the number of errors and fails is, the smaller the setting values ​​of K1, K2, and K3 are. For example,

[0080] For test scripts that have failed to produce errors for more than three months, K1 and K2 can be set to larger values, which means they get higher weights; in contrast, for test scripts that have only recently started to cause errors to fail, K1 and K2 can be set to smaller values. In some cases, K3 can also be set to a negative number to reflect the situation where the cause of the test failure has not been analyzed.

[0081] The third preset time period is a time period determined within the test script running cycle, which may be the same as or different from the first preset time period or the second preset time period. For example, the past six months are determined as the third preset time period.

[0082] The average running time of the test script in the third preset time period may be a ratio of the total running time of the test script to the total number of running times in the third preset time period.

[0083] The test script identifies the first number of target software defects within the third preset time period, that is, the total number of target software defects identified by executing the test script. The defects of the target software may be software errors, functional performance not meeting expectations, etc. Once the target software defects are identified, they can be counted.

[0084] a third test failure rate of test failures caused by defects in the target software within a third preset time period, and a ratio of the number of test failures caused by defects in the target software itself to the total number of test failures within the third preset time period.

[0085] The second number of code blocks or code lines of the test target software, that is, the total number of code blocks or code lines of the target software covered by the test software when executed, can also reflect the code coverage rate of the software test.

[0086] In a specific implementation, by obtaining the average running time, the first number, the third test failure rate, and the second number, and analyzing and processing these test data according to preset rules, algorithms, models, etc., a second indicator value can be determined to characterize the performance of the test script in identifying target software defects.

[0087] As an example, the average running time, the first number, the third test failure rate, and the weights corresponding to the second number may be determined respectively and the second indicator value may be determined to improve the accuracy of the second indicator value.

[0088] In a specific implementation, the second indicator value can be determined by the following formula:

[0089] SCORE2=K4*TIME P3 +K5*BUG P3 +K6*FF P3 +K7*CODE

[0090] Among them, SCORE2 is the second index value, P3 is the third preset time period, TIME P3 is the average running time, BUG P3 is the first quantity, FF P3 is the third test failure rate, CODE is the second quantity, K4, K5, K6, and K7 are the second weight values, among which K4 is the weight value corresponding to the average running time, K5 is the weight value corresponding to the first quantity, K6 is the weight value corresponding to the third test failure rate, and K7 is the weight value corresponding to the second quantity.

[0091] In actual applications, K4 can be set according to the priority of the test script running. The higher the priority of the test script running, the larger the value of K4, and vice versa; K5 can be determined according to the average number of times the test script is run within the third preset time period. The more the average number of runs, the smaller the value of K5, and vice versa; K6 can be determined according to the total number of test failures. The more the total number of test failures, the larger the value of K6, and vice versa; K7 can be determined according to the total number of code blocks or code lines covered by the test software when it is executed. The larger the total number, the smaller the value of K7, and vice versa.

[0092] In this embodiment, the first indicator value corresponding to each test script is determined by the number of runs, the first test failure rate, the second test failure rate, the number of test failures, and the first weight value, and / or the second indicator value corresponding to each test script is determined by the average running time, the first number, the third test failure rate, the second number, and the second weight value. This can improve the accuracy of the first indicator value and the second indicator value, thereby more realistically reflecting the relevant indicators of the test script and facilitating analysis or optimization of the test script.

[0093] In some embodiments of the present invention, the method further comprises:

[0094] Determining the development stage corresponding to the target software;

[0095] A weight value corresponding to each type of the target test data is determined according to the development stage.

[0096] In this embodiment, the importance of the relevant indicators of the test script reflected by the target test data is different for target software at different development stages. For example, in the early stage of software product development, when many test scripts lead to failures, the number of test failures due to undetermined reasons may not be a key indicator, so the weight value corresponding to the number of test failures will be set to a relatively small value. However, in the later stage of the software product development cycle, the number of test failures due to undetermined reasons becomes more important, and the weight value corresponding to the number of test failures can be increased accordingly. Therefore, by determining the weight value corresponding to each type of target test data according to the development stage of the current target software, the first indicator value and the second indicator value can reflect the relevant indicators of the test script more in line with the testing requirements of the development stage, so as to make it easier to find the target test script to be optimized and improve the quality and efficiency of software testing.

[0097] Step 103, determining a target test script from the multiple test scripts according to the first indicator value and / or the second indicator value;

[0098] In this embodiment, comparison, analysis and other processing are performed based on the first indicator value and / or the second indicator value corresponding to each test script in multiple test scripts, so as to determine the target test script that needs to be optimized and / or adjusted.

[0099] In some embodiments of the present invention, the first indicator value and the second indicator value are numerical indicator values, and determining a target test script from the plurality of test scripts according to the first indicator value and / or the second indicator value includes:

[0100] Sorting the multiple test scripts according to the first indicator value to obtain a first sorting, and sorting the multiple test scripts according to the second indicator value to obtain a second sorting;

[0101] The test scripts that conform to a first preset order in the first sorting and / or conform to a second preset order in the second sorting are determined as the target test scripts.

[0102] In a specific implementation, the first indicator value and the second indicator value may be numerical indicators such as score values. The larger the score value corresponding to the first indicator value, the greater the probability that the test script will fail due to misjudgment, and the lower the reliability of the test script. Exemplarily, the test scripts may be sorted from large to small according to the first indicator value to obtain a first sort and stored in a list. The higher the order, the lower the reliability of the test script. Furthermore, the test scripts that meet the first sort are determined as target test scripts, that is, the test scripts whose test components need to be optimized and / or adjusted. The first sort may be the first N-bit order in the first sort, or a specified order, such as the first or second place, etc.

[0103] Similarly, the larger the score value corresponding to the second indicator value, the higher the performance of the test script in identifying defects in the target software, and the test script can more effectively and accurately identify defects in the target software that do not meet expectations. Exemplarily, the test scripts can be sorted from small to large according to the second indicator value to obtain a second sort and stored in a list. The higher the order, the weaker the performance of the test script in identifying the target software. Further, the test scripts that meet the second order are determined as target test scripts, that is, the test scripts whose test components need to be optimized and / or adjusted. The second order can be the first N digits in the first sort, or a specified order, such as the first digit, the second digit, etc.

[0104] In this embodiment, by sorting the test scripts according to the first indicator value and the second indicator value, the reliability and performance comparison among multiple test scripts can be reflected, and then the target test script that needs to be optimized or the test component adjusted can be quickly located among multiple test scripts.

[0105] Step 104: optimize the target test script, and / or determine a target test suite from the multiple test suites according to the execution frequencies of the multiple test suites, and transfer the target test script to the target test suite.

[0106] In this embodiment, after determining the target test script that needs to be optimized or / transferred to the test suite, if the test script needs to be optimized, the target test script can be automatically optimized for reliability or performance by running the optimization software, or the relevant development or testing personnel can perform problem location analysis on the target test script and then optimize the source code or parameters of the target test script to iterate the target test script; if it is necessary to adjust the test suite to which the test script belongs, for example, transferring a test script with lower reliability or poorer performance to a test suite with lower execution frequency to avoid affecting the operating efficiency and test quality of the test suite to which it originally belongs, the target test suite can be determined from multiple test suites based on the execution frequency of the multiple test suites, and the target test script can be transferred to the target test suite to improve the overall quality and efficiency of software testing.

[0107] In actual applications, after a target test script is determined, if it is determined that the target test script has become invalid or cannot be optimized and maintained, the target test script may be removed from the test suite to which it belongs.

[0108] In some embodiments of the present invention, determining a target test suite from among the multiple test suites according to the execution frequencies of the multiple test suites includes:

[0109] Obtaining execution frequencies of the multiple test suites;

[0110] Among the multiple test suites, the test suite whose execution frequency is less than or equal to a preset execution frequency is determined as the target test suite.

[0111] In this embodiment, the execution frequencies of multiple test suites are obtained, and the execution frequencies can reflect the importance of the test suites in testing a certain functional module of the target software. The higher the execution frequency, the more important the tested function or module is to the test project.

[0112] Furthermore, among multiple test suites, a test suite with an execution frequency less than or equal to a preset execution frequency is determined as a target test suite, and the target test script is transferred to the target test suite. Exemplarily, if the first indicator value reflects that the target test script has a high probability of falsely reporting a target software defect, and / or the second indicator value reflects that the target test script has a low performance in identifying the target software defect, the target test script is transferred to the target test suite to reduce the adverse effects of the reliability, effectiveness, and other issues of the target test script on the entire test software process.

[0113] In practical applications, the first indicator value and the second indicator value may also be used to determine test scripts with high reliability and effectiveness, so as to transfer these test scripts to a test suite with a higher execution frequency, so as to fully and reasonably utilize computing resources.

[0114] In some embodiments of the present invention, before obtaining test data of multiple test suites when testing the target software, the method further includes:

[0115] Testing the target software using the multiple test suites;

[0116] After optimizing the target test script and / or determining a target test suite from among the multiple test suites according to the execution frequencies of the multiple test suites and transferring the target test script to the target test suite, the method further includes:

[0117] The target software is tested using the plurality of test suites including the target test scripts.

[0118] In a specific implementation, after the target software is tested using multiple test suites and test data is obtained, a first indicator value and / or a second indicator value is determined based on the test data, and then a target test script is determined based on the first indicator value and / or the second indicator value, and the target test script is further optimized, and / or the target test script is transferred to a target test suite, and finally the target software is tested again based on the adjusted multiple test suites containing the target test script. At this time, new test data will be generated, thereby realizing dynamic optimization of the test suite during the testing of the target software, and continuously improving the quality and efficiency of software testing.

[0119] In some embodiments of the present invention, reference Figure 2 , also provides a software testing system, which includes a development server and a test workstation, which are interconnected through a network, and the application program on the test workstation communicates with the application program on the development server through the network.

[0120] The development server includes a test database, an automated test server, a scoring program, and a ranking server. The test database is used to store test suites or test sets, test scripts, and test data (such as test results and execution data). The automated test server is used to automatically execute the test suites. The scoring program is used to interact with the test database to obtain test data and determine the first indicator value and the second indicator value corresponding to each test script. The first indicator value and the second indicator value can specifically be scores. The ranking server is used to rank the test scripts according to the first indicator value and / or the second indicator value.

[0121] The test workstation includes an automated test client, a ranking client, and the software under test (ie, target software). The automated test client is used to interact with the automated test server, and the ranking client is used to interact with the ranking server.

[0122] In actual applications, testers run automated test clients on test workstations to test target software. Automated test clients connect to automated test servers running on development servers. Automated test servers are further connected to test databases. Test scripts are stored in the test database. As mentioned above, test scripts in the test database are grouped into test suites based on their execution time, scope and importance of the tested functions.

[0123] Testers use the automated test client to select the test suite to be executed against the target software. The test suite can be executed immediately, delayed, or periodically based on the test requirements. When executing each test script in the test suite, the automated test client records the execution results and other execution data of the test script in the test database through the automated test server. The test results include the success or failure of the test script, and the execution data includes the date of script execution, execution duration, code blocks or lines of code tested, etc.

[0124] Furthermore, the test database maintains corresponding first and second index values ​​for each test script. These index values ​​are calculated and processed by the test data recorded in the test database each time the test script is executed, which indicates the reliability and effectiveness of the test script in testing the software product.

[0125] Furthermore, the first indicator value and the second indicator value are calculated by a scoring algorithm running on the development server, which can be specifically expressed as a score. Each time a new test result and execution data are recorded in the test database, the scoring algorithm calculates a score for the test script. The scoring algorithm can also be run periodically to update the scores of all test scripts in the database based on the latest available test results and execution data. Based on the calculated scores, the test scripts to be executed are ranked and prioritized, and the ranking server program retrieves a list of test scripts and their corresponding scores from the test database, and presents the list to the tester through the ranking client.

[0126] Testers can adjust parameters to customize which test scripts should be included in the list and according to which score and in what order the test scripts should be ranked. The ranked list of test scripts is then used to determine which test scripts should be included in each test suite based on the frequency of execution of the test suite and the relative value of the test scripts to the testing process.

[0127] Testers can also use the ranked list of test scripts to identify test scripts that need to be analyzed to determine whether to redevelop the script or remove it from the testing process. In addition, the ranked list of test scripts can also be used to prioritize the execution order of each script in the test suite to ensure that the most valuable test scripts are executed first during the testing cycle.

[0128] In this embodiment, refer to Figure 3 , also provides a hardware organizational architecture for software testing optimization, in which a capacity storage device is connected to a central processing unit through a storage controller to store applications such as an automated testing server, an automated testing client, a scoring program (i.e., a scoring program), a ranking server or a ranking client, and communicates with the Internet through a network interface unit.

[0129] In some embodiments of the present invention, reference Figure 4 , and also provides a specific process of software testing iteration, including:

[0130] S1: Use the test suite to test the target software;

[0131] S2: save test data;

[0132] S3: Determine a first indicator value and a second indicator value of the test script according to the test data;

[0133] S4: sorting the test scripts according to the first indicator value and / or the second indicator value to obtain a first sorting and a second sorting;

[0134] S5: Analyze and evaluate the first sorting and the second sorting to determine the target test script;

[0135] S6: Optimize the target test script and / or adjust the target test script to the target test suite.

[0136] When S6 is executed, it jumps to S1 again to execute, so as to realize continuous iteration and optimization of the test script during the entire test cycle of the software test.

[0137] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0138] The embodiments of the present invention have the following advantages: by acquiring test data of multiple test suites when testing target software, wherein each test suite includes multiple test scripts, a first indicator value and a second indicator value corresponding to each test script are determined according to the test data, wherein the first indicator value represents the probability of the test script falsely reporting a target software defect, and the second indicator value represents the performance of the test script in identifying the target software defect, a target test script is determined from multiple test scripts according to the first indicator value and / or the second indicator value, the target test script is optimized, and / or a target test suite is determined from multiple test suites according to the execution frequency of multiple test suites, and the target test script is transferred to the target test suite, thereby realizing automatic optimization of test scripts that need to be optimized, and / or dynamic adjustment of test scripts in the test suite, so that the test script can adapt to the testing requirements of the software in a timely manner, test the software in a more targeted manner, and effectively improve the testing efficiency of software testing.

[0139] Reference Figure 5 , shows a schematic diagram of the structure of a testing device provided by an embodiment of the present invention, which may specifically include the following modules:

[0140] The test data acquisition module 501 is used to acquire test data of multiple test suites when testing the target software; wherein each of the test suites includes multiple test scripts;

[0141] An indicator determination module 502 is used to determine a first indicator value and / or a second indicator value corresponding to each of the test scripts according to the test data; wherein the first indicator value represents the probability that the test script falsely reports a defect of the target software, and the second indicator value represents the performance of the test script in identifying defects of the target software;

[0142] A target test script determination module 503, configured to determine a target test script from among the plurality of test scripts according to the first indicator value and the second indicator value;

[0143] The target test script processing module 504 is used to optimize the target test script and / or determine a target test suite among the multiple test suites according to the execution frequencies of the multiple test suites and transfer the target test script to the target test suite.

[0144] In some embodiments of the present invention, the test data includes multiple types, and the indicator determination module 502 includes:

[0145] A weight value determination submodule, used to determine a plurality of target test data in the test data, and determine a weight value corresponding to each of the target test data;

[0146] The indicator value determination submodule is used to determine the first indicator value and / or the second indicator value corresponding to each of the test scripts according to the test data and the weight value.

[0147] In some embodiments of the present invention, the device further comprises:

[0148] A first testing module, configured to test the target software using the multiple test suites;

[0149] The second testing module is used to test the target software using the multiple testing suites including the target testing scripts.

[0150] In some embodiments of the present invention, the target test script determination module 503 includes:

[0151] A sorting submodule, used for sorting the multiple test scripts according to the size of the first indicator value to obtain a first sorting, and / or, sorting the multiple test scripts according to the size of the second indicator value to obtain a second sorting;

[0152] The target test script determination submodule is used to determine the test script that complies with the first preset order in the first sorting and / or complies with the second preset order in the second sorting as the target test script.

[0153] In some embodiments of the present invention, the target test script processing module 504 includes:

[0154] An execution frequency acquisition submodule, used to acquire the execution frequencies of the multiple test suites;

[0155] The target test suite determination submodule is used to determine, among the multiple test suites, a test suite whose execution frequency is less than or equal to a preset execution frequency as the target test suite.

[0156] In some embodiments of the present invention, the weight value determination submodule includes:

[0157] A development stage determination subunit, used to determine the development stage corresponding to the target software;

[0158] The second weight value determination subunit is used to determine the weight value corresponding to each type of test data according to the development stage.

[0159] In some embodiments of the present invention, the test data includes the number of times the test script is run within a first preset time period, a first test failure rate of test failures caused by defects other than the target software within the first preset time period, a second test failure rate of test failures caused by defects other than the target software within a second preset time period, and the number of test failures caused by undetermined causes within the first preset time period; the test data includes the average running time of the test script within a third preset time period, the first number of target software defects identified by the test script within the third preset time period, the third test failure rate of test failures caused by the target software defects within the third preset time period, and the second number of code blocks or code lines tested for the target software;

[0160] The weight value determination submodule also includes:

[0161] A weight value determination subunit, used to respectively determine the first test failure rate, the second test failure rate, and the first weight value corresponding to each of the test failures, and / or to respectively determine the second weight value corresponding to each of the average running time, the first number, the third test failure rate, and the second number;

[0162] The indicator value determination submodule includes:

[0163] An indicator value determination subunit is used to determine the first indicator value corresponding to each of the test scripts based on the number of runs, the first test failure rate, the second test failure rate, the number of test failures, and the first weight value, and / or to determine the second indicator value corresponding to each of the test scripts based on the average running time, the first number, the third test failure rate, the second number, and the second weight value.

[0164] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0165] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0166] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0167] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0168] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0169] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0170] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0171] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0172] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the above elements.

[0173] The above provides a detailed introduction to a testing method and device, an electronic device, and a storage medium. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for a person skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A testing method, characterized in that: The method comprises: Acquire test data of multiple test suites when testing the target software; wherein each of the test suites includes multiple test scripts; Determine a first indicator value and / or a second indicator value corresponding to each of the test scripts according to the test data; wherein the first indicator value represents the probability that the test script falsely reports the target software defect, and the second indicator value represents the performance of the test script in identifying the target software defect; Determine a target test script among the multiple test scripts according to the first indicator value and / or the second indicator value; The target test script is optimized, and / or a target test suite is determined among the multiple test suites according to execution frequencies of the multiple test suites, and the target test script is transferred to the target test suite.

2. The method according to claim 1, characterized in that The test data includes multiple types, and determining the first indicator value and / or the second indicator value corresponding to each test script according to the test data includes: In the test data, multiple target test data are determined, and a weight value corresponding to each of the target test data is determined; According to the test data and the weight value, a first indicator value and / or a second indicator value corresponding to each of the test scripts is determined.

3. The method according to claim 1, characterized in that Before obtaining test data of multiple test suites when testing the target software, the method further includes: Testing the target software using the multiple test suites; After optimizing the target test script and / or determining a target test suite from among the multiple test suites according to the execution frequencies of the multiple test suites and transferring the target test script to the target test suite, the method further includes: The target software is tested using the plurality of test suites including the target test scripts.

4. The method according to any one of claims 1 to 3, characterized in that: The first indicator value and the second indicator value are numerical indicator values, and determining a target test script from the plurality of test scripts according to the first indicator value and / or the second indicator value includes: Sorting the multiple test scripts according to the first indicator value to obtain a first sorting, and / or sorting the multiple test scripts according to the second indicator value to obtain a second sorting; The test scripts that conform to a first preset order in the first sorting and / or conform to a second preset order in the second sorting are determined as the target test scripts.

5. The method according to any one of claims 1 to 3, characterized in that: The step of determining a target test suite from among the multiple test suites according to the execution frequencies of the multiple test suites comprises: Obtaining execution frequencies of the multiple test suites; Among the multiple test suites, the test suite whose execution frequency is less than or equal to a preset execution frequency is determined as the target test suite.

6. The method according to claim 2, characterized in that Determining the weight value corresponding to each type of target test data includes: Determining the development stage corresponding to the target software; A weight value corresponding to each type of the target test data is determined according to the development stage.

7. The method according to claim 2, characterized in that The test data includes the number of times the test script is run within a first preset time period, a first test failure rate of test failures caused by defects other than the target software within the first preset time period, a second test failure rate of test failures caused by defects other than the target software within a second preset time period, and the number of test failures caused by undetermined causes within the first preset time period; the test data includes the average running time of the test script within a third preset time period, the first number of target software defects identified by the test script within the third preset time period, the third test failure rate of test failures caused by defects in the target software within the third preset time period, and the second number of code blocks or code lines tested for the target software; Determining a plurality of target test data in the test data, and determining a weight value corresponding to each of the target test data, includes: Determine first weight values ​​corresponding to the first test failure rate, the second test failure rate, and the number of test failures, respectively, and / or determine second weight values ​​corresponding to the average running time, the first number, the third test failure rate, and the second number, respectively; Determining the first indicator value and / or the second indicator value corresponding to each of the test scripts according to the test data and the weight value includes: Determine the first indicator value corresponding to each of the test scripts based on the number of runs, the first test failure rate, the second test failure rate, the number of test failures, and the first weight value, and / or determine the second indicator value corresponding to each of the test scripts based on the average running time, the first number, the third test failure rate, the second number, and the second weight value.

8. A testing device, characterized in that: The device comprises: A test data acquisition module, used to acquire test data of multiple test suites when testing the target software; wherein each of the test suites includes multiple test scripts; An indicator determination module, used to determine a first indicator value and / or a second indicator value corresponding to each of the test scripts according to the test data; wherein the first indicator value represents the probability that the test script falsely reports a defect of the target software, and the second indicator value represents the performance of the test script in identifying defects of the target software; a target test script determination module, configured to determine a target test script from among the plurality of test scripts according to the first indicator value and the second indicator value; The target test script processing module is used to optimize the target test script and / or determine a target test suite among the multiple test suites according to the execution frequencies of the multiple test suites and transfer the target test script to the target test suite.

9. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the testing method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the test method according to any one of claims 1 to 7 is implemented.

Images