Test Strategy Automatic Generation System and Method Based on Historical Data Statistics and Analysis

Abstract

The present invention discloses a system and method for automatically generating test strategies based on historical data statistics and analysis in the field of automatic testing of relay protection devices, including: retrieving test case information from a database for testers to select; after the tester selects all required test cases, saving them as a test plan; performing regression testing based on the strategy value of each test case in the test plan and the average value of all strategy values; in response to the strategy value of any test case being not less than the average value of all strategy values, putting this test case into the test execution output strategy set for this round of testing; in response to the strategy value of any test case being less than the average value of all strategy values, multiplying its influence value by a progression coefficient for updating, and comparing it with the average value of all strategy values again before calculating the strategy value for the next round of output. The present invention converts test experience into data information for storage, which is beneficial to the development of automatic testing technology and improves test intelligence.

Description

Technical Field

[0001] The present invention relates to a system and method for automatically generating test strategies based on historical data statistics and analysis, and belongs to the technical field of automatic testing of relay protection devices. Background Art

[0002] With the development of intelligent substations, the State Grid has gradually standardized the production and detection of secondary equipment for relay protection, forming a series of operation standards, which provides favorable conditions for the automated development of relay protection testing. The scope of application of the automatic testing technology for protection devices has gradually expanded. Taking the ATS automatic testing platform developed by Guodian NARI as an example, it has been widely used in various occasions such as R & D unit testing, integration testing, sending detection testing, and user-customized testing. Moreover, the standardization of the process enables test cases to be applicable to products of different manufacturers that execute the same standard. As the scope of application expands, the number of test cases will gradually accumulate to thousands or even tens of thousands. During the new product or product update stage, and during the product modification period, testers will conduct a large number of repetitions of regression testing and testing of problem points. Due to different personal experiences, only through personal or team manual judgment, the planned test plan may lead to problems such as incomplete test coverage and low test efficiency. Therefore, it is necessary to find an objective, scientific, and quantifiable method for selecting test cases and then selecting the optimal test plan. Summary of the Invention

[0003] The purpose of the present invention is to overcome the deficiencies in the prior art, and provide a system and method for automatically generating test strategies based on historical data statistics and analysis, which converts test experience into data information for storage, facilitates the development of automatic testing technology, and improves test intelligence.

[0004] To achieve the above purpose, the present invention is implemented by the following technical solutions:

[0005] In the first aspect, the present invention provides a method for automatically generating test strategies based on historical data statistics and analysis, including:

[0006] Retrieving test case information from the database for testers to select;

[0007] After the tester selects all the required test cases, save them as a test plan;

[0008] Conducting regression testing based on the strategy value of each test case in the test plan and the average value of all strategy values;

[0009] When the strategy value of any test case is not less than the average value of all strategy values, put this test case into the test execution output strategy set for this round of testing;

[0010] When the policy value of any test case is less than the average value of all policy values, its impact value is updated by multiplying it with a progression coefficient and compared again with the average value of all policy values before calculating the policy value for the next round of output.

[0011] After the regression test is completely finished, store the test result data accumulated during the test into the database.

[0012] Furthermore, retrieve test case information from the database for testers to select, including: displaying test case information through a tree structure, where the hierarchical levels of the tree structure include region, primary function, secondary function, test input, test output, test logic, and reverse test.

[0013] Furthermore, conduct regression testing based on the policy value of each test case in the test plan and the average value of all policy values, including:

[0014] Before the start of the first round of testing, assign an initial impact value to each test case according to the coefficient and weight of the test project.

[0015] When a new problem is discovered during the testing process, update its impact value by multiplying it with the progression coefficient m times, where m is the number of test rounds.

[0016] After the execution of this round of testing is completed, proceed to the analysis of the next round of testing strategy.

[0017] Furthermore, the calculation formula for the initial impact value is:

[0018]

[0019] where Si is the impact value of each test case, ss l is the coefficient given by the test project, sp l is the weight given by the test project.

[0020] Furthermore, the coefficients given by the test project include: the coefficient for setting the automation degree of the test case, the coefficient for setting the execution duration of the test case, the coefficient for setting the reverse test case corresponding to a higher probability of repeated execution of the test case, the coefficient for the test case associated with defect tracking in the system historical data, the coefficient for the test case corresponding to the completion of problem modification in unit testing, and the coefficient for the test case that is newly required to be tested during the initial plan selection by the tester.

[0021] Further, the weights given by the test items include: the weight of the automation degree setting coefficient of the test case, the weight of the execution duration setting coefficient of the test case, the weight of the reverse test case setting coefficient corresponding to the higher the repeated execution probability of the test case, the weight of the test case coefficient related to the defect tracking in the system historical data, the weight of the test case corresponding to the problem modification completed in the unit test, and the weight of the test case that must be newly tested during the initial plan selection of the tester.

[0022] Further, the policy value of each test case is:

[0023] P i =(S i ×k1) / T i

[0024]

[0025] Wherein, P i is the policy value of each test case, Si is the influence value of each test case, T i is the execution time of each test case, k1 is the balance ratio coefficient, and n is the number of test cases selected before the test;

[0026] The average value of all the policy values is:

[0027]

[0028] Wherein, P avg is the average value of all the policy values.

[0029] In a second aspect, the present invention provides a test strategy automatic generation system based on historical data statistics and analysis, including:

[0030] A selection module: used to retrieve test case information from the database for the tester to select;

[0031] A solution saving module: used to save as a test solution based on all the required test cases selected by the tester;

[0032] A test module: used to perform regression testing based on the policy value of each test case in the test solution and the average value of all the policy values;

[0033] When the policy value of any test case is not less than the average value of all the policy values, the test module puts the test case into the test execution output policy set of this round of testing;

[0034] When the policy value of any test case is less than the average value of all the policy values, the test module updates its influence value by multiplying it by a progressive coefficient, and compares it with the average value of all the policy values again before calculating the policy value in the next round of output;

[0035] Storage module: used for storing the test result data accumulated in the test into the database in response to the regression test being completely completed.

[0036] In a third aspect, the present invention provides a device for automatically generating a test strategy based on historical data statistics and analysis, comprising a processor and a storage medium;

[0037] The storage medium is used to store instructions;

[0038] The processor is used to operate according to the instructions to execute the steps of any of the methods described above.

[0039] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the above methods when executed by a processor.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The present invention provides a scientific and quantifiable test strategy output method for situations where a large number of test cases need to be repeatedly tested in the automatic test of relay protection devices, such as regression testing of product development. The black box test is used for multiple function points of the whole device. For a single function point, the test case can be selected according to the characteristics of the black box test. However, for a large number of function points, the combination selection of the overall test case is based on the experience of the tester. However, the experience of different testers is different, which is more subjective, and a large number of repeated tests must be selected by manpower, which is inefficient. The present invention is based on the statistics and analysis of the test process result data after the test execution by the automatic test platform, which is used as the output basis of the test case execution strategy. The execution strategy can be automatically calculated by the system, saving a lot of manpower and being more objective. In addition, the calculation and execution of the strategy are dynamic, and each round of testing will be adjusted according to the execution of the previous round of testing, so as to be more in line with the actual situation. After all tests are completed, the system will save the test case influence coefficient after multiple rounds of testing, help the tester adjust the test plan, and prevent the solidification of the classic test plan. The present invention converts the test experience into data information for storage, which is conducive to the development of automatic testing technology and improves the intelligence of testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a system module distribution diagram provided in Embodiment 1 of the present invention;

[0043] Figure 2 is a flow chart of generating an initial solution provided in the first embodiment of the present invention;

[0044] Figure 3 This is a flow chart of the test case strategy output provided by the first embodiment of the present invention. Specific implementation manner

[0045] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0046] Embodiment 1:

[0047] This embodiment discloses a method for automatically generating a test strategy based on historical data statistics and analysis, including the following steps:

[0048] Step 1) Divide the statistics of the test plan and test result data into two modules to complete. The data of the two modules is linked to the test case database. When testers select a test plan, they need to consider two aspects. On the one hand, the test plan pursues high coverage and tries to cover all kinds of test cases as much as possible; on the other hand, the test plan also needs to pursue high efficiency and reduce the test time as much as possible. High coverage leads to an increase in test time; high efficiency necessarily requires reducing the number of test cases. These two are in conflict. How to design an optimal strategy plan based on past test results and seek a dynamic balance between the two to achieve the best test plan. As Figure 1 shown in the module distribution diagram, the test plan automatic analysis module is responsible for continuously correcting the test case strategy output set during the regression test process and selecting the optimal solution. The input of this module requires testers to select a classic test plan before the test and check different test items with multiple functions. This module outputs the execution strategy of the next round of test case combination; the test result statistical analysis module is responsible for collecting data such as the execution duration, repeated test rate, unqualified test items, and whether there is corresponding defect tracking of test cases after multiple rounds of regression tests. These data will be used as a reference for the strategy calculation value of the next regression test. In addition, whether there are new test cases in this round also affects the adjustment of the classic test plan. Prevent the long-term solidification of the output test plan.

[0049] Step 2) After receiving the test requirements, the tester retrieves the test case data from the database, and the application program will display the test case information through a tree structure. The classification of the tree structure includes regions, primary functions, secondary functions, test inputs, test outputs, test logics, reverse tests, etc. These classifications are all mapped to specific products and their test items. The application program provides a classic test plan template, and each template will default to check the test cases of the corresponding test items. The tester browses the test items. If the key test items are not included, the tester can select and add checks in the test case library. If the key test items are not included and there are no used test cases in the test case library, the tester can add new test cases and pass the data management process of the test cases. Review whether the added test cases are only used temporarily or stored in the database. After selecting all the required test items, the system will save the initial test plan and execute the first round of tests. AsFigure 2 As shown in the process, the tester receives the test requirements. According to the functional points to be tested in the product, in addition to selecting the classic solution, it is also necessary to consider whether additional new test cases need to be added. Then, such test cases also belong to the key items and need to go through the management review process to be determined and stored in the database as test experience wealth.

[0050] Before the start of the first round in step 3), according to the importance of the tested project, an initial impact value is assigned to each test case. The initial impact value of each test case is Si, denoted as S1, S2…Sn; the calculation of the S value can refer to the following calculation:

[0051] The automation degree setting coefficient of the test case is SS1. The higher the automation degree, the higher the priority, and the weight is sp1;

[0052] The execution duration setting coefficient of the test case is SS2. The shorter the execution time, the higher the priority, and the weight is sp2;

[0053] The higher the probability of repeated execution of the test case, the corresponding reverse test case setting coefficient is SS3, and its priority is high, and the weight is sp3;

[0054] The test case coefficient SS4 related to the defect tracking in the system historical data has a high priority, and the weight is sp4;

[0055] The test case coefficient SS5 corresponding to the problem modification completed in unit testing has a high priority, and the weight is sp5;

[0056] During the initial scheme selection of the tester, the coefficient of the newly added mandatory test case is SS6, and the priority is relatively high, and the weight is sp6.

[0057] The coefficient ss satisfies 1≤ss≤5, and the value

[0058] In step 4), the total number of all test cases of the product is known. Assume that the number of test cases selected before testing is n;

[0059] In step 5), the execution time of each test case is known. The execution time of each test case is Ti, i≤n, denoted as T1, T2…Tn;

[0060] In step 6), construct a formula to calculate the balance ratio coefficient

[0061] In step 7), construct a formula to calculate the strategy value P i =(S i ×k1) / T i , the average value of all strategy values The system calculates the P i value of each test case and the average value Pavg Compare. If P i ≥P avg , then this test case will be selected and put into the test execution output strategy set for this round of testing;

[0062] Step 8) The test strategy plan is a dynamic process. For test cases not selected in the previous round of testing, before calculating the strategy value for the next round of output, their influence value S i will change to S i _>S i ×k2, where k2 is the progression coefficient. If it is still not selected in this round, S i will continue to be transformed in the next round;

[0063] Step 9) If new problems are found during the regression testing process, when conducting the next round of testing, first update the initial S i value through Step 3), and then perform S i -> initial S i × the m-th power of k2, where m represents which round of testing this is. This can ensure the pertinence of the test content, and the test cases related to the problems to be verified will surely be selected in this round;

[0064] As Figure 3 shown, this is the process of the test strategy execution for this round in the multi-round regression testing. After the previous round of testing, some test cases have never been selected according to the priority selection. To ensure the test coverage, the influence value of these test cases will be intervened as S i -> S i ×k2. If it has not been selected in m rounds, then this is the m-th intervention.

[0065] Some tests need to be focused on. For example, if there are unqualified items in the previous round of testing and it is found to be a device bug after investigation and feedback for modification, then this round should focus on it. To ensure that this item will surely be selected in this round, the influence value needs to be updated again according to Step 3) of the invention method, and on this basis, multiply by the progression coefficient k2 m times.

[0066] If in this round of testing, the tester has new key focus items and new test cases are added, then to ensure that the newly added test cases will surely be selected, similarly, the influence value is updated again according to Step 3) of the invention method, and on this basis, multiply by the progression coefficient k2 m times.

[0067] After the execution of this round of testing ends, it will loop again from Step 5) and enter the analysis of the test strategy for the next round. The entire regression testing process is under dynamic management.

[0068] Step 10) After the regression testing is completely finished, the test result statistical analysis module will complete the test result data accumulated in this test and store it in the database.

[0069] The present invention combines the subjective experience judgment of testers and the quantifiable data analysis of the historical statistics of the test system through a scientific and effective method in the scenario where regression testing requires a large number of repetitive tests, and maximizes the intelligent output of the test case strategy set. The test result data of each round will be saved in the database for management and used for the data analysis of the new round of testing, forming a good data closed-loop analysis application.

[0070] Embodiment 2:

[0071] The test strategy automatic generation system based on historical data statistics and analysis can implement the test strategy automatic generation method based on historical data statistics and analysis described in Embodiment 1, including:

[0072] Selection module: used to retrieve test case information from the database for testers to select;

[0073] Scheme saving module: used to save as a test scheme based on all the required test cases selected by the tester;

[0074] Testing module: used to perform regression testing based on the strategy value of each test case in the test scheme and the average value of all strategy values;

[0075] When the strategy value of any test case is not less than the average value of all strategy values, the testing module puts this test case into the test execution output strategy set of this round of testing;

[0076] When the strategy value of any test case is less than the average value of all strategy values, the testing module updates its influence value by multiplying it with the progression coefficient and compares it with the average value of all strategy values again before calculating the strategy value in the next round of output;

[0077] Storage module: used to store the test result data accumulated in the test in the database in response to the complete end of the regression testing.

[0078] Embodiment 3:

[0079] The embodiment of the present invention also provides a test strategy automatic generation device based on historical data statistics and analysis, which can implement the test strategy automatic generation method based on historical data statistics and analysis described in Embodiment 1, including a processor and a storage medium;

[0080] The storage medium is used to store instructions;

[0081] The processor is used to operate according to the instructions to execute the steps of the following method:

[0082] Retrieve test case information from the database for testers to select;

[0083] After the tester selects all the required test cases, save them as a test plan;

[0084] Based on the policy value of each test case in the test plan and the average value of all policy values, conduct regression testing;

[0085] In response to the policy value of any test case being not less than the average value of all policy values, put this test case into the test execution output policy set for this round of testing;

[0086] In response to the policy value of any test case being less than the average value of all policy values, multiply its influence value by a progression coefficient for update, and compare it with the average value of all policy values again before calculating the output policy value in the next round;

[0087] In response to the complete end of the regression testing, store the test result data accumulated during the testing into the database.

[0088] Embodiment 4:

[0089] The embodiment of the present invention also provides a computer-readable storage medium, which can implement the test strategy automatic generation method based on historical data statistics and analysis described in Embodiment 1. There is a computer program stored thereon, and when this program is executed by a processor, it implements the steps of the following method:

[0090] Retrieve test case information from the database for testers to select;

[0091] After the tester selects all the required test cases, save them as a test plan;

[0092] Based on the policy value of each test case in the test plan and the average value of all policy values, conduct regression testing;

[0093] In response to the policy value of any test case being not less than the average value of all policy values, put this test case into the test execution output policy set for this round of testing;

[0094] In response to the policy value of any test case being less than the average value of all policy values, multiply its influence value by a progression coefficient for update, and compare it with the average value of all policy values again before calculating the output policy value in the next round;

[0095] In response to the complete end of the regression testing, store the test result data accumulated during the testing into the database.

[0096] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0097] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in Figure 1 one or more of the processes Figure 1 or a plurality of processes and / or blocks

[0098] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device realizes the functions specified in Figure 1 one or more of the processes Figure 1 or a plurality of processes and / or blocks

[0099] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for realizing the functions specified in Figure 1 one or more of the processes Figure 1 or a plurality of processes and / or blocks

[0100] The above is only the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the technical principle of the present invention, several improvements and deformations can be made, and these improvements and deformations should also be regarded as the protection scope of the present invention.

Claims

1. An automatic generation method for test strategies based on historical data statistics and analysis, characterized in that It includes: Retrieve test case information from the database for testers to select; Based on all the required test cases selected by the testers, save them as a test plan; Based on the strategy value of each test case in the test plan and the average value of all strategy values, conduct regression testing; In response to before the start of the first round of testing, assign an initial impact value to each test case according to the coefficients and weights of the test items; In response to when a new problem is found during the testing process, update its impact value by multiplying it by the progressive coefficient m times, where m is the test round; In response to after the execution of this round of testing ends, enter the analysis of the next round of testing strategy; The calculation formula for the initial impact value is: ， Among them, Si is the influence value of each test case, is the coefficient given by the test item, is the weight given by the test item; In response to when the strategy value of any test case is not less than the average value of all strategy values, put this test case into the test execution output strategy set of this round of testing; In response to when the strategy value of any test case is less than the average value of all strategy values, update its impact value by multiplying it by the progressive coefficient, and compare it with the average value of all strategy values again before calculating the strategy value in the next round of output; In response to after the regression testing is completely over, store the test result data accumulated during the testing in the database; The strategy value of each test case is: ， ， Among them, is the strategy value for each test case, and Si is the influence value for each test case. is the execution time for each test case. is the balance ratio coefficient, and n is the number of test cases selected before testing. The average value of all strategy values is: ， Among them, is the average value of all policy values.

2. The method for automatically generating a test strategy based on historical data statistics and analysis according to claim 1, characterized in that, Retrieve test case information from the database for testers to select, including: display test case information through a tree structure, and the hierarchical levels of the tree structure include region, first-level function, second-level function, test input, test output, test logic, and reverse test.

3. The method for automatically generating a test strategy based on historical data statistics and analysis according to claim 1, wherein The coefficients given by the test items include: test case automation degree setting coefficient, test case execution duration setting coefficient, reverse test case setting coefficient corresponding to the higher the probability of repeated execution of the test case, test case coefficient related to defect tracking in system historical data, test case coefficient corresponding to the completion of problem modification in unit testing, and test case coefficient of newly added mandatory tests during the initial plan selection by testers.

4. The method for automatically generating a test strategy based on historical data statistics and analysis according to claim 1, characterized in that, The weights given by the test items include: test case automation degree setting coefficient weight, test case execution duration setting coefficient weight, reverse test case setting coefficient weight corresponding to the higher the probability of repeated execution of the test case, test case coefficient weight related to defect tracking in system historical data, test case coefficient weight corresponding to the completion of problem modification in unit testing, and test case coefficient weight of newly added mandatory tests during the initial plan selection by testers.

5. A test strategy automatic generation system based on historical data statistics and analysis, characterized in that, It includes: Selection module: used to retrieve test case information from the database for testers to select; Plan saving module: used to save as a test plan based on all the required test cases selected by the testers; Testing module: used to conduct regression testing based on the strategy value of each test case in the test plan and the average value of all strategy values; In response to before the start of the first round of testing, assign an initial impact value to each test case according to the coefficients and weights of the test items; In response to when a new problem is found during the testing process, update its impact value by multiplying it by the progressive coefficient m times, where m is the test round; In response to after the execution of this round of testing ends, enter the analysis of the next round of testing strategy; The policy value of each test case is as follows: ， ， Among them, is the strategy value for each test case, and Si is the influence value for each test case. is the execution time for each test case. is the balance ratio coefficient, and n is the number of test cases selected before the test. The average value of all the policy values is as follows: ， Among them, is the average value of all strategy values; The calculation formula for the initial impact value is: ， Among them, Si is the influence value of each test case, is the coefficient given by the test item, is the weight given by the test item; When the policy value of any test case is not less than the average value of all the policy values, the test module places the test case into the test execution output policy set for this round of testing; When the policy value of any test case is less than the average value of all the policy values, the test module updates its impact value by multiplying it by the progression coefficient and compares it again with the average value of all the policy values before calculating the policy value for the next round of output; Storage module: used to store the test result data accumulated during the test into the database in response to the complete end of the regression test.

6. An automatic test strategy generation device based on historical data statistics and analysis, characterized in that, Comprising a processor and a storage medium; The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 4.

Images