Methods and related equipment for evaluating the application level of automated testing

Abstract

The present application provides a method and related equipment for evaluating the application level of automated testing. The method includes: obtaining multiple test processes of automated testing applications; performing script mapping on each test process, and determining the verification automation rate of each test process based on the script mapping results; calculating the maximum automation rate of all test processes based on the verification automation rate of each test process; constructing a verification test statistics table based on the verification automation rate, maximum automation rate and test data of the verification test of each test process; performing actual testing on each test process, and calculating the actual automation rate of each test process in the actual test within a predetermined time period, and constructing an actual test statistics table based on the actual automation rate of each test process and the test data of the actual test; determining the evaluation results based on the verification test statistics table and the actual test statistics table. It can accurately and truly reflect the application level of automated testing.

Description

Technical Field

[0001] The present application relates to the field of data testing technology, and in particular to a method for evaluating the application level of automated testing and related equipment. Background Art

[0002] Software testing is the process of carefully designing a group of test cases (i.e., input data and its expected output results) based on the specifications of each stage of software development and the internal structure of the program, and using these test cases to run the program to discover program errors.

[0003] Automated testing, on the other hand, is the process of partially or completely replacing manual testing with test tools and test specifications, thereby improving testing efficiency. The primary advantage of automated testing over manual testing lies in the introduction of automated testing tools. Automated testing is generally defined as the management and implementation of various testing activities, including the development and execution of test scripts to verify test requirements using automated testing tools. Automating testing activities can achieve the greatest practical value in many situations, particularly during the test case development and assembly phases of automated testing, where test scripts are repeatedly called and reusable scripts may be run numerous times. Therefore, the benefits of automated testing can be substantial.

[0004] Currently, the effectiveness of automated testing is typically evaluated by counting the percentage of test cases that are automated, often at the system level. There's no clear distinction between the automation rate of a test case and the automation rate during actual execution. Summary of the Invention

[0005] In view of this, the purpose of this application is to propose a method for evaluating the application level of automated testing and related equipment to solve or partially solve the above technical problems.

[0006] Based on the above objectives, the first aspect of the present application provides a method for evaluating the application level of automated testing, comprising:

[0007] Get multiple test processes for automated testing applications;

[0008] Script each test process and determine the verification automation rate of each test process based on the script mapping results;

[0009] Calculate the maximum automation rate of the entire test process based on the verification automation rate of each test process;

[0010] Construct a verification test statistics table based on the verification automation rate, maximum automation rate and test data of the verification test of each test process;

[0011] Performing actual tests on each test process, calculating actual automation rates of each test process in the actual tests within a predetermined time period, and constructing an actual test statistics table based on the actual automation rates of each test process and the test data of the actual tests;

[0012] An evaluation result is determined based on the verification test statistics table and the actual test statistics table.

[0013] In a second aspect of the present application, a device for evaluating the application level of automated testing is provided, comprising:

[0014] A process acquisition module is configured to acquire multiple test processes of an automated testing application;

[0015] The verification test module is configured to perform script mapping on each test process and determine the verification automation rate of each test process based on the script mapping results;

[0016] A maximum automation rate calculation module is configured to calculate the maximum automation rate of the entire test process based on the verification automation rates of the respective test processes;

[0017] A verification test statistics module is configured to construct a verification test statistics table based on the verification automation rate, the maximum automation rate and the test data of the verification test of each test process;

[0018] an actual test statistics module configured to perform actual tests on each test process, calculate the actual automation rate of each test process in the actual test within a predetermined time period, and construct an actual test statistics table based on the actual automation rate of each test process and the test data of the actual test;

[0019] The evaluation result determination module is configured to determine the evaluation result according to the verification test statistics table and the actual test statistics table.

[0020] According to a third aspect of the present application, an electronic device is proposed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0021] The fourth aspect provided by the present application proposes a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the method described in the first aspect.

[0022] As can be seen from the above, the method and related equipment provided by this application for evaluating the application level of automated testing analyze the verification automation replacement rate of each test process during the verification test process to obtain the automation rate of the entire process; the actual automation rate of each test process actually executed automatically within a certain period of time is calculated to obtain the application level based on the automation rate. This method can eliminate functions that are currently not capable of automation, such as facial recognition and graphic verification codes, and can accurately and truthfully reflect the application level of automated testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in this application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are merely embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1A A flowchart of a method for evaluating the application level of automated testing according to an embodiment of the present application;

[0025] Figure 1B A table diagram showing the corresponding script relationships of the automated scripts of the smart insurance process according to an embodiment of the present application;

[0026] Figure 1C Verification automation replacement rate of each automation script of the smart insurance process in the embodiment of this application;

[0027] Figure 1D A table showing the verification automation rates of the nine test processes of the embodiment of the present application;

[0028] Figure 1E A table diagram of the verification test statistics table of the embodiment of the present application;

[0029] Figure 1F A table diagram of actual test statistics of an embodiment of the present application;

[0030] Figure 2 This is a structural block diagram of a device for evaluating the application level of automated testing according to an embodiment of the present application;

[0031] Figure 3 This is a schematic structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the objectives, technical solutions and advantages of this application more clear, this application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0033] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should have the usual meanings understood by people with ordinary skills in the field to which this application belongs. The "first", "second" and similar words used in the embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0034] In related technologies, automated testing mainly includes UI (User Interface) automated testing and interface automated testing. UI automated testing is based on the UI interface, which is closer to real user operations and has a higher cost than interface testing. The main reason is not the high difficulty of technical implementation, but because the UI is the terminal interface that connects to the user. It is the part that is adjusted most frequently and changed most drastically, so the maintenance cost is high. Interface automated testing is a gray box testing method that mainly tests the completeness of internal interface functions. Compared with UI automated testing, it has the characteristics of low automation cost and high testing efficiency. The working principle of interface testing is that the interface testing tool simulates the client sending a message request to the server, and the server accepts the request and responds. Then the response information is returned to the client, and the interface testing tool parses the response information.

[0035] The effectiveness of automated testing refers to the percentage of automated execution in actual testing, typically expressed as the percentage of automated test cases executed. The higher the percentage, the greater the automation replacement rate, the greater the manpower savings, and the better the effectiveness of automated testing.

[0036] Scenario testing is a software testing activity that uses scenarios. These tests generally differ from test cases because test cases consist of a single step, while scenarios cover multiple steps. In scenario testing, the correct result is achieved based on the correct operational flow, which is called the main flow. When a failure flow is encountered within the operational flow, this flow is called an alternative flow.

[0037] Verification testing, the final step before launch, typically uses the main process as the test scenario. Because it's scenario-based, testing often spans multiple systems. Using scenarios to describe system functionality or business processes can improve testing effectiveness.

[0038] The shortcomings of related technologies are that they cannot accurately reflect the actual implementation effects of automation and the replacement of manual labor, which is mainly reflected in three aspects:

[0039] 1) Due to objective constraints, some test cases could not be automated, such as biometric recognition like dynamic facial recognition and fingerprint recognition. These operations are designed to prevent automated attacks and enhance security. Related technologies do not fully consider the impact of these factors on the effectiveness of automated applications.

[0040] 2) It cannot reflect the application effect, but instead mixes the automation rate of the test execution phase with the automatability rate of the development phase. The application effect is a dynamic indicator, while the automatability rate is a relatively static indicator.

[0041] 3) It cannot comprehensively reflect the automation application of multiple test processes. Verification testing includes the verification of multiple processes, and each process has multiple test scenarios. Expressing the automation ratio based solely on test cases cannot fully reflect the automation application effect of the entire verification test.

[0042] Based on the above description, this application proposes a method for evaluating the application level of automated testing, such as Figure 1A As shown, the method includes:

[0043] Step 101: Acquire multiple test processes of an automated test application.

[0044] During implementation, multiple test processes are determined, specifically nine test processes, including:

[0045] Smart Insurance 2.0, self-service life insurance app, China Life Healthcare, Bank-Insurance Link, e-store, China Life e-learning, Easy Learning Hall, group sales and group customers.

[0046] Step 102 : Script mapping is performed on each test process, and the verification automation rate of each test process is determined based on the script mapping result.

[0047] In some embodiments, step 102 includes:

[0048] Step 1021 , retrieve the verification test sample set corresponding to each test process, and determine the corresponding script relationship between the verification test sample set corresponding to each test process and the automation script.

[0049] First, determine the automation type of each test process. The nine verification test processes include UI automation testing (APP (application) side and WEB (World Wide Web) side) and interface automation testing, as shown in Table 1.

[0050] Table 1

[0051] Testing Process Mobile UI Automation Web UI Automation Interface Automation Smart Insurance 2.0 √ Life insurance app self-service preservation √ China Life Healthcare √ Bank-Insurance Link √ e-store √ China Life e-Learning √ Yixuetang √ Group Sales √ Group tourists √

[0052] Then, sort out the corresponding script relationships between the test scenarios (i.e., verification test sample sets) of each test process and the automation scripts. First, number the verification test samples, then correspond them to the automation scripts, and list the verification test samples that have not been automated separately below. Figure 1B The following table shows the correspondence between the verification test samples and automation scripts of the intelligent insurance process.

[0053] Step 1022 : Calculate the verification automation replacement rate of each automation script based on each corresponding script relationship.

[0054] During specific implementation, each test process corresponds to multiple automation scripts, and each automation script is assigned a verification automation replacement rate.

[0055] In some embodiments, step 1022 includes:

[0056] Step 10221 : Determine the automation coverage data of the test case corresponding to each automation script according to the corresponding script relationship of the verification test sample set of each test process, wherein the automation coverage data includes: automation coverage or automation non-coverage.

[0057] Step 10222: Calculate the verification automation replacement rate p1 of each automation script based on the automation coverage data, where p1 = n1 / N1, where n1 is the number of cases covered by automation in the verification test sample set of each test process, and N1 is the total number of cases in the verification test sample set of each test process.

[0058] During specific implementation, after the correspondence between the verification test samples and the automation scripts is established, the automation coverage of the test cases contained in the verification test samples (test scenarios) is analyzed one by one, and the cases that are covered and not covered by automation are recorded. The uncovered cases are the parts that must be tested manually. The ratio of the number of covered cases to the total number of cases is the verification automation replacement rate.

[0059] like Figure 1C As shown, sample 1 (a sample of an automation script for smart insurance) corresponds to the automation script testAllOneCase1. Sample 1 contains 199 test cases, of which 12 test cases cannot be executed automatically due to reasons such as life insurance app follow-up (face recognition) and customer information page ID card recognition. The remaining 187 test cases can be executed automatically, so the verification automation replacement rate of the automation script is 187 / 199=94.0%.

[0060] Each test process obtains the verification automation replacement rate of its corresponding multiple automation scripts in the above manner. The specific process is the same as above and will not be repeated here.

[0061] Step 1023: weight the verification automation replacement rate to obtain the verification automation rate of each test process.

[0062] In specific implementation, the number of test cases contained in each verification test sample is used as the weight to calculate the weighted average of the verification automation replacement rate of each automation script in the test process, and the verification automation rate of the test process (that is, the automation rate of the process in the figure) can be obtained. Figure 1D As shown, column G represents the automation coverage of the process in the verification test sample dimension. However, considering that many test samples still require certain manual operations after the execution of the automation script, the actual automation rate (column H) will be less than the value of column G.

[0063] Step 103 : Calculate the maximum automatable rate of the entire test process based on the verification automation replacement rate and the verification automatable rate of each test process.

[0064] In some embodiments, step 103 specifically includes: taking the number of verification test cases included in the verification test sample set of each test process as a weight, calculating a weighted average of the verification automation replacement rate to obtain the maximum automatable rate of the entire test process.

[0065] During implementation, the nine verification test processes vary greatly, with multiple test types including apps, web, and interfaces. The number of test scenarios (i.e., verification test samples) cannot reflect the actual test workload of each process. For example, although smart insurance only has 6 verification test samples, it contains 269 valid test cases, and the test duration is relatively long. For different test processes, the workload of a single test case is similar. Therefore, using the number of test cases contained in each test process as a weight to obtain the verification automation rate of the entire verification test is relatively objective and accurate. After calculation, the maximum automation rate of the nine processes is 83.1%.

[0066] Through the above scheme, the maximum automation rate of each process and the entire verification test is obtained, which is also the maximum automation rate that can be achieved during the testing process.

[0067] Step 104 : constructing a verification test statistics table based on the verification automation rate, the maximum automation rate, and the test data of the verification test of each test flow.

[0068] In some embodiments, the test data of the verification test includes at least one of the following:

[0069] Verify the number of test samples, the number of automated scripts, the proportion of automated samples, and the total number and weight of verification test cases in the test process.

[0070] like Figure 1E As shown, a verification test statistics table is constructed that can display the automation application status of the entire verification test process. It can be displayed in a tabular form, which can be more intuitive. In addition, each rate value segment can be displayed by color, so that the user can intuitively determine the specific value range of various automation rates of the test process by color.

[0071] Step 105 , perform actual testing on each test process, count the actual automation rate of each test process in the actual testing within a predetermined time period, and construct an actual test statistics table based on the actual automation rate of each test process and the test data of the actual testing.

[0072] During actual implementation, it's often difficult to achieve maximum automation when testing each test process, often due to factors like environment outages, script bugs, and page changes in new versions. To accurately reflect the automation application in actual testing, we record the results of each automated execution when verification testing is required for a release.

[0073] In some embodiments, step 105 includes:

[0074] Step 1051 , retrieve the actual test samples corresponding to each test process, and determine the actual corresponding script relationship between the actual test samples corresponding to each test process and the automation script.

[0075] Step 1052 : Calculate the actual automation rate of each automation script based on each actual corresponding script relationship.

[0076] Step 1053: Perform weighted processing on the actual automatable rate to obtain the actual automation replacement rate of each test process.

[0077] Step 1054 , constructing an actual test statistics table based on the actual automation rate of each test process within the predetermined time period and the test data of the actual test.

[0078] In specific implementation, when verification testing is required for a version release, the results of each automated execution are recorded. The process for calculating the actual automation rate is similar to the calculation of the verification automation rate described above. I will not elaborate on this here. This allows the actual test results of each version release to be recorded.

[0079] The actual automation rate of each test process and the test data of the actual test recorded in the predetermined time period are used to build an actual test statistics table, as shown in the following example: Figure 1F shown.

[0080] Furthermore, by comparing the automation replacement rate (including the verified automation replacement rate or the actual automation replacement rate) with the automatable rate (including the verified automatable rate or the actual automatable rate), we can determine the effectiveness of automation without the impact of factors that cannot be automated (such as biometric recognition). Differentiating the automation rates during the script development phase from the script execution phase can help increase the automation ratio during test execution and save more manpower.

[0081] In some embodiments, the actual test data includes at least one of the following:

[0082] Test version, test start time, test report submission time, actual number of test samples, number of test cases, whether automation is used, and the number of samples using automated data in the test report.

[0083] The figure below is an example of a test statistics table that was constructed. The figure shows the automation application status of each test process.

[0084] Step 106: Determine an evaluation result based on the verification test statistics table and the actual test statistics table.

[0085] In specific implementation, the automation application corresponding to each test process can be evaluated (for example, non-automatable, partially automated, or fully automated, etc.) based on the automation application represented by the two tables obtained above (i.e., the verification test statistics table and the actual test statistics table).

[0086] Through the scheme of the above embodiment, it is possible to analyze the automation and manual testing conditions of all test processes, calculate the automatable rate of the test process (i.e., the maximum automation replacement rate), and obtain the application effect under the maximum automation replacement rate by recording the automation replacement rate during each test execution. Excluding functions such as face recognition and graphic verification codes that are currently not capable of automated execution, the application level of automated testing can be accurately and truly reflected. The automation rate of the entire verification test is obtained by weighting the number of use cases in the test scenarios of each test process, which can shield the differences between different test systems / processes / automation types and accurately reflect the overall average level of automated testing at the current stage.

[0087] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied in a distributed scenario and performed by multiple devices working together. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the method.

[0088] It should be noted that the above description is limited to some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0089] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a device for evaluating the level of automated testing application.

[0090] refer to Figure 2 , the device for evaluating the application level of automated testing includes:

[0091] A process acquisition module 21 is configured to acquire multiple test processes of an automated test application;

[0092] The verification test module 22 is configured to perform script mapping on each test process and determine the verification automation rate of each test process based on the script mapping results;

[0093] A maximum automation rate calculation module 23 is configured to calculate the maximum automation rate of the entire test process according to the verification automation rates of the respective test processes;

[0094] The verification test statistics module 24 is configured to construct a verification test statistics table based on the verification automation rate, the maximum automation rate and the test data of the verification test of each test process;

[0095] The actual test statistics module 25 is configured to perform actual tests on each test process, calculate the actual automation rate of each test process in the actual test within a predetermined time period, and construct an actual test statistics table based on the actual automation rate of each test process and the test data of the actual test;

[0096] The evaluation result determination module 26 is configured to determine an evaluation result according to the verification test statistics table and the actual test statistics table.

[0097] In some embodiments, the verification test module 22 includes:

[0098] The retrieving unit is configured to retrieve the verification test sample set corresponding to each test process, and determine the corresponding script relationship between the verification test sample set corresponding to each test process and the automation script;

[0099] a calculation unit configured to calculate a verification automation replacement rate of each automation script according to each corresponding script relationship;

[0100] The weighting unit is configured to perform weighted processing on the verification automation replacement rate to obtain the verification automation rate of each test process.

[0101] In some embodiments, the computing unit is specifically configured to:

[0102] According to the corresponding script relationship of the verification test sample set of each test process, the automation coverage data of the test case corresponding to each automation script is determined, wherein the automation coverage data includes: automation coverage or automation non-coverage; according to the automation coverage data, the verification automation replacement rate p1 of each automation script is calculated, p1=n1 / N1, wherein n1 is the number of automation-covered cases in the verification test sample set of each test process, and N1 is the total number of cases in the verification test sample set of each test process.

[0103] In some embodiments, the maximum automation rate calculation module 23 is specifically configured to:

[0104] The number of verification test cases included in the verification test sample set of each test process is used as a weight, and a weighted average of the verification automation replacement rate is calculated to obtain the maximum automation rate of the entire test process.

[0105] In some embodiments, the test data of the verification test includes at least one of the following:

[0106] Verify the number of test samples, the number of automated scripts, the proportion of automated samples, and the total number and weight of verification test cases in the test process.

[0107] In some embodiments, the actual test statistics module 25 includes:

[0108] The actual calling unit is configured to call the actual test samples corresponding to each test process and determine the actual corresponding script relationship between the actual test samples corresponding to each test process and the automation script;

[0109] an actual calculation unit configured to calculate an actual automatable rate of each automation script according to each actual corresponding script relationship;

[0110] an actual weighting unit configured to perform weighted processing on the actual automatable rate to obtain an actual automation replacement rate for each test process;

[0111] The statistical unit is configured to construct an actual test statistical table based on the actual automation rate of each test process and the test data of the actual test within the predetermined time period.

[0112] In some embodiments, the actual test data includes at least one of the following:

[0113] Test version, test start time, test report submission time, actual number of test samples, number of test cases, whether automation is used, and the number of samples using automated data in the test report.

[0114] For the convenience of description, the above devices are described as being divided into various modules according to their functions. Of course, when implementing this application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0115] The apparatus of the above embodiment is used to implement the corresponding method for evaluating the application level of automated testing in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0116] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, the method for evaluating the application level of automated testing as described in any of the above embodiments is implemented.

[0117] Figure 3 10 is a schematic diagram showing a more specific hardware structure of an electronic device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are communicatively connected to each other within the device via the bus 1050.

[0118] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0119] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0120] The input / output interface 1030 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0121] The communication interface 1040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0122] The bus 1050 comprises a path for transmitting information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0123] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0124] The electronic device of the above embodiment is used to implement the corresponding method for evaluating the application level of automated testing in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0125] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method for evaluating the level of automated testing application as described in any of the above embodiments.

[0126] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0127] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the method for evaluating the application level of automated testing as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0128] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. Within the scope of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0129] In addition, for simplicity of description and discussion, and in order not to make the embodiment of the application difficult to understand, the known power supply / ground connection with integrated circuit (IC) chip and other components may or may not be shown in the accompanying drawings provided. In addition, the device can be shown in the form of a block diagram to avoid making the embodiment of the application difficult to understand, and this also takes into account the following fact, that is, the details of the embodiment of these block diagram devices are highly dependent on the platform to be implemented in the embodiment of the application (that is, these details should be fully within the scope of understanding of those skilled in the art). When specific details (for example, circuit) are set forth to describe exemplary embodiments of the application, it will be apparent to those skilled in the art that the embodiment of the application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0130] Although the present invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may utilize the embodiments discussed.

[0131] The embodiments of the present application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of this application.

Claims

1. A method for evaluating the application level of automated testing, characterized in that: include: Get multiple test processes for automated testing applications; Script each test process and determine the verification automation rate of each test process based on the script mapping results; Calculate the maximum automation rate of the entire test process based on the verification automation rate of each test process; Construct a verification test statistics table based on the verification automation rate, maximum automation rate and test data of the verification test of each test process; Performing actual tests on each test process, calculating actual automation rates of each test process in the actual tests within a predetermined time period, and constructing an actual test statistics table based on the actual automation rates of each test process and the test data of the actual tests; Determine an evaluation result based on the verification test statistics table and the actual test statistics table; The script mapping of each test process and determining the verification automation rate of each test process based on the script mapping results include: Retrieve the verification test sample set corresponding to each test process, and determine the corresponding script relationship between the verification test sample set corresponding to each test process and the automation script; Calculate the verification automation replacement rate of each automation script based on each corresponding script relationship; Performing weighted processing on the verification automation replacement rate to obtain the verification automation rate of each test process; The maximum automation rate of the entire test process is calculated based on the verification automation rate of each test process, including: Taking the number of verification test cases included in the verification test sample set of each test process as a weight, calculating the weighted average of the verification automation replacement rate to obtain the maximum automation rate of the entire test process; The actual test is performed on each test process, and the actual automation rate of each test process in the actual test within a predetermined time period is counted, and an actual test statistics table is constructed based on the actual automation rate of each test process and the test data of the actual test, including: Retrieve the actual test samples corresponding to each test process and determine the relationship between the actual test samples corresponding to each test process and the actual corresponding script of the automation script; Calculate the actual automation replacement rate of each automation script based on each actual corresponding script relationship; Performing weighted processing on the actual automatable rate to obtain the actual automatable rate of each test process; The actual automation rate of each test process within the predetermined time period and the test data of the actual test are counted to build an actual test statistics table.

2. The method according to claim 1, characterized in that Calculating the verification automation replacement rate of each automation script according to each corresponding script relationship includes: Determine the automation coverage data of the test case corresponding to each automation script according to the corresponding script relationship of the verification test sample set of each test process, wherein the automation coverage data includes: automation coverage or automation non-coverage; According to the automation coverage data, the verification automation replacement rate p1 of each automation script is calculated. p1=n1 / N1, where n1 is the number of cases covered by automation in the verification test sample set of each test process, and N1 is the total number of cases in the verification test sample set of each test process.

3. The method according to claim 1, characterized in that The test data of the verification test includes at least one of the following: Verify the number of test samples, the number of automated scripts, the proportion of automated samples, and the total number and weight of verification test cases in the test process.

4. The method according to claim 1, wherein The actual test data includes at least one of the following: Test version, test start time, test report submission time, actual number of test samples, number of test cases, whether automation is used, and the number of samples using automated data in the test report.

5. A device for evaluating the application level of automated testing, characterized in that: include: A process acquisition module is configured to acquire multiple test processes of an automated testing application; The verification test module is configured to perform script mapping on each test process and determine the verification automation rate of each test process based on the script mapping results; specifically, the verification test sample set corresponding to each test process is retrieved and the corresponding script relationship between the verification test sample set corresponding to each test process and the automation script is determined; Calculate the verification automation replacement rate of each automation script based on each corresponding script relationship; Performing weighted processing on the verification automation replacement rate to obtain the verification automation rate of each test process; The maximum automation rate calculation module is configured to calculate the maximum automation rate of the entire test process based on the verification automation rate of each test process; specifically, the maximum automation rate of the entire test process is obtained by calculating the weighted average of the verification automation replacement rates using the number of verification test cases included in the verification test sample set of each test process as a weight; A verification test statistics module is configured to construct a verification test statistics table based on the verification automation rate, the maximum automation rate and the test data of the verification test of each test process; an actual test statistics module configured to perform actual tests on each test process, calculate the actual automation rate of each test process in the actual test within a predetermined time period, and construct an actual test statistics table based on the actual automation rate of each test process and the test data of the actual test; Specifically: retrieve the actual test samples corresponding to each test process, and determine the relationship between the actual test samples corresponding to each test process and the actual corresponding script of the automation script; Calculate the actual automation replacement rate of each automation script based on each actual corresponding script relationship; Performing weighted processing on the actual automatable rates to obtain the actual automatable rate of each test process; constructing an actual test statistics table based on the actual automatable rates of each test process within a predetermined time period and the test data of the actual test; The evaluation result determination module is configured to determine the evaluation result according to the verification test statistics table and the actual test statistics table.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 4.

Images