An automated testing method and apparatus
By stopping the error reporting step in automated testing and traversing the scenario scripts to determine the problem type, the problem of low analysis accuracy caused by the reliance on background logs in existing technologies is solved, enabling rapid handling of known product defects and improving the robustness of the test suite.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIVIEW TECH CO LTD
- Filing Date
- 2021-06-25
- Publication Date
- 2026-05-26
AI Technical Summary
Existing automated testing methods rely on backend log analysis, resulting in low analysis accuracy, inability to promptly address known product defects, and excessive dependence on developer code standards and log output accuracy, which can easily lead to script crashes or test failures.
By executing the business test scripts in the automated test suite, the error reporting steps are stopped, the scenario scripts are traversed to determine the execution scenario corresponding to the error code, and the problem type is determined based on the scenario. Methods such as environment recovery, script cleanup, and interface verification are used to handle the problem, thus eliminating the dependence on backend logs.
It improves the accuracy of problem analysis, enhances the robustness of the test suite, and can quickly identify and handle environmental issues, script issues, known product defects, and new product issues, thus avoiding test failures.
Smart Images
Figure CN115525531B_ABST
Abstract
Description
Technical Field
[0001] This article relates to automated testing technology, and more particularly to an automated testing method and apparatus. Background Technology
[0002] Currently, automated testing methods are commonly used in the software testing industry to replace the execution of some test points and the analysis of execution log results.
[0003] However, both simple extraction and filtering of script execution logs and analysis of product backend logs suffer from low accuracy in automated result analysis. This is because extracting script execution logs reveals a broad range of issues. Focusing solely on extracting and analyzing product backend logs also has significant drawbacks. Debug logs change as products evolve, making it difficult to guarantee accuracy when relying solely on backend logs for information extraction. Furthermore, developers may not address known issues promptly. If automated scripts fail to properly handle known product defects, execution may fail or even crash, affecting not only the current script but potentially causing the entire test task to fail.
[0004] Current AI-based automated testing methods primarily improve localization efficiency by extracting background logs from the testing system and analyzing error messages and corresponding defect types from the product logs.
[0005] However, this technology also has obvious drawbacks:
[0006] 1. It is highly dependent on the quality of the code written by the developers and the accuracy of the error message output; if the development output logs are not standardized or no logs are output, it will be difficult to obtain them through relevant technologies.
[0007] 2. During product iteration, developers may add or delete debugging log information. Testers often cannot directly control these log changes, which may lead to unexpected errors and crashes in the scripts.
[0008] 3. There are no effective countermeasures or handling measures for known product defects, and no function to quickly identify known product defects in historical versions. Summary of the Invention
[0009] This application provides an automated testing method and apparatus that can eliminate reliance on background logs, improve the accuracy of problem analysis, and ensure the robustness of the test suite.
[0010] This application provides an automatic testing method, which may include:
[0011] Execute the first business test script in the automated test suite;
[0012] If any step in the first business test script encounters an error, the execution of subsequent steps in the first business test script shall be stopped.
[0013] The scripts corresponding to the execution scenarios are traversed according to a preset order to determine the execution scenario corresponding to the error code.
[0014] When the execution scenario corresponding to the error code is determined to be the first execution scenario, the problem type is determined in the first execution scenario, and corresponding processing is performed based on the determination result.
[0015] In an exemplary embodiment of this application, the problem type may include any of the following: environmental problem, script problem, known product defect problem, and new product problem.
[0016] In an exemplary embodiment of this application, determining the problem type in the first execution scenario may include:
[0017] Clean up the script in the first execution scenario;
[0018] If cleaning up the script in the first execution scenario fails, report a new product issue; if cleaning up the script in the first execution scenario succeeds, verify the interface of the first execution scenario.
[0019] When the verification of the interface of the first execution scenario fails, the next execution scenario adjacent to the first execution scenario in the preset execution scenario sequence is verified to determine whether the next execution scenario is the execution scenario corresponding to the error code; when the verification of the interface of the first execution scenario succeeds, the problem type is determined by comparing with the baseline library.
[0020] In an exemplary embodiment of this application, the step of performing corresponding processing based on the judgment result may include:
[0021] When the problem type is the environmental problem, a preset environment recovery script is executed;
[0022] When the problem type is the script problem, execute the preset environment recovery script and report the script defect;
[0023] When the problem type is a known product defect problem, subsequent operations are performed based on the current resolution status of the known product defect problem;
[0024] When the problem type is the newly added product problem, the execution of the first business test script is terminated.
[0025] In an exemplary embodiment of this application, performing subsequent operations based on the current resolution status of the known product defect problem may include:
[0026] Search historical versions of the known product defect to see if it has been resolved.
[0027] When the issues in the previous versions have been resolved, report the new product issues.
[0028] If the historical version is not resolved, check if there is an evasion script in the preset database;
[0029] When the circumvention script exists, the circumvention script for the known product defect is executed; when the circumvention script does not exist, the execution of the first business test script is terminated.
[0030] In an exemplary embodiment of this application, the method may further include: after executing a preset environment recovery script, or after executing an avoidance script regarding the known product defect, executing a next step of the first business test script.
[0031] In an exemplary embodiment of this application, the method may further include:
[0032] If no error is reported in any step of the first business test script, the next step of the first business test script is executed after the current step is completed; or,
[0033] When it is determined that the execution scenario corresponding to the error code is not the first execution scenario, the next execution scenario adjacent to the first execution scenario in the preset execution scenario order is verified to determine whether the next execution scenario is the execution scenario corresponding to the error code.
[0034] In an exemplary embodiment of this application, the method may further include: obtaining the execution scenario using the following scheme:
[0035] A self-learning framework for developing single-interface test cases is provided, wherein the self-learning framework has input and output parameters for its learning interface and outputs general-purpose self-learning test cases.
[0036] Combine all parameters of a single interface and apply the general self-learning test cases to obtain N positive test cases and M negative test cases. Execute the N positive test cases and M negative test cases respectively. Save the scenarios corresponding to the positive test cases that fail to execute and the negative test cases that execute normally to the baseline library as the execution scenarios of the initial version; M and N are positive integers.
[0037] Establish a table mapping error codes and execution scenarios for a single interface, generate a script to detect whether each execution scenario is valid and a script to eliminate error execution scenarios, and embed the table mapping error codes and execution scenarios, the script to detect whether each execution scenario is valid and the script to eliminate error execution scenarios into a preset automatic testing system.
[0038] In an exemplary embodiment of this application, the method may further include: iteratively executing the following scheme to update the execution scenario:
[0039] When executing the automated test suite, if any execution scenario in the baseline library is hit by an error code, the occurrence count of that execution scenario is incremented by 1.
[0040] After the error code analysis is completed, all execution scenarios corresponding to each error code in the error code and execution scenario correspondence table are rearranged according to the number of occurrences.
[0041] In the current test, when the execution scenario in which the error code is hit is the first version, the first version is added to the baseline library, and an avoidance script for the current problem is developed;
[0042] This application also provides an automatic testing system, which may include a processor and a computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed by the processor, implement any of the automatic testing methods described above.
[0043] Compared with related technologies, the embodiments of this application may include: executing a first business test script in an automated test suite; stopping the execution of subsequent steps of the first business test script when any step in the first business test script reports an error; traversing the scripts corresponding to the execution scenarios according to a preset order to determine the execution scenario corresponding to the error code; when the execution scenario corresponding to the error code is determined to be the first execution scenario, judging the problem type under the first execution scenario, and performing corresponding processing according to the judgment result. This embodiment solves the drawback of existing technologies that are limited to judging the cause of product problems solely through querying error codes in the background logs, completely eliminating the dependence on background logs and allowing verification of the scenario corresponding to the error code through multiple methods; furthermore, by analyzing the errors of single-step execution and traversing and filtering each scenario according to the scenario matching method, the range of defect types is narrowed, and the accuracy of problem analysis is improved; in addition, the embodiment of this application decomposes the execution steps of the business script, and if an error is encountered during single-step execution, the execution of subsequent steps is paused, greatly ensuring the robustness of the test suite.
[0044] Other features and advantages of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the application. Other advantages of this application can be realized and obtained by means of the solutions described in the description and the accompanying drawings. Attached Figure Description
[0045] The accompanying drawings are used to provide an understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.
[0046] Figure 1 This is a flowchart of an automatic testing method according to an embodiment of this application;
[0047] Figure 2 This is an overall flowchart of the testing process for the business test script testcase_A_03 in this application embodiment;
[0048] Figure 3 This is a schematic diagram of the test process for the business test script testcase_A_03 in an embodiment of this application;
[0049] Figure 4 This is a flowchart of the scene acquisition and update method according to an embodiment of this application;
[0050] Figure 5 This is a block diagram of the automatic testing device according to an embodiment of this application. Detailed Implementation
[0051] This application describes several embodiments, but these descriptions are exemplary and not restrictive, and it will be apparent to those skilled in the art that many more embodiments and implementations are possible within the scope of the embodiments described herein. Although many possible combinations of features are shown in the drawings and discussed in the detailed description, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with, or may replace, any feature or element of any other embodiment.
[0052] This application includes and contemplates combinations of features and elements known to those skilled in the art. The embodiments, features, and elements disclosed in this application may also be combined with any conventional features or elements to form a unique inventive scheme as defined by the claims. Any feature or element of any embodiment may also be combined with features or elements from other inventive schemes to form another unique inventive scheme as defined by the claims. Therefore, it should be understood that any feature shown and / or discussed in this application may be implemented individually or in any suitable combination. Therefore, the embodiments are not limited except by the limitations imposed by the appended claims and their equivalents. Furthermore, various modifications and changes may be made within the scope of the appended claims.
[0053] Furthermore, in describing representative embodiments, the specification may have presented methods and / or processes as a specific sequence of steps. However, the method or process should not be limited to the specific order of steps described herein, to the extent that it does not depend on such a specific order. As will be understood by those skilled in the art, other sequences of steps are also possible. Therefore, the specific order of steps set forth in the specification should not be construed as a limitation of the claims. Moreover, the claims concerning the method and / or process should not be limited to the steps performed in the written order, and those skilled in the art will readily understand that these orders can be varied and still remain within the spirit and scope of the embodiments of this application.
[0054] This application provides an automatic testing method, such as... Figure 1 As shown, the method may include steps S01-S04:
[0055] S01. Execute the first business test script in the automated test suite;
[0056] S02. When any step in the first business test script reports an error, stop executing the subsequent steps of the first business test script.
[0057] S03. Traverse the scripts corresponding to the execution scenarios according to a preset order to determine the execution scenario corresponding to the error code;
[0058] S04. When it is determined that the execution scenario corresponding to the error code is the first execution scenario, the problem type is determined in the first execution scenario, and corresponding processing is performed according to the determination result.
[0059] In an exemplary embodiment of this application, the method may further include:
[0060] If no error is reported in any step of the first business test script, the next step of the first business test script will be executed after the current step is completed.
[0061] In an exemplary embodiment of this application, during automated testing, the execution steps of the business script can be broken down. If an error occurs during single-step execution, the execution of subsequent steps is paused, and the root cause of the error in the current step is analyzed using scenario matching. For example, if the error pertains to an environment-related issue (referred to as an environment problem), it is archived to provide a basis for business script optimization; if it pertains to a product-related issue (i.e., a product problem), it is reported, and the execution of subsequent steps in the current business script is terminated, effectively improving the accuracy of the business test script execution result analysis.
[0062] In an exemplary embodiment of this application, the scenario matching method is a method that gradually narrows down the range of error types by iteratively judging, eliminating error codes, and re-executing to screen and confirm the execution scenarios corresponding to each error code. Product problem: A problem found in the tested system or product during the testing process is called a product problem.
[0063] In exemplary embodiments of this application, the solutions of the embodiments of this application are described in detail below.
[0064] In the exemplary embodiments of this application, the testing process of the business test script testcase_A_03 is used as an example for description. The testing process of the business test script testcase_A_03 can be as follows: Figure 2 , Figure 3 As shown, the overall execution steps can be as follows: Figure 2 As shown, steps S10-S16 are included:
[0065] S10. Log in to the TDMS automated testing system and start executing the automated test suite;
[0066] S11. Execute the business test script testcase_A_03 in the automated test suite;
[0067] S12. When an interface called in the business test script reports an error, pause the execution of subsequent steps;
[0068] S13. Retrieve the execution scenario corresponding to the error code, and iterate through the scripts corresponding to each execution scenario (or error scenario, or simply scenario);
[0069] S14. Accurately locate the execution scenario corresponding to the error code, determine the problem type, and take appropriate action based on the determination result;
[0070] S15. Repeat steps S12-S14 until the exit condition is met, then exit the current business test script testcase_A_03.
[0071] S16. Start executing the business test script testcase_A_04 in the automated test suite until all business test scripts in the entire test suite have been executed.
[0072] In the exemplary embodiments of this application, the TDMS automated testing system is a system that intelligently builds an automated testing network environment, implements automated environment setup, and executes test scripts. A business test script is a test script written in code for a non-single-interface scenario designed for a specific function of a product. A test suite is a collection of business test scripts combined according to certain rules and order.
[0073] In an exemplary embodiment of this application, the problem type may include any of the following: environmental problem, script problem, known product defect problem, and new product problem.
[0074] In an exemplary embodiment of this application, determining the problem type in the first execution scenario may include:
[0075] Clean up the script in the first execution scenario;
[0076] If cleaning up the script in the first execution scenario fails, report a new product issue; if cleaning up the script in the first execution scenario succeeds, verify the interface of the first execution scenario.
[0077] When the verification of the interface of the first execution scenario fails, the next execution scenario adjacent to the first execution scenario in the preset execution scenario sequence is verified to determine whether the next execution scenario is the execution scenario corresponding to the error code; when the verification of the interface of the first execution scenario succeeds, the problem type is determined by comparing with the baseline library.
[0078] In an exemplary embodiment of this application, it is assumed that the testcase_A_03 script has 15 steps, and that the first step, step p1, has 10 execution scenarios for an error code (this is just an example; the specific number of execution scenarios and the rules for generating and sorting them will be described in detail later).
[0079] In exemplary embodiments of this application, such as Figure 3 As shown, when the testcase_A_03 script encountered an error in step 1, it paused the execution of step 2 and entered the verification process of the scenario matching method.
[0080] In an exemplary embodiment of this application, the execution scenario corresponding to the big data retrieval error code is first performed; the correspondence between single interface error codes and error scenarios stored in the database is shown in Table 1 below as an example:
[0081] Table 1
[0082]
[0083] In the exemplary embodiments of this application, Table 1 is merely an example of a general architecture for the correspondence between error codes and error scenarios, and the specific content does not affect the protection scope of the embodiments of this application.
[0084] In an exemplary embodiment of this application, the method may further include:
[0085] When the execution scenario corresponding to the error code is determined to be the first execution scenario, the problem type is determined under the first execution scenario, and corresponding processing is performed according to the determination result;
[0086] When it is determined that the execution scenario corresponding to the error code is not the first execution scenario, the next execution scenario adjacent to the first execution scenario in the preset execution scenario order is verified to determine whether the next execution scenario is the execution scenario corresponding to the error code.
[0087] In an exemplary embodiment of this application, it can first be verified whether it belongs to scenario 1. If so, the script for cleaning up scenario 1 is executed. If the execution of the script for cleaning up scenario 1 is successful and no error is reported, the interface for verifying scenario 1 is executed. If the execution of the interface for verifying scenario 1 is successful and no error is reported, the problem type is determined by comparing with the baseline library.
[0088] In an exemplary embodiment of this application, the step of performing corresponding processing based on the judgment result may include:
[0089] When the problem type is the environmental problem, a preset environment recovery script is executed;
[0090] When the problem type is the script problem, execute the preset environment recovery script and report the script defect;
[0091] When the problem type is a known product defect problem, subsequent operations are performed based on the current resolution status of the known product defect problem;
[0092] When the problem type is the newly added product problem, the execution of the first business test script is terminated.
[0093] In an exemplary embodiment of this application, the method may further include: executing the next step of the first business test script after executing a preset environment recovery script.
[0094] In the exemplary embodiments of this application, if the problem is an environment execution issue (environmental problem) and not a product problem, it can be added to the environment execution issue archive for easier script maintenance later. If it is a product problem, it is reported, and the execution of subsequent steps in the current business script is terminated. In this case, the current business script is exited directly, and subsequent steps are not executed, as it is meaningless. If the script for the cleanup scenario reports an error, it is a newly added product defect problem. In this case, the current business script is exited directly, and subsequent steps are not executed, as it is meaningless.
[0095] In an exemplary embodiment of this application, if the interface for performing scenario 1 verification fails, reports an error, or the verification does not belong to scenario 1, then the verification of scenario 2 can proceed.
[0096] In an exemplary embodiment of this application, performing subsequent operations based on the current resolution status of the known product defect problem may include:
[0097] Search historical versions of the known product defect to see if it has been resolved.
[0098] When the issues in the previous versions have been resolved, report the new product issues.
[0099] If the historical version is not resolved, check if there is an evasion script in the preset database;
[0100] When the circumvention script exists, the circumvention script for the known product defect is executed; when the circumvention script does not exist, the execution of the first business test script is terminated.
[0101] In an exemplary embodiment of this application, the method may further include: after executing an evasion script regarding the known product defect, executing a next step of the first business test script.
[0102] In an exemplary embodiment of this application, for a known product defect, if a script for an avoidance measure exists in the current database, the script is executed to eliminate the interference of the known product defect before proceeding to the next step; however, if no script for an avoidance measure exists, the current business script is skipped directly, and the next step is not executed. This is because if the current problem is not resolved, there may be various uncertainties in the environment, and continuing to execute may have a serious impact on the execution result of the entire test suite.
[0103] In an exemplary embodiment of this application, after traversing and verifying 10 scenarios, if none of them are found, it can be preliminarily determined to be a new product problem.
[0104] In an exemplary embodiment of this application, if the error in step 1 is found to be a non-product problem, the interfering data can be cleared and the subsequent step 2 can be performed.
[0105] In an exemplary embodiment of this application, when any step in the testcase_A_03 business script fails, the scenario matching method described above can be executed until all 15 steps involved in the testcase_A_03 business script have been verified (unless a product problem or a new product problem is detected, in which case the testcase_A_03 business script will be skipped directly). Then, the next business script, such as the execution loop of testcase_A_04, will be entered until the entire test suite is completed.
[0106] In exemplary embodiments of this application, such as Figure 4 As shown below, an example of the scene acquisition and update scheme is given.
[0107] In an exemplary embodiment of this application, the method may further include: obtaining the execution scenario using the following scheme:
[0108] A self-learning framework for developing single-interface test cases is provided, wherein the self-learning framework has input and output parameters for its learning interface and outputs general-purpose self-learning test cases.
[0109] Combine all parameters of a single interface and apply the general self-learning test cases to obtain N positive test cases and M negative test cases. Execute the N positive test cases and M negative test cases respectively. Save the scenarios corresponding to the positive test cases that fail to execute and the negative test cases that execute normally to the baseline library as the execution scenarios of the initial version; M and N are positive integers.
[0110] Establish a table mapping error codes and execution scenarios for a single interface, generate a script to detect whether each execution scenario is valid and a script to eliminate error execution scenarios, and embed the table mapping error codes and execution scenarios, the script to detect whether each execution scenario is valid and the script to eliminate error execution scenarios into a preset automatic testing system.
[0111] In an exemplary embodiment of this application, a self-learning framework for developing single-interface test cases is used to learn the interface's input and output parameters, and output general-purpose self-learning test cases (each parameter follows a basic test case design method, which can be any existing feasible method). Based on the actual parameters of the single interface, each parameter is fitted with a general-purpose self-learning test case. All parameters of the single interface are combined to generate N forward test cases and M reverse test cases. These test cases are executed, and the scenarios corresponding to the forward test cases that result in errors and the reverse test cases that execute normally are saved to the database. This generates the initial version of the scenarios.
[0112] In an exemplary embodiment of this application, the method may further include: iteratively executing the following scheme to update the execution scenario:
[0113] When executing the automated test suite, if any execution scenario in the baseline library is hit by an error code, the occurrence count of that execution scenario is incremented by 1.
[0114] After the error code analysis is completed, all execution scenarios corresponding to each error code in the error code and execution scenario correspondence table are rearranged according to the number of occurrences.
[0115] In the current test, when the execution scenario in which the error code is hit is the first version, the first version is added to the baseline library, and an avoidance script for the current problem is developed.
[0116] In the exemplary embodiments of this application, the concept of a version sequence table (which is the version iteration number of the product) needs to be introduced to record the test version iteration order, as shown in Table 2:
[0117] Table 2
[0118] D133 D133SP01 D135 D135SP01 D135SP02 D136 D137 D139 …
[0119] In an exemplary embodiment of this application, during the execution of the test suite, when an error code hits a certain scenario, the occurrence count of that scenario is incremented by 1.
[0120] In an exemplary embodiment of this application, after an error code analysis is completed, all scenarios corresponding to that error code can be sorted in descending order of "number of occurrences"; the error code and scenario correspondence table is shown in Table 3:
[0121] Table 3
[0122]
[0123]
[0124] In an exemplary embodiment of this application, when the hit scenario is earlier than the "first appearance version", the "first appearance version" of the scenario is added to the baseline library.
[0125] In an exemplary embodiment of this application, when the hit scenario belongs to the "removed version" or a version after the "removed version", that is, the scenario is not expected to appear in the current test version, the error code + scenario is classified as a new product issue and clearly marked that it has been resolved in a certain historical version, but has appeared again in this test.
[0126] In an exemplary embodiment of this application, after the test suite is executed, the newly added scene information is updated in the "Error Code and Scene Correspondence Table".
[0127] In exemplary embodiments of this application, the solutions of this application embodiments include at least the following advantages:
[0128] 1. It overcomes the shortcomings of existing technologies that rely solely on querying error codes in the backend logs to determine the cause of product problems, completely eliminating the dependence on backend logs and allowing verification of the scenarios corresponding to error codes through multiple methods;
[0129] 2. By analyzing the errors of single-step execution, we can traverse and filter each scenario according to the scenario matching method, narrowing down the range of defect types and improving the accuracy of problem analysis; improving the efficiency of the code in handling known problems (including product defects, script defects, environmental problems and execution problems), and improving the robustness of the test suite.
[0130] 3. Identify problem types such as known product issues, newly added product issues, environmental issues, and script issues to narrow down the range of problem types and improve the accuracy of automated analysis;
[0131] 4. For known product defects, take appropriate measures to avoid them, thereby preventing these known product defects from having a serious impact on the execution of the entire test suite;
[0132] 5. Deep learning obtains the scenarios corresponding to error codes and prioritizes scenarios based on their frequency of occurrence, providing a technical foundation for improving the accuracy of error judgment.
[0133] This application also provides an automatic testing system 1, such as... Figure 5 As shown, it may include a processor 11 and a computer-readable storage medium 12, wherein the computer-readable storage medium 12 stores instructions that, when executed by the processor 11, implement the automatic testing method described in any of the above.
[0134] In the exemplary embodiments of this application, any of the aforementioned automatic testing methods are applicable to this device embodiment, and will not be described in detail here.
[0135] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.
Claims
1. An automatic testing method, characterized in that, The method includes: Execute the first business test script in the automated test suite; If any step in the first business test script encounters an error, the execution of subsequent steps in the first business test script shall be stopped. The scripts corresponding to the execution scenarios are traversed in a preset order to determine the execution scenario corresponding to the error code. When the execution scenario corresponding to the error code is determined to be the first execution scenario, the problem type is determined under the first execution scenario, and corresponding processing is performed according to the determination result; The step of determining the problem type in the first execution scenario includes: Clean up the script in the first execution scenario; If cleaning up the script in the first execution scenario fails, report a new product issue; if cleaning up the script in the first execution scenario succeeds, verify the interface of the first execution scenario. When the verification of the interface of the first execution scenario fails, the next execution scenario adjacent to the first execution scenario in the preset execution scenario sequence is verified to determine whether the next execution scenario is the execution scenario corresponding to the error code; when the verification of the interface of the first execution scenario succeeds, the problem type is determined by comparing with the baseline library.
2. The automatic testing method according to claim 1, characterized in that, The problem types include any of the following: environmental problems, script problems, known product defect problems, and new product problems.
3. The automatic testing method according to claim 2, characterized in that, The corresponding processing based on the judgment result includes: When the problem type is the environmental problem, a preset environment recovery script is executed; When the problem type is the script problem, execute the preset environment recovery script and report the script defect; When the problem type is a known product defect problem, subsequent operations are performed based on the current resolution status of the known product defect problem; When the problem type is the newly added product problem, the execution of the first business test script is terminated.
4. The automatic testing method according to claim 3, characterized in that, The step of performing subsequent operations based on the current resolution status of the known product defect includes: Search historical versions of the known product defect to see if it has been resolved. When the issues in the previous versions have been resolved, report the new product issues. If the historical version is not resolved, check if there is an evasion script in the preset database; When the avoidance script exists, the avoidance script for the known product defect is executed; when the avoidance script does not exist, the execution of the first business test script is terminated.
5. The automatic testing method according to claim 4, characterized in that, The method further includes: after executing a preset environment recovery script, or after executing an avoidance script for the known product defect, executing the next step of the first business test script.
6. The automatic testing method according to claim 1, characterized in that, The method further includes: If no error is reported in any step of the first business test script, the next step of the first business test script is executed after the current step is completed; or, When it is determined that the execution scenario corresponding to the error code is not the first execution scenario, the next execution scenario adjacent to the first execution scenario in the preset execution scenario order is verified to determine whether the next execution scenario is the execution scenario corresponding to the error code.
7. The automatic testing method according to claim 1, characterized in that, The method further includes obtaining the execution scenario using the following scheme: A self-learning framework for developing single-interface test cases is provided, wherein the self-learning framework has input and output parameters for its learning interface and outputs general-purpose self-learning test cases. Combine all parameters of a single interface and apply the general self-learning test case to obtain N positive test cases and M negative test cases. Execute the N positive test cases and M negative test cases respectively. Save the scenarios corresponding to the positive test cases that fail to execute and the negative test cases that execute normally to the baseline library as the execution scenarios of the initial version; M and N are positive integers. Establish a table mapping error codes and execution scenarios for a single interface, generate a script to detect whether each execution scenario is valid and a script to eliminate error execution scenarios, and embed the table mapping error codes and execution scenarios, the script to detect whether each execution scenario is valid and the script to eliminate error execution scenarios into a preset automatic testing system.
8. The automatic testing method according to claim 7, characterized in that, The method further includes: iteratively executing the following scheme to update the execution scenario: When executing the automated test suite, if any execution scenario in the baseline library is hit by an error code, the occurrence count of that execution scenario is incremented by 1. After the error code analysis is completed, all execution scenarios corresponding to each error code in the error code and execution scenario correspondence table are rearranged according to the number of occurrences. In the current test, when the execution scenario in which the error code is hit is the first version, the first version is added to the baseline library, and an avoidance script for the current problem is developed.
9. An automatic testing system, characterized in that, The device includes a processor and a computer-readable storage medium storing instructions that, when executed by the processor, implement the automatic testing method as described in any one of claims 1-8.