Automatic testing method and device

Abstract

The invention discloses an automatic testing method and device. The method comprises the steps that in a historical time period, a correct version code is adopted to execute a test case, after stable operation, a benchmark result is obtained, and the benchmark result comprises benchmark test results corresponding to all steps of the test case; executing the test case by adopting the test version code to obtain an actual test result of each step of the test case; and comparing the corresponding actual test result with the benchmark test result to obtain a comparison result, and determining whether the test case can stably run in the process of executing the test version code or not according to the comparison result of each step of the test case. The technical problems that in the prior art, the compiling workload of UI automatic testing is large, checking points may be left, testing omission is caused, the false alarm rate is high, universality is not enough, and the UI automatic testing method and device cannot be suitable for various testing scenes are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of software testing, and in particular to an automated testing method and device. Background Art

[0002] UI automation testing simulates human behavior through code or tools to operate the user interface, realizing automated testing of the program's UI layer. This can save manual testing costs, improve testing efficiency, and is widely used in regression testing, compatibility testing, etc.

[0003] The current mainstream UI automated testing frameworks or tools rely on manually written assertions for result judgment, but the workload of assertion writing is large and may leave checkpoints unchecked, resulting in missed tests. There is also a solution in the prior art that uses image comparison to judge results, but this solution has a high false alarm rate and is not versatile enough to be applicable to a variety of test scenarios.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide an automated testing method and device to at least solve the technical problems in the prior art that the writing workload of UI automated testing is large, checkpoints may be left behind, resulting in missed tests, and the false alarm rate is high, the versatility is insufficient, and it cannot be applied to a variety of testing scenarios.

[0006] According to one aspect of an embodiment of the present invention, an automated testing method is provided, comprising: executing a test case with a correct version code in a historical time period and obtaining a benchmark result after the test case has run stably, the benchmark result including a benchmark test result corresponding to each step of the test case; executing the test case with a test version code to obtain an actual test result of each step of the test case; comparing the corresponding actual test result with the benchmark test result to obtain a comparison result, and determining whether the test case can run stably in the process of executing the test version code based on the comparison result of each step of the test case.

[0007] Optionally, a benchmark result is obtained after the test case is executed and stably runs with the correct version code in a historical time period, including: executing the test case with the correct version code in a historical time period, and in the process of executing the test case, obtaining an execution screenshot of each step of the test case, the execution screenshot being a screenshot of the corresponding direct result obtained after the test case stably runs the current step on the target interface; determining whether the execution result in the execution screenshot is correct; if the execution result in the execution screenshot is correct, determining the execution screenshot as the benchmark test result of the current step, and determining the benchmark test results of all the steps as the benchmark result.

[0008] Optionally, the corresponding actual test results are compared with the benchmark test results to obtain a comparison result, including: determining the interface that displays the execution result of the test case as the target interface; using automatic configuration and / or manual configuration, and determining a comparison area and a non-comparison area in the target interface based on the display content of each area in the target interface, the comparison area being an area for comparing the actual test results with the benchmark test results, and the non-comparison area being an area that does not need to be compared; determining a comparison accuracy based on the test scenario of the test case, and comparing the corresponding actual test results with the benchmark test results based on the comparison accuracy and the comparison area to obtain the comparison result.

[0009] Optionally, automatic configuration is adopted and based on the display content of each area in the target interface, the comparison area and the non-comparison area in the target interface are determined, including: using the correct version code to execute the test case a preset number of times to obtain a screenshot collection, the screenshot collection including execution screenshots of each step of the test case at each execution, and the number of execution screenshots corresponding to one step is the same as the preset number of times; comparing a preset number of the execution screenshots of the step, and obtaining an area with a similarity less than a first preset similarity as a random area; determining the random area as the non-comparison area, and determining the areas in all the execution screenshots except the random area as the comparison area.

[0010] Optionally, according to the test scenario of the test case, the comparison accuracy is determined, including: determining the accuracy requirement of the test scenario of the test case; when the accuracy requirement is greater than or equal to a first preset accuracy, determining the comparison accuracy to be an absolute comparison, and the absolute comparison is a method of comparing pixel by pixel; when the accuracy requirement is greater than or equal to a second preset accuracy and less than the first preset accuracy, determining the comparison accuracy to be a strict comparison, and the strict comparison is a method of strictly comparing operating parameters using an SSIM algorithm, and the operating parameters include at least the brightness and contrast of the target interface; when the accuracy requirement is less than the second preset accuracy, determining the comparison accuracy to be a content comparison, and the content comparison is a method of only comparing text.

[0011] Optionally, according to the comparison accuracy and the comparison area, the corresponding actual test results and the benchmark test results are compared to obtain a comparison result, including: obtaining an actual execution screenshot of the current step in the actual test result and a benchmark execution screenshot of the current step in the benchmark test result; according to the comparison area, cropping the actual execution screenshot and the benchmark execution screenshot to retain only the comparison area in the actual execution screenshot and the comparison area in the benchmark execution screenshot, which are the first area and the second area respectively; comparing the first area and the second area according to the comparison accuracy to obtain the comparison result.

[0012] Optionally, based on the comparison results of each of the steps of the test case, determining whether the test case can run stably in the process of executing the test version code includes: performing weighted averaging processing on the comparison results of all the steps of the test case to obtain a total similarity, the comparison result being the similarity between the actual test result of the corresponding step and the benchmark test result; when the total similarity is greater than or equal to a second preset similarity, determining that the test case can run stably in the process of executing the test version code; and when the total similarity is less than the second preset similarity, determining that the test case can run stably in the process of executing the test version code.

[0013] According to another aspect of an embodiment of the present invention, an automated testing device is also provided, including: a first execution module, used to execute the test case using the correct version code in a historical time period and obtain a benchmark result after stable operation, wherein the benchmark result includes the benchmark test result corresponding to each step of the test case; a second execution module, used to execute the test case using the test version code to obtain the actual test result of each step of the test case; a determination module, used to compare the corresponding actual test result with the benchmark test result to obtain a comparison result, and determine whether the test case can run stably in the process of executing the test version code according to the comparison result of each step of the test case.

[0014] Optionally, the first execution module includes: an execution unit, used to execute the test case using the correct version code in a historical time period, and in the process of executing the test case, obtain an execution screenshot of each step of the test case, the execution screenshot being a screenshot of the corresponding direct result obtained after the test case stably runs the current step on the target interface; a first determination unit, used to determine whether the execution result in the execution screenshot is correct; and a second determination unit, used to determine the execution screenshot as the benchmark test result of the current step if the execution result in the execution screenshot is correct, and determine the benchmark test results of all the steps as the benchmark result.

[0015] Optionally, the determination module includes: a third determination unit, used to determine the interface displaying the execution result of the test case as the target interface; a fourth determination unit, used to determine the comparison area and the non-comparison area in the target interface by automatic configuration and / or manual configuration and according to the display content of each area in the target interface, the comparison area is the area for comparing the actual test result with the benchmark test result, and the non-comparison area is the area that does not need to be compared; a comparison unit, used to determine the comparison accuracy according to the test scenario of the test case, and according to the comparison accuracy and the comparison area, compare the corresponding actual test result with the benchmark test result to obtain the comparison result.

[0016] Optionally, the fourth determination unit includes: an execution subunit, used to execute the test case a preset number of times using the correct version code to obtain a screenshot collection, wherein the screenshot collection includes execution screenshots of each step of the test case at each execution, and the number of execution screenshots corresponding to one step is the same as the preset number of times; a first comparison subunit, used to compare a preset number of the execution screenshots of the step, and obtain an area with a similarity less than a first preset similarity as a random area; a first determination subunit, used to determine the random area as the non-comparison area, and determine the area in all the execution screenshots except the random area as the comparison area.

[0017] Optionally, the comparison unit includes: a second determination subunit, used to determine the accuracy requirement of the test scenario of the test case; a third determination subunit, used to determine that the comparison accuracy is an absolute comparison when the accuracy requirement is greater than or equal to a first preset accuracy, and the absolute comparison is a pixel-by-pixel comparison method; a fourth determination subunit, used to determine that the comparison accuracy is a strict comparison when the accuracy requirement is greater than or equal to a second preset accuracy and less than the first preset accuracy, and the strict comparison is a strict comparison method that uses an SSIM algorithm to compare operating parameters, and the operating parameters include at least the brightness and contrast of the target interface; a fifth determination subunit, used to determine that the comparison accuracy is a content comparison when the accuracy requirement is less than the second preset accuracy, and the content comparison is a method of only comparing text.

[0018] Optionally, the comparison unit includes: an acquisition subunit, used to acquire an actual execution screenshot of the current step in the actual test result and a benchmark execution screenshot of the current step in the benchmark test result; a cropping subunit, used to crop the actual execution screenshot and the benchmark execution screenshot according to the comparison area, retaining only the comparison area in the actual execution screenshot and the comparison area in the benchmark execution screenshot, which are the first area and the second area respectively; a second comparison subunit, used to compare the first area and the second area according to the comparison accuracy to obtain the comparison result.

[0019] Optionally, the determination module includes: a weighted average processing unit, used to perform weighted average processing on the comparison results of all the steps of the test case to obtain a total similarity, wherein the comparison result is the similarity between the actual test result of the corresponding step and the benchmark test result; a fifth determination unit, used to determine that the test case can run stably in the process of executing the test version code when the total similarity is greater than or equal to a second preset similarity; and a sixth determination unit, used to determine that the test case can run stably in the process of executing the test version code when the total similarity is less than the second preset similarity.

[0020] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned automated testing methods.

[0021] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of the automated testing methods described above.

[0022] According to another aspect of an embodiment of the present invention, a computer program product is provided, comprising computer instructions, wherein when the computer instructions are executed by a processor, any one of the above-mentioned automated testing methods is executed.

[0023] In an embodiment of the present invention, after executing the test case with the correct version code in the historical time period and running stably, a benchmark result is obtained, and the benchmark result includes the benchmark test result corresponding to each step of the test case; the test version code is used to execute the test case to obtain the actual test result of each step of the test case; the corresponding actual test result and the benchmark test result are compared to obtain the comparison result, and according to the comparison result of each step of the test case, it is determined whether the test case can run stably in the process of executing the test version code. Through the technical solution provided by the present invention, the purpose of analyzing the UI automation test results by using the image comparison method, supporting automatic calculation of the image and manual configuration of the comparison area, and selecting the corresponding comparison accuracy according to the test scenario is achieved, thereby achieving the technical effect of not needing to manually write and maintain assertions and reducing false alarms, improving configuration efficiency and test versatility, and thus solving the technical problems of large workload of writing UI automation tests in the prior art, which may leave checkpoints and lead to missed tests, and high false alarm rate, insufficient versatility, and inability to be applied to a variety of test scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0025] Figure 1 It is a hardware structure block diagram of a mobile terminal of an automated testing method according to an embodiment of the present invention;

[0026] Figure 2 is a flow chart of an automated testing method according to an embodiment of the present invention;

[0027] Figure 3 is a flow chart of another automated testing method according to an embodiment of the present invention;

[0028] Figure 4 is a schematic diagram of the effect of the automated testing method according to an embodiment of the present invention;

[0029] Figure 5 is a flow chart of automatic calculation of contrast regions according to an embodiment of the present invention;

[0030] Figure 6 is a flowchart of picture comparison according to an embodiment of the present invention;

[0031] Figure 7 is a schematic diagram of an automated testing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0035] User Interface: User Interface, referred to as UI, is the medium for interaction and information exchange between the system and the user. It realizes the conversion between the internal form of information and the form acceptable to humans.

[0036] UI automation testing: simulates human behavior through code or tools to operate the user interface, and realizes automated testing of the program UI layer. It is divided into Web UI automation testing and App UI automation testing.

[0037] Stable code version: refers to the code version that has been tested and released and is considered to be correct.

[0038] Test version: refers to a version that is developed iteratively based on a stable code version, with some functions modified and needs to be tested.

[0039] Carousel: refers to a component that displays multiple images or text information in a fixed area in turn.

[0040] As introduced in the background technology, the writing workload of UI automated testing in the prior art is large, and checkpoints may be left behind, resulting in missed tests, and the false alarm rate is high. It is not versatile enough and cannot be applied to a variety of test scenarios. In view of the above defects, an automated testing method and device, an electronic device, a computer-readable storage medium, a processor, and a computer program product are provided in the embodiments of the present invention.

[0041] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0042] The method embodiments provided in the embodiments of the present invention can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure diagram of a mobile terminal of an automated testing method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0043] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the automated testing method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0044] According to an embodiment of the present invention, a method embodiment of an automated testing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] Figure 2is a flow chart of an automated testing method according to an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0046] Step S202, executing the test case with the correct version code in the historical time period and obtaining a benchmark result after stable operation, wherein the benchmark result includes a benchmark test result corresponding to each step of the test case.

[0047] In this embodiment, for the UI automation test case to be executed, a correct execution history is selected as the baseline result. The benchmark result is obtained by using the correct version code to execute the test case in the historical time period and running it stably. It can be used as a reference point for comparison to help verify the correctness and stability of the current version of the code. The benchmark result can be used as the expected result when the current version of the code executes the test case.

[0048] It should be noted that the above-mentioned UI (User Interface) is the medium for interaction and information exchange between the system and the user, and can realize the conversion between the internal form of information and the form acceptable to humans; the above-mentioned UI automation testing is to operate the user interface through code or tools to simulate human behavior, and realize automated testing of the program UI layer, which is divided into Web UI automation testing and App UI automation testing.

[0049] Step S204, executing the test case using the test version code to obtain actual test results of each step of the test case.

[0050] In this embodiment, the test version code is used to execute the test case, and the execution result of each step is automatically captured during the execution process to obtain the test execution result, which can be used to verify whether the functions of the tested software are running normally in different versions or environments.

[0051] It should be noted that the above-mentioned test version refers to a version that has been iteratively developed based on a stable code version and has some functions modified and needs to be tested. Among them, a stable code version refers to a code version that has been tested and launched and is considered to be correct.

[0052] Step S206, comparing the corresponding actual test results with the benchmark test results to obtain a comparison result, and determining whether the test case can run stably during the execution of the test version code based on the comparison results of each step of the test case.

[0053] In this embodiment, after the use case is executed on the test version, the execution screenshots of each step of the test version can be compared with the execution screenshots of the corresponding steps in the baseline results in turn to obtain the difference points, so as to determine whether the test case has passed the test. By comparing the results, the difference between the test version code and the baseline version code can be found, thereby determining whether the test case can run stably when executing the test version code.

[0054] In addition, in order to improve versatility and reduce false alarms, comparison area configuration and comparison accuracy configuration functions are also provided.

[0055] From the above, it can be seen that in an embodiment of the present invention, a benchmark result is obtained after the test case is executed and stably runs with the correct version code in a historical time period, and the benchmark result includes the benchmark test results corresponding to each step of the test case; the test case is executed with the test version code to obtain the actual test result of each step of the test case; the corresponding actual test result is compared with the benchmark test result to obtain a comparison result, and based on the comparison results of each step of the test case, it is determined whether the test case can run stably in the process of executing the test version code, thereby achieving the purpose of analyzing the UI automation test results by image comparison, supporting automatic calculation of images and manual configuration of comparison areas, and selecting the corresponding comparison accuracy according to the test scenario, thereby achieving the technical effect of not needing to manually write and maintain assertions and reducing false alarms, and improving configuration efficiency and test versatility.

[0056] The technical solution provided by the embodiment of the present invention solves the technical problems in the prior art that the writing workload of UI automation testing is large, verification points may be left behind, resulting in missed tests, and the false alarm rate is high, the versatility is insufficient, and it cannot be applied to a variety of test scenarios.

[0057] According to the above embodiment of the present invention, a benchmark result is obtained after the test case is executed and stably runs with the correct version code in a historical time period, including: executing the test case with the correct version code in a historical time period, and in the process of executing the test case, obtaining an execution screenshot of each step of the test case, the execution screenshot being a screenshot of the corresponding direct result obtained after the test case stably runs the current step on the target interface; determining whether the execution result in the execution screenshot is correct; if the execution result in the execution screenshot is correct, determining the execution screenshot as the benchmark test result of the current step, and determining the benchmark test results of all steps as the benchmark results.

[0058] In this embodiment, when executed for the first time, the test case is executed once on a stable code version. During the execution process, a screenshot of the execution of each step is automatically obtained and compared with the expected result. After confirming that the result is correct, it is set as the baseline (i.e., the benchmark test result), which can be used as a comparison benchmark for subsequent test results. By establishing the benchmark test result, it can be ensured that in the subsequent testing process, the function and performance of the system have not changed or that changes can be discovered in time.

[0059] According to the above-mentioned embodiment of the present invention, the corresponding actual test results are compared with the benchmark test results to obtain the comparison results, including: determining the interface displaying the execution results of the test case as the target interface; using automatic configuration and / or manual configuration, and according to the display content of each area in the target interface, determining the comparison area and the non-comparison area in the target interface, the comparison area is the area for comparing the actual test results with the benchmark test results, and the non-comparison area is the area that does not need to be compared; according to the test scenario of the test case, determining the comparison accuracy, and according to the comparison accuracy and the comparison area, comparing the corresponding actual test results with the benchmark test results to obtain the comparison results.

[0060] Figure 3 is a flow chart of an automated testing method according to an embodiment of the present invention. Figure 3 As shown, after the baseline is set, the image comparison configuration is performed to configure the area and accuracy of the image comparison.

[0061] Optionally, the above comparison area configuration may include two methods: automatic calculation and manual configuration.

[0062] Figure 4 Schematic diagram of the effect of the automated testing method according to an embodiment of the present invention. Figure 4 As shown, the a frame is a carousel, which refers to a component that displays multiple pictures or text information in a fixed area in turn. It will change randomly. If the whole picture is compared directly, false positives may occur. Therefore, it is necessary to exclude this part of the area for comparison.

[0063] It should be noted that due to the large number of user interface elements, there may be some random changes or unconcerned areas in the screenshots, which do not need to be compared, and there may also be situations where some areas need to be excluded. Therefore, the automatic calculation + manual configuration method is adopted to select the comparison area. It supports selecting the areas to be compared or not to be compared for the execution screenshots of each step of the test case according to the needs of the test scenario. The comparison area configuration is an optional step. If no configuration is performed, the entire screenshot is compared by default. After the first configuration, the configuration can be retained and there is no need to repeat the configuration later.

[0064] In this embodiment, the target interface can be determined as the interface that displays the test case execution results. The areas in the target interface that need to be compared and the areas that do not need to be compared are determined based on the display content of the actual test results and the benchmark test results. The corresponding comparison accuracy is determined based on the test scenario of the test case to facilitate comparative analysis of the test results. By comparing the actual test results with the benchmark test results, the stability and performance of the system can be evaluated, potential problems can be discovered and repaired in a timely manner.

[0065] In the above embodiment of the present invention, automatic configuration is adopted to determine the comparison area and non-comparison area in the target interface according to the display content of each area in the target interface, including: using the correct version code to execute the test case a preset number of times to obtain a screenshot collection, the screenshot collection includes execution screenshots of each step of the test case at each execution, and the number of execution screenshots corresponding to one step is the same as the preset number of times; comparing the preset execution screenshots of the step, and obtaining an area with a similarity less than a first preset similarity as a random area; determining the random area as the non-comparison area, and determining the area in all execution screenshots except the random area as the comparison area.

[0066] In this embodiment, the automatic calculation of the comparison area can automatically calculate the random area in the execution screenshot, without the need to manually configure the comparison area for each step screenshot of each test script, thereby reducing labor costs and improving configuration efficiency.

[0067] Figure 5 is a flow chart of automatic calculation of contrast area according to an embodiment of the present invention. Figure 5 As shown, first, execute the UI automation test case once on a stable code version to obtain the execution screenshot of each step; after waiting for a period of time, execute the test case again on the same code version to obtain the execution screenshot of each step, and repeat the execution three times to obtain a screenshot set; compare the execution screenshots of each step in the three execution results one by one to obtain inconsistent areas, which are random areas. The remaining areas after removing the random areas are the areas to be compared in subsequent test executions.

[0068] In addition, the preliminary comparison area can be obtained through automatic calculation of the comparison area. If adjustment is needed, it can be adjusted through manual configuration. Manual configuration supports users to select the comparison area, that is, only the selected area is compared when comparing pictures, and other parts do not need to be paid attention to. It also supports users to select areas not to be compared, that is, when comparing pictures, the selected area is ignored and only the remaining areas are compared.

[0069] In the above embodiment of the present invention, the comparison accuracy is determined according to the test scenario of the test case, including: determining the accuracy requirement of the test scenario of the test case; when the accuracy requirement is greater than or equal to the first preset accuracy, determining the comparison accuracy as absolute comparison, and the absolute comparison is a method of comparing pixel by pixel; when the accuracy requirement is greater than or equal to the second preset accuracy and less than the first preset accuracy, determining the comparison accuracy as strict comparison, and the strict comparison is a method of strictly comparing operating parameters using the SSIM algorithm, and the operating parameters at least include the brightness and contrast of the target interface; when the accuracy requirement is less than the second preset accuracy, determining the comparison accuracy as content comparison, and the content comparison is a method of only comparing text.

[0070] It should be noted that due to different test scenarios, the required comparison accuracy may also be different. Three comparison accuracy options are provided: absolute comparison, strict comparison, and content comparison to meet the needs of different test scenarios. The default is strict comparison.

[0071] Optionally, the first preset accuracy can be represented by a pixel-level error, for example, an error less than or equal to 1 pixel; the second preset accuracy can be represented by an SSIM indicator, for example, an SSIM value greater than 0.9; the accuracy less than the second preset accuracy can be represented by the accuracy of text comparison, for example, an accuracy greater than 95%.

[0072] In this embodiment, the corresponding comparison accuracy can be determined according to the accuracy requirements of the test scenario of the test case. By selecting appropriate comparison accuracy and accuracy requirements of the test scenario, it can be ensured that the design and execution of the test case are more accurate and effective, thereby improving the quality and reliability of the test.

[0073] The three specific comparison accuracies are as follows: 1) Absolute comparison: the baseline screenshot and the test screenshot are completely consistent, that is, every pixel is consistent, the comparison is successful, which is suitable for scenarios such as local refined detection, and is implemented by pixel-by-pixel comparison; 2) Strict comparison: the baseline screenshot and the test screenshot are similar in layout, content, color, resolution, etc., which is a successful comparison. Its verification granularity is closer to the user's real experience and has a wide range of applications. The SSIM algorithm is used to compare brightness, contrast, structure, etc., and a certain threshold is selected according to the test situation. If the calculation result exceeds the threshold, the comparison is successful, otherwise the comparison fails; 3) Content comparison: ignoring the differences in color and resolution, the baseline screenshot and the test screenshot are similar in content, which is a successful comparison. It is suitable for scenarios such as compatibility testing. The aHash algorithm is used to reduce the baseline screenshot and the test screenshot to the same size and convert them into grayscale images to obtain the hash value, and the Hamming distance between the two is calculated. A certain threshold is selected according to the test situation. If the distance is less than the threshold, the comparison is successful, otherwise the comparison fails.

[0074] It should be noted that the above comparison algorithm can be implemented by other similar algorithms to adjust the comparison accuracy according to actual application needs.

[0075] In the above-mentioned embodiment of the present invention, the corresponding actual test results and benchmark test results are compared according to the comparison accuracy and the comparison area to obtain the comparison result, including: obtaining the actual execution screenshot of the current step in the actual test result and the benchmark execution screenshot of the current step in the benchmark test result; according to the comparison area, the actual execution screenshot and the benchmark execution screenshot are cropped to retain only the comparison area in the actual execution screenshot and the comparison area in the benchmark execution screenshot, which are the first area and the second area respectively; the first area and the second area are compared according to the comparison accuracy to obtain the comparison result.

[0076] In this embodiment, an image comparison can be performed on the actual test results and the benchmark test results to check the difference between the actual execution screenshot and the benchmark execution screenshot in the comparison area. By cropping the comparison area of ​​the actual execution screenshot and the benchmark execution screenshot, the interference of irrelevant information can be reduced and the accuracy of the comparison can be improved. At the same time, by comparing the first area and the second area with comparison accuracy, the difference between the actual execution and the benchmark execution can be detected more accurately, so as to more quickly and accurately discover changes and problems in the UI interface.

[0077] Figure 6 is a flowchart of picture comparison according to an embodiment of the present invention, such as Figure 6 As shown, obtain the execution screenshot of the current step and the corresponding execution screenshot in the benchmark result, crop the screenshot according to the comparison area configuration, leaving only the comparison area, perform image comparison according to the comparison accuracy configuration, and record the comparison results, and compare the execution screenshot of each step in the test result (i.e., the first area) with the execution screenshot of the corresponding step in the baseline result (i.e., the second area) according to the configured comparison area and comparison accuracy.

[0078] According to the above embodiment of the present invention, based on the comparison results of each step of the test case, it is determined whether the test case can run stably in the process of executing the test version code, including: performing weighted averaging processing on the comparison results of all steps of the test case to obtain a total similarity, the comparison result is the similarity between the actual test result of the corresponding step and the benchmark test result; when the total similarity is greater than or equal to the second preset similarity, it is determined that the test case can run stably in the process of executing the test version code; when the total similarity is less than the second preset similarity, it is determined that the test case can run stably in the process of executing the test version code.

[0079] In this embodiment, the similarity between the actual test results and the benchmark test results of all steps corresponding to the test case can be weighted averaged to obtain the total similarity. If the test result execution screenshots of all steps of the test case are similar to the baseline result execution screenshots, the test case passes.

[0080] In addition, if the comparison fails, the test results are returned to the user.

[0081] like Figure 3 As shown, when it is confirmed that this test has passed, the test result is automatically set as a new baseline.

[0082] The technical solution provided by the above embodiment of the present invention solves the following problems: Commonly used UI automation test frameworks, such as Web UI automation test framework selenium and App UI automation test framework appium, support element search, simulation operation, etc., and also support assertion writing, such as judging whether an element exists, judging whether an element is selected, checking whether an element attribute is correct, etc. However, when using such test frameworks for UI automation testing, it is necessary to manually write assertions and manually set assertion schemes. Since they are manually written, they have the following disadvantages: first, the writing workload is large and the labor cost is high; second, due to manual omission of checkpoints or complex test scenarios, it may cause missed tests; third, changes in the front-end page may cause assertion failures and high maintenance costs. At the same time, it has the following beneficial effects: 1) The UI automation test results are analyzed by image comparison, without the need to manually write and maintain assertions, which reduces labor costs; 2) The comparison area adopts the method of automatic calculation + manual configuration, which can effectively reduce the false alarms caused by random areas or unconcerned areas; 3) The comparison accuracy configuration adapts to the comparison requirements of different test scenarios and improves the versatility of the solution.

[0083] From the above, it can be seen that in an embodiment of the present invention, the scheme of using image comparison for result judgment is optimized, and the comparison area can be set by automatic calculation to reduce false alarms caused by random areas. At the same time, the automatic calculation method saves the manpower cost of manual configuration and improves the configuration efficiency; in terms of image comparison accuracy, three different comparison accuracies are provided: absolute comparison, strict comparison, and content comparison, which can be selected according to different test scenarios, thereby improving the versatility of this test method; based on automatic calculation of comparison areas and multi-precision comparison, an optimized UI automation testing method and device are designed, which includes four modules: baseline setting, image comparison configuration, test case execution, and image comparison, which reduces the labor cost of assertion writing, reduces false alarms, and improves versatility.

[0084] That is, the technical solution provided by the embodiment of the present invention comprehensively considers various factors, judges the test results by comparing the test version execution screenshot with the baseline execution screenshot, and supports automatic calculation and manual configuration of image comparison areas, which can effectively reduce false alarms. At the same time, it provides three different comparison accuracy options to adapt to different test scenarios, improves versatility, and eliminates the need for manual assertions.

[0085] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0086] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0087] According to an embodiment of the present invention, there is also provided an automated testing device for implementing the automated testing method. Figure 7 is a schematic diagram of an automated testing device according to an embodiment of the present invention. Figure 7 As shown, the device includes: a first execution module 701, a second execution module 703 and a determination module 705. The automatic test device is described below.

[0088] The first execution module 701 is used to execute the test case with the correct version code in the historical time period and obtain the benchmark result after stable operation. The benchmark result includes the benchmark test result corresponding to each step of the test case.

[0089] The second execution module 703 is used to execute the test case using the test version code to obtain the actual test result of each step of the test case.

[0090] The determination module 705 is used to compare the corresponding actual test results with the benchmark test results to obtain a comparison result, and determine whether the test case can run stably during the execution of the test version code based on the comparison results of each step of the test case.

[0091] It should be noted here that the above-mentioned first execution module 701, second execution module 703 and determination module 705 correspond to steps S202 to S206 in the above-mentioned embodiment, and the three modules have the same instances and application scenarios implemented by the corresponding steps, but are not limited to the contents disclosed in the above-mentioned embodiment.

[0092] As can be seen from the above, in the scheme recorded in the above-mentioned embodiment of the present invention, first, the first execution module can be used to execute the test case with the correct version code in the historical time period and obtain the benchmark result after stable operation, and the benchmark result includes the benchmark test result corresponding to each step of the test case; then the second execution module can be used to execute the test case with the test version code to obtain the actual test result of each step of the test case; finally, the determination module can be used to compare the corresponding actual test result with the benchmark test result to obtain the comparison result, and according to the comparison result of each step of the test case, it is determined whether the test case can run stably in the process of executing the test version code, thereby achieving the purpose of analyzing the UI automation test results by using image comparison, supporting automatic calculation of images and manual configuration of comparison areas, and selecting the corresponding comparison accuracy according to the test scenario, thereby achieving the technical effect of not needing to manually write and maintain assertions and reducing false alarms, and improving the configuration efficiency and the versatility of the test.

[0093] The technical solution provided by the embodiment of the present invention solves the technical problems in the prior art that the writing workload of UI automation testing is large, verification points may be left behind, resulting in missed tests, and the false alarm rate is high, the versatility is insufficient, and it cannot be applied to a variety of test scenarios.

[0094] In an optional embodiment, a first execution module includes: an execution unit, which is used to execute the test case using the correct version code in a historical time period, and in the process of executing the test case, obtain an execution screenshot of each step of the test case, the execution screenshot being a screenshot of the corresponding direct result obtained after the test case stably runs the current step on the target interface; a first determination unit, which is used to determine whether the execution result in the execution screenshot is correct; and a second determination unit, which is used to determine the execution screenshot as the benchmark test result of the current step if the execution result in the execution screenshot is correct, and determine the benchmark test results of all steps as the benchmark results.

[0095] In an optional embodiment, the determination module includes: a third determination unit, which is used to determine the interface that displays the execution results of the test case as the target interface; a fourth determination unit, which is used to determine the comparison area and the non-comparison area in the target interface by automatic configuration and / or manual configuration, and according to the display content of each area in the target interface, the comparison area is the area for comparing the actual test results with the benchmark test results, and the non-comparison area is the area that does not need to be compared; a comparison unit, which is used to determine the comparison accuracy according to the test scenario of the test case, and compare the corresponding actual test results with the benchmark test results according to the comparison accuracy and the comparison area to obtain a comparison result.

[0096] In an optional embodiment, the fourth determination unit includes: an execution subunit, which is used to execute the test case a preset number of times using the correct version code to obtain a screenshot collection, wherein the screenshot collection includes execution screenshots of each step of the test case during each execution, and the number of execution screenshots corresponding to one step is the same as the preset number of times; a first comparison subunit, which is used to compare the preset execution screenshots of the step, and obtain an area with a similarity less than a first preset similarity as a random area; a first determination subunit, which is used to determine the random area as a non-comparison area, and determine the area in all execution screenshots except the random area as a comparison area.

[0097] In an optional embodiment, the comparison unit includes: a second determination subunit, which is used to determine the accuracy requirement of the test scenario of the test case; a third determination subunit, which is used to determine the comparison accuracy as absolute comparison when the accuracy requirement is greater than or equal to the first preset accuracy, and the absolute comparison is a method of comparing pixel by pixel; a fourth determination subunit, which is used to determine the comparison accuracy as strict comparison when the accuracy requirement is greater than or equal to the second preset accuracy and less than the first preset accuracy, and the strict comparison is a method of strictly comparing operating parameters using the SSIM algorithm, and the operating parameters include at least the brightness and contrast of the target interface; a fifth determination subunit, which is used to determine the comparison accuracy as content comparison when the accuracy requirement is less than the second preset accuracy, and the content comparison is a method of only comparing text.

[0098] In an optional embodiment, the comparison unit includes: an acquisition subunit, used to acquire an actual execution screenshot of the current step in the actual test result and a benchmark execution screenshot of the current step in the benchmark test result; a cropping subunit, used to crop the actual execution screenshot and the benchmark execution screenshot according to the comparison area, retaining only the comparison area in the actual execution screenshot and the comparison area in the benchmark execution screenshot, which are the first area and the second area respectively; a second comparison subunit, used to compare the first area and the second area according to the comparison accuracy to obtain a comparison result.

[0099] In an optional embodiment, the determination module includes: a weighted average processing unit, which is used to perform weighted average processing on the comparison results of all steps of the test case to obtain a total similarity, wherein the comparison result is the similarity between the actual test result of the corresponding step and the benchmark test result; a fifth determination unit, which is used to determine that the test case can run stably in the process of executing the test version code when the total similarity is greater than or equal to the second preset similarity; and a sixth determination unit, which is used to determine that the test case can run stably in the process of executing the test version code when the total similarity is less than the second preset similarity.

[0100] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include an automated testing method for executing any of the above items.

[0101] According to another aspect of an embodiment of the present invention, a processor is further provided, and the processor is used to run a program, wherein the program executes any one of the above-mentioned automated testing methods when it is run.

[0102] According to another aspect of an embodiment of the present invention, a computer program product is provided, including computer instructions, and when the computer instructions are executed by a processor, any one of the above-mentioned automated testing methods is executed.

[0103] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned automated testing methods.

[0104] Optionally, in this embodiment, the computer-readable storage medium may be located in any one of the computer terminals in a computer terminal group in a computer network, or in any one of the communication devices in a communication device group.

[0105] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining a benchmark result after executing the test case with the correct version code and running it stably in a historical time period, the benchmark result including the benchmark test result corresponding to each step of the test case; executing the test case with the test version code to obtain an actual test result of each step of the test case; comparing the corresponding actual test result with the benchmark test result to obtain a comparison result, and determining whether the test case can run stably during the execution of the test version code based on the comparison result of each step of the test case.

[0106] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: executing the test case using the correct version code in a historical time period, and in the process of executing the test case, obtaining an execution screenshot of each step of the test case, the execution screenshot being a screenshot of the corresponding direct result obtained after the test case stably runs the current step on the target interface; determining whether the execution result in the execution screenshot is correct; if the execution result in the execution screenshot is correct, determining the execution screenshot as the benchmark test result of the current step, and determining the benchmark test results of all steps as the benchmark results.

[0107] Optionally, in this embodiment, a computer-readable storage medium is configured to store program codes for executing the following steps: determining an interface that displays the execution results of a test case as a target interface; using automatic configuration and / or manual configuration, and based on the display content of each area in the target interface, determining a comparison area and a non-comparison area in the target interface, the comparison area being an area for comparing actual test results with benchmark test results, and the non-comparison area being an area that does not require comparison; determining a comparison accuracy based on the test scenario of the test case, and based on the comparison accuracy and the comparison area, comparing the corresponding actual test results with the benchmark test results to obtain a comparison result.

[0108] Optionally, in this embodiment, a computer-readable storage medium is configured to store program code for performing the following steps: executing a test case a preset number of times using the correct version code to obtain a screenshot collection, the screenshot collection including execution screenshots of each step of the test case during each execution, and the number of execution screenshots corresponding to a step is the same as the preset number of times; comparing the preset execution screenshots of the step, and obtaining an area with a similarity less than a first preset similarity as a random area; determining the random area as a non-comparison area, and determining the area in all execution screenshots except the random area as a comparison area.

[0109] Optionally, in this embodiment, a computer-readable storage medium is configured to store program code for performing the following steps: determining an accuracy requirement for a test scenario of a test case; when the accuracy requirement is greater than or equal to a first preset accuracy, determining the comparison accuracy to be an absolute comparison, in which the absolute comparison is performed pixel by pixel; when the accuracy requirement is greater than or equal to a second preset accuracy and less than the first preset accuracy, determining the comparison accuracy to be a strict comparison, in which the strict comparison is performed by using an SSIM algorithm to compare operating parameters, wherein the operating parameters include at least the brightness and contrast of the target interface; when the accuracy requirement is less than the second preset accuracy, determining the comparison accuracy to be a content comparison, in which the content comparison is performed only in which text is compared.

[0110] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining an actual execution screenshot of the current step in the actual test results and a benchmark execution screenshot of the current step in the benchmark test results; cropping the actual execution screenshot and the benchmark execution screenshot according to the comparison area, retaining only the comparison area in the actual execution screenshot and the comparison area in the benchmark execution screenshot, which are the first area and the second area respectively; comparing the first area and the second area according to the comparison accuracy to obtain a comparison result.

[0111] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: performing weighted averaging processing on the comparison results of all steps of the test case to obtain a total similarity, wherein the comparison result is the similarity between the actual test result of the corresponding step and the benchmark test result; when the total similarity is greater than or equal to a second preset similarity, determining that the test case can run stably during the execution of the test version code; when the total similarity is less than the second preset similarity, determining that the test case can run stably during the execution of the test version code.

[0112] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0113] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0114] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0115] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0116] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0117] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0118] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An automated testing method, characterized in that: include: The benchmark results are obtained after the test cases are executed and stably run using the correct version code in the historical time period, wherein the benchmark results include the benchmark test results corresponding to each step of the test cases; Execute the test case using the test version code to obtain actual test results of each step of the test case; The corresponding actual test results are compared with the benchmark test results to obtain a comparison result, and based on the comparison results of each step of the test case, it is determined whether the test case can run stably during the execution of the test version code.

2. The method according to claim 1, characterized in that: Benchmark results were obtained after running the test cases stably using the correct version of the code in the historical time period, including: The test case is executed using the correct version code in a historical time period, and during the execution of the test case, an execution screenshot of each step of the test case is obtained, wherein the execution screenshot is a screenshot of a corresponding direct result obtained after the test case stably runs the current step on the target interface; Determine whether the execution result in the execution screenshot is correct; In the case where the execution result in the execution screenshot is correct, the execution screenshot is determined as the benchmark test result of the current step, and the benchmark test results of all the steps are determined as the benchmark result.

3. The method according to claim 1, characterized in that Comparing the corresponding actual test results with the benchmark test results to obtain a comparison result, including: Determining an interface displaying the execution result of the test case as a target interface; Automatic configuration and / or manual configuration are adopted, and according to the display content of each area in the target interface, a comparison area and a non-comparison area in the target interface are determined, wherein the comparison area is an area where the actual test result and the benchmark test result are compared, and the non-comparison area is an area where no comparison is required; According to the test scenario of the test case, a comparison accuracy is determined, and according to the comparison accuracy and the comparison area, the corresponding actual test result is compared with the benchmark test result to obtain the comparison result.

4. The method according to claim 3, characterized in that Automatically configuring and determining the contrast area and the non-contrast area in the target interface according to the display content of each area in the target interface includes: Execute the test case a preset number of times using the correct version code to obtain a screenshot set, wherein the screenshot set includes execution screenshots of each step of the test case during each execution, and the number of execution screenshots corresponding to one step is the same as the preset number of times; Comparing the preset execution screenshots of the step, and obtaining a region with a similarity less than a first preset similarity as a random region; The random area is determined as the non-comparison area, and all areas in the execution screenshot except the random area are determined as the comparison area.

5. The method according to claim 3, characterized in that: According to the test scenario of the test case, the comparison accuracy is determined, including: Determining the accuracy requirements of the test scenarios of the test cases; When the accuracy requirement is greater than or equal to a first preset accuracy, determining that the comparison accuracy is an absolute comparison, and the absolute comparison is a pixel-by-pixel comparison; When the accuracy requirement is greater than or equal to the second preset accuracy and less than the first preset accuracy, the comparison accuracy is determined to be a strict comparison, and the strict comparison is a method of strictly comparing operating parameters using an SSIM algorithm, and the operating parameters at least include brightness and contrast of the target interface; When the accuracy requirement is less than the second preset accuracy, the comparison accuracy is determined to be content comparison, and the content comparison is a method of performing text comparison only.

6. The method according to claim 3, characterized in that According to the comparison accuracy and the comparison area, the corresponding actual test result is compared with the benchmark test result to obtain a comparison result, including: Obtaining an actual execution screenshot of the current step in the actual test result and a benchmark execution screenshot of the current step in the benchmark test result; According to the comparison area, the actual execution screenshot and the benchmark execution screenshot are cropped to retain only the comparison area in the actual execution screenshot and the comparison area in the benchmark execution screenshot, which are the first area and the second area respectively; The first region and the second region are compared according to the comparison accuracy to obtain the comparison result.

7. The method according to claim 1, characterized in that Determining whether the test case can run stably during the execution of the test version code according to the comparison results of each of the steps of the test case includes: Performing weighted average processing on the comparison results of all the steps of the test case to obtain a total similarity, wherein the comparison result is the similarity between the actual test result of the corresponding step and the benchmark test result; When the total similarity is greater than or equal to a second preset similarity, determining that the test case can run stably during the execution of the test version code; When the total similarity is less than the second preset similarity, it is determined that the test case can run stably during the execution of the test version code.

8. An automated testing device, characterized in that: include: A first execution module is used to execute the test case with the correct version code in the historical time period and obtain a benchmark result after stable operation, wherein the benchmark result includes a benchmark test result corresponding to each step of the test case; A second execution module is used to execute the test case using the test version code to obtain actual test results of each step of the test case; The determination module is used to compare the corresponding actual test results with the benchmark test results to obtain a comparison result, and determine whether the test case can run stably during the execution of the test version code based on the comparison results of each step of the test case.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the automated testing method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a program for executing the automated testing method described in any one of claims 1 to 7.

Images