An application program UI automatic test method and system

Abstract

The application discloses a kind of application program UI automation test method and system, the method comprises: obtaining prewritten test case, and intercepting current UI interface image of software to be tested;Extract the keyword in the test case, based on the keyword and current UI interface image, respectively by first inference model and second inference model, determine whether the test case can be executed;If two inference models are all determined to be executable and there is consistency in executable result, then the execution strategy output by the first inference model is used to test the UI interface of the software to be tested, and the associated information in the test execution is recorded;If the second inference model determines to be not executable, then a judgment is made by calling a decision inference model, and if the decision inference model still determines to be not executable, then the execution process of the current test case is aborted.The scheme of the present application can not only realize the automation test of UI interface, but also can guarantee the reliability and robustness of automation test.

Description

Technical Field

[0001] This invention belongs to the field of software testing technology, and in particular relates to an automated testing method and system for application UI. Background Technology

[0002] Traditional software testing models, especially UI automation testing, have long been plagued by a dilemma of "high theoretical value but poor practical results." On the one hand, business iteration cycles have been compressed from "monthly" to "daily," placing unprecedented demands on testing speed and frequency; on the other hand, the maintenance costs of traditional automated testing remain high. Data shows that each script failure requires an average of 30 minutes of manual troubleshooting. This "record and playback" type of automation essentially mechanically solidifies manual operations. Once there are minor changes to the application interface (UI) or adjustments to business logic, scripts will fail in large numbers, trapping the team in a vicious cycle of "writing-failure-fixing," making testing the most expensive and unpredictable link in the software delivery chain. Against this backdrop, the traditional model that simply pursues execution automation can no longer cope with complex business scenarios involving multi-platform collaboration and high-frequency iteration.

[0003] In recent years, artificial intelligence technologies such as Large Language Models (LLM), Retrieval Augmentation (RAG), Visual Large Models (VLM), and Multi-Agent have made breakthrough progress. However, testing schemes based on a single AI model, such as patent CN121636345A, are prone to misjudgment when faced with complex, dynamic, or ambiguous UI interfaces, making the testing operation prone to errors or omissions, resulting in low testing reliability. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide an application UI automated testing method and system to solve the problem of low reliability in current automated testing of single AI models.

[0005] In a first aspect of the present invention, an application UI automated testing method is provided, comprising: Obtain pre-written test cases and capture the current UI image of the software under test; Extract keywords from the test cases, and based on the keywords and the current UI image, determine whether the test cases can be executed using the first inference model and the second inference model, respectively. The first inference model and the second inference model are constructed based on the same multimodal model. The first inference model is a model that uses aggressive trial as an inference strategy, and the second inference model is a model that uses rigorous verification as an inference strategy. If both the first inference model and the second inference model are determined to be executable and the executable results are consistent, then the UI interface of the software to be tested will be tested according to the execution strategy output by the first inference model, and the related information during the test execution will be recorded. If the second inference model determines that the test case is not executable, the adjudication inference model is called to determine whether the test case can be executed. If the adjudication inference model still determines that the test case is not executable, the execution process of the current test case is stopped and an alarm is generated. The adjudication inference model is a third-party large language model.

[0006] In a second aspect of the present invention, an application UI automated testing system is provided, comprising: The data acquisition module is used to acquire pre-written test cases and capture images of the current UI interface of the software under test. The execution judgment module is used to extract keywords from the test cases, and based on the keywords and the current UI interface image, determine whether the test cases can be executed through the first inference model and the second inference model respectively. The first inference model and the second inference model are constructed based on the same multimodal model. The first inference model is a model that uses aggressive trial as an inference strategy, and the second inference model is a model that uses rigorous verification as an inference strategy. The test execution module is used to test the software under test according to the execution strategy output by the first inference model when both the first inference model and the second inference model are determined to be executable and the executable results are consistent, and to record the related information during the test execution. The execution adjudication module is used to call the adjudication inference model to determine whether the test case can be executed when the second inference model determines that it is not executable. If the adjudication inference model still determines that it is not executable, the execution process of the current test case is stopped and an alarm is generated. The adjudication inference model is a third-party large language model.

[0007] In a third aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor executes the computer program to implement the steps of the method as described in the first aspect of the present invention.

[0008] In a fourth aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method provided in the first aspect of the present invention.

[0009] In this embodiment of the invention, by adopting a multi-model collaborative reasoning decision-making mechanism, an adjudication model is introduced for verification based on sampling different reasoning strategies. This not only enables automated testing of the UI interface and improves testing efficiency, but also avoids problems such as operational errors and omissions that exist in testing that relies on a single model, thereby ensuring the reliability and robustness of automated testing. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating an application UI automation testing method according to an embodiment of the present invention; Figure 2 A schematic diagram of the structure of an application UI automated testing system provided in one embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0012] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0013] It should be understood that the terms "comprising" and other similar expressions in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, or apparatus that includes a series of steps or units and is not limited to the listed steps or units. Furthermore, "first" and "second" are used to distinguish different objects and are not intended to describe a specific order.

[0014] Please see Figure 1 The present invention provides a flowchart illustrating an application UI automated testing method, comprising: S101. Obtain pre-written test cases and capture the current UI image of the software under test; The test cases can be test cases described in natural language. For example, some test cases are described as: "Click the 'Start Navigation' button" and "Enter 'Beijing' in the search box and search". These test cases can verify the validity of the "Start Navigation" button and the "Search Box" on the interface.

[0015] The capture of UI interface images typically includes images before and after a certain operation is performed in the test case, such as the UI interface before the button is clicked and the interface displayed after the button is clicked.

[0016] Optionally, an image of the current UI interface of the software under test on the touch device can be captured using an external camera, ADB commands, or WDA tools.

[0017] The screen can be captured in real time by using an external camera, or a screenshot of the current touch device screen can be obtained through ADB (Android Debug Bridge) commands, WDA (WebDriverAgent) tools, etc., to obtain the interface image to be analyzed.

[0018] S102. Extract keywords from the test cases, and based on the keywords and the current UI interface image, determine whether the test cases can be executed using the first inference model and the second inference model respectively. The first inference model and the second inference model are constructed based on the same multimodal model. The first inference model is a model that uses aggressive trial as an inference strategy, and the second inference model is a model that uses rigorous verification as an inference strategy.

[0019] Natural language processing can be used to extract keywords from test cases. These keywords and UI images are then input into the first and second inference models, which output the judgment result on whether each step in the test case can be executed.

[0020] Both the first and second inference models are built on large-scale multimodal models, such as GPT-4V and Qwen-VL, and have different prompt word engineering and parameter configurations, and their corresponding decision-making strategies are also different.

[0021] S103. If both the first inference model and the second inference model are determined to be executable and the executable results are consistent, then the UI interface of the software to be tested shall be tested according to the execution strategy output by the first inference model, and the related information in the test execution shall be recorded. If both the first inference model and the second inference model determine that the same steps in the test case are executable and the executable results are consistent, then the test will be performed according to the execution strategy output by the first inference model. The execution strategy may include UI element coordinates and recognition results.

[0022] The associated information during test execution can include the original instructions, judgment results, execution coordinates, screenshots before and after execution, and execution time. This associated information can be packaged as log information for test case execution.

[0023] S104. If the second inference model determines that the test case is unexecutable, the adjudication inference model is called to determine whether the test case can be executed. If the adjudication inference model still determines that the test case is unexecutable, the execution process of the current test case is stopped and an alarm is generated. The adjudication inference model is a third-party large language model.

[0024] For example, the first inference model is configured with a prompt word "attempt to execute" to guide the inference model to actively attempt to execute test cases in the UI interface. For instance, if the prompt word of the first inference model is set to "find the coordinates of the UI element most likely related to the instruction, even if not completely certain", then the output will always contain an actionable screen coordinate (x, y), and the output format can be {"action": "tap", "coordinates": [x, y], "confidence": 0.75}.

[0025] The second inference model is configured with a "strict judgment" prompt to guide the inference model to objectively determine whether the test case execution is allowed on the UI interface. For example, if the prompt of the second inference model is set to "strictly judge whether the current interface allows the execution of the instruction, output 'executable' only when certain, otherwise output 'not executable'", then the output will be a binary classification result and an optional reason. The output format can be {"executable": false, "reason": "No 'Start Navigation' button found"}.

[0026] The adjudication inference model is a model with a larger parameter scale or stronger logical reasoning ability compared to the first and second inference models, such as third-party models like GPT-4 and Claude-3. The adjudication inference model receives the same interface images and test cases, and can refer to the "unexecutable reasons" in the output results given by the second inference model to perform secondary analysis and judgment, outputting the final executable or unexecutable decision.

[0027] When the adjudication inference model determines that the test case is unexecutable, the execution process of the current test case is terminated, and an alarm is generated so that testers can manually check and intervene. Simultaneously, the contextual data that led to the execution failure is recorded, which may include UI images, execution instructions, the intermediate process and results of the model's inference. Based on this contextual data, the first and second inference models can be fine-tuned or the prompts optimized to achieve closed-loop learning.

[0028] In this embodiment, multiple verifications of the test case execution process are achieved through collaborative judgment using inference models, thereby improving the reliability of automated testing, avoiding misjudgments by a single model, and ensuring the accuracy and robustness of automated testing.

[0029] In one embodiment, step S102, determining whether the test case can be executed using the first inference model and the second inference model respectively, includes: Identify UI elements in the current UI interface image, compare the UI elements with the keywords, and determine whether the UI elements and the execution objects in the keywords are the same; If they are different, the system actively searches and judges all possible UI elements related to the execution object in the current UI interface image based on the first inference model, calculates the confidence of each UI element as the execution object, and outputs all UI elements with a confidence higher than the preset value. If they are the same, the degree of matching between the UI elements and the keyword description is verified based on the second inference model, and the result of whether the test case can be executed is output based on the degree of matching.

[0030] The first inference model outputs all UI elements with confidence scores higher than a threshold based on the confidence scores of all relevant UI elements in the current UI image as execution objects. These UI elements are considered as potential test case execution objects. If the output UI element count is not zero, the test case can be considered to be executable; if it is zero, the test case is determined to be unexecutable.

[0031] The second inference model calculates the matching degree between UI elements and the keywords. If the matching degree is higher than a preset threshold, the test case is considered to be executable; otherwise, the test case is considered not executable.

[0032] In one embodiment, determining whether the test case can be executed in step S102 further includes: If both the first and second inference models are determined to be unexecutable, or if the executable results in the first and second inference models are inconsistent, the execution flow of the current test case will be paused, prompting manual intervention, and the prompt words of the first and second inference models can be optimized and adjusted.

[0033] In one embodiment, step S104, which involves calling the adjudication inference model to determine whether the test case can be executed, further includes: If the adjudication inference model determines that the test is executable, the execution object and its coordinates output by the first inference model are obtained. The test case is executed based on the object coordinates, and the original instructions, adjudication results, execution object coordinates, screenshots before and after execution, and execution timestamps of the test case are recorded. The recorded information is saved to the log file in a structured format.

[0034] For example, when the adjudication inference model determines that the operation is executable, the coordinates output by the first inference model can be converted into specific device control instructions. For instance, for an Android device, the ADB command is generated and executed: adb shell inputtap [x] [y]; for physical robotic arm control, the screen coordinates are converted into the spatial coordinates of the robotic arm's end effector through a coordinate transformation algorithm, and the click operation is driven.

[0035] In some embodiments, after all test cases have been executed, the complete execution log is analyzed. By comparing the interface changes before and after the test case execution, the effectiveness of the test case execution is verified, and the inference model can be further optimized. For example, for the "click the login button" test case, it is determined whether the user is successfully redirected to the login success page or whether the expected user information appears after execution.

[0036] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0037] Figure 2 This is a schematic diagram of the structure of an application UI automated testing system provided in an embodiment of the present invention. The system includes: The data acquisition module 210 is used to acquire pre-written test cases and capture the current UI interface image of the software under test; Optionally, an image of the current UI interface of the software under test on the touch device can be captured using an external camera, ADB commands, or WDA tools.

[0038] The execution judgment module 220 is used to extract keywords from the test cases and, based on the keywords and the current UI interface image, determine whether the test cases can be executed through the first inference model and the second inference model, respectively. The first inference model and the second inference model are constructed based on the same multimodal model. The first inference model is a model that uses aggressive trial as an inference strategy, and the second inference model is a model that uses rigorous verification as an inference strategy. The test execution module 230 is used to test the UI interface of the software to be tested according to the execution strategy output by the first inference model when both the first inference model and the second inference model are determined to be executable and the executable results are consistent, and to record the related information in the test execution. The execution decision module 240 is used to call the decision reasoning model to determine whether the test case can be executed when the second reasoning model determines that it is not executable. If the decision reasoning model still determines that it is not executable, the execution process of the current test case is stopped and an alarm is generated. The decision reasoning model is a third-party large language model.

[0039] In one embodiment, the execution judgment module 220 further includes: Identify UI elements in the current UI interface image, compare the UI elements with the keywords, and determine whether the UI elements and the execution objects in the keywords are the same; If they are different, the system actively searches and judges all possible UI elements related to the execution object in the current UI interface image based on the first inference model, calculates the confidence of each UI element as the execution object, and outputs all UI elements with a confidence higher than the preset value. If they are the same, the degree of matching between the UI elements and the keyword description is verified based on the second inference model, and the result of whether the test case can be executed is output based on the degree of matching.

[0040] In one embodiment, the execution judgment module 220 further includes: The suspend execution module is used to pause the execution flow of the current test case when both the first inference model and the second inference model are determined to be unexecutable, or when the executable results in the first inference model and the second inference model are inconsistent, prompting manual intervention, and allowing adjustment of the prompt words of the first inference model and the second inference model.

[0041] In one embodiment, the execution decision module 240 further includes: The log recording unit is used to obtain the execution object and object coordinates output by the first inference model when the adjudication inference model determines that it is executable, execute the test case based on the object coordinates, and record the original instructions of the test case, the adjudication result, the coordinates of the execution object, the screenshots before and after execution, and the execution timestamp. The recorded information is saved to the log file in a structured format.

[0042] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system and modules described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0043] Figure 3 This is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device is used for automated testing of software UI interfaces. Figure 3As shown, the electronic device 30 of this embodiment includes: a memory 310, a processor 320, and a system bus 330. The memory 310 includes an executable program 3101 stored thereon. As those skilled in the art will understand, Figure 3 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0044] The following is combined Figure 3 A detailed description of each component of the electronic device 30 is provided below: The memory 310 can be used to store software programs and modules. The processor 320 executes various functional applications and data processing of the electronic device by running the software programs and modules stored in the memory 310. The memory 310 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device (such as cached data), etc. In addition, the memory 310 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0045] An executable program 3101 containing automated testing methods is stored in memory 310. This executable program 3101 can be divided into one or more modules / units, which are stored in memory 310 and executed by processor 320 to perform UI interface testing, etc. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, describing the execution process of the executable program 3101 in the electronic device 30. For example, the executable program 3101 can be divided into functional modules such as a data acquisition module, an execution judgment module, a test execution module, and an execution adjudication module.

[0046] The processor 320 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 310, and by calling data stored in the memory 310, it performs various functions and processes data, thereby monitoring the overall status of the electronic device. Optionally, the processor 320 may include one or more processing units; preferably, the processor 320 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, application programs, etc., and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 320.

[0047] The system bus 330 is used to connect various functional components within the computer, transmitting data, address, and control information. Its type can be, for example, a PCI bus, an ISA bus, or a CAN bus. Instructions from the processor 320 are transmitted to the memory 310 via the bus, and the memory 310 sends data back to the processor 320. The system bus 330 is responsible for data and instruction exchange between the processor 320 and the memory 310. Of course, the system bus 330 can also connect to other devices, such as network interfaces and display devices.

[0048] In this embodiment of the invention, the executable program executed by the processor 320 included in the electronic device includes: Obtain pre-written test cases and capture the current UI image of the software under test; Extract keywords from the test cases, and based on the keywords and the current UI image, determine whether the test cases can be executed using the first inference model and the second inference model, respectively. The first inference model and the second inference model are constructed based on the same multimodal model. The first inference model is a model that uses aggressive trial as an inference strategy, and the second inference model is a model that uses rigorous verification as an inference strategy. If both the first inference model and the second inference model are determined to be executable and the executable results are consistent, then the UI interface of the software to be tested will be tested according to the execution strategy output by the first inference model, and the related information during the test execution will be recorded. If the second inference model determines that the test case is not executable, the adjudication inference model is called to determine whether the test case can be executed. If the adjudication inference model still determines that the test case is not executable, the execution process of the current test case is stopped and an alarm is generated. The adjudication inference model is a third-party large language model.

[0049] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0050] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0051] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automated testing of application UI, characterized in that, include: Obtain pre-written test cases and capture the current UI image of the software under test; Extract keywords from the test cases, and based on the keywords and the current UI image, determine whether the test cases can be executed using the first inference model and the second inference model, respectively. The first inference model and the second inference model are constructed based on the same multimodal model. The first inference model is a model that uses aggressive trial as an inference strategy, and the second inference model is a model that uses rigorous verification as an inference strategy. If both the first inference model and the second inference model are determined to be executable and the executable results are consistent, then the UI interface of the software to be tested will be tested according to the execution strategy output by the first inference model, and the related information during the test execution will be recorded. If the second inference model determines that the test case is not executable, the adjudication inference model is called to determine whether the test case can be executed. If the adjudication inference model still determines that the test case is not executable, the execution process of the current test case is stopped and an alarm is generated. The adjudication inference model is a third-party large language model.

2. The method according to claim 1, characterized in that, The process of capturing the current UI interface image of the software under test includes: Capture the current UI image of the software under test on the touch device using an external camera, ADB commands, or WDA tools.

3. The method according to claim 1, characterized in that, The step of determining whether the test case can be executed based on the keywords and the current UI image, using a first inference model and a second inference model respectively, includes: Identify UI elements in the current UI interface image, compare the UI elements with the keywords, and determine whether the UI elements and the execution objects in the keywords are the same; If they are different, the system actively searches and judges all possible UI elements related to the execution object in the current UI interface image based on the first inference model, calculates the confidence of each UI element as the execution object, and outputs all UI elements with a confidence higher than the preset value. If they are the same, the degree of matching between the UI elements and the keyword description is verified based on the second inference model, and the result of whether the test case can be executed is output based on the degree of matching.

4. The method according to claim 1, characterized in that, The step of determining whether the test case can be executed using the first inference model and the second inference model respectively also includes: If both the first and second inference models are determined to be unexecutable, or if the executable results in the first and second inference models are inconsistent, the execution flow of the current test case will be suspended, and manual intervention will be prompted.

5. The method according to claim 1, characterized in that, The step of calling the adjudication inference model to determine whether the test case can be executed also includes: If the adjudication inference model determines that the test is executable, the execution object and its coordinates output by the first inference model are obtained. The test case is executed based on the object coordinates, and the original instructions, adjudication results, execution object coordinates, screenshots before and after execution, and execution timestamps of the test case are recorded. The recorded information is saved to the log file in a structured format.

6. An application UI automated testing system, characterized in that, include: The data acquisition module is used to acquire pre-written test cases and capture images of the current UI interface of the software under test. The execution judgment module is used to extract keywords from the test cases, and based on the keywords and the current UI interface image, determine whether the test cases can be executed through the first inference model and the second inference model respectively. The first inference model and the second inference model are constructed based on the same multimodal model. The first inference model is a model that uses aggressive trial as an inference strategy, and the second inference model is a model that uses rigorous verification as an inference strategy. The test execution module is used to test the UI interface of the software to be tested according to the execution strategy output by the first inference model when both the first inference model and the second inference model are determined to be executable and the executable results are consistent, and to record the related information in the test execution. The execution adjudication module is used to call the adjudication inference model to determine whether the test case can be executed when the second inference model determines that it is not executable. If the adjudication inference model still determines that it is not executable, the execution process of the current test case is stopped and an alarm is generated. The adjudication inference model is a third-party large language model.

7. The system according to claim 6, characterized in that, The step of determining whether the test case can be executed based on the keywords and the current UI image, using a first inference model and a second inference model respectively, includes: Identify UI elements in the current UI interface image, compare the UI elements with the keywords, and determine whether the UI elements and the execution objects in the keywords are the same; If they are different, the system actively searches and judges all possible UI elements related to the execution object in the current UI interface image based on the first inference model, calculates the confidence of each UI element as the execution object, and outputs all UI elements with a confidence higher than the preset value. If they are the same, the degree of matching between the UI elements and the keyword description is verified based on the second inference model, and the result of whether the test case can be executed is output based on the degree of matching.

8. The system according to claim 6, characterized in that, The execution judgment module also includes: The suspend execution module is used to pause the execution flow of the current test case and prompt manual intervention when both the first inference model and the second inference model are determined to be unexecutable, or when the executable results in the first inference model and the second inference model are inconsistent.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of an application UI automated testing method as described in any one of claims 1 to 5.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed, it implements the steps of an application UI automation testing method as described in any one of claims 1 to 5.

Images