UI automatic test method and device, storage medium and program product

Abstract

The embodiment of the invention provides a UI automatic testing method and device, a storage medium and a program product. In the embodiment of the invention, the method does not depend on a test script, the language model used for UI automatic test directly and automatically identifies the interaction control needing to be tested according to the test requirement of the user on the target page and the screenshot of the target page, and the test instruction is generated for the interaction control needing to be tested; the terminal device where the target page is located is controlled through the test instruction to simulate user operation to conduct interaction control on the interaction control, and the test requirement of the user is met. According to the scheme, the to-be-tested control is automatically identified by the language model according to the test requirements and the page screenshot instead of depending on the test script, so that the test script does not need to be frequently modified due to the change of the page layout, the test cost can be reduced, and the test efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of UI automated testing technology, and in particular, to a UI automated testing method, device, storage medium, and program product. Background Art

[0002] With the continuous improvement of terminal capabilities, various applications (such as Web applications and mobile applications) have begun to highly rely on graphical user interfaces (UI), and user interaction paths have become more diverse (such as clicks, swipes, form inputs, etc.). Traditional manual testing is time-consuming and difficult to cover all scenarios, while UI automated testing based on automated test scripts has become a necessary means to ensure stability.

[0003] UI automated test scripts are usually closely related to the specific positions and attributes of UI elements. Once the UI interface changes, such as UI element position adjustment, attribute modification, or style update, the original test scripts are likely to require a large amount of modification and adjustment, thereby increasing the test cost and reducing the test efficiency. Summary of the Invention

[0004] Multiple aspects of this application provide a UI automated testing method, device, storage medium, and program product to improve test efficiency and reduce test cost.

[0005] An embodiment of this application provides a UI automated testing method, including: receiving an automated testing task submitted by a first terminal, where the automated testing task includes an identifier of a target application and test requirement information described in natural language, and the test requirement information is used to require an interaction test on an interaction control in a target page of the target application; according to the automated testing task, filling the test requirement information into a target prompt template to obtain a target model prompt, where the target prompt template at least includes: a requirement information filling item, and the requirement information filling item is used to carry the test requirement information; according to the automated testing task, sending a screenshot instruction to a second terminal where the target application is located for the second terminal to take a screenshot of the target page of the target application and return the screenshot of the target page; inputting the target model prompt and the screenshot of the target page into a language model based on artificial intelligence, and under the guidance of the target model prompt, identifying a target interaction control to be tested from the screenshot of the target page according to the test requirement information, and outputting target control information required for automated testing of the target interaction control; controlling the second terminal to perform an interaction operation on the target interaction control on the target page according to the target control information to complete the automated testing of the target interaction control.

[0006] An embodiment of the present application further provides an electronic device, including: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement the steps in the above method for parsing class inheritance relationships based on program execution or the code analysis method.

[0007] An embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to implement the steps in the above method for parsing class inheritance relationships based on program execution or the code analysis method.

[0008] In the embodiment of the present application, this case does not rely on test scripts. The language model for UI automated testing directly automatically identifies the interactive controls to be tested according to the user's test requirements for the target page and the screenshot of the target page, and generates test instructions for the interactive controls to be tested. Through the test instructions, the terminal device where the target page is located is controlled to simulate user operations to perform interactive control on the interactive controls, meeting the user's test requirements. In this solution, the language model automatically identifies the controls to be tested according to the test requirements and page screenshots, rather than relying on test scripts. Therefore, it is not necessary to frequently modify the test scripts due to changes in the page layout, which can reduce the test cost and improve the test efficiency.

[0009] Furthermore, in the case where some interactive controls are not marked or are not correctly marked in the XML document, this solution allows the language model to automatically identify the interactive controls to be tested according to the test requirements and page screenshots. If the test requirements require testing of "interactive controls not marked or not correctly marked in the XML document", the language model can identify the "interactive controls not marked or not correctly marked in the XML document" based on the page screenshots according to the test requirements, without being affected by the quality of the XML document, which can further improve the success rate and quality of UI testing. Description of the Drawings

[0010] The drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0011] Figure 1a is a schematic structural diagram of a UI automated testing system provided by an exemplary embodiment of the present application;

[0012] Figure 1b is a page schematic diagram of an automated test requirement filling page provided by an exemplary embodiment of the present application;

[0013] Figure 2a is a schematic flowchart of a UI automated testing method provided by an exemplary embodiment of the present application;

[0014] Figure 2b A flowchart of another UI automation testing method provided for another exemplary embodiment of the present application;

[0015] Figure 2c A page diagram of a target page after adding a status identifier provided for an exemplary embodiment of the present application;

[0016] Figure 3 A structural diagram of an electronic device provided for an exemplary embodiment of the present application. Detailed implementation manners

[0017] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with specific embodiments of the present application and the corresponding drawings. Apparently, the described embodiments are only a part rather than all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present application.

[0018] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties. The collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation entrances are provided for users to select authorization or rejection.

[0019] In addition, it should be noted that in the case of user interaction operations or trigger operations involved in the embodiments of the present application, the user interaction operations or trigger operations involved in the embodiments of the present application include but are not limited to: interaction operations in various ways such as touch operations, gesture operations, voice operations, head movement operations, and eye movement operations; among them, touch operations include but are not limited to: click operations, double-click operations, long-press operations, swipe operations, pinch operations, or mouse hover operations. Swipe operations include but are not limited to: straight swipes, curved swipes, etc.

[0020] Furthermore, it should be noted that in the case where the embodiments of the present application involve the jump between the first interface and the second interface, the jump methods involved in the embodiments of the present application include but are not limited to: directly jumping from the first interface to the second interface, first jumping from the first interface to the task interface and then jumping to the second interface when corresponding task operations are completed on the task interface; Completing the corresponding task operations on the task interface includes but is not limited to: when the task interface is implemented as a game interface, completing game operations on the game interface; when the task interface is implemented as an identity authentication interface, completing identity authentication on the identity authentication interface; when the task interface is implemented as a recharge interface, completing recharge operations on the recharge interface; and so on.

[0021] Continuing with the above background technology, in view of the technical problem that once the UI interface changes, such as the adjustment of the position of UI elements, the modification of attributes or the update of styles, the original test scripts are likely to require a large amount of modification and adjustment, thus increasing the test cost and reducing the test efficiency. In the embodiments of the present application, without relying on test scripts, the language model for UI automated testing directly automatically identifies the interactive controls to be tested according to the user's test requirements for the target page and the screenshot of the target page, and generates test instructions for the interactive controls to be tested. Through these test instructions, the terminal device where the target page is located is controlled to simulate user operations to perform interactive control on the interactive controls, meeting the user's test requirements. In this solution, the language model automatically identifies the controls to be tested based on the test requirements and page screenshots, rather than relying on test scripts. Therefore, it is not necessary to frequently modify the test scripts due to changes in the page layout, which can reduce the test cost and improve the test efficiency.

[0022] Further, in the case where some interactive controls are not marked or are not correctly marked in the XML document, in this solution, the language model automatically identifies the interactive controls to be tested according to the test requirements and page screenshots. If the test requirements require testing of the "interactive controls not marked or not correctly marked in the XML document", the language model can identify the "interactive controls not marked or not correctly marked in the XML document" based on the page screenshots according to the test requirements, without being affected by the quality of the XML document, which can further improve the success rate and quality of UI testing.

[0023] The following will describe in detail a solution provided by the embodiments of the present application with reference to the accompanying drawings.

[0024] Figure 1a It is a schematic structural diagram of the UI automated testing system provided by the exemplary embodiment of the present application. As Figure 1aAs described above, the system includes: an automated testing device 101, a first terminal 102, and a second terminal 103. The automated testing device 101 can establish communication connections with the first terminal 102 and the second terminal 103. Specifically, the automated testing device can execute the UI automated testing method as described in Figure 2.

[0025] The first terminal is a terminal on which the target application to be tested runs and displays the target page of the target application. The implementation form of the first terminal in the embodiments of the present application is not limited. For example, the first terminal can be a smart handheld device, such as a smartphone, a tablet computer, a laptop computer, or a desktop computer, etc.; or, the first terminal can also be a smart wearable device, such as a smart watch, a smart bracelet, etc.; or, the first terminal can also be various smart home appliances with display screens, such as a smart TV, a smart large screen, or a smart robot, etc. The implementation form of the target application in this embodiment is not limited. The implementation form of the target application can be an independently running APP or a small program that depends on the APP to run. The type of the target application in this embodiment is also not limited. For example, the target application can be an instant messaging application, a service application, a recruitment application, or a job application, etc.

[0026] The automated testing device can be regarded as the server corresponding to the Web page displayed by the above browser. The implementation form of this server in this embodiment is not limited. For example, it can be a physical server, a single cloud server, or a server array, etc.

[0027] The second terminal is the terminal used by the target user, and the target user can be a tester of the application. Similarly, the implementation form of the second terminal in this embodiment is not limited, and for specific details, please refer to the relevant description of the implementation form of the first terminal device, which will not be elaborated here. It should be noted that the first terminal and the second terminal can be the same type of terminal or different types of terminals, and this embodiment does not limit this. In addition, the implementation form and type of the application running on the second terminal are also not limited. The implementation form of the application can be an independently running APP or a small program that depends on the APP to run. The type of the application can be an instant messaging application, a service application, a recruitment application, or a job application, etc. Taking the application as a browser application as an example, the browser application can display the Web page provided by the automated testing device. Figure 1bIt is an exemplary schematic diagram of a Web page. A user can input automated test requirements on the Web page, generate an automated test task based on the automated test requirements, and upload the automated test task to an automated test device, so that the automated test device can perform automated testing on the target page of the target application running on the first terminal. For the specific testing process of the automated test device performing automated testing on the target page based on the automated test task, please refer to the description of the relevant embodiments in FIG. 2 below, which will not be elaborated here.

[0028] Figure 2a It is a schematic flowchart of a UI automated test method provided by an exemplary embodiment of the present application. Figure 2b It is a schematic flowchart of another UI automated test method provided by another exemplary embodiment of the present application. As Figure 2a and 2b shown, the method includes:

[0029] 201. Receive an automated test task submitted by a first terminal. The automated test task includes an identifier of a target application and test requirement information described in natural language. The test requirement information is used to require interactive testing of interactive controls in the target page of the target application.

[0030] 202. According to the automated test task, fill the test requirement information into a target prompt template to obtain a target model prompt. The target prompt template at least includes: a requirement information filling item, which is used to carry the test requirement information.

[0031] 203. According to the automated test task, send a screenshot instruction to a second terminal where the target application is located, so that the second terminal can take a screenshot of the target page of the target application and return the screenshot of the target page.

[0032] 204. Input the target model prompt and the screenshot of the target page into a language model based on artificial intelligence. Under the guidance of the target model prompt, identify the target interactive controls to be tested from the screenshot of the target page according to the test requirement information, and output the target control information required for automated testing of the target interactive controls.

[0033] 205. Control the second terminal to perform interactive operations on the target interactive controls on the target page according to the target control information, so as to complete the automated testing of the target interactive controls.

[0034] In some alternative embodiments, the automated testing device may send an access link to the test service page to the first terminal; in response to the triggering operation of the first terminal on the access link, the test service page is displayed on the first terminal. The test service page includes an application identification information item and a test requirement input box. The application identification information item is used for the test user to fill in the identification of the target application, and the test requirement input box is used to collect the test requirement information input by the test user; further, the first terminal may submit an automated testing task through the test service page. Correspondingly, receiving the automated testing task submitted by the first terminal includes: receiving the automated testing task submitted by the first terminal through the test service page.

[0035] In the embodiments of the present application, the identification of the target application included in the automated testing task may be a text identification with a unique identification function such as the name of the target application, or other types of identifications with a unique identification function, such as a digital identification, an identification combining numbers and letters, and so on. The test requirement information described in natural language may be a single test requirement information, or a composite test requirement information composed of multiple single test requirement information. For example, a single test requirement information may be "Please trigger the interaction control on the target page of the ** application", and a composite test requirement information may be "Search for the ** keyword in the search box (interaction control) of the ** instant messaging application, after the target page corresponding to the ** keyword is displayed, trigger the interaction control on the target page, and enter the ** mini program".

[0036] In the embodiments of the present application, there is a target prompt word template, and the target prompt word template at least includes: a requirement information filling item, and the requirement information filling item is used to carry the test requirement information. After receiving the automated testing task submitted by the first terminal, the target automated testing device fills the test requirement information into the target prompt word template to obtain a target model prompt word.

[0037] In the embodiments of the present application, the automated testing device may also send a screenshot instruction to the second terminal where the target application is located according to the automated testing task, so that the second terminal takes a screenshot of the target page of the target application and returns the screenshot of the target page.

[0038] In the embodiments of the present application, after obtaining the target model prompt and the screenshot of the target page, the target model prompt and the screenshot of the target page are input into an artificial intelligence-based language model. Under the guidance of the target model prompt, according to the test requirement information, the target interaction control to be tested is identified from the screenshot of the target page, and the target control information required for automated testing of the target interaction control is output. Without relying on test scripts, the language model for UI automated testing directly identifies the interaction controls to be tested according to the user's test requirements for the target page and the screenshot of the target page, generates test instructions for the interaction controls to be tested, and controls the terminal device where the target page is located to simulate user operations to perform interaction control on the interaction controls, meeting the user's test requirements.

[0039] Among them, the artificial intelligence-based language model can be a large language model (LLMs) that supports multi-modal, with cross-modal and cross-language deep semantic understanding and generation capabilities. Natural language processing can be used to understand the target model prompt and test requirement information. Under the guidance of the target model prompt, according to the test requirement information, the target interaction control to be tested is identified from the screenshot of the target page, and the target control information required for automated testing of the target interaction control is output. The large language model includes but is not limited to models of the GLM (Generalized Linear Model) series, models of the Qwen series, etc.

[0040] In some alternative embodiments, the target prompt template further includes: multiple interaction operations, the execution function corresponding to each interaction operation, and output result guiding information. The output result guiding information is used to guide the model to output text-based control information, and / or, to guide the model to output the execution function as control information. The control information is used to describe the interaction control that needs to be automatically tested and the interaction operation that needs to be performed on this interaction control. That is to say, under the guidance of the output result guiding information, the model can only output text-based control information, the model can also only output the execution function as control information, and the model can also output text-based control information and the execution function as control information. The multiple interaction operations can be, for example, click, swipe left, swipe right, double click, and so on. The text-based control information can be the name or identifier of the interaction control for automated testing, and the execution function refers to the function used to perform the corresponding interaction operation on the automated testing interaction space. An exemplary target prompt template is as follows:

[0041] "## Task background

[0042] You are an agent trained to perform specific tasks on a smartphone. You will be provided with screenshots of smartphone applications, where the interactive UI elements on the screenshots are labeled with numerical tags starting from 1. The task you need to complete is: <task_description>.

[0043] ## Available operation functions

[0044] Now, you need to combine all of the above and perform an operation on the current page, ensuring that the next operation does not repeat the previously executed operation. You must choose one of the following six actions:

[0045] 1. **tap(element: int)**

[0046] - Used to click on the UI element displayed on the smartphone screen.

[0047] - Example: `tap(5)` clicks on the UI element labeled with the number 5.

[0048] 2. **text(text_input: str)** - Used to insert text into an input field.

[0049] - Example: `text("Hello,world!")` inserts the string into the input area.

[0050] - Note: This function can only be called when the keyboard is visible. 3....

[0052] ## Output format

[0053] Your output should include the following two parts:

[0054] **Click copywriting**: If it is a click operation, output the copywriting corresponding to the click object.

[0055] **Operation**: Function call with the correct parameters, e.g., `tap(5)` or `FINISH` (if the task is completed or no operation is required). No other content can be output in this field except for the function call or FINISH.

[0056] Note You can only perform one operation at a time, so just call the function directly.

[0057] Among them, <task_description> is the requirement information filling item, and `tap(5)` is the execution function.

[0058] Accordingly, input the target model prompt and the screenshot of the target page into the artificial intelligence-based language model. Under the guidance of the target model prompt, identify the target interactive control to be tested from the screenshot of the target page according to the test requirement information, and output the target control information required for automated testing of the target interactive control, including: input the target model prompt and the screenshot of the target page into the artificial intelligence-based language model, and perform the following operations in the artificial intelligence-based language model: according to the test requirement information, identify the target interactive control to be tested and the target interactive operation to be performed on the target interactive control from the screenshot of the target page; according to the guidance of the output result guidance information, generate the target copy information for describing the execution of the target interactive operation on the target interactive control, and / or select the target execution function corresponding to the target interactive operation from the execution functions corresponding to each interactive operation; use the target copy information and / or the target execution function as the target control information, and output the target copy information and / or the target execution function.

[0059] In some alternative embodiments, the artificial intelligence-based language model includes: a requirement recognition network layer and a result generation network layer. Then input the target model prompt and the screenshot of the target page into the artificial intelligence-based language model. Under the guidance of the target model prompt, identify the target interactive control to be tested from the screenshot of the target page according to the test requirement information, and output the target control information required for automated testing of the target interactive control, including: input the target model prompt and the screenshot of the target page into the artificial intelligence-based language model, and perform the following operations in the artificial intelligence-based language model: input the target model prompt and the screenshot of the target page into the requirement recognition network layer, identify the test requirement information, output result guidance information, and each execution function included in the target model prompt, and according to the test requirement information, identify the target interactive control to be tested and the target interactive operation to be performed on the target interactive control from the screenshot of the target page; input the output result guidance information and each execution function into the result generation network layer, and according to the guidance of the output result guidance information, generate the target copy information for describing the execution of the target interactive operation on the target interactive control, and / or select the target execution function corresponding to the target interactive operation from the execution functions corresponding to each interactive operation; use the target copy information and / or the target execution function as the target control information, and output the target copy information and / or the target execution function.

[0060] In some optional embodiments, according to the test requirement information, the target interactive controls that need to be tested and the target interactive operations that need to be performed on the target interactive controls are identified from the screenshot of the target page, including: inputting the target model prompt words and the screenshot of the target page into the requirement identification network layer, identifying the test requirement information contained in the target model prompt words, and according to the test requirement information, identifying the target interactive controls that need to be tested and the target interactive operations that need to be performed on the target interactive controls from the screenshot of the target page. Further optionally, the target model prompt word and the screenshot of the target page are input into the requirement identification network layer, the test requirement information contained in the target model prompt word is identified, and according to the test requirement information, the target interactive control to be tested and the target interactive operation to be performed on the target interactive control are identified from the screenshot of the target page, including: semantically understanding the test requirement information to obtain the interactive control description information and the interactive operation description information in the test requirement information; identifying the graphic information of the screenshot of the target page to obtain multiple graphic elements existing on the target page; matching the graphic elements existing on the target page according to the interactive control description information, and taking the matching graphic elements as the target interactive control; and determining the target interactive operation to be performed on the target interactive control according to the interactive operation description information.

[0061] Further optionally, a resource file of a target page returned by the second terminal is received, wherein the resource file includes operational attributes of each page element in the target page and position information of the page element in the target page, and the operational attributes describe whether the corresponding page element supports interactive operations; based on the operational attributes of each page element in the resource file, the page elements in the target page that support interactive operations are determined as first interactive controls, and based on the position information of each first interactive control in the target page, a status identifier is added to each first interactive control in the screenshot of the target page. Among them, each page element in the target page is each interactive control. This embodiment does not limit the status identifier, which may be, for example, a number, a letter, or text. Taking the example of a number as the status identifier, an exemplary schematic diagram after adding a status identifier to each first interactive control in the screenshot of the target page is shown as follows. Figure 2c As shown. The resource file of the target page can be an XML file, but is not limited to this. Taking the page element "account" as an example, the content contained in its XML file is as follows:

[0062] “ <node index="1"text="账号,按钮,16之2"

[0063] resource-id=""class="android.view.View"

[0064] package="com.tencent.mm"content-desc=""

[0065] checkable = "false" checked = "false"

[0066] clickable = "false" enabled = "true"

[0067] focusable = "false" focused = "false"

[0068] scrollable = "false" long - clickable = "false"

[0069] password = "false" selected = "false"

[0070] bounds = "[245,214][418,349]" /

[0071] clickable = "false" focusable = "false”

[0072] Among them, clickable indicates whether the element can be touched, and focusable indicates whether the element can have a focus (i.e., touch positioning). When clickable or focusable = "true", it means that the element can support interactive operations. Bounds represents the coordinates of the element on the target page.

[0073] Correspondingly, the target model prompt and the screenshot of the target page are input into the artificial - intelligence - based language model, including: inputting the target model prompt and the screenshot of the target page with status identifiers added into the artificial - intelligence - based language model, so that the artificial - intelligence - based language model can identify interactive controls based on the status identifiers of each first interactive control. In this solution, the language model automatically identifies the controls to be tested according to the test requirements and the page screenshot, rather than relying on test scripts. Therefore, it is not necessary to frequently modify test scripts due to changes in page layout, which can reduce test costs and improve test efficiency.

[0074] In some alternative embodiments, according to the test requirement information, identifying the target interaction control to be tested and the target interaction operation to be performed on the target interaction control from the screenshot of the target page includes: identifying each first interaction control from the screenshot of the target page based on the status identifier included in the screenshot of the target page; identifying the graphic and text information of other areas of the screenshot of the target page except each first interaction control to obtain the remaining graphic and text elements existing on the target page; semantically understanding the test requirement information to obtain the interaction control description information and the interaction operation description information in the test requirement information; matching according to the interaction control description information among each first interaction control and the remaining graphic and text elements, and taking the first interaction control and / or graphic and text element in the match as the target interaction control; and determining the target interaction operation to be performed on the target interaction control according to the interaction operation description information. Using two methods to identify each page element in the target page can avoid the situation that due to incomplete or missing operation attribute information of at least some page elements in the resource file of the target page, the corresponding status identifier cannot be marked, resulting in the inability to identify the corresponding primary color and perform the corresponding interaction operation. That is to say, in the case where some interaction controls are not marked or are incorrectly marked in the XML document, this solution automatically identifies the interaction controls to be tested by the language model according to the test requirements and the page screenshot. If the test requirements require testing of "interaction controls not marked or incorrectly marked in the XML document", the language model can identify the "interaction controls not marked or incorrectly marked in the XML document" based on the page screenshot according to the test requirements, without being affected by the quality of the XML document, and can further improve the success rate and quality of UI testing.

[0075] Further optionally, the output result guiding information is used to guide the model to output copywriting-based control information and is used to guide the model to output an execution function as control information; then, controlling the second terminal to perform an interaction operation on the target interaction control on the target page according to the target control information to complete the automated test of the target interaction control includes: determining whether the target control information includes the target copywriting information and whether the target copywriting information is empty in the case of including the target copywriting information; if the judgment result is including and the target copywriting information is not empty, controlling the automated framework on the second terminal to perform an interaction operation on the target interaction control on the target page to complete the automated test of the target interaction control according to the target copywriting information; if the judgment result is not including or the target copywriting information is empty, controlling the automated framework on the second terminal to perform an interaction operation on the target interaction control on the target page to complete the automated test of the target interaction control according to the target execution function.

[0076] In an alternative embodiment, the automation framework on the second terminal is controlled according to the target copy information to perform interaction operations on the target interaction control on the target page, so as to complete the automation test of the target interaction control, including: sending the target copy information to the automation framework on the second terminal, so that the automation framework performs interaction operations on the target interaction control on the target page according to the target copy information, so as to complete the automation test of the target interaction control.

[0077] Further optionally, sending the target copy information to the automation framework on the second terminal, so that the automation framework performs interaction operations on the target interaction control on the target page according to the target copy information, so as to complete the automation test of the target interaction control, including: sending the target copy information to the automation framework on the second terminal, calling and running the preset function corresponding to the target copy information, so that the automation framework performs interaction operations on the target interaction control on the target page according to the target copy information, so as to complete the automation test of the target interaction control.

[0078] In another alternative embodiment, the automation framework on the second terminal is controlled according to the target execution function to perform interaction operations on the target interaction control on the target page, so as to complete the automation test of the target interaction control, including: calling the command line operation page, and running the target execution function in the command line operation page, so as to control the automation framework on the second terminal to perform interaction operations on the target interaction control on the target page, so as to complete the automation test of the target interaction control. Wherein, the command line can be an abd command line.

[0079] The technical solutions provided in the above embodiments of the present application do not rely on test scripts. The language model for UI automation testing directly automatically identifies the interaction controls to be tested based on the user's test requirements for the target page and the screenshot of the target page, and generates test instructions for the interaction controls to be tested. Through these test instructions, the terminal device where the target page is located is controlled to simulate user operations to perform interaction control on the interaction controls, meeting the user's test requirements. In this solution, the language model automatically identifies the controls to be tested according to the test requirements and page screenshots, rather than relying on test scripts. Therefore, it is not necessary to frequently modify the test scripts due to changes in the page layout, which can reduce the test cost and improve the test efficiency.

[0080] Furthermore, in the case where some interaction controls are not marked or are not correctly marked in the XML document, in this solution, the language model automatically identifies the interaction controls to be tested according to the test requirements and page screenshots. If the test requirements require testing of the "interaction controls not marked or not correctly marked in the XML document", the language model can identify the "interaction controls not marked or not correctly marked in the XML document" based on the page screenshot according to the test requirements, without being affected by the quality of the XML document, which can further improve the success rate and quality of UI testing.

[0081] Figure 3 A schematic structural diagram of an electronic device provided for an exemplary embodiment of the present application. As Figure 3 shown, the electronic device includes: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement the following steps:

[0082] Receive an automated test task submitted by a first terminal. The automated test task includes an identifier of a target application and test requirement information described in natural language. The test requirement information is used to require an interaction test to be performed on an interaction control in a target page of the target application; according to the automated test task, fill the test requirement information into a target prompt template to obtain a target model prompt. The target prompt template at least includes: a requirement information filling item, and the requirement information filling item is used to carry the test requirement information; according to the automated test task, send a screenshot instruction to a second terminal where the target application is located for the second terminal to take a screenshot of the target page of the target application and return the screenshot of the target page; input the target model prompt and the screenshot of the target page into a language model based on artificial intelligence, and under the guidance of the target model prompt, identify a target interaction control to be tested from the screenshot of the target page according to the test requirement information, and output target control information required for performing an automated test on the target interaction control; control the second terminal to perform an interaction operation on the target interaction control on the target page according to the target control information to complete the automated test on the target interaction control.

[0083] In some alternative embodiments, the target prompt template further includes: a variety of interaction operations, an execution function corresponding to each interaction operation, and output result guiding information. The output result guiding information is used to guide the model to output text-based control information and / or to guide the model to output the execution function as control information. The control information is used to describe the interaction control to be automatically tested and the interaction operation to be performed on the interaction control. Based on this, when the processor inputs the target model prompt and the screenshot of the target page into the artificial intelligence-based language model, and under the guidance of the target model prompt, identifies the target interaction control to be tested from the screenshot of the target page and outputs the target control information required for automatically testing the target interaction control, it specifically is used for: Inputting the target model prompt and the screenshot of the target page into the artificial intelligence-based language model, and performing the following operations in the artificial intelligence-based language model: According to the test requirement information, identifying the target interaction control to be tested and the target interaction operation to be performed on the target interaction control from the screenshot of the target page; Under the guidance of the output result guiding information, generating target text information for describing the performance of the target interaction operation on the target interaction control and / or selecting the target execution function corresponding to the target interaction operation from the execution functions corresponding to each interaction operation; Using the target text information and / or the target execution function as the target control information, and outputting the target text information and / or the target execution function.

[0084] In some alternative embodiments, when the processor identifies the target interaction control to be tested and the target interaction operation to be performed on the target interaction control from the screenshot of the target page according to the test requirement information, it specifically is used for: Semantically understanding the test requirement information to obtain the interaction control description information and the interaction operation description information in the test requirement information; Identifying the graphic and text information of the screenshot of the target page to obtain multiple graphic and text elements existing on the target page; Matching according to the interaction control description information among the graphic and text elements existing on the target page, and using the matched graphic and text elements as the target interaction control; And determining the target interaction operation to be performed on the target interaction control according to the interaction operation description information.

[0085] Further optionally, the processor is further used for receiving the resource file of the target page returned by the second terminal. The resource file includes the operation attributes of each page element in the target page and the position information in the target page. The operation attribute describes whether the corresponding page element supports interaction operations; According to the operation attributes of each page element in the resource file, determining the page elements that support interaction operations in the target page as the first interaction controls, and adding status identifiers to each first interaction control in the screenshot of the target page according to the position information of each first interaction control in the target page.

[0086] Accordingly, when the processor inputs the target model prompt and the screenshot of the target page into the artificial intelligence-based language model, it is also used to: input the target model prompt and the screenshot of the target page with the added status identifier into the artificial intelligence-based language model, so that the artificial intelligence-based language model can identify the interactive controls based on the status identifiers of each first interactive control.

[0087] Optionally, when the processor identifies the target interactive control to be tested and the target interactive operation to be performed on the target interactive control from the screenshot of the target page according to the test requirement information, it is specifically used to: identify each first interactive control from the screenshot of the target page based on the status identifier included in the screenshot of the target page; identify the graphic and text information of other areas except each first interactive control in the screenshot of the target page to obtain the remaining graphic and text elements existing on the target page; perform semantic understanding on the test requirement information to obtain the interactive control description information and the interactive operation description information in the test requirement information; match according to the interactive control description information among each first interactive control and the remaining graphic and text elements, and use the first interactive control and / or graphic and text element in the match as the target interactive control; and determine the target interactive operation to be performed on the target interactive control according to the interactive operation description information.

[0088] Further optionally, the output result guiding information is used to guide the model to output text-based control information and is used to guide the model to output an execution function as control information; then, when the processor controls the second terminal to perform an interactive operation on the target interactive control on the target page according to the target control information to complete the automated test of the target interactive control, it is specifically used to: determine whether the target control information contains the target text information and whether the target text information is empty in the case of containing the target text information; if the judgment result is containing and the target text information is not empty, control the automated framework on the second terminal to perform an interactive operation on the target interactive control on the target page according to the target text information to complete the automated test of the target interactive control; if the judgment result is not containing or the target text information is empty, control the automated framework on the second terminal to perform an interactive operation on the target interactive control on the target page according to the target execution function to complete the automated test of the target interactive control.

[0089] In an alternative embodiment, when the processor controls the automated framework on the second terminal to perform an interactive operation on the target interactive control on the target page according to the target text information to complete the automated test of the target interactive control, it is specifically used to: send the target text information to the automated framework on the second terminal, so that the automated framework performs an interactive operation on the target interactive control on the target page according to the target text information to complete the automated test of the target interactive control.

[0090] In another alternative embodiment, when the processor controls the automation framework on the second terminal to perform interaction operations on the target interaction control on the target page according to the target execution function to complete the automation test of the target interaction control, it is specifically configured to: call the command line operation page, and run the target execution function in the command line operation page to control the automation framework on the second terminal to perform interaction operations on the target interaction control on the target page to complete the automation test of the target interaction control.

[0091] Further optionally, the processor is further configured to send an access link of the test service page to the first terminal; in response to the trigger operation of the first terminal on the access link, display the test service page on the first terminal, where the test service page includes an application identification information item and a test requirement input box, the application identification information item is used for the test user to fill in the identification of the target application, and the test requirement input box is used to collect the test requirement information input by the test user; receive the automation test task submitted by the first terminal, including: receiving the automation test task submitted by the first terminal through the test service page.

[0092] The detailed implementation manners and beneficial effects of the modules in the embodiments of the present application have been described in detail in the foregoing embodiments, and will not be elaborated herein.

[0093] Further, as Figure 3 shown, the electronic device further includes: other components such as a communication component 30c, a display 30d, a power supply component 30e, an audio component 30f, etc. Figure 3 Only some components are schematically shown, which does not mean that the electronic device only includes Figure 3 the components shown. Additionally, Figure 3 the components within the dashed box are optional components, rather than essential components, and can be determined according to the product form of the electronic device. The electronic device in this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, or an IOT device, or can also be a server-side device such as a conventional server, a cloud server, or a server array. If the electronic device in this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, or a smart phone, it may include Figure 3 the components within the dashed box; if the electronic device in this embodiment is implemented as a server-side device such as a conventional server, a cloud server, or a server array, it may not include Figure 3 the components within the dashed box.

[0094] The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disc.

[0095] The above-mentioned communication component is configured to facilitate communication between the device where the communication component is located and other devices in a wired or wireless manner. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 6G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra wide band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0096] The above-mentioned display includes a screen, and the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions, but also detect the duration and pressure associated with the touch or swipe operations.

[0097] The above power supply component provides power for various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device where the power supply component is located.

[0098] The above audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC). When the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive external audio signals. The received audio signals can be further stored in a memory or transmitted via a communication component. In some embodiments, the audio component further includes a speaker for outputting audio signals.

[0099] Correspondingly, an exemplary embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to implement the steps in the above method.

[0100] Correspondingly, an exemplary embodiment of the present application further provides a computer program product. The computer program product includes computer programs / instructions, which, when executed by a processor, cause the processor to be able to implement the steps in the above method.

[0101] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk memories, compact disc read-only memories (CD-ROMs), optical memories, etc.) containing computer-usable program code.

[0102] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in one or more flows in the flowchart and / or one or more blocks in the block diagram.

[0103] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means that implement the function specified in one or more blocks of a flowchart and / or one or more blocks of a block diagram.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable apparatus provide steps for implementing the function specified in one or more blocks of a flowchart and / or one or more blocks of a block diagram.

[0105] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), an input / output interface, a network interface, and memory.

[0106] The memory may include non-permanent memory in the form of computer-readable media, random access memory (Random Access Memory, RAM) and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

[0107] Computer-readable media includes both permanent and non-permanent, removable and non-removable media implemented by any method or technology for storing information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change random access memory (Phase-change Random Access Memory, PRAM), static random access memory (SRAM), dynamic random access memory (Dynamic Random Access Memory, DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile discs (Digital Video Disc, DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0108] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, commodity or device comprising a series of elements not only includes those elements but also other elements not expressly listed, or elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the phrase "comprising an..." does not exclude the presence of additional identical elements in the process, method, commodity or device comprising the element.

[0109] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A UI automated testing method, characterized in that, Including: Receiving an automated test task submitted by a first terminal, where the automated test task includes an identifier of a target application and test requirement information described in natural language, and the test requirement information is used to request an interaction test on interaction controls in a target page of the target application; According to the automated test task, filling the test requirement information into a target prompt template to obtain a target model prompt. The target prompt template at least includes: a requirement information filling item, which is used to carry the test requirement information; According to the automated test task, sending a screenshot instruction to a second terminal where the target application is located, so that the second terminal takes a screenshot of the target page of the target application and returns the screenshot of the target page; Inputting the target model prompt and the screenshot of the target page into an artificial intelligence-based language model. Under the guidance of the target model prompt, identifying target interaction controls to be tested from the screenshot of the target page according to the test requirement information, and outputting target control information required for automated testing of the target interaction controls; Controlling the second terminal to perform interaction operations on the target interaction controls on the target page according to the target control information to complete the automated testing of the target interaction controls.

2. The method according to claim 1, wherein The target prompt template further includes: multiple interaction operations, an execution function corresponding to each interaction operation, and output result guiding information. The output result guiding information is used to guide the model to output text-based control information and / or to guide the model to output an execution function as control information. The control information is used to describe the interaction controls that need to be automatically tested and the interaction operations that need to be performed on the interaction controls; Inputting the target model prompt and the screenshot of the target page into an artificial intelligence-based language model. Under the guidance of the target model prompt, identifying target interaction controls to be tested from the screenshot of the target page according to the test requirement information, and outputting target control information required for automated testing of the target interaction controls, including: Inputting the target model prompt and the screenshot of the target page into an artificial intelligence-based language model, and performing the following operations in the artificial intelligence-based language model: According to the test requirement information, identifying target interaction controls to be tested from the screenshot of the target page and target interaction operations to be performed on the target interaction controls; Under the guidance of the output result guiding information, generating target text information for describing the execution of the target interaction operations on the target interaction controls and / or selecting a target execution function corresponding to the target interaction operation from the execution functions corresponding to each interaction operation; Taking the target text information and / or the target execution function as the target control information, and outputting the target text information and / or the target execution function.

3. The method according to claim 2, wherein According to the test requirement information, identifying target interaction controls to be tested from the screenshot of the target page and target interaction operations to be performed on the target interaction controls, including: Semantically understand the test requirement information to obtain the interactive control description information and interactive operation description information in the test requirement information; Identify the graphic and text information of the screenshot of the target page to obtain multiple graphic and text elements existing on the target page; Match according to the interactive control description information among the graphic and text elements existing on the target page, and use the matched graphic and text elements as the target interactive control; and Determine the target interactive operation to be performed on the target interactive control according to the interactive operation description information.

4. The method according to claim 2, wherein It further includes: Receive the resource file of the target page returned by the second terminal, where the resource file includes the operation attributes of each page element in the target page and the position information in the target page, and the operation attributes describe whether the corresponding page element supports interactive operations; Determine the page elements that support interactive operations in the target page as the first interactive controls according to the operation attributes of each page element in the resource file, and add status identifiers to each first interactive control in the screenshot of the target page according to the position information of each first interactive control in the target page; Correspondingly, input the target model prompt and the screenshot of the target page into the artificial intelligence-based language model, including: input the target model prompt and the screenshot of the target page after adding the status identifier into the artificial intelligence-based language model for the artificial intelligence-based language model to identify the interactive control based on the status identifier of each first interactive control.

5. The method according to claim 4, wherein Identify the target interactive control to be tested and the target interactive operation to be performed on the target interactive control from the screenshot of the target page according to the test requirement information, including: Identify each first interactive control from the screenshot of the target page based on the status identifier included in the screenshot of the target page; Identify the graphic and text information of other areas except each first interactive control in the screenshot of the target page to obtain the remaining graphic and text elements existing on the target page; Semantically understand the test requirement information to obtain the interactive control description information and interactive operation description information in the test requirement information; Match according to the interactive control description information among each first interactive control and the remaining graphic and text elements, and use the matched first interactive control and / or graphic and text elements as the target interactive control; and Determine the target interactive operation to be performed on the target interactive control according to the interactive operation description information.

6. The method according to any one of claims 2-5, characterized in that, The output result guiding information is used to guide the model to output text-based control information and is also used to guide the model to output an execution function as control information; Then, control the second terminal to perform interactive operations on the target interactive control on the target page according to the target control information to complete the automated test of the target interactive control, including: Judge whether the target control information contains the target copy information and whether the target copy information is empty in the case of containing the target copy information; If the judgment result is "included" and the target copywriting information is not empty, control the automation framework on the second terminal to perform interaction operations on the target interaction control on the target page according to the target copywriting information, so as to complete the automation test of the target interaction control; If the judgment result is "not included" or the target copywriting information is empty, control the automation framework on the second terminal to perform interaction operations on the target interaction control on the target page according to the target execution function, so as to complete the automation test of the target interaction control.

7. The method according to claim 6, wherein Controlling the automation framework on the second terminal to perform interaction operations on the target interaction control on the target page according to the target copywriting information, so as to complete the automation test of the target interaction control, includes: Sending the target copywriting information to the automation framework on the second terminal, so that the automation framework performs interaction operations on the target interaction control on the target page according to the target copywriting information, so as to complete the automation test of the target interaction control; Controlling the automation framework on the second terminal to perform interaction operations on the target interaction control on the target page according to the target execution function, so as to complete the automation test of the target interaction control, includes: Calling a command line operation page, and running the target execution function in the command line operation page, so as to control the automation framework on the second terminal to perform interaction operations on the target interaction control on the target page, so as to complete the automation test of the target interaction control.

8. The method according to any one of claims 1-5, 6 and 7, characterized in that, It further includes: Sending an access link of the test service page to the first terminal; Responding to the trigger operation of the first terminal on the access link, and displaying the test service page on the first terminal. The test service page includes an application identification information item and a test requirement input box. The application identification information item is used for the test user to fill in the identification of the target application, and the test requirement input box is used to collect the test requirement information input by the test user; Receiving the automation test task submitted by the first terminal, including: receiving the automation test task submitted by the first terminal through the test service page.

9. An electronic device, characterized in that, It includes: A memory and a processor; The memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement the steps in the method according to any one of claims 1-8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the processor is caused to implement the steps in the method according to any one of claims 1-8.

11. A computer program product, characterized in that, The computer program product includes computer programs / instructions. When the computer programs / instructions are executed by the processor, the processor is caused to be able to implement the steps in the method according to any one of claims 1-8.

Images