User interface automatic test method and related device

Abstract

The invention discloses a user interface automatic test method and a related device, and relates to the technical field of artificial intelligence, and the method comprises the steps: firstly obtaining test demand information input by a user, and a page control library corresponding to a to-be-tested mobile device; then an intention recognition model is called to analyze the test demand information based on the page control library to obtain user intention structure information, and the user intention structure information comprises control information and an execution action list; generating a corresponding test case according to the user intention structure information; finally, the test case is controlled to be executed, so that the user interface of the to-be-tested mobile equipment is automatically tested. According to the scheme, the test case is generated through the intention structure mechanism instead of direct generation of a large model, more intelligent intention recognition can be achieved, the accuracy and stability of generation of the test case can be comprehensively improved, and then the test efficiency and the test effect are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular, to a method and related device for automated testing of user interfaces. Background Art

[0002] With the rapid development of artificial intelligence technology, more and more testing tasks can be executed by machines, significantly improving the efficiency and accuracy of software testing.

[0003] Currently, in UI (User Interface) automated testing, the writing of test cases mostly relies on traditional programming languages and is manually written by programmers. This usually requires programmers to have certain programming skills and an in-depth understanding of the testing framework. This is not only time-consuming but also prone to human errors, resulting in low efficiency of UI automated testing and unable to guarantee the testing effect.

[0004] Therefore, how to provide a solution for automated testing of user interfaces to improve testing efficiency and testing effect has become a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0005] In view of the above problems, this application provides a method and related device for automated testing of user interfaces to achieve the accuracy and stability of automatically generating test cases, and further achieve the purpose of improving testing efficiency and testing effect. The specific solutions are as follows:

[0006] The first aspect of this application provides a method for automated testing of user interfaces, including:

[0007] Obtain the test requirement information input by the user and the page control library corresponding to the mobile device to be tested;

[0008] Call an intent recognition model to parse the test requirement information based on the page control library to obtain user intent structure information, where the user intent structure information includes control information and an execution action list; the intent recognition model is a trained large model with intent recognition capabilities;

[0009] Generate corresponding test cases according to the user intent structure information; at least one execution step is included in the test case, and each execution step is used to indicate performing an action on a control;

[0010] Control the execution of the test case to achieve automated testing of the user interface of the mobile device to be tested.

[0011] In a possible implementation, the calling of the intent recognition model to parse the test requirement information based on the page control library to obtain user intent structure information includes:

[0012] Input the test requirement information and the page control library into the intent recognition model to obtain an intent recognition result, which includes the position information of the control and a list of execution actions;

[0013] Use the page control library to supplement the control attribute information for the intent recognition result to obtain the user intent structure information.

[0014] In one possible implementation, the step of inputting the test requirement information and the page control library into the intent recognition model to obtain an intent recognition result includes:

[0015] Obtain a pre - constructed intent recognition prompt template; the intent recognition prompt template includes intent recognition task description information, a test requirement information filling slot, and a page control library filling slot; the intent recognition task description information includes length constraint information for the model output result;

[0016] Fill the test requirement information into the test requirement information filling slot and fill the page control library into the page control library filling slot to obtain an intent recognition prompt;

[0017] Input the intent recognition prompt into the intent recognition model to obtain an intent recognition result.

[0018] In one possible implementation, the step of generating corresponding test cases according to the user intent structure information includes:

[0019] Obtain the identifier corresponding to the control information in the user intent structure information;

[0020] Generate corresponding test cases according to the identifier and the list of execution actions in the user intent structure information.

[0021] In one possible implementation, the step of controlling the execution of the test cases includes:

[0022] Push the test cases to the local of the mobile device to be tested, and the local of the mobile device to be tested executes the test cases;

[0023] Or,

[0024] Remotely control the mobile device to be tested to execute the test cases.

[0025] In one possible implementation, if the test cases include multiple consecutive execution steps, then the step of controlling the execution of the test cases includes:

[0026] For the current execution step, before the current execution step, obtain the execution feedback information of the previous execution step of the current execution step;

[0027] After correcting and dynamically adjusting the current execution step according to the execution feedback information of the previous execution step, execute it;

[0028] After the current execution step is completed, obtain the execution feedback information of the current execution step and store it.

[0029] The second aspect of this application provides a user interface automation testing device, including:

[0030] An acquisition unit, configured to acquire the test requirement information input by the user and the page control library corresponding to the mobile device to be tested;

[0031] An intention recognition unit, configured to call an intention recognition model to parse the test requirement information based on the page control library to obtain user intention structure information, where the user intention structure information includes control information and an execution action list; the intention recognition model is a trained large model with intention recognition capabilities;

[0032] A test case generation unit, configured to generate corresponding test cases according to the user intention structure information; at least one execution step is included in the test case, and each execution step is used to indicate performing an action on a control;

[0033] A test case execution unit, configured to control the execution of the test case to implement user interface automation testing of the mobile device to be tested.

[0034] In a possible implementation, the intention recognition unit includes:

[0035] A model call unit, configured to input the test requirement information and the page control library into the intention recognition model to obtain an intention recognition result, where the intention recognition result includes the position information of the control and an execution action list;

[0036] A control information supplement unit, configured to supplement control attribute information to the intention recognition result by using the page control library to obtain the user intention structure information.

[0037] In a possible implementation, the model call unit is specifically configured to:

[0038] Obtain a pre-constructed intention recognition prompt template; the intention recognition prompt template includes intention recognition task description information, a test requirement information filling slot, and a page control library filling slot; the intention recognition task description information includes length constraint information on the model output result;

[0039] Fill the test requirement information into the test requirement information filling slot, and fill the page control library into the page control library filling slot to obtain an intent recognition prompt;

[0040] Input the intent recognition prompt into the intent recognition model to obtain an intent recognition result.

[0041] In a possible implementation, the test case generation unit is specifically configured to:

[0042] Obtain an identifier corresponding to the control information in the user intent structure information;

[0043] Generate corresponding test cases according to the identifier and the execution action list in the user intent structure information.

[0044] In a possible implementation, the test case execution unit is specifically configured to:

[0045] Push the test cases to the local of the mobile device to be tested, and the local of the mobile device to be tested executes the test cases;

[0046] Or,

[0047] Remotely control the mobile device to be tested to execute the test cases.

[0048] In a possible implementation, if the test cases include multiple consecutive execution steps, the test case execution unit is specifically configured to:

[0049] For the current execution step, before the current execution step, obtain the execution feedback information of the previous execution step of the current execution step;

[0050] Correct and dynamically adjust the current execution step according to the execution feedback information of the previous execution step and then execute it;

[0051] After the current execution step is executed, obtain the execution feedback information of the current execution step and store it.

[0052] A third aspect of the present application provides a computer program product, including computer-readable instructions, which when running on an electronic device, enable the electronic device to implement the user interface automation test method according to the first aspect or any implementation manner of the first aspect.

[0053] A fourth aspect of the present application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0054] The memory is used to store computer programs;

[0055] The processor is used to execute the computer program, so that the electronic device can implement the user interface automation testing method of the first aspect or any implementation manner of the first aspect as described above.

[0056] A fifth aspect of the present application provides a computer-readable storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the user interface automation testing method of the first aspect or any implementation manner of the first aspect as described above.

[0057] By means of the above technical solutions, a user interface automation testing method and related devices provided by the present application first obtain the test requirement information input by the user and the page control library corresponding to the mobile device to be tested; then call the intent recognition model to parse the test requirement information based on the page control library to obtain the user intent structure information, and the user intent structure information includes control information and an execution action list; the intent recognition model is a trained large model with intent recognition ability; then, according to the user intent structure information, corresponding test cases are generated; each test case includes at least one execution step, and each execution step is used to indicate an action to be performed on a control; finally, control is performed to execute the test cases to implement the user interface automation testing of the mobile device to be tested. In this solution, by generating test cases through the intent structure mechanism instead of directly generating them by the large model, more intelligent intent recognition can be achieved, and thus the accuracy and stability of generating test cases can be comprehensively improved, and further the test efficiency and test effect can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Combined with the drawings and referring to the following specific embodiments, the above and other features, advantages and aspects of the embodiments of the present disclosure will become more obvious. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic and the original components and elements are not necessarily drawn to scale.

[0059] Figure 1 It is a schematic structural diagram of a user interface automation testing system provided by an embodiment of the present application;

[0060] Figure 2 It is a schematic diagram of a test interface provided by an embodiment of the present application;

[0061] Figure 3 It is a schematic flowchart of a user interface automation testing method provided by an embodiment of the present application;

[0062] Figure 4 It is a schematic structural diagram of a user interface automation testing device provided by an embodiment of the present application;

[0063] Figure 5A schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0064] The embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. The terms used in the implementation manners part of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0065] The embodiments of the present application will be described below with reference to the accompanying drawings. Those skilled in the art know that with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0066] The terms "first", "second", etc. in the specification, claims and above-mentioned drawings of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be interchanged under appropriate circumstances, which is only a way of distinguishing objects with the same attributes when describing the embodiments of the present application. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, so that a process, method, system, product or device comprising a series of units does not have to be limited to those units, but may include other units not clearly listed or inherent to these processes, methods, products or devices.

[0067] With the rapid development of artificial intelligence technology, more and more testing tasks can be performed by machines, significantly improving the efficiency and accuracy of software testing.

[0068] Currently, in UI (User Interface) automated testing, the writing of test cases mostly relies on traditional programming languages and is manually written by programmers. This usually requires programmers to have certain programming skills and an in-depth understanding of the test framework, which is not only time-consuming but also prone to human errors, resulting in low efficiency of UI automated testing and the inability to guarantee the testing effect.

[0069] With the gradual maturity of artificial intelligence technology, a UI automated test case generation solution based on natural language processing has begun to be applied to UI automated testing. The test case generation solution based on natural language processing can automatically generate corresponding UI automated test cases by parsing the natural language description of users. The core of this method lies in understanding the user's intention and converting it into executable test steps, so as to achieve efficient and accurate writing of UI automated test cases.

[0070] Currently, there are mainly two types of UI automated test case generation solutions based on natural language processing, which are specifically as follows:

[0071] The first type: NLP (Natural Language Processing) semantic template matching technology to generate UI automated test cases

[0072] This generation scheme mainly determines the user's writing intention through keyword hit association + traditional NLP semantic template matching. For example, when the user clicks on the input field or enters the text "click", the NLP semantic template "click in the ××× direction" is automatically associated. The user selects the template that best suits their intention, fills in the specific text value according to the template, and then submits it to the background for generation.

[0073] The second type: Generate UI automated test cases based on the large model intelligent agent Agent

[0074] This generation scheme mainly constructs a large model intelligent agent Agent. Specifically, a programming framework of a large language model (LLM) such as langchain is generally selected to connect the implementations of the UI engine layer, semantic understanding layer, database layer, etc. The large model identifies the user's semantics intention, and then through device parsing, brings the page information and user intention to the large model, allowing the large model to generate executable action codes or directly select action code segments for execution, ultimately realizing the implementation of the user's intention.

[0075] However, the NLP semantic template matching technology is too dependent on predefined templates and cannot flexibly meet the diverse needs of users, easily leading to poor user experience. When the large model intelligent agent Agent identifies intentions, due to the limitations of the large model itself, the generated test cases are unstable in execution in specific scenarios and are difficult to adapt to the rapidly changing UI environment. In addition, these technologies also have deficiencies in the accuracy and maintainability of the generated test cases, restricting their wide application.

[0076] It can be seen that the existing UI automated test case generation schemes based on natural language processing do not generate test cases well, which still leads to low efficiency of UI automated testing and the inability to guarantee the testing effect.

[0077] To solve the above problems, the embodiments of the present application provide a user interface automated testing system. The user interface automated testing system of the embodiments of the present application will be introduced in detail below with reference to the accompanying drawings.

[0078] Refer to Figure 1 , Figure 1 is a schematic structural diagram of a user interface automated testing system provided by the embodiments of the present application, as Figure 1As shown in the figure, the user interface automation test system provided by the embodiment of the present application includes a user interface automation test terminal 11 and a user interface automation test server 12. A test interface is configured on the user interface automation test terminal, and the test interface can be in the form of a web page. The user interface automation test terminal can be a device such as a user's personal computer (PC) or a laptop computer.

[0079] If it is necessary to test the user interface of the mobile device 13 to be tested, the mobile device 13 to be tested can be connected to the user interface automation test terminal 11. The connection method can be a wired connection method (such as connecting through a USB interface or a Type-C interface, etc.), or a wireless connection method (such as connecting through wifi or Bluetooth, etc.). The mobile device to be tested can be a device with a user interface such as a mobile phone, a tablet computer, or a learning machine.

[0080] After connecting the mobile device 13 to be tested with the user interface automation test terminal 11, the user can start the local Agent service installed on the user interface automation test terminal. The local Agent service can search for local mobile devices. When the user selects to load the mobile device to be tested, the mobile device to be tested is forwarded through the adb (Android Debug Bridge, translated as Android debugging bridge) port, and at the same time, the user interface of the mobile device to be tested is synchronized to the user interface automation test server through the websocket protocol and rendered and displayed on the test interface. When the user opens the test interface, the user interface of the mobile device to be tested can be seen in real time on the test interface. The test interface can also include a natural language input field, where the user can input test requirement information in the form of natural language (such as "click on the camera"), as Figure 2 shown. The local Agent service can also send the test requirement information input by the user to the user interface automation test server 12.

[0081] In addition, the local Agent service can also parse the current user interface of the mobile device to be tested, generate a page control library corresponding to the mobile device to be tested, and provide the page control library to the user interface automation test server.

[0082] The local Agent service can generate the page control library by parsing the current user interface of the mobile device to be tested, including but not limited to at least one of the following three methods:

[0083] Method 1: hierarchy tree generation method

[0084] Obtain the control hierarchy information of the device's current user interface through adb (Android Debug Bridge) or the accessibility service, and then perform cropping through a refined algorithm. Delete useless control information such as clearly non-clickable and clearly disabled controls, or delete control attributes unnecessary for judging the control type. After completing the cropping, store the controls in the json data format.

[0085] Method 2: Page OCR (Optical Character Recognition) generation method

[0086] After taking a screenshot of the current user interface through an open-source OCR recognition service, perform OCR recognition, and store the text-type controls in the form of recognized text and corresponding coordinates in the json data format.

[0087] Method 3: Model generation method

[0088] By implementing the interception of the user interface materials used by the application, such as images of various control icons, and training and reasoning through open-source recognition models such as the yolo series, a control model library for the application is formed. Subsequently, when in use, take a screenshot of the current user interface and then hand it over to the recognition model, and store it in the json data format in the form of recognized icons and corresponding coordinates.

[0089] It should be noted that the above three methods for generating the page control library are complementary to each other due to the advantages and disadvantages of their respective technical solutions, and can maximize the scope and accuracy of control recognition.

[0090] After receiving the test requirement information and the page control library input by the user, the user interface automation test server 12 can execute the user interface automation test method of this application.

[0091] In a possible implementation, the user interface automation test server 12 can deploy an intent recognition engine, a test case generation engine, and a test case execution engine, the intent recognition engine. After obtaining the test requirement information input by the user, the intent recognition engine can call the intent recognition model to parse the test requirement information based on the page control library to obtain user intent structure information, and the user intent structure information includes control information and an execution action list; the intent recognition model is a trained large model with intent recognition capabilities; the test case generation engine can generate corresponding test cases according to the user intent structure information; at least one execution step is included in the test case, and each execution step is used to instruct to perform an action on a control; the test case execution engine can control the execution of the test case.

[0092] Based on the above, an embodiment of the present application provides a user interface automation testing method, and the execution subject of this method is a user interface automation testing server. The user interface automation testing method of the embodiment of the present application will be introduced in detail below with reference to the accompanying drawings.

[0093] Refer to Figure 3 , Figure 3 FIG. Figure 3 As shown in the flowchart of a user interface automation testing method provided by an embodiment of the present application, the user interface automation testing method provided by an embodiment of the present application may include the following steps, and these steps will be described in detail below.

[0094] S101: Obtain the test requirement information input by the user and the page control library corresponding to the mobile device to be tested;

[0095] In the present application, the test requirement information input by the user is used to indicate operations on the user interface of the mobile device to be tested. The test requirement information input by the user may be in text format. When inputting the test requirement information, the user may use the text input method or the voice input method. If the user uses the voice input method, the voice input by the user needs to be recognized to obtain the test requirement information in text format.

[0096] For ease of understanding, an example of the test requirement information input by the user is given in the present application as follows:

[0097] "Open the camera";

[0098] Or,

[0099] "First click the back button, and if the song has been favorited, then enter the following text in the comment area: A very nice song".

[0100] The page control library corresponding to the user interface to be tested may include information of all or part of the controls in the user interface to be tested. The page control library may be changed based on the test requirement information input by the user. In this regard, the present application does not make any limitations.

[0101] In a possible implementation, the page control library corresponding to the mobile device to be tested may be generated by the local Agent service installed on the user interface automation testing terminal parsing the current user interface of the mobile device to be tested. The local Agent service installed on the user interface automation testing terminal may provide the page control library to the user interface automation testing server.

[0102] S102: Invoke the intent recognition model to parse the test requirement information based on the page control library, and obtain user intent structure information, where the user intent structure information includes control information and an execution action list; the intent recognition model is a trained large model with intent recognition capabilities;

[0103] In this application, the intent recognition model can adopt large models of any structure, and this application does not make any limitations in this regard.

[0104] For ease of understanding, an example of user intent structure information is given in this application as follows:

[0105] Control information 1, execution action 1;

[0106] Or,

[0107] Control information 2, execution action 1, control information 2, execution action 2, control information 3, execution action 3.

[0108] S103: Generate corresponding test cases according to the user intent structure information; at least one execution step is included in the test cases, and each execution step is used to indicate performing an action on a control;

[0109] S104: Control the execution of the test cases to implement automated testing of the user interface of the mobile device to be tested.

[0110] In this application, controlling the execution of the test cases can achieve automated testing of the user interface of the mobile device to be tested.

[0111] As a possible implementation, the user interface automated testing server can push the test cases to the local of the mobile device to be tested, and the local of the mobile device to be tested executes the test cases. Considering that pushing the test cases to the mobile device to be tested for execution, the response time consumption and the preparation time consumption of the local execution environment of the mobile device to be tested are unacceptable at the interaction layer.

[0112] Therefore, in this application, as another possible implementation, the user interface automated testing server can remotely control the mobile device to be tested to execute the test cases. Specifically, the user interface automated testing server can send a control instruction to the mobile device to be tested, and the control instruction is used to control the mobile device to be tested to execute the test cases.

[0113] In a possible implementation, the control instruction may be an adb instruction. The control instruction can be sent to the local Agent service on the user interface automation test terminal. The local Agent service can use the adb protocol to execute the execution steps in the test case on the mobile device to be tested, such as clicking the camera button of the mobile device to be tested. The process of clicking the camera button will be displayed in real time on the test interface (the mobile device to be tested is screen-cast in real time on the test interface).

[0114] A user interface automation test method provided in this embodiment first obtains the test requirement information input by the user and the page control library corresponding to the mobile device to be tested; then calls the intent recognition model to parse the test requirement information based on the page control library to obtain the user intent structure information, and the user intent structure information includes control information and an execution action list; the intent recognition model is a trained large model with intent recognition ability; then according to the user intent structure information, a corresponding test case is generated; the test case includes at least one execution step, and each execution step is used to instruct to perform an action on a control; finally, the test case is controlled to be executed to implement the user interface automation test of the mobile device to be tested. In this solution, by generating test cases through the intent structure mechanism instead of directly generating them by the large model, more intelligent intent recognition can be achieved, thereby comprehensively improving the accuracy and stability of generating test cases, and further improving the test efficiency and test effect.

[0115] In another embodiment of the present application, a specific implementation manner of calling the intent recognition model to parse the test requirement information based on the page control library to obtain the user intent structure information is described, and this manner may include the following steps:

[0116] S201: Input the test requirement information and the page control library into the intent recognition model to obtain an intent recognition result, and the intent recognition result includes the position information of the control and an execution action list;

[0117] In the field of large models, prompt templatization is mainly used to improve the understanding of prompts by large models and is conducive to maintenance. Prompt templatization mainly divides the prompt into different blocks according to different functions, especially the variable parts, and uses matching templates to fill in, and the content can be dynamically replaced at any time. Therefore, in the present application, an intent recognition prompt template can be pre-constructed to frame the intent recognition task and dynamic replacement content of the large model.

[0118] Additionally, considering that technologies based on large models in the market often have long outputs, where the large model outputs almost all the required information, this brings technical convenience but also increases the response time and token costs. To improve the response time and save token costs, the length of the model output result can be restricted. Therefore, in this application, when constructing the intent recognition prompt template, the length of the model output result can be restricted.

[0119] In a possible implementation, the intent recognition prompt template includes intent recognition task description information, a test requirement information filling slot, and a page control library filling slot; the intent recognition task description information contains information on restricting the length of the model output result; then, inputting the test requirement information and the page control library into the intent recognition model to obtain the intent recognition result includes: obtaining the pre-constructed intent recognition prompt template; filling the test requirement information into the test requirement information filling slot, and filling the page control library into the page control library filling slot to obtain the intent recognition prompt; inputting the intent recognition prompt into the intent recognition model to obtain the intent recognition result.

[0120] For ease of understanding, in the embodiments of this application, an example of the output result of the intent recognition model is given as follows:

[0121] Assume that the test requirement information input by the user is: "Enter the text: 18854275569 in the search box"

[0122] The output result of the intent recognition model can be: write_on("[1235,997]","18854275569").

[0123] S202: Use the page control library to supplement control attribute information to the intent recognition result to obtain the user intent structure information.

[0124] In this application, due to the adoption of the short output scheme of the large model, the intent recognition result output by the large model only contains the position information of the control and the execution action list, and does not contain attribute information such as the name, description text, and list position of the control. However, in the actual user intent, there may be operations such as judging or extracting the above attribute information. Therefore, it is necessary to perform a reverse lookup to supplement the control attribute information. The specific implementation method of using the page control library to supplement control attribute information to the intent recognition result can be to match the position information of each control in the page control library with the control position information in the intent recognition result, find the corresponding control in the page control library, and then fill the attribute information of the corresponding control in the page control library into the intent recognition result.

[0125] In another embodiment of the present application, a specific implementation manner of generating corresponding test cases according to the user intention structure information is described. This manner may include the following steps:

[0126] S301: Obtain an identifier corresponding to the control information in the user intention structure information;

[0127] Although the control information in the user intention structure information can be directly assembled into test cases, considering the later case maintenance cost, for example, if 100 cases all use the same control, then when the style or functionality of the control changes, without control reuse, it is necessary to modify each place in these 100 cases where the control is involved, and the overall maintenance cost is very high. Therefore, in the present application, a reference relationship mapping of an identifier (ID) needs to be made for each control information. In this way, when the control changes, it only needs to be modified once. Since it is a reference relationship, other cases will also automatically execute using the modified result. Therefore, when generating corresponding test cases according to the user intention structure information, a reference relationship mapping of an ID needs to be made for the control information in the user intention structure information.

[0128] S302: Generate corresponding test cases according to the identifier and the execution action list in the user intention structure information.

[0129] For ease of understanding, an example of a test case is given in the present application, as follows:

[0130] Test case 1: Execute step 1, control ID1, execution action 1;

[0131] Or, test case 2:

[0132] Execute step 1: control ID2, execution action 1;

[0133] Execute step 2: control ID2, execution action 2;

[0134] Execute step 2: control ID3, execution action 3.

[0135] The execution steps in the test case may be single-step or a continuous segment. If it is single-step, only this single step needs to be executed. However, if it is a continuous execution step, it cannot be simply executed continuously. The problem is that in the single-step case, the corresponding execution page is static and the operated control is clear. Under continuous execution steps, the page after the first step execution is often uncontrollable due to different execution results, execution-time data (such as different accounts or dates, etc.). It is necessary to dynamically adjust the next execution step in combination with the context to ensure the accuracy of subsequent actions.

[0136] Therefore, in another embodiment of the present application, a specific implementation method for controlling the execution of the test case when the test case includes multiple consecutive execution steps is described. This method may include the following steps:

[0137] S401: For the current execution step, before the current execution step, obtain the execution feedback information of the previous execution step of the current execution step;

[0138] In the present application, the execution feedback information includes the execution result and the corresponding operation page.

[0139] S402: Correct and dynamically adjust the current execution step according to the execution feedback information of the previous execution step, and then execute it;

[0140] In the present application, correcting and dynamically adjusting the execution step includes the following aspects:

[0141] Error detection and handling: When a certain execution step fails to execute as expected (such as the target element is not found), the use case execution engine should have the ability to identify errors and take corresponding measures, such as re-trying to execute, finding an alternative path, or rolling back to the previous state.

[0142] Dynamic path planning: For complex multi-step tasks, the execution engine dynamically adjusts the subsequent steps based on the actual state of the current page to adapt to possible changes.

[0143] User interaction support: In some cases, further instructions from the user may be required to determine the correct operation direction. At this time, the use case execution engine pauses the execution process, provides a feedback request to the user, and continues to execute according to the user's input.

[0144] S403: After the current execution step is executed, obtain the execution feedback information of the current execution step and store it.

[0145] Specifically, the execution result of the current execution step and the corresponding operation page can be stored.

[0146] It should be noted that in the present application, the entire process of controlling the execution of the test case is a loop iteration process until all execution steps are executed. In this process, the use case execution engine continuously monitors the execution progress to ensure that each execution step can be completed accurately. If an unsolvable problem is encountered, an abnormal situation is reported and the execution is stopped so that the user can intervene in time. In the present application, the use case generated by real-time feedback during the execution process can be optimized, thereby reducing the use case maintenance cost and further improving the test efficiency.

[0147] In summary, the advantages of the technical solution of this application are that it significantly improves the usability of user interface automation testing, enabling non-technical personnel to easily achieve high-quality UI automation testing.

[0148] The above introduced a user interface automation testing method provided by an embodiment of this application. Next, a device for executing the above user interface automation testing method will be introduced.

[0149] Please refer to Figure 4 , Figure 4 which is a schematic structural diagram of a user interface automation testing device provided by an embodiment of this application. As Figure 4 shown, the user interface automation testing device includes:

[0150] An acquisition unit 101, configured to acquire test requirement information input by a user, and a page control library corresponding to a mobile device to be tested;

[0151] An intent recognition unit 102, configured to call an intent recognition model to parse the test requirement information based on the page control library, and obtain user intent structure information, where the user intent structure information includes control information and an execution action list; the intent recognition model is a trained large model with intent recognition capabilities;

[0152] A test case generation unit 103, configured to generate corresponding test cases according to the user intent structure information; at least one execution step is included in the test case, and each execution step is used to indicate performing an action on a control;

[0153] A test case execution unit 104, configured to control the execution of the test cases to implement user interface automation testing of the mobile device to be tested.

[0154] In a possible implementation, the intent recognition unit includes:

[0155] A model call unit, configured to input the test requirement information and the page control library into the intent recognition model, and obtain an intent recognition result, where the intent recognition result includes the position information of the control and an execution action list;

[0156] A control information supplement unit, configured to supplement control attribute information to the intent recognition result by using the page control library, and obtain the user intent structure information.

[0157] In a possible implementation, the model call unit is specifically configured to:

[0158] Obtain a pre-built intent recognition prompt template; the intent recognition prompt template includes intent recognition task description information, a test requirement information filling slot, and a page control library filling slot; the intent recognition task description information contains length constraint information for the model output result;

[0159] Fill the test requirement information into the test requirement information filling slot, and fill the page control library into the page control library filling slot to obtain an intent recognition prompt;

[0160] Input the intent recognition prompt into the intent recognition model to obtain an intent recognition result.

[0161] In a possible implementation, the test case generation unit is specifically configured to:

[0162] Obtain the identifier corresponding to the control information in the user intent structure information;

[0163] Generate corresponding test cases according to the identifier and the execution action list in the user intent structure information.

[0164] In a possible implementation, the test case execution unit is specifically configured to:

[0165] Push the test case to the local area of the mobile device to be tested, and the local area of the mobile device to be tested executes the test case;

[0166] Or,

[0167] Remotely control the mobile device to be tested to execute the test case.

[0168] In a possible implementation, if the test case includes multiple consecutive execution steps, the test case execution unit is specifically configured to:

[0169] For the current execution step, before the current execution step, obtain the execution feedback information of the previous execution step of the current execution step;

[0170] Correct and dynamically adjust the current execution step according to the execution feedback information of the previous execution step and then execute it;

[0171] After the current execution step is executed, obtain the execution feedback information of the current execution step and store it.

[0172] An electronic device is also provided in an embodiment of the present application. Refer to Figure 5As shown, it shows a schematic structural diagram of an electronic device suitable for implementing the electronic device in the embodiments of the present application. The electronic device in the embodiments of the present application may include, but is not limited to, fixed terminals such as mobile phones, laptop computers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), desktop computers, and the like. Figure 5 The electronic device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0173] As Figure 5 shown, the electronic device may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage device 608 into the random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. The input / output (I / O) interface 605 is also connected to the bus 604.

[0174] Generally, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 can allow the electronic device to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 5 it shows an electronic device with various devices, it should be understood that it is not required to implement or have all the shown devices. Instead, more or fewer devices may be implemented or had.

[0175] The embodiments of the present application also provide a computer program product including computer-readable instructions. When the computer-readable instructions run on an electronic device, the electronic device is enabled to implement any one of the user interface automation testing methods provided by the embodiments of the present application.

[0176] The embodiments of the present application also provide a computer-readable storage medium. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can be enabled to implement any one of the user interface automation testing methods provided by the embodiments of the present application.

[0177] It should be further noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the attached drawings of the device embodiments provided in this application, the connection relationships between the modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines.

[0178] Through the description of the above embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures for implementing the same function can also be various, such as analog circuits, digital circuits or dedicated circuits. However, for this application, in more cases, software program implementation is a better implementation method. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disc of a computer, and includes several instructions to enable a computer device (which can be a personal computer, training device, or network device, etc.) to execute the methods described in various embodiments of this application.

[0179] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product.

[0180] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be stored by a computer or a data storage device such as a training device or a data center that includes one or more integrated available media. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

Claims

1. A user interface automated testing method, characterized in that: include: Obtain the test requirement information input by the user and the page control library corresponding to the mobile device to be tested; The intention recognition model is called to parse the test requirement information based on the page control library to obtain user intention structure information, wherein the user intention structure information includes control information and an execution action list; the intention recognition model is a large model with intent recognition capability after training; Generate corresponding test cases according to the user intention structure information; The test case includes at least one execution step, each execution step is used to indicate the execution of an action on a control; The test case is controlled to be executed to realize the automatic test of the user interface of the mobile device to be tested.

2. The method according to claim 1, characterized in that The call intention recognition model parses the test requirement information based on the page control library to obtain user intention structure information, including: Input the test requirement information and the page control library into the intention recognition model to obtain an intention recognition result, wherein the intention recognition result includes the location information of the control and a list of execution actions; The page control library is used to supplement the control attribute information of the intention recognition result to obtain the user intention structure information.

3. The method according to claim 1, characterized in that The step of inputting the test requirement information and the page control library into the intent recognition model to obtain an intent recognition result includes: Obtain a pre-built intent recognition prompt template; the intent recognition prompt template includes intent recognition task description information, a test requirement information filling slot, and a page control library filling slot; the intent recognition task description information includes length constraint information for model output results; Fill the test requirement information into the test requirement information filling slot, and fill the page control library into the page control library filling slot to obtain an intent recognition prompt; The intention recognition prompt is input into the intention recognition model to obtain an intention recognition result.

4. The method according to claim 1, characterized in that: The generating corresponding test cases according to the user intention structure information includes: Obtaining an identifier corresponding to the control information in the user intention structure information; Generate a corresponding test case based on the identifier and the execution action list in the user intention structure information.

5. The method according to claim 1, characterized in that: The controlling and executing the test case comprises: Pushing the test case to the local mobile device to be tested, and executing the test case locally on the local mobile device to be tested; or, The mobile device to be tested is remotely controlled to execute the test case.

6. The method according to claim 1, characterized in that If the test case includes a plurality of consecutive execution steps, the control executes the test case, including: For a current execution step, before the current execution step, obtaining execution feedback information of a previous execution step of the current execution step; Correcting and dynamically adjusting the current execution step according to the execution feedback information of the previous execution step before executing it; After the current execution step is completed, execution feedback information of the current execution step is obtained and stored.

7. A user interface automated testing device, characterized in that: include: An acquisition unit, used to acquire the test requirement information input by the user and the page control library corresponding to the mobile device to be tested; An intention recognition unit is used to call an intention recognition model to parse the test requirement information based on the page control library to obtain user intention structure information, wherein the user intention structure information includes control information and an execution action list; the intention recognition model is a large model with intent recognition capability after training; A use case generation unit, used to generate corresponding test cases according to the user intention structure information; The test case includes at least one execution step, each execution step is used to indicate the execution of an action on a control; The test case execution unit is used to control the execution of the test case to realize the automatic test of the user interface of the mobile device to be tested.

8. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the user interface automation testing method as claimed in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the user interface automation testing method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the user interface automation testing method as described in any one of claims 1 to 6.

Images