Method and device for testing graphical user interface and electronic equipment

Abstract

The invention provides a method for testing a graphical user interface, and relates to the technical field of software testing, in particular to the technical field of testing of user graphical interfaces and artificial intelligence. According to the specific implementation scheme, in response to received database content synchronization requests sent by multiple pieces of target equipment, equipment state information, contained in the database content synchronization requests, of the target equipment is obtained; based on the device state information of the multiple target devices, database content data corresponding to the database content synchronization request is split into multiple data blocks corresponding to the multiple target devices; a plurality of data block synchronization tasks are generated based on the plurality of data blocks, the plurality of data block synchronization tasks are allocated to the plurality of target devices, and any one of the plurality of target devices is configured to respond to the received data block synchronization task sent by the server. And executing the data block synchronization task to synchronize the corresponding data block to a local target device.

Description

Technical Field

[0001] The present disclosure relates to the field of software testing technology, and in particular to the field of user graphical interface testing and artificial intelligence technology. Background Art

[0002] The graphical user interface (GUI) of an application plays an important role in software usage, directly affecting the realization of software functions and user experience. When an application is updated, the GUI usually changes, and these changes may affect the normal operation of existing functions. Therefore, in order to ensure that the new version does not have a negative impact on the original functions, GUI testing is very necessary.

[0003] Currently, manual testing is mainly used. Manual testing requires manual operations in frequent testing scenarios, which is inefficient and prone to human errors, resulting in inaccurate test results. Summary of the invention

[0004] The present disclosure provides a database content synchronization method, device, and electronic device for solving at least one of the above technical problems.

[0005] According to one aspect of the present disclosure, there is provided 1. a method for testing a graphical user interface, wherein the method comprises:

[0006] Tailoring the target test case to obtain the tailored target test case;

[0007] Generate test actions based on the graphical user interface to be tested and the tailored target test case;

[0008] The test action is executed to automatically test the graphical user interface to be tested to obtain a test result.

[0009] According to one aspect of the present disclosure, a graphical user interface testing device is provided, wherein the device comprises:

[0010] A cutting module, used for cutting the target test case to obtain the cut target test case;

[0011] A generating module, used for generating a test action based on the graphical user interface to be tested and the tailored target test case;

[0012] The test module is used to execute the test action to automatically test the graphical user interface to be tested and obtain the test result.

[0013] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0017] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the above method.

[0018] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0021] Figure 1 It is a flowchart of a graphical user interface testing method provided by the first embodiment of the present disclosure;

[0022] Figure 2 is a schematic diagram of a branch of an exemplary original target test case;

[0023] Figure 3 is a schematic diagram of a branch of an exemplary pruned target test case;

[0024] Figure 4 is a schematic diagram of an exemplary generated test action;

[0025] Figure 5 is a flowchart of a graphical user interface testing method provided by the second embodiment of the present disclosure;

[0026] Figure 6 This is an exemplary process diagram of S203;

[0027] Figure 7 is an exemplary interface diagram of a graphical user interface detection interface element;

[0028] Figure 8is a schematic diagram of an exemplary test system;

[0029] Fig. 9 is a structural schematic diagram of a graphical user interface testing device provided in a third embodiment of the present disclosure;

[0030] Fig.10 The block diagram is a block diagram of an electronic device for implementing the method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0033] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0034] The terminology used herein is for describing particular embodiments only and is not intended to limit the present disclosure.As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0035] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0036] The graphical user interface testing method according to the present disclosure can be executed by an electronic device such as a terminal device or a server, and the terminal device can be a vehicle-mounted device, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor calling a computer-readable program instruction stored in a memory. Alternatively, the graphical user interface testing method provided by the present disclosure can be executed by a server.

[0037] In the first embodiment disclosed, see Figure 1 , Figure 1 A flowchart of a method for testing a graphical user interface provided by the first embodiment of the present disclosure is shown. The method comprises:

[0038] S101. Tailor a target test case to obtain a tailored target test case.

[0039] S102: Generate test actions based on the graphical user interface to be tested and the tailored target test cases.

[0040] S103: Execute a test action to automatically test the graphical user interface to be tested and obtain a test result.

[0041] A test case is a description of the test task for application software, reflecting the test plan, method, technology and strategy. The content of a test case can include at least one of the following: test objectives, test environment, input data, test steps, expected results, test scripts, etc., and finally form a document. In other words, a test case is a set of test inputs, execution conditions and expected results compiled for a special goal, used to test whether a certain requirement of a specific application software is met.

[0042] A Graphical User Interface (GUI) is a computer user interface that is displayed in a graphical manner. A GUI is an interface display format for users to communicate with application software. A GUI contains multiple interface elements, and users can operate at least some of the interface elements to interact with the application software. Therefore, in order to verify a certain function of a GUI, it is necessary to form a test case based on the requirements of the GUI.

[0043] Here, a test case set containing multiple test cases can be formed for the GUI to be tested. Since the test case may contain multiple expressions (or "descriptions") that are irrelevant to the test execution steps, such as summary expressions, in S101, the target test case can be trimmed first, and some expressions irrelevant to the test execution can be removed to obtain a trimmed target test case. This can reduce the information that needs to be processed and resolved later, and make the generation of subsequent test actions clearer to reduce errors.

[0044] Furthermore, after obtaining the tailored target test case, a test action is generated based on the actual GUI to be tested and the target test case, and then the action is executed to automatically test the GUI, thereby realizing the automation of GUI testing.

[0045] See also Figure 2-Figure 4 , Figure 2A schematic diagram showing a branch of an exemplary original target test case; Figure 3 A schematic diagram showing a branch of an exemplary pruned target test case; Figure 4 A schematic diagram showing an exemplary generated test action is shown. As an example, see Figure 2 The original test cases are written by relevant testers according to their requirements, so the format is not uniform and contains many summaries, markers, or other expressions irrelevant to automated execution. Figure 3 , get the pruned test cases, retaining the expressions of the test cases relevant to the execution. For example, Figure 2 Part of the test case is: "Main dialogue test case (case) → dialogue page → welcome logic and adjustment → function → map-related historical data → input how to get to the Forbidden City → normal display of welcome words and map history content". In the expression of this part of the test case, "welcome logic and adjustment → function → map-related historical data" are all summary expressions, which are irrelevant to the subsequent generation of test actions, so they need to be cut to get Figure 3 The test case that has been trimmed in the above example retains the expression of the test case that needs to be executed to actually generate the test action: "Dialogue page → input how to get to the Forbidden City → normal display of welcome message and map history content".

[0046] Further, based on Figure 3 After the test cases are tailored, determine the test actions that need to be performed to test these expressions. For example, see Figure 4 , we get five test actions: cold start the application software (app) → click the dialog tab (tab) → enter "how to get to the Forbidden City" in the main dialog input box → on the main dialog page, click the send button → on the current page, verify that there is a welcome message. So the executor can complete these five actions by executing these five actions. Figure 3 Test the test case and get the test results.

[0047] The method provided by the present invention can tailor the original target test cases to obtain the target test cases that actually need to be tested, and then automatically generate test actions for the target test cases that actually need to be tested, and then automatically test the GUI application test actions to be tested, thereby realizing automatic testing of the GUI, thereby realizing the automation of GUI testing and simplifying the testing process.

[0048] In the related art, there are two main methods for automatic testing of GUI, the recording and playback method and the model-driven method. In the recording and playback method, the tester needs to manually test first, and the recording tool records these operations, and then performs playback testing; in the model-driven method, it is necessary to input descriptions including interface elements, business logic, etc., and the mathematical model automatically determines the test cases or scripts. Of these two methods, the recording and playback method has poor flexibility and is intelligently performed according to pre-recording; while the model-driven method has poor semantic understanding ability and is complex to drive. Based on this, the present disclosure proposes a testing method for a graphical user interface, see below.

[0049] In disclosing the second embodiment, see Figure 5 , Figure 5 A flowchart of a method for testing a graphical user interface provided by a second embodiment of the present disclosure is shown. The method comprises:

[0050] S201. Based on preset screening conditions, target test cases that can be used for automatic testing are screened out from a preset test case set.

[0051] The preset screening conditions include at least one of the following:

[0052] (1) The number of preconditions exceeds the preset threshold.

[0053] Preconditions are conditions that must be true before a test case is executed. If there are too many or too stringent preconditions, they are not suitable for automatic testing, so they need to be filtered. Presets can be set as needed, such as 3 or 5, and there is no limit here.

[0054] (2) Test cases rely on complex data input. Complex data, such as dynamic data, is more difficult to automatically test in this case, so it can be filtered.

[0055] (3) The test case requires complex interactive operations. In this case, the accuracy of the test is low, so it can be filtered.

[0056] The above conditions are only exemplary descriptions and do not constitute limitations, and each condition can be omitted or combined. In this way, one or more target test cases suitable for automatic testing are screened out from the test case set, and subsequent operations can be performed for any target test case, thereby screening out test cases that are not suitable for automatic testing, improving the accuracy of the test, and avoiding invalid testing.

[0057] S202: Input the target test case and preset prompt words into the large model, and use the large model to remove invalid expressions in the target test case to obtain a trimmed target test case.

[0058] Since the test case may contain multiple expressions (or "descriptions") that are not related to the test execution steps, such as summary expressions, the target test case can be trimmed in S202. Specifically, the preset prompt words (prompt) can be combined with the generalization ability and semantic parsing ability of the reason model. Through the guidance of the prompt words, some expressions that are not related to the test execution in the original target test case can be removed to obtain the trimmed target test case. This can reduce the information that needs to be processed and resolved later, and make the generation of subsequent test actions clearer and reduce errors.

[0059] The large model includes at least one of the following: a machine learning model with a large number of parameters and a complex structure, a large language model, a language model based on an encoder (Transformer) architecture, etc., which are not limited here.

[0060] It should be noted that S202 is an implementation of S101. Currently, S101 also has other implementations, which are not limited here.

[0061] S203: Automatically test the graphical user interface to be tested based on historical test actions corresponding to historical test cases similar to the target test case.

[0062] See also Figure 6 , Figure 6 An exemplary flow chart of S203 is shown, where S203 includes:

[0063] S2031. Based on the target test case, perform similarity search on historical test cases contained in a preset automatic test case library to obtain historical test cases whose similarity to the target test case is higher than a preset threshold.

[0064] S2032. Automatically test the graphical user interface to be tested by using historical test actions corresponding to historical test cases.

[0065] S2033: Determine an untested portion in the graphical user interface to be tested, and execute a test case based on the graphical user interface to be tested and the tailored target test case based on the untested portion to generate a test action.

[0066] In order to make the tested test cases and their test results continue to be used and realize the data flywheel, an automatic test case library can be established, in which historical test cases and related information are stored, such as historical test actions, historical test results, historical graphical user interfaces, historical test results, etc. corresponding to the historical test cases, which are not limited here. After each test, the test cases and related information are stored in the automatic test case library as historical test cases. Thus, when a new target test case is carried out, it is possible to retrieve whether there is a historical test case with similarity to the current target test case in the automatic test case library. If so, the historical test case is taken out, the corresponding historical test action is obtained, and then the historical test action is executed for automatic testing. However, the test cases are usually not completely consistent, so the historical test action may only test a part of the functions of the current target test case, thereby locating the part of the current target test case that the historical test action has not tested, and executing the subsequent S204 test. If the historical test action can completely cover the current target test case, the execution of the subsequent steps is empty.

[0067] The preset threshold of similarity can be set as required, such as 70%, etc., which is not limited here.

[0068] In this way, on the one hand, historical test data can be reused to achieve data flywheel; on the other hand, subsequent test steps can be simplified.

[0069] In some examples, the historical test cases are in the form of text vectors, that is, the historical test cases in the automatic test case library are stored in a vector manner, so that fast retrieval can be achieved. In this case, S2031 includes:

[0070] Sub-step 1: Convert the target test case into text vector form.

[0071] Sub-step 2: Perform vector similarity retrieval based on the target test case in text vector form and historical test cases in text vector form in a preset automatic test case library to obtain historical test cases whose vector similarity with the target test case is higher than a preset threshold.

[0072] First, the target test case is also converted into a vector form, and then the vector similarity search is performed with the historical test cases in vector form. In this way, the search and matching can be quickly realized. Among them, the preset threshold of the vector similarity can be set as needed, such as 80%, etc., which is not limited here.

[0073] In some examples, S203 includes:

[0074] S2031′, based on the target test case, perform similarity retrieval on historical test cases contained in a preset automatic test case library to obtain historical test cases whose similarity to the target test case is higher than a preset threshold;

[0075] S2032', input the target test case and the historical test case into the big model, and verify through the big model whether the description of the target test case matches the description of the historical test case.

[0076] If there is a match, execute S2033; if there is no match, end the process, skip S203, and execute S204.

[0077] S2033', automatically testing the graphical user interface to be tested through historical test actions corresponding to historical test cases;

[0078] S2034′, determining an untested portion in the graphical user interface to be tested, and executing a test case based on the graphical user interface to be tested and the cut target test case based on the untested portion to generate a test action.

[0079] In this implementation, step S2032' is introduced, that is, after retrieving historical test cases similar to the current target test case, a discrimination step is added to determine whether the description of the target test case matches the description of the historical test case with a similar threshold through the large model, thereby avoiding test errors. Since the description has semantic information, the introduction of the large model for verification can ensure accuracy.

[0080] It should be noted that, in some cases, S203 (including S2031-S2033 and S2031'-S2034') can be omitted, which is not limited here.

[0081] S204: Detect interface elements included in the graphical user interface to be tested.

[0082] S205. Input the big model based on the interface elements, target test cases, and preset callable function sets to obtain the test actions output by the big model.

[0083] Among them, the test action at least includes: the target interface element corresponding to the test action and the position of the target interface element in the graphical user interface to be tested, and the callable function corresponding to the test action. The purpose of the test action is to call the corresponding callable function on the target interface element to execute the test action and obtain the result returned by the callable function. In S205, through the large model, combined with the target test case and the available callable function set, the large model is input for analysis to obtain which test actions need to be completed under the current GUI to be tested in order to achieve the test requirements of the target test case, thereby breaking down the entire test path into multiple single-step test actions, and executing the test action can complete the target test case under the GUI to be tested and obtain the test result.

[0084] GUI refers to a computer operation user interface that is displayed in a graphical manner. GUI is an interface display format for users to communicate with application software. GUI contains multiple interface elements, and users can operate at least some of the interface elements to interact with the application software. In this step, the GUI to be tested can be obtained by running the application software using a device loaded with the application software, and then taking a real-time screenshot. To verify a certain function of the GUI, it is necessary to locate the associated interface elements that need to be operated or read, so that the graphical user interface to be tested is first identified to obtain the interface elements contained in the GUI, and then through the large model, combined with the target test case and the available callable function set, the large model is input for analysis to obtain which test actions need to be completed under the current GUI to be tested in order to achieve the test requirements of the target test case, thereby disassembling the entire test path into multiple single-step test actions, and executing the test actions can complete the target test case under the GUI to be tested and obtain the test results.

[0085] In some examples, the interface elements include interactive interface elements and text interface elements. The interactive interface elements are interface elements included in the GUI that users can use to interact with application software; the text interface elements are interface elements included in the GUI that are used for description. Since the characteristics of the two are inconsistent, two models can be used to identify the two interface elements respectively; S204 includes:

[0086] Step 1: Input the graphical user interface to be tested into the interactive element detection (GUIICON) model to obtain the interactive interface elements and their positions.

[0087] Step 2: Input the graphical user interface to be tested into the text extraction (OCR) model to obtain the text interface elements and their positions.

[0088] See also Figure 7 , Figure 7An exemplary graphical user interface for detecting interface elements is shown. The recognition results of step 1 and step 2 include: the recognized interface elements, the rectangular frames of the interface elements (used to indicate their positions), and the digital identifiers assigned to the rectangular frames. Figure 7 In the detection interface shown, the graphical user interface includes five interface elements: Image 1, Figure 2 , sample text, button 1 and button 2. Each recognized interface element has a rectangular frame and a digital logo in the upper left corner. This information will then be input into the large model, and the test action will be generated in combination with the large model. Although the large model has graphic analysis capabilities, due to the complexity of the GUI, the target recognition capability of the large model usually results in recognition errors. Therefore, the interface elements can be identified through pre-trained GUIICON models and OCR models, and then input into the large model. This can significantly improve the recognition accuracy and further ensure the accuracy of the location where the test action is performed on the GUI to be tested.

[0089] In some examples, the test action includes multiple steps; S205 includes:

[0090] Based on the interface elements, target test cases, and preset callable function sets, the big model is input to obtain multiple test actions output by the big model and the execution order of the multiple test actions.

[0091] In some examples, to complete a target test case, it is usually necessary to perform multiple test actions. The large model is used to integrate interface elements, target test cases, and callable function sets, analyze which test actions are required to implement the target test cases in the current GUI, and determine the operation location of each test action and the corresponding callable function.

[0092] For example, see Figure 3 , the target test case to be tested: "Dialogue page → input the way to the Forbidden City → normal display of welcome message and map history content"; see Figure 4 , the test actions that need to be performed are: get five execution actions: cold start application software (app) → click the dialog label (tab) → enter "how to get to the Forbidden City" in the main dialog input box → on the main dialog page, click the send button → on the current page, verify the existence of the welcome message. In other words, Figure 3 The target test case shown is "Dialogue page → input how to get to the Forbidden City → normal display of welcome message and map history content", the identified interface elements, the callable function set input big model, the big model output test actions and their execution order: see Figure 4 , cold start the application software (app) → click the dialog tab (tab) → enter "how to get to the Forbidden City" in the input box of the main dialog → on the main dialog page, click the Send button → on the current page, verify that there is a welcome message.

[0093] In some examples, the callable function set includes multiple callable functions. Multiple callable functions can be preset, each of which has a corresponding operation function. Exemplarily, the callable function set includes at least one of the following callable functions:

[0094] (1) The first callable function is used to click any interface element.

[0095] Exemplarily, the first callable function may be expressed as tap(element:int). The first callable function is used to click an interactive interface element displayed on the GUI, where "element" is a digital identifier assigned to the interactive interface element displayed on the GUI.

[0096] (2) A second callable function is used to input text for any interface element.

[0097] Exemplarily, the second callable function can be expressed as input(element: int, text_input: str). The second callable function is used for text input, "element" is a digital identifier assigned to an interactive interface element displayed by the GUI, and "text_input" is an input string.

[0098] (3) A third callable function is used to represent the sliding trajectory of the finger on the graphical user interface to be tested.

[0099] For example, the third callable function can be expressed as swipe(direction:str). This third callable function is used to slide the screen of a smartphone, indicating the sliding track of a finger on the screen. In some examples, the sliding distance defaults to half the screen. "direction" is a string representing four directions: up (slide up), down (slide down), left (slide to the left), and right (slide to the right).

[0100] (4) The fourth callable function is used to long press any interface element.

[0101] Exemplarily, the fourth callable function may be expressed as long_press(element:int). The fourth callable function is used to long press an interface element displayed on the GUI. "element" is a digital identifier assigned to an interactive interface element displayed on the GUI.

[0102] (5) A fifth callable function is used to verify whether any interface element exists in the graphical user interface to be tested.

[0103] Exemplarily, the fifth callable function can be expressed as assert_exist(element: int, content: str). According to the target test case, it is necessary to verify the existence of a certain interface element on a GUI page. "content" indicates the verification content. "element" is a digital identifier assigned to the interactive interface element displayed by the GUI.

[0104] (6) A sixth callable function is used to verify that no interface element exists in the graphical user interface to be tested.

[0105] Exemplarily, the sixth callable function can be expressed as assert_not_exist(content:str). According to the target test case, it is necessary to verify that a certain interface element does not exist on a GUI page. "content" represents the verification content. "element" is a digital identifier assigned to the interactive interface element displayed by the GUI.

[0106] The above conditionally callable functions are merely exemplary descriptions and are not intended to be limiting, and each callable function may be omitted or combined.

[0107] It should be noted that S204-S205 is one implementation of S102. Currently, S102 also has other implementations, which are not limited here.

[0108] S206. Call the callable function corresponding to the test action in the execution order, execute the test action for the target interface element, and obtain the parameters returned by the callable function as the test result.

[0109] In S205, when the large model generates test actions, it allocates corresponding callable functions to each test action for implementing the test action. In S206, the executor calls the corresponding functions in sequence according to the test actions and execution order output by the large model, that is, the test path, and executes the callable functions on the target interface elements to obtain the parameters returned by the callable functions as the test results, thereby realizing automatic testing of each test case of the GUI.

[0110] It should be noted that S206 is an implementation method of S103. Currently, S103 also has other implementation methods, which are not limited here.

[0111] S207: Establish a mapping relationship among the target test case, the test action, the graphical user interface to be tested, and the test result, and store the mapping relationship in at least one of the case storage and the automatic test case library.

[0112] In the method provided by the present disclosure, each time a target test case is executed, a mapping relationship between the target test case, the test action, the graphical user interface to be tested, and the test result can be established, and the above data with the mapping relationship can be stored as historical data in at least one of the use case storage and the automatic test case library. After obtaining the tailored target test case, a similarity search is performed in the automatic test case library based on the historical test case to obtain a historical test case whose similarity to the target test case is higher than a preset threshold. Exemplarily, the target test case, the test action, the graphical user interface to be tested, and the test result can be stored in the automatic test case library in the form of a vector, which is convenient for retrieval and archiving; the original data of the target test case, the test action, the graphical user interface to be tested, and the test result can be stored in the use case storage, which is convenient for extended application based on the original data.

[0113] S208. Input the target test case, test action, graphical user interface to be tested, graphical user interface after the automatic test and test results into the big model, determine the correctness of the automatic test based on the big model, and obtain the correction suggestion information output by the big model.

[0114] In some cases, in order to avoid errors in the generation and execution of test actions, a correction mechanism is introduced. That is, the target test case, test action, graphical user interface to be tested, graphical user interface after the automatic test and test results are input into the big model, and the correctness of the automatic test is judged through the generalization ability of the big model. In other words, it is judged whether the automatic test is carried out according to the expected evolution; then based on the judgment result, the correction suggestion information is generated. If the correctness is judged to be compliant, the correction suggestion can be empty. The correction suggestion information indicates the errors that occur in the generation and execution of the test action, as well as the way to correct the errors. In this way, test errors can be avoided and the accuracy of the test results can be improved.

[0115] S209, input the target test case, test action, graphical user interface to be tested, test results and preset test requirements into the big model, determine whether the automatic test meets the test requirements through the big model, and obtain the test requirement summary information output by the big model.

[0116] The requirement summary information is information indicating whether the automatic test meets the test requirements and the progress of the automatic test. The target test case, test action, graphical user interface to be tested, test results and preset test requirements are input into the big model. Through the generalization ability of the big model, it is judged whether the automatic test meets the test requirements, and the test progress is recorded in the process of executing each test action. Finally, the test requirement summary information is generated based on the judgment result and the test progress. By recording the test progress of each test action and whether the test requirements are met, it can be used as guidance information to guide the generation of subsequent test actions.

[0117] It should be noted that at least any one of steps S207-209 can be omitted, and the execution order of S207-S09 can be swapped, which is not limited here.

[0118] See also Figure 8 , Figure 8 The schematic diagram of the structure of an exemplary test system of the method provided by the present disclosure is shown in FIG. The method of the present disclosure can be applied in, for example, Figure 8 The test system shown in FIG. 1 includes: a judgment optimization module, a retrieval module and a generation module, wherein the retrieval module can be omitted in some examples. The judgment optimization module includes a judge and a clipper; the retrieval module includes a vector retriever and an executor, and is linked to an automatic test case set; the generation module includes a visual parser, an action generator, an executor, a case storage, a corrector, a summarizer, and is connected to a device for loading application software.

[0119] Among them, in the judgment optimization module:

[0120] The judger is used to filter out target test cases that can be used for automatic testing from a preset test case set based on preset filtering conditions.

[0121] The trimmer is used to input the target test case and preset prompt words into the large model, and remove invalid expressions in the target test case through the large model to obtain the trimmed target test case.

[0122] In the search module:

[0123] The vector retriever is used to perform similarity retrieval on the historical test cases contained in a preset automatic test case library based on the target test case, and obtain the historical test cases whose similarity with the target test case is higher than a preset threshold.

[0124] The executor is used to automatically test the graphical user interface to be tested through historical test actions corresponding to historical test cases, determine untested parts of the graphical user interface to be tested, and input them into the generation module.

[0125] In the generated module:

[0126] The visual parser is used to detect the interface elements contained in the graphical user interface to be tested.

[0127] The action generator is used to input the large model based on interface elements, target test cases, and preset callable function sets to obtain test actions output by the large model.

[0128] The executor is used to call the callable function corresponding to the test action in the execution order, execute the test action for the target interface element, and obtain the parameters returned by the callable function as the test result.

[0129] The use case memory is used to establish a mapping relationship among a target test case, a test action, a graphical user interface to be tested, and a test result, and save the mapping relationship in the use case memory.

[0130] The corrector is used to input the target test cases, test actions, the graphical user interface to be tested, the graphical user interface after the automatic test and the test results into the big model, determine the correctness of the automatic test based on the big model, and obtain the correction suggestion information output by the big model.

[0131] The summarizer is used to input the target test cases, test actions, graphical user interfaces to be tested, test results and preset test requirements into the big model, determine whether the automatic test meets the test requirements through the big model, and obtain the test requirement summary information output by the big model.

[0132] In the third embodiment disclosed, based on Figure 1 The same principle, Fig. 9 A graphical user interface testing device 90 provided in the third embodiment of the present disclosure is shown, and the device includes:

[0133] A cutting module 901 is used to cut the target test case to obtain a cut target test case;

[0134] A generating module 902, for generating a test action based on the graphical user interface to be tested and the tailored target test case;

[0135] The testing module 903 is used to execute a testing action to automatically test the graphical user interface to be tested and obtain a testing result.

[0136] In some examples, the generation module is specifically used to:

[0137] Detect the interface elements contained in the graphical user interface to be tested;

[0138] Input the big model based on the interface elements, target test cases, and preset callable function sets to obtain the test actions output by the big model;

[0139] The test action at least includes: a target interface element corresponding to the test action and the position of the target interface element in the graphical user interface to be tested, and a callable function corresponding to the test action.

[0140] In some examples, the test action includes a plurality of;

[0141] The generation module is used to input the large model based on the interface elements, target test cases, and preset callable function sets, and obtain the test actions output by the large model:

[0142] Based on the interface elements, target test cases, and preset callable function sets, the big model is input to obtain multiple test actions output by the big model and the execution order of the multiple test actions.

[0143] In some examples, the interface elements include interactive interface elements and textual interface elements;

[0144] The generation module detects the interface elements contained in the graphical user interface to be tested, and is used to:

[0145] Input the graphical user interface to be tested into the interactive element detection model to obtain the interactive interface elements and their positions;

[0146] The graphical user interface to be tested is input into the text extraction model to obtain the text interface elements and their positions.

[0147] In some examples, the set of callable functions includes at least one of the following callable functions:

[0148] The first callable function is used to click any interface element;

[0149] The second callable function is used to input text for any interface element;

[0150] A third callable function is used to represent a sliding track of a finger on the graphical user interface to be tested;

[0151] The fourth callable function is used to long press any interface element;

[0152] A fifth callable function is used to verify that any interface element exists in the graphical user interface to be tested;

[0153] The sixth callable function is used to verify that the graphical user interface to be tested does not have any interface element.

[0154] In some examples, the test module is specifically used to:

[0155] In accordance with the execution order, the callable function corresponding to the test action is called, the test action is executed for the target interface element, and the parameters returned by the callable function are obtained as the test result.

[0156] In some examples, the cropping module is used to:

[0157] The target test case and the preset prompt words are input into the big model, and the invalid expressions in the target test case are removed by the big model to obtain the trimmed target test case.

[0158] In some examples, the apparatus further comprises:

[0159] A retrieval module is used to perform similarity retrieval on historical test cases contained in a preset automatic test case library based on a target test case, and obtain historical test cases whose similarity to the target test case is higher than a preset threshold;

[0160] Automatically test the graphical user interface to be tested through historical test actions corresponding to historical test cases;

[0161] An untested portion of the graphical user interface to be tested is determined, and a test action is generated based on the untested portion and the graphical user interface to be tested and the tailored target test case.

[0162] In some examples, the apparatus further comprises:

[0163] A verification module is used to input the target test case and the historical test case into the big model, and verify whether the description of the target test case matches the description of the historical test case through the big model;

[0164] In the case of a match, the historical test actions corresponding to the historical test cases are executed, and the graphical user interface to be tested is automatically tested.

[0165] In some examples, the apparatus further comprises:

[0166] The storage module is used to establish a mapping relationship among target test cases, test actions, graphical user interfaces to be tested and test results, and store the mapping relationship in at least one of a case storage and an automatic test case library.

[0167] In some examples, the apparatus further comprises:

[0168] The correction module is used to input the target test cases, test actions, the graphical user interface to be tested, the graphical user interface after the automatic test and the test results into the big model, determine the correctness of the automatic test based on the big model, and obtain the correction suggestion information output by the big model.

[0169] In some examples, the apparatus further comprises:

[0170] A module is required to input the target test cases, test actions, the graphical user interface to be tested, the test results and the preset test requirements into the big model, determine whether the automatic test meets the test requirements through the big model, and obtain the test requirement summary information output by the big model.

[0171] In some examples, the apparatus further comprises:

[0172] The screening module is used to screen out target test cases that can be used for automatic testing from a preset test case set based on preset screening conditions.

[0173] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0174] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0175] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0176] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from the storage unit 1002 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0177] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0178] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a test method for a graphical user interface. For example, in some embodiments, the test method for a graphical user interface may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the test method for the graphical user interface described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured with the graphical user interface testing method in any other appropriate manner (eg, by means of firmware).

[0179] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0180] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0181] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0182] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0183] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0184] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0185] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0186] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for testing a graphical user interface, wherein: The method comprises: Tailoring the target test case to obtain the tailored target test case; Generate test actions based on the graphical user interface to be tested and the tailored target test case; The test action is executed to automatically test the graphical user interface to be tested to obtain a test result.

2. The method according to claim 1, wherein: The generating of the test action based on the graphical user interface to be tested and the tailored target test case includes: Detecting interface elements included in the graphical user interface to be tested; Inputting a large model based on the interface elements, the target test cases, and a preset callable function set to obtain a test action output by the large model; The test action at least includes: a target interface element corresponding to the test action and a position of the target interface element in the graphical user interface to be tested, and a callable function corresponding to the test action.

3. The method according to claim 2, wherein: The test actions include multiple; The step of inputting the large model based on the interface elements, the target test cases, and the preset callable function set to obtain the test actions output by the large model includes: The large model is input based on the interface elements, the target test cases, and the preset callable function set to obtain the multiple test actions output by the large model and the execution order of the multiple test actions.

4. The method according to claim 2 or 3, wherein: The interface elements include interactive interface elements and text interface elements; The detecting the interface elements contained in the graphical user interface to be tested includes: Inputting the graphical user interface to be tested into an interactive element detection model to obtain the interactive interface elements and their positions; The graphical user interface to be tested is input into a text extraction model to obtain the text interface elements and their positions.

5. The method according to any one of claims 2 to 4, wherein: The callable function set includes at least one of the following callable functions: A first callable function is used to click any of the interface elements; A second callable function is used to input text for any of the interface elements; A third callable function, used to represent a sliding track of a finger on the graphical user interface to be tested; A fourth callable function is used to long press any of the interface elements; A fifth callable function, used to verify that any of the interface elements exists in the graphical user interface to be tested; The sixth callable function is used to verify that the graphical user interface to be tested does not contain any of the interface elements.

6. The method according to any one of claims 2 to 5, wherein: The executing the test action to automatically test the graphical user interface to be tested to obtain a test result includes: In accordance with the execution order, the callable function corresponding to the test action is called, the test action is performed on the target interface element, and the parameter returned by the callable function is obtained as the test result.

7. The method according to any one of claims 1 to 6, wherein: The step of tailoring the target test case to obtain the tailored target test case includes: The target test case and preset prompt words are input into the large model, and invalid expressions in the target test case are removed by the large model to obtain the trimmed target test case.

8. The method according to any one of claims 1 to 7, wherein: After the target test case is tailored to obtain the tailored target test case, and before the test action is generated based on the graphical user interface to be tested and the tailored target test case, the method further includes: Based on the target test case, similarity retrieval is performed on historical test cases contained in a preset automatic test case library to obtain historical test cases whose similarity to the target test case is higher than a preset threshold; Automatically test the graphical user interface to be tested through historical test actions corresponding to historical test cases; An untested portion in the graphical user interface to be tested is determined, and based on the untested portion, the test case based on the graphical user interface to be tested and the tailored target test case is executed to generate a test action.

9. The method according to claim 8, wherein: After performing similarity retrieval on the historical test cases contained in the preset automatic test case library based on the target test case to obtain the historical test cases whose similarity with the target test case is higher than the preset threshold, before automatically testing the graphical user interface to be tested through the historical test actions corresponding to the historical test cases, the method further includes: Inputting the target test case and the historical test case into a large model, and verifying through the large model whether the description of the target test case matches the description of the historical test case; In the case of a match, the historical test action corresponding to the historical test case is executed, and the graphical user interface to be tested is automatically tested.

10. The method according to any one of claims 1 to 9, wherein: After the test action is executed to automatically test the graphical user interface to be tested and the test result is obtained, the method further includes: A mapping relationship among the target test case, the test action, the graphical user interface to be tested and the test result is established and stored in at least one of a case storage and an automatic test case library.

11. The method according to any one of claims 1 to 10, wherein: After the test action is executed to automatically test the graphical user interface to be tested and the test result is obtained, the method further includes: The target test case, the test action, the graphical user interface to be tested, the graphical user interface after the automatic test and the test result are input into the big model, the correctness of the automatic test is determined based on the big model, and the correction suggestion information output by the big model is obtained.

12. The method according to any one of claims 1 to 11, wherein: After the test action is executed to automatically test the graphical user interface to be tested and the test result is obtained, the method further includes: The target test case, the test action, the graphical user interface to be tested, the test result and the preset test requirements are input into the big model, and whether the automatic test meets the test requirements is determined by the big model to obtain the test requirement summary information output by the big model.

13. The method according to any one of claims 1 to 12, wherein: Before the target test case is clipped to obtain the clipped target test case, the method further includes: Based on the preset filtering conditions, target test cases that can be used for automatic testing are filtered out from the preset test case set.

14. A graphical user interface testing device, wherein: The device comprises: A cutting module, used for cutting the target test case to obtain the cut target test case; A generating module, used for generating a test action based on the graphical user interface to be tested and the tailored target test case; The test module is used to execute the test action to automatically test the graphical user interface to be tested and obtain the test result.

15. The device according to claim 14, wherein: The generation module is specifically used for: Detecting interface elements included in the graphical user interface to be tested; Inputting a large model based on the interface elements, the target test cases, and a preset callable function set to obtain a test action output by the large model; The test action at least includes: a target interface element corresponding to the test action and a position of the target interface element in the graphical user interface to be tested, and a callable function corresponding to the test action.

16. The device according to claim 15, wherein: The test actions include multiple; The generation module is used to input the large model based on the interface elements, the target test cases, and the preset callable function set to obtain the test actions output by the large model: The large model is input based on the interface elements, the target test cases, and the preset callable function set to obtain the multiple test actions output by the large model and the execution order of the multiple test actions.

17. The device according to claim 15 or 16, wherein: The interface elements include interactive interface elements and text interface elements; The generating module detects the interface elements included in the graphical user interface to be tested, and is used to: Inputting the graphical user interface to be tested into an interactive element detection model to obtain the interactive interface elements and their positions; The graphical user interface to be tested is input into a text extraction model to obtain the text interface elements and their positions.

18. The device according to any one of claims 15 to 17, wherein: The callable function set includes at least one of the following callable functions: A first callable function is used to click any of the interface elements; A second callable function is used to input text for any of the interface elements; A third callable function, used to represent a sliding track of a finger on the graphical user interface to be tested; A fourth callable function is used to long press any of the interface elements; A fifth callable function, used to verify that any of the interface elements exists in the graphical user interface to be tested; The sixth callable function is used to verify that the graphical user interface to be tested does not contain any of the interface elements.

19. The device according to any one of claims 15 to 18, wherein: The test module is specifically used for: In accordance with the execution order, the callable function corresponding to the test action is called, the test action is performed on the target interface element, and the parameter returned by the callable function is obtained as the test result.

20. The device according to any one of claims 14 to 19, wherein: The cutting module is used to: The target test case and preset prompt words are input into the large model, and invalid expressions in the target test case are removed by the large model to obtain the trimmed target test case.

21. The device according to any one of claims 14 to 20, wherein: The device also includes: A retrieval module is used to perform similarity retrieval on historical test cases contained in a preset automatic test case library based on the target test case, and obtain historical test cases whose similarity with the target test case is higher than a preset threshold; Automatically test the graphical user interface to be tested through historical test actions corresponding to historical test cases; An untested portion in the graphical user interface to be tested is determined, and based on the untested portion, the test case based on the graphical user interface to be tested and the tailored target test case is executed to generate a test action.

22. The device according to any one of claims 14 to 21, wherein: The device also includes: A verification module, used for inputting the target test case and the historical test case into a large model, and verifying through the large model whether the description of the target test case matches the description of the historical test case; In the case of a match, the historical test action corresponding to the historical test case is executed, and the graphical user interface to be tested is automatically tested.

23. The device according to any one of claims 14 to 22, wherein: The device also includes: The storage module is used to establish a mapping relationship among the target test case, the test action, the graphical user interface to be tested and the test result, and store it in at least one of a case storage and an automatic test case library.

24. The device according to any one of claims 14 to 23, wherein: The device also includes: A correction module is used to input the target test case, the test action, the graphical user interface to be tested, the graphical user interface after the automatic test and the test result into a large model, determine the correctness of the automatic test based on the large model, and obtain the correction suggestion information output by the large model.

25. The device according to any one of claims 14 to 24, wherein: The device also includes: A module is required, which is used to input the target test case, the test action, the graphical user interface to be tested, the test results and the preset test requirements into the big model, determine whether the automatic test meets the test requirements through the big model, and obtain the test requirement summary information output by the big model.

26. The device according to any one of claims 14 to 25, wherein: The device also includes: The screening module is used to screen out target test cases that can be used for automatic testing from a preset test case set based on preset screening conditions.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 14.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-13.

29. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 13.

Images