Graphical user interface testing method, device, and electronic device

Abstract

The present disclosure provides a method for testing a graphical user interface, which relates to the field of software testing technology, and in particular to the field of testing a graphical user interface and the field of artificial intelligence technology. A specific implementation scheme is as follows: in response to receiving a database content synchronization request sent by multiple target devices, obtaining device status information of the target device contained in the database content synchronization request; based on the device status information of the multiple target devices, splitting the database content data corresponding to the database content synchronization request into multiple data blocks corresponding to the multiple target devices; generating multiple data block synchronization tasks based on the multiple data blocks, and assigning the multiple data block synchronization tasks to the multiple target devices, wherein any one of the multiple target devices is configured to execute the data block synchronization task in response to receiving the data block synchronization task sent by the server to synchronize the corresponding data block to the local target device.

Description

Technical Field

[0001] The present disclosure relates to the field of software testing technology, and in particular to the field of user graphical interface testing and artificial intelligence technology. Background Art

[0002] An application's graphical user interface (GUI) plays a crucial role in software usage, directly impacting functionality and user experience. When an application is updated, the GUI often undergoes changes, which can impact the proper functioning of existing functionality. Therefore, GUI testing is essential to ensure that the new version doesn't negatively impact existing functionality.

[0003] Currently, manual testing is mainly used. Manual testing requires manual operations in frequent testing scenarios, which is inefficient and prone to human errors, resulting in inaccurate test results. Summary of the Invention

[0004] The present disclosure provides a database content synchronization method, device, and electronic device for solving at least one of the above technical problems.

[0005] According to one aspect of the present disclosure, there is provided 1. a method for testing a graphical user interface, wherein the method comprises:

[0006] Tailoring the target test case to obtain the tailored target test case;

[0007] Generate test actions based on the graphical user interface to be tested and the tailored target test case;

[0008] The test action is executed to automatically test the graphical user interface to be tested and obtain a test result.

[0009] According to one aspect of the present disclosure, a graphical user interface testing device is provided, wherein the device includes:

[0010] A cutting module, used for cutting the target test case to obtain the cut target test case;

[0011] A generation module, configured to generate test actions based on the graphical user interface to be tested and the tailored target test case;

[0012] The testing module is used to execute the testing action to automatically test the graphical user interface to be tested and obtain a test result.

[0013] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0017] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the above method.

[0018] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0021] Figure 1 This is a flowchart of a graphical user interface testing method provided by the first embodiment of the present disclosure;

[0022] Figure 2 is a schematic diagram of a branch of an exemplary original target test case;

[0023] Figure 3 is a schematic diagram of a branch of an exemplary pruned target test case;

[0024] Figure 4 is a schematic diagram of an exemplary generated test action;

[0025] Figure 5 is a flowchart of a method for testing a graphical user interface provided by the second embodiment of the present disclosure;

[0026] Figure 6 This is an exemplary process diagram of S203;

[0027] Figure 7 is an exemplary interface diagram of a graphical user interface detection interface element;

[0028] Figure 8is a schematic structural diagram of an exemplary test system;

[0029] Figure 9 1 is a schematic structural diagram of a graphical user interface testing device provided in a third embodiment of the present disclosure;

[0030] Figure 10 is a block diagram of an electronic device for implementing the method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0033] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0034] The terminology used herein is for describing particular embodiments only and is not intended to limit the present disclosure.As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0035] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless expressly defined as such herein.

[0036] The graphical user interface testing method according to the present disclosure can be executed by an electronic device such as a terminal device or a server. The terminal device can be an in-vehicle device, user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, etc. The method can be implemented by a processor calling computer-readable program instructions stored in a memory. Alternatively, the graphical user interface testing method provided by the present disclosure can be executed by a server.

[0037] In the first embodiment of the disclosure, see Figure 1 , Figure 1 A flowchart of a method for testing a graphical user interface provided by the first embodiment of the present disclosure is shown. The method includes:

[0038] S101. Tailor a target test case to obtain a tailored target test case.

[0039] S102: Generate test actions based on the graphical user interface to be tested and the tailored target test cases.

[0040] S103: Execute a test action to automatically test the graphical user interface to be tested and obtain a test result.

[0041] A test case is a description of a testing task for application software, embodying the test plan, methods, techniques, and strategies. A test case can include at least one of the following: test objectives, test environment, input data, test steps, expected results, test scripts, and other items, ultimately documented. In other words, a test case is a set of test inputs, execution conditions, and expected results compiled for a specific purpose, used to test whether a specific requirement of a specific application software is met.

[0042] A graphical user interface (GUI) is a computer user interface that uses graphics. A GUI is an interface display format for users to communicate with application software. A GUI consists of multiple interface elements, at least some of which can be manipulated by the user to interact with the application. Therefore, to verify a specific GUI function, it is necessary to develop test cases tailored to the GUI's requirements.

[0043] Here, a test case set containing multiple test cases can be formed for the GUI to be tested. Since the test case may contain multiple expressions (or "descriptions") that are not related to the execution steps of the test, such as summary expressions, in S101, the target test case can be trimmed first, and some expressions that are not related to the execution of the test can be removed to obtain the trimmed target test case. This can reduce the information that needs to be processed and resolved later, and make the generation of subsequent test actions clearer and reduce errors.

[0044] Furthermore, after obtaining the tailored target test case, a test action is generated based on the actual GUI to be tested and the target test case, and then the action is executed to automatically test the GUI, thereby realizing the automation of GUI testing.

[0045] See also Figure 2-Figure 4 , Figure 2A schematic diagram showing a branch of an exemplary original target test case; Figure 3 A schematic diagram showing a branch of an exemplary tailored target test case; Figure 4 A schematic diagram showing an exemplary generated test action is shown. As an example, see Figure 2 The original test cases are written by relevant testers according to their needs, so the format is not uniform and contains many summaries, tags, or other expressions that are irrelevant to the automation execution. Figure 3 , we get the pruned test cases, which retain the expressions of the test cases that are relevant to the execution. For example, Figure 2 Part of the test case is: "Main dialogue test case (case) → dialogue page → welcome logic and adjustment → function → map related historical data → input the direction to the Forbidden City → normal display of welcome message and map history content". In the expression of this part of the test case, "welcome logic and adjustment → function → map related historical data" are all summary expressions and have nothing to do with the subsequent generation of test actions, so they need to be cut to get Figure 3 The trimmed test case retains the expression of the test case that needs to be executed to actually generate the test action: "Dialogue page → Enter the directions to the Forbidden City → Normally display the welcome message and map history content."

[0046] Further, based on Figure 3 After the test cases are tailored, determine the test actions that need to be performed to test these expressions. For example, see Figure 4 , we get five test actions: cold start the application software (app) → click the dialog tab (tab) → enter "how to get to the Forbidden City" in the main dialog input box → on the main dialog page, click the send button → on the current page, verify that there is a welcome message. So the executor can perform these five actions to complete Figure 3 Test the test case and get the test results.

[0047] The method provided by the present invention can tailor the original target test cases to obtain the target test cases that actually need to be tested, and then automatically generate test actions for the target test cases that actually need to be tested, and then automatically test the GUI application test actions to be tested, thereby realizing automatic testing of the GUI, thereby realizing the automation of GUI testing and simplifying the testing process.

[0048] In the related art, there are two main methods for automatic testing of GUIs: the recording and playback method and the model-driven method. In the recording and playback method, the tester needs to manually test first, and the recording tool records these operations, and then performs playback testing; in the model-driven method, it is necessary to input descriptions including interface elements, business logic, etc., and the mathematical model automatically determines the test cases or scripts. Of these two methods, the recording and playback method has poor flexibility and is intelligently performed according to pre-recording; while the model-driven method has poor semantic understanding capabilities and is complex to drive. Based on this, the present disclosure proposes a testing method for graphical user interfaces, see below.

[0049] In disclosing the second embodiment, see Figure 5 , Figure 5 A flowchart of a method for testing a graphical user interface provided by a second embodiment of the present disclosure is shown. The method includes:

[0050] S201 : Based on preset screening conditions, target test cases that can be used for automatic testing are screened out from a preset test case set.

[0051] The preset screening conditions include at least one of the following:

[0052] (1) The number of preconditions exceeds the preset threshold.

[0053] Preconditions are conditions that must be true before a test case can be executed. Excessive or stringent preconditions are unsuitable for automated testing, so they need to be filtered. Preconditions can be set as needed, for example, to 3 or 5, and there is no limit here.

[0054] (2) Test cases rely on complex data input. Complex data, such as dynamic data, is more difficult to automatically test, so filtering is recommended.

[0055] (3) Test cases require complex interactive operations. In this case, the accuracy of the test is low, so it can be filtered.

[0056] The above conditions are merely illustrative and non-limiting, and individual conditions may be omitted or combined. In this way, one or more target test cases suitable for automated testing are screened from the test case set. Subsequent operations can be performed on any target test case, thereby filtering out test cases unsuitable for automated testing, improving test accuracy, and avoiding ineffective testing.

[0057] S202: Input the target test case and the preset prompt words into the large model, and use the large model to remove invalid expressions in the target test case to obtain a trimmed target test case.

[0058] Since the test case may contain multiple expressions (or "descriptions") that are not related to the test execution steps, such as summary expressions, in S202, the target test case can be trimmed first. Specifically, the preset prompt words (prompt) can be combined with the generalization ability and semantic parsing ability of the reason model. Through the guidance of the prompt words, some expressions in the original target test case that are not related to the test execution can be removed to obtain the trimmed target test case. This can reduce the information that needs to be processed and distinguished later, and make the generation of subsequent test actions clearer, reducing errors.

[0059] The large model includes at least one of the following: a machine learning model with a large number of parameters and a complex structure, a large language model, a language model based on an encoder (Transformer) architecture, etc., which are not limited here.

[0060] It should be noted that S202 is one implementation of S101. Currently, S101 also has other implementations, which are not limited here.

[0061] S203 : Automatically test the graphical user interface to be tested based on historical test actions corresponding to historical test cases similar to the target test case.

[0062] See also Figure 6 , Figure 6 An exemplary flow chart of S203 is shown, where S203 includes:

[0063] S2031 : Based on the target test case, perform similarity search on historical test cases contained in a preset automatic test case library to obtain historical test cases whose similarity to the target test case is higher than a preset threshold.

[0064] S2032. Automatically test the graphical user interface to be tested using historical test actions corresponding to historical test cases.

[0065] S2033: Determine an untested portion of the graphical user interface to be tested, execute a test case based on the graphical user interface to be tested and the tailored target test case based on the untested portion, and generate a test action.

[0066] In order to ensure that previously tested test cases and their test results can be continuously used and achieve a data flywheel, an automatic test case library can be established, which stores historical test cases and related information. Related information includes historical test actions, historical test results, historical graphical user interfaces, historical test results, etc. corresponding to the historical test cases, which are not limited here. After each test, the test case and related information are stored as historical test cases in the automatic test case library. Therefore, when performing a new target test case, the automatic test case library can be searched to see if there is a historical test case similar to the current target test case. If so, the historical test case is retrieved, the corresponding historical test action is obtained, and the historical test action is then executed to perform the automatic test. However, test cases are usually not completely consistent. Therefore, the historical test action may only test a portion of the functions of the current target test case. Therefore, the portion of the current target test case that has not been tested by the historical test action is located, and the subsequent test in S204 is performed. If the historical test action can fully cover the current target test case, the execution of the subsequent steps is empty.

[0067] The preset threshold of similarity can be set as needed, for example, 70%, etc., which is not limited here.

[0068] In this way, on the one hand, historical test data can be reused to achieve data flywheel; on the other hand, subsequent test steps can be simplified.

[0069] In some examples, historical test cases are in the form of text vectors, that is, historical test cases in the automatic test case library are stored in a vector format, thereby enabling fast retrieval. In this case, S2031 includes:

[0070] Sub-step 1: Convert the target test case into text vector form.

[0071] Sub-step 2: Perform vector similarity retrieval based on the target test case in text vector form and historical test cases in text vector form in a preset automatic test case library to obtain historical test cases whose vector similarity with the target test case is higher than a preset threshold.

[0072] First, the target test case is converted into a vector form. Then, a vector similarity search is performed with the historical vector test cases. This method allows for rapid retrieval and matching. The preset threshold for vector similarity can be set as needed, for example, to 80%, which is not limited here.

[0073] In some examples, S203 includes:

[0074] S2031′, based on the target test case, perform similarity search on historical test cases contained in a preset automatic test case library to obtain historical test cases whose similarity to the target test case is higher than a preset threshold;

[0075] S2032′: Input the target test case and the historical test case into the big model, and verify through the big model whether the description of the target test case matches the description of the historical test case.

[0076] If there is a match, execute S2033; if there is no match, end the process, skip S203, and execute S204.

[0077] S2033′, automatically testing the graphical user interface to be tested using historical test actions corresponding to historical test cases;

[0078] S2034′: determine an untested portion of the graphical user interface to be tested, execute the untested portion based on the graphical user interface to be tested and the tailored target test case, and generate a test action.

[0079] This implementation introduces step S2032'. After retrieving historical test cases similar to the current target test case, a verification step is added. Using the large model, the description of the target test case is determined to match the description of historical test cases with similarity thresholds, thus avoiding test errors. Because descriptions contain semantic information, using the large model for verification ensures accuracy.

[0080] It should be noted that, in some cases, S203 (including S2031-S2033 and S2031'-S2034') can be omitted, which is not limited here.

[0081] S204: Detecting interface elements included in the graphical user interface to be tested.

[0082] S205 , input the big model based on the interface elements, target test cases, and preset callable function sets to obtain the test actions output by the big model.

[0083] Among them, the test action at least includes: the target interface element corresponding to the test action and the position of the target interface element in the graphical user interface to be tested, and the callable function corresponding to the test action. The purpose of the test action is to call the corresponding callable function on the target interface element to execute the test action and obtain the result returned by the callable function. In S205, through the large model, combined with the target test case and the available callable function set, the large model is input for analysis to obtain which test actions need to be completed under the current GUI to be tested in order to achieve the test requirements of the target test case, thereby breaking down the entire test path into multiple single-step test actions, and executing the test action can complete the target test case under the GUI to be tested and obtain the test results.

[0084] GUI refers to a computer operation user interface that is displayed in a graphical manner. GUI is an interface display format for users to communicate with application software. GUI contains multiple interface elements, and users can operate at least some of the interface elements to interact with the application software. In this step, the GUI to be tested can be obtained by running the application software on a device loaded with the application software and then taking a real-time screenshot. To verify a certain function of the GUI, it is necessary to locate the associated interface elements that need to be operated or read, so that the graphical user interface to be tested is first identified to obtain the interface elements contained in the GUI. Then, through the large model, combined with the target test case and the available callable function set, the large model is input for analysis to obtain which test actions need to be completed under the current GUI to be tested in order to achieve the test requirements of the target test case. Thus, the entire test path is disassembled into multiple single-step test actions. Executing the test actions can complete the target test case under the GUI to be tested and obtain the test results.

[0085] In some examples, interface elements include interactive interface elements and text interface elements. Interactive interface elements are interface elements included in the GUI that users can use to interact with application software; text interface elements are interface elements included in the GUI that are used for description. Because the characteristics of the two are different, two models can be used to identify the two types of interface elements respectively. S204 includes:

[0086] Step 1: Input the graphical user interface to be tested into the interactive element detection (GUIICON) model to obtain the interactive interface elements and their positions.

[0087] Step 2: Input the graphical user interface to be tested into the text extraction (OCR) model to obtain the text interface elements and their positions.

[0088] See also Figure 7 , Figure 7The following is a schematic diagram showing an exemplary graphical user interface for detecting interface elements. The recognition results of step 1 and step 2 include: the recognized interface elements, the rectangular boxes of the interface elements (used to indicate their positions), and the digital identifiers assigned to the rectangular boxes. For example Figure 7 In the detection interface shown, the graphical user interface includes five interface elements: Image 1, Figure 2 , sample text, button 1 and button 2. Each recognized interface element has a rectangular frame and a digital identifier in the upper left corner. This information will then be input into the large model, and the test action will be generated in combination with the large model. Although the large model has graphic analysis capabilities, due to the complexity of the GUI, the target recognition ability of the large model will usually produce recognition errors. Therefore, the interface elements can be identified first through pre-trained GUIICON models and OCR models, and then input into the large model. This can significantly improve the recognition accuracy and further ensure the accuracy of the location where the test action is performed on the GUI to be tested.

[0089] In some examples, the test action includes multiple steps; S205 includes:

[0090] Based on the interface elements, target test cases, and preset callable function sets, the big model is input to obtain multiple test actions output by the big model and the execution order of the multiple test actions.

[0091] In some examples, to complete a target test case, it is usually necessary to perform multiple test actions. The large model is used to integrate interface elements, target test cases and callable function sets, analyze which test actions are required to implement the target test case in the current GUI, and determine the operation location and corresponding callable function of each test action.

[0092] For example, see Figure 3 , the target test case to be tested: "Dialogue page → input the directions to the Forbidden City → normal display of welcome message and map history content"; see Figure 4 , the test actions that need to be performed are: get five execution actions: cold start application software (app) → click the dialog label (tab) → enter "how to get to the Forbidden City" in the main dialog input box → on the main dialog page, click the send button → on the current page, verify the existence of the welcome message. In other words, Figure 3 The target test case shown is "Dialogue page → input directions to the Forbidden City → normal display of welcome message and map history content", the identified interface elements, the callable function set input large model, the large model output test actions and their execution order: see Figure 4 , cold start the application software (app) → click the dialog tab (tab) → enter "how to get to the Forbidden City" in the input box of the main dialog → on the main dialog page, click the send button → on the current page, verify that there is a welcome message.

[0093] In some examples, the callable function set includes multiple callable functions. Multiple callable functions can be preset, each of which has a corresponding operation function. Exemplarily, the callable function set includes at least one of the following callable functions:

[0094] (1) The first callable function is used to click any interface element.

[0095] For example, the first callable function may be expressed as tap(element:int). The first callable function is used to click an interactive interface element displayed on the GUI, where "element" is a digital identifier assigned to the interactive interface element displayed on the GUI.

[0096] (2) The second callable function is used to input text for any interface element.

[0097] For example, the second callable function may be expressed as input(element:int, text_input:str). The second callable function is used for text input, where "element" is a digital identifier assigned to an interactive interface element displayed by the GUI, and "text_input" is an input string.

[0098] (3) A third callable function is used to represent the sliding trajectory of the finger on the graphical user interface to be tested.

[0099] For example, the third callable function can be expressed as swipe(direction:str). This third callable function is used to swipe the smartphone screen, indicating the path of a finger sliding across the screen. In some examples, the default sliding distance is half the screen. "direction" is a string representing one of four directions: up (swipe up), down (swipe down), left (swipe left), and right (swipe right).

[0100] (4) The fourth callable function is used to long press any interface element.

[0101] For example, the fourth callable function can be expressed as long_press(element:int). The fourth callable function is used to long press an interface element displayed on the GUI. "element" is a numeric identifier assigned to an interactive interface element displayed on the GUI.

[0102] (5) A fifth callable function is used to verify whether any interface element exists in the graphical user interface to be tested.

[0103] For example, the fifth callable function can be expressed as assert_exist(element:int, content:str). According to the target test case, the existence of a certain interface element on a GUI page needs to be verified. "content" represents the verification content. "element" is the numeric identifier assigned to the interactive interface element displayed by the GUI.

[0104] (6) A sixth callable function is used to verify that no interface element exists in the graphical user interface to be tested.

[0105] For example, the sixth callable function can be expressed as assert_not_exist(content:str). According to the target test case, it is necessary to verify the absence of a certain interface element on a GUI page. "content" represents the verification content. "element" is the numeric identifier assigned to the interactive interface element displayed by the GUI.

[0106] The above conditionally callable functions are merely exemplary descriptions and do not constitute a limitation, and the various callable functions may be omitted or combined.

[0107] It should be noted that S204-S205 is one implementation of S102. Currently, S102 also has other implementations, which are not limited here.

[0108] S206. Call the callable function corresponding to the test action in the execution order, execute the test action for the target interface element, and obtain the parameters returned by the callable function as the test result.

[0109] In S205, when the large model generates test actions, it assigns corresponding callable functions to each test action to implement the test action. In S206, the executor calls the corresponding functions in sequence according to the test actions and execution order output by the large model, i.e., the test path. The callable functions are executed on the target interface elements, and the parameters returned by the callable functions are obtained as test results, thereby achieving automatic testing of each test case of the GUI.

[0110] It should be noted that S206 is one implementation of S103. Currently, S103 also has other implementations, which are not limited here.

[0111] S207: Establish a mapping relationship among the target test case, the test action, the graphical user interface to be tested, and the test result, and save the mapping relationship in at least one of the case storage and the automatic test case library.

[0112] In the method provided by the present disclosure, each time a target test case is executed, a mapping relationship between the target test case, the test action, the graphical user interface to be tested, and the test result can be established, and the above data with the mapping relationship can be stored as historical data in at least one of the use case memory and the automatic test case library. After obtaining the tailored target test case, a similarity search is performed in the automatic test case library based on the historical test case to obtain a historical test case whose similarity to the target test case is higher than a preset threshold. Exemplarily, the target test case, the test action, the graphical user interface to be tested, and the test result can be stored in the automatic test case library in the form of a vector to facilitate retrieval and archiving; the original data of the target test case, the test action, the graphical user interface to be tested, and the test result can be stored in the use case memory to facilitate extended application based on the original data.

[0113] S208. Input the target test case, test action, graphical user interface to be tested, graphical user interface after automatic testing and test results into the big model, determine the correctness of the automatic test based on the big model, and obtain the correction suggestion information output by the big model.

[0114] In some cases, a correction mechanism is introduced to avoid errors during the generation and execution of test actions. Specifically, the target test case, test action, graphical user interface to be tested, the graphical user interface after the automated test, and the test results are input into a large model. The large model's generalization capabilities are then used to determine the correctness of the automated test—in other words, to determine whether the automated test progressed as expected. Based on the judgment result, correction suggestion information is generated. If the correctness is determined to be compliant, the correction suggestion can be left blank. The correction suggestion information indicates errors that occurred during the generation and execution of the test action, as well as how to correct them. This approach can avoid test errors and improve the accuracy of test results.

[0115] S209: Input the target test case, test action, graphical user interface to be tested, test results and preset test requirements into the big model, determine whether the automatic test meets the test requirements through the big model, and obtain the test requirement summary information output by the big model.

[0116] Requirements summary information indicates whether automated testing meets test requirements and its progress. The target test case, test action, the graphical user interface to be tested, test results, and pre-set test requirements are input into the macro model. Leveraging the macro model's generalization capabilities, it determines whether automated testing meets test requirements. It also records the progress of each test action, and finally generates test requirement summary information based on the judgment results and test progress. Recording the progress of each test action and whether it meets test requirements serves as guidance for the generation of subsequent test actions.

[0117] It should be noted that at least any one of steps S207-209 can be omitted, and the execution order of S207-S09 can be swapped, which is not limited here.

[0118] See also Figure 8 , Figure 8 The schematic diagram of the structure of an exemplary test system of the method provided by the present disclosure is shown in FIG. The method of the present disclosure can be applied in, for example, Figure 8 The test system shown in the figure includes a judgment and optimization module, a retrieval module, and a generation module. In some examples, the retrieval module can be omitted. The judgment and optimization module includes a judger and a cropper; the retrieval module includes a vector retriever and an executor, and is linked to an automated test case set; the generation module includes a visual parser, an action generator, an executor, a case storage, a corrector, and a summarizer, and is connected to a device that loads application software.

[0119] Among them, in the judgment optimization module:

[0120] The judge is used to filter out target test cases that can be used for automatic testing from a preset test case set based on preset filtering conditions.

[0121] The trimmer is used to input the target test case and preset prompt words into the large model, and remove invalid expressions in the target test case through the large model to obtain the trimmed target test case.

[0122] In the search module:

[0123] The vector retriever is used to perform similarity retrieval on the historical test cases contained in the preset automatic test case library based on the target test case, and obtain historical test cases whose similarity with the target test case is higher than a preset threshold.

[0124] The executor is used to automatically test the graphical user interface to be tested through historical test actions corresponding to historical test cases, determine untested parts of the graphical user interface to be tested, and input them into the generation module.

[0125] In the generated module:

[0126] The visual parser is used to detect the interface elements contained in the graphical user interface to be tested.

[0127] The action generator is used to input the large model based on interface elements, target test cases, and preset callable function sets to obtain test actions output by the large model.

[0128] The executor is used to call the callable function corresponding to the test action in the execution order, perform the test action on the target interface element, and obtain the parameters returned by the callable function as the test result.

[0129] The use case memory is used to establish a mapping relationship among target test cases, test actions, the graphical user interface to be tested and the test results, and save the mapping relationship in the use case memory.

[0130] The corrector is used to input the target test cases, test actions, the graphical user interface to be tested, the graphical user interface after the automatic test and the test results into the big model, determine the correctness of the automatic test based on the big model, and obtain the correction suggestion information output by the big model.

[0131] The summarizer is used to input the target test cases, test actions, graphical user interfaces to be tested, test results and preset test requirements into the big model, determine whether the automatic test meets the test requirements through the big model, and obtain the test requirement summary information output by the big model.

[0132] In the third embodiment disclosed, based on Figure 1 The same principle, Figure 9 A graphical user interface testing device 90 provided in a third embodiment of the present disclosure is shown, the device comprising:

[0133] A cutting module 901 is used to cut the target test case to obtain a cut target test case;

[0134] A generating module 902 is configured to generate test actions based on the graphical user interface to be tested and the tailored target test case;

[0135] The testing module 903 is used to execute a testing action to automatically test the graphical user interface to be tested and obtain a test result.

[0136] In some examples, the generation module is specifically configured to:

[0137] Detect the interface elements contained in the graphical user interface to be tested;

[0138] Input the big model based on the interface elements, target test cases, and preset callable function sets to obtain the test actions output by the big model;

[0139] The test action at least includes: a target interface element corresponding to the test action and the position of the target interface element in the graphical user interface to be tested, and a callable function corresponding to the test action.

[0140] In some examples, the test action includes multiple;

[0141] The generation module is used to input the large model based on the interface elements, target test cases, and preset callable function sets, and obtain the test actions output by the large model.

[0142] Based on the interface elements, target test cases, and preset callable function sets, the big model is input to obtain multiple test actions output by the big model and the execution order of the multiple test actions.

[0143] In some examples, the interface elements include interactive interface elements and textual interface elements;

[0144] The generation module detects the interface elements contained in the graphical user interface to be tested and is used to:

[0145] Input the graphical user interface to be tested into the interactive element detection model to obtain the interactive interface elements and their positions;

[0146] The graphical user interface to be tested is input into the text extraction model to obtain the text interface elements and their positions.

[0147] In some examples, the set of callable functions includes at least one of the following callable functions:

[0148] The first callable function is used to click any interface element;

[0149] The second callable function is used to input text for any interface element;

[0150] A third callable function is used to represent a sliding trajectory of a finger on the graphical user interface to be tested;

[0151] The fourth callable function is used to long press any interface element;

[0152] A fifth callable function is used to verify that any interface element exists in the graphical user interface to be tested;

[0153] The sixth callable function is used to verify that no interface element exists in the graphical user interface to be tested.

[0154] In some examples, the test module is specifically used to:

[0155] In accordance with the execution order, the callable function corresponding to the test action is called, the test action is executed for the target interface element, and the parameters returned by the callable function are obtained as the test result.

[0156] In some examples, the cropping module is used to:

[0157] The target test case and the preset prompt words are input into the large model, and the invalid expressions in the target test case are removed by the large model to obtain the trimmed target test case.

[0158] In some examples, the apparatus further includes:

[0159] A retrieval module is used to perform similarity retrieval on historical test cases contained in a preset automatic test case library based on the target test case, and obtain historical test cases whose similarity to the target test case is higher than a preset threshold;

[0160] Automatically test the graphical user interface to be tested through historical test actions corresponding to historical test cases;

[0161] An untested portion of the graphical user interface to be tested is determined, and a test case based on the graphical user interface to be tested and the tailored target test case is executed based on the untested portion to generate a test action.

[0162] In some examples, the apparatus further includes:

[0163] The verification module is used to input the target test case and the historical test case into the big model, and verify whether the description of the target test case matches the description of the historical test case through the big model;

[0164] If a match is found, the historical test actions corresponding to the historical test cases are executed, and the graphical user interface to be tested is automatically tested.

[0165] In some examples, the apparatus further includes:

[0166] The storage module is used to establish a mapping relationship among target test cases, test actions, graphical user interfaces to be tested, and test results, and store the mapping relationship in at least one of a case storage and an automatic test case library.

[0167] In some examples, the apparatus further includes:

[0168] The correction module is used to input the target test case, test action, graphical user interface to be tested, graphical user interface after the automatic test and test results into the big model, determine the correctness of the automatic test based on the big model, and obtain the correction suggestion information output by the big model.

[0169] In some examples, the apparatus further includes:

[0170] A module is required to input the target test cases, test actions, graphical user interfaces to be tested, test results and preset test requirements into the big model, determine whether the automatic test meets the test requirements through the big model, and obtain the test requirement summary information output by the big model.

[0171] In some examples, the apparatus further includes:

[0172] The screening module is used to screen out target test cases that can be used for automatic testing from a preset test case set based on preset screening conditions.

[0173] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0174] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0175] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0176] like Figure 10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from the storage unit 1002 into a random access memory (RAM) 1003. Various programs and data required for the operation of the device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0177] Various components in device 1000 are connected to I / O interface 1005, including an input unit 1006, such as a keyboard, mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, optical disk, etc.; and a communication unit 1009, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1009 allows device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0178] The computing unit 1001 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as the testing method for a graphical user interface. For example, in some embodiments, the testing method for a graphical user interface can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the testing method for the graphical user interface described above can be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured with the graphical user interface testing method in any other appropriate manner (eg, by means of firmware).

[0179] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0180] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0181] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0182] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0183] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0184] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0185] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0186] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for testing a graphical user interface, wherein: The method comprises: Tailoring the target test case to obtain the tailored target test case; wherein the target test case and preset prompt words are input into a large model, and invalid expressions in the target test case are removed by the large model to obtain the tailored target test case; Generate test actions based on the graphical user interface to be tested and the tailored target test case; Executing the test action to automatically test the graphical user interface to be tested, and obtaining a test result; Wherein, after the target test case is tailored to obtain the tailored target test case, and before the test action is generated based on the graphical user interface to be tested and the tailored target test case, the method further includes: Based on the target test case, similarity retrieval is performed on historical test cases contained in a preset automatic test case library to obtain historical test cases whose similarity to the target test case is higher than a preset threshold; Automatically test the graphical user interface to be tested through historical test actions corresponding to historical test cases; An untested portion of the graphical user interface to be tested is determined, and based on the untested portion, the target test case based on the graphical user interface to be tested and the tailored test case is executed to generate a test action.

2. The method according to claim 1, wherein The generating of test actions based on the graphical user interface to be tested and the tailored target test case includes: Detecting interface elements included in the graphical user interface to be tested; Inputting a large model based on the interface elements, the target test cases, and a preset callable function set to obtain a test action output by the large model; The test action at least includes: a target interface element corresponding to the test action and a position of the target interface element in the graphical user interface to be tested, and a callable function corresponding to the test action.

3. The method according to claim 2, wherein: The test actions include multiple; The step of inputting a large model based on the interface elements, the target test case, and a preset callable function set to obtain a test action output by the large model includes: The large model is input based on the interface elements, the target test cases, and the preset callable function set to obtain the multiple test actions output by the large model and the execution order of the multiple test actions.

4. The method according to claim 2 or 3, wherein: The interface elements include interactive interface elements and text interface elements; The detecting the interface elements included in the graphical user interface to be tested includes: Inputting the graphical user interface to be tested into an interactive element detection model to obtain the interactive interface elements and their positions; The graphical user interface to be tested is input into a text extraction model to obtain the text interface elements and their positions.

5. The method according to any one of claims 2-3, wherein: The callable function set includes at least one of the following callable functions: A first callable function is used to click any of the interface elements; A second callable function is used to input text for any of the interface elements; A third callable function is used to represent a sliding trajectory of a finger on the graphical user interface to be tested; A fourth callable function is used to long press any of the interface elements; A fifth callable function, configured to verify that any of the interface elements exists in the graphical user interface to be tested; The sixth callable function is used to verify that none of the interface elements exist in the graphical user interface to be tested.

6. The method according to any one of claims 2-3, wherein: The executing the test action to automatically test the graphical user interface to be tested and obtain a test result includes: In accordance with the execution order, the callable function corresponding to the test action is called, the test action is executed for the target interface element, and the parameters returned by the callable function are obtained as the test result.

7. The method according to claim 1, wherein After performing similarity retrieval on historical test cases contained in a preset automatic test case library based on the target test case to obtain historical test cases having a similarity with the target test case greater than a preset threshold, and before automatically testing the graphical user interface to be tested using historical test actions corresponding to the historical test cases, the method further includes: Inputting the target test case and the historical test case into a large model, and verifying whether the description of the target test case matches the description of the historical test case through the large model; In the case of a match, the historical test action corresponding to the historical test case is executed, and the graphical user interface to be tested is automatically tested.

8. The method according to any one of claims 1 to 3, wherein: After automatically testing the graphical user interface to be tested by executing the test action and obtaining the test result, the method further includes: A mapping relationship among the target test case, the test action, the graphical user interface to be tested, and the test result is established and stored in at least one of a case memory and an automatic test case library.

9. The method according to any one of claims 1 to 3, wherein: After automatically testing the graphical user interface to be tested by executing the test action and obtaining the test result, the method further includes: The target test case, the test action, the graphical user interface to be tested, the graphical user interface after the automatic test and the test result are input into the big model, the correctness of the automatic test is determined based on the big model, and the correction suggestion information output by the big model is obtained.

10. The method according to any one of claims 1 to 3, wherein: After automatically testing the graphical user interface to be tested by executing the test action and obtaining the test result, the method further includes: The target test case, the test action, the graphical user interface to be tested, the test results and the preset test requirements are input into the big model, and the big model is used to determine whether the automatic test meets the test requirements, and the test requirement summary information output by the big model is obtained.

11. The method according to any one of claims 1 to 3, wherein: Before tailoring the target test case to obtain the tailored target test case, the method further includes: Based on the preset filtering conditions, the target test cases that can be used for automatic testing are filtered out from the preset test case set.

12. A graphical user interface testing device, wherein: The device comprises: A cutting module, used for cutting the target test case to obtain the cut target test case; A generation module, configured to generate test actions based on the graphical user interface to be tested and the tailored target test case; A testing module, configured to execute the test action to automatically test the graphical user interface to be tested and obtain a test result; The cutting module is used to: Inputting the target test case and preset prompt words into a large model, removing invalid expressions in the target test case through the large model, and obtaining the tailored target test case; The device further comprises: A retrieval module is configured to perform a similarity search on historical test cases contained in a preset automatic test case library based on the target test case, and obtain historical test cases whose similarity to the target test case is higher than a preset threshold; Automatically test the graphical user interface to be tested through historical test actions corresponding to historical test cases; An untested portion of the graphical user interface to be tested is determined, and based on the untested portion, the target test case based on the graphical user interface to be tested and the tailored test case is executed to generate a test action.

13. The device according to claim 12, wherein The generation module is specifically used for: Detecting interface elements included in the graphical user interface to be tested; Inputting a large model based on the interface elements, the target test cases, and a preset callable function set to obtain a test action output by the large model; The test action at least includes: a target interface element corresponding to the test action and a position of the target interface element in the graphical user interface to be tested, and a callable function corresponding to the test action.

14. The device according to claim 13, wherein The test actions include multiple; The generation module is used to input a large model based on the interface elements, the target test case, and a preset callable function set to obtain a test action output by the large model: The large model is input based on the interface elements, the target test cases, and the preset callable function set to obtain the multiple test actions output by the large model and the execution order of the multiple test actions.

15. The device according to claim 13 or 14, wherein The interface elements include interactive interface elements and text interface elements; The generation module detects the interface elements included in the graphical user interface to be tested, and is used to: Inputting the graphical user interface to be tested into an interactive element detection model to obtain the interactive interface elements and their positions; The graphical user interface to be tested is input into a text extraction model to obtain the text interface elements and their positions.

16. The device according to any one of claims 13-14, wherein: The callable function set includes at least one of the following callable functions: A first callable function is used to click any of the interface elements; A second callable function is used to input text for any of the interface elements; A third callable function is used to represent a sliding trajectory of a finger on the graphical user interface to be tested; A fourth callable function is used to long press any of the interface elements; A fifth callable function, configured to verify that any of the interface elements exists in the graphical user interface to be tested; The sixth callable function is used to verify that none of the interface elements exist in the graphical user interface to be tested.

17. The device according to any one of claims 13-14, wherein: The test module is specifically used for: In accordance with the execution order, the callable function corresponding to the test action is called, the test action is executed for the target interface element, and the parameters returned by the callable function are obtained as the test result.

18. The device according to any one of claims 12 to 14, wherein: The device further comprises: A verification module, configured to input the target test case and the historical test case into a large model, and verify through the large model whether the description of the target test case matches the description of the historical test case; In the case of a match, the historical test action corresponding to the historical test case is executed, and the graphical user interface to be tested is automatically tested.

19. The device according to any one of claims 12 to 14, wherein: The device further comprises: A storage module is used to establish a mapping relationship among the target test case, the test action, the graphical user interface to be tested and the test result, and store the mapping relationship in at least one of a case memory and an automatic test case library.

20. The device according to any one of claims 12 to 14, wherein: The device further comprises: The correction module is used to input the target test case, the test action, the graphical user interface to be tested, the graphical user interface after the automatic test and the test results into the big model, determine the correctness of the automatic test based on the big model, and obtain the correction suggestion information output by the big model.

21. The device according to any one of claims 12 to 14, wherein: The device further comprises: A module is required, which is used to input the target test case, the test action, the graphical user interface to be tested, the test results and the preset test requirements into the big model, determine whether the automatic test meets the test requirements through the big model, and obtain the test requirement summary information output by the big model.

22. The device according to any one of claims 12 to 14, wherein: The device further comprises: The screening module is used to screen out target test cases that can be used for automatic testing from a preset test case set based on preset screening conditions.

23. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-11.

25. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images