Application gui testing method, system, electronic device, and storage medium

Abstract

The application discloses an application program GUI test method and system, electronic equipment and storage medium, and relates to the technical field of mobile application testing. The method comprises the following steps: determining whether a current GUI image has defects; if yes, storing the defect type in the current track memory, starting the next track memory, and determining whether a test stop condition is reached; if not, determining whether the current track memory is cut off; if yes, starting the next track memory, determining whether the test stop condition is reached; if not or the test stop condition is not reached, searching for important memories from a memory warehouse; inputting the important memories, image information memories, target settings and exploration tendencies into a multimodal large model to obtain a current exploration tendency and a current operation, adding the current exploration tendency and the current operation into the memory warehouse and the current track memory respectively, and applying the current operation to the current GUI; and updating the current GUI image into a next GUI image and performing a test. The application improves the efficiency of application program GUI testing.

Description

Technical Field

[0001] This invention relates to the field of mobile application testing technology, and in particular to an application GUI testing method, system, electronic device, and storage medium. Background Technology

[0002] Currently, deep learning technology has many different approaches in the field of Graphical User Interface (GUI) testing. Li Yuanchun et al. proposed a deep learning-based automated Android application testing method that learns test sequence generation from human-generated interaction event sequences. David Adamo et al. proposed a reinforcement learning-based automated Android application GUI testing method that uses a Q-learning-based test generation algorithm to select events and interpret the GUI of the application under test. Yavuz Koroglu et al. proposed FARLEAD-Android, which uses reinforcement learning to generate tests that satisfy a given linear temporal logic specification. However, these methods do not identify and locate GUI graphical elements, and cannot accurately manipulate these elements like human testers, potentially leading to accidental operations on invalid areas and test redundancy. Furthermore, these methods do not detect or classify GUI defects, affecting the efficiency and quality of automated test case generation. Summary of the Invention

[0003] The purpose of this invention is to provide an application GUI testing method, system, electronic device, and storage medium, which improves the automation level and efficiency of application GUI testing.

[0004] To achieve the above objectives, the present invention provides the following solution:

[0005] An application GUI testing method includes:

[0006] Use multiple defective GUI images and their corresponding defect types as prompts to fine-tune the multimodal large model;

[0007] Obtain the current GUI image and its corresponding information; the current GUI image is the image of the application's current GUI.

[0008] The current GUI image is assigned a current number, and the current number, the current GUI image, and the current image information are stored in a memory repository; the memory repository includes trajectory memory, image memory, image information memory, and strategies; the strategies include: goal setting, exploration tendency, and operation;

[0009] Based on the current GUI image and the multimodal large model, determine whether there are defects in the current GUI image to obtain a first judgment result;

[0010] If the first judgment result is yes, store the corresponding defect type in the current trajectory memory, end the current trajectory memory, start the next trajectory memory, update the current trajectory memory to the next trajectory memory, and determine whether the test stop condition is reached to obtain a third judgment result;

[0011] If the first judgment result is no, determine whether the current trajectory memory has ended to obtain a second judgment result;

[0012] If the second judgment result is yes, end the current trajectory memory, start the next trajectory memory, update the current trajectory memory to the next trajectory memory, and determine whether the test stop condition is reached to obtain a third judgment result;

[0013] If the third judgment result is yes, stop the test;

[0014] If the third judgment result is no or the second judgment result is no, then:

[0015] Search for the important memory of the current GUI image in the memory warehouse;

[0016] Input the important memory, the image information memory, the target setting, and the exploration tendency as prompts into the multimodal large model to obtain the current exploration tendency and the current operation, add the current exploration tendency to the strategy of the memory warehouse, store the current operation in the current trajectory memory, and apply the current operation to the current GUI;

[0017] Update the current GUI image to the next GUI image and return "Obtain the current GUI image and the corresponding current image information".

[0018] Optionally, the image information corresponding to the image of any GUI includes: component elements, interaction logic, and attributes and states.

[0019] Optionally, based on the current GUI image and the multimodal large model, determining whether there are defects in the current GUI image to obtain a first judgment result specifically includes:

[0020] Input the current GUI image into the multimodal large model and input the prompt "<Is there a defect in the GUI image number>? If so, what is the defect?" Thus, determine whether there are defects in the current GUI image to obtain a first judgment result.

[0021] Optionally, search for important memories of the current GUI image from the memory repository, including:

[0022] Retrieve the image information memory, the target setting, and the exploration tendency corresponding to the current GUI image from the memory repository, perform an importance score on the trajectory memory, and determine all trajectory memories with importance scores in the top preset rankings as important memories.

[0023] Optionally, store the current number, the current GUI image, and the current image information in the memory repository, including:

[0024] Store the current number in the current trajectory memory in the memory repository, store the current GUI image in the image memory in the memory repository, and store the current image information in the image information memory in the memory repository.

[0025] Optionally, any trajectory memory includes one or more natural languages corresponding to GUIs, and the natural language is: "Perform <operation> at <position> on <GUI number> to obtain <GUI number>, <GUI number> is defect-free / has defects, and the defect is <defect description>.

[0026] An application GUI testing system includes:

[0027] A fine-tuning module for fine-tuning a multimodal large model with multiple defective GUI images and corresponding defect types as prompts;

[0028] An image acquisition module for acquiring the current GUI image and the corresponding current image information; the current GUI image is an image of the current GUI of the application;

[0029] A storage module for assigning a current number to the current GUI image and storing the current number, the current GUI image, and the current image information in the memory repository; the memory repository includes trajectory memory, image memory, image information memory, and strategies; the strategies include: target setting, exploration tendency, and operations;

[0030] A first judgment module for judging whether the current GUI image has defects according to the current GUI image and the multimodal large model, and obtaining a first judgment result;

[0031] A first execution module for, if the first judgment result is yes, storing the corresponding defect type in the current trajectory memory, ending the current trajectory memory, starting the next trajectory memory, updating the current trajectory memory to the next trajectory memory, and judging whether the test stop condition is reached, and obtaining a third judgment result;

[0032] The second execution module is used to determine whether the current trajectory memory has expired if the first judgment result is negative, and to obtain a second judgment result.

[0033] The third execution module is used to, if the second judgment result is yes, terminate the current trajectory memory, start the next trajectory memory, update the current trajectory memory to the next trajectory memory, and determine whether the test stop condition has been met, and obtain the third judgment result;

[0034] The fourth execution module is used to stop the test if the third judgment result is yes;

[0035] The fifth execution module is configured to, if the third judgment result is negative or the second judgment result is negative, then:

[0036] Search the memory repository for important memories of the current GUI image;

[0037] The important memories, the image information memories, the target settings, and the exploration tendencies are input into the multimodal large model as prompts to obtain the current exploration tendencies and current operations. The current exploration tendencies are added to the strategy of the memory warehouse, the current operations are stored in the current trajectory memory, and the current operations are applied to the current GUI.

[0038] Update the current GUI image to the next GUI image and return to the image acquisition module.

[0039] An electronic device, comprising:

[0040] One or more processors;

[0041] A storage device on which one or more programs are stored;

[0042] When the one or more programs are executed by the one or more processors, the one or more processors implement the application GUI testing method as described above.

[0043] A storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the application GUI testing method as described above.

[0044] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0045] This invention discloses an application GUI testing method, system, electronic device, and storage medium. By storing the application's GUI image, corresponding GUI information, standard natural language format trajectories, and strategies in memory, and retrieving important memories to input into a multimodal large model, the invention outputs exploration tendencies. These exploration tendencies guide the application of operations to the GUI, enabling targeted testing of the application like a human. Furthermore, the invention allows for the input of relevant text into input boxes, significantly improving testing efficiency. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a schematic diagram of the application GUI testing method provided in Embodiment 1 of the present invention;

[0048] Figure 2 Flowchart of a method for intelligent testing of application GUIs based on multimodal large models

[0049] Figure 3 A diagram illustrating the mobile application state update process;

[0050] Figure 4 This is a schematic diagram of the architecture of an intelligent testing system for application GUI based on a multimodal large model. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] The purpose of this invention is to provide an application GUI testing method, system, electronic device, and storage medium, aiming to improve the automation level and efficiency of application GUI testing.

[0053] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] Example 1

[0055] Figure 1This is a schematic diagram of the application GUI testing method provided in Embodiment 1 of the present invention. Figure 1 As shown, the application GUI testing method in this embodiment includes:

[0056] Step 1: Use multiple defective GUI images and their corresponding defect types as prompts to fine-tune the multimodal large model.

[0057] Step 2: Obtain the current GUI image and its corresponding information.

[0058] The current GUI image is the image of the application's current GUI.

[0059] Step 3: Assign the current number to the current GUI image, and store the current number, the current GUI image, and the current image information in the memory repository.

[0060] The memory repository includes trajectory memory, image memory, image information memory, and strategies; strategies include: goal setting, exploration tendencies, and operations.

[0061] Step 4: Determine if the current GUI image has defects and obtain the first judgment result.

[0062] Step 4 specifically involves determining whether the current GUI image has defects based on the current GUI image and the multimodal large model, and obtaining the first judgment result.

[0063] If the first judgment result is yes, then proceed to step 5 and determine whether the test stop condition has been met to obtain the third judgment result.

[0064] Step 5 includes: storing the corresponding defect type into the current trajectory memory, terminating the current trajectory memory, starting the next trajectory memory, and updating the current trajectory memory to the next trajectory memory.

[0065] If the first judgment result is negative, then determine whether the current trajectory memory has expired, and obtain the second judgment result.

[0066] If the second judgment result is yes, then proceed to step 6 and determine whether the test stop condition has been met to obtain the third judgment result.

[0067] Step 6 includes: terminating the current trajectory memory, starting the next trajectory memory, and updating the current trajectory memory to the next trajectory memory.

[0068] If the third judgment result is yes, then stop the test.

[0069] If the third judgment result is negative or the second judgment result is negative, then proceed to step 7 and return to step 2.

[0070] Step 7 includes:

[0071] Search for important memories of the current GUI image from the memory repository.

[0072] Input important memories, image information memories, target settings, and exploration tendencies as prompts into the multi-modal large model to obtain the current exploration tendency and the current operation. Add the current exploration tendency to the strategy in the memory repository, store the current operation in the current trajectory memory, and apply the current operation to the current GUI.

[0073] Update the current GUI image to the next GUI image.

[0074] As an optional implementation manner, the image information corresponding to the image of any GUI includes: component elements, interaction logic, and attributes and states.

[0075] As an optional implementation manner, the multi-modal large model is the CPM-Bee-1B model.

[0076] As an optional implementation manner, based on the current GUI image and the multi-modal large model, determine whether there are defects in the current GUI image to obtain the first judgment result, specifically including:

[0077] Input the current GUI image into the multi-modal large model and input the prompt "Are there any defects in <GUI image number>? If so, what are the defects?" Thus, determine whether there are defects in the current GUI image to obtain the first judgment result.

[0078] As an optional implementation manner, searching for important memories of the current GUI image from the memory repository includes:

[0079] Retrieve the image information memory, target setting, and exploration tendency corresponding to the current GUI image from the memory repository, perform an importance scoring on the trajectory memory, and determine all trajectory memories with importance scores in the top preset ranking as important memories.

[0080] Specifically, the scoring method includes: assuming the current GUI image is GUI1, then the weight of GUI1 is 1. Some GUIs can be converted to GUI1 through one operation, then the weights of these GUIs are 0.9. Some GUIs can be converted to GUI1 through two operations, then the weights of these GUIs are 0.9 2 , and the weights of those GUIs that require three steps to convert to GUI1 are 0.9 3 , and so on. Finally, add up the weights of the GUIs in each trajectory memory, multiply the trajectory score of the discovered defect by a weight of 1.2, so as to obtain the importance score of each trajectory memory. By default, take the trajectory memories with the top 10 highest scores (i.e., the top preset ranking), and the weights and the top preset ranking can be adjusted according to the actual situation.

[0081] As an optional implementation, storing the current number, the current GUI image, and the current image information in the memory repository includes:

[0082] Storing the current number in the current trajectory memory in the memory repository, storing the current GUI image in the image memory in the memory repository, and storing the current image information in the image information memory in the memory repository.

[0083] As an optional implementation, any trajectory memory includes the natural language corresponding to one or more GUIs, and the natural language is: "Performing <operation> at <position> on <GUI number> results in <GUI number>, <GUI number> is defect-free / has defects, and the defect is <defect description>.

[0084] In addition to the above application program GUI testing method, as Figure 2 shown, another implementation is provided, that is, a method for intelligent testing of application program GUIs based on a multimodal large model, including:

[0085] Step 101: Fine-tuning the multimodal large model with the defect types of the GUI and the GUI images with defects as prompts.

[0086] Step 102: Obtaining the GUI images and corresponding image information of the mobile application.

[0087] Step 103: Numbering the GUI images and the corresponding image information and storing them in the memory repository, where the memory repository includes trajectory memory, image memory, image information memory, and strategy.

[0088] Actually, the number refers to the unique number corresponding to each GUI image. The significance of numbering: a. In step 104, what is actually input is the number corresponding to the image, rather than the image itself, which can reduce the input content. Through the interface, the multimodal large model can index the corresponding image according to the number. b. The number is recorded in the trajectory memory. First, the memory repository can be considered as a main folder, which contains 4 sub-folders, namely trajectory memory, image memory, image information memory, and strategy. The strategy folder contains 3 sub-folders, namely target setting, exploration tendency, and operation. Among them, the target setting is set by humans, and the contents of the other folders in the memory repository are initially empty. During the testing process, every time a GUI is experienced, it will be stored in the trajectory memory, image memory, and image information memory. The exploration tendency and operation in the strategy are the outputs of the multimodal large model.

[0089] Step 104: Inputting the numbered GUI images into the multimodal large model to determine whether there are defects. If so, record the defects and store the defect description in the trajectory memory.

[0090] There are two scenarios for trajectory termination: 1. If a defect is found, the trajectory terminates. In this case, the GUI where the defect is found is the last GUI of the current trajectory, and the next GUI is used as the start of a new trajectory; 2. If a duplicate GUI appears, the trajectory terminates. In this case, the duplicate GUI is the last GUI of the current trajectory.

[0091] Step 105: Determine whether the trajectory termination condition has been met. If yes, the trajectory terminates and proceeds to step 106. If no, proceed to step 107.

[0092] Step 106: Determine whether the test stop condition has been met (the test stop condition can be set according to the test objective, such as "no new GUI appears within 10 minutes"). If yes, the test stops; if no, the current trajectory memory expires, a new trajectory memory is started, and the process jumps to step 107.

[0093] Step 107: Search the memory repository for important memories of the current GUI image.

[0094] Searching memories includes: directly retrieving current image information memories, goal setting, exploration tendencies, scoring the importance of trajectory memories, with higher scores for trajectories that are highly relevant to the current GUI image, higher scores for trajectories that discover defects, and higher scores for longer trajectories, and retrieving memories with high importance scores.

[0095] Step 108: Input important memories, current image information memories, goal setting, and exploration tendencies as prompts into the multimodal large model. The multimodal large model outputs exploration tendencies and actions.

[0096] The strategy includes goal setting, exploration tendencies, and operations. Goal setting is manually set, such as "find as many GUI defects as possible"; exploration tendencies are output from the multimodal large model and are used to guide the focus of testing, such as "check the responsiveness and interactivity of the interface"; operations are also output from the large model and are used to perform single-step operations on the GUI, such as "click on the back button / double-click on ** pixels".

[0097] Step 109: Update the exploration tendency output by the multimodal large model to the policy and store the operation in the trajectory memory. That is, the exploration tendency is a JSON file under the policy folder. Every time the large model outputs an exploration tendency, add that exploration tendency to the JSON file. This is the update. Each test uses the latest exploration tendency to guide the process.

[0098] Step 110: Apply the operations of the multimodal large model output to the current GUI and transform it into a new GUI.

[0099] Step 111: Return "Obtain the GUI images of the mobile application and the corresponding image information". The iteration stops only when the test stop condition is met. As for whether the test stop condition is met, it is for the large model to judge by itself.

[0100] Preferably, the image information includes view hierarchy information such as component elements, attributes and states, and interaction logic.

[0101] Preferably, the GUI images are stored in the image memory in the memory warehouse, and the image information is stored in the image information memory in the memory warehouse.

[0102] Preferably, the trajectory memory includes a series of GUI image numbers, actions, positions where actions are applied, and defect situations. The trajectory memory is composed of a standard natural language format, which is "Performing <operation> at <position> on <GUI number> results in <GUI number>, <GUI number> is defect-free / has defects, and the defect is <defect description>."

[0103] Preferably, the actions include click, double-click, long-press, swipe, and input; the swipe includes swiping left, right, up, and down, and the position where the action is applied includes coordinate points and component elements.

[0104] Preferably, the component elements include: text buttons, icons, radio buttons, check boxes, sliders, switches, page indicators, input boxes; the defects include: display problems, function problems, and user experience problems.

[0105] Preferably, the strategy includes goal setting, exploration tendency, and operations; the goal setting includes test goals, test stop conditions, and trajectory cut-off conditions. The trajectory cut-off condition is that duplicate GUIs appear in the current trajectory and defects are found. The goal setting is set manually; the exploration tendency is the operation tendency for the current GUI trajectory; the operations include actions and positions where actions are applied.

[0106] Preferably, the search memory includes: directly retrieving the current image information memory, goal setting, and exploration tendency, performing an importance score on the trajectory memory, with higher scores for trajectories that are more relevant to the current GUI image, higher scores for trajectories where defects are found, and higher scores for longer trajectories, and retrieving memories with high importance scores.

[0107] As Figure 3 shown, a flowchart for updating the state of the mobile application is also provided:

[0108] Step 1, Connect the mobile application automation testing tool Appium and the virtual device Android Virtual Device (AVD) through Pycharm. Take AVD with Android 7.0 as an example. Start and connect the application in AVD and enter the home page of the application.

[0109] Step 2: Take a screenshot of the current page of the application using the driver.get_screenshot_as_file function in Python, and obtain the view hierarchy information of the current page using the driver.getPageSource() method of Appium.

[0110] Step 3: Number the screenshot and the view hierarchy information and store them in the memory repository. The memory repository is stored in the form of a folder. The screenshot is named with the number and stored in the image memory folder, and the view hierarchy information is stored in the image information memory folder in JSON format.

[0111] Step 4: Input the screenshot into the fine-tuned CPM-Bee-1B and input the prompt "Is there a defect in <GUI image number>? If so, what is the defect?", and determine whether there is a defect. If there is no defect, go to Step 5. If there is a defect, output the description content of the defect, store it in the trajectory txt file in the trajectory memory folder, and go to Step 6.

[0112] Step 5: Input the prompt "Has the trajectory cut-off condition been reached?", if yes, go to Step 6, if no, go to Step 7.

[0113] Step 6: The current trajectory memory ends, start a new trajectory memory, input the prompt "Has the test stop condition been reached?", if yes, the test stops.

[0114] Step 7: Retrieve the current image information memory, target setting, and exploration tendency from the memory repository, score the importance of the trajectory memory sequence according to the coordinate values of the normal distribution, and retrieve the trajectory memory with the top 10 scores.

[0115] Step 8: Input the retrieved memory as a prompt into the multi-modal large model, and input the prompt "Update the current exploration tendency and give the next operation according to the exploration tendency", and the multi-modal large model outputs the updated exploration tendency and the next operation.

[0116] Step 9: Update the exploration tendency output by the multi-modal large model with a serial number to the strategy.

[0117] Step 10: Store the operation output by the multi-modal large model in the trajectory memory, and apply the operation to the current GUI to switch to a new GUI.

[0118] Step 11: Go back to Step 3. The goal is to traverse the GUI and components of the application to the greatest extent, detect GUI defects to the greatest extent, and generate a JSON file for recording GUI defects.

[0119] Example 2

[0120] The application GUI testing system in this embodiment includes:

[0121] The fine-tuning module is used to fine-tune the multimodal large model by using multiple defective GUI images and their corresponding defect types as prompts.

[0122] The image acquisition module is used to acquire the current GUI image and its corresponding information; the current GUI image is the image of the application's current GUI.

[0123] The storage module is used to assign a current number to the current GUI image and store the current number, the current GUI image, and the current image information into the memory warehouse. The memory warehouse includes trajectory memory, image memory, image information memory, and strategies. The strategies include: goal setting, exploration tendency, and operation.

[0124] The first judgment module is used to determine whether there are defects in the current GUI image based on the current GUI image and the multimodal large model, and obtain the first judgment result.

[0125] The first execution module is used to, if the first judgment result is yes, store the corresponding defect type into the current trajectory memory, close the current trajectory memory, open the next trajectory memory, update the current trajectory memory to the next trajectory memory, and determine whether the test stop condition has been met, and obtain the third judgment result.

[0126] The second execution module is used to determine whether the current trajectory memory has expired if the first judgment result is negative, and to obtain the second judgment result.

[0127] The third execution module is used to, if the second judgment result is yes, terminate the current trajectory memory, start the next trajectory memory, update the current trajectory memory to the next trajectory memory, and determine whether the test stop condition has been met, and obtain the third judgment result.

[0128] The fourth execution module is used to stop the test if the result of the third judgment is yes.

[0129] The fifth execution module is used to: if the result of the third judgment is negative or the result of the second judgment is positive, then:

[0130] Search the memory repository for important memories of the current GUI image.

[0131] Important memories, image information memories, goal settings, and exploration tendencies are input into the multimodal large model as prompts to obtain the current exploration tendency and current operation. The current exploration tendency is added to the strategy in the memory warehouse, the current operation is stored in the current trajectory memory, and the current operation is applied to the current GUI.

[0132] Update the current GUI image to the next GUI image and return to the image acquisition module.

[0133] In addition to the aforementioned application GUI testing system, such as Figure 4 As shown, another implementation method is also provided, namely, an intelligent testing system for application GUI based on a multimodal large model, including:

[0134] The prompt fine-tuning module 201 is used to fine-tune the multimodal large model by taking the types of defects in the GUI and the GUI images with defects as prompts.

[0135] The GUI image and corresponding image information acquisition module 202 is used to acquire the GUI image and corresponding image information of the mobile application.

[0136] The memory storage module 203 is used to number the GUI images and corresponding image information and store them in the memory warehouse, which includes trajectory memory, image memory, image information memory, and strategy.

[0137] Input GUI image module 204 is used to input numbered GUI images into the multimodal large model.

[0138] The defect detection module 205 is used to detect whether there is a defect. If there is a defect, the defect is recorded and its description is stored in the trajectory memory. If the trajectory cutoff condition is met, the system jumps to module 207. Otherwise, the system jumps to module 206.

[0139] The trajectory termination condition judgment module 206 is used to determine whether the trajectory termination condition has been met. If yes, the trajectory terminates and the process jumps to module 207; otherwise, the process jumps to module 208.

[0140] The test stop condition judgment module 207 is used to determine whether the test stop condition has been met. If yes, the test stops; if no, the current trajectory memory expires, a new trajectory memory is started, and the process jumps to module 208.

[0141] The memory retrieval module 208 is used to search for important memories of the current GUI image from the memory repository.

[0142] The multimodal large model processing module 209 is used to input important memories as prompts into the multimodal large model, and the multimodal large model outputs exploration tendencies and operations.

[0143] The exploration tendency update module 210 is used to update the exploration tendency output by the multimodal large model into the policy.

[0144] Operation saving and application module 211 is used to store the operation output of the multimodal large model into the trajectory memory, apply the operation to the current GUI, and switch to the new GUI.

[0145] The return module 212 is used to return the GUI image and the corresponding image information acquisition module 202.

[0146] Example

[0147] The mobile application in this embodiment comes from an app store.

[0148] In this embodiment, the multimodal large model adopts a fine-tuned version of the CPM-Bee-1B model.

[0149] Example 3

[0150] An electronic device, comprising:

[0151] One or more processors.

[0152] A storage device on which one or more programs are stored.

[0153] When one or more programs are executed by one or more processors, the one or more processors implement the application GUI testing method as described in Example 1.

[0154] Example 4

[0155] A storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements an application GUI testing method as described in Example 1.

[0156] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0157] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for testing an application GUI, characterized in that, The method includes: Fine-tuning a multimodal large model with multiple defective GUI images and corresponding defect types as prompts; Obtaining the current GUI image and corresponding current image information; the current GUI image is an image of the current GUI of an application; Assigning a current number to the current GUI image, and storing the current number, the current GUI image, and the current image information in a memory repository; the memory repository includes trajectory memory, image memory, image information memory, and strategies; the strategies include: goal setting, exploration tendency, and operations; Judging whether the current GUI image has defects based on the current GUI image and the multimodal large model, and obtaining a first judgment result; If the first judgment result is yes, storing the corresponding defect type in the current trajectory memory, terminating the current trajectory memory, starting the next trajectory memory, updating the current trajectory memory to the next trajectory memory, and judging whether the test stop condition is reached, obtaining a third judgment result; If the first judgment result is no, judging whether the current trajectory memory is terminated, obtaining a second judgment result; If the second judgment result is yes, terminating the current trajectory memory, starting the next trajectory memory, updating the current trajectory memory to the next trajectory memory, and judging whether the test stop condition is reached, obtaining a third judgment result; if the third judgment result is yes, stopping the test; If the third judgment result is no or the second judgment result is no, then: Searching for important memories of the current GUI image in the memory repository; Inputting the important memories, the image information memory, the goal setting, and the exploration tendency as prompts into the multimodal large model, obtaining the current exploration tendency and the current operation, adding the current exploration tendency to the strategies in the memory repository, storing the current operation in the current trajectory memory, and applying the current operation to the current GUI; Updating the current GUI image to the next GUI image, and returning to "Obtaining the current GUI image and corresponding current image information".

2. The application GUI testing method according to claim 1, characterized in that, The image information corresponding to the image of any GUI includes: component elements, interaction logic, and attributes and states.

3. The application GUI testing method according to claim 1, characterized in that, Judging whether the current GUI image has defects based on the current GUI image and the multimodal large model, and obtaining a first judgment result, specifically including: Inputting the current GUI image into the multimodal large model, and inputting the prompt "<Is there a defect in the GUI image number>? If so, what is the defect?", thereby judging whether the current GUI image has defects, and obtaining a first judgment result.

4. The application GUI testing method according to claim 1, characterized in that, Searching for important memories of the current GUI image in the memory repository, including: Retrieving the image information memory, the goal setting, and the exploration tendency corresponding to the current GUI image from the memory repository, scoring the importance of the trajectory memory, and determining all the trajectory memories with importance scores in the top preset ranking as important memories.

5. The application GUI testing method according to claim 1, characterized in that, Store the current number, the current GUI image, and the current image information in the memory repository, including: Store the current number in the current trajectory memory in the memory repository, store the current GUI image in the image memory in the memory repository, and store the current image information in the image information memory in the memory repository.

6. The application GUI testing method according to claim 1, characterized in that, Any trajectory memory includes the natural language corresponding to one or more GUIs, and the natural language is: Perform <operation> at <position> on <GUI number> to obtain <GUI number>, <GUI number> is defect-free / has defects, and the defect is <defect description>.

7. An application GUI testing system, characterized in that, The system includes: A fine-tuning module for fine-tuning the multi-modal large model with multiple defective GUI images and corresponding defect types as prompts; An image acquisition module for acquiring the current GUI image and the corresponding current image information; the current GUI image is the image of the current GUI of the application; A storage module for assigning a current number to the current GUI image, and storing the current number, the current GUI image, and the current image information in the memory repository; the memory repository includes trajectory memory, image memory, image information memory, and strategies; the strategies include: goal setting, exploration tendency, and operations; A first judgment module for judging whether the current GUI image has defects according to the current GUI image and the multi-modal large model, and obtaining a first judgment result; A first execution module for, if the first judgment result is yes, storing the corresponding defect type in the current trajectory memory, ending the current trajectory memory, starting the next trajectory memory, updating the current trajectory memory to the next trajectory memory, and judging whether the test stop condition is reached, and obtaining a third judgment result; A second execution module for, if the first judgment result is no, judging whether the current trajectory memory is ended, and obtaining a second judgment result; A third execution module for, if the second judgment result is yes, ending the current trajectory memory, starting the next trajectory memory, updating the current trajectory memory to the next trajectory memory, and judging whether the test stop condition is reached, and obtaining a third judgment result; A fourth execution module for stopping the test if the third judgment result is yes; A fifth execution module for, if the third judgment result is no or the second judgment result is no, then: Search for the important memory of the current GUI image in the memory repository; Input the important memory, the image information memory, the goal setting, and the exploration tendency as prompts into the multi-modal large model to obtain the current exploration tendency and the current operation, add the current exploration tendency to the strategies in the memory repository, store the current operation in the current trajectory memory, and apply the current operation to the current GUI; Update the current GUI image to the next GUI image and return to the image acquisition module.

8. An electronic device, characterized in that, Including: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the application GUI testing method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, It stores a computer program, wherein the computer program, when executed by a processor, implements the application GUI testing method as described in any one of claims 1 to 6.

Images