GUI (Graphical User Interface) test method and device of application program, electronic equipment and storage medium

Abstract

The invention relates to the technical field of software testing, and provides a GUI testing method and device of an application program, electronic equipment and a storage medium. In the process of running the to-be-tested application program, multiple pieces of GUI state information are collected; for each piece of GUI state information, obtaining each piece of matched target test knowledge from a test knowledge base according to the GUI state information, and generating a corresponding test prompt word in combination with the GUI state information; and generating a test strategy based on the GUI state information and the corresponding test cue word by using the multi-modal large model, and carrying out GUI test on the to-be-tested application program according to the test strategy by using the test tool so as to carry out GUI traversal test on the to-be-tested application program. Related test knowledge is obtained through the test knowledge base and cue words are generated to guide the multi-modal large model to output the test strategy so as to explore more interaction paths to carry out GUI test, so that the test operation is simplified, and the test efficiency and coverage rate are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of software testing, and in particular to a GUI testing method, device, electronic device and storage medium for an application program. Background Art

[0002] For mobile platforms such as Android platform applications, it is usually necessary to test the GUI (Graphical User Interface) of the application. At present, there are three commonly used testing methods: the first is to randomly generate user operation sequences through test tools for testing, but this method lacks pertinence, will generate a large number of repeated test cases and is difficult to cover complex interaction paths, resulting in low test efficiency. The second is to simulate user operations for testing by writing scripts, but this method requires frequent updates of scripts as the interface changes, resulting in high maintenance costs. The third is to record the user's actual operations and perform corresponding operations with reference to the recorded video for testing, but this method is often difficult to simulate the user's real usage scenarios and behavior patterns, and may ignore some key interaction paths, resulting in low test effectiveness. Summary of the invention

[0003] In view of this, an object of the present invention is to provide a GUI testing method, device, electronic device and storage medium for an application.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0005] In a first aspect, the present invention provides a GUI testing method for an application, the method comprising:

[0006] During the running of the application under test, multiple GUI status information is collected;

[0007] For any of the GUI state information, each matching target test knowledge is obtained from a pre-built test knowledge base according to the GUI state information, and a corresponding test prompt word is generated according to each of the target test knowledge and the GUI state information;

[0008] Using a preset multimodal macro model, generating a test strategy based on the GUI state information and its corresponding test prompt words, and using a preset test tool to perform GUI testing on the application to be tested according to the test strategy;

[0009] Each of the GUI state information is traversed, and a GUI traversal test is performed on the application to be tested.

[0010] In an optional implementation manner, the step of collecting a plurality of GUI state information during the running of the application to be tested includes:

[0011] In the process of running the application to be tested, the image of the current GUI is collected in real time and the control tree of the current GUI is obtained, so as to obtain multiple GUI images and their control trees;

[0012] The visual element corresponding to the triggered control is marked in each of the GUI images, the marked GUI image is obtained, and each GUI state information is obtained, wherein the GUI state information includes the marked GUI image and its control tree.

[0013] In an optional implementation, the test knowledge base includes test knowledge sets of multiple applications, and the test knowledge sets include multiple test knowledge and their labels;

[0014] The step of acquiring each matching target test knowledge from a pre-built test knowledge base according to the GUI state information, and generating a corresponding test prompt word according to each target test knowledge and the GUI state information includes:

[0015] Acquire the test knowledge set of the application to be tested in the test knowledge base to obtain a target test knowledge set;

[0016] In the target test knowledge set, each test knowledge whose label matches the control tree in the GUI state information is obtained to obtain each target test knowledge;

[0017] Based on the GUI image marked in the GUI state information, interface details and test purposes are generated, and according to a preset first template, corresponding test prompt words are generated based on the interface details, the test purpose and each of the target test knowledge.

[0018] In an optional embodiment, the step of using a preset multimodal macro model to generate a test strategy based on the GUI state information and its corresponding test prompt words, and using a preset test tool to perform a GUI test on the application to be tested according to the test strategy includes:

[0019] Using the multimodal large model, a test strategy is generated based on the GUI state information and its corresponding test prompt words, wherein the test strategy includes a control to be operated and an operation type;

[0020] The test tool is used to generate corresponding test instructions based on the test strategy, and the corresponding interface of the application to be tested is entered according to the test instructions, and the control to be operated is triggered according to the operation type.

[0021] In an optional embodiment, the method further comprises:

[0022] In the process of running the application to be tested, GUI images are collected according to a preset time length and a preset interval to obtain multiple groups of GUI image sequences;

[0023] For each group of the GUI image sequences, corresponding detection prompt words are generated based on the GUI image sequence and a preset second template, and the multimodal large model is used to perform non-crash anomaly detection on each GUI image in the GUI image sequence based on the detection prompt words to obtain detection results, thereby obtaining detection results for each group of the GUI image sequences.

[0024] In an optional implementation, the test knowledge base is constructed in the following manner:

[0025] Get multiple test information for each preset application;

[0026] For each of the application programs, each of the test information of the application program is cleaned and standardized to obtain each test knowledge, and each of the test knowledge is labeled to obtain a test knowledge set including each of the test knowledge and its label, thereby obtaining a test knowledge set for each of the application programs;

[0027] The test knowledge set of each application is stored to obtain the test knowledge base.

[0028] In a second aspect, the present invention provides a GUI testing device for an application, the device comprising:

[0029] The acquisition module is used to collect multiple GUI status information during the running of the application program to be tested;

[0030] A test module, for obtaining, for any one of the GUI state information, each matching target test knowledge from a pre-built test knowledge base according to the GUI state information, and generating a corresponding test prompt word according to each of the target test knowledge and the GUI state information;

[0031] Using a preset multimodal macro model, generating a test strategy based on the GUI state information and its corresponding test prompt words, and using a preset test tool to perform GUI testing on the application to be tested according to the test strategy;

[0032] Each of the GUI state information is traversed, and a GUI traversal test is performed on the application to be tested.

[0033] In an optional embodiment, the device further comprises a detection module, which is used to:

[0034] In the process of running the application to be tested, GUI images are collected according to a preset time length and a preset interval to obtain multiple groups of GUI image sequences;

[0035] For each group of the GUI image sequences, corresponding detection prompt words are generated based on the GUI image sequence and a preset second template, and the multimodal large model is used to perform non-crash anomaly detection on each GUI image in the GUI image sequence based on the detection prompt words to obtain detection results, thereby obtaining detection results for each group of the GUI image sequences.

[0036] In a third aspect, the present invention provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the GUI testing method of the application described in any one of the aforementioned embodiments is implemented.

[0037] In a fourth aspect, the present invention provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the GUI testing method of the application described in any one of the aforementioned embodiments.

[0038] The present invention provides a GUI testing method, device, electronic device and storage medium for an application. The method comprises: collecting multiple GUI state information during the operation of the application to be tested; for any GUI state information, obtaining each matching target test knowledge from a pre-built test knowledge base according to the GUI state information, and generating corresponding test prompt words according to each target test knowledge and the GUI state information; using a preset multimodal large model, generating a test strategy based on the GUI state information and its corresponding test prompt words, and using a preset test tool, performing GUI testing on the application to be tested according to the test strategy; traversing each GUI state information, and performing GUI traversal testing on the application to be tested. Relevant test knowledge is obtained through GUI state information and the test knowledge base, and prompt words are generated to guide the multimodal large model to output a test strategy, so as to explore more interactive paths to perform GUI traversal testing on the application, thereby simplifying the test operation and improving the efficiency and coverage of the test.

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 A block diagram of an electronic device provided by an embodiment of the present invention is shown;

[0042] Figure 2 One of the flow charts of the GUI testing method of the application provided by the embodiment of the present invention is shown;

[0043] Figure 3 The second flowchart of the GUI testing method for an application provided by an embodiment of the present invention is shown;

[0044] Figure 4 A functional module diagram of a GUI testing device for an application provided by an embodiment of the present invention is shown.

[0045] Icons: 100 - electronic device; 110 - bus; 120 - processor; 130 - memory; 150 - I / O module; 170 - communication module; 300 - GUI test device for application; 310 - building module; 330 - acquisition module; 350 - testing module; 370 - detection module. DETAILED DESCRIPTION

[0046] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0047] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0048] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0049] See also Figure 1 , is a block diagram of an electronic device provided by an embodiment of the present invention. The electronic device 100 includes a bus 110 , a processor 120 , a memory 130 , an I / O module 150 and a communication module 170 .

[0050] The bus 110 is a circuit that connects the above elements to each other and transfers communications between the above elements.

[0051] The processor 120 can receive commands from the above-mentioned other elements (such as the memory 130, the I / O module 150, the communication module 170, etc.) through the bus 110, can interpret the received commands, and can perform calculations or data processing according to the interpreted commands. The processor 120 can be an integrated circuit chip with signal processing capabilities, which can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0052] The memory 130 may store commands or data received from the processor 120 or other components (e.g., the I / O module 150, the communication module 170, etc.) or commands or data generated by the processor 120 or other components. The memory 130 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), or an electrically erasable read-only memory (EEPROM).

[0053] The I / O module 150 may receive commands or data input from a user via input-output means (e.g., a sensor, a keyboard, a touch screen, etc.), and may transmit the received commands or data to the processor 120 or the memory 130 via the bus 110. It is also used to display various information (e.g., multimedia data, text data) received, stored, and processed from the above elements, and may display videos, images, data, etc. to the user.

[0054] The communication module 170 may be used to communicate signals or data with other devices.

[0055] Understandably, Figure 1 The structure shown is only a schematic diagram of the structure of the electronic device 100. The electronic device 100 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0056] The electronic device provided in the embodiment of the present invention may be a personal computer, a tablet computer, a netbook, a personal digital assistant, etc. The embodiment of the present invention does not impose any limitation on this.

[0057] The above-mentioned electronic device 100 will be used as an execution subject to execute each step in each method provided in the embodiment of the present invention and achieve corresponding technical effects.

[0058] See also Figure 2 , is a flow chart of a GUI testing method for an application provided in an embodiment of the present invention.

[0059] Step S202: collecting a plurality of GUI status information during the running of the application program to be tested.

[0060] It is understood that GUI refers to a user interface that is displayed in a graphical manner. For applications on the Android platform, the state of the GUI is managed through the Activity component. The state of the GUI can be understood as the layout and style of the interface. In the process of running the application, performing different interactive operations will cause the GUI state to change.

[0061] For ease of understanding, the embodiment of the present invention is described by taking the application to be tested as a live broadcast application a as an example. For example, in the process of running the live broadcast application a, multiple GUI state information generated by executing different interactive operations can be collected. The GUI state information can be understood as information representing the layout and style of the interface.

[0062] Step S204: for any GUI state information, each matching target test knowledge is obtained from a pre-built test knowledge base according to the GUI state information, and a corresponding test prompt word is generated according to each target test knowledge and the GUI state information.

[0063] Step S206, using the preset multimodal macro model, generating a test strategy based on the GUI state information and its corresponding test prompt words, and using the preset test tool to perform GUI testing on the application to be tested according to the test strategy.

[0064] Step S208, traversing each GUI state information, and performing a GUI traversal test on the application to be tested.

[0065] In this embodiment, the multimodal large language model (MLLM) refers to an artificial intelligence model that can process and integrate multiple modal information, such as text, images, audio and video. Then, a test knowledge base can be built in advance based on the test-related information of multiple applications, and the test knowledge in the test knowledge base can be used as sample data to fine-tune the basic multimodal large model to obtain a multimodal large model adapted to GUI testing.

[0066] It is understandable that the processing method for each GUI state information in the embodiment of the present invention is similar. For the sake of simplicity, a GUI state information is used as an example for explanation. For example, for any GUI state information collected from the live broadcast application a, each target test knowledge matching it can be obtained from the pre-built test knowledge base according to the GUI state information, and a corresponding test prompt word is generated according to each target test knowledge and the GUI state information.

[0067] Then, the GUI state information and its corresponding test prompt words are input into the preset multimodal large model, and the multimodal large model is guided to output the test strategy to explore more interaction paths. Then, the preset test tool is used to perform GUI testing on the live broadcast application a according to the test strategy output by the multimodal large model. In a similar manner, each GUI state information is processed to perform GUI traversal testing on the live broadcast application a.

[0068] It can be seen that based on the above steps, in the process of running the application to be tested, multiple GUI state information is collected; for any GUI state information, each matching target test knowledge is obtained from the pre-built test knowledge base according to the GUI state information, and corresponding test prompt words are generated according to each target test knowledge and GUI state information; using the preset multimodal large model, a test strategy is generated based on the GUI state information and its corresponding test prompt words, and using the preset test tool, a GUI test is performed on the application to be tested according to the test strategy; each GUI state information is traversed to perform a GUI traversal test on the application to be tested. Relevant test knowledge is obtained through GUI state information and the test knowledge base, and prompt words are generated to guide the multimodal large model to output a test strategy to explore more interactive paths to perform GUI traversal testing on the application, thereby simplifying the test operation and improving the efficiency and coverage of the test.

[0069] Optionally, for step S202, an embodiment of the present invention provides a possible implementation method.

[0070] Step S202-1, in the process of running the application to be tested, the image of the current GUI is collected in real time and the control tree of the current GUI is obtained to obtain multiple GUI images and their control trees;

[0071] Step S202-3, marking the visual element corresponding to the triggered control in each GUI image, obtaining the marked GUI image, and obtaining each GUI state information, the GUI state information including the marked GUI image and its control tree.

[0072] For ease of understanding, the above-mentioned live broadcast application a is used for further explanation. During the running of live broadcast application a, a screenshot of the current GUI can be taken in real time to obtain multiple GUI images, and a control tree corresponding to the GUI image can be obtained by extracting the controls of the current GUI. The control tree represents a data structure of the hierarchical relationship between the controls in the current GUI. For example, the ADB (Android Debug Bridge) command of the Android platform can be used to obtain the GUI image and the control tree.

[0073] The command to obtain the GUI image is as follows:

[0074] adb shell screencap-p / sdcard / screenshot.png

[0075] adb pull / sdcard / screenshot.png

[0076] The command for obtaining the GUI image indicates that the image of the current interface is captured by the screencap tool, and is saved to a file named screenshot.png on the local storage card, and the file is copied to the specified device.

[0077] The command to get the control tree is as follows:

[0078] adb shell uiautomator dump / sdcard / window_dump.xml

[0079] adb pull / sdcard / window_dump.xml

[0080] The command to obtain the control tree indicates that the control of the current interface is obtained through the uiautomator tool, and the control tree is recorded in a file in XML (eXtensible Markup Language) format, and then saved in a file named window_dump.xml on the local storage card, and the file is copied to the specified device. It should be understood that a file in JSON (Java Script Object Notation) format can also be used to record the control tree, that is, the file in XML format can be converted into a file in JSON format.

[0081] After obtaining each GUI image, it is also necessary to mark the visual element corresponding to the triggered control in the GUI image to match the triggered control with the visual element in the GUI image, and then obtain the marked GUI image. And the marked GUI image and its control tree are used as GUI state information, that is, multiple GUI state information is obtained. The visual element refers to the icon such as button, text box, etc. used to represent the control in the GUI image. The visual element can reflect the position of the control in the GUI image, as well as the style of the control such as color, shape, etc.

[0082] Optionally, for step S204, an embodiment of the present invention provides a possible implementation method.

[0083] Step S204-1, obtaining a test knowledge set of the application to be tested in a test knowledge base to obtain a target test knowledge set.

[0084] Step S204-3, in the target test knowledge set, obtain each test knowledge whose label matches the control tree in the GUI state information to obtain each target test knowledge.

[0085] Step S204-5, based on the GUI image marked in the GUI state information, generate interface details and test purpose, and generate corresponding test prompt words based on the interface details, test purpose and each target test knowledge according to the preset first template.

[0086] In this embodiment, the test knowledge base includes test knowledge sets of multiple application programs, and each test knowledge set includes multiple test knowledge and their labels.

[0087] For ease of understanding, we continue to use the live broadcast application a mentioned above for illustration. Assume that the GUI image marked in a certain GUI state information is a live broadcast room interface, which marks the visual elements corresponding to the bullet screen input box and the bullet screen send button, and the control tree in the GUI state information consists of three controls of the same level, namely the live broadcast room list, the bullet screen input box and the bullet screen send button.

[0088] In the test knowledge base, the test knowledge set of the live broadcast application a is obtained, and the target test knowledge set is obtained. Then, in the target test knowledge set, each control contained in the control tree in the GUI state information is searched, and each matching test knowledge is obtained as the target test knowledge, and each target test knowledge such as test knowledge 1 and test knowledge 2 is obtained.

[0089] Then, based on the GUI image marked in the GUI status information, the interface details and test purpose are generated. For example, the generated interface details include the current interface name, i.e., the live broadcast room interface, and the control information, i.e., the live broadcast room list, the bullet screen input box, and the bullet screen sending button; and the generated test purpose, i.e., verifying the bullet screen sending function.

[0090] Then, according to the preset first template, generate corresponding test prompts based on the interface details, test purpose, and each target test knowledge. For example, the generated test prompt is: Please output the test strategy based on the interface details {current interface [live studio interface], control information [live studio list, bullet screen input box, and bullet screen send button]}, test purpose {verify the bullet screen send function}, and combine the relevant information retrieved from the test knowledge base {test knowledge 1, test knowledge 2}, and include the specific controls and operation types to be operated. Please ensure that the output test strategy meets the test purpose and can effectively explore the functions of the application.

[0091] It should be understood that the first template can be set according to the actual application, and the embodiment of the present invention is not limited. For example, if the GUI images are collected in chronological order, the historical operation can be generated based on the triggered control marked in the previous GUI image and used as a kind of information in the interface details. In addition, the output test strategy can also be required to include the test reason and expected results, so as to understand the role of the test strategy and verify the test results.

[0092] It can be understood that the embodiment of the present invention improves the test efficiency by establishing a test knowledge base as an external knowledge source of the multimodal large model and generating prompt words that guide the multimodal large model to refer to relevant test information to output the test strategy. And because the multimodal large model will explore various interactive paths of the test GUI based on the prompt words, the coverage and effectiveness of the test are improved.

[0093] Optionally, for step S206, an embodiment of the present invention provides a possible implementation method.

[0094] Step S206-1, using the multimodal large model, generating a test strategy based on the GUI state information and its corresponding test prompt words, the test strategy including the control to be operated and the operation type.

[0095] Step S206-3: Generate corresponding test instructions based on the test strategy using the test tool, enter the corresponding interface of the application to be tested according to the test instructions, and trigger the control to be operated according to the operation type.

[0096] For ease of understanding, the above-mentioned live broadcast application a is continued for explanation. The obtained GUI state information, i.e., the marked GUI image and its control tree, and the generated test prompt words are input into the preset multimodal large model to guide the multimodal large model to output a test strategy, and the test strategy includes the specific control to be operated, i.e., the control to be operated and the operation type.

[0097] The test strategy output by the multimodal large model is then transmitted to the preset test tool, which is a tool that can accurately perform various interactive operations. The test tool then parses the test strategy to generate corresponding test instructions, and enters the corresponding interface of the live broadcast application a according to the test instructions, and then triggers the control to be operated according to the operation type in the interface to perform the corresponding test operation, so as to perform GUI testing on the live broadcast application a.

[0098] It should be noted that each GUI status information has a corresponding test result. The embodiment of the present invention can also generate new GUI status information according to the test result corresponding to the GUI status information, and execute steps S204 and S206 based on the new GUI status information to perform GUI testing on the application under test to improve the test coverage.

[0099] Optionally, after step S208, the embodiment of the present invention further provides an implementation method.

[0100] Step S210 , during the running of the application program to be tested, GUI images are collected according to a preset duration and a preset interval to obtain a plurality of groups of GUI image sequences.

[0101] Step S212: For each group of GUI image sequences, a corresponding detection prompt word is generated based on the GUI image sequence and a preset second template, and a multimodal large model is used to perform non-crash anomaly detection on each GUI image in the GUI image sequence based on the detection prompt word to obtain a detection result, thereby obtaining a detection result for each group of GUI image sequences.

[0102] It should be noted that the existing testing methods are all about testing crash exceptions in the application, such as serious program errors that cause application operation interruption, invalid interactive operations, data discard, etc. However, for non-crash exceptions, such as black screen, white screen, flowery screen, style overlap, component missing and other problems, although such problems will not cause the application to crash, they will seriously affect the user experience. Therefore, the embodiment of the present invention will also test non-crash exceptions for the application to be tested to provide information support for optimizing the application to be tested.

[0103] For ease of understanding, the above-mentioned live broadcast application a is used for further explanation. During the running of the live broadcast application a, GUI images are collected according to a preset duration and a preset interval to obtain multiple groups of GUI image sequences. That is, the number of images in each group of GUI image sequences is the same.

[0104] It can be understood that the processing method for each group of GUI image sequences in the embodiment of the present invention is similar. For the sake of simplicity, an example is given below in which a group of GUI image sequences is 5 time-connected live broadcast room screenshots.

[0105] Based on the set of GUI image sequences and the preset second template, a corresponding detection prompt word is generated. For example, the generated detection prompt word is: Please select the target image according to the given set of images {live studio screenshot Figure 1 , live studio screenshot Figure 2 , live studio screenshot Figure 3 , live studio screenshot Figure 4, live broadcast room 5}, detect whether these images have abnormalities and output the detection results. The types of abnormalities are: 1. Black screen (the image is completely black); 2. White screen (the image is completely white); 3. Flower screen (the image has serious color distortion or image distortion); 4. Style overlap (interface elements cover each other, making the content difficult to recognize); 5. Component missing (key buttons or controls of the interface disappear). And the detection results need to include which image has an abnormality and the type of abnormality, the specific table items of the abnormality and the cause of the abnormality, as well as the potential impact on the user experience. It should be understood that the second template can be set according to the actual application, and the embodiment of the present invention is not limited.

[0106] Then, the group of GUI image sequences and their corresponding detection prompt words are input into a preset multimodal large model, and the multimodal large model is used to perform non-crash anomaly detection on each GUI image in the group of GUI image sequences based on the detection prompt words to output a detection result, that is, to obtain the detection result of the group of GUI image sequences. Each group of GUI image sequences is processed in a similar manner to obtain the detection result of each group of GUI image sequences.

[0107] Optionally, for the above-mentioned test knowledge base, the embodiment of the present invention also provides a method for constructing a test knowledge base, please refer to Figure 3 .

[0108] Step S222, obtaining a plurality of test information of each preset application.

[0109] Step S224, for each application, each test information of the application is cleaned and standardized to obtain each test knowledge, and each test knowledge is labeled to obtain a test knowledge set including each test knowledge and its label, thereby obtaining a test knowledge set for each application.

[0110] Step S226, storing the test knowledge set of each application to obtain a test knowledge base.

[0111] In this embodiment, the application programs for which test-related information is to be collected can be pre-set, and the test information of these application programs can be collected from multiple sources to obtain multiple test information of each application program. The test information can be a manual test case, an automated test script, a GUI screenshot and its control tree, or a common program error. A manual test case refers to a test step and expected result manually described in text form. An automated test script refers to a script written in a programming language that can automatically execute test steps.

[0112] For each application, each test information of the application can be cleaned and standardized to obtain each test knowledge; then, according to the set labeling rules, each test knowledge is labeled to obtain the label of each test knowledge, and each test knowledge and its label of the application are obtained, that is, the test knowledge set of the application is obtained. All the test information of each application is processed in a similar way to obtain the test knowledge set of each application. Then, the test knowledge set of each application is stored to obtain the test knowledge base.

[0113] For ease of understanding, the present invention provides a live broadcast application a as an example, and provides several examples of test knowledge and its labels.

[0114] One of the test knowledge of live broadcast application a is a manual test case: Case ID: TC001. Case name: Send barrage. Steps: 1. Open live broadcast application a; 2. Scroll the list of live broadcast rooms on the homepage of live broadcast application a; 3. Select and enter a live broadcast room; 4. Wait for 5 seconds on the live broadcast room page to ensure that the screen is loaded normally; 5. Click the barrage input box; 6. Enter the text "Test barrage 123"; 7. Click the barrage send button; 8. Verify whether the barrage is correctly displayed on the live broadcast screen. Expected result: The barrage "Test barrage 123" is correctly displayed on the live broadcast screen. And the labels of this test knowledge are: live broadcast room, barrage, barrage input box, barrage send button.

[0115] One of the testing concepts of live streaming application A is automated test scripts:

[0116] def enter_live_room(room_id):

[0117] find_element_by_id("live_room_list").click()

[0118] find_element_by_id(f"room_{room_id}").click()

[0119] wait_for_element_visible("live_stream_view", timeout=10)

[0120] def send_danmaku(text):

[0121] find_element_by_id("danmaku_input").click()

[0122] find_element_by_id("danmaku_input").send_keys(text)

[0123] find_element_by_id("send_button").click()

[0124] wait_for_element_with_text(text,timeout=5)

[0125] And the labels of this test knowledge are: live broadcast room operation, barrage operation.

[0126] One of the test knowledge of live broadcast application a is GUI screenshot and its control tree, where the relevant information of the GUI screenshot is: Description: Contains live video screen, bullet screen scrolling area, bullet screen input box at the bottom and bullet screen send button. Key area coordinates example: bullet screen input box: (50,1800,900,1900); bullet screen send button: (920,1800,1070,1900). The control tree of the GUI screenshot is: live broadcast room list-bullet screen input box-bullet screen send button. And the labels of the test knowledge are: live broadcast room interface, bullet screen area, bullet screen input box.

[0127] One of the test knowledge of live broadcast application a is common program errors: Problem type: Black screen in live broadcast room. Description: After entering the live broadcast room, the screen is completely black and nothing is displayed. Possible reasons: 1. Network connection problem; 2. Video decoding failure; 3. Server-side live stream interruption. Detection method: Analyze the screenshot of the live broadcast room interface to check whether there is a large black area. And the label of this test knowledge is: black screen, live broadcast room failure.

[0128] In order to execute the corresponding steps in the above embodiments and various possible methods, a method for implementing a GUI testing device for an application is provided below. Figure 4 , is a functional module diagram of the GUI testing device for an application provided in an embodiment of the present invention. It should be noted that the basic principle and technical effects of the GUI testing device 300 for an application provided in this embodiment are the same as those in the above embodiment. For the sake of brief description, for matters not mentioned in this embodiment, reference may be made to the corresponding contents in the above embodiment. The GUI testing device 300 for an application includes:

[0129] The collection module 330 is used to collect a plurality of GUI state information during the running of the application program to be tested;

[0130] The test module 350 is used to obtain each matching target test knowledge from a pre-built test knowledge base according to the GUI state information for any GUI state information, and generate corresponding test prompt words according to each target test knowledge and the GUI state information; generate a test strategy based on the GUI state information and its corresponding test prompt words using a preset multimodal large model, and perform GUI testing on the application to be tested according to the test strategy using a preset test tool; traverse each GUI state information and perform GUI traversal testing on the application to be tested.

[0131] Optionally, the acquisition module 330 is also used to: acquire the image of the current GUI and obtain the control tree of the current GUI in real time during the running of the application to be tested, so as to obtain multiple GUI images and their control trees; mark the visual elements corresponding to the triggered controls in each GUI image, obtain the marked GUI image, and obtain each GUI status information, wherein the GUI status information includes the marked GUI image and its control tree.

[0132] Optionally, the test module 350 is also used to: in the target test knowledge set, obtain each test knowledge whose label matches the control tree in the GUI state information to obtain each target test knowledge; based on the marked GUI image in the GUI state information, generate interface details and test purposes, and according to the preset first template, generate corresponding test prompt words based on the interface details, test purposes and each target test knowledge.

[0133] Optionally, the test module 350 is also used to: utilize a multimodal large model to generate a test strategy based on GUI status information and its corresponding test prompt words, the test strategy including controls to be operated and operation types; utilize a test tool to generate corresponding test instructions based on the test strategy, enter the corresponding interface of the application to be tested according to the test instructions, and trigger the controls to be operated according to the operation type.

[0134] Optionally, the GUI testing device 300 of the application also includes a detection module 370, which is used to: during the running of the application to be tested, collect GUI images according to a preset duration and a preset interval to obtain multiple groups of GUI image sequences; for each group of GUI image sequences, generate corresponding detection prompt words based on the GUI image sequence and a preset second template, and use a multimodal large model to perform non-crash anomaly detection on each GUI image in the GUI image sequence based on the detection prompt words to obtain detection results, thereby obtaining detection results for each group of GUI image sequences.

[0135] Optionally, the GUI testing device 300 of the application also includes a construction module 310, which is used to: obtain multiple test information of each preset application; for each application, clean and standardize each test information of the application to obtain each test knowledge, and label each test knowledge to obtain a test knowledge set containing each test knowledge and its label, to obtain a test knowledge set for each application; store the test knowledge set of each application to obtain a test knowledge base.

[0136] An embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the GUI testing method of the application program disclosed in the embodiment of the present invention is implemented.

[0137] The embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the GUI testing method of the application program disclosed in the embodiment of the present invention is implemented.

[0138] In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0139] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0140] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0141] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A GUI testing method for an application, characterized in that: The method comprises: During the running of the application under test, multiple GUI status information is collected; For any of the GUI state information, each matching target test knowledge is obtained from a pre-built test knowledge base according to the GUI state information, and a corresponding test prompt word is generated according to each of the target test knowledge and the GUI state information; Using a preset multimodal macro model, generating a test strategy based on the GUI state information and its corresponding test prompt words, and using a preset test tool to perform GUI testing on the application to be tested according to the test strategy; Each of the GUI state information is traversed, and a GUI traversal test is performed on the application to be tested.

2. The method according to claim 1, characterized in that The step of collecting a plurality of GUI status information during the running of the application to be tested includes: In the process of running the application to be tested, the image of the current GUI is collected in real time and the control tree of the current GUI is obtained, so as to obtain multiple GUI images and their control trees; The visual element corresponding to the triggered control is marked in each of the GUI images, the marked GUI image is obtained, and each GUI state information is obtained, wherein the GUI state information includes the marked GUI image and its control tree.

3. The method according to claim 2, characterized in that The test knowledge base includes test knowledge sets of multiple application programs, and the test knowledge sets include multiple test knowledge and their labels; The step of acquiring each matching target test knowledge from a pre-built test knowledge base according to the GUI state information, and generating a corresponding test prompt word according to each target test knowledge and the GUI state information includes: Acquire the test knowledge set of the application to be tested in the test knowledge base to obtain a target test knowledge set; In the target test knowledge set, each test knowledge whose label matches the control tree in the GUI state information is obtained to obtain each target test knowledge; Based on the GUI image marked in the GUI state information, interface details and test purposes are generated, and according to a preset first template, corresponding test prompt words are generated based on the interface details, the test purpose and each of the target test knowledge.

4. The method according to claim 1, characterized in that: The step of using a preset multimodal macro model to generate a test strategy based on the GUI state information and its corresponding test prompt words, and using a preset test tool to perform a GUI test on the application to be tested according to the test strategy includes: Using the multimodal large model, a test strategy is generated based on the GUI state information and its corresponding test prompt words, wherein the test strategy includes a control to be operated and an operation type; The test tool is used to generate corresponding test instructions based on the test strategy, and the corresponding interface of the application to be tested is entered according to the test instructions, and the control to be operated is triggered according to the operation type.

5. The method according to claim 1, characterized in that The method further comprises: In the process of running the application to be tested, GUI images are collected according to a preset time length and a preset interval to obtain multiple groups of GUI image sequences; For each group of the GUI image sequences, corresponding detection prompt words are generated based on the GUI image sequence and a preset second template, and the multimodal large model is used to perform non-crash anomaly detection on each GUI image in the GUI image sequence based on the detection prompt words to obtain detection results, thereby obtaining detection results for each group of the GUI image sequences.

6. The method according to claim 1, characterized in that The test knowledge base is constructed in the following way: Get multiple test information for each preset application; For each of the application programs, each of the test information of the application program is cleaned and standardized to obtain each test knowledge, and each of the test knowledge is labeled to obtain a test knowledge set including each of the test knowledge and its label, thereby obtaining a test knowledge set for each of the application programs; The test knowledge set of each application is stored to obtain the test knowledge base.

7. A GUI testing device for an application, characterized in that: The device comprises: The acquisition module is used to collect multiple GUI status information during the running of the application program to be tested; A test module, for obtaining, for any one of the GUI state information, each matching target test knowledge from a pre-built test knowledge base according to the GUI state information, and generating a corresponding test prompt word according to each of the target test knowledge and the GUI state information; Using a preset multimodal macro model, generating a test strategy based on the GUI state information and its corresponding test prompt words, and using a preset test tool to perform GUI testing on the application to be tested according to the test strategy; Each of the GUI state information is traversed, and a GUI traversal test is performed on the application to be tested.

8. The device according to claim 7, characterized in that The device also includes a detection module, which is used to: In the process of running the application to be tested, GUI images are collected according to a preset time length and a preset interval to obtain multiple groups of GUI image sequences; For each group of the GUI image sequences, corresponding detection prompt words are generated based on the GUI image sequence and a preset second template, and the multimodal large model is used to perform non-crash anomaly detection on each GUI image in the GUI image sequence based on the detection prompt words to obtain detection results, thereby obtaining detection results for each group of the GUI image sequences.

9. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the GUI testing method of the application program according to any one of claims 1 to 6 is implemented.

10. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the GUI testing method for an application program according to any one of claims 1 to 6.

Images