A graphical user interface testing method, apparatus, device and medium
By using an AI agent to parse and execute graphical user interface test cases, generating and saving test scripts, the problem of low testing efficiency and high barriers to entry in existing technologies is solved, achieving efficient and low-barrier graphical user interface testing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Filing Date
- 2026-03-02
- Publication Date
- 2026-07-10
Smart Images
Figure CN122364061A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent testing, and more particularly to a graphical user interface testing method, apparatus, device, and medium. Background Technology
[0002] In the field of graphical user interface testing, existing testing methods mainly include three types: manual testing, automated testing using code scripts, and robotic arm testing. All three types of testing methods have significant technical defects, which ultimately lead to low efficiency in the overall testing process and high technical barriers, making it difficult to adapt to the needs of efficient and large-scale testing. Manual testing requires testers to manually execute operations according to test cases throughout the entire process. This not only requires a significant investment of manpower to complete repetitive testing work but also makes continuous 24 / 7 testing impossible. Furthermore, manual operation and result judgment are prone to mistesting, and these mistests are difficult to detect in a timely manner, significantly reducing testing efficiency. Automated testing using code scripts requires testers to have professional coding and maintenance skills. Test scripts must be manually written based on test cases. The development, debugging, and subsequent maintenance of scripts all consume a lot of time and manpower. The high coding ability requirement creates a significant technical barrier, making it impossible for non-professionals to participate in the development and execution of test cases. At the same time, this method can only test a single device at a time, further limiting testing efficiency. Robotic arm testing requires testers to specially program the robotic arm's movements. The installation and deployment process is complex, and the equipment cannot be moved. This not only requires a high investment of manpower but also suffers from the problem of limited testing scenarios. Similarly, it can only test a single device at a time, making it difficult to improve testing efficiency. In summary, existing GUI testing methods all suffer from high barriers to entry and low efficiency due to their over-reliance on manual participation, professional skill requirements, or limited testing scenarios. Summary of the Invention
[0003] This invention provides a graphical user interface (GUI) testing method, apparatus, computer equipment, and storage medium to solve the problems of low efficiency and high threshold in GUI testing in the prior art.
[0004] Firstly, a graphical user interface testing method is provided, including: Obtain test cases for graphical user interfaces in natural language format; The test cases are analyzed by an AI agent to generate corresponding test operation instructions and inspection operation instructions; Based on the test operation instructions, the corresponding test operations are executed through the AI agent in the graphical user interface; After the test operation is completed, based on the inspection operation instructions, the AI agent performs the corresponding inspection operation on the interface content of the graphical user interface to obtain the test results.
[0005] Optionally, after the test and inspection operations are completed, the following will be included: Generate test reports for the testing process; Receive the review results of the test report. If the review results are approved, encapsulate the test operation instructions and inspection operation instructions into a test script and save it.
[0006] Optionally, before parsing test cases and generating corresponding test operation instructions and inspection operation instructions through an AI agent, the method further includes: Determine if a complete script that matches the test case and has been saved exists; If a complete script exists, the complete script will be invoked to complete the graphical user interface test.
[0007] Optionally, before parsing test cases and generating corresponding test operation instructions and inspection operation instructions through the AI agent, the following steps are also included: Determine if a complete script that matches the test case and has been saved exists; If no complete script exists, then determine if there is a test case containing the same steps as the current test case. If it exists, reuse the script corresponding to the same step in that use case; The remaining different steps are generated by the AI agent, which generates corresponding test and inspection instructions and executes the test and inspection operations.
[0008] Optionally, before parsing test cases and generating corresponding test operation instructions and inspection operation instructions through an AI agent, the method further includes: Determine if a complete script that matches the test case and has been saved exists; If no complete script exists, check if there is a test case that is identical to the current test case from the first step onwards; if so, reuse the script corresponding to all steps in that test case from the first step until the first different step appears. The remaining different steps are generated by the AI agent, which generates corresponding test and inspection instructions and executes the test and inspection operations.
[0009] Optionally, after receiving the review results of the test report, the following may also be included: If the review result is not approved, the revised graphical user interface test cases will be accepted. The AI agent re-parses the adjusted test cases to generate adjusted test operation instructions and inspection operation instructions. Based on the adjusted test and inspection instructions, the AI agent executes the adjusted test and inspection operations and regenerates the test report until the review result is approved.
[0010] Optionally, before obtaining graphical user interface test cases in natural language form, the method further includes: Receive imported test case files; The test case file is converted into a test case file of a preset format using a preset conversion script; Extract test cases in natural language form from the converted test case file.
[0011] Secondly, a graphical user interface testing apparatus is provided, comprising: The acquisition module is used to acquire graphical user interface test cases in natural language format. The parsing module is used to parse test cases through an AI agent and generate corresponding test operation instructions and inspection operation instructions; The testing module is used to execute corresponding test operations through an AI agent in the graphical user interface based on test operation instructions; The inspection module is used to perform corresponding inspection operations on the interface content of the graphical user interface based on the inspection operation instructions after the test operation is completed, and obtain the test results.
[0012] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the graphical user interface testing method described above.
[0013] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the graphical user interface testing method described above.
[0014] The aforementioned graphical user interface (GUI) testing method, apparatus, equipment, and storage medium define test cases in natural language, replacing the code writing requirements of existing script testing and the motion programming requirements of robotic arm testing. Testers do not need coding skills or specialized robotic arm operation abilities, eliminating the professional technical barriers of existing testing methods. By using an AI agent to replace manual labor, the process from parsing test cases to generating test / inspection operation instructions, and then executing test operations and completing inspection operations, abandons the manual execution of tests, manual writing of maintenance scripts, and manual programming of robotic arm movements found in existing technologies. Furthermore, it can simultaneously test multiple devices and multiple systems, significantly improving overall testing efficiency. This effectively solves the problems of low efficiency and high barriers to entry in existing GUI testing technologies. Attached Figure Description
[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 This is a flowchart illustrating a graphical user interface testing method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a graphical user interface testing device according to an embodiment of the present invention. Detailed Implementation
[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0018] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.
[0019] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0020] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," or "in response to determination." Similarly, the phrase "if determined" or "if matched to [described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once matched to [described condition or event]," or "in response to matched to [described condition or event]."
[0021] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0022] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.
[0023] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0024] To illustrate the technical solution of the present invention, specific embodiments are described below.
[0025] Please see Figure 1 As shown, Figure 1 A flowchart illustrating a graphical user interface testing method provided in an embodiment of the present invention includes the following steps: S11: Obtain test cases for graphical user interfaces in natural language format; S12: The test cases are analyzed by the AI agent to generate corresponding test operation instructions and inspection operation instructions; S13: Based on test operation instructions, the AI agent executes the corresponding test operations in the graphical user interface; S14: After the test operation is completed, based on the inspection operation instructions, the AI agent performs the corresponding inspection operation on the interface content of the graphical user interface to obtain the test results.
[0026] For ease of understanding, the following explains some key terms in this embodiment: A graphical user interface (GUI) test case is a document or description that describes the steps and expected results of performing a specific test task on a graphical user interface. Test cases can be expressed in natural language to facilitate understanding and creation by non-experts.
[0027] An AI agent is an entity with artificial intelligence capabilities that can understand, analyze, and execute specific tasks. In this embodiment, the AI agent is configured to parse test cases in natural language and generate and execute corresponding operation instructions based on the parsing results.
[0028] Test operation instructions refer to the sequence of commands generated by the AI agent based on test cases, which guide the AI agent to perform specific operations on the graphical user interface. These instructions can include various interactive behaviors such as clicking (i.e., AI simulating human clicking on the screen), swiping (i.e., AI simulating human swiping on the screen), inputting text (i.e., AI inputting text content in a specified text input box), pressing keys (i.e., AI simulating device key operations), opening an application (i.e., AI automatically launching a specified application), searching for controls (i.e., AI automatically searching for specific UI controls in a scrollable page), and waiting (i.e., AI pausing execution and waiting for a specified time).
[0029] The inspection operation instructions refer to the sequence of commands generated by the AI agent based on test cases, used to guide the AI agent in verifying the interface content of the graphical user interface. These instructions can include various verification methods such as image inspection (i.e., whether the visual appearance of the AI verifies whether the element matches the benchmark, such as checking whether a static image matches), text inspection (i.e., whether the text content contained in the AI verifies whether the AI verifies whether the element matches, such as checking whether fixed text matches), change inspection (i.e., whether the AI verifies whether the element changes after a period of time, such as checking whether the appearance of the element changes dynamically over time), difference inspection (i.e., whether the AI verifies whether the element differs from the previous operation, such as whether the wallpaper is different after swiping to change the wallpaper), existence inspection (i.e., whether the AI verifies whether the element exists), and regular expression inspection (i.e., whether the text content contained in the AI verifies whether the element matches a regular expression, such as checking whether a clock is in 12-hour or 24-hour format).
[0030] Test operations refer to a series of interactive behaviors performed by an AI agent on a graphical user interface according to test operation instructions, with the aim of simulating user operations to trigger specific system responses.
[0031] The inspection operation refers to the AI agent analyzing and verifying the current state or content of the graphical user interface according to the inspection operation instructions in order to determine whether the test operation has achieved the expected results.
[0032] Test results refer to the judgments or reports made by the AI agent regarding whether the functions and behaviors of the graphical user interface meet expectations after performing test and inspection operations.
[0033] This embodiment provides a graphical user interface testing method, the specific implementation process of which may include the following steps: First, obtain graphical user interface test cases in natural language. Test cases can be added directly to the system by testers, including the actions and expected outcomes. For example, in automotive infotainment testing, testers might first add action 1 "Open system settings," then action 2 "Click the connection tab," then action 3 "Click the WLAN card," and finally the expected outcome "Display the WLAN pop-up." Alternatively, test cases can be read from a pre-prepared text file, which may contain a series of test steps described in natural language.
[0034] Subsequently, the AI agent analyzes the test cases to generate corresponding test operation instructions and inspection operation instructions. Specifically, the AI agent is configured to receive natural language text and use its language understanding capabilities to identify key information such as verbs and nouns in the text, converting them into structured, AI-executable instructions. For example, for the description "Open system settings," the AI agent can generate the test operation instruction "Launch application: System Settings"; for the descriptions "Click the Connection tab" and "Click the WLAN card," the AI agent can generate the test operation instructions "Click card: Connect" and "Click card: WLAN"; and for the description "Display WLAN pop-up," the AI agent converts it into the inspection operation instruction "Check: Does the pop-up image and text match WLAN?"
[0035] Next, based on the generated test operation instructions, the AI agent executes the corresponding test operations on the graphical user interface. The AI agent is configured to simulate user interaction behavior on the graphical user interface. For example, when receiving the test operation instruction "Launch application: System Settings", the AI agent autonomously finds the "System Settings" application among all applications and performs the opening operation; when receiving the instruction "Click card: Connect", the AI agent finds the card named "Connect" among all elements on the interface and performs the clicking operation.
[0036] After the test operation is completed, based on the inspection operation instructions, the AI agent performs corresponding inspection operations on the interface content of the graphical user interface and obtains the test results. The AI agent is configured to analyze and verify the current state of the graphical user interface according to the inspection operation instructions. For example, when receiving the inspection operation instruction "Check: Does the pop-up image and text WLAN match?", the AI agent identifies whether the image being inspected on the interface is a pop-up image and whether the text being inspected is "WLAN". If they match, the inspection passes; if they do not match, the inspection fails. Finally, based on the execution of all inspection operations, a comprehensive test result is generated, indicating the pass or fail status of the entire test case.
[0037] This embodiment effectively solves the problems of excessive reliance on manual labor, high professional barriers, and low efficiency in traditional graphical user interface (GUI) testing by introducing an AI agent to automate the parsing, execution, and inspection of test cases in natural language form. This method allows non-professionals to easily create and execute test cases, significantly lowering the technical threshold for testing. Simultaneously, the AI agent can generate operation instructions for different systems and test multiple devices at the same time, significantly improving testing efficiency and accuracy. It addresses the problems of low efficiency and high barriers to entry in existing GUI testing technologies.
[0038] In some of the solutions mentioned above in this application, AI agents are used to perform test and inspection operations to automate the testing process. However, in this process, the test results lack a systematic recording and review mechanism, and the test scripts are not saved for reuse, resulting in low testing efficiency, potential errors not being discovered in a timely manner, and the inability to reuse scripts, which limits the optimization of the overall testing process.
[0039] In response, this application further proposes a graphical user interface testing method, which includes, after the test operation and inspection operation are completed, generating a test report of the test process; receiving the review result of the test report; and if the review result is passed, encapsulating the test operation instructions and inspection operation instructions into a test script and saving it.
[0040] Specifically, after the AI agent executes corresponding test operations on the graphical user interface based on test operation instructions, and performs corresponding inspection operations on the interface content of the graphical user interface based on inspection operation instructions after the test operation is completed, this method generates a test report of the test process. This test report aims to systematically record the detailed situation of test execution, including but not limited to test steps, operation logs, key interface screenshots, checkpoint results (pass / fail), error messages, and execution time. Its purpose is to provide comprehensive and objective data support for subsequent test result review, problem tracing, and test process optimization. In specific implementation, various methods can be adopted: for example, the system can automatically collect various data generated by the AI agent during the execution of test and inspection operations, and integrate them into a structured report file according to a preset report template, corresponding to the corresponding test cases and presented visually. The report can be exported as a PDF file for easy manual review, or exported to a database for viewing on another device; alternatively, the system can write key events and data during the test process into a dedicated test database in real time, and after the test is completed, extract and organize the data from the database into the final test report using data query and report generation tools.
[0041] Subsequently, this method receives the review results of the test report. These review results are feedback after evaluating the generated test report, used to determine the effectiveness of the test execution, the accuracy of the results, and whether they meet expectations, ensuring the quality and reliability of the test results and avoiding the use of inaccurate or problematic test results as the final basis. In practical applications, receiving review results can be achieved through: for example, a user review interface allowing testers to view detailed test report content and manually select a "pass" or "fail" review conclusion; or, the system can integrate automated review rules, for example, when all preset critical checkpoints pass and no serious errors occur, the system automatically suggests a "pass" review result, but still requires final manual confirmation to balance efficiency and accuracy.
[0042] If the review is successful, the test operation instructions and inspection operation instructions are encapsulated into test scripts and saved. The core of this step is to solidify the validated and successful test logic into reusable automated test scripts, thereby improving testing efficiency, reducing repetitive work, and building a reliable automated test script library. Specifically, this can include: for example, packaging the test operation instruction sequence and inspection operation instruction sequence generated by the AI agent based on natural language test cases, which have been successfully executed and passed review, along with their execution order, parameter configuration, and necessary metadata (such as script name, version, creator, associated test case ID, etc.), into a single, executable script file and storing it in a version control system or dedicated script management platform; or, these instructions can be stored in a backend database as structured data (such as JSON, YAML) and assigned a unique identifier for subsequent direct invocation and execution.
[0043] In some of the embodiments described above in this application, it is proposed to generate instructions by parsing test cases using an AI agent to execute tests. However, in its implementation, if a complete script that has been saved is not used, it will lead to repeated parsing of the same test cases, increasing processing time and resource consumption, and reducing test efficiency.
[0044] In response, this application further proposes that, before parsing test cases through an AI agent to generate corresponding test operation instructions and inspection operation instructions, the method also includes: determining whether a complete script that matches the test cases and has been saved exists; if a complete script exists, then calling the complete script to complete the graphical user interface test.
[0045] Specifically, upon receiving a graphical user interface test case in natural language format, the system first determines whether a complete script matching the test case exists and has been saved. This determination aims to avoid repeatedly generating existing test scripts, thereby improving testing efficiency and resource utilization. One implementation involves the system maintaining a script library that stores approved and saved complete test scripts. When a new test case is received, the system can extract features from it, such as key operation sequences, target elements, and expected results using natural language processing techniques. This information is then compared with the metadata (such as script name, associated test case description, hash value of the operation sequence, or semantic embedding) of all complete scripts in the script library. If the matching degree reaches a preset threshold (e.g., through a semantic similarity algorithm or operation sequence matching algorithm), a matching complete script is considered to exist. Another implementation involves pre-storing the fingerprint information of the corresponding natural language test case for each saved complete script, such as hash value, keyword vector, or semantic embedding. When a new natural language test case is input, its fingerprint information is calculated and quickly compared with the fingerprint information of all complete scripts in the script library. If a complete script with completely identical or highly similar fingerprint information exists, then a matching complete script is considered to exist.
[0046] If a complete script is found, the system will directly invoke it to complete the graphical user interface test. This step reuses existing, validated test scripts, avoiding the AI agent from repeatedly parsing and generating instructions, thus saving computational resources and time. One implementation is that, upon determining the existence of a matching complete script, the system directly loads that script from the script library. This complete script contains predefined test operation instructions and check operation instructions. The system then executes the corresponding test and check operations on the target graphical user interface according to these instructions through the AI agent (or automated test executor) until the test is completed and the test results are obtained.
[0047] Through the above technical solution, before the AI agent parses test cases to generate instructions, the system can first check whether there is a saved complete script that matches the current test case. This effectively avoids repeated parsing of the same or similar test cases, significantly reducing the computational load and processing time of the AI agent. When a matching complete script exists, the system can directly call that script to execute the test, thus eliminating the step of the AI agent parsing and generating instructions, greatly improving testing efficiency and optimizing the utilization of computing resources. This not only speeds up the testing process but also reduces the uncertainty that may be introduced by repeated parsing, ensuring the stability and reliability of the test.
[0048] In some embodiments, this application further proposes that, before the AI agent parses the test cases and generates corresponding test operation instructions and inspection operation instructions, the method further includes: determining whether there is a complete script that matches the test case and has been saved; if there is no complete script, determining whether there is a test case that contains the same steps as the current test case; if there is, reusing the script corresponding to the same steps in the test case; and generating corresponding test operation instructions and inspection operation instructions for the remaining different steps by the AI agent, and executing the test operation and inspection operation by the AI agent.
[0049] This application adds a script reuse judgment before the AI agent generates test instructions: priority is given to reusing complete scripts that match the test cases; if no complete script is available, script fragments with the same steps are reused; only the remaining different steps are generated and executed by the AI agent, which ensures test coverage while reducing repeated generation and improving execution efficiency and reliability.
[0050] In other specific embodiments, it may also include: determining whether there is a test case that is continuously identical to the current test case from the first step; if so, reusing the scripts corresponding to all steps in the test case from the first step to the first occurrence of different steps; the remaining different steps are generated by the AI agent with corresponding test operation instructions and inspection operation instructions, and the AI agent executes the test operation and inspection operation.
[0051] The specific implementation is as follows: When a new test case is received, the description text of each step is extracted starting from the first test step. This description is then compared with the steps in the saved test cases to determine if the step content and execution order are the same. If, starting from the first step, the content and execution order of several consecutive steps are identical, then a matching script exists for these consecutive steps. For example, if the current test case contains four steps (A, B, C, D), and the saved test case contains five steps (A, B, C, F, G) with a saved and approved script, and it is determined that steps A, B, and C have the same content and execution order starting from the first step, then steps A, B, and C can use the saved script to execute the test. The remaining step, D, is completed by an AI agent generating instructions. If the current test case contains four steps (A, B, C, D), and the saved test case contains five steps (B, C, D, G) with a saved and approved script, then the script for the saved test case cannot be reused because the first step of the current test case and the saved test case are different. Additionally, it should be noted that if there are multiple test cases that are consecutively identical to the current test case from the first step, the test case with the most consecutive identical steps should be selected as the best test case, and the scripts corresponding to all steps from the first step to the first occurrence of a different step in the best test case should be reused.
[0052] If a script exists that matches the content and execution order of several consecutive steps starting from the first step, the system will directly call the script to complete the testing of those steps in the graphical user interface. This step can directly reuse existing, verified test scripts, avoiding the AI agent repeatedly parsing and generating instructions, thus saving computing resources and time. One implementation is that when a test case is found to have several consecutive steps identical to the current test case starting from the first step, the system directly loads the script for the corresponding steps of that test case from the script library. This script contains the test operation instructions and check operation instructions that the AI agent has already executed. These steps can be tested directly according to these operation instructions. The remaining steps are then tested by the AI agent generating test operation instructions and check operation instructions, and automatically executing the test operations and check operations until the test is completed and the test results are obtained. In particular, if all steps have the same content and execution order and a saved script exists, all steps can be tested and checked according to the script, without the AI agent needing to parse, generate instructions, and execute the test.
[0053] In some of the solutions mentioned above in this application, a test report is generated and the audit results are received after the test and inspection operations are completed. If the results are passed, the script is packaged and saved. However, in this process, if the audit results are not passed, there is a lack of an effective mechanism to automatically adjust the test instructions and re-execute the test, which leads to low test efficiency and increased manual intervention, thereby prolonging the test cycle and increasing resource consumption.
[0054] In response, this application further proposes that, after receiving the review results of the test report, if the review results are not satisfactory, the adjusted graphical user interface test cases will be received; the adjusted test cases will be re-parsed by the AI agent to generate adjusted test operation instructions and inspection operation instructions; based on the adjusted test operation instructions and inspection operation instructions, the adjusted test operations and inspection operations will be executed by the AI agent, and the test report will be regenerated until the review results are satisfactory.
[0055] Specifically, the purpose of receiving adjusted graphical user interface test cases is to provide an adjustment entry point for the testing process. This allows for correction or optimization instructions based on specific reasons for failure or deviations from expected behavior when a test report fails review. The adjusted test cases can be manually modified by human testers, who can either change the step descriptions or add necessary steps to ensure more accurate parsing by the AI agent. Alternatively, they can be automatically optimized by a large AI model first, followed by further optimization by humans based on the AI-optimized test cases.
[0056] Furthermore, based on the adjusted test cases, the AI agent executes the adjusted test and inspection operations, and regenerates the test report until the review result is passed. This step describes a closed-loop automated correction and verification process. After generating the adjusted test and inspection operation instructions, the AI agent will execute the corresponding test and inspection operations on the graphical user interface according to these new instructions. For example, the AI agent can simulate user input, clicks, swipes, and other interactive behaviors, and check the text, images, layout, and other content on the interface. After completing one round of adjusted testing, the system will regenerate a test report reflecting the results of the adjusted test execution. Subsequently, the system will receive the review result of this new test report again. If the review result still fails, it will re-enter the stage of receiving adjusted test cases, forming an iterative optimization loop until the test report finally passes the review, ensuring the correctness and effectiveness of the test cases.
[0057] In addition, the number of adjustments can be set. If the AI agent still fails the test result review after the number of adjustments is reached, it will be transferred to a human for retesting.
[0058] In some of the solutions mentioned above in this application, graphical user interface test cases in natural language form are proposed for subsequent AI agent parsing and test execution. However, in the implementation process, if the test cases are provided in the form of files in non-natural language format, users need to manually convert or perform additional processing, which increases the testing threshold and reduces the overall testing efficiency.
[0059] In response, this application further proposes a graphical user interface testing method, which includes, before obtaining the graphical user interface test cases in natural language form, receiving an imported test case file; converting the test case file into a test case file in a preset format using a preset conversion script; and extracting the test cases in natural language form from the converted test case file.
[0060] Specifically, receiving imported test case files aims to allow the system to obtain test case data from external sources, thereby avoiding manual input or copy-pasting by users and improving data import efficiency. This step can be implemented in several ways. For example, the system can provide a file selector on a graphical user interface (GUI) that allows users to browse and select test case files in the local file system for import; alternatively, the system can be configured to receive file streams or file paths from other test management systems or version control systems through specific API interfaces, enabling automated or batch import. Furthermore, it can automatically detect and import test case files placed in preset directories by monitoring specific directories.
[0061] If the imported test case file is not in a standard format supported by the system, the system will convert it into a pre-defined format using a pre-defined conversion script. The purpose is to automate the process by unifying test cases of different formats into a standard format that the system can recognize and process, thus ensuring compatibility and standardization in subsequent processing and reducing the requirements for the input file format. For example, the pre-defined conversion script could be a parser written in programming languages such as Python or Java, specifically designed to parse common test case file formats such as XML, CSV, and Excel, and then reconstruct their content into an internally defined unified file format, such as JSON.
[0062] Furthermore, natural language test cases are extracted from the converted test case file. The purpose is to identify and extract user-understandable text content describing test steps and expected results from the standardized test case data. This provides direct natural language input for subsequent AI agent parsing, ensuring that the AI agent can accurately understand the test intent. In specific implementations, in a preset format, natural language test cases may be stored in specific fields, such as "step description" or "expected result" fields, and the system can directly read the contents of these fields. Alternatively, if the preset format is structured text, the natural language description portion can be identified and extracted through specific markers (such as [STEP], [ACTION], [EXPECT], etc.) or paragraph separators.
[0063] Through the above technical solution, this application effectively solves the problem that the system cannot extract file content when test cases are provided in non-standard file formats. Receiving imported test case files allows users to directly import test case files of various formats, greatly reducing manual intervention and tedious file processing operations. Subsequently, a preset conversion script automatically converts these files into a unified preset format, ensuring that test cases from different sources and formats can be standardized, without requiring users to have professional format conversion skills. Finally, test cases in natural language form are extracted from the converted files, providing standardized and directly understandable input for the subsequent efficient parsing and test execution of the AI agent, thereby significantly lowering the technical threshold for testing and improving the automation level and overall efficiency of the entire graphical user interface testing process.
[0064] The following example will provide a more detailed explanation of the above technical solution: In automotive infotainment system function testing, test engineers need to test the system settings module, one of the tests being whether the media audio cards can be opened normally.
[0065] Test engineers first receive an imported test case file. This file may be a spreadsheet containing multiple test cases. The system uses a pre-defined conversion script to convert the file into a pre-defined standard format and extracts the graphical user interface test cases in natural language form, such as: "Action - Open System Settings, Action - Click the Sound tab, Action - Find the Media Sound Effects card, Expected - Display background image and title: Media Sound Effects, Expected - Subtitle: Immersive Sound Effects." This step avoids the tedious work of testers manually entering or converting test cases, improving the efficiency of test case preparation.
[0066] After obtaining the natural language test case, the system first checks if a test case exists that is identical to the current test case from the very first step. If so, it reuses the scripts corresponding to all steps from the first step until the first different step appears, thus avoiding redundant parsing and instruction generation and improving testing efficiency. For example, if the steps "open system settings" and "click the sound tab" in the test case are exactly the same as the first two steps of the saved test case A, then the scripts for those two consecutive steps are invoked to execute the corresponding operations. For the remaining steps in the test case (such as "Operation - find media sound effect card, Expected - display background image and title: media sound effect, Expected - subtitle: immersive sound effect"), the system uses an AI agent to parse them and generate corresponding test operation instructions and inspection operation instructions. This approach fully utilizes validated scripts, reduces the scope of parsing required by the AI agent, and improves efficiency and accuracy.
[0067] If no matching steps exist from the first step, the system will directly parse the entire natural language test case using an AI agent. The AI agent performs semantic understanding on statements such as "open system settings," "click the sound tab," and "find the media sound card," generating a series of specific test operation instructions. For example: open the "System Settings" application, click the "Sound tab" icon, swipe the screen to find the "Media Sound Card" button and click it, check the "background image," check if the title text is "Media Sound Effects," and check if the subtitle text is "Immersive Sound Effects." Simultaneously, the AI agent will also parse the expected descriptions "display background image and title: Media Sound Effects" and "subtitle: Immersive Sound Effects," generating corresponding inspection operation instructions, such as: check the "background image," check if the title text is "Media Sound Effects," and check if the subtitle text is "Immersive Sound Effects." Compared to traditional code-based automated testing, this natural language parsing method significantly lowers the technical barrier for testers, eliminating the need to write complex code and avoiding errors caused by improper operation in manual testing.
[0068] Based on these generated test operation instructions, the AI agent executes corresponding test operations on the target vehicle's graphical user interface. For example, the AI agent simulates user interactions such as clicking and swiping, and completes the entire process of opening an application, clicking an icon, swiping to find and click a button step by step according to the instruction sequence. This process is fully automated, overcoming the limitation that manual testing cannot be performed continuously around the clock, and also avoiding the problems of complex equipment deployment and limited scenarios in robotic arm testing.
[0069] After all testing operations are completed, the AI agent performs corresponding checks on the graphical user interface content based on the previously generated inspection instructions. For example, the AI agent analyzes whether the background image of the current screen is a media sound effect card, whether the title text is "media sound effects," and whether the subtitle text is "immersive sound effects." In this way, the AI agent can automatically judge the test results, obtain a conclusion of whether the test passed or failed, avoid the misjudgment that may occur with manual judgment, and can detect problems in a timely manner.
[0070] After the testing and inspection operations are completed, the system will automatically generate a detailed test report, which includes the execution steps of the test cases, the operation records of the AI agent, the inspection results, and the final test conclusions. This test report is then submitted for review.
[0071] If the review is successful, the system will encapsulate each step of the test and inspection processes generated by the AI agent in this test into a test script and save it. These saved scripts can be directly called in subsequent tests, further improving testing efficiency.
[0072] If the review fails, for example, if the reviewer finds that the AI agent cannot find the media sound effect card, the step in the test case can be optimized by describing it as "swipe the screen and find the media sound effect card". Based on the optimized test case, the AI agent will re-parse and generate test operation instructions and inspection operation instructions. Subsequently, the AI agent will re-execute the adjusted test operation and inspection operation based on these instructions and regenerate the test report. This iterative process will continue until the review is passed, ensuring the accuracy and reliability of the test while avoiding the tedious work of testers manually modifying the script.
[0073] In one embodiment, this application also provides a graphical user interface testing apparatus. For example... Figure 2 As shown, it includes an acquisition module 21, a parsing module 22, a testing module 23, and an inspection module 24. Detailed descriptions of each functional module are as follows: Module 21 is used to acquire graphical user interface test cases in natural language form; The parsing module 22 is used to parse test cases through an AI agent and generate corresponding test operation instructions and inspection operation instructions; Test module 23 is used to execute corresponding test operations on the graphical user interface through an AI agent based on test operation instructions; The inspection module 24 is used to perform corresponding inspection operations on the interface content of the graphical user interface through an AI agent based on the inspection operation instructions after the test operation is completed, and obtain the test results.
[0074] This invention also provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the aforementioned graphical user interface testing method; to avoid repetition, this will not be described again here. Alternatively, the electronic device can implement the functions of each module in this embodiment of the graphical user interface testing device; this will not be described again here.
[0075] This invention also provides a readable storage medium storing a program. When the program is executed by a processor, it implements the aforementioned graphical user interface testing method. To avoid repetition, this will not be described again here. Alternatively, when the program is executed by a processor, it implements the functions of each module in this embodiment of the graphical user interface testing device, which will also not be described again here.
[0076] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
[0077] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0078] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
Claims
1. A graphical user interface testing method, characterized in that, include: Obtain test cases for graphical user interfaces in natural language format; The test cases are analyzed by an AI agent to generate corresponding test operation instructions and inspection operation instructions. Based on the test operation instructions, the AI agent executes the corresponding test operation in the graphical user interface. After the test operation is completed, based on the inspection operation instruction, the AI agent performs the corresponding inspection operation on the interface content of the graphical user interface to obtain the test result.
2. The graphical user interface testing method according to claim 1, characterized in that, After the test and inspection operations are completed, the following are included: Generate a test report for the testing process; The system receives the review result of the test report. If the review result is satisfactory, the system encapsulates the test operation instructions and inspection operation instructions into a test script and saves it.
3. The graphical user interface testing method according to claim 1, characterized in that, Before the step of parsing the test cases using an AI agent to generate corresponding test operation instructions and inspection operation instructions, the method further includes: Determine if a complete script matching the test case exists and has been saved; If the complete script exists, then the complete script is invoked to complete the graphical user interface test.
4. The graphical user interface testing method according to claim 1, characterized in that, Before the step of parsing the test cases using an AI agent to generate corresponding test operation instructions and inspection operation instructions, the method further includes: Determine if a complete script matching the test case exists and has been saved; If the complete script does not exist, then determine whether there is a test case containing the same steps as the current test case; If it exists, reuse the script corresponding to the same step in that use case; The remaining different steps are generated by the AI agent, which generates corresponding test and inspection instructions and executes the test and inspection operations.
5. The graphical user interface testing method according to claim 1, characterized in that, Before the step of parsing the test cases using an AI agent to generate corresponding test operation instructions and inspection operation instructions, the method further includes: Determine if a complete script matching the test case exists and has been saved; If the complete script does not exist, determine if there is a test case that is identical to the current test case from the first step onwards; if so, reuse the script corresponding to all steps in that test case from the first step until the first occurrence of a different step. The remaining different steps are generated by the AI agent, which generates corresponding test and inspection instructions and executes the test and inspection operations.
6. The graphical user interface testing method according to claim 2, characterized in that, After receiving the review result of the test report, the process also includes: If the review result is not approved, the revised graphical user interface test cases will be accepted. The adjusted test cases are re-analyzed by an AI agent to generate adjusted test operation instructions and inspection operation instructions; Based on the adjusted test operation instructions and inspection operation instructions, the AI agent executes the adjusted test operations and inspection operations, and regenerates the test report until the review result is approved.
7. The graphical user interface testing method according to claim 1, characterized in that, Before obtaining the graphical user interface test cases in natural language form, the method further includes: Receive imported test case files; The test case file is converted into a test case file of a preset format using a preset conversion script; Extract the test cases in natural language form from the converted test case file.
8. A graphical user interface testing device, characterized in that, include: The acquisition module is used to acquire graphical user interface test cases in natural language format. The parsing module is used to parse the test cases through an AI agent and generate corresponding test operation instructions and inspection operation instructions; The testing module is used to execute corresponding test operations on the graphical user interface through the AI agent based on the test operation instructions. The inspection module is used to perform corresponding inspection operations on the interface content of the graphical user interface through the AI agent based on the inspection operation instructions after the test operation is completed, and obtain the test results.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the graphical user interface testing method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the graphical user interface testing method as described in any one of claims 1 to 7.