Test method, device and electronic equipment of application program

Abstract

This application provides a testing method, apparatus, and electronic device for an application. The testing method includes: acquiring test cases for testing the application in the device under test; the test cases include action steps, and the format of the test cases is a standardized format adapted to a first test model; executing corresponding test operations according to the action steps; the test operations include: acquiring a first test operation instruction in the current action step and a current screenshot of the device under test, jointly parsing the first test operation instruction and the current screenshot through the first test model, generating a second test operation instruction based on the parsing result, and sending the second test operation instruction to the device under test to drive the device under test to perform corresponding interactive operations on the application according to the second test operation instruction. The technical solution of this application can improve the automation level and execution efficiency of the testing process.

Description

Technical Field

[0001] This application relates to the field of smart terminal testing, and more particularly to a method, apparatus, and electronic device for testing applications. Background Technology

[0002] With the ever-accelerating pace of mobile application development, traditional automated testing solutions have revealed significant limitations in handling frequent UI (User Interface) updates and business logic changes. When the application's user interface is adjusted or the functional flow is refactored, test engineers need to spend a lot of time and manpower to update and debug a large number of test cases, resulting in high maintenance costs for test cases and severely restricting testing efficiency and release cycles, with poor iterative adaptability. Summary of the Invention

[0003] In view of the above, this application provides a method, apparatus, and electronic device for testing applications to solve the problems existing in the prior art. The technical solution is as follows:

[0004] In a first aspect, embodiments of this application provide a method for testing an application, including:

[0005] Obtain test cases for testing the application in the device under test; the test cases include action steps, and the format of the test cases is a standardized format adapted to the first test model;

[0006] The test operation is performed according to the action steps. The test operation includes: obtaining the first test operation instruction in the current action step and the current screenshot of the device under test; performing joint analysis of the first test operation instruction and the current screenshot through the first test model; generating the second test operation instruction based on the analysis result; and sending the second test operation instruction to the device under test to drive the device under test to perform the corresponding interactive operation of the application according to the second test operation instruction.

[0007] The first test model is a large visual language model with the ability to understand the visual interface of a graphical user interface.

[0008] Secondly, embodiments of this application provide an application testing apparatus, comprising:

[0009] The test case acquisition module is used to acquire test cases for testing the application in the device under test; the test cases include action steps, and the format of the test cases is a standardized format adapted to the first test model;

[0010] The test case execution module is used to execute corresponding test operations according to the action steps. The test operations include: obtaining the first test operation instruction in the current action step and the current screenshot of the device under test; performing joint parsing of the first test operation instruction and the current screenshot through the first test model; generating a second test operation instruction based on the parsing result; and sending the second test operation instruction to the device under test to drive the device under test to perform corresponding interactive operations on the application according to the second test operation instruction.

[0011] The first test model is a large visual language model with the ability to understand the visual interface of a graphical user interface.

[0012] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor, wherein the memory stores a computer program, which is loaded and executed by the processor to implement the method provided in the first aspect of embodiments of this application.

[0013] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the first aspect of embodiments of this application.

[0014] The above-mentioned technical solution of this application can achieve at least the following beneficial effects:

[0015] A first test model with graphical user interface (GUI) visual understanding capabilities is employed. This model performs joint analysis of the first test operation command and the image under test based on visual and semantic understanding, achieving functional semantic understanding and dynamic recognition of user interface elements. Based on this, an executable second test operation command is automatically generated, driving the device under test to perform corresponding interactive operations. This approach effectively overcomes the problems of script breakage and high maintenance costs associated with traditional automated testing in scenarios involving changes to the GUI and version iterations. It significantly reduces manual maintenance costs, improves testing efficiency, and exhibits good robustness and generalization capabilities. Even if the application interface is restructured or the control positions are adjusted, the first test model can still accurately infer the operation target based on visual context and task intent, achieving continuous testing capabilities of "modeling once, applicable to multiple versions." Furthermore, the embodiments of this application use a standardized format to uniformly organize test cases, which facilitates model parsing and batch processing, effectively improving the automation level and execution efficiency of the testing process. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the hierarchical architecture of the test plan in the embodiments of this application;

[0018] Figure 2 This is a flowchart illustrating the execution of a test plan in an embodiment of this application;

[0019] Figure 3 This is a schematic diagram of the hierarchical architecture of the test system provided in the embodiments of this application;

[0020] Figure 4 A flowchart illustrating a testing method for an application provided in this application embodiment;

[0021] Figure 5 This is a schematic diagram illustrating the principle of cyclic testing in an embodiment of this application;

[0022] Figure 6 This is a schematic diagram of the system prompts in the embodiments of this application;

[0023] Figure 7 This is a table illustrating historical test cases in the embodiments of this application;

[0024] Figure 8 This is a schematic diagram of the conversion template in the embodiments of this application;

[0025] Figure 9 This is a schematic diagram illustrating the merging of historical test cases and system prompts in an embodiment of this application;

[0026] Figure 10 and Figure 11 This is a schematic diagram illustrating text parsing and conversion in an embodiment of this application;

[0027] Figure 12 This is a schematic diagram of the structural framework of a testing device for an application provided in an embodiment of this application. Detailed Implementation

[0028] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0029] It should be noted that, unless otherwise specified, the following embodiments and features can be combined with each other; and, based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] It should be noted that this document describes various aspects of embodiments within the scope of the appended claims. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on the embodiments of this application, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0031] The relevant terminology in this application is explained below:

[0032] Test case (or TestCase): A set of steps and expected results designed to verify a specific software function.

[0033] Action steps: A step used to implement a specific test in the test plan.

[0034] Inspection step: A step used to confirm the result of the execution of an action step.

[0035] Instant check steps: Check steps to verify whether the current screenshot meets a specific condition.

[0036] Action check step: A check step to verify whether the previous action step was executed successfully.

[0037] Multi-image inspection steps: This step verifies a change by comparing two screenshots taken at different times.

[0038] Standard image check steps: This step verifies whether the interface display is correct by comparing the current screenshot with a pre-saved standard image.

[0039] Test cases are typically included in the test plan, such as Figure 1 As shown, a test plan can be divided into several levels: test case group, test case, section, and step. A test case group can include multiple test cases. For example, a mobile phone model upgrade test group includes multiple test cases for upgrading the mobile phone. One of the test cases may be, for example, Google Play (a digital content distribution platform) switch verification. A test case can include multiple section; a section can include multiple steps.

[0040] Step sections are used to mark different phases of a test case. Different phases can have different names. In one example, the step section for the setup / preparation phase could be named Setup, the step section for the test step phase could be named TestStep, and the step section for the cleanup / recovery phase could be named Teardown. When executing step sections, they can be executed in the order of Setup, TestStep, and Teardown. If an error occurs during Setup, the remaining two phases' step sections will not be executed; if an error occurs during TestStep, Teardown needs to continue execution since Setup has already completed. When executing a specific step section, the steps within it can be executed sequentially. If a problem occurs in a step, the remaining steps within its scope will not be executed.

[0041] Setup contains steps that create prerequisites for test cases, representing all the preparation and preconditions before the test cases are executed. The value of this step section is a string containing the environment, data, or state that needs to be met to execute the test. Multiple steps or conditions are separated within the string by newline characters \n. Each line is usually a specific action or status check, such as "Open Settings, search for GMS (Google Mobile Services), and turn on GMS".

[0042] TestStep contains the core test steps, representing the core test steps and the corresponding expected results (ExpectedResult) verification. This step section is also a multi-line string separated by newline characters \n. This string contains the specific action (starting with action:) and checkpoints or assertions used to verify whether the action is successful (starting with check:), such as "Turn on the Google Basics switch, click the top left corner to return to the Post & Sync page", "The page exists on Google", etc.

[0043] Teardown includes steps to close out test cases, ensuring their completion. This section represents all cleanup work done after the test cases have finished executing. Its main purpose is to restore the application to its initial state before test execution or a known stable state to ensure that tests do not interfere with each other. This typically includes operations such as deleting test data, closing connections, and logging out users, for example, "return to desktop". The value of this section is also a multi-line string separated by newline characters \n, containing only the action steps (starting with action:), because cleanup steps usually do not require verification (check).

[0044] When executing a test plan, such as Figure 2 As shown, execution can proceed layer by layer in the hierarchical order of test case groups, test cases, step segments, and steps. First, select the test case group to be tested, traverse and execute each test case in the test case group until there are no remaining test cases to execute. When traversing each test case, traverse and execute each step segment of the test case until there are no remaining step segments to execute. When traversing each step segment, traverse and execute each step sub-flow in the step segment until there are no remaining steps in the step segment.

[0045] The technical solution of this application can be applied to a testing system that can execute test cases to test applications in the device under test. The testing system may include a testing device and a server, which can communicate via a network. The testing device and the device under test can also communicate via a network or wired connection. Specifically, the testing device can execute the application testing methods provided in the embodiments of this application to test the applications in the device under test. The server trains the intelligent model (e.g., a large visual-language model) required for testing to enable the relevant intelligent model to possess the testing capabilities required by the testing solution of this application. During the testing process, the testing device can call the trained intelligent model provided by the server to execute corresponding test operations, or the testing device and the server can jointly execute corresponding test operations. For example, the testing device can obtain and send the request data required by the intelligent model to the server. The server inputs the request data into the intelligent model for processing, and after processing, returns the instructions or information output by the intelligent model to the testing device. The testing device then executes subsequent operations or terminates the test based on the instructions.

[0046] Both the test device and the device under test can be any type of terminal device, such as a smartphone, tablet, or large-screen terminal device. These can be portable, pocket-sized, handheld, built-in computer devices, or vehicle-mounted mobile devices. There can be one or more devices under test. Based on the unique identification number (i.e., serial number) of one or more terminal devices provided by the tester, the test device can use a dedicated connection tool to locate the one or more terminal devices corresponding to the unique identification number among all connected terminal devices, designate them as the device under test, and click the test button to begin the test.

[0047] Figure 3This document illustrates a specific architecture example of the test system described in this application. The architecture is divided into four layers: PC, Browser, Server, and Infrastructure. The PC layer deploys an Android Debug Bridge Service (ADB Server), responsible for establishing connections with Android devices and handling functions such as device control, screenshotting, and command execution. The Browser layer deploys a Chrome Extension and a Vue.js built-in frontend. The Chrome Extension is used to capture page information, trigger frontend operations, or interact with backend services. The frontend provides a visual interface for operations such as test plan configuration and result display. The server layer deploys a FastAPI (Application Programming Interface) framework, functional modules, and tool libraries. The FastAPI framework handles frontend requests and interface routing, serving as a communication hub between frontend and backend modules. Functional modules include an Artificial Intelligence (AI) testing module and an error detection module. The AI ​​testing module tests the application on the device under test, while the error detection module identifies errors during testing. Tool libraries include Adbutils (a Python wrapper for ADB tools), SQLAlchemy (a Python ORM framework), LangChain (a large model application development framework), jieba (a Chinese word segmentation tool), APScheduler (a scheduled task library), tenacity (a retry mechanism library), Pillow (a Python image processing library), and PyTorch (a deep learning framework), enabling simplified debugging, database operations, connection to large language models (LLMs), word segmentation, and other functions. The infrastructure layer is equipped with a Vision-Language Model (VLM), a text-based large language model, and a lightweight database (SQLite). The Vision-Language Model enables GUI visual understanding and semantic analysis, the text-based large language model enables natural language interaction and test logic generation, and the lightweight database can store test data, task records, result logs, etc.

[0048] The technical solution of this application and how the technology of this application solves the above-mentioned technical problems are described in detail below with specific embodiments.

[0049] This application provides a method for testing applications, such as... Figure 4As shown, the method may include the following steps S401 and S402.

[0050] S401, Obtain test cases for testing the application in the device under test.

[0051] Test cases include action steps, and the format of the test cases is a standardized format adapted to the first test model. The standardized format can be a semi-structured format or a structured format. The first test model is a large visual language model with GUI visual understanding capabilities.

[0052] Test cases can be provided by testers, who can input test case information through input devices, visual input interfaces, etc.

[0053] S402, execute the corresponding test operation according to the action steps in the test case.

[0054] The aforementioned test operations include: acquiring a first test operation instruction and a current screenshot of the device under test in the current action step; jointly parsing the first test operation instruction and the current screenshot using a first test model; generating a second test operation instruction based on the parsing result; and sending the second test operation instruction to the device under test to drive the device under test to perform corresponding interactive operations on the application according to the second test operation instruction. The first test operation instruction may be a natural language instruction.

[0055] After receiving the second test operation instruction, the device under test will complete the specific interactive operation according to the instruction type of the second test operation instruction. The instruction type includes, but is not limited to, one or more types such as click, long press, type, drag, scroll, open app, press back, and press home. Click and long press instructions can be used to instruct the device under test to perform click and long press operations at specified coordinate positions. Type input instructions can be used to instruct the device under test to enter the text content to be entered in the current input box. Drag and scroll instructions can be used to instruct the device under test to perform drag and scroll operations according to the start and end coordinates. Open app instructions can be used to instruct the device under test to launch the corresponding application according to the application name. Back and return to home screen instructions can be used to instruct the device under test to perform back and return to home screen operations.

[0056] When there are multiple test cases to be executed, the embodiments of this application can execute the steps in each test case in a pre-set order. For example, multiple test modules can be set on the test device, and each test case can be executed by one test module. The execution of each test case is independent of each other, and the execution of each test case does not affect the execution of subsequent test cases.

[0057] The application testing method provided in this application embodiment employs a first test model with GUI visual understanding capabilities. It performs joint analysis of the first test operation command and the image under test based on visual and semantic understanding, achieving functional semantic understanding and dynamic recognition of user interface elements. Based on this, it automatically generates executable second test operation commands, which drive the device under test to perform corresponding interactive operations. This approach effectively overcomes the problems of script breakage and high maintenance costs in traditional automated testing scenarios involving changes in the graphical user interface and version iterations. It significantly reduces manual maintenance costs, improves testing efficiency, and possesses good robustness and generalization capabilities. Even if the application interface is restructured or the control positions are adjusted, the first test model can still accurately infer the operation target based on visual context and task intent, achieving continuous testing capabilities of "one-time modeling, applicable to multiple versions." Furthermore, this application embodiment uses a standardized format to uniformly organize test cases, which facilitates model parsing and batch processing, effectively improving the automation level and execution efficiency of the testing process.

[0058] The visual language model used in this application uses screenshots (purely visual) as input, does not require reading an accessibility tree to assist in path planning, and is easier to apply to devices on various platforms, such as Android devices and iOS (a mobile operating system) devices, thus having a wide range of applications.

[0059] In an optional implementation, step S402 above, performing the corresponding test operation according to the action step, may include: repeatedly executing the corresponding test operation according to the current action step until the current action step is successfully executed or the number of iterations of the test operation exceeds a preset upper limit for the number of iterations. For each action step in the test case, the specific test operation can be repeatedly executed according to this implementation method. The successful execution of the action step is determined as the completion of the task or goal of the action step. If the task or goal of the action step is still not completed when the number of iterations exceeds the preset upper limit for the number of iterations, it is marked as the execution failure of the action step. The upper limit for the number of iterations of the test operation corresponding to a single action step can be preset according to actual needs.

[0060] Figure 5 This shows a specific example of repeatedly performing test operations on a certain action step. (Refer to...) Figure 5For example, for a certain action step, a natural language instruction describing the specific task or goal, such as "login account," can be extracted from that action step and used as the first test operation instruction. A screenshot of the current screen of the device under test is taken. The first test operation instruction and the current screen screenshot are input into the first test model as a test request. The first test model parses and infers from the first test operation instruction and the image under test, predicting what specific test operation should be performed next, such as clicking, inputting, or swiping, and outputs the predicted instruction sequence for the next specific test operation, i.e., the second test operation instruction. This second test operation instruction is sent to the device under test, which can then perform the corresponding test operation according to the second test instruction. After the test operation, it is determined whether the current action step was successful, such as whether the account was successfully logged in. If successful, the loop test for the current action step ends. If unsuccessful, it is determined whether the number of loop tests for the current action step exceeds a preset loop count limit. If it exceeds the limit, the loop test for the current action step stops; if it does not exceed the limit, the next test operation of the loop test begins.

[0061] The aforementioned loop testing mechanism for individual action steps can automatically trigger retry when the current action step fails, re-executing test operations including screenshot acquisition, model parsing, instruction generation, and device control. This effectively addresses temporary failures caused by factors such as interface loading delays, event packet loss, and brief interaction anomalies, improving the fault tolerance and environmental adaptability of the automated testing process. The introduction of a loop count limit provides an effective control mechanism for the testing scheme in this application embodiment, avoiding infinite loops and invalid repetitions.

[0062] In an optional implementation, step S402, which involves performing the corresponding test operation based on the action step, may further include: during the Nth execution of the test operation, performing a diagnostic analysis on the historical operation record of the current action step using a first test model. Here, N is an integer greater than 1, and the analysis result of the joint parsing may include the result of this diagnostic analysis.

[0063] For example, in Figure 5Based on the illustrated iterative testing operation, information from each test operation can be recorded as a historical operation record after completion. During the Nth test operation, the historical operation records corresponding to previously executed test operations can be input into the first test model, along with the first test operation command and the current screenshot. The first test model can then diagnose and analyze these historical operation records to determine the reasons for failure. Based on these reasons, the generation strategy for the second test command is readjusted, and a new second test command is generated. This "observation-thinking-action" iterative mechanism improves the test pass rate and adaptive decision-making level in complex dynamic scenarios, supports continuous system learning, and provides a more accurate model foundation for subsequent testing processes.

[0064] In an optional implementation, the test cases of this application embodiment may further include a checking step. Correspondingly, the application testing method provided in this application embodiment may further include performing corresponding checking operations according to the checking steps. The checking operation includes: obtaining the test assertion in the checking step and a screenshot of the device under test in a specified state; determining, through a second test model, whether the screen state corresponding to the screenshot conforms to the expected state described by the test assertion; generating a judgment result; and recording the checking step as "successful" or "failed" based on the judgment result.

[0065] The second test model can be a large visual language model, such as a general visual language model, or an instance of the same model as the first test model. Test assertions can describe the expected state of the device under test in natural language. The generated judgment result can include a "pass" or "fail" conclusion and the reason for the judgment.

[0066] Introducing inspection steps into test cases and combining them with a second test model to automatically verify the semantic matching ability of screen states and test assertions can enhance the intelligent perception and quality control capabilities of the test system.

[0067] In one alternative implementation, the above-mentioned inspection steps may include at least one of the following steps: an immediate inspection step, an action-check step, a multi-check step, and an image-check step.

[0068] When performing an inspection operation according to the instant inspection steps, the image to be inspected can be a current screenshot of the device under test, and the test assertion can describe the expected state of the current screenshot, such as "an incoming call pop-up appears on the screen" or "a confirmation message for successful login has appeared".

[0069] When performing an inspection operation according to the action inspection steps, the image to be inspected can be the last screenshot of the previous action step's execution process, such as the screenshot after the last test operation in a loop test operation. It can be obtained from the execution record of the previous action step (i.e., the test operation record). The test assertion can describe the expected state of the last screenshot, such as "After the action is completed, the flashlight is on and the icon background is yellow".

[0070] When performing an inspection operation according to the multi-image inspection steps, the images to be inspected can be the current screenshot of the device under test and screenshots during the execution of specified historical action steps. The second test model can obtain the screenshots from the execution records of specified historical action steps according to the inspection intent. The screenshots obtained by the second test model can show the screen state before a certain test operation in the corresponding loop test operation of the historical action step. The test assertions can describe the expected changes in cross-step inspection, such as "Step X | Do not display deleted information content" or "Verify whether the items in the list have been successfully deleted".

[0071] When performing the inspection operation according to the standard image inspection steps, the images to be inspected can be a current screenshot of the device under test and a pre-stored standard image. The standard image is a reference image that reflects the expected screen state and can serve as a correctness benchmark. The standard image can be pre-stored in a database. The test assertion can describe the expected consistency relationship between the current screenshot and the standard image, such as "verify whether the page UI layout is consistent with the design draft" or "img.png (an image file name) | screenshot is consistent with the image". The second test model can perform pixel-level comparison and analysis of the current screenshot and the standard image. During the comparison and analysis, dynamic or irrelevant differences such as time, battery level, and signal strength are ignored, focusing on finding substantive interface problems such as layout errors, missing elements, and abnormal colors. The comparison results output by the second test model can include a description of the differences between the current screenshot and the standard image and whether it "passes", based on which the inspection step can be recorded as "successful" or "failed".

[0072] This application embodiment introduces the aforementioned various types of inspection steps, supporting their arbitrary combination and flexible invocation, thus constructing a multi-layered, multi-dimensional automated verification system. This significantly enhances the testing system's coverage of complex application scenarios. Immediate inspection steps can be triggered immediately after an action step is completed to quickly verify short-term response states and improve the real-time performance of testing. Action inspection steps can be combined with prior interactive behaviors for context-aware verification to ensure the continuity of functional flows. Multi-image inspection steps support cross-step and cross-interface logical judgments. Standard image inspection steps can use standard images as a benchmark to achieve UI consistency verification, effectively supporting front-end quality control.

[0073] In an optional implementation, the application testing method provided in this application embodiment may further include: obtaining historical test cases written in natural language and preset system prompt words, parsing and formatting the historical test cases and system prompt words through a large text model to obtain test cases in the standardized format used in step S401.

[0074] In some application scenarios, the business side has many historical test cases written for manual testing scenarios. These historical test cases are usually in Excel format. When used directly as test cases for the testing methods provided in this application's embodiments, problems such as inconsistent formatting, abbreviated steps, and lack of project consensus context easily arise. To more effectively utilize these historical test assets, this application processes historical test cases written in natural language using a large text model in the above embodiments. The large text model can convert the format of historical test cases into standardized, executable test cases compatible with this application's testing system based on system prompts. This solution greatly reduces the workload of manual rewriting and debugging, enables rapid reuse of customers' historical test assets, and improves the import efficiency of the testing system.

[0075] An example of a system suggestion word is as follows: Figure 6 As shown, it can include input instructions, output instructions, user input instructions, and other instructions required for format conversion.

[0076] When obtaining information about historical test cases, you can first obtain the original data source (such as spreadsheets, documents, etc.), and then obtain the information of individual historical test cases from the original data source.

[0077] Before inputting historical test cases and system prompts into the large text model, the column names of the historical test cases in the Excel file can be modified according to a conversion template with fixed column names to match the fixed column names in the conversion template. Then, the large text model is called to rewrite and format the historical test cases with modified column names, resulting in standardized test cases. The modification of column names in historical test cases can be done by testers before uploading to the testing system, or the historical test cases can be uploaded to the testing system and automatically modified by the system.

[0078] The fixed column names mentioned above can be set according to actual needs. For example, each row can be named Precondition (or Pretreatment Condition), Test Steps, and Expected Result. Correspondingly, the modified historical test cases can include three parts of information: preconditions, test steps, and expected results. Preconditions define the environment, state, and data required to execute the test. They describe the prerequisites that the test system must meet before the test can begin so that it can proceed smoothly. For example, preconditions could be "the user must be logged in" or "specific data must be on a specific page." Preconditions ensure that the starting point of the test is clear and reproducible. Test steps describe in detail and clearly a series of specific operations that need to be followed when executing the test. Each step is an independent, executable action. This is the core part of the test case, guiding testers (or automated scripts) to interact with the application under test to verify its functionality. For example, test steps could be "click the device button," "enter 'abc' in the input box," or "click the save button." The expected outcome describes the correct behavior or state the system should exhibit after all "test steps" are completed. It provides a clear criterion for judging the success or failure of the test. Testers compare the actual performance of the application under test with this expected outcome to determine whether the test passed. For example, the expected outcome could be "the system indicates successful saving," "the page redirects to the profile page," or "the value of a field in the database is updated." In some application scenarios, the fixed column name can also be a column name for other content.

[0079] Before inputting historical test cases and system prompts into the large text model, you can merge the historical test cases and system prompts, and then use the merged information as the final input content into the large text model.

[0080] In an optional implementation, the application testing method provided in this application embodiment may further include preprocessing historical test cases. This preprocessing may include: extracting multiple text fragments from historical test cases and concatenating these fragments into a single complete text message. Correspondingly, parsing and formatting historical test cases and system prompts using a large text model may include: parsing and formatting the preprocessed historical test cases and system prompts using a large text model.

[0081] The scope of text fragment extraction can be set according to actual needs. For example, it can be extracted from the modified column names of historical test cases, with the text information corresponding to each column name being considered as a text fragment. For instance, as shown... Figure 7As shown, in a row of an Excel spreadsheet containing historical test cases, there are three cells: cell A1, cell A2, and cell A3. Cell A1 corresponds to the column name "PretreatmentCondition," cell A2 corresponds to the column name "TestSteps," and cell A3 corresponds to the column name "ExpectedResult." When extracting text fragments, the text content in these three cells can be precisely extracted from the Excel spreadsheet as three independent text fragments. These three independent text fragments can then be concatenated into a larger string according to a fixed format, forming a unified, unprocessed text block, which serves as the complete text information mentioned above.

[0082] against Figure 7 In the table shown, in another example, the extraction and concatenation of text fragments can be synchronously achieved based on a unified transformation template, which can be... Figure 8 The format shown on the left includes a title (e.g.) Figure 8 Plain Text in the middle), fixed column names (such as Figure 8 The left side displays fixed text content such as Precondition, Test Steps, and Expected Result. Placeholders can also be used to reserve space for dynamic content that needs to be filled in. When extracting and concatenating text fragments, the content of each column name in the historical test cases (after modifying column names) can be filled into the placeholders corresponding to the column names in the conversion template. For example, the content of cell A1 in the historical test cases can be filled into... Figure 8 Fill in the placeholder corresponding to the Precondition on the left with the content of Cell A2 from the historical test cases. Figure 8 Fill in the placeholder for Test Steps on the left with the content of Cell A3 from the historical test cases. Figure 8 After filling in the relevant content in the placeholder corresponding to Expected Result on the left, the data that was originally scattered in three cells is combined into one, as shown below. Figure 8 The large, coherent, easy-to-read, and easy-to-process text block shown on the right represents the simultaneous extraction and splicing of three text fragments.

[0083] This preprocessing method of extracting and splicing text fragments helps eliminate the problems of semantic dispersion and inconsistent format in the original test cases, and improves the coherence and completeness of the input content. On this basis, by using a large text model to parse and convert the preprocessed historical test cases and system prompts, the contextual semantics can be understood more accurately, the key elements of the test scenario can be effectively identified, and they can be uniformly converted into standardized format test cases to improve the automation and accuracy of test case generation.

[0084] In an alternative implementation, preprocessing may further include adding identification information to each text segment within the complete text information. The identification information may be column names from historical test cases or other information.

[0085] For example in Figure 7 In the example, Precondition, Test Steps, and Expected Result were added as identifiers for cell A1, cell A2, and cell A3 respectively, so that the large text model could better understand the meaning of each part.

[0086] like Figure 9 As shown, when merging historical test cases and system prompts, the system prompts and preprocessed historical test cases can be combined to obtain the final input content. The final input content includes operation instructions and data that needs to be processed. The text model will deeply understand, analyze and transform the data to be processed based on the operation instructions in the final input content, and output text composed of standardized test cases. This facilitates the computer to perform subsequent structured mapping and parsing to obtain data with clear classification labels (such as Setup, TestStep, and Teardown labels) for subsequent testing.

[0087] For semi-structured text, this application embodiment can parse it and convert it into a fully structured data object, which is ultimately used for database storage. The execution flow is as follows: Figure 10 and Figure 11As shown, it includes data reception, line splitting, line-by-line parsing, data extraction, and structured mapping. First, during data reception, the data structure output by the large text model is received. Each key corresponds to a string containing multiple lines of text. Then, during line segmentation, each value in the received data structure is traversed, using a newline character (\n) as a delimiter to split the large string into an array (list) of single-line strings. During line-by-line parsing, this array is traversed, and a predefined regular expression rule is applied to each line of text (item). During data extraction, the regular expression identifies and extracts multiple target data fragments from each line of text. For example, for the text "1. action[default](1,2,3): xxxxxxx", the regular expression will extract four target data fragments: "1、action", "default", "1,2,3", and "xxxxxxx". During structured mapping, the extracted target data fragments can be assigned to specific fragments of a new data object. For example, data fragment "1" becomes the value of "idx (index)", and data fragment "action (action)" becomes the value of "idx (index)". This becomes the value of "type (step type)", and so on, forming an object similar to JSON (JavaScript Object Notation).

[0088] In the above parsing and transformation process, in order to match strings like "1. action[default](1,2,3):xxxxxxx", a typical regular expression is as follows:

[0089] Plain Text

[0090] / ^(\d+)\.\s*(\w+)\[(\w+)\]\((.*?)\):\s*(.*)$ / ”

[0091]

[0092] Referring to the table above, from the beginning (^) to the end ($) of a line, sequentially search for and capture one or more numbers, a dot, any space, a word, a word within square brackets, any content within parentheses, a colon, any space, and all remaining content up to the end of the line. Using these rules and capture groups (), the program can accurately break down a line of text into multiple meaningful data segments.

[0093] After the preprocessing in this embodiment is completed, the preprocessed test cases can be manually verified. The manual verification method can be, for example, to first automatically execute the test cases, which can be executed only once without looping. After automatic execution, it can be confirmed whether the expected effect has been achieved. If the expected effect has not been achieved, the tester can modify the test cases based on the model capability guideline of the corresponding test model.

[0094] In an optional implementation, the application testing method provided in this application embodiment may further include: detecting whether an error has occurred during the execution of a test operation; determining the step segment to which the error belongs in response to the occurrence of an error in the test operation; and processing the error accordingly based on the step segment to which the error belongs.

[0095] The steps in a test case can be divided into different step segments, such as the setup / preparation phase, the test step phase, and the cleanup / recovery phase, as shown in the reference. Figure 2 For example, during the execution of the test operations corresponding to each step in the test case, the test system can continuously detect whether errors occur in the test operations. When an error is detected in a certain test operation, it can check which step segment the step corresponding to the erroneous test operation belongs to, that is, check which step segment the error occurs in. If the error occurs in the setup / preparation phase or the cleanup / recovery phase, it can be determined that the current test case cannot continue, and all steps in the current test case that have not yet been executed can be skipped directly. If the error occurs in the test step phase, the test can be stopped immediately, and the steps in the cleanup / recovery phase can be started directly to ensure that the device environment is restored. If no error is detected, the steps can continue to be executed in the predetermined order of each step in the test case until all steps in the current test case or all steps in the current test plan are completed.

[0096] Based on the above approach, errors in test operations are detected in real time during test execution, and differentiated processing is performed according to the step segment where the error occurred. This can effectively improve the stability and reliability of test execution, while also enhancing overall test efficiency and execution security.

[0097] In an optional implementation, the application testing method provided in this application embodiment may further include: sending a device configuration instruction to the device under test before performing the test operation to trigger the device under test to configure the test environment.

[0098] By configuring the test environment for the equipment under test, it can be ensured that the test begins in a unified and controllable environment, which effectively improves the consistency and reliability of the test environment, as well as the accuracy and repeatability of the test process.

[0099] The configuration of the test environment for the device under test may include, for example, installing and enabling an input engine on the device under test, and / or restoring the device under test to the home screen interface. The input engine can be configured to input text into the application on the device under test and display an input interface on the test device, eliminating the need for the input interface to be displayed on the device under test. This avoids the input interface obscuring the display content of the device under test, ensuring the accuracy and visibility of the testing process. It can also be configured to prevent garbled characters when the input content is displayed on the device under test. The home screen interface can serve as a known and defined initial interface to ensure the uniformity and reproducibility of the test environment.

[0100] The visual language big model in this application embodiment can be any one or more types of visual language big models, and the text big model can be any one or more types of text big models. For different types of visual language big models and text big models, the corresponding prompt words, parsing logic, etc. can be replaced or adjusted according to the specific characteristics of the model.

[0101] In this embodiment of the application, the test method provided in this embodiment can be used to test the application on a certain device under test by running the log.

[0102] Although the steps of the methods in the embodiments of this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0103] Based on the same technical concept, embodiments of this application also provide an application testing device, such as... Figure 12 As shown, the testing device 1200 may include: a test case acquisition module 1201 and a test case execution module 1202.

[0104] The test case acquisition module 1201 is used to: acquire test cases for testing the application in the device under test. The test cases include action steps. The format of the test cases is a standardized format adapted to the first test model. The first test model is a large visual language model with graphical user interface visual understanding capabilities.

[0105] The test case execution module 1202 is used to: execute corresponding test operations according to the action steps. The test operations include: obtaining a first test operation instruction in the current action step and a current screenshot of the device under test; jointly parsing the first test operation instruction and the current screenshot using a first test model; generating a second test operation instruction based on the parsing result; and sending the second test operation instruction to the device under test to drive the device under test to perform corresponding interactive operations on the application according to the second test operation instruction.

[0106] In an optional implementation, when performing the corresponding test operation according to the action step, the test case execution module 1202 can be used to: repeatedly execute the corresponding test operation according to the current action step until the current action step is successfully executed or the number of times the test operation is repeated exceeds the preset number of times.

[0107] In an optional implementation, when performing the corresponding test operation according to the action steps, the test case execution module 1202 can also be used to: perform diagnostic analysis on the historical operation record of the current action step through the first test model during the Nth execution of the test operation; N is an integer greater than 1, and the parsing result of the first test model includes the result of the diagnostic analysis.

[0108] In an optional implementation, the test case further includes an inspection step. Correspondingly, the test case execution module 1202 can also be used to: perform corresponding inspection operations according to the inspection steps. The inspection operations include: obtaining the test assertion in the inspection steps and a screenshot of the device under test in a specified state; determining whether the screen state corresponding to the screenshot conforms to the expected state described by the test assertion using a second test model; and generating a judgment result. The second test model can be a large visual language model, such as a general visual language model or a model instance identical to the first test model.

[0109] In one alternative implementation, the inspection steps may include at least one type of steps selected from immediate inspection steps, action inspection steps, multi-image inspection steps, and standard image inspection steps.

[0110] In an optional implementation, the testing device 1200 may further include a historical test case processing module, used to: acquire historical test cases written in natural language and preset system prompts, and parse and convert the historical test cases and system prompts using a large text model to obtain standardized test cases.

[0111] In an optional implementation, the historical test case processing module described above can also be used to: preprocess historical test cases, and parse and convert the preprocessed historical test cases and system prompts using a large text model. Preprocessing includes: extracting multiple text fragments from the historical test cases and concatenating these fragments into a single complete text message.

[0112] In one alternative implementation, the preprocessing further includes adding identification information to each text segment within the complete text information.

[0113] In an optional embodiment, the testing apparatus 1200 may further include an error detection module, used to detect whether an error has occurred during the execution of the testing operation; in response to an error occurring in the testing operation, to determine the step segment to which the error belongs; and to process the error accordingly based on the step segment to which the error belongs.

[0114] In an optional embodiment, the testing apparatus 1200 may further include a device configuration module, which is used to send a device configuration instruction to the device under test before performing the test operation, so as to trigger the device under test to configure the test environment.

[0115] The functions of each module in the device of this application embodiment can be referred to the corresponding descriptions in the above method embodiments, and will not be repeated here.

[0116] Based on the same technical concept, embodiments of this application also provide an electronic device, which can be manifested as a general-purpose computing device. This electronic device may include a memory and a processor. The memory stores a computer program, which is loaded and executed by the processor to implement any of the testing methods provided in the embodiments of this application. The memory and processor may contain one or more data.

[0117] The aforementioned memory may include at least one of non-volatile memory and volatile memory. Non-volatile memory may include at least one of the following: read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, etc. Volatile memory may include random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), sync link dynamic random access memory (SLDRAM), direct memory bus RAM (DR RAM), etc.

[0118] The aforementioned memory may also include programs / utilities having a set (at least one) of program modules, including but not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The aforementioned memory may also be referred to as a storage medium or storage device, and the embodiments of this application do not impose any limitations on this.

[0119] The processor mentioned above can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor mentioned above can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0120] Optionally, if the memory and processor are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be at least one of the following: Industry Standard Architecture (ISA) bus, Peripheral Component Interconnect (PCI) bus, Extended Industry Standard Architecture (EISA) bus, etc. The bus can represent one or more of several bus architectures, including a memory bus or memory controller, peripheral bus, graphics acceleration port, processor, or a local bus using any of the various bus architectures.

[0121] Optionally, if the memory and processor are integrated on a single chip, they can communicate with each other through an internal interface.

[0122] The aforementioned electronic device can communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), and also with one or more devices that enable users to interact with the electronic device, and / or with any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed via input / output (I / O) interfaces. Furthermore, the aforementioned electronic device can also communicate with one or more networks via a network adapter, such as a Local Area Network (LAN), a Wide Area Network (WAN), or a public network (e.g., the Internet). The network adapter communicates with other modules of the electronic device via a bus. It should be understood that other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) systems, tape drives, and data backup storage systems.

[0123] The electronic device according to this embodiment of the present application is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present application.

[0124] Based on the same technical concept, embodiments of this application also provide a computer-readable storage medium storing program code, which, when executed by a processor, implements any of the test methods provided in embodiments of this application.

[0125] In exemplary embodiments of this application, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible implementations, various aspects of this application may also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of this application described in the "Exemplary Methods" section above.

[0126] The above-described program product can be produced using any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium.

[0127] A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a readable storage medium (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0128] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0129] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0130] Through the description of the above embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods provided according to the embodiments of this application.

[0131] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0132] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0133] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this application, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0134] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0135] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for testing an application, characterized in that, include: Obtain test cases for testing the application on the device under test; The test cases include action steps, and the format of the test cases is a standardized format adapted to the first test model; Perform the corresponding test operations according to the described action steps; The test operation includes: obtaining a first test operation instruction in the current action step and a current screenshot of the device under test; performing joint analysis on the first test operation instruction and the current screenshot through the first test model; generating a second test operation instruction based on the analysis result; and sending the second test operation instruction to the device under test to drive the device under test to perform corresponding interactive operations on the application according to the second test operation instruction. The first test model is a large visual language model with the ability to understand the visual interface of a graphical user interface.

2. The method according to claim 1, characterized in that, The step of performing the corresponding test operation according to the action steps includes: The corresponding test operation is executed repeatedly according to the current action step until the current action step is successfully executed or the number of times the test operation is repeated exceeds the preset limit of the number of times.

3. The method according to claim 2, characterized in that, The step of performing the corresponding test operation according to the action steps further includes: during the Nth execution of the test operation, performing a diagnostic analysis on the historical operation record of the current action step through the first test model; N is an integer greater than 1. The analysis results include the results of the diagnostic analysis.

4. The method according to claim 1, characterized in that, The test cases also include inspection steps; The method further includes performing corresponding inspection operations according to the inspection steps; The inspection operation includes: acquiring the test assertion in the inspection step and a screenshot of the device under test in a specified state; determining whether the screen state corresponding to the screenshot conforms to the expected state described by the test assertion through a second test model; and generating a judgment result; the second test model is a visual language large model.

5. The method according to any one of claims 1-4, characterized in that, Also includes: Retrieve historical test cases written in natural language and preset system prompts; The historical test cases and system prompts are parsed and formatted using a large text model to obtain the standardized test cases.

6. The method according to claim 5, characterized in that, Also includes: The historical test cases are preprocessed; The preprocessing includes: extracting multiple text fragments from the historical test cases and concatenating the multiple text fragments into a complete text information; The historical test cases and system prompts are parsed and their formats converted using the large text model, including: The preprocessed historical test cases and system prompts are parsed and their formats converted using the large text model.

7. The method according to any one of claims 1-4, characterized in that, Also includes: During the execution of the test operation, it is detected whether an error has occurred. In response to an error occurring during the test operation, determine the step segment to which the error belongs; The error is processed accordingly based on the step section to which it belongs.

8. The method according to any one of claims 1-4, characterized in that, Also includes: Before performing the test operation, a device configuration command is sent to the device under test to trigger the device under test to configure the test environment.

9. A testing apparatus for an application, characterized in that, include: The test case acquisition module is used to acquire test cases for testing the application in the device under test; The test cases include action steps, and the format of the test cases is a standardized format adapted to the first test model; The test case execution module is used to perform corresponding test operations according to the action steps; The test operation includes: obtaining a first test operation instruction in the current action step and a current screenshot of the device under test; performing joint analysis on the first test operation instruction and the current screenshot through the first test model; generating a second test operation instruction based on the analysis result; and sending the second test operation instruction to the device under test to drive the device under test to perform corresponding interactive operations on the application according to the second test operation instruction. The first test model is a large visual language model with the ability to understand the visual interface of a graphical user interface.

10. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, which is loaded and executed by the processor to implement the method as described in any one of claims 1-8.

Images