Automatic testing method and device, computer program product and computer readable storage medium

Abstract

The invention provides an automatic testing method and device, a computer program product and a computer readable storage medium. Comprising the steps of determining page information of at least one page and control information contained in each page based on a source code of a to-be-tested application; if the to-be-tested application is tested for the first time, generating a test operation sequence based on the historical operation data; for each operation step in the test operation sequence, determining a target position of a to-be-operated control associated with the operation step in a current page corresponding to the operation step by adopting a plurality of large language models, and executing an operation corresponding to the operation step based on the target position to obtain an operation result; if the operation result represents that the operation is successful, storing an identifier of a large language model adopted in the operation step, a target position and an operation parameter used for executing the operation, and forming reusable structured data; and if the to-be-tested application is not tested for the first time, obtaining a test operation sequence, and executing each operation step in the test operation sequence based on the structured data.

Description

Technical Field

[0001] This application relates to automated testing technology, and more particularly to an automated testing method, equipment, computer program product, and computer-readable storage medium. Background Technology

[0002] Currently, AI-based automated testing technology for user interfaces (UI) still faces significant challenges in practice. Existing solutions typically rely on AI models trained using general knowledge, making it difficult to accurately plan personalized operation paths for specific applications (Apps), leading to difficulties in building test processes. At the execution level, existing technologies often cannot automatically and reliably verify the success of each operation, and the control recognition and positioning capabilities of individual AI models are limited, easily leading to operation failures due to recognition errors. Furthermore, existing technologies lack a mechanism for solidifying successful operation experiences. Each time a test is executed, even when facing the same controls and paths, model recognition and attempts must be repeated, resulting in significant consumption of computing resources and time, leading to low overall testing efficiency and insufficient stability. Therefore, there is an urgent need for a technical solution that can improve the accuracy of path planning and control positioning, and can solidify and reuse successful practical experiences to achieve efficient and stable automated testing. Summary of the Invention

[0003] This application provides an automated testing method, device, computer program product, and computer-readable storage medium, which not only improves the accuracy of path planning, component location, and execution success rate of UI automated testing, but also fundamentally solves the problem that successful paths cannot be solidified and reused, resulting in a significant improvement in test execution efficiency and stability.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides an automated testing method, the method comprising: Based on the source code of the application under test, determine the page information of at least one page and the control information contained in each page; If the application under test is being tested for the first time, a test operation sequence is generated based on historical operation data; wherein, the historical operation data contains a sequence of operation steps mapped from the user's operation intention, and the operation steps in the test operation sequence are associated with controls corresponding to the control information; For each operation step in the test operation sequence, multiple large language models are used to determine the target position of the control to be operated associated with the operation step in the current page corresponding to the operation step, and the operation corresponding to the operation step is executed based on the target position to obtain the operation result; If the operation result indicates that the operation was successful, the identifier of the large language model used in the operation steps, the target location, and the operation parameters used to perform the operation are stored to form reusable structured data; If this is not the first time the application under test is being tested, obtain the test operation sequence and execute each operation step in the test operation sequence based on the structured data.

[0005] In the above scheme, determining the page information of at least one page and the control information contained in each page based on the source code of the application under test includes: The source code is parsed using a large language model to obtain page information and control information for at least one page; wherein, the page information includes at least a page identifier, a page name, and a page type; and the control information includes at least a control identifier, a control name, and a control type.

[0006] In the above scheme, generating a test operation sequence based on historical operation data includes: The historical operation data is parsed to obtain the user's operation intention and the corresponding operation step sequence; For each operation step in the sequence of operation steps, a target control is matched from the control information based on the control description in the operation step. The operation steps are associated with the target control to generate the test operation sequence.

[0007] In the above scheme, the step of determining the target position of the control to be operated associated with the operation step in the current page corresponding to the operation step using multiple large language models for each operation step in the test operation sequence includes: Obtain at least one screen image of the current page; The at least one screen image is processed using the multiple large language models to obtain the candidate position of the control to be operated in the current page corresponding to the operation step, output by each large language model. The target location is determined from a plurality of candidate locations.

[0008] In the above scheme, the step of processing the at least one screen image using the multiple large language models to obtain the candidate position of the control to be operated on in the current page corresponding to the operation step, output by each large language model, includes: The display area corresponding to each screen image is divided into multiple sub-regions; wherein, the multiple sub-regions include upper left, upper center, upper right, middle left, middle, middle right, lower left, lower center, and lower right. For each of the large language models, the target screen image where the control to be operated is located is determined based on the large language model, and the target sub-region where the control to be operated is located in the target screen image is determined; wherein, the plurality of sub-regions includes the target sub-region; The location information consisting of the target screen image and the target sub-region is determined as the candidate location.

[0009] In the above scheme, storing the identifier of the large language model used in the operation steps, the target location, and the target operation parameters to form reusable structured data includes: The identifier, the target location, and the operation parameters are associated and recorded; The data associated with the records is then stored in a fixed format to form the structured data.

[0010] In the above scheme, executing each operation step in the test operation sequence based on the structured data includes: For each operation step in the test operation sequence, the identifier, target location, and operation parameters of the large language model corresponding to the operation step are obtained from the structured data; Based on the identifier of the large language model, the target location, and the operation parameters, the target control associated with the operation step is driven to execute the operation step.

[0011] This application provides an automated testing device, the device comprising: The acquisition unit is used to determine the page information of at least one page and the control information contained in each page based on the source code of the application under test. The generation unit is used to generate a test operation sequence based on historical operation data if the application under test is being tested for the first time; wherein the historical operation data includes a sequence of operation steps mapped from user operation intentions, and the operation steps in the test operation sequence are associated with controls corresponding to the control information; The processing unit is used to determine the target position of the control to be operated associated with the operation step in the current page corresponding to the operation step for each operation step in the test operation sequence using multiple large language models, and to perform the operation corresponding to the operation step based on the target position to obtain the operation result. A storage unit is used to store the identifier of the large language model used in the operation steps, the target location, and the operation parameters used to perform the operation if the operation result indicates that the operation is successful, forming reusable structured data; The processing unit is further configured to, if not testing the application under test for the first time, acquire the test operation sequence and execute each operation step in the test operation sequence based on the structured data.

[0012] This application embodiment provides an automated testing device, the device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method provided in the embodiments of this application.

[0013] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the method provided in this application when executed by a processor.

[0014] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implement the method provided in this application.

[0015] The embodiments of this application have the following beneficial effects: During the first execution, the target position of the control is determined by using multiple large language models in parallel processing and voting mechanism, and the correctness of the step execution is ensured by verifying the operation results. This significantly improves the accuracy and success rate of a single operation step. Furthermore, the identifier, target position and operation parameters of the large language model on which the operation is successful are associated, recorded and solidified into structured data, forming reusable test assets. This allows these solidified data to be directly called to drive the test in subsequent non-first executions, completely avoiding the repeated AI recognition and trial process. As a result, the overall UI automation test has been improved in terms of path planning accuracy, component positioning and execution success rate, and the problem of successful paths not being solidified and reused has been fundamentally solved, resulting in a significant improvement in test execution efficiency and stability. Attached Figure Description

[0016] Figure 1 This is a first flowchart illustrating the automated testing method provided in this application embodiment; Figure 2 This is a schematic diagram of the second process of the automated testing method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the third process of the automated testing method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the fourth process of the automated testing method provided in the embodiments of this application; Figure 5 This is a schematic diagram of the fifth process of the automated testing method provided in the embodiments of this application; Figure 6 This is a schematic diagram of the sixth process of the automated testing method provided in the embodiments of this application; Figure 7 This is a schematic diagram of the seventh process of the automated testing method provided in the embodiments of this application; Figure 8 This is a schematic diagram of region division in the automated testing method provided in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of the automated testing device provided in the embodiments of this application; Figure 10 This is a schematic diagram of the structure of the automated testing equipment provided in the embodiments of this application.

[0017] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] It should be noted that the functionality of each app is relatively personalized. Currently, UI automation testing for apps typically utilizes AI agents trained with industry-standard knowledge (such as tools like UI TARS and AUITestAgent), which has the following problems: 1. The accuracy of planning operation paths for a specific app is not high; 2. It cannot automatically verify whether a single step is executed successfully; 3. The ability of a single AI to identify and locate controls is limited; 4. App software testing requires multiple repetitive operations, and existing AI technologies require multiple attempts for each execution, with successful but unsuccessful practices not being solidified for reuse in the next iteration.

[0020] Based on this, embodiments of this application provide an automated testing method, referring to... Figure 1 and Figure 2 As shown, this method, applied to automated testing equipment, can specifically include the following steps: Step 101: Based on the source code of the application under test, determine the page information of at least one page and the control information contained in each page.

[0021] In this embodiment, the application under test refers to a software application that requires UI automation testing, such as an App or desktop software; the source code refers to the original code of the application under test, which typically includes structural information such as UI layout, control definitions, and event handling; page information refers to the descriptive information of each interface in the application under test; and control information refers to interactive or visible elements on the page, such as buttons, input boxes, and text labels. An artificial intelligence model can be used to parse the source code of the application under test, extract the structured information of all pages (specifically including page identifier, page name, and page type), and store all page information in a page table. Simultaneously, all control information on each page (specifically including control identifier, control name, control type, and the page to which the control belongs) can be extracted, and the control information on each page can be stored in a control table.

[0022] Step 102: If this is the first time testing the application under test, generate a test operation sequence based on historical operation data.

[0023] The historical operation data contains a sequence of operation steps mapped from the user's operation intentions, and the operation steps in the test operation sequence are associated with controls corresponding to the control information.

[0024] In this embodiment, historical operation data refers to the records of operations performed by users or testers in the application under test, typically existing in the form of "operation intent -- operation steps," such as customer service Q&A pairs, user behavior logs, etc.; user operation intent refers to the description of the task the user wishes to complete, such as checking stock holdings; operation step sequence refers to a series of specific interface operations required to achieve a certain operation intent, such as clicking "My" -- clicking "Account" -- clicking "Stocks"; test operation sequence refers to a fully structured task list that can be directly executed by the automated testing engine, and each step in the sequence not only includes operation actions and result verification, but also explicitly associates with information such as the identifier, type, and expected position of the target control. By collecting and parsing historical operation data and transforming it into a structured operation flow table, each operation step is associated with the previously extracted control information to form an executable test operation sequence. In this way, by using real user operation history to generate test paths, the accuracy and practicality of path planning are improved. At the same time, by transforming the operation intent described in natural language into an executable control operation sequence, the automated parsing and conversion of test requirements are realized.

[0025] Step 103: For each operation step in the test operation sequence, use multiple large language models to determine the target position of the control to be operated associated with the operation step in the current page corresponding to the operation step, and execute the operation corresponding to the operation step based on the target position to obtain the operation result.

[0026] In this embodiment, the large language model refers to the AI ​​model used to identify the position of controls; the target position refers to the specific position of the control on the current page screen, usually represented by a screen area (e.g., which screen, which area); the operation result refers to the result returned after the operation is performed, used to verify whether the operation was successful. Upon initial execution, for each operation step, multiple screenshots of the current page are taken first, then multiple AI models are used to identify the position of the control on the screen, and a voting mechanism is used to determine the final position. Then, an operation (e.g., clicking, input) is performed based on the determined position, and the operation result is verified to ensure it matches the expectation. This multi-model voting mechanism improves the accuracy and robustness of control positioning, reduces the risk of a single model failing to identify the control, and ensures the correct execution of each step by verifying the operation result in real time, thus improving the reliability of the test.

[0027] Step 104: If the operation result indicates that the operation was successful, store the identifier of the large language model used in the operation steps, the target location, and the operation parameters used to perform the operation, forming reusable structured data.

[0028] In this embodiment, "operation success" means that the operation result is consistent with expectations, such as correct page redirection or control status meeting expectations; "identifier" refers to the number or type of the large language model, such as AI1, AI3, etc.; "operation parameters" refers to the parameters required to perform the operation, such as click coordinates, input text, waiting time, etc.; "structured data" refers to the operation records stored in tabular form, which facilitates subsequent querying and reuse. When a step is successfully executed, the AI ​​model number, control position, operation parameters, and other information used in this execution can be recorded in the step table to form reusable test data. In this way, the successful operation path can be solidified, avoiding repeated attempts, improving the efficiency of subsequent testing, and forming a knowledge accumulation mechanism. The automated testing equipment becomes more accurate with use, gradually reducing the dependence on AI models.

[0029] Step 105: If this is not the first time testing the application under test, obtain the test operation sequence and execute each operation step in the test operation sequence based on the structured data.

[0030] In this embodiment of the application, "non-first execution" means that the test operation sequence of the application under test has a successful execution record, and historical data can be directly reused. When executing a test operation sequence that has been successfully executed, the identifier, target position, operation parameters, etc. of the previously recorded large language model are directly read from the step table, and the target control is driven to perform the operation. There is no need to perform control positioning and retrying again. This can not only greatly improve the test execution speed and avoid repeated attempts and resource waste, but also improve test stability and reduce execution failures caused by AI recognition fluctuations.

[0031] The automated testing method provided in this application significantly improves the accuracy and success rate of individual operation steps by employing parallel processing and voting mechanisms of multiple large language models to determine the target position of controls during the first execution and verifying the operation results to ensure the correct execution of steps. Furthermore, the identifiers, target positions, and operation parameters of the large language models on which the operation is successful are associated, recorded, and stored as structured data, forming reusable test assets. This allows these solidified data to be directly called to drive testing in subsequent non-first executions, completely avoiding repeated AI identification and trial processes. As a result, the overall UI automated testing achieves improvements in path planning accuracy, component positioning, and execution success rate, and fundamentally solves the problem of unreusable successful paths, resulting in a significant improvement in test execution efficiency and stability.

[0032] Based on the foregoing embodiments, this application provides yet another automated testing method, referring to... Figure 3 , Figure 4 and Figure 5 As shown, this method, applied to automated testing equipment, can specifically include the following steps: Step 201: Use a large language model to parse the source code and obtain page information and control information for at least one page.

[0033] The page information includes at least the page identifier, page name, and page type; the control information includes at least the control identifier, control name, and control type.

[0034] In the embodiments of this application, such as Figure 6 As shown, the complete source code of the application under test is taken as input and provided to a specially trained or code-understanding Large Language Model (LLM). The LLM then analyzes the UI framework-specific syntax, class definitions, layout file references, etc. in the source code to automatically identify all pages and all controls within each page, and outputs them in a structured manner. That is, the LLM transforms the identified unstructured code information into structured data in a predefined format. Specifically, each page and its attributes (i.e., page identifier (specifically, page number), page name, and page type) are filled into the page table shown in Table 1 below, and each control and its key attributes (i.e., control identifier (specifically, control number), control name, control type, and page identifier) ​​are filled into the control table shown in Table 2 below. This completely replaces the traditional method of manually writing or semi-automatically collecting page and control information through screen recording tools in UI automated testing, greatly reducing the cost of building and maintaining test scripts and improving the degree of automation in the construction process.

[0035] Table 1

[0036] Table 2

[0037] Step 202: If this is the first time testing the application under test, analyze the historical operation data to obtain the user's operation intent and the corresponding sequence of operation steps.

[0038] Historical operation data includes a sequence of operation steps mapped from the user's operation intentions.

[0039] In the embodiments of this application, such as Figure 7 As shown, LLM can be used to parse historical operation data and extract user operation intents from it as the targets of test cases. The complete description of achieving the intent is decomposed into a linear, independent sequence of operation steps. Then, the parsed user operation intent-operation step sequence pairs are stored in a structured operation flow table. This not only realizes the automatic conversion from business language to test language and transforms scattered, non-standard historical records into standardized test requirements that can be processed by the system, but also uses real user operation history to generate test paths, which greatly improves the business relevance and planning accuracy of the paths (because the test simulates the real paths that users most often take).

[0040] In one feasible implementation, historical operation data can include all historical question-and-answer pairs, i.e., user operation intent -- operation step sequence, as shown in Table 3 below. When the user operation intent is: to view my stock holdings, the corresponding operation step sequence (operation step - operation -- result) can include: click My [Result: enter My page] -- click Account [Result: enter account information page] -- click Stock [Result: enter stock holdings page].

[0041] Table 3

[0042] Step 203: For each operation step in the sequence of operation steps, match the target control from the control information based on the control description in the operation step.

[0043] In this embodiment, the control description refers to the textual description of the interface element to be operated in the sequence of operation steps, such as "My," "Account," "Stocks," etc.; the target control refers to the specific, structured control record (including control identifier, name, type, etc.) that best corresponds to the control description, found by a matching algorithm from the control information library (i.e., control table) determined in step 201. For each step in the sequence of operation steps, the following operations are performed: first, the control description representing the control is extracted from the operation step; then, using the accurate "control name" and other fields stored in the control table and obtained from source code parsing, the most matching target control (i.e., its unique control identifier, such as a003b00006) is found for the current control description through text similarity calculation, synonym mapping, and other techniques. This allows the ambiguous natural language instructions to be precisely defined as specific UI elements.

[0044] Step 204: Associate the operation steps with the target control to generate a test operation sequence.

[0045] In this test operation sequence, the operation steps are associated with the controls corresponding to the control information.

[0046] In this embodiment, each operation step in the test operation sequence is associated with the complete information of its corresponding target control. Then, all the associated steps can be combined sequentially to form a complete test operation sequence with rich context. This test operation sequence is the direct basis for subsequent AI execution and path solidification. In this way, an enhanced test script rich in semantic information is generated, which not only knows "where to click", but also "what control (button or text) is clicked" and "what page will be taken after success". This makes test execution and result verification more intelligent and reliable, and provides a complete data object for the subsequent "solidification and reuse" mechanism. It should be noted that after successful execution, the entire structured sequence generated this time, along with the execution parameters, can be completely saved, realizing the solidification of the successful path, which can be directly and stably called for future similar test needs.

[0047] Step 205: Obtain at least one screen image of the current page.

[0048] In this embodiment, the current page refers to the UI page corresponding to the current operation step during the execution of the test operation sequence. Since applications typically support scrolling, split-screen, or swipe switching, a page may contain multiple screen images (i.e., multi-screen content). A screen image refers to a static image obtained by taking a screenshot of the current display interface, usually obtained through an automated testing framework (such as Appium, Selenium, etc.) or the device's own screenshot function. When the test operation sequence is executed for the first time, when an operation step needs to be executed, a visual snapshot (i.e., screen image) of the current page is first obtained through an automated tool. Since a page may exceed the display range of a screen, at least one screen image needs to be obtained. This is usually done by simulating scrolling operations and continuously capturing multiple screen images to cover all possible areas of the page. In this way, not only is the complete capture of the page content ensured, providing comprehensive visual input for subsequent control positioning and avoiding positioning failures caused by controls not being in the current visible area, but it also supports processing long pages or pages with complex layouts, improving the adaptability and robustness of the method.

[0049] Step 206: Process at least one screen image using multiple large language models to obtain the candidate positions of the control to be operated on in the current page corresponding to the operation step, as output by each large language model.

[0050] In this embodiment, multiple large language models refer to multiple different AI models (such as computer vision-based object detection models, multimodal large models, etc.). Each model can independently analyze images and identify the position of controls. These models may have different architectures, training data, or expertise, forming a complementary relationship. The candidate position refers to the position of the control to be operated, output by each large language model after recognizing the same control to be operated. It should be noted that due to the differences between the large language models, the same control to be operated may obtain multiple different candidate positions. The acquired screen images (there may be multiple images) can be simultaneously input into multiple large language models. Then, each large language model independently analyzes the screen image, locates the control to be operated in the screen image based on the semantic description (such as control name, type) or visual features of the control to be operated, and outputs its position information as a candidate position.

[0051] It should be noted that a single model may make mistakes due to factors such as image quality, changes in control style, and occlusion, while parallel recognition by multiple models can reduce the overall error rate and improve the reliability and accuracy of localization.

[0052] It should be noted that step 206 can be achieved in the following way: Step 206A1: Divide the display area corresponding to each screen image into multiple sub-regions.

[0053] The sub-regions include top left, top center, top right, left center, center, right center, bottom left, bottom center, and bottom right.

[0054] In this embodiment, the display area refers to the entire display area corresponding to a screen image, usually represented by a pixel matrix; the sub-region is the area of ​​the entire display area divided into sub-regions. The grid is divided into nine equally or approximately equally divided regions, named top left, top center, top right, left center, center, right center, bottom left, bottom center, and bottom right. To simplify the representation of control positions and facilitate model recognition and subsequent processing, each screen image can be divided into the following regions: Figure 8 The nine sub-regions shown (i.e., the nine-square grid) are a way to discretize the continuous coordinate space into a finite number of regions, which reduces the complexity of position description, thereby reducing the difficulty of AI model recognition tasks and improving recognition speed and consistency.

[0055] Step 206A2: For each large language model, determine the target screen image where the control to be operated is located based on the large language model, and determine the target sub-region where the control to be operated is located in the target screen image.

[0056] Among them, multiple sub-regions include the target sub-region.

[0057] In this embodiment, the target screen image refers to the screen image on which the control to be operated is located among multiple screen images (which may correspond to different scroll positions on the page); the target sub-region refers to the sub-region (such as "bottom left", "middle", etc.) on which the control to be operated is located within the target screen image, as determined by the large language model. For each large language model, it needs to complete two levels of tasks: determining the target screen image, that is, selecting the screen image containing the control to be operated (i.e., the screen where the control is located) from multiple screen images, and determining the target sub-region, that is, determining which sub-region the control to be operated is located within the selected target screen image according to the nine sub-regions. The final output result can be represented as (screen number, region name), for example, (screen number 1, bottom left region) indicates that the control is located in the bottom left region of the first screen image.

[0058] Step 206A3: Determine the location information composed of the target screen image and the target sub-region as candidate locations.

[0059] In this embodiment, the target screen image and target sub-region output by each large language model can be combined to form a complete location information, which serves as a candidate location for the space to be operated on by the model. This not only standardizes the output format of each model, allowing the outputs of different models to be directly compared and aggregated, but also simplifies the storage of location information, making it easier to record in the control table later and to use directly during execution.

[0060] Step 207: Determine the target location from multiple candidate locations.

[0061] In this embodiment, the target position refers to the position information selected from multiple candidate positions through a decision-making mechanism (such as voting) and ultimately used for execution. By collecting candidate positions output by all large language models, a voting mechanism is adopted, that is, counting the number of times each candidate position appears and selecting the position with the most votes as the final target position. If there is a tie, the target position can be determined according to preset rules (e.g., prioritizing models in the foreground or specific areas). In this way, 1. through a democratic voting mechanism, the opinions of multiple models are integrated, reducing the impact of errors in the recognition of a single model and improving the overall accuracy of control positioning; 2. an adaptive model selection is achieved: for different controls and pages, different models may perform differently, and the voting mechanism can automatically select the position that the majority of models consider correct.

[0062] It should be noted that after obtaining the target position of the control to be operated, the target position of the control to be operated can be stored in the control table determined in step 201, resulting in the control table shown in Table 4 below (i.e., the updated control table): Table 4

[0063] Step 208: Perform the corresponding operation based on the target location to obtain the operation result.

[0064] In the embodiments of this application, an LLM can be used to perform the operation corresponding to the operation step to obtain the operation result. If the first LLM operation result verification fails, the process returns to the same step and tries to use the second LLM, the third LLM, and so on, until the operation result indicates that the operation is successful, and then the following steps are executed.

[0065] Step 209: If the operation result indicates that the operation was successful, the identifier, target location, and operation parameters of the large language model will be associated and recorded.

[0066] In this embodiment of the application, the associated record refers to binding the three key information items—the identifier of the large language model, the target location of the space to be operated on, and the operation parameters—together as a data tuple and writing it into the step table of step 202, resulting in the step table shown in Table 5 below (i.e., the updated step table). This binding relationship clarifies "which model (identifier), in which location (target location), and how to operate (operation parameters)" in order to successfully complete this step. After successfully executing a step in the test sequence for the first time, the process data is not immediately discarded. Instead, an "experience solidification" operation is performed. This involves extracting the core decision information (which model was correctly identified) and execution information (where and how the operation was performed) upon which the success depended, and storing them in a structured manner. This is equivalent to taking a data snapshot of the "best practice" for the current step. In this way, 1. Precise accumulation of test knowledge is achieved: instead of simply recording the success of a step, the "recipe" of success—the model, location, and parameters—is precisely recorded, making accurate reproduction possible in the future; 2. Directly usable data units are provided for "solidification and reuse": the recorded associated data constitutes the minimum set of operation instructions that can be directly called in subsequent non-first executions.

[0067] Table 5

[0068] Step 210: Solidify and store the data after association to form structured data.

[0069] In this embodiment, persistent storage refers to the continuous saving of associated record data to non-volatile storage media (such as databases or file systems). The term "persistent" emphasizes the durability and stability of the storage, meaning that this experiential data will be retained long-term and will not be lost due to program restarts or session termination, becoming a long-term reliable asset for the system. Structured data specifically refers to data stored in database tables such as step tables and control tables. This data is organized in rows and columns, with clearly defined fields (such as operation step ID, large language model identifier (i.e., AI number), target location, operation parameters, etc.), facilitating programmatic querying, retrieval, and retrieval. The associated records generated above can be written to persistent storage such as databases according to a predefined data model. For example, inserting a record containing (operation step identifier, large language model number, target location (screen number, screen area), operation parameters, etc.) into the step table, along with the previously stored page table and control table, constitutes a complete and queryable UI automation knowledge base. This achieves the following: 1. Establishing a reusable test asset library: Transforming a one-time successful execution into a permanently reusable structured asset is a core technical means to solve the problem of unfixed and reusable successful paths; 2. Significantly improving the efficiency and stability of subsequent tests: When executing the same test later, there is no need to call multiple AI models for time-consuming identification, voting, and trials. "Best practice" data can be directly read from this knowledge base and executed, which greatly reduces computational resource consumption and execution time, while avoiding execution failures caused by AI recognition fluctuations, making the testing process fast and stable; 3. As tests are continuously executed, this knowledge base will become increasingly rich and complete, and the automated testing equipment will become increasingly "intelligent" and reliable, demonstrating continuous learning capabilities.

[0070] Step 211: If this is not the first time testing the application under test, obtain the test operation sequence, and for each operation step in the test operation sequence, obtain the identifier, target location, and operation parameters of the large language model corresponding to the operation step from the structured data.

[0071] In this embodiment, "non-first execution" refers to determining that the test operation sequence to be executed has a history of successful execution. This determination is typically based on the result of querying structured data (especially a "step table") with the unique identifier of the test operation sequence (such as user operation intent or operation steps). When a test request is received, it is first determined whether the test operation sequence corresponding to the request is "non-first execution". If so, for each operation step in the test operation sequence, a complex AI recognition process is no longer initiated. Instead, it directly functions as a "query engine": based on the unique key of the step (e.g., operation intent + step number), the best recipe recorded during a previous successful execution is quickly retrieved from the local structured knowledge base. In one feasible implementation, for the first step of "viewing stock holdings"—"click My"—the automated testing device will directly query the historical records: Large language model identifier: AI1, target position of the control to be operated: screen 1 - bottom left area, operation parameters: click "My" to enter the "My" page, wait 2 seconds after operation; Large language model identifier: AI3, target position of the control to be operated: screen 3 - middle area, operation parameters: click the "Account" button to enter the "Account" page, if no match is found, pull down to refresh the page and try again; Large language model identifier: AI2, target position of the control to be operated: screen 2 - bottom right area, operation parameters: click "Stock" to enter the stock holdings page.

[0072] It should be noted that: 1. "Short-circuit" optimization for test execution has been implemented: The computationally intensive AI model calls, image processing, and voting decision-making processes in the initial execution are skipped, shortening the execution path to a simple data query, greatly improving execution speed. 2. The accuracy and consistency of execution instructions are ensured: The acquired data consists of historically validated, accurate parameters, avoiding execution failures caused by random fluctuations in AI model recognition results or minor page changes (but with unchanged control positions and logic), significantly improving the stability and reliability of the test.

[0073] Step 212: Based on the identifier, target location, and operation parameters of the large language model, drive the target control associated with the operation step to execute the operation step.

[0074] In this embodiment, "driver" refers to the automated testing framework (such as Appium, Selenium, or a system-defined driver engine) simulating the user's actual operation on the application interface according to a given set of instructions; "target control" refers to the interface element with a unique identifier that has been statically associated with the operation steps through step 203, and whose logical identity is known to the automated testing device (e.g., a003b00006). This step addresses the question of "how to operate it." The structured instruction set obtained in step 210 is converted into native commands that the automated testing framework can understand and execute: 1. Locating: Based on the target location (e.g., the lower left area of ​​screen 1) and combined with screen size information, it is converted into specific screen coordinates or control search logic (e.g., "swipe to screen 1, and then search for a control of type 'text' and name 'my' in the lower left area"); 2. Execution: Combined with the operation parameters, the driver framework executes the specified operation on the located control. For example, execute the "click" operation and then execute the "wait 2 seconds" logic defined in the parameters after clicking; or execute the "enter text 'XXX'" operation; 3. Control logic application: At the same time, apply the enhanced control logic that may be included in the parameters, such as "if no match is found, pull down to refresh the page and retry". At this time, since historical successful positions are used, the probability of matching failure is extremely low, and this logic is more of a guarantee. In this way, 1. It achieves efficient and stable execution of "instant query and use": The entire execution process does not rely on real-time AI analysis, but is entirely based on historical solidified data, with extremely fast execution speed and a success rate close to 100%, perfectly solving the core defects mentioned in the background technology, such as "each AI operation often requires multiple attempts, consuming a lot of time" and "failure to solidify successful operations". 2. It transforms the "intelligence" of AI into traditional "precise" automation: In the first execution, the role of AI is exploration and decision-making; in subsequent executions, the wisdom of AI (identifiers, positions, parameters) is transformed into repeatable and precise instructions. 3. A virtuous cycle of "learning-application" is formed: new knowledge is continuously learned and solidified through steps 208 and 209, while existing knowledge is efficiently reused through steps 210 and 211. As testing continues, the proportion of test scenarios that can be directly reused increases, and the overall efficiency and return on investment of the system will show a continuous optimization trend.

[0075] It should be noted that the specific implementation process of the steps performed by each module in the embodiments of this application can be referred to Figure 1 and Figure 3 The implementation process of the automated testing method provided in the corresponding embodiment will not be described in detail here.

[0076] The automated testing method provided in this application significantly improves the accuracy and success rate of individual operation steps by employing multiple large language models in parallel processing and a voting mechanism to determine the target position of the control during the first execution and verifying the operation results to ensure the correct execution of the steps. Furthermore, the large language model identifiers, target positions, and operation parameters on which the operation is successful are associated, recorded, and stored as structured data, forming reusable test assets. This allows these solidified data to be directly called to drive testing in subsequent non-first executions, completely avoiding repeated AI identification and trial processes. As a result, the overall UI automated testing achieves improvements in path planning accuracy, component positioning, and execution success rate, and fundamentally solves the problem that successful paths cannot be solidified and reused, resulting in a significant improvement in test execution efficiency and stability.

[0077] Based on the foregoing embodiments, this application provides a battery state-of-charge estimation device, which can be applied to... Figure 1 and Figure 3 In the corresponding embodiment, the battery state-of-charge estimation method is provided, referring to Figure 9 As shown, the battery state-of-charge estimation device 3 may include: an acquisition unit 31, a generation unit 32, a processing unit 33, and a storage unit 34, wherein: The acquisition unit 31 is used to determine the page information of at least one page and the control information contained in each page based on the source code of the application under test. The generation unit 32 is used to generate a test operation sequence based on historical operation data if the application under test is being tested for the first time; wherein, the historical operation data contains a sequence of operation steps mapped from the user's operation intention, and the operation steps in the test operation sequence are associated with controls corresponding to control information; The processing unit 33 is used to determine the target position of the control to be operated associated with the operation step in the current page corresponding to the operation step for each operation step in the test operation sequence using multiple large language models, and to perform the operation corresponding to the operation step based on the target position to obtain the operation result. Storage unit 34 is used to store the identifier of the large language model used in the operation steps, the target location, and the operation parameters used to perform the operation if the operation result characterizes the operation as successful, forming reusable structured data; The processing unit 33 is also used to obtain a test operation sequence if it is not the first time testing the application under test, and to execute each operation step in the test operation sequence based on the structured data.

[0078] In other embodiments of this application, the acquisition unit 31 is further configured to perform the following steps: The source code is parsed using a large language model to obtain page information and control information for at least one page; the page information includes at least the page identifier, page name, and page type; the control information includes at least the control identifier, control name, and control type.

[0079] In other embodiments of this application, the generation unit 32 is further configured to perform the following steps: Analyze historical operation data to obtain user operation intent and the corresponding sequence of operation steps; For each operation step in the sequence of operation steps, the target control is matched from the control information based on the control description in the operation step. Associate the operation steps with the target control to generate a test operation sequence.

[0080] In other embodiments of this application, the processing unit 33 is further configured to perform the following steps: Get at least one screen image of the current page; At least one screen image is processed using multiple large language models to obtain the candidate positions of the control to be operated on in the current page corresponding to the operation step, as output by each large language model. Determine the target location from multiple candidate locations.

[0081] In other embodiments of this application, the processing unit 33 is further configured to perform the following steps: The display area corresponding to each screen image is divided into multiple sub-regions; among which, the multiple sub-regions include top left, top center, top right, left center, center, right center, bottom left, bottom center, and bottom right. For each major language model, the target screen image where the control to be operated is located is determined based on the major language model, and the target sub-region where the control to be operated is located in the target screen image is determined; wherein, multiple sub-regions include the target sub-region; The location information consisting of the target screen image and the target sub-region is used to determine the candidate location.

[0082] In other embodiments of this application, the storage unit 34 is also used to perform the following steps: The identifier, target location, and operation parameters are associated and recorded; The data associated with the records is then stored in a fixed format to form structured data.

[0083] In other embodiments of this application, the processing unit 33 is further configured to perform the following steps: For each operation step in the test operation sequence, the identifier, target location, and operation parameters of the large language model corresponding to the operation step are obtained from the structured data; Based on the identifier, target location, and operation parameters of the large language model, the target control associated with the operation steps is driven to execute the operation steps.

[0084] It should be noted that the specific implementation process of the steps performed by each module in the embodiments of this application can be referred to Figure 1 and Figure 3 The implementation process of the automated testing method provided in the corresponding embodiment will not be described in detail here.

[0085] The automated testing device provided in this application significantly improves the accuracy and success rate of single operation steps by employing multiple large language models in parallel processing and a voting mechanism to determine the target position of the control during the first execution and verifying the operation results to ensure the correct execution of the steps. Furthermore, the large language model identifiers, target positions, and operation parameters on which the operation is successful are associated, recorded, and stored as structured data, forming reusable test assets. This allows these solidified data to be directly called to drive testing in subsequent non-first executions, completely avoiding repeated AI recognition and trial processes. As a result, the overall UI automated testing achieves improvements in path planning accuracy, component positioning, and execution success rate, and fundamentally solves the problem that successful paths cannot be solidified and reused, resulting in a significant improvement in test execution efficiency and stability.

[0086] Based on the foregoing embodiments, embodiments of this application provide an automated testing device that can be applied to... Figure 1 and Figure 3 In the automated testing method provided in the corresponding embodiment, refer to Figure 10 As shown, the automated testing device 4 may include: a processor 41, a memory 42, and a communication bus 43, wherein: Communication bus 43 is used to realize the communication connection between processor 41 and memory 42; The processor 41 is used to execute an automated test program for the battery in the memory 42 to perform the following steps: Based on the source code of the application under test, determine the page information of at least one page and the control information contained in each page; If this is the first time testing the application under test, a test operation sequence is generated based on historical operation data. The historical operation data contains a sequence of operation steps mapped from the user's operation intentions, and the operation steps in the test operation sequence are associated with controls corresponding to the control information. For each operation step in the test operation sequence, multiple large language models are used to determine the target position of the control to be operated associated with the operation step in the current page corresponding to the operation step, and the operation corresponding to the operation step is executed based on the target position to obtain the operation result; If the operation result indicates that the operation was successful, the identifier of the large language model used in the operation steps, the target location, and the operation parameters used to perform the operation are stored to form reusable structured data. If this is not the first time testing the application under test, obtain the test operation sequence and execute each operation step in the test operation sequence based on the structured data.

[0087] In other embodiments of this application, processor 41 is used to execute the source code of the automated test program in memory 42 based on the application under test, to determine page information of at least one page and control information contained in each page, in order to perform the following steps: The source code is parsed using a large language model to obtain page information and control information for at least one page; the page information includes at least the page identifier, page name, and page type; the control information includes at least the control identifier, control name, and control type.

[0088] In other embodiments of this application, the processor 41 is used to execute an automated test program in the memory 42 to generate a test operation sequence based on historical operation data, in order to implement the following steps: Analyze historical operation data to obtain user operation intent and the corresponding sequence of operation steps; For each operation step in the sequence of operation steps, the target control is matched from the control information based on the control description in the operation step. Associate the operation steps with the target control to generate a test operation sequence.

[0089] In other embodiments of this application, the processor 41 is used to execute each operation step in the test operation sequence of the automated test program in the memory 42, and uses multiple large language models to determine the target position of the control to be operated associated with the operation step in the current page corresponding to the operation step, so as to achieve the following steps: Get at least one screen image of the current page; At least one screen image is processed using multiple large language models to obtain the candidate positions of the control to be operated on in the current page corresponding to the operation step, as output by each large language model. Determine the target location from multiple candidate locations.

[0090] In other embodiments of this application, the processor 41 is used to execute an automated test program in the memory 42 to process at least one screen image using multiple large language models, and to obtain the candidate positions of the control to be operated on in the current page corresponding to the operation step output by each large language model, so as to achieve the following steps: The display area corresponding to each screen image is divided into multiple sub-regions; among which, the multiple sub-regions include top left, top center, top right, left center, center, right center, bottom left, bottom center, and bottom right. For each major language model, the target screen image where the control to be operated is located is determined based on the major language model, and the target sub-region where the control to be operated is located in the target screen image is determined; wherein, multiple sub-regions include the target sub-region; The location information consisting of the target screen image and the target sub-region is used to determine the candidate location.

[0091] In other embodiments of this application, the processor 41 is used to execute the storage operation steps of the automated test program in the memory 42, using the identifier, target location, and target operation parameters of the large language model to form reusable structured data to achieve the following steps: The identifier, target location, and operation parameters are associated and recorded; The data associated with the records is then stored in a fixed format to form structured data.

[0092] In other embodiments of this application, processor 41 is used to execute each operation step in the structured data-based test operation sequence of the automated test program in memory 42 to achieve the following steps: For each operation step in the test operation sequence, the identifier, target location, and operation parameters of the large language model corresponding to the operation step are obtained from the structured data; Based on the identifier, target location, and operation parameters of the large language model, the target control associated with the operation steps is driven to execute the operation steps.

[0093] It should be noted that a detailed description of the steps performed by the processor can be found in [reference needed]. Figure 1 and Figure 3 The implementation process of the automated testing method provided in the corresponding embodiment will not be described in detail here.

[0094] The automated testing equipment provided in this application, upon initial execution, determines the target position of the control by employing parallel processing and voting mechanisms of multiple large language models. It also ensures the correct execution of steps by verifying the operation results, significantly improving the accuracy and success rate of single operation steps. Furthermore, it associates, records, and stores the identifiers, target positions, and operation parameters of the large language models upon which the operation success depends as structured data, forming reusable test assets. This allows for direct use of this solidified data to drive testing in subsequent non-first-time executions, completely avoiding repetitive AI identification and trial processes. As a result, it improves the accuracy of path planning, component positioning, and execution success rate in UI automated testing, fundamentally solving the problem of unreusable successful paths and significantly improving test execution efficiency and stability.

[0095] Based on the foregoing embodiments, this application provides a computer program product, including a computer program, which implements [the following] when executed by a processor. Figure 1 and Figure 3 The steps in the automated testing method provided in the corresponding embodiment.

[0096] Based on the foregoing embodiments, this application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to achieve... Figure 1 and Figure 3 The steps in the automated testing method provided in the corresponding embodiment.

[0097] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An automated testing method, characterized in that, The method includes: Based on the source code of the application under test, determine the page information of at least one page and the control information contained in each page; If the application under test is being tested for the first time, a test operation sequence is generated based on historical operation data; wherein, the historical operation data contains a sequence of operation steps mapped from the user's operation intention, and the operation steps in the test operation sequence are associated with controls corresponding to the control information; For each operation step in the test operation sequence, multiple large language models are used to determine the target position of the control to be operated associated with the operation step in the current page corresponding to the operation step, and the operation corresponding to the operation step is executed based on the target position to obtain the operation result; If the operation result indicates that the operation was successful, the identifier of the large language model used in the operation steps, the target location, and the operation parameters used to perform the operation are stored to form reusable structured data; If this is not the first time the application under test is being tested, obtain the test operation sequence and execute each operation step in the test operation sequence based on the structured data.

2. The method according to claim 1, characterized in that, Based on the source code of the application under test, the page information of at least one page and the control information contained in each page are determined, including: The source code is parsed using a large language model to obtain page information and control information for at least one page; wherein, the page information includes at least a page identifier, a page name, and a page type; and the control information includes at least a control identifier, a control name, and a control type.

3. The method according to claim 2, characterized in that, The generation of test operation sequences based on historical operation data includes: The historical operation data is parsed to obtain the user's operation intention and the corresponding operation step sequence; For each operation step in the sequence of operation steps, a target control is matched from the control information based on the control description in the operation step. The operation steps are associated with the target control to generate the test operation sequence.

4. The method according to claim 1, characterized in that, For each operation step in the test operation sequence, multiple large language models are used to determine the target position of the control to be operated associated with the operation step in the current page corresponding to the operation step, including: Obtain at least one screen image of the current page; The at least one screen image is processed using the multiple large language models to obtain the candidate position of the control to be operated in the current page corresponding to the operation step, output by each large language model. The target location is determined from a plurality of candidate locations.

5. The method according to claim 4, characterized in that, The step of processing the at least one screen image using the multiple large language models to obtain the candidate position of the control to be operated on in the current page corresponding to the operation step, output by each large language model, includes: The display area corresponding to each screen image is divided into multiple sub-regions; wherein, the multiple sub-regions include upper left, upper center, upper right, middle left, middle, middle right, lower left, lower center, and lower right. For each of the large language models, the target screen image where the control to be operated is located is determined based on the large language model, and the target sub-region where the control to be operated is located in the target screen image is determined; wherein, the plurality of sub-regions includes the target sub-region; The location information consisting of the target screen image and the target sub-region is determined as the candidate location.

6. The method according to claim 1, characterized in that, The storage of the identifier of the large language model used in the operation steps, the target location, and the target operation parameters forms reusable structured data, including: The identifier, the target location, and the operation parameters are associated and recorded; The data associated with the records is then stored in a fixed format to form the structured data.

7. The method according to claim 1, characterized in that, The execution of each operation step in the test operation sequence based on the structured data includes: For each operation step in the test operation sequence, the identifier, target location, and operation parameters of the large language model corresponding to the operation step are obtained from the structured data; Based on the identifier of the large language model, the target location, and the operation parameters, the target control associated with the operation step is driven to execute the operation step.

8. An automated testing device, characterized in that, The device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method according to any one of claims 1 to 7.

9. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs that can be executed by one or more processors to implement the method of any one of claims 1 to 7.

Images