An agent-based mobile application automatic dialing test method and system

Abstract

The application provides an agent-based mobile terminal application automatic dialing test method and system, and belongs to the technical field of mobile terminal application automatic test. The method is executed by an agent of a control terminal, and comprises the following steps: acquiring a current interface image of a mobile terminal in real time; performing multi-modal analysis on the interface image, detecting UI components in the interface, extracting the bounding boxes and category labels of the components, extracting text content in the interface, and fusing the UI components and the text content into a structured UI representation; inputting the representation, a preset task instruction and a historical operation memory into a large language model, so that the model generates a next operation instruction in a sequence decision manner; the mobile terminal executes the next operation instruction to complete automatic interactive operation; then, the step of acquiring the current interface image is returned, and the next round of operation is executed until a task completion or termination condition is met. Based on the method, a corresponding system is also provided. The application realizes high-coverage and strong-generalization automatic dialing test of mobile terminal applications.

Description

Technical Field

[0001] This invention belongs to the field of automated testing technology for mobile applications, and specifically relates to an automated testing method and system for mobile applications based on intelligent agents. Background Technology

[0002] With the rapid development of the mobile internet, the number and complexity of smartphone applications continue to rise, with more diverse application interaction logic, more complex interface structures, and a significantly increased version iteration frequency. To test and collect application traffic, the demand for automated testing of mobile applications is becoming increasingly urgent. However, current mainstream automated testing solutions in the industry still have significant limitations, failing to meet the testing needs of multiple scenarios and application types, and struggling to support the high-frequency probing and continuous monitoring tasks brought about by the rapid evolution of the application ecosystem. Traditional script-based automation relies on manually writing test scripts or recording operation steps. Such methods are highly dependent on specific interface structures; once the application is upgraded, the layout is adjusted, or the interaction logic changes, the scripts are prone to failure, requiring extensive manual maintenance, resulting in high costs and limited scalability. While UI automation frameworks based on control trees can directly manipulate controls, they still rely on explicit control IDs, hierarchical structures, and other information. Many commercial applications, for the sake of interface performance, often exhibit dynamic control changes, control obfuscation, or even the use of self-drawn controls, making it difficult for automation frameworks to identify and causing probing failures. Furthermore, such methods still rely on static rules and lack the ability to understand task intent or autonomously plan operation steps.

[0003] Graphical User Interfaces (GUIs) have always been at the core of human-computer interaction, providing users with an intuitive, visually driven way to access and operate digital systems. Traditional automated GUI interaction relies on script-based or rule-based methods, such as Monkey Testing, which generates random input operations on the interface to discover potential problems and is often used for robustness testing of mobile apps. Rule-based methods generate goal-oriented operation sequences by explicitly modeling GUI state transition logic. While these systems improve the degree of test automation, they generally suffer from poor flexibility, weak generalization ability, and heavy reliance on human intervention, making it difficult to adapt to dynamically changing interface content and complex user tasks. With the widespread application of large language models and multimodal models, agent-based interactive systems are gradually becoming an important direction for intelligent automation. An agent is an intelligent entity capable of perceiving the environment, understanding task intent, making autonomous decisions, and executing operations. Multimodal agents possess the ability to recognize visual elements on the interface, understand text and task intent, and generate executable operation steps based on reasoning, making it possible to "operate mobile applications like a human." Its essential advantage lies in its ability to break free from dependence on control trees and script rules, achieving a higher level of intelligence through the four stages of "perception—understanding—decision—operation." A GUI Agent is an intelligent agent system that simulates human user behavior, completing intelligent interaction and task execution on a graphical interface through clicking, swiping, and text input. This type of intelligent agent can perceive visual elements on the screen, understand the interface layout and semantics, and make decisions and operations accordingly, achieving autonomous control of applications on desktop, mobile, and other platforms.

[0004] However, existing solutions mostly rely on large cloud models to directly process images, which suffers from high costs, high latency, and poor stability. Therefore, there is an urgent need in this field for a new automatic dialing test mechanism that can automatically understand interface content, autonomously generate task operation steps, and stably execute the entire task chain without script or control dependencies, thereby achieving real-world usage scenario simulation and reliable dialing test of mobile applications. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes an automatic testing method and system for mobile applications based on intelligent agents. This method enables low-cost, robust, and automated testing of a large number of mobile applications, effectively reducing testing costs while improving testing stability and versatility.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention proposes an automatic dialing test method for mobile applications based on an intelligent agent, executed by an intelligent agent on the control end, comprising the following steps:

[0008] Real-time capture of the current screen image on the mobile device;

[0009] The current interface image is subjected to local multimodal analysis to detect UI components in the interface and extract the bounding boxes and category labels of each component. At the same time, the text content in the interface is extracted through optical character recognition. The UI components and text content are associated and fused into a structured UI representation.

[0010] The structured UI representation, preset task instructions, and historical operation memory are input into the large language model, which then generates the next operation instruction in a sequential decision-making manner.

[0011] The next operation instruction is executed via the mobile device to complete the automated interactive operation; then the process returns to the step of obtaining the current interface image of the mobile device in real time, and continues to execute the next round of operations until the task completion condition is met or the termination condition is triggered.

[0012] Furthermore, local multimodal parsing is performed on the current interface image, specifically as follows:

[0013] Perform lightweight preprocessing on the current interface image to obtain the preprocessed interface image;

[0014] The UI component detection algorithm is used to detect the UI components in the preprocessed interface image, identify each UI component in the interface, and extract the bounding box and category label of each component.

[0015] Optical character recognition is performed on the preprocessed interface image to extract the text content and corresponding text location information in the interface.

[0016] Based on the bounding boxes and text position information of each component, the identified text content is associated and fused with the corresponding UI component, and each UI component is associated with its corresponding text content to form a structured UI representation.

[0017] Furthermore, the structured UI representation is expressed as:

[0018] ;

[0019] in, express The structured UI representation of the timeline interface; Indicates the number of times in the current interface One UI element; Indicates the coordinate center of the bounding box; Represented as semantic category labels; Represented as text content; This is represented as an interactive property; This indicates the total number of UI elements.

[0020] Furthermore, the large language model generates the next operation instruction using a sequence decision-making method, specifically as follows:

[0021] The system analyzes the category labels and text content of each UI element in the current structured UI representation, identifies the functional attributes of each UI element, and forms the semantic understanding result of the current interface.

[0022] The semantic understanding results of the current interface and the preset task instructions as well as UI representation of time Together, they serve as contextual input, and the large language model is used to compute candidate operations in action space A. conditional probability distribution ;

[0023] Select the candidate operation with the highest conditional probability As the output of the next operation instruction, it satisfies:

[0024] ;

[0025] in, For the current moment UI representation.

[0026] Furthermore, the operation instructions include one or more of the following: clicking on a specified coordinate, sliding from the starting point to the ending point, entering specified text, returning to the previous screen, and returning to the main screen.

[0027] Furthermore, before executing the next operation instruction via the mobile device, a status anomaly detection step is also included:

[0028] The current interface image is input into a multimodal large model for state determination, detecting whether there are abnormal pop-ups or interface obstruction on the current interface, and obtaining the anomaly detection results. ;

[0029] Determine the operation instructions to be executed based on the anomaly detection results:

[0030] ;

[0031] in, This indicates a recovery operation command; This indicates the next step instruction generated.

[0032] Furthermore, the state anomaly detection step also includes task timeout control, specifically:

[0033] Record the execution time of the current task;

[0034] When the execution time exceeds a preset threshold, the current task will be forcibly terminated and the process will switch to the next task.

[0035] Furthermore, the method also includes:

[0036] Record the operation type, operation parameters, timestamp, interface change information and execution result of each operation to form an operation record;

[0037] When an anomaly occurs, the anomaly type and corresponding recovery operation information are recorded to form an anomaly log.

[0038] A test report is generated based on the operation record and the anomaly record.

[0039] Furthermore, the task completion condition is to complete all the operation steps corresponding to the preset task instruction;

[0040] The termination conditions include task execution timeout, detection of an unrecoverable anomaly, or receipt of an external stop command.

[0041] Secondly, this invention also proposes an automatic dialing test system for mobile applications based on intelligent agents, comprising:

[0042] On the control end, an intelligent agent is deployed. The intelligent agent is used to perform the following: real-time acquisition of the current interface image of the mobile terminal; local multimodal analysis of the current interface image, detection of UI components in the interface and extraction of bounding boxes and category labels of each component, and extraction of text content in the interface through optical character recognition, and association and fusion of the UI components and text content into a structured UI representation; inputting the structured UI representation, preset task instructions, and historical operation memory into a large language model, which generates the next operation instruction in a sequential decision-making manner; executing the next operation instruction through the mobile terminal to complete the automated interactive operation; and then returning to the step of real-time acquisition of the current interface image of the mobile terminal to continue to execute the next round of operations until the task completion condition is met or the termination condition is triggered.

[0043] And at least one mobile terminal, which is communicatively connected to the control terminal, for responding to operation commands sent by the intelligent agent to perform automated interactive operations, and providing interface images for the intelligent agent to collect.

[0044] The effects described in the invention are merely those of the embodiments, and not all the effects of the invention. One of the above technical solutions has the following advantages or beneficial effects:

[0045] This invention proposes an automated testing method and system for mobile applications based on intelligent agents, belonging to the field of automated testing technology for mobile applications. The method is executed by an intelligent agent on the control end and includes the following steps: real-time acquisition of the current interface image of the mobile device; local multimodal analysis of the current interface image to detect UI components and extract the bounding boxes and category labels of each component, while simultaneously extracting text content from the interface through optical character recognition; associating and fusing the UI components and text content into a structured UI representation; inputting the structured UI representation, preset task instructions, and historical operation memories into a large language model, which generates the next operation instruction using a sequential decision-making approach; executing the next operation instruction on the mobile device to complete the automated interactive operation; then returning to the step of real-time acquisition of the current interface image of the mobile device to continue executing the next round of operations until the task completion condition is met or the termination condition is triggered. Based on this method, an automated testing system for mobile applications based on intelligent agents is also proposed. This invention, by introducing GUIAgent technology, breaks the dependence of traditional script-based automated testing on rules, control structures, and manual maintenance, achieving high coverage and strong generalization of automated testing capabilities for mobile applications. This invention can automatically understand the interface, automatically generate and execute operation steps, and has universal applicability across scenarios and applications. Compared with existing methods, this invention significantly improves the success rate of dial-up testing, reduces maintenance costs, and provides a new, efficient, and feasible technical approach for large-scale mobile application traffic collection.

[0046] This invention enables automated task execution and performance verification for various types of apps by constructing a GUI Agent with perception, understanding, and operation capabilities. This eliminates the need for manually writing test scripts or recording operation steps, significantly reducing test configuration and maintenance costs.

[0047] This invention employs local multimodal parsing technology, utilizing only screenshots of the mobile interface for UI component detection and text recognition. It does not rely on control tree structures and script rules, enabling the GUI Agent to understand the application interface in a manner close to human vision, and possessing cross-interface, cross-version, and cross-layout adaptability.

[0048] This invention uses structured UI representations as input to a large language model for sequence decision-making, replacing the traditional testing method that relies on static rules. It can automatically understand task intent, autonomously plan operation chains, and achieve a complete closed loop from task parsing and interface understanding to execution testing.

[0049] This invention performs local visual analysis on interface images and inputs the compressed structured UI representation into a large language model, which significantly reduces inference complexity and API call overhead, while ensuring high performance and stability in large-scale continuous probing scenarios.

[0050] This invention designs an anomaly detection and automatic recovery mechanism, which can detect anomalies such as pop-ups and interface blockages and automatically perform recovery operations. It also has task timeout control to ensure the overall stability and controllability of the dialing and testing process. Attached Figure Description

[0051] Figure 1 This is a flowchart of an automatic dialing test method for mobile applications based on intelligent agents, as proposed in Embodiment 1 of the present invention.

[0052] Figure 2 This is the first figure of a specific dialing test example in Embodiment 1 of the present invention;

[0053] Figure 3 This is the second figure of a specific dialing test example in Embodiment 1 of the present invention;

[0054] Figure 4 This is the third figure of a specific dialing test example in Embodiment 1 of the present invention;

[0055] Figure 5 This is the fourth figure of a specific dialing test example in Embodiment 1 of the present invention;

[0056] Figure 6 This is the fifth figure of a specific dialing test example in Embodiment 1 of the present invention;

[0057] Figure 7 This is a schematic diagram of an automatic dialing system for mobile applications based on intelligent agents, according to Embodiment 2 of the present invention. Detailed Implementation

[0058] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific embodiments and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure of the invention, components and arrangements of specific examples are described below. Furthermore, reference numerals and / or letters may be repeated in different examples. This repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. It should be noted that the components illustrated in the drawings are not necessarily drawn to scale. Descriptions of well-known components, processing techniques, and processes are omitted in this invention to avoid unnecessarily limiting the invention.

[0059] Example 1

[0060] Embodiment 1 of this invention proposes an automatic testing method for mobile applications based on intelligent agents, which solves the technical problem that existing automated testing of mobile applications relies on scripts, rules, or control tree structures, making it difficult to cope with the increasing number of application versions, frequent interface changes, and large differences in the structure of various types of applications.

[0061] Embodiment 1 of this invention proposes an automated testing method for mobile applications based on an intelligent agent. This method is executed by an intelligent agent at the control end. By constructing a multimodal agent that supports visual recognition, text parsing, intent understanding, chained reasoning, and operation execution, it solves problems such as rigid rules, fragile scripts, and difficulty in adapting to changes in traditional automated testing. In this invention, the agent can directly analyze screenshots of the mobile application interface, identify visual elements such as button text, icon meanings, and page structure, and autonomously plan multi-step operation chains based on user task instructions or preset testing tasks, achieving a complete closed loop from task parsing and interface understanding to execution testing.

[0062] Figure 1 This is a flowchart of an automatic dialing test method for mobile applications based on intelligent agents, as proposed in Embodiment 1 of the present invention.

[0063] In step S1, the dialing test task begins.

[0064] The system initiates the testing process, initializes the intelligent agent's operating environment, establishes a communication connection between the control terminal and the mobile terminal (e.g., via the ADB debugging interface), and prepares to execute the preset testing task queue.

[0065] Specifically, this includes: starting the intelligent agent service on the control terminal and loading configuration parameters (such as timeout threshold, number of retries, etc.); establishing a communication link with the mobile terminal and verifying the connection status; initializing the task queue and preparing to execute each test task in sequence.

[0066] In step S2, task instructions are generated.

[0067] The agent obtains the description of the currently pending test task and parses it into a understandable task instruction format. The task instruction can be a task description in natural language, such as "Search for photography-related posts in a lifestyle sharing community APP and open the first one".

[0068] Specific implementation: Read the preset testing tasks from the task queue; the task instructions exist in the form of natural language text, describing the operation objectives to be completed; the agent stores the task instructions as input parameters T for subsequent decisions.

[0069] By using natural language task instructions, this invention eliminates the need for manual writing of test scripts or recording of operation steps. The intelligent agent can autonomously understand the task intent and plan the execution path, significantly reducing the configuration cost of testing tasks.

[0070] In step S3, the current interface image of the mobile device is acquired in real time;

[0071] The intelligent agent on the control end acquires a screenshot of the current interface from the mobile device via the Android Debug Bridge (ADB) debugging interface. To ensure real-time performance, this acquisition process can be completed within milliseconds, and the image is then transmitted to the control end for processing.

[0072] In step S4, local multimodal analysis is performed on the current interface image.

[0073] The intelligent agent at the control end performs local multimodal analysis on the acquired interface images. Specifically, this includes:

[0074] Lightweight preprocessing of the interface image, including size scaling and color space conversion, is performed to reduce the computational burden of subsequent processing.

[0075] A target detection algorithm is used to detect UI components in the preprocessed interface image, identifying each UI component and extracting its bounding box and category label. The category labels include, but are not limited to, buttons, icons, input fields, and list items.

[0076] Optical Character Recognition (OCR) algorithm is used to perform text recognition on the preprocessed interface image, extracting the text content and corresponding text location information in the interface.

[0077] Based on the bounding boxes and text position information of each component, the positional overlap or spatial distance between each UI component and each text content is calculated. The text content is then associated with the UI component that best matches its position, and each UI component is associated with its corresponding text content, forming a structured set of UI elements.

[0078] The structured UI representation is as follows:

[0079] ;

[0080] in, express The structured UI representation of the timeline interface; Indicates the number of times in the current interface One UI element; Indicates the coordinate center of the bounding box; Represented as semantic category labels; Represented as text content; This is represented as an interactive property; This indicates the total number of UI elements.

[0081] in, ,in Indicates the coordinates of the center point of the UI element; ; This represents the text content extracted via OCR. It will be empty if there is no text. This is an interactive attribute. The entire parsing process is executed locally on the control terminal, without needing to call large cloud models for visual reasoning, thus significantly reducing system operating costs and ensuring high performance and stability in large-scale continuous probing scenarios.

[0082] In step S5, the structured UI representation, preset task instructions, and historical operation memory are input into the large language model, which then generates the next operation instruction using a sequential decision-making approach. In this application, the large language model is located at the control terminal. The input to the large language model is no longer the original image, but a compressed structured UI representation, thus significantly reducing reasoning complexity and API call overhead.

[0083] The agent first analyzes the semantic function of each interactive element in the interface, such as identifying functional components like the "search box," "back button," "playback control," and "category filter," and determines whether these components are relevant to the current testing target. Then, based on task requirements and historical operation trajectories, the agent generates the optimal operation strategy; its decision-making process can be modeled as a sequence decision problem.

[0084] The large language model generates the next operation instruction using a sequence decision method, specifically as follows:

[0085] The system analyzes the category labels and text content of each UI element in the current structured UI representation, identifies the functional attributes of each UI element, and forms the semantic understanding result of the current interface.

[0086] The semantic understanding results of the current interface and the preset task instructions as well as UI representation of time Together, they serve as contextual input, and the large language model is used to compute candidate operations in action space A. conditional probability distribution ;

[0087] Select the candidate operation with the highest conditional probability As the output of the next operation instruction, it satisfies:

[0088] ;

[0089] in, For the current moment UI representation.

[0090] The operation instructions include one or more of the following: clicking on a specified coordinate, sliding from the starting point to the ending point, entering specified text, returning to the previous screen, and returning to the main screen.

[0091] This application selects the action with the highest probability. As the currently executed instruction, it enables continuous reasoning and decision-making based on context understanding, and can generate and execute the detailed operations shown in Table 1.

[0092] Table 1: Operations that the Agent can generate and execute

[0093]

[0094] After the Agent generates specific operations, the actual operation is performed on the mobile device through the test execution module. A single test task will be completed by an action set S consisting of N operations.

[0095]

[0096] in, Represents a set of actions; a single test task consists of a complete sequence of operations arranged in order.

[0097] Indicates the first An operation instruction is a single operation generated by the agent, such as clicking, swiping, or entering text. Indicates the total number of operations;

[0098] This represents the action space, a collection of all executable operation types, including click, swipe, text input, back, and return to the main page.

[0099] In step S6, anomaly detection and handling are performed. During testing, this invention performs anomaly detection before each operation, such as checking if the interface refreshes normally, if the expected page is entered, or if a blocking pop-up appears. This ensures the system can continuously advance the task without getting stuck in an infinite loop or prolonged stagnation. When the system detects an abnormal pop-up on the page, the UI screenshot is first input into a multimodal large model for state judgment.

[0100] ;

[0101] in, This indicates a recovery operation command; Indicates the generated next operation instruction

[0102] If the system passes the anomaly detection function D( When an intrusive pop-up is detected, the Agent will automatically generate and execute a "close pop-up" command. This restores the normal task flow; when no anomalies are detected, the system continues to execute the next operation planned in the test task. During long-duration tasks or continuous tests, recoverable errors may occur due to network latency, interface failures, etc. To address this, the system is designed with an automatic recovery mechanism that can reload the current task state and continue execution after a repairable anomaly is detected. Additionally, to avoid infinite loops, each task has a time limit.

[0103]

[0104] When a single test task execution time Exceeding a given threshold When this happens, the system will forcibly terminate the current task and jump to the next task to ensure the overall stability and controllability of the testing process.

[0105] When this invention is executed:

[0106] Record the operation type, operation parameters, timestamp, interface change information and execution result of each operation to form an operation record;

[0107] When an anomaly occurs, the anomaly type and corresponding recovery operation information are recorded to form an anomaly log.

[0108] A test report is generated based on the operation record and the anomaly record.

[0109] In this invention, to achieve accurate detection of anomalies in mobile application interfaces, a multimodal large model is introduced as the core component for state judgment. A multimodal large model refers to a large neural network model capable of simultaneously processing multiple modalities of data, such as images and text, and possessing the ability to understand interface semantics and identify abnormal states from visual input.

[0110] Specifically, the multimodal large model in this invention adopts a vision-language architecture, using mobile interface screenshots as input. The model's visual encoder extracts visual features from the images, including pop-up shapes, button layouts, text regions, and occlusion patterns. The language decoder then outputs classification results or descriptive information about the interface state. This model can be pre-tuned on a large amount of mobile interface screenshot data to accurately identify various abnormal scenarios, including but not limited to:

[0111] Pop-up ads: various promotional, coupon, and event recommendations that are not native to the application;

[0112] Permission request pop-up: A prompt box indicating that the application is requesting access to system permissions such as camera, storage, and location;

[0113] System pop-up windows: system-level dialog boxes such as network error, version update, and error message;

[0114] Interface blocking: A state where the application is unable to continue interacting, such as a blank screen, infinite loading, or crash message.

[0115] In step S7, it is determined whether the task instruction has been completed. The task completion condition in this step is to complete all the operation steps corresponding to the preset task instruction; the termination trigger conditions include task execution timeout, detection of an unrecoverable abnormality, or receipt of an external stop instruction.

[0116] If all task instructions are completed, proceed to step S8; otherwise, return to step S3 and continue to the next round of operations until the task completion condition is met or the termination condition is triggered.

[0117] In step S8, the testing task is completed. Table 2 below provides an example of the test task, showing good performance in the automatic testing task for mobile applications. Table 3 below provides detailed performance metrics for the platform test.

[0118] Table 2: Examples of Test Tasks

[0119]

[0120] Table 3: Performance Indicators of the Testing Platform

[0121]

[0122] Embodiment 1 of this invention proposes an automated mobile application testing method based on intelligent agents. Through GUIAgent, it completes local multimodal parsing, structured UI representation, large-model inference and decision-making, and automated execution linkage, realizing an automated testing platform that requires no scripts, source code, or manual configuration. Against the backdrop of the explosive growth in the number of mobile internet applications, this invention can significantly reduce the human resource investment in application testing, making the testing process sustainable, automated, and intelligent, and possesses significant engineering practice value and industrial application prospects.

[0123] To fully illustrate the implementation process of Embodiment 1 of the present invention, a specific testing task case is created. The testing object is the "Lifestyle Sharing Community" mobile application, and the target task instruction is: "Search for photography-related posts in the 'Lifestyle Sharing Community' and open the first one." This task covers multiple typical interaction steps, including opening a specific application, homepage recognition, search entry location, text input, list parsing, and content page navigation, and can comprehensively reflect the automatic testing capability of the present invention in a real application environment. The detailed steps are as follows:

[0124] The user requested to "search for photography-related posts in the lifestyle sharing community and open the first one." Therefore, the first step is to open the "Lifestyle Sharing Community" app. Figure 2This is the first figure of a specific dialing test example in Embodiment 1 of the present invention;

[0125] The lifestyle sharing community app has been opened. The next step is to enter "photography-related posts" in the search box and then open the first search result. Figure 3 This is the second figure of a specific dialing test example in Embodiment 1 of the present invention;

[0126] I've already opened the "Lifestyle Sharing Community" app and clicked the search icon to begin my search. Next, I need to enter "photography" in the search box and open the first relevant post. Figure 4 The third figure is a specific example of dialing test in Embodiment 1 of the present invention.

[0127] You've already completed the steps of opening the "Lifestyle Sharing Community" app, clicking the search icon, and entering "photography" in the search box. Next, you need to open the first post in the search results. Figure 5 The fourth figure is a specific example of dialing test in Embodiment 1 of the present invention.

[0128] The completed steps included opening the "Lifestyle Sharing Community" app, clicking the search icon, entering "photography" in the search box, and clicking the first post in the search results. Figure 6 This is the fifth image in the specific dialing test example of Embodiment 1 of the present invention. The content of this post has already been displayed on the current page. All requirements have been completed according to the user's instructions, and no further operation is required.

[0129] Example 2

[0130] Based on the agent-based automatic dialing test method for mobile applications proposed in Embodiment 1 of this invention, Embodiment 2 of this invention also proposes an agent-based automatic dialing test system for mobile applications. Figure 7 This is a schematic diagram of an automatic dialing test system for mobile applications based on intelligent agents, according to Embodiment 2 of the present invention. The system includes:

[0131] On the control end, an intelligent agent is deployed. The intelligent agent is used to perform the following: real-time acquisition of the current interface image of the mobile terminal; local multimodal analysis of the current interface image, detection of UI components in the interface and extraction of bounding boxes and category labels of each component, and extraction of text content in the interface through optical character recognition, and association and fusion of the UI components and text content into a structured UI representation; inputting the structured UI representation, preset task instructions, and historical operation memory into a large language model, which generates the next operation instruction in a sequential decision-making manner; executing the next operation instruction through the mobile terminal to complete the automated interactive operation; and then returning to the step of real-time acquisition of the current interface image of the mobile terminal to continue to execute the next round of operations until the task completion condition is met or the termination condition is triggered.

[0132] And at least one mobile terminal, which is communicatively connected to the control terminal, for responding to operation commands sent by the intelligent agent to perform automated interactive operations, and providing interface images for the intelligent agent to collect.

[0133] The agent performs local multimodal parsing of the current interface image, specifically as follows:

[0134] Perform lightweight preprocessing on the current interface image to obtain the preprocessed interface image;

[0135] The UI component detection algorithm is used to detect the UI components in the preprocessed interface image, identify each UI component in the interface, and extract the bounding box and category label of each component.

[0136] Optical character recognition is performed on the preprocessed interface image to extract the text content and corresponding text location information in the interface.

[0137] Based on the bounding boxes and text position information of each component, the identified text content is associated and fused with the corresponding UI component, and each UI component is associated with its corresponding text content to form a structured UI representation.

[0138] The structured UI representation is expressed as:

[0139] ;

[0140] in, express The structured UI representation of the timeline interface; Indicates the number of times in the current interface One UI element; Indicates the coordinate center of the bounding box; Represented as semantic category labels; Represented as text content; This is represented as an interactive property; This indicates the total number of UI elements.

[0141] The large language model generates the next operation instruction using a sequence decision method, specifically as follows:

[0142] The system analyzes the category labels and text content of each UI element in the current structured UI representation, identifies the functional attributes of each UI element, and forms the semantic understanding result of the current interface.

[0143] The semantic understanding results of the current interface and the preset task instructions as well as UI representation of time Together, they serve as contextual input, and the large language model is used to compute candidate operations in action space A. conditional probability distribution ;

[0144] Select the candidate operation with the highest conditional probability As the output of the next operation instruction, it satisfies:

[0145] ;

[0146] in, For the current moment UI representation.

[0147] The operation instructions include one or more of the following: clicking on a specified coordinate, sliding from the starting point to the ending point, entering specified text, returning to the previous screen, and returning to the main screen.

[0148] Before executing the next operation instruction via the mobile device, a status anomaly detection step is also included:

[0149] The current interface image is input into a multimodal large model for state determination, detecting whether there are abnormal pop-ups or interface obstruction on the current interface, and obtaining the anomaly detection results. ;

[0150] Determine the operation instructions to be executed based on the anomaly detection results:

[0151] ;

[0152] in, This indicates a recovery operation command; This indicates the next step instruction generated.

[0153] The state anomaly detection process also includes task timeout control, specifically:

[0154] Record the execution time of the current task; when the execution time exceeds a preset threshold, forcibly terminate the current task and switch to the next task.

[0155] The intelligent agent is also used to record the operation type, operation parameters, timestamp, interface change information and execution result of each operation to form an operation record; when an exception occurs, it records the exception type and the corresponding recovery operation information to form an exception record; and generates a test report based on the operation record and the exception record.

[0156] In this invention, the task completion condition is to complete all the operation steps corresponding to the preset task instruction; the termination trigger conditions include task execution timeout, detection of an unrecoverable abnormality, or receipt of an external stop instruction.

[0157] The description of the relevant parts of the agent-based automatic mobile application testing system provided in Embodiment 2 of this application can be found in the detailed description of the corresponding parts of the agent-based automatic mobile application testing method provided in Embodiment 1 of this application, and will not be repeated here.

[0158] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that the elements inherent in a process, method, article, or apparatus that includes a list of elements are included. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Additionally, portions of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.

[0159] While specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art can make other modifications or variations based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. An automatic dialing test method for mobile applications based on intelligent agents, characterized in that, Executed by the intelligent agent at the control end, the process includes the following steps: Real-time capture of the current screen image on the mobile device; The current interface image is subjected to local multimodal analysis to detect UI components in the interface and extract the bounding boxes and category labels of each component. At the same time, the text content in the interface is extracted through optical character recognition. The UI components and text content are associated and fused into a structured UI representation. The structured UI representation, preset task instructions, and historical operation memory are input into the large language model, which then generates the next operation instruction in a sequential decision-making manner. The structured UI representation is expressed as: ； in, express The structured UI representation of the timeline interface; Indicates the number of times in the current interface One UI element; Indicates the coordinate center of the bounding box; Represented as semantic category labels; Represented as text content; This is represented as an interactive property; Indicates the total number of UI elements; The large language model generates the next operation instruction using a sequence decision method, specifically as follows: The system analyzes the category labels and text content of each UI element in the current structured UI representation, identifies the functional attributes of each UI element, and forms the semantic understanding result of the current interface. The semantic understanding results of the current interface and the preset task instructions as well as UI representation of time Together, they serve as contextual input, and the large language model is used to compute candidate operations in action space A. conditional probability distribution ; Select the candidate operation with the highest conditional probability As the output of the next operation instruction, it satisfies: ； in, For the current moment UI representation; The next operation instruction is executed via the mobile device to complete the automated interactive operation; then the process returns to the step of obtaining the current interface image of the mobile device in real time, and continues to execute the next round of operations until the task completion condition is met or the termination condition is triggered.

2. The automatic dialing test method for mobile applications based on intelligent agents according to claim 1, characterized in that, Perform local multimodal analysis on the current interface image, specifically as follows: Perform lightweight preprocessing on the current interface image to obtain the preprocessed interface image; The UI component detection algorithm is used to detect the UI components in the preprocessed interface image, identify each UI component in the interface, and extract the bounding box and category label of each component. Optical character recognition is performed on the preprocessed interface image to extract the text content and corresponding text location information in the interface. Based on the bounding boxes and text position information of each component, the identified text content is associated and fused with the corresponding UI component, and each UI component is associated with its corresponding text content to form a structured UI representation.

3. The automatic dialing test method for mobile applications based on intelligent agents according to claim 1, characterized in that, The operation instructions include one or more of the following: clicking on a specified coordinate, sliding from the starting point to the ending point, entering specified text, returning to the previous screen, and returning to the main screen.

4. The automatic dialing test method for mobile applications based on intelligent agents according to claim 1, characterized in that, Before executing the next operation instruction via the mobile device, a status anomaly detection step is also included: The current interface image is input into a multimodal large model for state determination, detecting whether there are abnormal pop-ups or interface obstruction on the current interface, and obtaining the anomaly detection results. ; Determine the operation instructions to be executed based on the anomaly detection results: ； in, This indicates a recovery operation command; This indicates the generated next operation instruction.

5. The automatic dialing test method for mobile applications based on intelligent agents according to claim 4, characterized in that, The state anomaly detection step also includes task timeout control, specifically: Record the execution time of the current task; When the execution time exceeds a preset threshold, the current task will be forcibly terminated and the process will switch to the next task.

6. The automatic dialing test method for mobile applications based on intelligent agents according to claim 1, characterized in that, The method further includes: Record the operation type, operation parameters, timestamp, interface change information and execution result of each operation to form an operation record; When an anomaly occurs, the anomaly type and corresponding recovery operation information are recorded to form an anomaly log. A test report is generated based on the operation record and the anomaly record.

7. The automatic dialing test method for mobile applications based on intelligent agents according to claim 1, characterized in that, The task completion condition is to complete all the operation steps corresponding to the preset task instruction. The termination conditions include task execution timeout, detection of an unrecoverable anomaly, or receipt of an external stop command.

8. An automatic dialing test system for mobile applications based on intelligent agents, used to execute the automatic dialing test method for mobile applications based on intelligent agents as described in any one of claims 1 to 7, characterized in that, include: The control terminal is equipped with an intelligent agent, which is used to perform the following: real-time acquisition of the current interface image of the mobile terminal; The current interface image is subjected to local multimodal analysis to detect UI components in the interface and extract the bounding boxes and category labels of each component. At the same time, the text content in the interface is extracted through optical character recognition. The UI components and text content are associated and fused into a structured UI representation. The structured UI representation, preset task instructions and historical operation memory are input into a large language model, which generates the next operation instruction in a sequential decision-making manner. The next operation instruction is executed through the mobile device to complete the automated interactive operation; then the process returns to the step of obtaining the current interface image of the mobile device in real time, and continues to execute the next round of operations until the task completion condition is met or the termination condition is triggered. And at least one mobile terminal, which is communicatively connected to the control terminal, for responding to operation commands sent by the intelligent agent to perform automated interactive operations, and providing interface images for the intelligent agent to collect.

Images