Agent-based application testing method and program product

Abstract

The application discloses an application program testing method and program product based on an agent. The method comprises the following steps: receiving target description information and setting a testing termination condition, wherein the target description information comprises description information of a testing task, and the testing task is used at least for indicating an application program to be tested; acquiring an interface screenshot of the application program, performing visual element identification and text identification on the interface screenshot, and generating interface element information according to the identification result; providing the target description information, the interface screenshot and the interface element information to a multi-agent system, dynamically generating a testing operation instruction based on the target description information, the interface screenshot and the interface element information by the multi-agent system, executing the testing operation instruction and recording an instruction execution result of the testing operation instruction; and in response to the testing termination condition being met, generating a testing result of the application program based on the instruction execution result. The technical scheme improves the efficiency, flexibility and stability of the application program testing based on the agent.

Description

Technical Field

[0001] This invention relates to the field of software performance testing technology, and in particular to an application testing method and program product based on intelligent agents. Background Technology

[0002] As mobile applications become increasingly complex, application testing plays a vital role in software quality assurance. Efficiently and accurately completing testing has become a key aspect of the development process.

[0003] Existing application testing solutions typically fall into two categories: manual testing and script-based automated testing. Manual testing relies primarily on testers manually executing test procedures, while script-based automated testing relies mainly on predefined test scripts and a graphical user interface tree structure to locate and manipulate controls.

[0004] Manual testing suffers from low efficiency, repetitive work, and a high risk of missed tests. While script-based automated testing can improve efficiency, it typically relies on predefined test scripts and graphical user interface tree-structured controls. When the application's interface layout, component styles, or page structure changes, the test scripts are prone to becoming invalid, requiring re-maintenance and resulting in high maintenance costs. Summary of the Invention

[0005] This invention provides an application testing method and program product based on intelligent agents, in order to improve the flexibility and stability of testing.

[0006] According to one aspect of the present invention, an agent-based application testing method is provided, comprising: Receive target description information and set test termination conditions, wherein the target description information includes description information of the test task, and the test task is used to at least indicate the application to be tested; Obtain a screenshot of the application's interface, perform visual element recognition and text recognition on the screenshot, and generate interface element information based on the recognition results; The target description information, the interface screenshot, and the interface element information are provided to the multi-agent system. The multi-agent system dynamically generates test operation instructions based on the target description information, the interface screenshot, and the interface element information, executes the test operation instructions, and records the instruction execution results. In response to the fulfillment of the test termination condition, test results for the application are generated based on the execution result of the instruction.

[0007] According to another aspect of the present invention, an agent-based application testing apparatus is provided, the apparatus comprising: The target description information receiving module is used to receive target description information and set test termination conditions. The target description information includes description information of the test task, and the test task is used to at least indicate the application to be tested. The interface screenshot recognition module is used to acquire screenshots of the application interface, perform visual element recognition and text recognition on the screenshots, and generate interface element information based on the recognition results. The test operation instruction generation module is used to provide the target description information, the interface screenshot, and the interface element information to the multi-agent system. The multi-agent system dynamically generates test operation instructions based on the target description information, the interface screenshot, and the interface element information, executes the test operation instructions, and records the instruction execution results. The test result generation module is used to generate test results for the application based on the instruction execution result in response to the satisfaction of the test termination condition.

[0008] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the agent-based application testing method according to any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the agent-based application testing method according to any embodiment of the present invention.

[0010] According to another aspect of the present invention, embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements the agent-based application testing method as described in any of the embodiments of this disclosure.

[0011] The technical solution of this invention receives target description information and sets test termination conditions. The target description information includes a description of the test task, which at least indicates the application to be tested. This provides a clear task orientation and controllable execution boundary for the subsequent autonomous exploration of the multi-agent system, forming the fundamental premise for the efficient and secure operation of the entire testing process. It acquires screenshots of the application interface, performs visual element recognition and text recognition on the screenshots, and generates interface element information based on the recognition results. This solves the problems of script failure and high maintenance costs caused by interface changes, providing a reliable perceptual basis for subsequent multi-agent decision-making. The multi-agent system is provided with target description information, screenshots, and interface element information. Based on these information, the multi-agent system dynamically generates test operation instructions, executes them, and records the execution results. Through the multi-agent system, the test path can be dynamically generated and adaptively adjusted according to the actual state of the interface, while ensuring that each step is traceable, solving the problems of high maintenance costs and poor adaptability to interface changes in preset scripts. In response to the fulfillment of the test termination conditions, the test results of the application are generated based on the instruction execution results. In summary, the technical solution of this invention generates interface element information by performing visual element recognition and text recognition on interface screenshots, and generates test operation instructions based on a multi-agent system, thereby reducing reliance on graphical user interface tree information parsing and fixed test scripts. Compared with traditional testing solutions, this invention can adaptively complete interface element recognition and testing operations when the interface layout, component style, or page structure changes, improving testing efficiency, flexibility, and stability.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of an application testing method based on an intelligent agent, as described in an embodiment of the present invention. Figure 2 This is an architecture diagram of agent-based application testing in an embodiment of the present invention; Figure 3This is a collaborative flowchart of agent-based application testing in an embodiment of the present invention; Figure 4 This is a flowchart of an agent-based application testing method in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an application testing device based on an intelligent agent according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0016] It should be noted that the terms "target," "original," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0020] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0021] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0022] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0023] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0024] Figure 1 This invention provides a flowchart of an agent-based application testing method. This embodiment is applicable to applications and can be executed by an agent-based application testing device. This device can be implemented in hardware and / or software, optionally through an electronic device such as a mobile terminal, PC, or server. Figure 1 As shown, the method may specifically include: S110. Receive target description information and set test termination conditions. The target description information includes description information of the test task, which is used to at least indicate the application to be tested.

[0025] In this embodiment of the invention, the target description information can be specifically understood as the description information input to implement the test task. Specifically, the target description information includes a description of the test task. The target description information is not a traditional structured test script or code instruction, but rather a semantic description. The target description information is used to transform the test intent into information recognizable by a multi-agent system. The test task can be specifically understood as the task extracted and parsed from the target description information. The test task is used to indicate the application to be tested and the test objectives to be completed. The test termination condition can be specifically understood as a preset condition used to control when the test process ends. The test termination condition is used to prevent the test task from executing indefinitely and to trigger the test termination process when a specific condition is met. For example, a preset number of tests can be set; when the number of tests exceeds the preset number, the test terminates. The test termination condition can be preset according to the actual application scenario. Setting the test termination condition prevents test runaway and avoids the problem of insufficient testing due to a fixed number of steps.

[0026] Specifically, it receives target description information input by the user and pre-sets test termination conditions, providing clear test tasks for subsequent steps, while ensuring the flexibility and reliability of the test process through test termination conditions.

[0027] S120. Obtain a screenshot of the application interface, perform visual element recognition and text recognition on the screenshot, and generate interface element information based on the recognition results.

[0028] In this embodiment of the invention, a screenshot can be specifically understood as a screenshot of the current interface of the application. A screenshot is a pixel-level representation of the current interface state. The screenshot constitutes the basic input data for subsequent visual element recognition and text recognition operations. Interface element information can be specifically understood as the information generated by fusing visual element recognition and text recognition of the screenshot. Interface element information includes at least the location information of interactive components, the type of interactive components, and associated text content. The interface element set is used to provide a unified description of the interactive components and their associated semantic information in the current interface, providing a reliable perceptual basis for agent decision-making.

[0029] Specifically, the system captures real-time image data (screenshots) of the application under test via the device control interface. Visual element and text recognition are performed on these screenshots, and interface element information is generated based on the recognition results, thus resolving the issue in testing scenarios where graphical user interface tree information is unavailable.

[0030] Optionally, the step of performing visual element recognition and text recognition on the screenshot and generating interface element information based on the recognition results includes: performing visual element recognition on the screenshot to obtain first position information of interactive components in the screenshot, and performing text recognition on the screenshot to obtain text content in the screenshot and second position information corresponding to the text content; associating the text content with the corresponding interactive component based on the overlap relationship between the first position information and the second position information, and combining the component type of the interactive component, the first position information, and the text content associated with the interactive component to generate interface element information.

[0031] In this embodiment of the invention, interactive components can be specifically understood as interactive components in the interface screenshot. Interactive components can be operated through touch, click, swipe, or input to trigger application responses or interface state changes. Interactive components include multiple component types. Interactive components include, but are not limited to, buttons, image buttons, text buttons, input boxes, text input boxes, clickable list items, list entries, sliders, switch controls, checkboxes, radio buttons, dropdown selectors, swivelable areas, scroll views, clickable cards, and operation buttons in pop-ups. The first location information can be specifically understood as the location information of interactive components obtained after visual element recognition of the interface screenshot. The first location information is used to identify the specific location of each interactive component in the interface. The first location information includes at least the bounding box coordinates of the interactive component. The first location information can obtain the location data of interactive components in the interface based on a purely visual method, providing coordinate basis for subsequent instruction generation and actual operation. The second location information can be specifically understood as the location information of text content in the interface screenshot. The second location information is used to identify the specific location of each text block in the interface. The second location information includes at least the bounding box coordinates of the text content. The second location information is matched with the first location information through spatial overlap to achieve semantic association between text content and interactive components, providing a basis for text attribution for the generation of interface element information.

[0032] Specifically, visual element recognition is performed on the screenshot to obtain the first position information of interactive components in the screenshot. Text recognition is performed on the screenshot to obtain the text content and its corresponding second position information. The text recognition method can be OCR (Optical Character Recognition). The overlap relationship between the first and second position information is calculated. For example, the degree of overlap is measured by calculating the intersection-union ratio (IUR) of the bounding boxes of the interactive components and the text blocks. According to a preset overlap threshold, if the IUR is greater than or equal to the overlap threshold, it is determined that the text content and the interactive component are associated; if the IUR is less than the overlap threshold, it is determined that the text content and the interactive component are not associated. Based on the overlap relationship between the first and second position information, the text content is associated with the corresponding interactive component, that is, the text content is assigned to the corresponding interactive component. The component type, first position information, and associated text content of all interactive components are combined into structured interface element information for subsequent use by the multi-agent system.

[0033] Optionally, the method further includes: when graphical user interface tree information is available, obtaining the graphical user interface tree information of the application, and correcting the first position information of the interactive component and the component type according to the graphical user interface tree information.

[0034] In this embodiment of the invention, the graphical user interface tree information can be specifically understood as information describing the control hierarchy structure of the current application interface. The graphical user interface tree information describes the nesting relationships, type attributes, and coordinate positions of all components in the interface in a tree structure. The graphical user interface tree information is used to correct and calibrate the first position information and the component types.

[0035] Specifically, in scenarios where graphical user interface (GUI) tree information is available, the GUI tree information of the application is obtained. Based on the GUI tree information, the first position information and component type of interactive components are corrected to improve the accuracy of visual element recognition.

[0036] S130. Provide the target description information, the interface screenshot, and the interface element information to the multi-agent system. The multi-agent system dynamically generates test operation instructions based on the target description information, the interface screenshot, and the interface element information, executes the test operation instructions, and records the instruction execution results.

[0037] In this embodiment of the invention, the multi-agent system can be specifically understood as a system composed of multiple intelligent agent units with different functions and cooperating with each other, arranged according to a preset calling order and coordination mechanism. The multi-agent system is used to dynamically generate test operation instructions based on target description information, interface screenshots, and interface element information. The core characteristic of the multi-agent system lies in the decoupling of functions among the agents and their collaborative decision-making. Its decision-making process does not rely on preset scripts but dynamically generates operation instructions based on the current interface state and historical operation records, thereby achieving autonomous exploratory testing of the application. The test operation instructions can be specifically understood as the operation instructions output by the multi-agent system. These test operation instructions enable the entire testing process to run adaptively without preset scripts, fundamentally solving the technical problems of high script maintenance costs and poor cross-version compatibility in traditional automated testing. The instruction execution result is specifically understood as a description of the result after the actual execution of the test operation instruction. The instruction execution result includes at least information on whether the operation corresponding to the test operation instruction was successfully executed, information on whether the application interface changed, and exception information.

[0038] Specifically, the system provides target description information, screenshots, and interface element information to the multi-agent system. Before inputting the screenshots into the system, image encoding is performed to obtain the image code. Based on the target description information, screenshots, and interface element information, the multi-agent system dynamically generates test operation instructions. After generating the instructions, they are executed, and the execution results are recorded. This multi-agent system achieves intelligent transformation from user intent to executable instructions, enabling automated testing to autonomously complete the testing process in complex interaction scenarios without the need for pre-set scripts, significantly reducing reliance on manually written and maintained test scripts.

[0039] Optionally, the multi-agent system includes an interface analysis agent, an operation analysis agent, a result analysis agent, a task planning agent, and a history memory agent.

[0040] In this embodiment of the invention, the interface analysis agent can be specifically understood as an agent in a multi-agent system responsible for visual understanding and semantic parsing of the current application interface. The interface analysis agent transforms interface screenshots and interface element information into scene understanding results that can be used for subsequent operation decisions. The interface analysis agent is used to analyze the interface of the application to be tested. Through the interface analysis agent, pixel-level visual data can be abstracted into semantic-level scene descriptions, providing a clear contextual basis for the generation of subsequent operation instructions. The operation analysis agent can be specifically understood as an agent in a multi-agent system responsible for generating specific test operation instructions. The operation analysis agent achieves intelligent mapping from interface perception to operation execution, enabling automated testing to flexibly generate operation instructions based on the current interface state without relying on preset scripts, fundamentally solving the problem of script failure caused by changes in interface layout in traditional solutions. The result analysis agent can be specifically understood as an agent in a multi-agent system responsible for verifying the effect of executed operations. The result analysis agent outputs operation judgment results based on the interface screenshots before and after the operation. The result analysis agent is responsible for judging whether the operation meets expectations. The results analysis agent enables real-time quality verification for each round of testing, providing crucial feedback signals for the autonomous testing loop. The task planning agent, specifically, can be understood as the agent in the multi-agent system responsible for updating the task completion status of the test tasks. The task planning agent obtains the task completion status based on the target description information and operation judgment results. Through the task planning agent, the testing process is autonomously planned and dynamically adjusted, giving the entire test execution task-oriented adaptive capabilities. The history memory agent, specifically, can be understood as the agent in the multi-agent system responsible for persistently storing key information from the testing process. The history memory agent assists the task planning agent in judging the effectiveness of strategies and provides experience disambiguation basis for the operation analysis agent. Through the history memory agent, repeated invalid operations are avoided, enabling the testing process to accumulate experience and continuously optimize.

[0041] Optionally, the step of dynamically generating test operation instructions by the multi-agent system based on the target description information, the interface screenshot, and the interface element information includes: calling the interface analysis agent, inputting the interface screenshot and the interface element information into the interface analysis agent, and obtaining the interface scene understanding result output by the interface analysis agent; calling the operation analysis agent, inputting the interface scene understanding result, the target description information, and historical operation records into the operation analysis agent, and obtaining the test operation instructions output by the operation analysis agent; calling the result analysis agent, generating an operation judgment result based on the interface screenshots before and after the operation, the operation judgment result being used to indicate whether the operation performed for the test operation instructions is qualified; calling the task planning agent, inputting the target description information and the operation judgment result into the task planning agent, and obtaining the task completion status output by the task planning agent; and calling the history memory agent, recording the operation information of the current round and storing it in the historical operation record for subsequent rounds to generate the test operation instructions.

[0042] In this embodiment of the invention, the interface scene understanding result can be specifically understood as the result used to describe the functional attributes and operational context of the current interface. The interface scene understanding result includes, but is not limited to, the scene type of the current interface, the description of the core functions of the interface, and the semantic annotation of interactive elements. As one of the core inputs of the operation analysis agent, the interface scene understanding result provides a semantic-level environmental cognitive foundation for the dynamic generation of subsequent operation instructions. The historical operation record can be specifically understood as a record used to record all operation information executed from the start of the test to the current round. The historical operation record is retrieved by the operation analysis agent in subsequent rounds, thereby assisting the operation analysis agent in avoiding repetitive and invalid operations. Through the historical operation record, the test process possesses experience-driven dynamic optimization capabilities, and the effectiveness of operation decisions gradually improves as the test execution progresses. The operation judgment result can be specifically understood as the result indicating whether the operation in the current round has achieved the expected effect. The operation judgment result includes, but is not limited to, a status indicator indicating whether the operation is qualified. The operation judgment result is transmitted to the task planning agent and the historical memory agent respectively, so that the task planning agent can update the task completion status accordingly, and the historical memory agent can record the operation of this round as experience data, thereby realizing real-time quality verification of each round of operation and feedback optimization of subsequent decisions. Task completion status can be understood as an indication of the overall progress of the current test task. Task completion status includes, but is not limited to, completed, partially completed, and incomplete. As the core basis for the task planning agent to coordinate the work rhythm of various agents, the task completion status enables the entire testing process to have task-driven autonomous scheduling capabilities.

[0043] Specifically, the process involves several steps: First, the UI analysis agent is invoked, inputting screenshots and element information to obtain the UI scene understanding results. Second, the operation analysis agent is invoked, inputting the UI scene understanding results, target descriptions, and historical operation records to obtain test operation instructions. Third, the result analysis agent is invoked, generating operation judgment results based on screenshots before and after the operation. Fourth, the task planning agent is invoked, inputting target descriptions and operation judgment results to obtain the task completion status. Fifth, the history memory agent is invoked, recording the operation information of the current round and storing it in the history record for generating test operation instructions in subsequent rounds. These agents are invoked in a preset order—UI analysis, operation analysis, result analysis, task planning, and history memory—and the subsequent test behavior is dynamically adjusted based on historical operation records and the current test status, thereby achieving autonomous, exploratory, and automated testing of the application under test.

[0044] Optionally, the method further includes: after executing the test operation instruction, calling the result analysis agent to determine whether the current operation meets expectations based on the screenshots of the interface before and after the operation is executed; if it does not meet expectations, recording the abnormal information.

[0045] In this embodiment of the invention, the anomaly information can be specifically understood as the information generated by the structured recording when the result analysis agent determines that the operation does not meet expectations. The anomaly information includes, but is not limited to, anomaly type, anomaly description, anomaly occurrence timestamp, the operation instruction that caused the anomaly, and screenshot paths of the interface before and after the operation. The anomaly information is used to provide a basis for decision-making regarding the strategy adjustment of the task planning agent.

[0046] Specifically, a screenshot of the interface before the test operation command is executed is obtained as the first screenshot. After the test operation command is completed, a screenshot of the current interface is obtained again as the second screenshot. Both the first and second screenshots are input into the result analysis agent, which analyzes the screenshots before and after the operation to determine if the current operation meets expectations. If the result analysis agent determines that the current operation meets expectations, it outputs an operation judgment result containing a pass / fail flag. If the result analysis agent determines that the operation does not meet expectations, it outputs an operation judgment result with a fail / unqualified flag and records the anomaly information. This process achieves real-time quality verification and accurate anomaly capture for each test operation command, ensuring that any operation deviating from expectations is fully recorded, providing crucial feedback signals for the autonomous decision-making closed loop of the testing process.

[0047] Optionally, the target description information includes user description information and system description information. The user description information is a test target and auxiliary prompt information described in natural language input by the user, used to indicate the test target that the application to be tested needs to complete. The system description information is preset responsibility specification information, used to limit the data processing logic of the interface analysis agent, the operation analysis agent, the result analysis agent, the task planning agent, and the history memory agent.

[0048] In this embodiment of the invention, user description information can be specifically understood as information including test objectives described in natural language by the user and auxiliary prompts. User description information is used to convey the test objectives that the application under test needs to complete to the multi-agent system. Auxiliary prompts are used to supplement the test objectives. Auxiliary prompts can provide constraints such as preconditions, test data, business rules, or exploration preferences. System description information can be specifically understood as information used to define the behavioral boundaries of each agent. System description information includes, but is not limited to, the responsibility specifications of each agent and output format constraints. System description information and user description information constitute a unity of rigid specifications and flexible tasks, jointly ensuring the stability of multi-agent collaborative decision-making and the parsability of the output results.

[0049] Optionally, executing the test operation instruction and recording the instruction execution result includes: parsing the test operation instruction, extracting the operation type and corresponding operation parameters of the test operation instruction, assembling the operation parameters into a device control command according to the operation type; executing the device control command on the test device, obtaining a screenshot of the interface after executing the device control command, and generating the instruction execution result based on the screenshot of the interface before and after executing the command. The instruction execution result includes information on whether the operation corresponding to the test operation instruction was successfully executed, information on whether the interface of the application changed, and exception information.

[0050] In this embodiment of the invention, the operation type can be specifically understood as the type of action to be performed on the test device in the test operation instruction. The operation type includes at least one of the following: clicking a specified coordinate, double-clicking a specified coordinate, long-pressing a specified coordinate, a sliding operation implemented using start and end coordinates, inputting specified text content in an input box, and simulating system key operations. By extracting the operation type field from the test operation instruction, the specific interaction category to be executed for the current instruction can be accurately specified from a variety of predefined interaction methods. Specifically, clicking a specified coordinate, double-clicking a specified coordinate, and long-pressing a specified coordinate are mainly completed by specifying interface coordinates; the sliding operation is implemented using start and end coordinates; the operation of inputting specified text content in an input box is used to input specified text content in the input box; and the simulated system key operation is used to execute operations such as the back button. The operation parameter can be specifically understood as the parameter in the test operation instruction used to carry the specific data required to execute the operation. The operation parameter provides the necessary execution parameters for the assembly device control command. The operation parameter is related to the operation type. For example, when the operation type is clicking a specified coordinate, double-clicking a specified coordinate, or long-pressing a specified coordinate, the operation parameter at least includes the coordinate value of the specified coordinate. When the operation type is a sliding operation, the operation parameters must include at least the start and end coordinates. When the operation type is entering specified text content in an input box, the operation parameters must include at least the specified text content; when the operation type is simulating system key operations, the operation parameters must include at least the key type identifier. By extracting the operation parameter fields from the test operation instructions, a precise parameter mapping from the test operation instructions output by the agent to the device's executable control commands is achieved. Specifically, the device control command can be understood as a control command generated by assembling the operation type and operation parameters. The device control command can be directly recognized and executed by the device control interface of the test device. The device control command enables precise simulation and control of interactive behaviors such as screen touch, text input, and system key presses on the test device. Through the conversion of device control commands, effective decoupling between high-level decision-making and low-level execution is achieved. Specifically, the test device can be the device running the application to be tested. The test device is the actual execution carrier of the test operation instructions, used to simulate user interaction behavior and verify the functional correctness of the application in a real operating environment. For example, the test device can be a real mobile phone or tablet device running the Android operating system, a real mobile phone or tablet device running the iOS operating system, etc.

[0051] Specifically, the process involves acquiring the test equipment and initializing its control interface, then connecting to the test equipment via this interface. After connection, the system extracts the operation type and corresponding parameters from the test operation instructions through the established communication channel. Based on the operation type, the parameters are assembled into a device control command according to the test equipment's control interface protocol. The device control command is then executed on the test equipment, and a screenshot of the interface after execution is captured. The command execution result is generated based on the screenshots before and after execution and stored. This result includes information on whether the operation corresponding to the test operation instruction was successfully executed, whether the application interface changed, and any exception information. By converting the test operation instructions output by the agent into device control commands, a standardized conversion between high-level decision-making and low-level execution is achieved. This ensures end-to-end traceability of each instruction from generation to execution, guaranteeing accurate implementation and closed-loop feedback of multi-agent decision-making results on the physical device.

[0052] Optionally, before executing the device control command on the test device, the method further includes: detecting currently available test devices and determining the device to be tested.

[0053] Specifically, since the test environment may contain multiple connected devices, a single connected device, or no connected devices, it is necessary to detect the currently available devices and determine the device to be tested before executing device control commands on the test device. Detecting the currently available test devices and determining the device to be tested avoids compatibility issues caused by fixed device connection information.

[0054] S140. In response to the satisfaction of the test termination condition, generate the test result of the application based on the instruction execution result.

[0055] Specifically, when the test termination condition is met, the test results of the application are generated based on the instruction execution results, thus realizing a complete closed loop from test execution to the generation of test result output of the application.

[0056] Optionally, the step of generating the test result of the application based on the instruction execution result in response to meeting the test termination condition includes: saving the stored test information as structured data in response to meeting the test termination condition, generating a test report display page according to a preset template and the structured data, wherein the display information of the test report display page includes at least one of the following: statistical information on the total number of test steps, the number of exceptions, the number of pages covered, and the total test time; a page state transition diagram showing the page jump relationship; and screenshots before and after each operation, a visual processing result diagram, and an operation diagram corresponding to each operation.

[0057] In this embodiment of the invention, test information can be specifically understood as information generated throughout the entire testing process. Test information serves as the basic data source for generating the test report display page after the test. Test information includes, but is not limited to, the content of test operation instructions, operation types and parameters, screenshots of the interface before and after operation execution, operation judgment results, exception information records, and timestamp information corresponding to each step of the operation. Structured data can be specifically understood as data with a standardized format generated after the test information is uniformly organized and encoded. Data extracted and mapped from the test information into data with fixed field definitions is called structured data. Structured data is used as the standard input for the test report generation module, ensuring the reproducible rendering of test results and consistency with subsequent data analysis. The preset template can be specifically understood as a template used to standardize the structure and style of the test report display page. The preset template is pre-set. As the skeleton framework for generating the test report display page, the preset template defines the layout structure, data filling positions, chart types, and visual styles of the test report display page. By setting the preset template, the format of the test report display page is defined, making the test report generation process completely automated, eliminating the need for manual data processing or layout. The preset templates can be customized to meet the diverse requirements of different projects for test report formats. The test report display page can be understood as a page that displays the test report generated based on the preset template and structured data. This page showcases the testing process and results. By eliminating the need for manually organizing test data or writing report documents, the time cost of test result analysis and problem localization is significantly reduced, while providing a standardized data presentation format for the quantitative evaluation of test quality.

[0058] The test report display page includes at least one of the following: statistics on the total number of test steps, the number of anomalies, the number of pages covered, and the total test time; a page state transition diagram showing the page navigation relationships; and screenshots before and after each operation, a visual processing result diagram, and an operation diagram. The total number of test steps can be understood as the total number of valid operation rounds executed and recorded during the test. The total number of test steps is the sum of the number of times device control commands are executed from the start of the test until the test termination condition is met. The number of anomalies can be understood as the total number of anomaly messages recorded. The result analysis agent judges whether the current operation meets expectations based on the interface screenshots before and after the operation execution. If it does not meet expectations, the number of times anomaly messages are recorded is the number of anomalies. The number of anomalies is equivalent to the frequency of unexpected behavior of the application during the test. The number of pages covered can be understood as the total number of unique pages accessed and recorded during the test. The number of unique application pages first entered or jumped to during the test termination condition is the number of pages covered. The number of pages covered is used to measure the breadth of exploration and the sufficiency of interface coverage in the test. The total test time can be understood as the time elapsed from the start of the test to the fulfillment of the test termination condition. The total test time is the total duration from receiving the target description information to the fulfillment of the test termination condition. The total test time is used to evaluate test execution efficiency. For example, the total test time can be displayed in the format of "X minutes X seconds" or "X seconds". The page state transition diagram can be understood as a transition diagram rendered based on the jump relationships between pages recorded during the test. The page state transition diagram is displayed on the test report page. The page state transition diagram is used to visually demonstrate the jump relationships between different pages during the test. The visual processing result diagram can be understood as image data generated by marking the detected interactive component bounding boxes, component type labels, and associated text on the original screenshot after visual element recognition of the interface screenshot. The visual processing diagram is used to intuitively demonstrate the perception and recognition results of interface elements. The operation diagram can be understood as an image that visually marks the actual operation location and operation type on the corresponding interface screenshot. The operation diagram is used to intuitively demonstrate the operation in this operation. The operation diagrams help to eliminate the need to guess the operation intent and actual landing point when analyzing test paths or locating the cause of anomalies, significantly improving the efficiency and accuracy of test result backtracking.

[0059] Specifically, when the test termination conditions are met, the stored test information is read. All test information is saved as structured data. A test report display page is generated based on a preset template and the structured data. The test report display page includes at least one of the following: statistics on the total number of test steps, the number of exceptions, the number of pages covered, and the total test time; a page state transition diagram showing the page navigation relationships; and screenshots before and after each operation, visual processing result diagrams, and operation diagrams. After the test is completed, the system reads all test information, assembles it into structured data, and then generates a test report display page based on the preset template for testers to review and analyze.

[0060] Optionally, the test termination conditions include: the task completion status output by the task planning agent is "completed", or the current number of operations reaches a preset maximum number of operations.

[0061] Specifically, the test termination condition can be either that the task completion status output by the task planning agent is "completed," or that the current number of operations has reached a preset limit. If the task completion status output by the task planning agent is "completed," the test result of the application is generated based on the instruction execution result; or, if the current number of operations has reached the preset limit, the test result of the application is generated based on the instruction execution result. Setting test termination conditions achieves a synergy between the agent's autonomous decision-making and engineering safety safeguards, balancing test efficiency and system reliability.

[0062] Optionally, the method further includes: generating a test directory based on the test timestamp, storing screenshots of the interface before and after the execution of the test operation instruction, the test operation instruction, timestamp information, exception information, and test status in the test directory; recording interface jump information when the application interface jumps, the interface jump information including at least the interface identifiers before and after the jump; constructing an interface jump path graph based on the interface identifiers before and after the jump, with each interface as a node and the jump relationship between interfaces as an edge, and storing it in the test directory.

[0063] In this embodiment of the invention, the test timestamp can be specifically understood as the time sequence identifier information of the test start time recorded when each test is started. The test timestamp is used to identify a test and generate test targets based on the test timestamp. The test timestamp provides a clear time for test report generation and historical data tracing. The test directory can be specifically understood as a directory created based on the test timestamp at the start of each test, dedicated to the storage path of this test. The test directory is used to centrally store the test information generated during the execution of this test. Under this directory, subdirectories are further divided according to data type for itemized storage, including a subdirectory for storing screenshots of the interface before and after the execution of test operation instructions, a subdirectory for storing operation records, a subdirectory for storing timestamp information, a subdirectory for storing exception information, and a subdirectory for storing test status. The test directory realizes the orderly storage of test information, providing a clear storage structure foundation for the generation of test reports. The interface jump information can be specifically understood as the information recorded when the application interface jump is detected during the test execution. The interface jump information includes at least the interface identifier before the jump and the interface identifier after the jump. The interface identifier before the jump and the interface identifier after the jump are used to distinguish different interface nodes. A UI navigation path graph can be understood as a topological graph structure built based on recorded UI navigation information, describing the navigation relationships between pages accessed during testing. The UI navigation path graph uses UIs as nodes and the navigation relationships between UIs as directed edges, visually displaying all page paths traversed from the starting UI through each round of operations, as well as the reachability relationships between pages. In the UI navigation path graph, each node represents a UI, and each edge represents a UI navigation event. The UI navigation path graph serves as the raw data input for the test report generation module to render the page state transition graph, and as the data basis for subsequent path coverage analysis and uncovered page identification.

[0064] Specifically, a test directory is generated based on the test timestamps, storing screenshots of the interface before and after the execution of test operation commands, the test operation commands themselves, timestamp information, exception information, and test status. When the application interface transitions, the transition information is recorded. Based on the interface identifiers before and after the transition in the transition information, an interface transition path graph is constructed, with each interface as a node and the transition relationships between interfaces as edges, and this graph is stored in the test directory. Storing the graph in the test directory ensures process traceability, and constructing the interface transition path graph ensures coverage assessability.

[0065] Optionally, the interface jump information also includes operation type and operation timestamp. The operation type is used to record the operation that triggers the jump to trace the test execution process, and the operation timestamp is used to statistically analyze the time consumption distribution of each step of the test.

[0066] Specifically, the interface transition information also includes the operation type and the operation timestamp. The operation type records the type of operation that triggered the current interface transition. Recording the operation type allows for identification of the operation that triggered each transition edge during subsequent test path analysis. The operation timestamp records the exact moment the interface transition occurred. By calculating the difference between the timestamps before and after the operation, the time distribution of each step can be determined, providing a quantitative basis for identifying performance bottlenecks and locating slow operations during testing.

[0067] The technical solution of this invention, by receiving target description information and setting test termination conditions, whereby the target description information includes a description of the test task, which at least indicates the application to be tested, provides a clear task orientation and controllable execution boundary for the subsequent autonomous exploration of the multi-agent system. This is a fundamental prerequisite for the efficient and secure operation of the entire testing process. The solution involves acquiring screenshots of the application's interface, performing visual element and text recognition on the screenshots, and generating interface element information based on the recognition results. This solves the problems of script failure and high maintenance costs caused by interface changes, providing a reliable perceptual basis for subsequent multi-agent decision-making. The multi-agent system is provided with target description information, screenshots, and interface element information. Based on these information, the multi-agent system dynamically generates test operation instructions, executes them, and records the execution results. Through the multi-agent system, the test path can be dynamically generated and adaptively adjusted according to the actual state of the interface, while ensuring that each step is traceable, solving the problems of high maintenance costs and poor adaptability to interface changes in preset scripts. In response to the fulfillment of the test termination conditions, the test results of the application are generated based on the instruction execution results. In summary, the technical solution of this invention generates interface element information by performing visual element recognition and text recognition on interface screenshots, and generates test operation instructions based on a multi-agent system, thereby reducing reliance on graphical user interface tree information parsing and fixed test scripts. Compared with traditional testing solutions, this invention can adaptively complete interface element recognition and testing operations when the interface layout, component style, or page structure changes, improving testing efficiency, flexibility, and stability.

[0068] Based on the above embodiments, an optional example is provided, which can be used to test applications. The architecture diagram for agent-based application testing is as follows: Figure 2As shown. The architecture of application testing based on intelligent agents is divided into several modules. (1) State perception module, which is used to obtain screenshots of the application interface, perform visual element recognition and text recognition on the screenshots, and generate interface element information based on the recognition results. (2) Intelligent decision module, which is used to generate test operation instructions based on the current interface state, historical operation records, test tasks and interface semantic information output by the state perception module. This module includes a multi-agent system, which includes an interface analysis agent, an operation analysis agent, a result analysis agent, a task planning agent and a historical memory agent. (3) Device interaction module, which is used to receive test operation instructions generated by the intelligent decision module and execute the test operation instructions. The new interface state data is transmitted to the state perception module, and the new interface state data includes the screenshots of the interface after the operation. (4) Test record module, which is used to persistently store test information during the test process, including screenshots of the interface before and after the operation, the processed interface information, test operation instructions, timestamp information and exception information, etc. At the same time, the interface jump information is recorded to form an interface jump path diagram for subsequent test path analysis and test coverage calculation. The external environment and tools include testing equipment, a large language model interface, and a database. The testing equipment is used to execute test operation commands and provide interface screenshot data; the large language model interface is used to connect to external large language models, providing model reasoning capabilities for the intelligent decision-making module; and the database is used to persistently store structured data and test records generated during the testing process.

[0069] Figure 3 This is a collaborative flowchart for agent-based application testing. The intelligent decision-making module acts as the process scheduling center, completing the testing process by calling the state awareness module, device interaction module, and test recording module. Specifically, the state awareness module is responsible for acquiring screenshots of the application interface, component identification information, and runtime status information; the intelligent decision-making module generates test operation instructions; the device interaction module performs control operations on the corresponding test devices according to the test operation instructions and provides feedback on the instruction execution results; and the test recording module is responsible for persistently storing test information during the testing process. An iterative testing mechanism is adopted, dynamically analyzing the current task completion status in each round of testing. The testing process terminates when the task planning agent outputs a completed task status or when the current number of operations reaches a preset limit.

[0070] Figure 4This is a flowchart of an application testing process based on intelligent agents. At the start of the test, the user first inputs the target description information and sets the test termination condition. For example, the test termination condition could be the maximum number of tests. Then, the currently available test devices are detected, and the device to be tested is determined. During the test, a screenshot of the application interface is first acquired, and visual element recognition and text recognition are performed. Interface element information is generated based on the recognition results. Subsequently, the multi-agent system generates test operation instructions based on the interface screenshot, target description information, and interface element information. The device interaction module executes the corresponding operation on the test device. After each round of operation, a new interface screenshot is acquired, and the result analysis agent generates an operation judgment result to determine whether the current operation meets expectations. Simultaneously, the task planning agent outputs the task completion status and dynamically adjusts the subsequent test process based on the current task completion status, while the history memory agent continuously records the operation information of the current round. When the number of tests reaches the preset maximum number of tests, or the task planning agent outputs a task completion status of "completed," the test process ends, the test information is saved, and a test report display page is generated.

[0071] Figure 5 This is a schematic diagram of a device for testing applications based on intelligent agents, provided in an embodiment of the present invention. This embodiment is applicable to applications undergoing testing. The device can be implemented using software and / or hardware, and can be integrated into any device that provides application testing functionality based on intelligent agents, such as… Figure 5 As shown, the device for testing applications based on intelligent agents specifically includes: a target description information receiving module 210, an interface screenshot recognition module 220, a test operation instruction generation module 230, and a test result generation module 240.

[0072] The target description information receiving module 210 is used to receive target description information and set test termination conditions. The target description information includes description information of the test task, and the test task is used to at least indicate the application to be tested. The interface screenshot recognition module 220 is used to acquire screenshots of the application interface, perform visual element recognition and text recognition on the screenshots, and generate interface element information based on the recognition results. The test operation instruction generation module 230 is used to provide the target description information, the interface screenshot and the interface element information to the multi-agent system, and the multi-agent system dynamically generates test operation instructions based on the target description information, the interface screenshot and the interface element information, executes the test operation instructions and records the instruction execution results of the test operation instructions; The test result generation module 240 is used to generate test results for the application based on the instruction execution result in response to the satisfaction of the test termination condition.

[0073] The technical solution of this invention receives target description information and sets test termination conditions. The target description information includes a description of the test task, which at least indicates the application to be tested. This provides a clear task orientation and controllable execution boundary for the subsequent autonomous exploration of the multi-agent system, forming the fundamental premise for the efficient and secure operation of the entire testing process. It acquires screenshots of the application interface, performs visual element recognition and text recognition on the screenshots, and generates interface element information based on the recognition results. This solves the problems of script failure and high maintenance costs caused by interface changes, providing a reliable perceptual basis for subsequent multi-agent decision-making. The multi-agent system is provided with target description information, screenshots, and interface element information. Based on these information, the multi-agent system dynamically generates test operation instructions, executes them, and records the execution results. Through the multi-agent system, the test path can be dynamically generated and adaptively adjusted according to the actual state of the interface, while ensuring that each step is traceable, solving the problems of high maintenance costs and poor adaptability to interface changes in preset scripts. In response to the fulfillment of the test termination conditions, the test results of the application are generated based on the instruction execution results. In summary, the technical solution of this invention generates interface element information by performing visual element recognition and text recognition on interface screenshots, and generates test operation instructions based on a multi-agent system, thereby reducing reliance on graphical user interface tree information parsing and fixed test scripts. Compared with traditional testing solutions, this invention can adaptively complete interface element recognition and testing operations when the interface layout, component style, or page structure changes, improving testing efficiency, flexibility, and stability.

[0074] Based on the above embodiments, optionally, the interface screenshot recognition module 220 is configured to: perform visual element recognition on the interface screenshot to obtain first position information of interactive components in the interface screenshot; and perform text recognition on the interface screenshot to obtain text content in the interface screenshot and second position information corresponding to the text content; and associate the text content with the corresponding interactive component according to the overlap relationship between the first position information and the second position information, and combine the component type of the interactive component, the first position information, and the text content associated with the interactive component to generate interface element information.

[0075] Based on the above embodiments, optionally, the device further includes a graphical user interface tree information acquisition module, used to: acquire the graphical user interface tree information of the application when the graphical user interface tree information is available, and modify the first position information of the interactive component and the component type according to the graphical user interface tree information.

[0076] Based on the above embodiments, optionally, the multi-agent system includes an interface analysis agent, an operation analysis agent, a result analysis agent, a task planning agent, and a history memory agent.

[0077] Based on the above embodiments, optionally, the test operation instruction generation module 230 is configured to: invoke the interface analysis agent, input the interface screenshot and the interface element information into the interface analysis agent, and obtain the interface scene understanding result output by the interface analysis agent; invoke the operation analysis agent, input the interface scene understanding result, the target description information, and historical operation records into the operation analysis agent, and obtain the test operation instruction output by the operation analysis agent; invoke the result analysis agent, generate an operation judgment result based on the interface screenshots before and after the operation, the operation judgment result being used to indicate whether the operation performed for the test operation instruction is qualified; invoke the task planning agent, input the target description information and the operation judgment result into the task planning agent, and obtain the task completion status output by the task planning agent; invoke the history memory agent, record the operation information of the current round and store it in the history operation record for subsequent rounds to generate the test operation instruction.

[0078] Optionally, based on the above embodiments, the device further includes an operation judgment module, used to: after executing the test operation instruction, call the result analysis agent to judge whether the current operation meets expectations based on the interface screenshots before and after the operation is executed; if it does not meet expectations, record the abnormal information.

[0079] Based on the above embodiments, optionally, the target description information includes user description information and system description information. The user description information is a test target and auxiliary prompt information described in natural language input by the user, used to indicate the test target that the application to be tested needs to complete. The system description information is preset responsibility specification information, used to limit the data processing logic of the interface analysis agent, the operation analysis agent, the result analysis agent, the task planning agent, and the history memory agent.

[0080] Based on the above embodiments, optionally, the test operation instruction generation module 230 is used to: parse the test operation instruction, extract the operation type and corresponding operation parameters of the test operation instruction, assemble the operation parameters into a device control command according to the operation type; execute the device control command on the test device, obtain a screenshot of the interface after executing the device control command, and generate the instruction execution result based on the screenshot of the interface before and after executing the command. The instruction execution result includes information on whether the operation corresponding to the test operation instruction was successfully executed, information on whether the interface of the application changed, and error information.

[0081] Based on the above embodiments, optionally, the test result generation module 240 is used to: in response to the satisfaction of the test termination condition, save the stored test information as structured data, generate a test report display page according to a preset template and the structured data, wherein the display information of the test report display page includes at least one of the following: statistical information on the total number of test steps, the number of exceptions, the number of covered pages and the total test time, a page state transition diagram showing the page jump relationship, and screenshots before and after each operation, a visual processing result diagram and an operation diagram corresponding to each operation.

[0082] Based on the above embodiments, optionally, the device further includes a test directory storage module, configured to: generate a test directory based on a test timestamp, and store screenshots of the interface before and after the execution of the test operation instruction, the test operation instruction, timestamp information, exception information, and test status in the test directory; record interface jump information when the application interface jumps, the interface jump information including at least the interface identifiers before and after the jump; construct an interface jump path graph based on the interface identifiers before and after the jump, with each interface as a node and the jump relationship between interfaces as an edge, and store it in the test directory.

[0083] The above-described products can perform the methods provided in any embodiment of the present invention, and have the corresponding functional modules and beneficial effects for performing the methods.

[0084] Figure 6 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0085] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0086] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0087] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as agent-based application testing methods.

[0088] In some embodiments, the agent-based application testing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the agent-based application testing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the agent-based application testing method by any other suitable means (e.g., by means of firmware).

[0089] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0090] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0091] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0092] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0093] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0094] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0095] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.

[0096] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0097] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for testing applications based on intelligent agents, characterized in that, include: Receive target description information and set test termination conditions, wherein the target description information includes description information of the test task, and the test task is used to at least indicate the application to be tested; Obtain a screenshot of the application's interface, perform visual element recognition and text recognition on the screenshot, and generate interface element information based on the recognition results; The target description information, the interface screenshot, and the interface element information are provided to the multi-agent system. The multi-agent system dynamically generates test operation instructions based on the target description information, the interface screenshot, and the interface element information, executes the test operation instructions, and records the instruction execution results. In response to the fulfillment of the test termination condition, test results for the application are generated based on the execution result of the instruction.

2. The application testing method based on intelligent agents according to claim 1, characterized in that, The step of performing visual element recognition and text recognition on the screenshot, and generating interface element information based on the recognition results, includes: Visual element recognition is performed on the screenshot to obtain the first position information of the interactive components in the screenshot, and text recognition is performed on the screenshot to obtain the text content in the screenshot and the second position information corresponding to the text content. Based on the overlap between the first location information and the second location information, the text content is associated with the corresponding interactive component, and the component type of the interactive component, the first location information, and the text content associated with the interactive component are combined to generate interface element information.

3. The application testing method based on intelligent agents according to claim 2, characterized in that, Also includes: In scenarios where graphical user interface tree information is available, the graphical user interface tree information of the application is obtained, and the first position information of the interactive component and the component type are corrected based on the graphical user interface tree information.

4. The application testing method based on intelligent agents according to claim 1, characterized in that, The multi-agent system includes an interface analysis agent, an operation analysis agent, a result analysis agent, a task planning agent, and a history memory agent. The process of dynamically generating test operation instructions by the multi-agent system based on the target description information, the interface screenshot, and the interface element information includes: Invoke the interface analysis agent, input the interface screenshot and the interface element information into the interface analysis agent, and obtain the interface scene understanding results output by the interface analysis agent; Invoke the operation analysis agent, input the interface scene understanding result, the target description information and historical operation records into the operation analysis agent, and obtain the test operation instructions output by the operation analysis agent; The result analysis agent is invoked to generate an operation judgment result based on the screenshots of the interface before and after the operation. The operation judgment result is used to indicate whether the operation performed in response to the test operation instruction is qualified. Invoke the task planning agent, input the target description information and the operation judgment result into the task planning agent, and obtain the task completion status output by the task planning agent; The historical memory agent is invoked to record the operation information of the current round and store it in the historical operation record for use in generating the test operation instructions in subsequent rounds.

5. The application testing method based on intelligent agents according to claim 4, characterized in that, Also includes: After executing the test operation instruction, the result analysis agent is invoked to determine whether the current operation meets expectations based on the screenshots of the interface before and after the operation. If it does not meet expectations, the abnormal information is recorded.

6. The application testing method based on intelligent agents according to claim 4, characterized in that, The target description information includes user description information and system description information. The user description information is a test target and auxiliary prompt information described in natural language input by the user, used to indicate the test target that the application to be tested needs to complete. The system description information is a preset responsibility specification information, used to limit the data processing logic of the interface analysis agent, the operation analysis agent, the result analysis agent, the task planning agent, and the history memory agent.

7. The application testing method based on intelligent agents according to claim 1, characterized in that, The step of executing the test operation instruction and recording the execution result of the test operation instruction includes: The test operation instruction is parsed, the operation type and corresponding operation parameters of the test operation instruction are extracted, and the operation parameters are assembled into a device control command according to the operation type. Execute the device control command on the test device, obtain a screenshot of the interface after executing the device control command, and generate the instruction execution result based on the screenshot of the interface before and after executing the command. The instruction execution result includes information on whether the operation corresponding to the test operation command was successfully executed, information on whether the interface of the application has changed, and error information.

8. The application testing method based on intelligent agents according to claim 1, characterized in that, The step of generating test results for the application based on the instruction execution result in response to the satisfaction of the test termination condition includes: In response to the fulfillment of the test termination condition, the stored test information is saved as structured data, and a test report display page is generated based on the preset template and the structured data. The test report display page displays information including at least one of the following: statistical information on the total number of test steps, the number of exceptions, the number of pages covered, and the total test time; a page state transition diagram showing the page jump relationship; and screenshots before and after each operation, a visual processing result diagram, and an operation diagram corresponding to each operation.

9. The application testing method based on intelligent agents according to claim 1, characterized in that, Also includes: A test directory is generated based on the test timestamp, and the interface screenshots before and after the execution of the test operation command, the test operation command, timestamp information, exception information and test status are stored in the test directory; When the application interface changes, the interface change information is recorded, and the interface change information includes at least the interface identifiers before and after the change. Based on the interface identifiers before and after the jump, an interface jump path graph is constructed with each interface as a node and the jump relationship between interfaces as an edge, and stored in the test directory.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the agent-based application testing method as described in any one of claims 1-9.

Images