A multi-agent cooperative interface automation testing method, device and medium

Abstract

The application provides a multi-agent cooperative interface automatic test method, device and medium, and belongs to the technical field of software test automation. The method comprises the following steps: receiving interface information of a user interface to be tested uploaded by a user; calling an image analysis agent to analyze and identify the interface information to generate an interface analysis result; based on the interface analysis result, calling a test case generation agent to automatically generate a test case; based on the test case and a selection instruction, calling a test script generation agent to generate a test script in a selected script format; calling a script execution agent to run the test script to simulate user operation, execute the test on the user interface to be tested, and record the execution process; and generating a test report according to the execution result of the test script. The application realizes automatic and efficient execution of UI automatic test.

Description

Technical Field

[0001] This invention relates to the field of software testing automation technology, specifically to a multi-agent collaborative interface automation testing method, device, and medium. Background Technology

[0002] Software UI (User Interface) automated testing refers to simulating end-user interactions (such as clicking, typing, and swiping) on ​​software interfaces (e.g., web pages, mobile application interfaces) by writing automated scripts or using specialized tools. This replaces manual repetitive functional verification, compatibility testing, and regression testing, thereby ensuring the functional integrity, correct interactive responses, and compatibility of the user interface under different environments. This technology is a key means to ensure software quality and improve testing efficiency.

[0003] Currently, mainstream UI automation testing technologies (such as those based on testing frameworks like Selenium and Appium) typically follow this implementation logic: First, the target UI element is located using the unique attributes of the control (such as ID, name, XPath, etc.); second, corresponding event simulation operations (such as clicking a button or entering text) are executed through programming scripts; finally, assertions are performed on the system's response results (such as page redirection or data update) to determine whether the test passes. Although the above methods improve testing efficiency to some extent, existing UI automation testing technologies have the following drawbacks in practical applications: high development and maintenance costs, strong environment dependence, poor stability, high skill threshold, and low team collaboration efficiency.

[0004] Therefore, there is an urgent need in this field for a new UI automation testing solution that can overcome the above-mentioned defects, aiming to significantly reduce the cost of script writing and maintenance, improve the ability to identify complex and dynamic UI elements, enhance the stability and cross-platform adaptability of the testing process, and reduce the dependence on the programming skills of testers. Summary of the Invention

[0005] In view of this, it is necessary to provide a method, device and medium for automated testing of interfaces with multi-agent collaboration, in order to solve the technical problems of high development and maintenance costs, poor adaptability to complex interaction scenarios, strong environmental dependence, poor stability, high skill threshold and low team collaboration efficiency in the existing technology.

[0006] To address the aforementioned technical problems, in a first aspect, the present invention provides an automated testing method for multi-agent collaborative interfaces, comprising: Receive interface information of the user interface to be tested uploaded by the user; The image analysis agent is invoked to analyze and identify the interface information and generate interface analysis results; Based on the interface analysis results, the test case generation agent is invoked to automatically generate test cases. Based on the test cases and selection instructions, the test script generation agent is invoked to generate test scripts that conform to the selected script format; The script is invoked to execute the test script on the intelligent agent to simulate user operations, perform tests on the user interface to be tested, and record the execution process. A test report is generated based on the execution results of the test script.

[0007] In one possible implementation, the interface information includes a screenshot of the user interface to be tested and a webpage address.

[0008] In one possible implementation, the step of invoking the image analysis agent to analyze and identify the interface information and generate interface analysis results includes: The page screenshot is preprocessed, and element detection and localization, text extraction and visual attribute recognition are performed based on the processed image; Based on the element detection and localization results, text extraction results, and visual attribute recognition results, the interface analysis results, including page metadata and a list of page elements, are integrated and generated.

[0009] In one possible implementation, the step of invoking the image analysis agent to analyze and identify the interface information and generate interface analysis results includes: The headless browser is launched in the background to load the webpage address and capture the page screenshot and document object model tree data; The page screenshot is preprocessed, and element detection and localization, text extraction and visual attribute recognition are performed based on the processed image. Element extraction information is obtained based on the document object model tree data. Based on the extracted element information, element detection and localization results, text extraction results, and visual attribute recognition results, the interface analysis results, including page metadata and a list of page elements, are integrated and generated.

[0010] In one possible implementation, the step of automatically generating test cases by invoking a test case generation agent based on the interface analysis results includes: Based on the interface analysis results, the functional information of the user interface to be tested is obtained. Based on the aforementioned functional information, a test process is generated that includes at least forward test scenarios and reverse test scenarios; The test operation instructions are generated according to the test process, and the test cases are obtained according to the test operation instructions, test data and expected verification results.

[0011] In one possible implementation, the step of invoking a test script generation agent to generate a test script conforming to the selected script format based on the test cases and selection instructions includes: The test operation instructions are mapped to the interface call commands of the test framework corresponding to the selected script format; Based on the page element attribute information in the interface analysis results, generate locator code in the selected script format; Generate a test script that conforms to the interface call command, the locator code, and the test data.

[0012] In one possible implementation, after the step of invoking the test script generation agent to generate a test script conforming to the selected script format based on the test cases and selection instructions, the method further includes: Receive editing instructions and modify at least one of the following aspects of the test script: script content, category, and priority, according to the editing instructions.

[0013] One possible implementation also includes: Receive configuration information for the created scheduled task, including associated test scripts and execution scheduling plans; The associated test script is executed according to the execution scheduling plan.

[0014] In a second aspect, the present invention also provides an electronic device, including a memory and a processor, wherein, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the multi-agent collaborative interface automation testing method described in any of the above implementations.

[0015] Thirdly, the present invention also provides a computer-readable storage medium for storing a computer-readable program or instruction, which, when executed by a processor, can implement the steps in the multi-agent collaborative interface automation testing method described in any of the above implementations.

[0016] The beneficial effects of this invention are as follows: The multi-agent collaborative interface automation testing method provided by this invention allows multiple agents to work collaboratively. Image analysis agents, test case generation agents, test script generation agents, and script execution agents can run in parallel, improving task execution efficiency. The entire testing process, from receiving interface information to generating test reports, achieves a high degree of automation. There is no need for manual writing of test cases and scripts, reducing a large amount of repetitive work and significantly shortening test preparation and execution time, enabling faster testing and accelerating software development iteration. Furthermore, due to the high degree of automation, reliance on professional testers is reduced, especially the requirements for testers' programming skills and mastery of test frameworks. Even non-professional testers can quickly generate test cases and scripts, execute tests, and obtain reports through this system, thereby saving labor costs. Furthermore, when the user interface under test changes, the system can re-analyze the interface information through the image analysis agent and automatically adjust the test cases and scripts, eliminating the need for frequent manual script modifications and effectively reducing the maintenance costs of test scripts. Furthermore, detailed test reports are generated based on the execution results of the test scripts, including information such as the execution status of test cases and problems found during the testing process. This provides testers and developers with comprehensive reference information, which helps to better evaluate the quality and performance of the software and formulate subsequent development and optimization strategies. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic flowchart of an embodiment of the multi-agent collaborative interface automated testing method provided by the present invention; Figure 2 A schematic diagram of an embodiment of the multi-agent collaborative interface automated testing system provided by the present invention; Figure 3 For the present invention Figure 1 A schematic diagram of an embodiment of S200; Figure 4 For the present invention Figure 1 Another embodiment of the S200 flowchart is shown below; Figure 5 For the present invention Figure 1 A schematic diagram of an embodiment of S300; Figure 6 For the present invention Figure 1A schematic diagram of an embodiment of the S400; Figure 7 A schematic diagram of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0020] In the description of the embodiments of the present invention, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0021] The terms "first," "second," etc., used in the embodiments of this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] Before demonstrating the embodiments, the following terms will be explained.

[0024] AI (Artificial Intelligence) is a comprehensive technological science that aims to simulate, extend, and expand human intelligence.

[0025] AI Agent: An AI intelligent agent refers to an artificial intelligence system capable of autonomously perceiving its environment, making decisions, and taking actions. Driven by a large language model, an AI Agent possesses the ability to autonomously understand, perceive, plan, remember, and use tools, enabling it to automatically execute and complete complex tasks. Unlike traditional artificial intelligence, an AI Agent has the ability to gradually achieve a given goal through independent thinking and the invocation of tools.

[0026] UI (User Interface) refers to the medium through which users interact with digital products (such as software, websites, mobile applications, etc.), encompassing the overall design of visual design, interaction logic, and user experience.

[0027] URL (Uniform Resource Locator): is an address format used to identify the location of resources on the Internet. Simply put, a URL is the address of a webpage. You can find specific webpages, images, files or other resources on the network through a URL.

[0028] The DOM (Document Object Model Tree) is used to represent the structure of HTML or XML documents. It organizes elements, attributes, text, and other elements in a hierarchical, tree-like structure. The DOM tree is a hierarchical structure where each node has a parent node (except the root node) and possible child nodes. This structure is similar to a directory tree in a file system. The DOM tree consists of various types of nodes, including element nodes (such as...) 、 ), attribute nodes (such as class="example"), text nodes (such as "Hello World"), etc. The DOM tree of an HTML document is usually rooted at the element, and divided into two main branches: the root and the branch. The root branch usually contains the document's metadata, such as the title and stylesheet links; the branch contains the document's main content, such as text, images, tables, etc.

[0029] This invention provides an automated testing method, device, and medium for multi-agent collaborative interfaces, which will be described below.

[0030] Figure 1 This is a schematic flowchart of an embodiment of the multi-agent collaborative interface automated testing method provided by the present invention. Figure 2 A schematic diagram of an embodiment of the multi-agent collaborative interface automated testing system provided by the present invention is shown below. Figure 1 As shown, the automated testing methods for multi-agent collaborative interfaces include: S100: Receive interface information of the user interface to be tested uploaded by the user.

[0031] It should be noted that this invention is a UI automation testing intelligent platform (hereinafter referred to as the platform) based on multimodal large models and multi-agent collaboration. The platform consists of 6 parts, such as... Figure 2 As shown, these are the page management module 201, the test creation module 202, the test execution module, the test result module 204, the test report module 205, and the scheduled task template. Clicking the upload button on the page management module 201's interface brings up an input interface, allowing users to obtain the interface information of the user-uploaded user interface to be tested via a web upload button, API interface, or other methods. Interface information can include key information such as the interface layout, component types, and styles.

[0032] S200: Call the image analysis agent to analyze and identify the interface information and generate interface analysis results.

[0033] It should be noted that the page management module 201 can automatically analyze interface information and manage analysis results, providing an interface for viewing and deleting interface analysis results. The input interface of the page management module 201 can be used to add page information for the user interface to be tested. Page information can include the page name, page description, page URL, etc. After uploading the interface information, the image analysis agent is triggered to complete multimodal image analysis and UI element recognition. That is, the image analysis agent automatically identifies page elements and displays all interface analysis results in the page list interface. The interface analysis results include page name, description, parsing status, number of elements, creation time, etc. Each interface analysis result has two operation buttons: a delete button and a view button. Clicking the delete button deletes the interface analysis result, and clicking the view button allows viewing the detailed information of the interface analysis result.

[0034] S300. Based on the interface analysis results, call the test case generation agent to automatically generate test cases.

[0035] It should be noted that the test creation module 202 automatically generates test cases and test scripts based on the interface analysis results. After the interface information is uploaded, the image analysis agent corresponding to the page management module 201 automatically analyzes the uploaded interface information, and the test case generation agent corresponding to the test creation module 202 generates structured test cases. Test cases include an interface overview, analysis of key elements, test scenarios, expected results, supplementary explanations, etc. The operation interface of the test creation module 202 provides editing and saving functions for the test cases.

[0036] S400. Based on the test cases and selection instructions, the test script generation agent is invoked to generate a test script that conforms to the selected script format.

[0037] It should be noted that after test cases are automatically generated, the format requirements for the generated test scripts can be set, such as generating a MidScene.js YAML script or a Playwright TypeScript script. The interface for creating test module 202 also has a script generation button to support test script generation.

[0038] S500: Call the script execution agent to run the test script to simulate user operations, perform tests on the user interface to be tested, and record the execution process.

[0039] It should be noted that after the test scripts are automatically generated, the script execution agent in the test execution module 203 can be invoked to query, execute, view, edit, and delete the generated test scripts. The interface of the test execution module 203 includes a test script list and test script execution statistics. The statistics include the total number of scripts, the number of scripts currently executing, the number of scripts that have been completed, and the number of scripts with reports. For example, it might display a total of 4 scripts, with 0 scripts currently executing and 4 scripts that have been completed, and all scripts have corresponding test reports. Additionally, the test script list displays all generated test script records, each including the script name, format, execution status, and update time. The interface of the test execution module 203 also provides a script query function. By opening the interface and using conditional queries, specific test script information can be displayed. Users can quickly locate and view specific test script information by entering the script name and clicking the query button. Below each test script entry are four operation buttons: view, execute, edit, and delete. Clicking the "View" button allows users to view detailed information about the test script, including its name, description, and content. Clicking the "Delete" button allows users to delete the test script. Clicking the "Edit" button opens the script editing interface, where users can modify and adjust the test script's content, category (e.g., functional test script, performance test script, security test script, compatibility test script), and priority (e.g., highest priority, high priority, medium priority, low priority). Clicking the "Execute" button starts the script execution agent. The system will display "Test script execution task started" and "Test script execution task completed" upon completion. The execution status of the test script will be recorded in the historical execution record list of the test results module 204, while the generated test report will appear in the report list of the test report module 205.

[0040] S600. Generate a test report based on the execution results of the test script.

[0041] It should be noted that the Test Results module 204 is primarily used to view and manage the historical execution history of test scripts. The interface of the Test Results module 204 is divided into two parts: a list of historical test script execution records and statistics on historical test script execution. In the statistics area, users can view relevant statistical data. For example, the system might display that a certain type of script (such as a Playwright TypeScript script) was executed a total of 4 times, with 2 successful executions and 2 failures, resulting in a success rate of 50%. The list of historical test script execution records displays detailed execution history information for all test scripts. Users can use the conditional search function to enter the script name and other conditions, and click the search button to quickly locate and view specific test script execution records. Each test script execution record includes key information such as the test script ID, status (e.g., execution failed or completed), format (e.g., MidScene.js YAML script or Playwright TypeScript script), start time, and execution duration. Furthermore, each test script execution record has three corresponding operation buttons: view, download, and delete. Clicking the "View" button allows users to see detailed information about the test script's execution process, including test description, specific input data, behavior, parameters, and output. Clicking the "Download" button provides detailed information about the test script's execution process in .JS format. For example, this could be the detailed execution process for a user registration scenario. Clicking the "Delete" button removes the execution record of the test script.

[0042] Additionally, in the test report module 205, users can query, view, download, and delete test reports automatically generated after the test scripts are executed. After opening the interface of the test report module 205, users can view the record information of all test reports in the test report record list. Similarly, this interface also provides a report query function; users can quickly locate and view specific test reports by entering search criteria such as script name or test report name and clicking the query button. Each test report includes key information such as the test report name, test script name, script format (e.g., MidScene.js YAML script or Playwright TypeScript script), test status (e.g., pass or fail), test case execution status (including the execution result of each test case, including pass / fail status information, such as a total of 5 test cases, 4 passed, 1 failed, and 0 skipped), and execution time. Each test report record has three corresponding operation buttons: view, download, and delete. Clicking the "View" button allows users to see detailed information about the test report. This information includes an overview (summarizing the overall test execution, including the number of test cases, pass rate, etc.), defect information (including defects and errors found during testing), and test execution results (displaying the execution result of each test case, including pass / fail status information). Clicking the "Download" button will download the test report, typically saved in Word format; clicking the "Delete" button will delete the test report.

[0043] In summary, the multi-agent collaborative interface automation testing method provided by this invention firstly involves multiple agents working collaboratively. Image analysis agents, test case generation agents, test script generation agents, and script execution agents can run in parallel, improving task execution efficiency. The entire testing process, from receiving interface information to generating test reports, achieves a high degree of automation. There is no need for manual writing of test cases and scripts, reducing a large amount of repetitive work and significantly shortening test preparation and execution time, enabling faster testing and accelerating software development iteration. Furthermore, due to the high degree of automation, reliance on professional testers is reduced, especially the requirements for testers' programming skills and mastery of testing frameworks. Even non-professional testers can quickly generate test cases and scripts, execute tests, and obtain reports through this system, thereby saving labor costs. Furthermore, when the user interface under test changes, the system can re-analyze the interface information through the image analysis agent and automatically adjust the test cases and scripts, eliminating the need for frequent manual script modifications and effectively reducing test script maintenance costs. Furthermore, detailed test reports are generated based on the execution results of the test scripts, including information such as the execution status of test cases and problems found during the testing process. This provides testers and developers with comprehensive reference information, which helps to better evaluate the quality and performance of the software and formulate subsequent development and optimization strategies.

[0044] In some embodiments of the present invention, the interface information includes a screenshot of the user interface to be tested and a webpage address.

[0045] It's important to note that screenshots refer to image files obtained after a user takes a screenshot of the user interface to be tested. Screenshots contain visual information about the user interface in a specific state, such as the appearance and position of interface elements like buttons, input boxes, text, and layout. Through screenshots, the image analysis agent can intuitively identify and analyze the interface's layout and elements, providing a foundation for subsequent test case and script generation. A URL (webpage address) refers to the webpage address of the user interface, i.e., the network resource identifier where the user interface resides. Through the URL, the agent can access and load the user interface under test for real-time interactive testing. Simultaneously, the URL can also provide contextual information, helping the agent better understand the functionality and purpose of the user interface under test.

[0046] In some embodiments of the present invention, such as Figure 3 As shown, step S200 includes: S211. Perform image preprocessing on the page screenshot, and perform element detection and localization, text extraction and visual attribute recognition based on the processed image.

[0047] It should be noted that if a page screenshot is uploaded in the page management module 201, the image analysis agent automatically performs page image analysis and UI element (i.e., page element) recognition, and displays the analysis results. The method for performing page image analysis and UI element recognition mainly relies on the system calling the image analysis agent in the background. The image analysis agent is an executable program; input image data, execute the program, and output the interface analysis results. Calling the image analysis agent performs image preprocessing on the page screenshot, including resizing (scaling the page screenshot to a standard size suitable for the model), normalization (standardizing pixel values ​​for easier model processing), and noise reduction. Image analysis agents use pre-trained object detection models (such as YOLO and Faster R-CNN) to identify common UI elements in page screenshots. UI elements include buttons, input elements (such as input boxes and date pickers), dropdown menus, icons, and selection boxes (toggle switches and dropdown menus). The model outputs element detection and localization results in list format, including the type of UI element, its bounding box coordinates in the page screenshot, and confidence score. The image analysis agent can also use Optical Character Recognition (OCR) technology to identify all visible text in the page screenshot and obtain text extraction results. These results are also in list format, including the text content and its position in the page screenshot. The text content can include text on buttons, placeholder text for input boxes, labels, etc. In addition, the image analysis agent can also use image classification or segmentation models to analyze the visual features of UI elements to obtain visual attribute recognition results. The visual attribute recognition results are in list form, including whether they are disabled (grayed out), whether they are selected (such as checkboxes), element styles (element colors and shapes, etc.), and element grouping attributes. Element grouping attributes can infer which elements belong to the same group (such as form fields) through the alignment, spacing, and similarity between UI elements.

[0048] S212. Based on the element detection and localization results, text extraction results, and visual attribute recognition results, integrate and generate the interface analysis results including page metadata and a list of page elements.

[0049] It's important to note that after the image analysis agent identifies the element detection and localization results, text extraction results, and visual attribute recognition results for each UI element in a page screenshot, it combines the identified text and element type to infer the UI element's function and analyzes the relative position of the UI elements, using alignment relationships to infer the interface layout (such as navigation bars, sidebars, main content areas, etc.). Thus, the image analysis agent integrates the element detection and localization results, text extraction results, and visual attribute recognition results to generate a structured list of page elements. Each object in the page element list represents an interactive or identifiable UI element on the user interface under test. The image analysis agent performs element detection and localization, text extraction, and visual attribute recognition tasks simultaneously through parallel computation. Subsequently, in a sequentially executed information fusion layer, all these parallel outputs are aligned, matched, and combined based on spatial location and semantic relationships, ultimately generating a complete and structured list of UI elements (i.e., the page element list). Furthermore, the image analysis agent can obtain page metadata based on the element detection and localization results, text extraction results, and visual attribute recognition results. The page metadata includes the page name, description, number of elements, creation time, etc., and the page element category includes multiple page metadata records. Thus, the image analysis agent fuses and encapsulates the element detection and localization results, text extraction results, and visual attribute recognition results to generate a structured interface analysis result (JSON format).

[0050] The page management module 201's interface is divided into two parts: a page list and page analysis statistics. For example, the page analysis statistics include a total of 3 pages, 3 pages analyzed successfully, 1 page under analysis, and 0 pages analyzed unsuccessfully. On the page management module 201's interface, clicking the "Upload Page Screenshot" button brings up an input interface. In this interface, users enter the page name, description, and URL, then click and drag to upload the screenshot. After uploading, clicking the "AI Analysis" button triggers the image analysis agent to perform multimodal image analysis and UI element recognition. For example, when the system displays "File uploaded successfully!", AI analysis has started in the background. The page analysis statistics show: 1 page under analysis. When the system displays "AI analysis completed," a new page metadata record is added to the page list. The image analysis agent automatically identifies page elements and displays the interface analysis results for all analyzed pages in the page list interface. Each page analysis record (i.e., interface analysis result) includes the specific page name, page description, parsing status (e.g., analysis completed), number of elements, and creation time. Each interface analysis result has two corresponding operation buttons: a delete button and a view button. Clicking the "delete" button deletes the interface analysis result, while clicking the "view" button allows you to view the page details of the interface analysis result. The page details include basic information and a list of page elements. Basic information includes the page name, page description, page URL, parsing status (e.g., analysis complete), number of elements, creation time, and update time. The page element list information includes: specific element names, detailed test data, and status (e.g., testable).

[0051] In this embodiment, image preprocessing of the page screenshot, such as grayscale conversion, binarization, and denoising, removes noise and interference, enhancing image contrast and clarity. This makes subsequent operations like element detection and localization, text extraction, and visual attribute recognition more accurate and reliable. Furthermore, by integrating the element detection and localization results, text extraction results, and visual attribute recognition results, an interface analysis result including page metadata and a list of page elements is generated. This information integration method, through image preprocessing and multi-dimensional information, allows for a more comprehensive understanding of the page's structure and content. The generated interface analysis result contains rich element information, providing more accurate input for the test case generation agent, enabling the generated test cases to more comprehensively cover various interaction scenarios. Moreover, when page elements change, the image analysis agent can automatically update the element detection and localization, text extraction, and visual attribute recognition results by re-analyzing the processed image. This allows the test script to automatically adjust based on the latest interface analysis results, reducing the workload of manual script maintenance. Furthermore, through image preprocessing and multi-dimensional information integration, the test scripts rely more heavily on the visual features and text content of elements. These features are relatively stable, reducing script failures caused by attribute changes. This makes the scripts suitable not only for static pages but also for dynamic pages and complex interactive interfaces. The generated interface analysis results and test scripts are better adapted to different testing environments and platforms. Moreover, the generated interface analysis results include page metadata and a detailed list of page elements. This information provides testers with rich visual references, allowing them to intuitively understand the page structure and element distribution, thereby better understanding and optimizing test cases and scripts.

[0052] In some embodiments of the present invention, such as Figure 4 As shown, step S200 includes: S221. Launch a headless browser in the background to load the webpage address and capture a screenshot of the page and document object model tree data.

[0053] It's important to note that if the input is a webpage URL (i.e., a webpage address), the image analysis agent will launch a headless browser (controlled by tools such as Puppeteer, Playwright, or Selenium) on the backend to redirect to and load the webpage URL. It will then wait for the page corresponding to the URL to fully load (including JavaScript execution, dynamic content loading, and network request completion). Afterward, it will obtain the page's Document Object Model (DOM) data and take a screenshot of the rendered page to obtain the final visual result. The DOM data includes the DOM structure (the HTML document object model of the webpage URL, i.e., the source code skeleton of the user interface under test), CSS style information, and the Accessibility Tree, among other things.

[0054] S222. Perform image preprocessing on the page screenshot, perform element detection and localization, text extraction and visual attribute recognition on the processed image, and obtain element extraction information based on the document object model tree data.

[0055] It should be noted that the parts that are the same as those in the above embodiments are referred to in the above embodiments, and will not be repeated here. The element extraction information includes the parsed data of the DOM structure (including element tags, IDs, class names, attributes, etc.), rich semantic information extracted from the accessibility tree, etc.

[0056] S223. Based on the element extraction information, element detection and positioning results, text extraction results, and visual attribute recognition results, integrate and generate the interface analysis results including page metadata and a list of page elements.

[0057] It should be noted that: the bounding boxes of elements obtained from visual analysis of page screenshots (including element detection and localization, text extraction, and visual attribute recognition) are matched with the coordinates of UI element nodes extracted from the Document Object Model (DOM) tree data to determine which DOM node in the tree corresponds to the button identified by visual analysis at position (x, y). <button>Nodes allow for information exchange and enhancement, generating a complete and structured list of UI elements (i.e., a page element list) based on extracted element information, element detection and localization results, text extraction results, and visual attribute recognition results. Additionally, the image analysis agent can obtain page metadata based on these data. Finally, the image analysis agent fuses and encapsulates these elements to generate a structured interface analysis result (JSON format).

[0058] In this embodiment, combining page screenshots and Document Object Model (DOM) data for analysis allows for the acquisition of page information from both visual and structural perspectives. Preprocessed page screenshots provide intuitive visual information for element detection and localization, text extraction, and visual attribute recognition. DOM data, on the other hand, provides structural information about the page, such as element hierarchy and attributes. For pages with a large amount of dynamic content, complex layouts, and interactions, relying solely on image analysis may have limitations. By combining DOM data, a better understanding of the page structure and the relationships between elements can be achieved. Therefore, this multi-source data fusion approach makes element extraction more accurate and reliable. Furthermore, by integrating element extraction information, element detection and localization results, text extraction results, and visual attribute recognition results, the generated page analysis results contain richer element information. This provides more accurate and detailed input for test cases and script generation agents, enabling the generated test cases to more comprehensively cover various interaction scenarios and functionalities. Furthermore, by combining image analysis and DOM data extraction, test scripts can rely more heavily on the visual features, text content, and structural information of elements. These features are relatively stable, reducing script failures caused by attribute changes. Furthermore, when the page structure or elements change, the image analysis agent and DOM data extraction can automatically update the results of element detection and localization, text extraction, and visual attribute recognition. Test scripts can be automatically adjusted based on the latest page analysis results, reducing the workload of manual script maintenance. Moreover, from loading the webpage address, obtaining page screenshots and DOM data, to image preprocessing, element detection and localization, text extraction, and visual attribute recognition, the entire process is completed automatically in the background, greatly reducing the pre-test preparation work for testers and improving testing efficiency. Testers do not need to manually capture page screenshots or parse DOM data; they only need to upload the webpage address, and the system can automatically generate detailed page analysis results. Furthermore, by loading webpages and obtaining data in real time through a headless browser, the current status and problems of the page can be quickly reported. Testers can adjust testing strategies and optimize test cases in a timely manner based on the real-time feedback of page analysis results, thereby improving the relevance and effectiveness of testing.

[0059] If the uploaded content is a screenshot, image recognition techniques from deep learning, such as Convolutional Neural Networks (CNNs), are used. The image is input into a trained image analysis agent, which can identify the position, size, and type of various elements in the interface, such as buttons, input boxes, text boxes, and dropdown menus. For interface information in code form, the interface structure and component information are extracted by parsing the code. For example, an HTML parsing library (such as Python's BeautifulSoup library) is used to parse the HTML code, obtain the page's DOM structure, and extract the tags, attributes, and other information of each component. For image analysis, a large number of user interface screenshots need to be labeled, noting the component types and positions, and then this labeled data is used to train the CNN model. During actual analysis, the uploaded image is input into the trained model to obtain the interface analysis results. For code parsing, corresponding parsing scripts are written, and appropriate parsing libraries are used to extract interface information based on different code types (such as HTML, CSS, JavaScript, etc.).

[0060] In some embodiments of the present invention, such as Figure 5 As shown, step S300 includes: S310. Based on the interface analysis results, the functional information of the user interface to be tested is obtained; S320. Generate a test process that includes at least forward test scenarios and reverse test scenarios based on the functional information; S330. Generate corresponding test operation instructions according to the test process, and obtain the test cases according to the test operation instructions, test data and expected verification results.

[0061] It's important to note that the process involves extracting page elements from the page analysis results and grouping related elements together. For example, it identifies the "username input box," "password input box," and "login button" as a form group. Based on the type, layout (e.g., vertical arrangement), and text labels of the page elements, it infers the interaction relationships between them. Combining the overall page layout, form groups, interaction relationships, and functional information, and utilizing Natural Language Processing (NLP) techniques to understand the textual semantics of the elements, it infers the functional information of the user interface under test. Reverse engineering testing scenarios simulate various abnormal operations, illegal inputs, or edge cases to verify the software's fault tolerance and robustness. Based on the identified functional information, the intelligent agent automatically designs test paths covering various scenarios using software testing theory. The forward engineering testing scenario simulates an ideal user, following the design intent and correctly completing the main functional flow. Then, the intelligent agent generates corresponding test operation instructions based on the test flow, and subsequently generates corresponding test cases based on the test operation instructions, test data, and expected verification results.

[0062] In this embodiment, test cases are automatically generated based on interface analysis results, reducing the workload of testers manually writing test cases and improving testing efficiency. After test cases are generated, tests can be executed immediately to obtain results. Testers can quickly optimize test cases based on the results, improving the relevance and effectiveness of the tests. Furthermore, by analyzing the interface analysis results, the functional information of the page can be accurately extracted, and the generated test cases are more consistent with the actual functions of the page, reducing test errors caused by misjudging functions. Moreover, test cases generated based on interface analysis results can automatically adapt to page changes. When page functions or layouts change, test cases can be automatically adjusted according to the latest interface analysis results, reducing maintenance costs.

[0063] In some embodiments of the present invention, such as Figure 6 As shown, step S400 includes: S410. Map the test operation instructions to the interface call commands of the test framework corresponding to the selected script format; S420. Generate locator code in the selected script format based on the page element attribute information in the interface analysis results. S430. Generate a test script that conforms to the interface call command, the locator code, and the test data.

[0064] It should be noted that: users select a target testing framework on the testing platform, such as Selenium, Playwright, or Appium. Based on the selected testing framework, test operation commands (such as input, clicks, selections, etc.) are mapped to the framework's interface call commands. Attribute information of page elements, such as ID, class name, tag name, and text content, is extracted from the interface analysis results. According to the selected script format and the requirements of the testing framework, corresponding locator code is generated. The mapped interface call commands, the generated locator code, and test data are integrated together, and according to the selected script format (such as JavaScript, TypeScript, Python, etc.), the corresponding test script file is generated.

[0065] In this embodiment, by automatically mapping test operation instructions to interface call commands of the test framework and generating locator code based on page element attribute information, the workload of testers manually writing test scripts is greatly reduced, improving the efficiency of test script generation. Automated script generation reduces errors that may be introduced by manual script writing, improving script quality and reliability. Furthermore, it can generate corresponding scripts based on the user-selected test framework, supporting multiple mainstream test frameworks such as Selenium, Playwright, and Appium, meeting different testing environments and needs. Furthermore, the locator code generated based on interface analysis results can automatically adapt to changes in page element attributes, reducing script failures caused by page updates and improving script adaptability and maintainability. Furthermore, testers do not need to deeply understand the interfaces and syntax of the test framework; they only need to focus on test case design and test data preparation, lowering the technical threshold for testers and enabling more people to participate in testing.

[0066] In some embodiments of the present invention, after the step of calling the test script generation agent to generate a test script conforming to the selected script format based on the test cases and selection instructions, the method further includes: Receive editing instructions and modify at least one of the following aspects of the test script: script content, category, and priority, according to the editing instructions.

[0067] It should be noted that a user-friendly interface is provided, allowing users to view and edit the generated test scripts. Users can select the test script to edit and enter specific editing commands through this interface. Editing commands include at least one of the following: script content modification, category modification, and priority modification. Script content modification allows users to modify the specific code content of the test script, such as adding, deleting, or modifying test operation commands. Category modification allows users to change the category of the test script, such as changing it from "functional test" to "performance test." Priority modification allows users to adjust the priority of the test script, such as changing it from "medium priority" to "high priority."

[0068] In this embodiment, users are allowed to customize the generated test scripts according to actual needs, enabling the test scripts to better adapt to specific test scenarios and requirements. Furthermore, if deficiencies or adjustments are found in the generated test scripts during testing, users can quickly modify them without regenerating the entire script, improving the flexibility and adaptability of the test scripts. Moreover, users can review and optimize the generated test scripts through editing commands, ensuring that the script's logic and content meet expectations, improving the quality and accuracy of the test scripts. During editing, users can identify and correct any errors or unreasonable aspects in the generated scripts, further enhancing the reliability of the test scripts. Furthermore, users can view the execution process and results of the test scripts in real time, gaining a clear understanding of the testing process. Simultaneously, during testing, users can pause and modify test cases and scripts as needed, avoiding test deviations caused by agent misoperation, enhancing the transparency and controllability of the testing process, and making the testing work more in line with actual needs.

[0069] In some embodiments of the present invention, step S600 is followed by: S700: Receive the configuration information of the created scheduled task, the configuration information including the associated test script and the execution schedule plan; S800, The associated test script is executed according to the execution scheduling plan.

[0070] It should be noted that the Scheduled Task Module 206 is needed when you want to execute test scripts on a scheduled basis. The Scheduled Task Module 206 provides the function of creating and managing scheduled tasks. The interface of the Scheduled Task Module 206 allows you to search for and display specific scheduled tasks by entering search keywords (such as the task name). Clicking the "Create Scheduled Task" button in the Scheduled Task Module 206 interface brings up an input interface for adding task information. You can enter the task name, associated test script, execution schedule, task description, and other configuration information. For scheduling configuration, you need to enter the scheduling type and valid time range; for execution configuration, you need to enter the timeout (in seconds), number of retries, and retry interval (in seconds). After successful creation, the scheduled task is displayed in the Scheduled Task Management List. The Scheduled Task Management List displays detailed information for each scheduled task, including the task name, schedule start time, schedule end time, execution frequency, Cron expression, and scheduling status (e.g., not scheduled, scheduled). Each scheduled task has three corresponding operation buttons: run record, edit, and delete. Clicking the "Run Log" button allows you to view information such as the task's start time, end time, execution status, and execution time. Clicking the "Edit" button allows you to modify the scheduled task based on your user actions. Clicking the "Delete" button allows you to delete the scheduled task based on your user actions.

[0071] This invention is a UI automation testing intelligent platform based on a multimodal large model and multi-agent collaboration. The platform supports multimodal AI analysis, including image analysis and webpage URL analysis, enabling automatic identification of UI elements and automatic generation of test cases and multi-format test scripts. UI elements refer to various controls, components, or objects that users can interact with or see when interacting with the software interface. Specifically, the image analysis agent analyzes page screenshots and performs webpage URL analysis to identify UI elements; this is a single agent performing these functions. The platform completes automated UI testing through multi-agent collaboration and real-time feedback. The multiple agents in this invention include an image analysis agent, a test case generation agent, a test script generation agent, and a script execution agent. The image analysis agent performs element identification, while other agents do not. Technically, this platform achieves agent division of labor, AI adaptability, and human-machine controllability. This solves the problems of high maintenance costs and difficult cross-platform adaptation, and ensures a transparent and controllable testing process through real-time intervention, avoiding the risks of black-box operation. This invention provides functions such as UI page management, test creation, automatic generation of test cases and scripts, automatic generation of test reports after test execution, and creation and execution of scheduled test tasks. This invention helps small businesses and large enterprises perform a large number of UI tests without errors, saving manpower, reducing manual labor, and improving work efficiency, thereby benefiting the organization. The UI automation testing platform based on multi-agent collaboration significantly optimizes testing efficiency and reduces maintenance costs through mechanisms such as task division and collaboration, dynamic adaptation, and visual enhancement. The beneficial effects are as follows: 1) Breakthroughs in core technologies enable multi-agent collaboration. Independent agents are responsible for tasks such as image analysis, element recognition, automatic generation of test cases and test scripts, and automatic generation of test reports after test execution. Compatibility testing and dynamic interactive verification are processed in parallel, improving task execution efficiency by more than 40%.

[0072] 2) AI technology enables deep visual understanding, breaks through the dependence on traditional controls, locates elements through image / text recognition, supports one script to adapt to multiple platforms (Web / Android / iOS), and lowers the threshold for script writing.

[0073] 3) Supports real-time manual intervention. Test cases and test scripts can be paused and modified during execution to avoid the risk of AI misoperation.

[0074] 4) Cost and resource optimization: The AI ​​self-healing system automatically repairs scripts that fail due to element changes, reducing maintenance costs by 40-50%.

[0075] 5) The entire execution process of the test script is visualized, with real-time screenshot tracking, enhancing the credibility of the test process.

[0076] To better implement the multi-agent collaborative interface automated testing method in the embodiments of the present invention, based on the multi-agent collaborative interface automated testing method, correspondingly, as follows: Figure 7 As shown, the present invention also provides an electronic device 700. The electronic device 700 includes a processor 701, a memory 702, and a display 703. Figure 7 Only some components of the electronic device 700 are shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0077] In some embodiments, processor 701 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 702 or process data, such as the multi-agent collaborative interface automation testing method of the present invention.

[0078] In some embodiments, processor 701 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 701 may be local or remote. In some embodiments, processor 701 may be implemented on a cloud platform. In one embodiment, the cloud platform may include a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, intranet, multi-cloud, etc., or any combination thereof.

[0079] In some embodiments, memory 702 may be an internal storage unit of electronic device 700, such as a hard disk or memory of electronic device 700. In other embodiments, memory 702 may also be an external storage device of electronic device 700, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 700.

[0080] Furthermore, the memory 702 may include both internal storage units of the electronic device 700 and external storage devices. The memory 702 is used to store application software and various types of data installed on the electronic device 700.

[0081] In some embodiments, display 703 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 703 is used to display information from electronic device 700 and to display a visual user interface. Components 701-703 of electronic device 700 communicate with each other via a system bus.

[0082] In one embodiment, when processor 701 executes the multi-agent collaborative interface automation test program stored in memory 702, the following steps can be implemented: Receive interface information of the user interface to be tested uploaded by the user; The image analysis agent is invoked to analyze and identify the interface information and generate interface analysis results; Based on the interface analysis results, the test case generation agent is invoked to automatically generate test cases. Based on the test cases and selection instructions, the test script generation agent is invoked to generate test scripts that conform to the selected script format; The script is invoked to execute the test script on the intelligent agent to simulate user operations, perform tests on the user interface to be tested, and record the execution process. A test report is generated based on the execution results of the test script.

[0083] It should be understood that when the processor 701 executes the multi-agent collaborative interface automation testing method program in the memory 702, in addition to the functions mentioned above, it can also implement other functions, as detailed in the description of the corresponding method embodiments above.

[0084] Furthermore, this embodiment of the invention does not specifically limit the type of electronic device 700 mentioned. Electronic device 700 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the invention, electronic device 700 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0085] Accordingly, this application also provides a computer-readable storage medium for storing computer-readable programs or instructions. When the programs or instructions are executed by a processor, they can implement the steps or functions of the multi-agent collaborative interface automation testing method provided in the above-described method embodiments.

[0086] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0087] The above provides a detailed description of the automated testing method, equipment, and medium for multi-agent collaborative interfaces provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.< / button>

Claims

1. A method for automated testing of interfaces through multi-agent collaboration, characterized in that, include: Receive interface information of the user interface to be tested uploaded by the user; The image analysis agent is invoked to analyze and identify the interface information and generate interface analysis results; Based on the interface analysis results, the test case generation agent is invoked to automatically generate test cases. Based on the test cases and selection instructions, the test script generation agent is invoked to generate test scripts that conform to the selected script format; The script is invoked to execute the test script on the intelligent agent to simulate user operations, perform tests on the user interface to be tested, and record the execution process. A test report is generated based on the execution results of the test script.

2. The automated testing method for multi-agent collaborative interfaces according to claim 1, characterized in that, The interface information includes a screenshot of the user interface to be tested and its web address.

3. The automated testing method for multi-agent collaborative interfaces according to claim 2, characterized in that, The step of calling the image analysis agent to analyze and identify the interface information and generate interface analysis results includes: The page screenshot is preprocessed, and element detection and localization, text extraction and visual attribute recognition are performed based on the processed image; Based on the element detection and localization results, text extraction results, and visual attribute recognition results, the interface analysis results, including page metadata and a list of page elements, are integrated and generated.

4. The automated testing method for multi-agent collaborative interfaces according to claim 2, characterized in that, The step of calling the image analysis agent to analyze and identify the interface information and generate interface analysis results includes: The headless browser is launched in the background to load the webpage address and capture screenshots of the page and document object model tree data. The page screenshot is preprocessed, and element detection and localization, text extraction and visual attribute recognition are performed based on the processed image. Element extraction information is obtained based on the document object model tree data. Based on the extracted element information, element detection and localization results, text extraction results, and visual attribute recognition results, the interface analysis results, including page metadata and a list of page elements, are integrated and generated.

5. The automated testing method for multi-agent collaborative interfaces according to claim 1, characterized in that, The step of automatically generating test cases by calling the test case generation agent based on the interface analysis results includes: Based on the interface analysis results, the functional information of the user interface to be tested is obtained. Based on the aforementioned functional information, a test process is generated that includes at least forward test scenarios and reverse test scenarios; The test operation instructions are generated according to the test process, and the test cases are obtained according to the test operation instructions, test data and expected verification results.

6. The automated testing method for multi-agent collaborative interfaces according to claim 5, characterized in that, The step of calling the test script generation agent to generate a test script conforming to the selected script format based on the test cases and selection instructions includes: The test operation instructions are mapped to the interface call commands of the test framework corresponding to the selected script format; Based on the page element attribute information in the interface analysis results, generate locator code in the selected script format; Generate a test script that conforms to the interface call command, the locator code, and the test data.

7. The automated testing method for multi-agent collaborative interfaces according to claim 1, characterized in that, After the process of generating a test script conforming to the selected script format by calling the test script generation agent based on the test cases and selection instructions, the process further includes: Receive editing instructions and modify at least one of the following aspects of the test script: script content, category, and priority, according to the editing instructions.

8. The automated testing method for multi-agent collaborative interfaces according to claim 1, characterized in that, Also includes: Receive configuration information for the created scheduled task, including associated test scripts and execution scheduling plans; The associated test script is executed according to the execution scheduling plan.

9. An electronic device, characterized in that, Including memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the multi-agent collaborative interface automated testing method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the multi-agent collaborative interface automation testing method according to any one of claims 1 to 8.

Images