Method and device for testing interface visual automation exploration based on large model

Abstract

The invention relates to the field of automatic testing, and provides an interface vision automatic exploration testing method and device based on a large model.The method comprises the steps that interface data of software to be tested are collected and preprocessed, and a standardized interface data set is obtained; inputting the standardized interface data set into a pre-trained multi-modal large model to execute identification and classification of interface elements, exploration of interaction paths and marking operation of exploration results, and outputting an exploration result set with marks; on the basis of key information in the marked exploration result set, inquiring the constructed historical use case library, and matching to obtain related historical test cases; and matching and optimizing the matched historical test cases, executing the matched and optimized test cases, and generating a test result report. The problems of low test coverage rate, high maintenance cost and low efficiency in the prior art are solved, intelligent exploration on the UI interface is realized, the historical use cases are automatically matched and reused, and self-adaptive automatic test is completed.

Description

Technical Field

[0001] This invention relates to the field of automated testing, and in particular to a testing method and apparatus for automated exploration of interface vision based on a large model. Background Technology

[0002] With the increasing complexity and rapid iteration of software application interfaces, automated testing of user interfaces (UI) has become a crucial step in ensuring software quality and improving release efficiency. Traditional UI automated testing mainly relies on script recording and playback technology or code-based testing frameworks, where testers pre-write test scripts, define element location methods and operation flows, and then execute them through an automation engine. While these methods improve test execution efficiency to some extent, they are essentially still "static" and "rule-driven" testing models.

[0003] In existing technologies, test scripts rely entirely on manual design and maintenance, making it difficult to simulate real and diverse user interactions. This is especially true for new, modified, or boundary interaction scenarios, easily creating testing blind spots and leading to missed defects. Furthermore, frequent UI iterations cause changes in element attributes, rendering many existing test scripts invalid. This necessitates manual updating and debugging of location information, resulting in a large workload, significant time consumption, and the inability to directly reuse historical test cases across versions, leading to a serious waste of testing resources. Finally, writing and maintaining stable UI automation scripts typically requires testers to possess certain programming skills and element location technology knowledge, making implementation and maintenance costs high for testing teams without technical backgrounds or sufficient experience.

[0004] Therefore, how to achieve intelligent exploration, adaptive test case matching and reuse of UI interfaces without deep human intervention, thereby significantly improving test coverage, reducing maintenance costs and lowering the technical threshold, has become an important technical issue that urgently needs to be addressed in the current software testing field. Summary of the Invention

[0005] This invention provides a testing method and apparatus for automated visual exploration of interfaces based on a large model. It solves the problems of low test coverage, high maintenance costs and low efficiency caused by the reliance on manual script writing and maintenance in existing UI automated testing technologies. It realizes intelligent exploration of UI interfaces based on a large model, automatic matching and reuse of historical test cases and completion of adaptive automated testing.

[0006] This invention provides a testing method for automated exploration of interface vision based on a large model, comprising the following steps: Collect the interface data of the software to be tested, and preprocess the interface data to obtain a standardized interface dataset. The standardized interface dataset is input into a pre-trained multimodal large model, which performs interface element recognition and classification, interaction path exploration, and exploration result labeling operations, and outputs a labeled exploration result set. Based on the key information in the labeled exploration result set, the existing historical test case library is queried to match and obtain relevant historical test cases; The matched historical test cases are adapted and optimized, and the adapted and optimized test cases are executed to generate a test result report.

[0007] According to the present invention, a test method for automated exploration of interface vision based on a large model, after generating a test result report, the method further includes: updating and iteratively optimizing the training data of the multimodal large model and the historical test case library based on the test result report and the exploration scenarios where no test cases were matched.

[0008] According to the present invention, a testing method for automated exploration of interface vision based on a large model is provided. The method involves collecting interface data of the software under test and preprocessing the interface data to obtain a standardized interface dataset. Specifically, this includes: acquiring image frames, element attribute sets, and sequences of interactive events triggered by users or scripts during the operation of the software under test using an interface capture tool; the element attribute set includes element type, identifier, and location information; performing noise reduction processing on the image frames; normalizing the format of the element attribute set; and classifying the interactive event sequences into operation categories to generate a standardized interface dataset with a unified data pattern.

[0009] According to the present invention, a testing method for automated visual exploration of an interface based on a large model is provided. The method involves the multimodal large model performing interface element identification and classification, interaction path exploration, and exploration result labeling operations, outputting a labeled exploration result set. Specifically, the method includes: performing element-level detection on image frames in the standardized interface dataset using the visual branch of the multimodal large model to obtain a set of interface elements and their corresponding functional semantic labels; performing semantic parsing on the element attribute set and interaction event sequence using the language branch of the multimodal large model to generate an element-event association graph; generating multiple candidate interaction paths based on the element-event association graph according to a preset user behavior logic, and comparing interface state changes in real time during the exploration process. If path duplication is detected or no new interface state is generated, the current interaction path is terminated from further expansion; and labeling the interface scene corresponding to each retained interaction path to generate a labeled exploration result set.

[0010] According to the present invention, a testing method for automated visual exploration of interfaces based on a large model includes the following steps: querying a pre-built historical test case library based on key information in the marked exploration result set to obtain relevant historical test cases; extracting key information from the marked exploration result set, the key information including scene type, covered interface element information, and current interface version identifier; comparing the extracted key information with a pre-established index in the historical test case library; and selecting historical test cases related to the current exploration scene from the historical test case library according to preset matching rules.

[0011] According to the present invention, a testing method for automated visual exploration of interfaces based on a large model includes the following steps: adapting and optimizing the matched historical test cases, executing the adapted and optimized test cases, and generating a test result report. Specifically, this includes: if the interface element location information in the historical test cases is inconsistent with the current interface to be tested, then inputting the element attribute information of the current interface into the multimodal large model, which automatically updates the location information based on semantic understanding; if the interaction step sequence of the historical test cases is missing relative to the currently explored interaction path, the multimodal large model automatically supplements the corresponding interaction steps based on the interaction path; executing the adapted and optimized test cases using an automated testing engine; collecting actual results in real time during execution and comparing them with the expected results of the test cases; and generating a corresponding test result report based on the comparison results.

[0012] This invention also provides a testing device for automated exploration of interface vision based on a large model, comprising the following modules: The interface data acquisition module is used to collect interface data of the software under test and preprocess the interface data to obtain a standardized interface dataset. The model exploration module is used to input the standardized interface dataset into a pre-trained multimodal large model, which performs interface element recognition and classification, interaction path exploration, and exploration result labeling operations, and outputs a labeled exploration result set. The test case matching module is used to query the constructed historical test case library based on the key information in the labeled exploration result set and match relevant historical test cases. The test case module is used to adapt and optimize the matched historical test cases, execute the adapted and optimized test cases, and generate a test result report.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the test method for automated exploration of interface vision based on a large model as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a test method for automated exploration of interface vision based on a large model as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a test method for automated exploration of interface vision based on a large model as described above.

[0016] This invention provides a testing method and apparatus for automated visual exploration of interfaces based on a large model, which offers the following advantages: By integrating computer vision and natural language understanding capabilities through a large model, it automatically identifies and classifies interface elements, simulates real user interaction logic to generate multiple interaction paths, and structurally labels the exploration results. This achieves autonomous, comprehensive, and efficient dynamic exploration of the software interface, significantly improving the coverage of test scenarios and the exploration efficiency of automated testing. Through matching key information from the labeled exploration result set with a historical test case library, the system can intelligently retrieve relevant historical test cases. For the matched test cases, the semantic understanding and adaptation capabilities of the large model automatically update element location information and supplement and optimize interaction steps, thereby driving the automated testing engine to execute tests. This process enables the automated reuse and adaptation of historical test cases across versions and scenarios, even with UI iterations or changes in interface elements. This significantly reduces the cost of manually modifying and maintaining test scripts due to interface changes, improves the utilization efficiency of test resources, and lowers the technical threshold for UI automated testing in terms of programming skills and element location experience for testers. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the testing method for automated exploration of interface vision based on a large model provided by the present invention.

[0019] Figure 2This is a schematic diagram of the structure of the testing device for automated exploration of interface vision based on a large model provided by the present invention.

[0020] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0022] The following is combined with Figures 1-3 The embodiments of the present invention are described in detail.

[0023] The testing method for automated exploration of interface vision based on a large model provided in this embodiment of the invention is executed by a testing device for automated exploration of interface vision based on a large model. This device can be configured in a computer, which can be a local computer or a cloud computer. The local computer can be a computer, tablet, etc., and no specific limitation is made here.

[0024] Figure 1 This is a flowchart illustrating the testing method for automated exploration of interface vision based on a large model provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps: S110. Collect the interface data of the software to be tested, and preprocess the interface data to obtain a standardized interface dataset.

[0025] According to the present invention, a testing method for automated exploration of interface vision based on a large model is provided. The method involves collecting interface data of the software under test and preprocessing the interface data to obtain a standardized interface dataset. Specifically, this includes: acquiring image frames, element attribute sets, and sequences of interactive events triggered by users or scripts during the operation of the software under test using an interface capture tool; the element attribute set includes element type, identifier, and location information; performing noise reduction processing on the image frames; normalizing the format of the element attribute set; and classifying the interactive event sequences into operation categories to generate a standardized interface dataset with a unified data pattern.

[0026] Specifically, the deployed Selenium automated testing tool was used to collect interface data from a mobile shopping application under test. After the application started and entered the main interface, Selenium automatically captured the image frames of the current screen and extracted the attribute sets of all interactive and static elements in the interface through its WebDriver protocol. This included the type of each element (such as "button", "text input box", "image"), unique ID (such as "btn_login", "input_search"), name (such as "login", "search box"), and its precise position information in the screen coordinate system (such as coordinate range). In addition, user operations were simulated (such as clicking the "category" button, entering the text "phone" in the search box, and scrolling through the product list) and the sequence of these script-triggered interactive events was recorded, including the event type, the affected element, and the event parameters. Subsequently, the acquired raw image frames were denoised using a median filter algorithm to reduce pixel interference caused by screen reflections or compression. The element attribute sets were normalized, for example, by standardizing all element type names to lowercase English letters ("button", "input") and converting position coordinates to a unified relative coordinate representation. Interaction event sequences were categorized into "navigation," "input," and "trigger" types based on their purpose. Finally, the processed image frames, the standardized element attribute JSON document, and the categorized interaction event list were combined to generate a well-structured and uniformly formatted standardized interface dataset, providing high-quality input for subsequent analysis and exploration of large-scale models.

[0027] This embodiment achieves accurate acquisition of multi-dimensional interface data by integrating automated testing tools, and effectively ensures the quality and consistency of input data by combining targeted preprocessing steps. Image noise reduction improves the readability of interface screenshots and reduces interference in subsequent visual recognition; element attribute format normalization eliminates parsing ambiguities caused by differences in data sources, providing reliable structured information for element location and matching; and the classification of interaction event sequences gives the original operations clear semantics, which helps to understand user behavior patterns.

[0028] S120. Input the standardized interface dataset into the pre-trained multimodal large model, and have the multimodal large model perform the identification and classification of interface elements, the exploration of interaction paths, and the labeling of exploration results, and output a labeled exploration result set.

[0029] According to the present invention, a testing method for automated visual exploration of an interface based on a large model is provided. The method involves the multimodal large model performing interface element identification and classification, interaction path exploration, and exploration result labeling operations, outputting a labeled exploration result set. Specifically, the method includes: performing element-level detection on image frames in the standardized interface dataset using the visual branch of the multimodal large model to obtain a set of interface elements and their corresponding functional semantic labels; performing semantic parsing on the element attribute set and interaction event sequence using the language branch of the multimodal large model to generate an element-event association graph; generating multiple candidate interaction paths based on the element-event association graph according to a preset user behavior logic, and comparing interface state changes in real time during the exploration process. If path duplication is detected or no new interface state is generated, the current interaction path is terminated from further expansion; and labeling the interface scene corresponding to each retained interaction path to generate a labeled exploration result set.

[0030] Specifically, a preprocessed standardized interface dataset of an online document editor is input into a fine-tuned multimodal large model. The model's visual branch performs pixel-level analysis on image frames, detecting all interface elements, such as a button with a "bold" icon, a text label displaying an "unsaved" status, and an input box with the content "Please enter a title...", and generates functional semantic labels for them, such as "Formatting toolbar - Bold button", "Status indicator - Unsaved label", and "Main editing area - Title input box".

[0031] The model's language branch performs semantic parsing on element attribute sets (such as element type and name) and recorded interaction event sequences (such as "clicking the 'File' menu" and "entering the text 'Test'"). It understands that the "File" menu may contain sub-items such as "New" and "Save," and that the "Enter" action must act on an editable area. Based on this, it constructs an element-event relationship graph, which describes the hierarchical relationships between elements (such as menus containing sub-items) and the possible flow of event triggering (such as clicking "File" triggering "clicking 'New'"). Based on this relationship graph, the model simulates and generates multiple candidate interaction paths according to the pre-defined "user creates and formats a document" behavioral logic. For example, path A: "Click 'File' → Click 'New' → Enter text in the title input box → Click the 'Bold' button"; path B: "Enter text directly in the main editing area → Click 'File' → Click 'Save'." When simulating the exploration of path A, after clicking "New", the model detects that the interface state changes from "Untitled Document" to "Document 1", which is a new state, so the exploration continues; if when exploring a certain path, after simulating "Clicking 'Save'", the interface state is found to be exactly the same as the state after a previous exploration (for example, both pop up a "Save Successful" prompt and the editing area does not change), it is determined that the path is duplicated, and the further expansion of that branch is terminated.

[0032] For each valid interaction path that is retained, the model marks the corresponding final or intermediate interface scenario. For example, path A is marked as "normal process - create and format a new document", which covers elements such as "file menu", "new item", "title input box" and "bold button". Based on the coreness of these elements, it is assessed as "medium risk" and finally outputs an exploration result set containing these marking information.

[0033] By deeply fusing visual and textual information through a multimodal large-scale model, accurate and automated parsing and deduction of UI elements and their interaction logic are achieved. Precise detection and functional semantic annotation of visual branches replace traditional component recognition based on fixed rules, improving the robustness of recognizing diverse and dynamic UI elements. The element-event relationship graph constructed by the language branches semantically characterizes the possibilities of interface interactions, providing structured knowledge for intelligent path exploration. Based on this, path generation and real-time state deduplication mechanisms ensure the comprehensiveness and efficiency of the exploration, avoiding the generation of invalid or repetitive test scenarios. The final output, a set of exploration results with rich semantic tags, provides high-quality, structured input for subsequent test case matching and reuse, thereby significantly improving the intelligence level and exploration efficiency of the entire UI automated testing process.

[0034] S130. Based on the key information in the labeled exploration result set, query the constructed historical test case library and match the relevant historical test cases.

[0035] According to the present invention, a testing method for automated visual exploration of interfaces based on a large model includes the following steps: querying a pre-built historical test case library based on key information in the marked exploration result set to obtain relevant historical test cases; extracting key information from the marked exploration result set, the key information including scene type, covered interface element information, and current interface version identifier; comparing the extracted key information with a pre-established index in the historical test case library; and selecting historical test cases related to the current exploration scene from the historical test case library according to preset matching rules.

[0036] Specifically, after the large model completes the intelligent exploration of the login interface of a new version of the "Enterprise Reimbursement System" and generates a labeled exploration result set, the system extracts key information from it: the scenario type is "normal process scenario - user login", the interface element information covered includes "username input box (id: user_name)", "password input box (id: pwd)", "login button (id: login_btn)", and the current interface version is identified as "V2.1.5".

[0037] The system compares this key information with a pre-built multi-dimensional index in a historical UI test case library. This library contains test cases accumulated from multiple past versions, and each test case has an index based on scenario type, associated interface elements, and software version. The system filters according to preset matching rules, for example, prioritizing test cases with the same scenario type of "user login"; within these test cases, it further compares the covered interface elements and calculates their overlap with the element set in the current exploration results; simultaneously, it considers interface version compatibility and evaluates the degree of UI structure difference between version V2.1.5 and the associated versions of the historical test cases. Ultimately, the system successfully matched a historical test case from version V2.0, which also covers username input, password input, and login button operations, and whose UI structure differences are within the compatibility range, thus being selected as a historical test case relevant to the current exploration scenario.

[0038] This embodiment extracts structured key information from the semantically rich exploration results generated by a large model and uses a pre-built multi-dimensional index of historical test case database for efficient comparison. The system can intelligently and accurately locate the most relevant test cases from massive historical test assets. This avoids the problems of test case omissions or misselections caused by relying entirely on manual memory or simple keyword searches in traditional testing. Filtering based on preset multi-dimensional matching rules (scenario, element, version) not only improves the accuracy and relevance of test case matching but also lays a reliable foundation for subsequent test case adaptation and execution, thereby significantly improving the reuse efficiency of test cases and the intelligence level of the entire automated testing process.

[0039] S140. Adapt and optimize the matched historical test cases, execute the adapted and optimized test cases, and generate a test result report.

[0040] According to the present invention, a testing method for automated visual exploration of interfaces based on a large model includes the following steps: adapting and optimizing the matched historical test cases, executing the adapted and optimized test cases, and generating a test result report. Specifically, this includes: if the interface element location information in the historical test cases is inconsistent with the current interface to be tested, then inputting the element attribute information of the current interface into the multimodal large model, which automatically updates the location information based on semantic understanding; if the interaction step sequence of the historical test cases is missing relative to the currently explored interaction path, the multimodal large model automatically supplements the corresponding interaction steps based on the interaction path; executing the adapted and optimized test cases using an automated testing engine; collecting actual results in real time during execution and comparing them with the expected results of the test cases; and generating a corresponding test result report based on the comparison results.

[0041] Specifically, the system matched an order submission history test case from version "V1.5" to test the new "V2.0" e-commerce application. In this test case, the original "Submit Order" button's location information was `id="submit_order_old"`, while in the current "V2.0" interface, the ID of the button with the same function has changed to `data-testid="checkout-btn"`. The system inputs the attribute information (such as type, name, location, and new ID) of relevant elements in the current interface, including this button, into the multimodal large model.

[0042] Based on semantic understanding, the large-scale model identifies that the "Submit Order" button in the original use case is semantically consistent with the button with the ID "checkout-btn" in the current interface. Therefore, it automatically updates the element location information in historical use cases to `id="checkout-btn"` (or the adapted new location method). Simultaneously, the system discovers that the new interface adds a "Select Delivery Method" step before submitting the order in the currently explored interaction path, a step missing in the historical use case's interaction sequence. Based on the currently explored complete interaction path (e.g., "Fill in Address → Select Delivery Method → ​​Submit Order"), the large-scale model automatically generates and supplements the missing interaction step "Click the 'Standard Delivery' option" and its expected result (e.g., "Display Delivery Fee"). Subsequently, using the integrated Selenium automated testing engine, this optimized use case, updated with new element location information and supplemented interaction steps, is executed. The engine simulates the operation and collects the actual results in real time (e.g., whether clicking the new button successfully redirects to the order completion page), comparing these results with the updated expected results in the use case. Finally, the system automatically generates a clear test result report based on the comparison results (if they match, the test passes; if they don't match, the differences are recorded).

[0043] By leveraging the semantic understanding and path awareness capabilities of the large-scale model, intelligent, precise adaptation and dynamic optimization of historical test cases are achieved. This effectively solves two core challenges caused by software interface iterations: element location failure and changes in operational processes. Historical test cases can regain usability in new versions of the interface without manual intervention, while maintaining the integrity of the test logic. The execution of the automated testing engine and the comparison of real-time results ensure the objectivity and efficiency of test verification. This embodiment significantly reduces the maintenance cost of test cases caused by UI changes, greatly improves the reusability of test assets across versions, and enhances the automation and intelligence of the entire regression testing process.

[0044] According to the present invention, a test method for automated exploration of interface vision based on a large model is provided. After generating a test result report, the training data of the multimodal large model and the historical test case library are updated and iteratively optimized based on the test result report and the exploration scenarios where no test cases were matched.

[0045] Specifically, after a complete round of automated testing, the system generated a test result report containing multiple items. For example, the report indicated that in an exploration scenario of "user changing profile picture," the actual result (successful profile picture update) was consistent with the expected result of historical test cases, and the test passed. Simultaneously, in a scenario of "shopping cart discount calculation," the report also showed that the expected result of historical test cases was "displaying a discount of 30 yuan," while the actual result was "displaying a discount of 25 yuan." The system fed this difference—"expected discount of 30 yuan, actual discount of 25 yuan"—back to the large model, which initially analyzed the reason as "the discount rules have changed." This round of exploration also discovered a completely new scenario not stored in the historical test case library: "completing an order using third-party payment (such as digital RMB)." For this new scenario where no test case was matched, the system marked its complete exploration results (including interface screenshots, element states, and interaction event sequences) as a "new payment process scenario."

[0046] The system supplements the training dataset of the multimodal large model with the aforementioned test anomaly reports (including discrepancy information and preliminary analysis) and the complete exploration results of the new "third-party payment" scenario, as new training data. Simultaneously, based on the exploration results of this new "third-party payment" scenario, the large model automatically generates a new test case, including a test scenario description, specific interaction steps (such as "selecting digital RMB payment" and "confirming payment"), and expected results (such as "redirecting to the payment success page"). This new test case is added to the historical test case library, and a corresponding scenario type and element association index is created for it. The system uses this supplementary data (anomaly reports and new scenario data) to perform an incremental fine-tuning of the large model to improve its future recognition accuracy for similar preferential rule interfaces and its ability to explore new payment processes. After this round of optimization, the system automatically triggers a new round of UI interface exploration and testing.

[0047] By systematically feeding test execution results (especially anomalies) and newly explored, uncovered scenarios back to the model and test case library, a complete "test-learn-optimization" closed loop is formed. This not only enables the historical test case library to be dynamically and automatically enriched and expanded as software functions evolve, ensuring the test assets' coverage of the latest business scenarios; but also, by continuously incrementally fine-tuning the multimodal large model using real test feedback, the large model's interface understanding, exploration, and test case matching capabilities continuously evolve and improve in practice, thereby enhancing the adaptive capability and long-term effectiveness of the entire automated testing system. This iterative optimization mechanism fundamentally transforms the system from a static automated execution tool into an intelligent testing system with continuous learning and self-improvement capabilities.

[0048] The following describes the testing device for automated exploration of interface vision based on a large model provided by the present invention. The testing device for automated exploration of interface vision based on a large model described below and the testing method for automated exploration of interface vision based on a large model described above can be referred to in correspondence.

[0049] like Figure 2 The image shows a testing device for automated visual exploration of interfaces based on a large model, provided by the present invention, comprising: The interface data acquisition module 210 is used to collect interface data of the software under test and preprocess the interface data to obtain a standardized interface dataset. The model exploration module 220 is used to input the standardized interface dataset into a pre-trained multimodal large model, and the multimodal large model performs the identification and classification of interface elements, the exploration of interaction paths, and the labeling of exploration results, and outputs a labeled exploration result set. The test case matching module 230 is used to query the constructed historical test case library based on the key information in the labeled exploration result set and match relevant historical test cases. The test case module 240 is used to adapt and optimize the matched historical test cases, execute the adapted and optimized test cases, and generate a test result report.

[0050] Specifically, the functions of each module in the user account management system provided in this embodiment of the invention correspond one-to-one with the operation flow of each step in the above method-like embodiments, and the achieved effects are also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment of the invention.

[0051] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340. The processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a test method for automated visual exploration of the interface based on a large model. This method includes: collecting interface data of the software under test and preprocessing the interface data to obtain a standardized interface dataset; inputting the standardized interface dataset into a pre-trained multimodal large model, where the multimodal large model performs interface element recognition and classification, interaction path exploration, and labeling of exploration results, outputting a labeled exploration result set; based on key information in the labeled exploration result set, querying a constructed historical test case library to match relevant historical test cases; adapting and optimizing the matched historical test cases, executing the adapted and optimized test cases, and generating a test result report.

[0052] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0053] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the testing method for automated interface visual exploration based on a large model provided by the above methods. The method includes: collecting interface data of the software to be tested and preprocessing the interface data to obtain a standardized interface dataset; inputting the standardized interface dataset into a pre-trained multimodal large model, whereby the multimodal large model performs interface element recognition and classification, interaction path exploration, and labeling of exploration results, and outputs a labeled exploration result set; based on the key information in the labeled exploration result set, querying a constructed historical test case library to match relevant historical test cases; adapting and optimizing the matched historical test cases, executing the adapted and optimized test cases, and generating a test result report.

[0054] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a testing method for automated visual exploration of interfaces based on a large model, as provided by the methods described above. The method includes: collecting interface data of the software under test and preprocessing the interface data to obtain a standardized interface dataset; inputting the standardized interface dataset into a pre-trained multimodal large model, whereby the multimodal large model performs identification and classification of interface elements, exploration of interaction paths, and labeling of exploration results, outputting a labeled exploration result set; querying a constructed historical test case library based on key information in the labeled exploration result set to match relevant historical test cases; adapting and optimizing the matched historical test cases, executing the adapted and optimized test cases, and generating a test result report.

[0055] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0056] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0057] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A testing method for automated visual exploration of interfaces based on large models, characterized in that, include: Collect the interface data of the software to be tested, and preprocess the interface data to obtain a standardized interface dataset. The standardized interface dataset is input into a pre-trained multimodal large model, which performs interface element recognition and classification, interaction path exploration, and exploration result labeling operations, and outputs a labeled exploration result set. Based on the key information in the labeled exploration result set, the existing historical test case library is queried to match and obtain relevant historical test cases; The matched historical test cases are adapted and optimized, and the adapted and optimized test cases are executed to generate a test result report.

2. The testing method for automated exploration of interface vision based on a large model according to claim 1, characterized in that, After generating the test result report, the method further includes: Based on the test result report and the exploration scenarios where no test cases were matched, the training data of the multimodal large model and the historical test case library are updated and iteratively optimized.

3. The testing method for automated exploration of interface vision based on a large model according to claim 1, characterized in that, The process involves collecting interface data from the software under test and preprocessing the interface data to obtain a standardized interface dataset, specifically including: The interface capture tool is used to acquire image frames, element attribute sets, and sequences of interactive events triggered by the user or script during the operation of the software under test; the element attribute set includes element type, identifier, and location information; The image frames are denoised, the element attribute set is formatted, and the interaction event sequence is classified into operation categories to generate a standardized interface dataset with a unified data pattern.

4. The testing method for automated exploration of interface vision based on a large model according to claim 3, characterized in that, The process of performing interface element identification and classification, interaction path exploration, and exploration result labeling operations by the multimodal large model, and outputting a labeled exploration result set, specifically includes: The visual branch of the multimodal large model is used to perform element-level detection on image frames in the standardized interface dataset to obtain a set of interface elements and their corresponding functional semantic labels. The element attribute set and interaction event sequence are semantically parsed using the language branch of the multimodal large model to generate an element-event association graph; Based on the element-event association graph, multiple candidate interaction paths are generated according to the preset user behavior logic. During the exploration process, the interface state changes are compared in real time. If a path is detected to be repeated or no new state is generated on the interface, the current interaction path is terminated and further expansion is not allowed. The interface scene corresponding to each retained interaction path is marked, and a set of marked exploration results is generated.

5. The testing method for automated exploration of interface vision based on a large model according to claim 1, characterized in that, Based on the key information in the tagged exploration result set, the existing historical test case library is queried to match relevant historical test cases, specifically including: Key information is extracted from the labeled exploration results set, including scene type, information on the covered interface elements, and the current interface version identifier. The extracted key information is compared with a pre-established index in the historical use case library; Based on preset matching rules, historical test cases related to the current exploration scenario are selected from the historical test case library.

6. The testing method for automated exploration of interface vision based on a large model according to claim 1, characterized in that, The process of adapting and optimizing the matched historical test cases, executing the adapted and optimized test cases, and generating a test result report specifically includes: If the interface element location information in the historical test cases is inconsistent with the current interface to be tested, the element attribute information of the current interface is input into the multimodal big model, and the multimodal big model automatically updates the location information based on semantic understanding. If the interaction step sequence of the historical test cases is missing relative to the currently explored interaction path, the multimodal big model will automatically supplement the corresponding interaction steps based on the interaction path. Execute adapted and optimized test cases using an automated testing engine; During execution, actual results are collected in real time and compared with the expected results of the test cases; A corresponding test result report is generated based on the comparison results.

7. A testing device for automated visual exploration of interfaces based on large models, characterized in that, include: The interface data acquisition module is used to collect interface data of the software under test and preprocess the interface data to obtain a standardized interface dataset. The model exploration module is used to input the standardized interface dataset into a pre-trained multimodal large model, which performs interface element recognition and classification, interaction path exploration, and exploration result labeling operations, and outputs a labeled exploration result set. The test case matching module is used to query the constructed historical test case library based on the key information in the labeled exploration result set and match relevant historical test cases. The test case module is used to adapt and optimize the matched historical test cases, execute the adapted and optimized test cases, and generate a test result report.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the test method for automated exploration of interface vision based on a large model as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the test method for automated exploration of interface vision based on a large model as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the test method for automated exploration of interface vision based on a large model as described in any one of claims 1 to 6.

Images