Automated UI interaction exploration method based on multimodal large model

Abstract

The present invention discloses an automated UI interactive exploration method based on a multimodal large model, comprising the following steps: (1) collecting text, images, XML files and historical operation record information, analyzing the above information by the multimodal large model and generating a single-step exploration action, and realizing correct recognition, conversion and execution of the action through an execution system; (2) providing the multimodal large model with page information before and after the execution of the action and the current visual map, and the multimodal large model updates the visual map by extracting page features and identifying the relationship between pages; (3) using a page-level recursive exploration strategy to systematically construct a visual map of an application. By using the present invention, mobile applications can be automatically explored and visual maps can be constructed, thereby improving the accuracy and reliability of UI exploration.

Description

Technical Field

[0001] The present invention belongs to the field of automated UI interaction, and in particular relates to an automated UI interaction exploration method based on a multimodal large model. Background Art

[0002] Automatically exploring applications, analyzing program structures, and generating visual maps can help developers accurately locate and navigate the various pages and functional modules in the application. The purpose is to enhance visibility and understandability during the software development process, especially for complex application structures. This approach not only enables developers to quickly grasp the overall architecture of the application, but also provides a practical tool to track and solve problems.

[0003] Visual maps provide an intuitive representation of the application structure, helping developers quickly locate specific functional modules and pages. For developers who are new to the project, visual maps can serve as an effective navigation tool to help them familiarize themselves with the project structure more quickly. When problems are discovered during the development process, visual maps can help quickly locate the specific page or module where the problem occurred, simplifying the steps to reproduce the problem. Visual maps can serve as a bridge for communication within the development team and with non-technical stakeholders, helping to improve the efficiency of problem discussion and resolution. Automated exploration can discover pages or functions that may be missed by manual exploration, ensuring more comprehensive exploration coverage. For complex applications with rich functions and many pages, automated exploration and visual map generation are particularly important to effectively manage and reduce complexity.

[0004] Research work on building visual maps usually involves the comprehensive use of technologies from multiple fields such as automated testing, computer vision, graph theory, natural language processing (NLP), and artificial intelligence (AI). The combination of these technologies enables automated exploration of the user interface (UI) of an application, identification and classification of UI elements, understanding of the relationships between them, and ultimately building a visual map that reflects the structure of the application. The following are some existing UI interaction exploration methods: Random-based exploration strategy: Simulate user behavior by generating non-deterministic events (such as user operations such as clicks and slides) to trigger various states of the application, thereby achieving the purpose of exploring and revealing the structure and behavior of the application. The disadvantage is that since the events are randomly generated, this method may over-explore some areas and under-explore other areas, which affects the depth and quality of the exploration. Automatically generating and executing a large number of random events may result in high resource consumption, including CPU and memory, especially when exploring large or resource-intensive applications.

[0005] Model-driven exploration method: By running the application and dynamically generating a model (which can be regarded as a visual map), the model maps the application's states and transitions between states. For example, Stoat, the tool first uses a series of predefined or randomly generated events to interact with the application, then processes the captured UI elements through a hash function to generate a hash value representing the UI state, and finally, based on the recorded states and events, Stoat builds a state transition diagram. Its defects are: the state recognition algorithm is inaccurate, especially when the UI changes slightly or the visual differences between states are not obvious. This may lead to incorrect classification of different states as the same state (state merging error), or incorrect identification of the same state as multiple states (state splitting error). Asynchronous operations and time-dependent behaviors in the application may not be accurately captured by the model. For example, delayed responses to network requests may result in different application states at different points in time, which may be difficult to accurately represent through static models. Summary of the invention

[0006] The present invention provides an automated UI interactive exploration method based on a multimodal large model, which can automatically explore mobile applications and build visual maps, thereby improving the accuracy and reliability of UI exploration.

[0007] An automated UI interaction exploration method based on a multimodal large model comprises the following steps:

[0008] (1) Collect text, images, XML files, and historical operation records, analyze the information using a multimodal large model and generate single-step exploration actions, and then correctly identify, transform, and execute the actions through the execution system;

[0009] (2) Providing the multimodal model with page information before and after the action and the current visual map. The multimodal model updates the visual map by extracting page features and identifying the relationship between pages.

[0010] (3) A page-level recursive exploration strategy is adopted to systematically build a visual map of the application.

[0011] The method of the present invention realizes a comprehensive exploration of complex applications from scratch by combining a multimodal large model and automated exploration technology, ensuring comprehensive coverage of each page module and functional module of the application, and providing a reliable foundation for software development and functional analysis. Automated exploration reduces the cost and time consumption of manual exploration, making the software exploration process more efficient and economical. The conversion of exploration actions and the generation of execution instructions can ensure the accurate execution of exploration actions, eliminate execution deviations caused by human factors, and ensure the consistency and reliability of results. By constructing and updating visual maps, the structure of the application and the relationship between pages can be better understood, providing a comprehensive view for further development and analysis, and helping to discover potential problems and optimization solutions. By adopting a page-level recursive exploration strategy, a visual map of the application is systematically constructed, ensuring that each page module is fully explored, and improving the comprehensiveness and effectiveness of the exploration.

[0012] Furthermore, in step (1), the text includes the application overview material provided by the developer / publisher and the application runtime information queried through the developer tool; the image is a screenshot file; the XML file is a preprocessed view hierarchy file; and the historical operation record is an array containing multiple record items, each of which represents a separate user operation and is gradually improved during the iterative regression exploration process.

[0013] Each record contains a unique identifier for each operation, the exact time the operation occurred, the type of operation, a brief description of the operation, the page where the operation occurred, the specific elements of the operation, the specific parameters for input and sliding operations, and the operation result page.

[0014] The preprocessing of the view hierarchy file includes filtering and numbering specific elements, specifically:

[0015] First, focus on interactive elements. By retaining only clickable and swipable elements and all leaf node elements, the number of elements that need to be explored is reduced, making the multimodal large model exploration process more focused on elements that are critical to user interaction.

[0016] Then, accurately locate and set a unique ID for each retained element so that the multimodal large model and execution system can accurately locate and reference these elements;

[0017] Finally, the hierarchical relationship between elements is preserved, so that the multimodal large model can understand the interface layout and logic of the application, thereby generating exploratory actions to simulate the user's real operation path.

[0018] In step (1), a standard action type is defined to the multimodal large model so that the multimodal large model can output a single-step exploration action. The multimodal large model accurately locates the operation element according to the element ID in the view hierarchy file.

[0019] In step (1), the correct recognition, conversion and execution of the action are realized by the execution system, specifically:

[0020] First, the execution system needs to parse the actions output by the multimodal large model, including the type of action and the target of the action; then, the identified actions are converted into specific execution instructions; finally, the target action is executed using an application programming interface, software development kit, or other control interface; after the action is executed, the execution system collects data about the execution results, including the success or failure status, error information during the execution process, and corresponding page information.

[0021] In step (2), the visual map is a structured representation that depicts the relationships between the various pages of the application, including how they are linked through user interactions;

[0022] The visual map data structure is an array that contains descriptions of all pages in the application. Each page object contains: a unique identifier for the page, a description of the page, a set of interactive elements on the page, a set of identifiers of subpages to which the current page can directly navigate, and a set of identifiers of the parent page of the current page.

[0023] The interactive elements on the page in the visual map are composed of four parts, namely the unique identifier of the element, the type of operation performed on the element, the identifier of the target page, and the description or document of the operation.

[0024] In step (2), the multimodal large model extracts and recognizes page features based on the input information, and outputs the updated visual map and historical operation records. The update logic is based on the following five points: adding new pages, updating elements, modifying page jump paths, and maintaining element actions and goals.

[0025] The specific process of step (3) is as follows:

[0026] The first step is to set the starting point. The exploration script sets the default entry of the application as the starting page of the exploration.

[0027] The second step is to generate and execute single-step exploration actions;

[0028] In the third step, the multimodal large model updates the visual map based on the exploration results. If a new page is generated, the exploration script adds the new page to the queue to be explored.

[0029] Step 4: Complete the initial page exploration. The multimodal model determines whether a page jump occurs based on the page information before and after the action is executed. If a page jump occurs, the exploration script returns to the initial page and repeatedly requests the multimodal model to generate non-repetitive exploration actions, repeating the dynamic exploration process of the initial page until the multimodal model determines that all interactive elements have been explored.

[0030] Step 5: After the initial page exploration is completed, the exploration script finds new pages that have not been explored from the queue to be explored and repeats the above page exploration process;

[0031] Step 6: The full exploration is complete. When the exploration script confirms that all pages in the queue to be explored have been explored, the exploration process ends.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. Comprehensiveness and systematicity: Based on a multimodal large model, the present invention combines multiple data such as text, images, XML files, and historical operation records to systematically explore applications and build their visual maps from scratch, ensuring comprehensive exploration coverage of each page module and functional module of the application. This helps to discover and solve various defects and problems in the application, and improves the quality and stability of the software.

[0034] 2. Structured data conversion: The present invention converts the application into structured data, i.e., a visual map. The visual map is used to define the application page structure and describe the relationship and interaction between pages. It is conducive to the subsequent development work, such as path planning and optimization, anomaly detection and repair, visualization and understanding, continuous integration and continuous development, thereby improving the efficiency and quality of developers' work, accelerating the discovery and repair of problems, and providing strong support for the development and maintenance of applications.

[0035] 3. Page-level recursive exploration strategy: The present invention adopts a page-level recursive exploration strategy to systematically build a visual map of the application, thereby ensuring a comprehensive exploration of the application structure. This strategy can effectively discover the various page modules and functional modules in the application, providing a basis for further software development and functional analysis.

[0036] 4. Accurate interaction simulation: The present invention accurately locates and references page elements by preprocessing the view hierarchy file, and generates accurate exploration actions to simulate the user's real operation path. This helps to ensure the correct execution of UI exploration actions and improve the accuracy and reliability of UI exploration. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of an automated UI interaction exploration method based on a multimodal large model in an embodiment of the present invention;

[0038] Figure 2 A flowchart of the generation and execution of a single-step exploration action in an embodiment of the present invention;

[0039] Figure 3 A schematic diagram of a process of updating a visual map in an embodiment of the present invention;

[0040] Figure 4 A schematic diagram of the process of systematically constructing a visual map in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0042] like Figure 1 As shown, an automated UI interaction exploration method based on a multimodal large model includes the following steps:

[0043] S10: Collect information such as text, images, XML files, and historical operation records. The multimodal large model analyzes the above information and generates single-step exploration actions. The execution system realizes correct recognition, conversion, and execution of the actions. The exploration actions are used to systematically analyze the relationship between the functional modules and pages of the application. The use of historical operation records helps to avoid repeated exploration. The specific content of this structure is gradually improved during the exploration process.

[0044] S20: Provide the multimodal big model with page information before and after the action is executed and the current visual map. The multimodal big model updates the visual map by extracting page features and identifying the relationship between pages. This includes identifying new pages, updating page elements, modifying page jump paths, etc., to ensure that the visual map reflects the latest status of the application structure and layout.

[0045] S30: Use page-level recursive exploration strategy to systematically build a visual map of the application. Starting from a single page, request the multimodal big model to generate a single-step exploration action and execute the action by the automatic execution system. Repeatedly visit the multimodal big model to ensure that all interactive elements on the page are explored. The exploration script will add other pages to the queue to be explored based on the page jump relationship and interaction path. After completing the exploration of the current page, it will conduct in-depth exploration based on the link relationship between pages to reveal the overall structure of the application.

[0046] In the embodiment of the present invention, the multimodal big model used may be an open source big model such as BLIP, MiniGPT, VisualGLM, or other commercial big models.

[0047] like Figure 2As shown, the specific implementation of S10 is:

[0048] S101, collecting input information, including text, images, XML files and historical operation records.

[0049] Specifically, the input information includes application overview materials provided by the developer / publisher, application runtime information (such as appPackage, appActivity) queried through developer tools, screenshot files, view hierarchy files, and historical operation records.

[0050] The historical operation record is an array containing multiple record items, each of which represents a separate user operation, which will be gradually improved during the exploration process of iterative regression. The record item structure contains the following fields: actionId, timestamp, actionType, description, pageId, elementId, inputValue, outcomePage, which specifically represent the following: the unique identifier of each operation, the exact time when the operation occurred, the operation type, a brief description of the operation, the page where the operation occurred, the specific elements of the operation, the specific parameters for input and sliding operations, and the operation result page.

[0051] Purpose of each input information: Help the multimodal big model understand the function and purpose of the application through the application introduction, which is conducive to the preliminary determination of the key areas of exploration, such as the functions that are particularly emphasized; help the multimodal big model identify the development team of the application through the application runtime information (appPackage, appActivity), understand the startup point and main activities of the application; provide the multimodal big model with visual context through the screenshot file so that it can recognize and understand the user interface elements and layout; enable the multimodal big model to understand the hierarchy and properties of UI elements, such as clickability, swiping, etc., by providing the view hierarchy file, optimize the element selection strategy, and ensure the precise positioning of the operation elements; provide historical operation records to feedback historical exploration actions to avoid repeated exploration.

[0052] S102, pre-processing the view hierarchy file, screening and numbering specific elements. The specific operations are:

[0053] Focus on interactive elements: By retaining only clickable and slidable elements and all leaf node elements, the number of elements that need to be explored is significantly reduced, so that the multimodal large model exploration process can be more focused on those elements that are critical to user interaction, thereby improving the efficiency and effectiveness of exploration.

[0054] Precise positioning: Set a unique ID for each retained element so that the multimodal large model and exploration action execution system can accurately locate and reference these elements. This is the prerequisite for achieving precise interaction (click, input, slide, etc.) and ensuring the correct execution of exploration actions.

[0055] Preserving element relationships: Preserving the hierarchical relationships between elements helps the multimodal large model understand the interface layout and logic of the application, thereby generating exploration actions to simulate the user's real operation path.

[0056] S103, providing the multimodal big model with defined standard action types. For example, in a touch screen device, standard actions such as click, input, long press, swipe, and back can be designed. The multimodal big model accurately locates the operation element according to the element ID in the view hierarchy file.

[0057] S104, correctly identifying, converting and executing the action through the execution system.

[0058] First, the execution system needs to parse the actions output by the multimodal large model, including the type of action (such as click, input, slide, etc.), and the target of the action (such as the specific element or screen area of ​​the operation); then, the identified action is converted into a specific execution instruction; finally, the target action is executed using an application programming interface (API), software development kit (SDK) or other control interface, such as the operating system's accessibility features or automation framework (such as Android's UI Automator or iOS's XCUITest). After executing the action, the system collects data about the execution result, including the success or failure status, error information during the execution process, and the corresponding page information.

[0059] like Figure 3 As shown, the specific implementation of S20 is:

[0060] S201, input information into the multimodal large model: page information (images, XML files) before and after the use case execution and existing visual map data.

[0061] Among them, the visual map is a structured representation that depicts the relationship between the various pages of the application, including how they are linked through user interaction. The visual map data structure is an array called visualMap, which contains the description of all the pages in the application. Each page object contains the following fields: pageId, description, elements, children, parent, which specifically represent the following: the unique identifier of the page, the description of the page, the set of interactive elements on the page, the set of identifiers of the sub-pages that the current page can directly navigate to, and the set of identifiers of the parent page of the current page.

[0062] The interactive elements of the page in the visual map are composed of four parts: id, action, target, and document, which respectively represent the unique identifier of the element, the type of operation performed on the element, the identifier of the target page, and the description or document of the operation.

[0063] S202: The multimodal large model extracts and recognizes page features based on the input information. The multimodal large model outputs an updated visual map and historical operation records. The update logic is based on the following five points:

[0064] Add new page: Determine whether to add a new page based on the page description in the existing visual map. If the application adds a new page, add a new page object to the pages array and fill in relevant description, elements and other information.

[0065] Update elements: If the elements of the page change (such as addition, deletion, function change), you need to update the elements array in the corresponding page object to ensure that the id, action, target, document and other information of each element are accurate.

[0066] Modify the page jump path: If the page jump structure of the application changes, you need to update the children and parent fields to reflect these changes.

[0067] Page description updates: If the functionality or role of a page changes, the description field should be updated accordingly.

[0068] Maintain element actions and targets: Ensure that each element's action and target (if any) correctly reflect the actions that a user can perform on that element and the results of those actions.

[0069] like Figure 4 As shown, the specific implementation of S30 is:

[0070] S301, setting the starting point: the exploration script sets the default entry of the application (usually the home page or login page) as the starting page of the exploration;

[0071] S302, generating a single-step exploration action: the exploration script uses the developer tool to read the XML file and screenshot of the start page, and the multimodal big model identifies all interactive elements of the start page and the actions that may be associated with the elements based on the above page information. At the same time, the multimodal big model generates a non-repetitive single-step exploration action based on the understanding of historical operation actions;

[0072] S303, executing a single-step exploration action: the exploration script uses the action execution system to execute the generated exploration action;

[0073] S304, visual map update: the multimodal large model updates the visual map according to the exploration results, including adding new pages, updating elements, modifying page jump paths, updating page descriptions, and clarifying element actions of maintenance elements. If a new page is generated, the exploration script will add the new page to the queue to be explored.

[0074] S305, completing the initial page exploration: the multimodal big model determines whether a page jump occurs based on the page information before and after the execution of the action. If a page jump occurs, the exploration script returns to the initial page, repeatedly requests the multimodal big model to generate exploration actions, and repeats the dynamic exploration process of the initial page until the multimodal big model determines that all interactive elements have been explored;

[0075] S306, new page exploration: after the initial page exploration is completed, the exploration script finds new pages that have not been explored from the queue to be explored, and repeats the above page exploration process;

[0076] S307, end of comprehensive exploration: when the exploration script confirms that all pages in the queue to be explored have been explored, the exploration process ends.

[0077] This approach ensures that every page module and functional module of the application is carefully examined and explored, while also providing a comprehensive view of the application structure, laying a solid foundation for further software development and functional analysis.

[0078] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An automated UI interaction exploration method based on a multimodal large model, characterized in that: The steps include: (1) Collect text, images, XML files, and historical operation records, analyze the information using a multimodal large model and generate single-step exploration actions, and then correctly identify, transform, and execute the actions through the execution system; (2) Providing the multimodal model with page information before and after the action and the current visual map. The multimodal model updates the visual map by extracting page features and identifying the relationship between pages. A visual map is a structured representation that depicts the relationship between the various pages of an application, including how they are linked through user interaction. The visual map data structure is an array that contains descriptions of all the pages in the application. Each page object contains: a unique identifier for the page, a description of the page, a set of interactive elements on the page, a set of identifiers for subpages that the current page can directly navigate to, and a set of identifiers for the parent page of the current page. The interactive elements on the page in the visual map are composed of four parts, namely, the unique identifier of the element, the type of operation performed on the element, the identifier of the target page, and the description or documentation of the operation; (3) Adopting a page-level recursive exploration strategy to systematically build a visual map of the application; The specific process is: The first step is to set the starting point. The exploration script sets the default entry of the application as the starting page of the exploration. The second step is to generate and execute single-step exploration actions; In the third step, the multimodal large model updates the visual map based on the exploration results. If a new page is generated, the exploration script adds the new page to the queue to be explored. Step 4: Complete the initial page exploration. The multimodal model determines whether a page jump occurs based on the page information before and after the action is executed. If a page jump occurs, the exploration script returns to the initial page and repeatedly requests the multimodal model to generate non-repetitive exploration actions, repeating the dynamic exploration process of the initial page until the multimodal model determines that all interactive elements have been explored. Step 5: After the initial page exploration is completed, the exploration script finds new pages that have not been explored from the queue to be explored and repeats the above page exploration process; Step 6: The full exploration is complete. When the exploration script confirms that all pages in the queue to be explored have been explored, the exploration process ends.

2. The method for automated UI interaction exploration based on a multimodal large model according to claim 1, characterized in that: In step (1), the text includes application overview materials provided by the developer / publisher and application runtime information queried through developer tools; the image is a screenshot file; The XML file is a preprocessed view hierarchy file; the historical operation record is an array containing multiple record items, each of which represents a separate user operation and is gradually improved during the iterative regression exploration process.

3. The method for automated UI interaction exploration based on a multimodal large model according to claim 2, characterized in that: Each record contains a unique identifier for each operation, the exact time the operation occurred, the type of operation, a brief description of the operation, the page where the operation occurred, the specific elements of the operation, the specific parameters for input and sliding operations, and the operation result page.

4. The method for automated UI interaction exploration based on a multimodal large model according to claim 2, characterized in that: The preprocessing of the view hierarchy file includes filtering and numbering specific elements, specifically: First, focus on interactive elements. By retaining only clickable and swipable elements and all leaf node elements, the number of elements that need to be explored is reduced, making the multimodal large model exploration process more focused on elements that are critical to user interaction. Then, accurately locate and set a unique ID for each retained element, so that the multimodal large model and execution system can accurately locate and reference these elements; Finally, the hierarchical relationship between elements is preserved, so that the multimodal large model can understand the interface layout and logic of the application, thereby generating exploratory actions to simulate the real operation path of users.

5. The method for automated UI interaction exploration based on a multimodal large model according to claim 4, characterized in that: In step (1), a standard action type is defined to the multimodal large model so that the multimodal large model can output a single-step exploration action. The multimodal large model accurately locates the operation element according to the element ID in the view hierarchy file.

6. The method for automated UI interaction exploration based on a multimodal large model according to claim 1, characterized in that: In step (1), the correct recognition, conversion and execution of the action are realized by the execution system, specifically: First, the execution system needs to parse the actions output by the multimodal large model, including the type of action and the target of the action; then, the identified actions are converted into specific execution instructions; finally, the target action is executed using an application programming interface, software development kit, or other control interface; after the action is executed, the execution system collects data about the execution results, including the success or failure status, error information during the execution process, and corresponding page information.

7. The method for automated UI interaction exploration based on a multimodal large model according to claim 1, characterized in that: In step (2), the multimodal large model extracts and recognizes page features based on the input information, and outputs the updated visual map and historical operation records. The update logic is based on the following five points: adding new pages, updating elements, modifying page jump paths, and maintaining element actions and goals.

Images