Multi-interface mobile application evaluation method and device based on multi-modal model

By using a multi-interface mobile application evaluation method based on a multimodal model, we can systematically detect cross-page information consistency issues in mobile applications, improve the accuracy and efficiency of detection, overcome the limitations of traditional methods, and meet the development needs of rapid iteration.

CN121326772AActive Publication Date: 2026-01-13ZHEJIANG UNIV
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202511882466.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-12-15
Publication Date
2026-01-13
Estimated Expiration
2045-12-15

AI Technical Summary

Technical Problem

Existing technologies struggle to automate and systematically detect cross-page information consistency issues in mobile applications, especially in large-scale applications, leading to a decline in user experience quality.

Method used

A multi-interface mobile application evaluation method based on a multimodal model is adopted. Interaction data is collected through graphical user interface agents, and structured information is extracted by combining large language models and tool calls. A cross-interface information association graph is constructed, and consistency problem reports are generated by using multi-agent parallel analysis and knowledge base filtering technology.

Benefits of technology

It enables end-to-end consistency detection of information in mobile applications, improving the targeting and accuracy of problem detection, reducing false positives, significantly improving testing efficiency and coverage, and alleviating the pressure of manual review.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121326772A_ABST
    Figure CN121326772A_ABST
Patent Text Reader

Abstract

The invention relates to a multi-interface mobile application evaluation method and device based on a multi-modal model, and belongs to the field of mobile application quality testing. The method comprises the steps of test item collection, information extraction, review information synthesis, information analysis and result filtering and output, specifically, test link collection is completed through a graphical user interface proxy technology, accurate information extraction is completed based on a multi-modal large language model and tool calling, and the result is output. User demand analysis and review task planning are completed through multi-agent cooperation, deep consistency analysis is completed based on thinking chain reasoning, and finally a high-quality problem report is output through an intelligent filtering mechanism. According to the method, automatic extraction, intelligent analysis and systematic evaluation of cross-page information are realized, the problem of information inconsistency in the mobile application can be efficiently identified, the pertinence and effectiveness of problem discovery are improved, the high confidence of a test result is ensured, the manual review pressure is relieved, and technical support is provided for application quality optimization.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present application relates to the field of mobile application quality testing, in particular to a multi-interface mobile application evaluation method and device based on a multi-modal model. BACKGROUND

[0002] In the current field of mobile application development, as the complexity of applications increases and the iteration speed accelerates, traditional manual quality assurance (QA) methods have been difficult to fully cover all test scenarios. Especially in large-scale applications, due to functional differentiation, team size expansion, and high customization of functions, information content, and business services, inconsistencies between different pages and modules are prone to occur, which violates the consistency principle in user experience design and seriously affects the usability and user experience quality of the application.

[0003] From the perspective of interactive experience, information inconsistency (such as conflicting prices on different pages) can significantly damage user trust and cognitive fluency, thereby disrupting the construction of user mental models, leading to cognitive dissonance and decision paralysis. This experience problem involves deep cognitive consistency and system credibility principles, having a significant negative impact on user trust, task completion rate, and brand perception. Existing testing methods have many limitations: 1. Manual review methods are costly, have low coverage, and are highly subjective, making it difficult to meet the needs of rapid iteration development; 2. Traditional automated testing mainly focuses on functional verification and cannot understand semantic-level information inconsistency issues; 3. Existing automated GUI agent testing mainly focuses on task completion and functional executability, and conducts evaluation at the accessibility level, but lacks deep analysis capabilities for cross-page information consistency; 4. Information consistency detection requires comprehensive understanding capabilities across pages and modalities (text and vision), and relies on deep cognitive understanding of application information architecture, and existing methods have obvious shortcomings in these areas.

[0004] For example, the invention patent with publication number CN119883888A discloses a user interface defect detection method, device, equipment and medium, relating to the field of smart home / smart home technology, which includes: obtaining a user interface image and a standard interface image corresponding to the user interface image, and extracting first features of the standard interface image; determining a first detection rule corresponding to the standard interface image according to the first features of the standard interface image and historical defect data, and recommending the first detection rule to a test end; performing consistency detection on the user interface image and the standard interface image according to a second detection rule fed back by the test end to obtain defect information of the user interface image. This method improves the accuracy and efficiency of interface consistency detection through intelligent recommendation and automatic detection, but it mainly focuses on the comparison between a single interface and a standard interface, and cannot solve the problem of cross-page and cross-process information consistency detection.

[0005] The application No. CN120336183A discloses an application testing method, device, storage medium and program product, which comprises the following steps: controlling a user interface of a to-be-tested application to slide in a specified direction; after the user interface slides in the specified direction, determining whether the interface screenshot of the user interface before the sliding is consistent with the interface screenshot of the user interface after the sliding by using a trained preset model; when the two interface screenshots are consistent, determining that the user interface has been slid to a boundary position. The method mainly focuses on the boundary detection of the interface sliding behavior, and does not analyze the semantic consistency. Figure One

[0006] The application No. CN120371694A discloses a user interface automatic assertion method, system and program product based on a multi-modal large model, which comprises the following steps: obtaining a screen image of a to-be-tested UI and a text instruction describing an expected state; inputting the image and the text instruction into a multi-modal large model configured to jointly process the image and the text; analyzing whether the actual UI state in the image meets the expectation of the text instruction by using the image-text joint understanding capability of the multi-modal large model; and generating a final assertion result based on the model determination result. Although the method applies the multi-modal large model to verify the UI state, it mainly focuses on the expected matching of a single interface state, and lacks the systematic detection capability for the cross-page information flow and semantic consistency.

[0007] Therefore, there is an urgent need for a technical solution capable of automatically and systematically detecting cross-page information consistency problems of mobile applications. SUMMARY

[0008] The purpose of the present application is to provide a multi-interface mobile application evaluation method and device based on a multi-modal model, aiming to solve the problem that the prior art only focuses on a single interface and cannot automatically and systematically detect cross-page information consistency problems of mobile applications.

[0009] The technical solution adopted by the present application to solve the technical problems is as follows: A multi-interface mobile application evaluation method based on a multi-modal model, comprising the following steps: Test item collection: executing the interaction of a specified test link through a graphical user interface agent, collecting interface screenshots and user interaction history records in the interaction process; Information extraction: using an information extractor combining a large language model and tool calling to extract structured information from the interface screenshots, and generating structured data containing multiple information groups; Review information synthesis: based on the structured data, using a large language model to analyze user information requirements, establishing a cross-interface information association graph based on the parsed user information requirements, and combining preset information consistency checkpoints to investigate cross-interface information content, and generating a review file; ​Information analysis: based on the generated review files, different analysis agents are used according to the nature of the review task, and the reasoning mode of thought chain is used for parallel analysis, and the reasoning process and analysis results of information consistency problems are output; Result filtering and output: construct heuristic rules based on keywords to preliminarily screen the analysis results, and then screen through the filter based on search enhancement generation technology, output the final problem report, and complete the evaluation.

[0010] In one embodiment, in the test item collection, the user interaction history record includes: Interface summary information, a simplified natural language description of each interface generated by the graphical user interface agent, which extracts the key functional areas and main information content of the interface, providing a basis for subsequent interaction decisions of the graphical user interface agent; Interaction operation record, used to record the specific operation behavior of the graphical user interface agent on each interface, each interaction operation record at least contains operation type and operation object.

[0011] In one embodiment, in the information extraction, the information extractor based on the combination of large language model and tool calling performs structured information extraction on the interface screenshot, including: Perform macro semantic understanding on the interface screenshot by the large language model, perform context-aware block division to form information groups, and analyze the content and visual semantics of the information groups to realize basic information extraction, wherein each information group contains a unique identifier, block position, information summary, and component list; Then the large language model calls external tool set to verify and enhance the extracted basic information, and completes the extraction of structured information.

[0012] In one embodiment, the external tool set includes one or more of an optical character recognition tool, a color picker tool, or a pixel-level visual analysis tool, which is used to supplement and confirm the extracted information; wherein the supplemented and confirmed information includes color value, accurate text content and pixel-level visual features.

[0013] In one embodiment, in the information extraction, the organizational structure of the structured data contains three layers, which are: user interaction history record, information group summary, and specific information content of information group; The user interaction history record includes interaction path, interface summary and operation record, which is used to provide a macro business process view; The information group summary includes the overall description and block division of each information group in the interface, which is used to provide a medium interface structure view; The specific information content of the information group includes detailed information and visual features of each component in the information group, and is used to provide a microscopic view of information details.

[0014] In one embodiment, in the information synthesis, the construction process of the cross-interface information association graph includes: According to the user interaction history record, the complete operation sequence of the user between interfaces is extracted as the process skeleton of the cross-interface information association graph; at the same time, the structured information unit defining the specific business information contained in each interface is taken as the semantic node library of the cross-interface information association graph; Based on the process skeleton and the semantic node library, the operation path of the user between interfaces is extracted, and the information group involved in the path is located, forming an operation sequence with enhanced information; The association between the information groups is inferred through two-dimensional logic, and the association includes establishing a process association with a front-back sequence relationship according to the sequence of the information groups in the user operation path, and analyzing the business content carried by the information groups to determine whether they point to the same business object or describe the same business attribute semantic association; All information groups are taken as nodes, and the inferred process association and semantic association are taken as edges to establish a cross-interface information association graph.

[0015] In one embodiment, in the information synthesis, the preset information consistency check point includes numerical information contradiction, text information contradiction, visual style contradiction, information loss, and information increase. The numerical information contradiction includes numerical conflict and specification conflict; wherein the numerical conflict refers to the numerical information of the product displayed in different interfaces being inconsistent, including numerical contradiction of time, price, quantity or geographical location distance; the specification conflict refers to the product specification information being contradictory in different display positions, including non-uniform unit or coexistence of ambiguous description and accurate description; The text information contradiction includes using different text expressions for the same semantics, contradiction between texts, and ambiguity of texts; The visual style contradiction includes different visual styles of information of the same level and similar visual styles of information of different levels; wherein the different visual styles of information of the same level refer to the use of different visual styles for interface elements with the same content or the same function; the similar visual styles of information of different levels refer to the use of the same or similar visual styles for information of different levels or different states, resulting in unclear information level distinction; The information loss is the disappearance of key information when the page jumps; The information increase is the sudden appearance of new important information in the later stage of the process.

[0016] In one embodiment, in the result filtering and output, the filtering through the filter based on the retrieval enhancement generation technology includes: receive the analysis result after preliminary screening, and abstract the analysis result to remove specific business information to extract problem pattern features, and generate a query vector; In the pre-built low-quality output knowledge base, the most similar historical low-quality output cases to the query vector are searched through vector similarity retrieval; Based on the retrieved similar cases, it is judged whether the current analysis result belongs to a known low-quality output pattern; If the similarity between the current analysis result and the known low-quality output pattern exceeds a preset threshold, it is filtered; if it does not exceed, the analysis result is retained as a high-confidence output, and a final problem report is obtained.

[0017] In one embodiment, the construction process of the low-quality output knowledge base includes: Collect invalid outputs identified by preliminary screening settlement and low-quality problem reports found by manual review; Use a large language model to summarize and classify the collected low-quality problem reports, and remove specific business information in the cases while retaining the abstract features of the problem patterns; The abstracted problem patterns are subjected to text embedding processing and stored in the knowledge base for retrieval.

[0018] In addition, to achieve the above-mentioned purpose, the application also provides a multi-modal model-based multi-interface mobile application evaluation device, which is used to implement the multi-modal model-based multi-interface mobile application evaluation method, and includes: A test item acquisition module is configured to execute the interaction of a specified test link through a graphical user interface agent, and to acquire interface screenshots and user interaction history records in the interaction process; An information extraction module is configured to extract structured information from the interface screenshots based on an information extractor combining a large language model and a tool call, and to generate structured data containing multiple information groups; An audit information synthesis module is configured to analyze user information needs based on the structured data using a large language model, to establish a cross-interface information association graph based on the parsed user information needs, and to combine a preset information consistency checkpoint to investigate cross-interface information content and generate an audit file; An information analysis module is configured to use different analysis agents according to the nature of the audit task based on the generated audit file, and to use a thinking chain reasoning mode for parallel analysis, and to output the reasoning process and analysis results of information consistency problems; A result filtering module is configured to construct heuristic rules based on keywords to preliminarily screen the analysis results, and to use a filter based on retrieval enhancement generation technology for screening, and to output a final problem report to complete the evaluation.

[0019] The application has the following effects by adopting the technical scheme: (1) Systematic cross-page information consistency detection capability: The application automatically collects multiple interface information in the user interaction process through a graphical user interface agent, combines a multi-modal large language model to perform structured information extraction and cross-page association analysis, realizes systematic detection of full-process information consistency of a mobile application, and overcomes the limitation of traditional methods that can only perform single-page or functional testing.

[0020] (2) The application analyzes user needs in depth, establishes a cross-interface information association graph, systematically locates a cross-interface information set that needs to be compared for subsequent consistency review, and establishes an information consistency checkpoint system, which can comprehensively cover typical information consistency problem types in a mobile application, making consistency detection more close to actual user experience and improving the pertinence and effectiveness of problem discovery.

[0021] (3) Through a multi-level information extraction and analysis architecture, progressive analysis from macro to micro is realized, improving the accuracy of problem positioning.

[0022] (4) Through parallel analysis agents and combining thought chain reasoning and knowledge base-based result filtering mechanism, false positive rate is effectively reduced, high confidence of test results is ensured, and manual review pressure is reduced.

[0023] (5) The entire evaluation process realizes full automation from test item collection to result output, can quickly adapt to different types of mobile applications and test scenarios, significantly improves test efficiency and coverage, and reduces manual cost. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description.

[0025] Figure 1 is a framework diagram of the multi-interface mobile application evaluation method based on a multi-modal model provided by the embodiment of the application.

[0026] Figure 2 is a flowchart of the multi-interface mobile application evaluation method based on a multi-modal model provided by the embodiment of the application.

[0027] Figure 3 is a structural schematic diagram of the multi-interface mobile application evaluation device based on a multi-modal model provided by the embodiment of the application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific implementation described herein is only used to explain the present application and does not limit the protection scope of the present application.

[0029] As shown in Figure 1 , the overall flowchart of the multi-interface mobile application evaluation method based on the multi-modal model provided in the present embodiment. The present application realizes the automatic identification and evaluation of the cross-page information consistency problem in the mobile application through the multi-modal large language model technology. The method includes five core steps of test item collection, information extraction, review information synthesis, information analysis and result filtering, forming a complete evaluation closed loop.

[0030] S1, test item collection: through the graphical user interface agent, the interaction of the specified test link is executed, and the interface screenshots and user interaction history records in the interaction process are collected. Specifically as follows: As shown in Figure 2 , this step completes the automatic interaction of the specified test link through the intelligent graphical user interface agent (GUI), and collects the interface screenshots and user interaction history records in the interaction process. This step serves as the data collection link of the method of the present application, and provides a basic data source for the subsequent information consistency evaluation.

[0031] In the present embodiment, the system can complete the automatic collection of test links for various application scenarios (such as e-commerce shopping, social media, financial services, etc.). The present embodiment takes clothes shopping in e-commerce shopping as an application scenario. The test link is defined in the form of a task goal, and the GUI agent can independently complete the full-process interaction from the starting page to the target page, and record the complete user interaction history record.

[0032] The user interaction history record mainly includes two types of core information: interface summary information and interaction operation record. The interface summary information describes the key content of the page in natural language form, and the interaction operation record details the operation behavior and object of the user on the interface. These two types of information together with the interface screenshots form a structured test data set, which provides a complete data basis for the subsequent cross-interface information consistency analysis.

[0033] Through this intelligent test item collection method, the system can efficiently simulate the interaction path of real users and obtain complete interface information sequences, thereby providing comprehensive and accurate raw data input for subsequent information extraction and consistency evaluation.

[0034] S2, information extraction: based on the information extractor combining large language model and tool calling, the interface screenshots are subjected to structured information extraction to generate structured data containing multiple information groups. Specifically as follows: The core of this step is to build a "semantic understanding-precision enhancement" two-stage information extraction architecture. This architecture combines the macro semantic understanding ability of multi-modal large language model with the micro data accuracy of special tool set, realizes the high-quality and structured analysis of interface information, and provides reliable data foundation for subsequent cross-interface consistency evaluation.

[0035] First stage: macro semantic understanding based on expert role simulation. In this stage, through precise prompt tuning, the multi-modal large language model is configured as a senior UI (user interface) analysis expert agent. The core task of this expert agent is not simply to identify elements, but to deconstruct the interface from the perspective of business logic and user experience, mainly reflected in the following two aspects: context-aware chunking: the model goes beyond traditional segmentation methods based on geometric position, and instead adopts an intelligent grouping strategy based on semantic relevance. It aggregates logically related elements in the interface (such as product pictures, titles, prices, and promotion labels) into independent "information groups". For example, in a list page containing multiple product cards, the model will explicitly indicate in the description of each information group that "the product card with the model wearing a white T-shirt in the picture" or "the product card with the title 'Pure Cotton Short Sleeve Round Neck T-shirt'". This division ensures the business integrity of the information and provides a stable anchor point for subsequent cross-interface tracking of specific objects (such as information about the same product on different pages).

[0036] Multi-dimensional visual semantics analysis: the model not only extracts literal information such as text and numerical values, but also deeply analyzes the business implications contained in the visual forms it carries. The significant visual features of deep analysis include but are not limited to: bold font, italic font, text color, strikethrough, underline, background color, font size, corner size, transparency, border style, etc. The model identifies these visual attributes to infer the importance and business meaning of the information. For example: the model can understand that the strikethrough on the price usually means "original price", while highlighted or specially colored text represents "promotion information" or "key tips". This decoding ability of visual language makes the extracted information go beyond the original data and contain rich business status and information level annotations.

[0037] In the process of handling interfaces containing multiple similar objects (such as product lists, SKU selectors, coupon lists), the model has the ability to bind scattered information to specific objects. This ability ensures that the extracted data is not only accurate but also has a clear attribution relationship, avoiding confusion and mismatch of information. This ability of association recognition is particularly important when dealing with product lists. When multiple product cards appear on a product list page, each card contains information such as product images, prices, titles, and promotion labels. The model needs to identify which price, image, and description belong to which product. The model distinguishes similar objects through descriptive features, such as "the product with the model wearing a white T-shirt, priced at ¥99" and "the product with the model wearing a black T-shirt, priced at ¥129."

[0038] Second stage: micro-precision enhancement based on function calling. To overcome the inherent limitations of large language models in pixel-level precision, the architecture introduces a function calling mechanism as a key second stage. After completing macro semantic understanding, the model can intelligently identify which information needs higher precision verification and actively call external specialized tool sets for confirmation and enhancement.

[0039] The workflow of this collaborative mode is as follows: the large language model performs macro semantic understanding on the interface screenshot, performs context-aware block division to form information groups, and analyzes the content and visual semantics of the information groups to achieve basic information extraction and generate a call request. Then the system executes tools such as optical character recognition (OCR) models, color pickers, or pixel-level visual models, and returns accurate results (such as accurate text content, hexadecimal color codes, and element pixel sizes) to the model for supplement and confirmation of the extracted information. The supplemented and confirmed information includes color values, accurate text content, and pixel-level visual features. The model finally integrates the information from both stages and outputs a fully structured data that combines macro semantic understanding and micro data precision. Through this two-stage architecture, the invention not only solves the problem of traditional automation technology failing to understand interface semantics, but also makes up for the shortcoming of relying solely on large language models that cannot guarantee absolute data accuracy.

[0040] Finally, each interface is converted into a three-layer information architecture consisting of "user interaction history records," "information group summaries," and "component specific information." The user interaction history records include interaction paths, interface summaries, and operation records, providing a macro view of the business process. The information group summaries include the overall description and block division of each information group in the interface, providing a meso view of the interface structure. The specific information content of the information group includes detailed information and visual features of each component in the information group, for providing a microscopic view of information details. This three-layer information architecture provides a solid foundation for multi-level, cross-interface information consistency analysis in subsequent steps.

[0041] S3, review information synthesis: based on structured data, using large language model to analyze user information demand, based on the parsed user information demand to establish cross-interface information association graph, and combined with the preset information consistency check point to investigate cross-interface information content, generate review file.

[0042] The innovation of this step is to build a multi-agent collaborative, hierarchical and progressive cognitive synthesis framework. This framework aims to simulate the complete mental model of experts conducting consistency review: first, understand the user's core intent and information demand from the macro process; second, based on the intent, build a cross-interface review hypothesis at the mesoscopic level; finally, package the hypothesis with the precise data at the microscopic level to form a structured analysis file. This process makes full use of the three-layer information architecture generated in the previous stage.

[0043] The first stage: user intent simulation based on user interaction history records. This stage focuses on answering "why review". The invention uses the first level of information architecture (user interaction history records) to deeply simulate the user's cognitive and decision-making process in the interaction process through an expert user agent (Expert User Agent). This agent analyzes the user's operation sequence, not only to record behavior, but also to infer the underlying goals and key information needs at different decision nodes.

[0044] The core function of this agent is that it can distinguish and identify information types that have a high impact on user decision-making (such as price, delivery time, discount conditions, etc.), giving priority and targeting based on the user's perspective for subsequent review tasks. This makes consistency detection no longer a indiscriminate full comparison, but rather an intelligent review focused on ensuring core user experience.

[0045] For example, taking the clothes shopping scenario as an example, the user's interaction history contains the following key nodes: browsing multiple product cards on the product list page → clicking on a white T-shirt to enter the product detail page → viewing product parameters and user reviews → selecting size and color → clicking the "Buy Now" button → entering the order confirmation page → viewing delivery information and coupons → clicking "Submit Order" to enter the payment page. By analyzing this operation sequence, the user's decision-making evolution process is inferred, and special attention is paid to the key turning points and information acquisition nodes in the user interaction path. And combined with the inference mechanism of differentiated user needs, it is judged which information has decision-making influence on specific types of users. This ability enables the agent to go beyond the surface information list and identify key information that is specific to the scene; and combined with the decision-making influence evaluation framework, the influence degree of the information is judged by analyzing the user's behavior changes before and after acquiring a certain type of information, for example, if the user sees a product marked "limited time 5 off" on the product list page and immediately clicks into the detail page, it means that the promotion information has high decision-making influence; if the user's stay time is significantly prolonged after viewing the return policy on the detail page, it means that the guarantee information has high experience influence.

[0046] The second stage: review task construction based on information group association. After understanding the user's intention, this stage focuses on answering "what to review" and "where to review". The present invention uses the second level of information architecture (information group overview) to build specific review tasks through a task planning agent.

[0047] The core capability of this agent is to build a cross-interface information association graph. It tracks the flow and evolution of "information groups" describing the same business object between different interfaces according to the user's actual interaction path, and associates them. Then, according to the preset five consistency check point system (covering numerical, text, visual, etc.), the agent automatically generates structured review suggestions (Analysis Proposal) for these associated information groups. Each suggestion is a specific, executable, and verified consistency check hypothesis. Specifically: According to the user's interaction history record, the complete operation sequence of the user between interfaces is extracted as the process skeleton of the cross-interface information association graph; at the same time, based on the structured information unit defining the specific business information contained in each interface as the semantic node library of the cross-interface information association graph; Based on the process skeleton and semantic node library, the user's operation path between interfaces is extracted, and the information groups involved in the path are located, forming an information-enhanced operation sequence; For example, in the clothes shopping scenario, the agent tracks the complete operation path of the user: the user browses the second commodity card on the commodity list page → clicks to enter the commodity detail page to view price information → selects the M code specification to view the corresponding price → adds the commodity to the shopping cart to view the commodity information on the shopping cart page → enters the order confirmation page to view the order amount → views the payable amount on the payment page.

[0048] The agent identifies all information groups related to the same commodity in this path and establishes a correlation relationship. For example, the commodity card on the commodity list page and the commodity information area on the commodity detail page have semantic correlation, describing the basic information of the same commodity; The price information on the commodity detail page and the specification price in the SKU selector have semantic correlation, describing the price of different specifications of the same commodity; The commodity information on the shopping cart page, the commodity list on the order confirmation page, and the order details on the payment page have both process and semantic correlation, describing the information presentation of the selected commodity at different transaction stages.

[0049] This graphed information organization provides a clear object positioning for subsequent expert analyzers, ensuring that the same object information across interfaces can be systematically identified and compared.

[0050] Through two-dimensional logical inference of the correlation between information groups, the correlation includes process correlation established according to the sequence of information groups in the user operation path and semantic correlation analyzed by the business content carried by the information groups to determine whether they point to the same business object or describe the same business attribute; For example, in the clothes shopping scenario, the business side may explicitly require the inspection of "commodity price", "delivery fee", "return policy", and other legal and regulatory and platform rule requirements that must be accurately displayed; the user side analysis identifies "coupon use conditions", "estimated arrival time", "inventory quantity", and other information that has a significant impact on user decision-making. After the agent integrates these two dimensions, the information types such as price information and delivery information that appear in both dimensions are determined as the core inspection objects, while the information types such as coupon use restrictions that are only concerned in the user dimension are determined as supplementary inspection objects.

[0051] All information groups are taken as nodes, and the inferred process correlation and semantic correlation are taken as edges to establish a cross-interface information correlation graph.

[0052] Subsequently, according to the preset information consistency checkpoints, structured review suggestions are automatically generated for these correlated information groups: Checkpoint one: numerical information contradiction, including two subtypes of numerical conflict and specification conflict.

[0053] The numerical conflict refers to the inconsistency of numerical information of a product displayed on different pages, including but not limited to: time limit numerical conflict (such as the list page showing "tomorrow" and the detail page showing "3-5 days to arrive"), price numerical conflict (such as the list page showing "¥99" and the detail page showing "¥109"), quantity numerical conflict (such as the detail page showing "in stock" and the order prompt showing "only 1 left"), and geographical location distance numerical conflict (such as the list page showing "3.2 kilometers away" and the map page showing "5.1 kilometers away"). In the clothes shopping scenario, if the product card on the product list page shows a promotional price of "¥99", but the user finds that the price is "¥109" after clicking into the product detail page, or the price changes to "¥119" after selecting the M size specification without any indication of the price difference between different specifications on the list page, it constitutes a price numerical conflict.

[0054] The specification conflict refers to the contradiction of product specification information in different display positions, including but not limited to: inconsistent units (such as the detail page showing "500g" and the parameter table showing "0.5kg", although the numerical values are equivalent, but the expression methods are inconsistent), and coexistence of vague description and accurate description (such as the list page showing "about 3 days to arrive" and the order page showing "expected to arrive on December 25", the time ranges of the two may not match). In the clothes shopping scenario, if the specification parameters on the product detail page show "fabric composition: 95% cotton", and the product description text mentions "100% pure cotton material", it constitutes a specification conflict.

[0055] Checkpoint two: contradictory information in the text, including three subtypes of using different texts to express the same semantics, contradictory texts before and after, and ambiguous texts.

[0056] Using different texts to express the same semantics refers to using different word descriptions for the same concept or the same function in different interfaces, which may cause user understanding difficulties or doubts. In the clothes shopping scenario, if the list page shows "7-day no reason return", the detail page shows "supports 7-day worry-free return and exchange", and the order confirmation page shows "return guarantee: one week return", although the semantics are the same, the differences in expression may make the user doubt whether the specific policy content is consistent.

[0057] The contradictory texts before and after include two cases of commitment invalidation contradiction and positive and negative expression difference. Commitment invalidation contradiction refers to the commitment or explanation made on the previous page being invalid or appearing opposite expression on the subsequent page. In the clothes shopping scenario, if the detail page promises "free shipping for orders over ¥99", but the user adds a product priced at ¥99 to the shopping cart, and the order confirmation page shows "shipping fee ¥10", it constitutes a commitment invalidation contradiction. Positive and negative expression difference refers to the same information appearing opposite expressions of affirmation and negation on different pages, such as the detail page marking "does not support 7-day no reason return", and the order confirmation page showing "enjoy worry-free return service".

[0058] The ambiguity of the copy refers to the description of the copy with multiple understanding ways, which may lead to user's wrong expectation. In the clothes shopping scene, if the product detail page describes "free shipping" but does not specify whether there are regional or amount restrictions, the user may understand it as unconditional free shipping, while in fact it may be "free shipping over 99 yuan" or "free shipping in some areas only". Such ambiguous description is easy to cause user dissatisfaction.

[0059] Checkpoint three: visual style contradiction, including two subtypes of sibling information style difference and different level information style similarity.

[0060] The sibling information style difference refers to the use of different visual styles for interface elements with the same content or the same function, which breaks the consistency rules of the interface. In the clothes shopping scene, if in the product list page, the first product's promotional price is displayed in red 20-point bold font, and the second product's promotional price is displayed in orange 18-point normal font, although both are promotional price information, the difference in visual style may make the user mistakenly think that the two products have different promotional strength or importance.

[0061] For example, if the "buy now" button on the product detail page uses a red background with white text, and the "go to checkout" button on the shopping cart page uses an orange background with black text, although both are main operation buttons, the difference in style will weaken the user's coherent awareness of the operation process.

[0062] The different level information style similarity refers to the use of the same or similar visual styles for information of different levels or different states, which leads to unclear information level distinction. In the clothes shopping scene, if in the product detail page, the original price of the product (the reference price which has expired) uses gray 14-point font, and the secondary explanation text in the product parameters (such as "subject to actual goods") also uses gray 14-point font, it will be difficult for the user to distinguish the importance difference between price information and explanation information.

[0063] For example, if the promotional label "limited time 5 off" and the ordinary product attribute label "pure cotton" use the same visual style (same background color, font size and border style), the promotional information will be weakened, and the user's perception of the discount information will be affected.

[0064] Checkpoint four: information loss refers to the disappearance of key information with interface jump in the user task flow, which leads to the user's inability to obtain the information in the subsequent decision-making link. The key information includes but is not limited to at least one of the following: coupon information, time-limited activity information, delivery benefit information, and delivery status information.

[0065] In the clothes shopping scenario, typical information loss cases include: 1. Coupon loss: the user sees a certain product marked "available for full-reduced coupon" on the product list page, and after clicking into the product detail page, the coupon information disappears, and the user cannot understand the specific coupon use conditions and the discount strength; 2. Time-limited activity loss: the user sees the activity countdown "limited time 5 times, remaining 2 hours" on the product detail page, and after joining the shopping cart, the countdown information disappears on the shopping cart page, the user cannot perceive the activity urgency, and may miss the preferential period; 3. Delivery benefit loss: the user sees the "member exclusive free shipping" mark on the product detail page, but the benefit description disappears on the order confirmation page, resulting in the user being unable to confirm whether to enjoy the free shipping benefit before payment; 4. Delivery status loss: the user sees the "present stock fast delivery, expected tomorrow" delivery promise on the product detail page, but only "standard delivery" is displayed on the order confirmation page without specifying the specific time limit, and the user cannot verify whether the time limit promise is valid. The core of the information loss problem is that the user makes a decision (such as selecting the product, joining the shopping cart) based on the information on the previous page, but cannot confirm the information again in the subsequent link, resulting in the loss of decision basis and the reduction of user trust.

[0066] Checkpoint five: information increase refers to the sudden appearance of important limiting conditions that have not been mentioned before the key decision node of the user's task flow, affecting the effectiveness of the user's decision. The key decision node includes but is not limited to the order confirmation page (the bill page) or the payment page, and the important limiting condition includes but is not limited to at least one of the product delivery restrictions, group purchase use restrictions, and benefit enjoyment restrictions.

[0067] In the clothes shopping scenario, typical information increase cases include: 1. Delivery restriction increase: the user does not see any delivery restriction information on the product list page and the detail page, and normally selects the product and adds it to the shopping cart, but suddenly prompts "the product does not support delivery to your address" or "remote area needs to pay an additional ¥20 freight" on the order confirmation page. At this time, the user has invested a lot of time cost, and this sudden restriction will seriously affect the user experience; 2. Increase of discount restrictions: the user sees the "reduce 20 yuan for 99 yuan" discount information on the product detail page, selects two products with a total price of ¥118, but suddenly prompts "this coupon is only for purchasing a single product" or "this discount is not available with member discounts" when using the discount on the order confirmation page. This type of previously unmentioned usage restriction causes the user's discount expectations to fall through; 3. Payment restriction increase: the user does not see any payment method restriction information on the previous page, but suddenly discovers "the product does not support cash on delivery" or "additional 3% handling fee for credit card payment" on the payment page. The sudden appearance of such payment restrictions may cause the user to abandon the purchase. The core of the information increase problem is that the user does not receive complete decision-making information in the early stages of the decision-making path, and only discovers important restriction conditions at the critical node just before completing the transaction. At this time, the user has invested a lot of time and psychological cost, and the sudden appearance of restrictions will cause strong frustration and damage to trust.

[0068] The third stage: analysis case packaging based on accurate content. Finally, this stage focuses on "how to review", that is, to provide complete input for subsequent in-depth analysis. The system integrates the "review suggestions" generated in the previous stage with the third level of information architecture (specific information content), and automatically packages it into an independent review file (Review File) through a script.

[0069] Each review file is a complete analysis case, which not only contains the review instructions to be executed, but also encapsulates all the context required to execute the instructions: including macro-intentions derived from user interaction history records, meso-review guidelines generated by task planning agents, and all micro-detail data extracted from related "information groups".

[0070] Through this cognitive synthesis framework, the invention systematically transforms raw and scattered multi-interface information into a series of structured analysis tasks with clear goals, complete data, and rich context, laying a solid foundation for high-quality, automated reasoning analysis in step S4.

[0071] S4, based on the generated review file, different analysis agents are used according to the nature of the review task, and the thought chain reasoning mode is used for parallel analysis, outputting the reasoning process and analysis results of the information consistency problem.

[0072] This step, as the core of the system's automated reasoning and judgment, is responsible for executing the structured review tasks generated in the previous step. Its innovation lies in efficient task distribution and interpretable reasoning mechanisms.

[0073] This step receives the "review files" from step S3 and intelligently distributes them to analysis agents (Analysis Agent) with corresponding expertise according to the nature of the review tasks (such as numerical contradictions, visual style contradictions, etc.), for parallel processing, thereby achieving optimal utilization of resources and specialization of analysis.

[0074] The core working mode of each analysis agent is Chain-of-Thought (CoT) reasoning. This mechanism forces the agent to first generate a complete and clear logical reasoning process before giving the final judgment. This "process-first" output mode ensures that every problem discovered has transparent explainability and traceability, allowing human auditors to clearly verify the basis of their judgments. Finally, the agent outputs a structured problem report, which not only contains problem details but also includes a complete reasoning chain, realizing an intelligent closed loop from "finding problems" to "arguing problems".

[0075] S5, Result filtering and output: Constructing keyword-based heuristic rules to preliminarily filter analysis results, and then using search enhancement generation technology-based filters for screening, outputting the final problem report, and completing the evaluation.

[0076] To ensure the high fidelity and high credibility of the final output report, this step introduces an innovative, self-optimizing dual-filter quality assurance framework.

[0077] First, efficient heuristic filtering. The system first uses keyword-based fast heuristic rules, including but not limited to: "inconsistency", "contradiction", "conflict", "difference", "error", "problem", "risk", etc. Through keyword detection, the system can identify model outputs that do not contain problem report keywords. This stage aims to efficiently filter out obviously invalid or format error outputs as the first quality threshold, reducing the pressure of subsequent fine filtering.

[0078] Second round: intelligent filtering based on retrieval enhancement. The core innovation of this step is an intelligent filtering agent based on a retrieval-augmented generation (RAG) technology. The invention builds and maintains a dynamic "low-quality output knowledge base", the initial data of which comes from invalid outputs identified in the rule filtering stage and low-quality problem reports found through manual review. All outputs are archived in the knowledge base after manual review to ensure the accuracy and representativeness of the knowledge base content. The knowledge base is managed using a large language model, which summarizes and classifies low-quality outputs. When processing low-quality outputs, the system removes specific business information (such as product names, specific price values, page IDs, etc.), retains the abstract features of the problem pattern, and then performs text embedding processing. This abstraction process allows the knowledge base to identify common patterns in low-quality outputs without being limited by specific business scenarios.

[0079] Typical low-quality output patterns include but are not limited to: 1. Over-speculation: the model makes excessive speculations in the absence of sufficient information, reporting problems that do not exist in reality; 2. Misunderstanding of business logic: the model lacks domain knowledge and misjudges normal business logic as inconsistent problems (such as believing that different specifications of a product should have the same price); 3. Fuzzy positioning: the problem report lacks precise page and information group positioning, making it impossible to reproduce the problem; 4. Inadequate reasoning: the reasoning process lacks logical support, and the basis for problem judgment is unclear; 5. Duplicate reporting: repeated reporting of the same problem from different angles increases the burden on manual review.

[0080] When an analysis result enters this filtering stage, the filtering agent abstracts the analysis result to be detected, removes specific business information, extracts problem pattern features, and generates a query vector. Next, the system performs vector similarity retrieval in the low-quality output knowledge base to find a number of historical low-quality output cases that are most similar to the result to be detected; Then, the filtering agent combines the retrieved similar cases and the current analysis result to determine whether the current result belongs to the low-quality output patterns identified in the knowledge base. If the similarity exceeds a preset threshold (set to 0.8 in this embodiment), it is determined to be a low-quality output and is filtered out; Finally, the system retains high-confidence problem reports that have not been filtered out and outputs them as the final evaluation results.

[0081] The low-quality output knowledge base supports continuous iterative optimization. The sources of knowledge base updates include: 1. Automatic accumulation: invalid outputs identified by rule filtering are automatically archived to the knowledge base; 2. Manual labeling: when reviewing the filtering results, testers can manually add newly discovered low-quality output patterns to the knowledge base; 3. Business customization: testers can manually access specific non-concerned content to avoid repeated reporting on problems that repeatedly occur but have no optimization plan in the short term.

[0082] The self-optimization feature of this framework is reflected in the fact that the knowledge base can be continuously iterated through manual feedback and the addition of new samples. This enables the filtering system to continuously learn and gradually adapt to the business logic of a specific application, thereby continuously improving the accuracy of filtering and ensuring that the final delivery to developers and testers is a high-confidence, high-value problem report.

[0083] So far, the five core steps of the information consistency evaluation method for multi-interface mobile applications provided by the present application have been completed. This method collects test links through GUI proxy technology, performs accurate information extraction based on multi-modal large language models and tool calls, completes user demand analysis and review task planning through multi-agent collaboration, performs deep consistency analysis based on thought chain reasoning, and finally outputs high-quality problem reports through intelligent filtering mechanism. The entire process realizes the automated identification and evaluation of mobile application information consistency problems, providing effective technical support for application quality optimization.

[0084] Please refer to Figure 3 , based on the above method, the present application also provides a multi-interface mobile application evaluation device based on a multi-modal model, the multi-interface mobile application evaluation system is used to realize the multi-interface mobile application evaluation method, comprising: a test item collection module for executing the interaction of a specified test link through a graphical user interface agent, collecting interface screenshots and user interaction history records in the interaction process; an information extraction module based on an information extractor combining a large language model and tool calls to extract structured information from the interface screenshots, generating structured data containing multiple information groups; a review information synthesis module for analyzing user information needs based on structured data using a large language model, establishing a cross-interface information association graph based on the parsed user information needs, and checking cross-interface information content in combination with pre-set information consistency checkpoints to generate a review file; an information analysis module for using different analysis agents according to the nature of the review tasks based on the generated review file, and using a thought chain reasoning mode for parallel analysis, outputting the reasoning process and analysis results of the information consistency problems; The result filtering module is configured to construct keyword-based heuristic rules to preliminarily filter the analysis results, and then filter the results through a filter based on a search enhancement generation technology, and output a final problem report, thereby completing the evaluation.

[0085] It should be noted that the multi-modal model-based multi-interface mobile application evaluation method and the multi-modal model-based multi-interface mobile application evaluation device provided in the above embodiments belong to the same concept, and the specific implementation process is described in the multi-modal model-based multi-interface mobile application evaluation method, which will not be repeated here.

[0086] It should be noted that in this document, the terms "comprise", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or terminal. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of another identical element in the process, method, article or terminal that includes the element.

[0087] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware (such as a processor, a controller, etc.) to complete, and the program can be stored in a computer-readable storage medium. The program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a magnetic disc, an optical disc, etc.

[0088] It should be understood that the application of the present application is not limited to the above examples, and those skilled in the art can improve or change it according to the above description, and all these improvements and changes should belong to the protection scope of the appended claims of the present application.

Claims

1. A method for evaluating multi-interface mobile applications based on a multimodal model, characterized in that, Includes the following steps: Test item collection: The interaction of the specified test link is executed through the graphical user interface agent, and the interface screenshots and user interaction history are collected during the interaction process; Information Extraction: An information extractor based on a combination of large language model and tool calls performs structured information extraction on interface screenshots, generating structured data containing multiple information groups; Review information synthesis: Based on structured data, large language models are used to analyze user information needs. Based on the parsed user information needs, a cross-interface information association graph is established. Combined with preset information consistency checkpoints, cross-interface information content is checked and a review document is generated. Information Analysis: Based on the generated review documents, different analysis agents are used according to the nature of the review task, and parallel analysis is performed using a thought chain reasoning mode to output the reasoning process and analysis results of the information consistency problem; Results filtering and output: Construct keyword-based heuristic rules to initially screen the analysis results, then use a filter based on search enhancement generation technology to further filter the results, and finally output a problem report to complete the evaluation.

2. The multi-interface mobile application evaluation method based on a multimodal model according to claim 1, characterized in that, The user interaction history recorded in the test item collection includes: Interface overview information, a simplified natural language description of each interface generated by the graphical user interface agent. The simplified natural language description extracts the key functional areas and main information content of the interface, providing a basis for the graphical user interface agent's subsequent interaction decisions. Interaction logs are used to record the specific actions performed by the graphical user interface agent on each interface. Each interaction log contains at least the action type and the action object.

3. The multi-interface mobile application evaluation method based on a multimodal model according to claim 1, characterized in that, In the information extraction process, the information extractor based on a combination of a large language model and tool calls performs structured information extraction from the interface screenshot, including: The large language model performs macro-semantic understanding of the interface screenshot, performs context-aware block division to form information groups, and analyzes the content and visual semantics of the information groups to extract basic information. Each information group contains a unique identifier, block location, information overview and a list of its components. Next, the large language model calls an external toolset to verify and enhance the accuracy of the extracted basic information, thus completing the extraction of structured information.

4. The multi-interface mobile application evaluation method based on a multimodal model according to claim 3, characterized in that, The external toolset includes one or more of optical character recognition tools, color picker tools, or pixel-level visual analysis tools, which are used to supplement and confirm the extracted information; wherein, the supplemented and confirmed information includes color values, precise text content, and pixel-level visual features.

5. The multi-interface mobile application evaluation method based on a multimodal model according to claim 4, characterized in that, In information extraction, the structured data organization architecture consists of three layers: user interaction history, information group overview, and specific information content of the information group. The user interaction history includes interaction path, interface overview and operation record, which is used to provide a macro view of the business process; The information group overview includes an overall description and block division of each information group in each interface, outlining the content, visual features and location features of each information group, and is used to provide a meso view of the interface structure. The specific information content of the information group includes detailed information and significant visual features of each component within the information group, used to provide a microscopic view of information details.

6. The multi-interface mobile application evaluation method based on a multimodal model according to claim 1, characterized in that, In the process of reviewing information synthesis, the construction of the cross-interface information association map includes: Based on user interaction history, the complete operation sequence of the user between interfaces is extracted as the process skeleton of the cross-interface information association graph; at the same time, the structured information units based on the specific business information contained in each interface are used as the semantic node library of the cross-interface information association graph. Based on the process skeleton and semantic node library, the operation path of the user between interfaces is extracted, and the information groups involved in the path are located to form an information-enhanced operation sequence. The association between information groups is inferred through two-dimensional logic. The association includes establishing a process association with a sequential relationship based on the sequential position of the information groups on the user's operation path, and a semantic association that analyzes the business content carried by the information groups to determine whether they point to the same business object or describe the same business attribute. By treating all information groups as nodes and the inferred process and semantic relationships as edges, a cross-interface information association graph is established.

7. The multi-interface mobile application evaluation method based on a multimodal model according to claim 6, characterized in that, In the process of reviewing information synthesis, the preset information consistency checkpoints include: numerical information contradictions, textual information contradictions, visual style contradictions, information loss, and information additions; The numerical information contradictions include numerical conflicts and specification conflicts; wherein, the numerical conflicts refer to inconsistencies in the numerical information of the product displayed on different interfaces, including numerical contradictions in timeliness, price, quantity, or geographical distance; the specification conflicts refer to contradictions in the product specification information displayed in different locations, including inconsistent units or the coexistence of vague descriptions and precise descriptions. The contradictions in the text include using different text to express the same meaning, contradictions within the text, and ambiguity in the text. The visual style contradictions include different styles for information at the same level and similar styles for information at different levels; wherein, different styles for information at the same level means that interface elements with the same content or the same function use different visual styles; and similar styles for information at different levels means that information at different levels or in different states uses the same or similar visual styles, resulting in unclear distinction between information levels. The information loss refers to the disappearance of key information when the page redirects; The information is added when new and important information suddenly appears later in the process.

8. The multi-interface mobile application evaluation method based on a multimodal model according to claim 1, characterized in that, In the result filtering and output, the filtering performed using a filter based on retrieval enhancement generation technology includes: Receive the analysis results after preliminary screening, and abstract the analysis results to remove specific business information in order to extract problem pattern features and generate query vectors; In a pre-built low-quality output knowledge base, vector similarity retrieval is used to find several historical low-quality output cases that are most similar to the query vector. Based on the retrieved similar cases, determine whether the current analysis results belong to a known low-quality output pattern; If the similarity between the current analysis result and a known low-quality output pattern exceeds a preset threshold, it is filtered out; otherwise, the analysis result is retained as a high-confidence output, resulting in the final problem report.

9. The multi-interface mobile application evaluation method based on a multimodal model according to claim 8, characterized in that, The construction process of the low-quality output knowledge base includes: Collect reports of invalid outputs identified during the initial screening and settlement process, as well as reports of low-quality issues discovered during manual review; The collected low-quality problem reports are summarized and categorized using a large language model. In this process, specific business information in the cases is removed, while the abstract features of their problem patterns are retained. The abstracted problem pattern is then embedded into text and stored in a knowledge base for retrieval.

10. A multi-interface mobile application evaluation device based on a multimodal model, characterized in that, The multi-interface mobile application evaluation system is used to implement the multi-interface mobile application evaluation method according to any one of claims 1-9, including: The test item collection module is used to execute the interaction of the specified test link through the graphical user interface agent, and collect the interface screenshots and user interaction history during the interaction process; The information extraction module, based on an information extractor that combines a large language model with tool calls, extracts structured information from screenshots of the interface and generates structured data containing multiple information groups. The review information synthesis module is used to analyze user information needs based on structured data using a large language model, establish a cross-interface information association graph based on the parsed user information needs, and check the cross-interface information content in conjunction with preset information consistency checkpoints to generate review documents. The information analysis module is used to analyze generated review documents using different analysis agents according to the nature of the review task, and adopts a thinking chain reasoning mode for parallel analysis, outputting the reasoning process and analysis results of information consistency issues; The results filtering module is used to construct keyword-based heuristic rules to initially screen the analysis results, and then use filters based on search enhancement generation technology to further filter the results, outputting the final problem report and completing the evaluation.

Citation Information

Patent Citations

  • User interface defect detection method and device, equipment and medium

    CN119883888A

  • Application program test mode and device, storage medium and program product

    CN120336183A

  • Automatic user interface assertion method and system based on multi-modal large model and program product

    CN120371694A

  • Product UI function identification and defect detection method based on multi-modal large model

    CN118467349A

  • Automatic GUI (Graphical User Interface) testing method and system based on multi-modal model

    CN119759771A